Crawler Pages

Some search engines, notably Yahoo! and altavista, only allow you to submit one Web page for rating. They then examine that page and index the keywords in it before following up the Web links it mentions. Ideally, this crawler page contains links to all the key pages on the site that you wish to be indexed. This should be done without any text. It only exists to help the search engines, so it doesn't have to be viewable. To prevent the crawler page from appearing in the search engine results, put a redirect on it so the searcher is passed straight to your home page.

If your site is extensive, link the master crawler page to sub-crawler pages, which list other working pages.

The maximum indexable content per page is limited to 100K. Any excess cannot be indexed.

If you are using framed pages, make sure they don't exceed the limit by getting your crawlers to test the links between the <noframes> and </noframes> sections in the master page that generates framed pages.

Search Engine Optimization Overview

Search Engine Optimization Overview

This is the 2nd volume of a 9 volume series called the Webmasters Toolbox package. Search engines are the number one way that internet users find websites. In most cases, a listing in a search engine is free. So, it's no surprise that Search Engine Optimization SEO is often the first priority when marketing a website.

Get My Free Ebook

Post a comment