Some search engines, notably Yahoo! and altavista, only allow you to submit one Web page for rating. They then examine that page and index the keywords in it before following up the Web links it mentions. Ideally, this crawler page contains links to all the key pages on the site that you wish to be indexed. This should be done without any text. It only exists to help the search engines, so it doesn't have to be viewable. To prevent the crawler page from appearing in the search engine results, put a redirect on it so the searcher is passed straight to your home page.
If your site is extensive, link the master crawler page to sub-crawler pages, which list other working pages.
The maximum indexable content per page is limited to 100K. Any excess cannot be indexed.
If you are using framed pages, make sure they don't exceed the limit by getting your crawlers to test the links between the <noframes> and </noframes> sections in the master page that generates framed pages.
Was this article helpful?