Blogging Roller

Dave Johnson on open web technologies, social software and software development

Scooter.

You probably did not notice this, but my site was getting a tremendous number of hits this week. The reason is Scooter. Scooter is the AltaVista search robot. Scooter is persistent but not very smart. Scooter is not content to index a page, follow the links, and then index the linked pages. Scooter tries to guess the URLs to pages on your site and does a terrible job at the guessing game. If Scooter gets thousands of 404 PAGE NOT FOUND errors in one day, Scooter does not care. He just keep on guessing. According to Seach Engine World, he may be here for several months:

Search Engine World The Altavista search engine starts by spidering your entire site with its spider Scooter. Although lately Scooter hasn't been scooting too well. Scooter may take up to three months to spider and index your entire site (if it is going to crawl your site at all). It normally spiders about 2-10 pages per site in any week. Sometimes Scooter needs a good swift kick to get it to index certain pages.

Yep, I would like to give Scooter a good swift kick. Scooter obeys a robots.txt, if you have one. I'm hoping that the following addition to my robots.txt file will make him go away.

User-agent: Scooter/3.3
Disallow: /

User-agent: Scooter/3.2.SF0
Disallow: /
Dave Johnson in General • 🕒 02:02AM Jan 31, 2003
Tags: General
Comments:

Post a Comment:
  • HTML Syntax: NOT allowed