The Web Robots Pages
Web Robots (also known as Web Wanderers, Crawlers, or Spiders),
are programs that traverse the Web automatically. Search engines such
as Google use them to index the web
content, spammers use them to scan for email addresses, and they have
many other uses.
On this site you can learn more about web robots.