Google:
Googlebot- Crawls web pages (it’s the most important of the bunch)
Googlebot-Mobile- Crawls pages specifically designed for mobile devices
Googlebot-Image Crawls images for inclusion in image search results
Mediapartners-Google Crawls AdSense content
AdsBot-Google Crawls AdWords landing pages to measure their quality
Yahoo!:
Slurp Crawls web pages
Yahoo-MMAudVid Crawls video files
Yahoo-MMCrawler Crawls images
Yahoo-MMAudVid Crawls video files
Yahoo-MMCrawler Crawls images
Bing:
MSNBot- Crawls web pages
MSNBot-Media Crawls media files
MSNBot-News Crawls news feeds
Thousands of crawlers are operating on the Internet. It would make no sense to pay
attention to all of them. Depending on your site’s content and the region you are targeting, you may need to pay more attention to other crawlers.
attention to all of them. Depending on your site’s content and the region you are targeting, you may need to pay more attention to other crawlers.
Not all crawlers are created equal. Some crawlers crawl your web pages, whereas others
crawl your images, news feeds, sound files, video files, and so forth.
crawl your images, news feeds, sound files, video files, and so forth.