Yes, that’s right! If you are not very careful with the robots.txt file, you could be blocking
web spiders from crawling content you do want to be indexed. Most reputable web
spiders will obey your instructions within robots.txt. So, be very careful when you use
and edit the robots.txt file. Also note that Google Webmaster Tools provides a
robots.txt checker, so you can enter a URL and confirm whether a given URL will be
crawled.
web spiders from crawling content you do want to be indexed. Most reputable web
spiders will obey your instructions within robots.txt. So, be very careful when you use
and edit the robots.txt file. Also note that Google Webmaster Tools provides a
robots.txt checker, so you can enter a URL and confirm whether a given URL will be
crawled.