Not everyone has access to their web server, but they still want to have control over how crawlers
behave on their web site. If you’re one of those, you can still control the crawlers that come to your
site. Instead of using the robots.txt file, you use a robots meta tag to make your preferences known
to the crawlers.
The robots meta tag is a small piece of HTML code that is inserted into the <HEAD> tag of your
web site and it works generally in the same manner that the robots.txt file does. You include your
instructions for crawlers inside the tags. The following example shows you how your robots meta
tag might look:
<html>
<head>
<meta name=”robots” content=”noindex, nofollow”>
<meta name=”description” content=”page description.”>
<title>
Web Site Title
</title>
</head>
<body>
This bit of HTML tells crawlers not to index the content on the site and not to follow the links on
the site. Of course, that might not be exactly what you had in mind. You can also use several other
robots meta tags for combinations of following, not following, indexing, and not indexing:
<meta name=”robots” content=”index,follow”>
<meta name=”robots” content=”noindex,follow”>
behave on their web site. If you’re one of those, you can still control the crawlers that come to your
site. Instead of using the robots.txt file, you use a robots meta tag to make your preferences known
to the crawlers.
The robots meta tag is a small piece of HTML code that is inserted into the <HEAD> tag of your
web site and it works generally in the same manner that the robots.txt file does. You include your
instructions for crawlers inside the tags. The following example shows you how your robots meta
tag might look:
<html>
<head>
<meta name=”robots” content=”noindex, nofollow”>
<meta name=”description” content=”page description.”>
<title>
Web Site Title
</title>
</head>
<body>
This bit of HTML tells crawlers not to index the content on the site and not to follow the links on
the site. Of course, that might not be exactly what you had in mind. You can also use several other
robots meta tags for combinations of following, not following, indexing, and not indexing:
<meta name=”robots” content=”index,follow”>
<meta name=”robots” content=”noindex,follow”>