in your root .htaccess file add the following
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^Amazon.CloudFront$
RewriteRule ^robots\.txt$ robots-cdn.txt
And then create a separate robots-cdn.txt:
User-agent: *
Disallow: /
When accessed through via http://cdn.domain.com/robots.txt will
return the contents of the robots-cdn.txt file... otherwise the rewrite won't kick in
and the true robots.txt will kick in.
advanced seo interview questions
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^Amazon.CloudFront$
RewriteRule ^robots\.txt$ robots-cdn.txt
And then create a separate robots-cdn.txt:
User-agent: *
Disallow: /
When accessed through via http://cdn.domain.com/robots.txt will
return the contents of the robots-cdn.txt file... otherwise the rewrite won't kick in
and the true robots.txt will kick in.
advanced seo interview questions