WordPress Robots.txt Resources

In particular, I use the following to prevent the crawling of “test” sites:

[code]User-agent: *
Disallow: /[/code]

Preventing your site from being indexed, the right way! – Yoast

The BETTER WAY to make sure Google does not list my “test” sites. Instead of the above-mentioned Robots.txt method, Yoast recommends adding this bit of coolness to HTAccess:

[code]Header set X-Robots-Tag "noindex, nofollow"[/code]

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top