WordPress Robots.txt Resources

In particular, I use the following to prevent the crawling of “test” sites:

User-agent: *
Disallow: /

Preventing your site from being indexed, the right way! – Yoast

The BETTER WAY to make sure Google does not list my “test” sites. Instead of the above-mentioned Robots.txt method, Yoast recommends adding this bit of coolness to HTAccess:

Header set X-Robots-Tag "noindex, nofollow"

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *