WordPress Robots.txt Resources
- Better Robots.txt Rules for WordPress – Perishable Press
- WordPress robots.txt Example – Yoast
- Robotstxt.org
- Disallow Robots Using Robots.txt – David Walsh
- Block or remove pages using a robots.txt file – Google
In particular, I use the following to prevent the crawling of “test” sites:
User-agent: * Disallow: /
Preventing your site from being indexed, the right way! – Yoast
The BETTER WAY to make sure Google does not list my “test” sites. Instead of the above-mentioned Robots.txt method, Yoast recommends adding this bit of coolness to HTAccess:
Header set X-Robots-Tag "noindex, nofollow"
