PDA

View Full Version : robots.txt do you need to have one?


Klove
2004-09-29, 04:04 PM
do you need to have one, or can you survive without one?

DangerDave
2004-09-29, 05:27 PM
Klove you don't NEED to have one.. but if you have /dirs/ you want robots to stay out of... then yes... it's worth having one.

DD

Klove
2004-09-29, 05:27 PM
Thank you, I want it to sniff my entire domain! ;P