Blocking Unknown Robots/Spiders?
 

So you guys do care on what scans your domains?

if so what techniques do you guys use?

I came across this blog post:

http://michael.langley.id.au/blog/posts/28

Dunno if it is the better way to do this kind of thing...

I'm interested in this because I check the stats on one of mine domains and got a lot of unknown robots...

http://dl.dropboxusercontent.com/u/1...otsspiders.jpg

Why would I care?

You can set the rules in robots.txt in your root of a domain but most both don't obey those.

You can deny them in .htaccess (or robots.txt but as pc said, only the useful/nice bots are likely to comply) or use something like http://www.spambotsecurity.com/zbblock.php to stop them ...