What the f.....?

ecchi · 2013-08-17, 06:33 AM

I have a domain http://www.torture.jp.net/ and checking my logs I found a hit on the index page from http://www.torture.jp.net/robots.txt. Which would mean someone checked robots.txt before looking at the main page. Only problem with this is that I do NOT have a robots.txt file on that domain!!! In other words, the hit came from a page that does not exist!

Is it possible that there is a "hidden" robots.txt file somewhere that I did not create, and does not show up on my FTP client? Because I am not only getting this strange referrer, but also Google refuses to index this domain, and I have got no hits from any search engine to this domain, ever. But all three things would make sense if there was a robots.txt file somewhere with a "no spider" rule set.

2013-08-17, 06:33 AM	#1
ecchi Banned Join Date: Oct 2003 Location: About to be evicted!!!! Posts: 4,082	What the f.....? I have a domain http://www.torture.jp.net/ and checking my logs I found a hit on the index page from http://www.torture.jp.net/robots.txt. Which would mean someone checked robots.txt before looking at the main page. Only problem with this is that I do NOT have a robots.txt file on that domain!!! In other words, the hit came from a page that does not exist! Is it possible that there is a "hidden" robots.txt file somewhere that I did not create, and does not show up on my FTP client? Because I am not only getting this strange referrer, but also Google refuses to index this domain, and I have got no hits from any search engine to this domain, ever. But all three things would make sense if there was a robots.txt file somewhere with a "no spider" rule set.