robot.txt file

[BigVikingAl]

Can someone explain the value of robot.txt. Is it important?
I have created a robot.txt file and uploaded it into the root of my domain but when I do a search for it using one of the many tools available I get an error message:

robots.txt file does not appear to be valid

and the text points to my 404 page. What am I doing wrong? Should I be concerned?

[Mutant]

Make sure it's robots.txt not robot.txt

http://googleblog.blogspot.com/2007/...es-access.html

[johnnybg]

robots.txt tells search engines what NOT to index on your site. It's useful to "hide" your site administration consoles (for example) from SEs.

What tools do you use to search for the file? And why?

[BigVikingAl]

I used the google tool and another website that I didn't bookmark.
Both showed an error message of:

Success (200)

robots.txt file does not appear to be valid

Then it shows the default text of my robots.txt file but it shows the text on my 404 page not my blank robots.txt page.

[johnnybg]

you should double check if your robots.txt file is properly uploaded in domain root (that you probably done already)

Do you have any redirections in .htaccess file?

According to the documentation, blank robots.txt file is equal to allowing all bots to spider everything on your site.

[BigVikingAl]

It is in my root directory. Yes I have a re-direct in my .htaccess file to a 404 page. Do I need to add something so the bots can read my blank robots.txt file?

[johnnybg]

if you have something like this in .htaccess

ErrorDocument 404 http://www.yourdmain.com/404page.html

then you don't need to change it.

When you upload blank robots.txt to your domain root you should be able open http://yourdomain.com/robots.txt in bowser and get a blank page. If that's not the case you are probably redirecting somewhere.

Maybe it's best to post contents of your .htaccess just to be sure something is not hiding in there.

[BigVikingAl]

Tjid is all I have in my .htaccess file:

ErrorDocument 404 http://www.stars4porn.com/missing.html
AddType application/x-httpd-php .html

When I type in: http://www.stars4porn.com/robots.txt I get my empy (almost) robots.txt file:

###############################
#
#
User-agent: *
#
# list folders robots are not allowed to index
#
Disallow:
Disallow:
Disallow:
Disallow:
#
# list specific files robots are not allowed to index
#
Disallow:
Disallow:
#
#
###############################


Everything seems to be working fine my problem is only with google:
When I use their robots.txt tool I get:

robots.txt file does not appear to be valid

Then it shows the default text of my robots.txt file but it shows the text on my 404 page not my blank (almost) robots.txt page.

[cd34]

http://validator.czweb.org/robots-txt.php

while that finds the robots.txt file valid, I'm not quite sure if google might have a problem with the multiple blank disallows.

try the following:

User-agent: *
Disallow:

Other than that, make sure you upload in ascii, though, I saw no indication that you had any stray ^M's in there.

[JK]

I would also change this line

ErrorDocument 404 http://www.stars4porn.com/missing.html

to
ErrorDocument 404 /missing.html

as you're currently sending Google and all other search engines a '302 Found' message, not '404 Not Found'.

[BigVikingAl]

Not following JK. This is my 404 page and it is where I want to send my 404 traffic. Where does:

ErrorDocument 404 /missing.html

send them?

[JK]

It will send visitors to the same page.

Because you have the http://www..., the server is not returning the status code "404 Not Found", which can cause problems.

[BigVikingAl]

Just tried it JK but when i mistyped in one of my sites I didn't go to my 404 page. Just a google page not found page.

[JK]

Sorry, I should have mentioned it will only work on the domain that hosts the missing.html file. If you have multiple domains sending 404 traffic to that page then you will either need to mirror the page on each domain, or keep it as it was.

Is your robots.txt now validating with Google?

[BigVikingAl]

Now working on main domain and 2 sub-domains. Still not on 3 subs. Now get google 404 error - missing page