robot.txt

[Mike-mijen]

Does any one use robot.txt files on their free sites. I was just wondering. I am thinking about doing this myself.
Peace Mike|peace|

[stuveltje]

Does any one use robot.txt files on their free sites. I was just wondering. I am thinking about doing this myself.
Peace Mike|peace|yes and since i use that one, i am much better found and got way much hits:D

[LeRoy]

No I haven't used that at all on freesites or any of my sites. I have some decent/fair Google placements so I'm thinking I should add this one to my "To Do"

On freesites do you just put the text file in the same directory as the free site? Or do you just add it to the root of the domain?

[Mike-mijen]

Good question. I am not sure. I think it would be better if put in with the free site itself. But I am not knowledge on this one. I need to look deeper for this answer on the web.

[nate]

Im not sure there is a reason to unless you don't want them indexed. I'm not even sure you need a link to it on your main site as SEs will pick it up from the link lists/.

[Maj. Stress]

The robots.txt should go in the root of the domain. It's a good idea to use one even if you want the entire domain indexed. Some overlooked uses for robots.txt are not indexing anything in your cgi-bin folder or 2257 pages.

[rooty]

Im not using this method yet as im not building freesites at the rate most people do but this sounds like a reasonable way of doing it maybe

adding this to your robots.txt


Sitemap: http://www.urdomain.tld/sitemap.xml

User-agent: *
Disallow: /freesites/porn/freesite-mirror.html
Disallow: /freesites/porn/freesite-mirror2.html


then adding the url's you would like to be indexed to the sitemap file

[Wazza]

Be very careful in your use of robots.txt - if you're trying to limit the number of mirrors being indexed, someone on your "non-indexed" mirrors will spot it, they will call you out as a cheat, and a lot of places will ban you...

Don't do this...


User-agent: *
Disallow: /freesites/porn/freesite-mirror.html
Disallow: /freesites/porn/freesite-mirror2.html


Do do it for 2257 pages and cgi-bin etc

SE traffic on free sites is a nice little bonus (to gain a top 10 spot for a worthwhile term is exceptionally rare and fleeting), your most valuable source of filtered targeted traffic will always be LLs

[rooty]

Be very careful in your use of robots.txt - if you're trying to limit the number of mirrors being indexed, someone on your "non-indexed" mirrors will spot it, they will call you out as a cheat, and a lot of places will ban you...

Don't do this...


User-agent: *
Disallow: /freesites/porn/freesite-mirror.html
Disallow: /freesites/porn/freesite-mirror2.html


Do do it for 2257 pages and cgi-bin etc

SE traffic on free sites is a nice little bonus (to gain a top 10 spot for a worthwhile term is exceptionally rare and fleeting), your most valuable source of filtered targeted traffic will always be LLs

very good point I didnt even think about that... only the SEO values...

[Mike-mijen]

Be very careful in your use of robots.txt - if you're trying to limit the number of mirrors being indexed, someone on your "non-indexed" mirrors will spot it, they will call you out as a cheat, and a lot of places will ban you...

Don't do this...


User-agent: *
Disallow: /freesites/porn/freesite-mirror.html
Disallow: /freesites/porn/freesite-mirror2.html


Do do it for 2257 pages and cgi-bin etc

SE traffic on free sites is a nice little bonus (to gain a top 10 spot for a worthwhile term is exceptionally rare and fleeting), your most valuable source of filtered targeted traffic will always be LLs

I was unaware of that. I will have to make sure of that

[Mr Spock]

If you disallow the spiders onto your mirrors isn't that the same effect as nofollowing the recip links on those mirrors - Great way to lose friends!

I don't bother and I still get a fair bit of traffic from SE

[faxxaff]

A lot of robots including Googlebot will do nothing but ignore a robots.txt file.

[Maj. Stress]

A lot of robots including Googlebot will do nothing but ignore a robots.txt file.

Do you have any factual data to back that comment up?

[JK]

If you disallow the spiders onto your mirrors isn't that the same effect as nofollowing the recip links on those mirrors - Great way to lose friends!

Disallowing the spiders completely, and getting pages indexed but not passing link juice to link lists are totally different.

A lot of robots including Googlebot will do nothing but ignore a robots.txt file.

You may see "Googlebot" visiting pages you have disallowed in your robots.txt, I would hazard a guess that this is not really Google. It's quite easy to spoof the user-agent.