robot.txt

Mike-mijen · 2009-07-09, 03:51 PM

Does any one use robot.txt files on their free sites. I was just wondering. I am thinking about doing this myself.
Peace Mike

stuveltje · 2009-07-09, 03:59 PM

Quote:

Originally Posted by Mike-mijen

Does any one use robot.txt files on their free sites. I was just wondering. I am thinking about doing this myself.
Peace Mike

yes and since i use that one, i am much better found and got way much hits

LeRoy · 2009-07-09, 04:46 PM

No I haven't used that at all on freesites or any of my sites. I have some decent/fair Google placements so I'm thinking I should add this one to my "To Do"

On freesites do you just put the text file in the same directory as the free site? Or do you just add it to the root of the domain?

Mike-mijen · 2009-07-09, 05:07 PM

Good question. I am not sure. I think it would be better if put in with the free site itself. But I am not knowledge on this one. I need to look deeper for this answer on the web.

nate · 2009-07-09, 05:31 PM

Im not sure there is a reason to unless you don't want them indexed. I'm not even sure you need a link to it on your main site as SEs will pick it up from the link lists/.

Maj. Stress · 2009-07-09, 05:38 PM

The robots.txt should go in the root of the domain. It's a good idea to use one even if you want the entire domain indexed. Some overlooked uses for robots.txt are not indexing anything in your cgi-bin folder or 2257 pages.

rooty · 2009-07-10, 11:37 AM

Im not using this method yet as im not building freesites at the rate most people do but this sounds like a reasonable way of doing it maybe

adding this to your robots.txt

Code:

Sitemap: http://www.urdomain.tld/sitemap.xml

User-agent: *
Disallow: /freesites/porn/freesite-mirror.html
Disallow: /freesites/porn/freesite-mirror2.html

then adding the url's you would like to be indexed to the sitemap file

Wazza · 2009-07-10, 11:09 PM

Be very careful in your use of robots.txt - if you're trying to limit the number of mirrors being indexed, someone on your "non-indexed" mirrors will spot it, they will call you out as a cheat, and a lot of places will ban you...

Don't do this...

Code:

User-agent: *
Disallow: /freesites/porn/freesite-mirror.html
Disallow: /freesites/porn/freesite-mirror2.html

Do do it for 2257 pages and cgi-bin etc

SE traffic on free sites is a nice little bonus (to gain a top 10 spot for a worthwhile term is exceptionally rare and fleeting), your most valuable source of filtered targeted traffic will always be LLs

rooty · 2009-07-12, 08:40 AM

Quote:

Originally Posted by Wazza

Be very careful in your use of robots.txt - if you're trying to limit the number of mirrors being indexed, someone on your "non-indexed" mirrors will spot it, they will call you out as a cheat, and a lot of places will ban you...

Don't do this...

Code:

User-agent: *
Disallow: /freesites/porn/freesite-mirror.html
Disallow: /freesites/porn/freesite-mirror2.html

Do do it for 2257 pages and cgi-bin etc

SE traffic on free sites is a nice little bonus (to gain a top 10 spot for a worthwhile term is exceptionally rare and fleeting), your most valuable source of filtered targeted traffic will always be LLs

very good point I didnt even think about that... only the SEO values...

Mike-mijen · 2009-07-12, 10:13 AM

Quote:

Originally Posted by Wazza

Be very careful in your use of robots.txt - if you're trying to limit the number of mirrors being indexed, someone on your "non-indexed" mirrors will spot it, they will call you out as a cheat, and a lot of places will ban you...

Don't do this...

Code:

User-agent: *
Disallow: /freesites/porn/freesite-mirror.html
Disallow: /freesites/porn/freesite-mirror2.html

Do do it for 2257 pages and cgi-bin etc

SE traffic on free sites is a nice little bonus (to gain a top 10 spot for a worthwhile term is exceptionally rare and fleeting), your most valuable source of filtered targeted traffic will always be LLs

I was unaware of that. I will have to make sure of that

Mr Spock · 2009-07-12, 12:18 PM

If you disallow the spiders onto your mirrors isn't that the same effect as nofollowing the recip links on those mirrors - Great way to lose friends!

I don't bother and I still get a fair bit of traffic from SE

faxxaff · 2009-07-13, 12:48 AM

A lot of robots including Googlebot will do nothing but ignore a robots.txt file.

Maj. Stress · 2009-07-13, 01:36 AM

Quote:

Originally Posted by faxxaff

A lot of robots including Googlebot will do nothing but ignore a robots.txt file.

Do you have any factual data to back that comment up?

JK · 2009-07-14, 07:13 PM

Quote:

Originally Posted by Mr Spock

If you disallow the spiders onto your mirrors isn't that the same effect as nofollowing the recip links on those mirrors - Great way to lose friends!

Disallowing the spiders completely, and getting pages indexed but not passing link juice to link lists are totally different.

Quote:

Originally Posted by faxxaff

A lot of robots including Googlebot will do nothing but ignore a robots.txt file.

You may see "Googlebot" visiting pages you have disallowed in your robots.txt, I would hazard a guess that this is not really Google. It's quite easy to spoof the user-agent.

2009-07-09, 03:51 PM	#1
Mike-mijen What do you mean, my birth certificate expired? Join Date: Aug 2004 Location: Fla Posts: 995	robot.txt Does any one use robot.txt files on their free sites. I was just wondering. I am thinking about doing this myself. Peace Mike

2009-07-09, 04:46 PM	#3
LeRoy "Young dumb and full of cum" Join Date: Jun 2007 Location: Porn Valley Posts: 2,370	No I haven't used that at all on freesites or any of my sites. I have some decent/fair Google placements so I'm thinking I should add this one to my "To Do" On freesites do you just put the text file in the same directory as the free site? Or do you just add it to the root of the domain? __________________ JAPANESE ADULT AFFILIATE PROGRAM

2009-07-09, 05:31 PM	#5
nate I can now put whatever you want in this space :) Join Date: Mar 2009 Location: Merica! Posts: 543	Im not sure there is a reason to unless you don't want them indexed. I'm not even sure you need a link to it on your main site as SEs will pick it up from the link lists/. __________________ Its just a jump to the left.

2009-07-10, 11:37 AM	#7
rooty A woman is like beer. They look good, they smell good, and you'd step over your own mother just to get one! Join Date: Sep 2008 Posts: 51	Im not using this method yet as im not building freesites at the rate most people do but this sounds like a reasonable way of doing it maybe adding this to your robots.txt Code: Sitemap: http://www.urdomain.tld/sitemap.xml User-agent: * Disallow: /freesites/porn/freesite-mirror.html Disallow: /freesites/porn/freesite-mirror2.html then adding the url's you would like to be indexed to the sitemap file __________________ Nothing to see here really - My boring blog

2009-07-10, 11:09 PM	#8
Wazza I'm a jaded evil bastard, I wouldn't piss on myself if I was on fire... Join Date: Apr 2003 Location: Melbourne, Australia Posts: 808	Be very careful in your use of robots.txt - if you're trying to limit the number of mirrors being indexed, someone on your "non-indexed" mirrors will spot it, they will call you out as a cheat, and a lot of places will ban you... Don't do this... Code: User-agent: * Disallow: /freesites/porn/freesite-mirror.html Disallow: /freesites/porn/freesite-mirror2.html Do do it for 2257 pages and cgi-bin etc SE traffic on free sites is a nice little bonus (to gain a top 10 spot for a worthwhile term is exceptionally rare and fleeting), your most valuable source of filtered targeted traffic will always be LLs __________________ I sale Internet My sites have no traffic and no PR - let's trade - PM me

2009-07-09, 05:07 PM	#4
Mike-mijen What do you mean, my birth certificate expired? Join Date: Aug 2004 Location: Fla Posts: 995	Good question. I am not sure. I think it would be better if put in with the free site itself. But I am not knowledge on this one. I need to look deeper for this answer on the web.

2009-07-09, 05:38 PM	#6
Maj. Stress Progress rarely comes in buckets, it normally comes in teaspoons Join Date: Jun 2005 Location: Dark Side Of Naboo Posts: 1,289	The robots.txt should go in the root of the domain. It's a good idea to use one even if you want the entire domain indexed. Some overlooked uses for robots.txt are not indexing anything in your cgi-bin folder or 2257 pages.

2009-07-12, 12:18 PM	#11
Mr Spock You can now put whatever you want in this space :) Join Date: Nov 2006 Location: Vulcan Posts: 695	If you disallow the spiders onto your mirrors isn't that the same effect as nofollowing the recip links on those mirrors - Great way to lose friends! I don't bother and I still get a fair bit of traffic from SE

2009-07-13, 12:48 AM	#12
faxxaff I've always wondered if there was a god. And now I know there is -- and it's me Join Date: Mar 2005 Location: Vietnam Posts: 326	A lot of robots including Googlebot will do nothing but ignore a robots.txt file. __________________ Adult Webmaster Blog Fine Asian Pussy