robots txt to prevent crawling of freesites

[Mr. Blue]

btw Halfdeck, this is a purely academic debate on my part and I respect your opinions...you may be completely right and I can be wrong on it. On the surface, at least to me, the robot.txt thing seems wrong, but I do understand your points on the topic and I'd be curious to see what others think on it as well.

[DJilla]

I've generally been of the mind that if its not in the rules you can pretty much do what you want and LL's when they encounter these odd situations (obviously only a few are doing this kind of thing) the LL has the responsibility to keep their listings up after confirming by hand or whatever. I admire Preacher for going that extra mile of rekeying his sites in when his script showed this links were disallowed by robots.txt. Also, there is the real politick that its probably not a good idea if LL's begin just ignoring your sites because you use this technique.

I guess what I became worried about was if picXX was aware of the other topics we were pointing out here and what they mean to him. He obviously is searching and taking advice on "best practices" by using the robots scheme but it seems he may have been frightened into taking it down immediately BEFORE he was able to get his pages in possible order to avoid the duplicate content penalty.

Does he clearly understand now that he should go back to his pages and change the text content, alt tags, and metatags on the mirrors etc AND maybe even re-submit some of them?. I hope so.

[Preacher]

Originally Posted by Halfdeck

...Even then, if you're linking to each other, chances are the link is completely ignored by Google.

I don't think the link is completley ignored, but I do think it becomes highly devalued.

[Halfdeck]

Out of those options the most sane and easy one is to submit to a very small high quality group of linklists that you know will list you regularly.

I agree Mr Blue.

btw Halfdeck, this is a purely academic debate on my part and I respect your opinions...you may be completely right and I can be wrong on it. On the surface, at least to me, the robot.txt thing seems wrong, but I do understand your points on the topic and I'd be curious to see what others think on it as well.

Same here. I don't think we're talking about right/wrong anyway. We're comparing odds.

So, there's a certain futility in following the robot.txt advice if it prevents you from easily getting listed at the LLs you're submitting to.

In any case, robots.txt isn't the best way to keep pages out of Google's index. Matt Cutts has stated that even if you disallow urls using robots.txt, if other sites link to that url, Google may list it in its index, albeit url only (no title/description). If you want to hide the url completely from Google's index, the commonly recommended course of action is labeling pages using META noindex tag. I don't know LL scripts, but I doubt META robots tag would interfere with their crawling. Else I dare say they should be rewritten.

Essentially you are breaking a Link Lists rules because you're completely negating any benefit the Link List owner was trying to get by having category specific recips.

If robots.txt is one of your LL rules.

My point though, is if you accept mirror free sites, chances are you're getting linked from a supplemental page which does you no good anyway. Also, Google seems to be getting pickier about duplicate content especially from unknown, untrusted, 1 month old domains, so just tweaking the title/meta tag and on-page text may not always be enough to keep a page in the main index.

Let me post an example.

I have a list of free sites here:

http://www.nastyxvids.com/sitemap/

Mind you, I built these free sites before I was even aware of search engines, so this isn't exactly scientific (also, site: search is a bit quirky lately, and you may see something different from what I'm seeing depending on which DC you're hitting). The domain is a little short of 2 years old.

Pages listed in Google's main index:

http://www.google.com/search?q=site%...en-US:official
http://www.google.com/search?hs=6Db&...2F&btnG=Search
http://www.google.com/search?hs=7tv&...2F&btnG=Search
http://www.google.com/search?hs=duv&...2F&btnG=Search
http://www.google.com/search?hs=SaG&...2F&btnG=Search
http://www.google.com/search?hs=JGb&...2F&btnG=Search
http://www.google.com/search?hs=YbG&...2F&btnG=Search
http://www.google.com/search?hs=ewv&...2F&btnG=Search
http://www.google.com/search?hs=eHb&...2F&btnG=Search
http://www.google.com/search?hs=Hxv&...2F&btnG=Search

Most of the LLs I submitted to are getting no link love from my submissions on that domain.

------------------------------------

The way I'd go about free site mirrors now would be this:

/index.html
/main.html
/gallery1.html
/gallery2.html
/doorway1.html -> links to main.html
/doorway2.html -> link to main.html

Provided doorway1.html is significantly different from /index.html, and assuming 100s of templates a submitter uses are significantly different from each other, (and assuming 10,000s of submitted free sites are unique enough in terms of on-page text/HTML structure), and assuming further that a submitter build free sites on a one year+ old, trusted, TBPR 3+ domain, there are plenty of unique text (200-300 words+) on each page ..... I think all pages will be indexed as unique pages in Google, and no robots.txt disallow is needed.

Still, my main objection would be against tactics aimed at artificially boosting your SE rankings. I wouldn't assume grey hat methods like recips (they're not citations or "votes" with minimal traffic value) will work indefinitely.

I don't think the link is completley ignored, but I do think it becomes highly devalued.

I don't think anything - which is why I said "chances are" - because I have no concrete evidence either way.

Advertising's fine
Buying links for PR: bad
Google senses much

Adam Lasnik (Google's new PR guy):
http://www.webmasterworld.com/google/3079355.htm

Whether he's bluffing or not who knows. I do know Google already detects and kills PageRank transfers on *some* bought links, and I assume the same to be happening with some traded, "made for SE ranking" links.

Another relevant quote (Matt Cutts):

After looking at the example sites, I could tell the issue in a few minutes. The sites that fit “no pages in Bigdaddy” criteria were sites where our algorithms had very low trust in the inlinks or the outlinks of that site. Examples that might cause that include excessive reciprocal links, linking to spammy neighborhoods on the web, or link buying/selling.

http://www.mattcutts.com/blog/indexing-timeline/

I still do not agree with the mentality "how can I improve/optimize my ranking without getting penalized?" whch seems to be driving this robots.txt discussion. A better question imo would be "how can I make my site more valuable to visitors, and more visible, so more people will find what they're looking for?"

Bottom line: I see nothing wrong with blocking duplicate content pages using robots.txt or meta noindex tag - that's commonly recommended SEO practice. A free site submitter doesn't gain PageRank by disallowing / noindexing a page. It only prevents duplicate content from being indexed. Tagging a free site page with NOFOLLOW would send me a different signal (a free site submitter trying to hog PageRank), but that's another issue.

P.S. Off topic, but if I ran a LL, I would think about tagging links to free sites with NOFOLLOW, as does Technorati tag pages, which are starting to rank very well on Google. You eliminate the reciprocal linking issue (turn all free site links into one way links), and possible negative trust brought on by linking to supplemental/duplicate content pages on untrusted domains.

[Mr. Blue]

Excellent post Halfdeck, you've really explained the topic perfectly and now I do agree with your stance on it. Just a few things.

Originally Posted by Halfdeck

The way I'd go about free site mirrors now would be this:

/index.html
/main.html
/gallery1.html
/gallery2.html
/doorway1.html -> links to main.html
/doorway2.html -> link to main.html

That's the way I used to do my freesite linking for mirrored pages but it seems like a lot of linklist don't like that format for linking. I started to do the linking differently to meet with the requirements, but it annoyed me enough where I decided to go down to a small tight focused group of LinkLists instead.

P.S. Off topic, but if I ran a LL, I would think about tagging links to free sites with NOFOLLOW, as does Technorati tag pages, which are starting to rank very well on Google. You eliminate the reciprocal linking issue (turn all free site links into one way links), and possible negative trust brought on by linking to supplemental/duplicate content pages on untrusted domains.

Just two comments on it. When you refer to tagging links you mean the links coming in would have a rel="tag" type tagging? So, to use your above example domain if I were a LL owner...the link would be included as

<a href="http://www.nastyxvids.com/keyword/index.html" rel="tag">Keyword</a>

Something like that?

Also, since we're really talking about small link lists and their rules (as big linklist probably don't have to worry about the No Follow rule). Wouldn't it be wiser for smaller link lists to use recips more like TGPs use recips? Sorry, I'm more of a tgp guy so I'm going to explain this in those terms...but for TGPs we use a single recip with almost no hope of getting SE off that recip. What we use it for is more or less getting a percentage of traffic from other tgps that list the same gallery (hopefully getting listed with tgps of equal size or bigger than your own). So the focus of the recip is heavily on branding the domain name / name of the tgp or if it's a niche tgp highlighting the niche quality of your tgp that might draw people off of a general tgp to your site if they have a specific fetish.

Shouldn't smaller link lists move to that single recip that heavily brands instead of a category recip?

[virgohippy]

Originally Posted by Halfdeck

Still, my main objection would be against tactics aimed at artificially boosting your SE rankings. I wouldn't assume grey hat methods like recips (they're not citations or "votes" with minimal traffic value) will work indefinitely.

For the moment, there may be some mutual benefit, albeit small, but it's certainly reassuring to know that in the future things can only get more strict/difficult. |shocking|

Originally Posted by Halfdeck

P.S. Off topic, but if I ran a LL, I would think about tagging links to free sites with NOFOLLOW, as does Technorati tag pages, which are starting to rank very well on Google. You eliminate the reciprocal linking issue (turn all free site links into one way links), and possible negative trust brought on by linking to supplemental/duplicate content pages on untrusted domains.

I can think of a few other wembasters who stopped submitting to a few lists because of this. As a word of caution, just as a LL owner wouldn't want to recieve a nofollow in their link, I don't think a submitter would be too happy to get one either.

If I had a choice, I'd rather link to fewer sites and send more of my traffic to submitters who make quality sites and unique warning pages. IMO, doing so not only benefits both of us as far as SEO is concerned, but it also maintains a quality brand for my own LL.

[virgohippy]

Originally Posted by Mr. Blue

Just two comments on it. When you refer to tagging links you mean the links coming in would have a rel="tag" type tagging? So, to use your above example domain if I were a LL owner...the link would be included as

<a href="http://www.nastyxvids.com/keyword/index.html" rel="tag">Keyword</a>

Something like that?

I think halfdeck is referring to this:

<a href="http://www.nastyxvids.com/keyword/index.html" rel="nofollow">Keyword</a>

A nofollow link is like wearing a condom.

[Linkster]

Well - I guess I better fess up as to why I went to the cat recips - and it actually had nothing whatsoever to do with SEO - as a matter of fact the submitters will be happy to know that Im switching back - but it will be a text link not a gif file

The reason I did it is that I knew that a certain gif file of mine was in some autosubmitters programs - I used it as a tool to find the autosubmitters (at least the first part of the detective work anyway) and it worked

I would say that based on what I have experienced and some other LLs have seen, that category recips dont mean shit unless you also go the route of requesting that the recips be on a page already in Google - similar to what one sucessful LL owner did to get some really nice spots - but I have to agree that branding in this day and age is ten times as important (building bookmarkers) than trying to run a LL based on SE's only - it may work for a few months, but then when Google tweaks something you try to fight Google instead of expending the energy on making your brand and getting "return surfer trust" - believe me - that group spends way more money

/added - I know that putting the submitters through having to do the cat recips was a little extra work ontheir end - but I think the benefit of having less competition for spots on LLs gives a bigger benefit to free site makers

[Halfdeck]

I think halfdeck is referring to this:

<a href="http://www.nastyxvids.com/keyword/index.html" rel="nofollow">Keyword</a>

Yep.

I can think of a few other wembasters who stopped submitting to a few lists because of this. As a word of caution, just as a LL owner wouldn't want to recieve a nofollow in their link, I don't think a submitter would be too happy to get one either.

If I had a choice, I'd rather link to fewer sites and send more of my traffic to submitters who make quality sites and unique warning pages. IMO, doing so not only benefits both of us as far as SEO is concerned, but it also maintains a quality brand for my own LL.

As a submitter, I wouldn't like it either. I'm not advocating the idea; just pondering it. Not something I'd try with a new LL with just a handful of submitters.

However, LL aren't meant to be cheap directories, PageRank boosters, or reciprocal linking networks. LL is a source of traffic, plain and simple.

Slapping a nofollow on your category page links *may* result in higher SERP positions, because your incoming links from free sites are all one-way.

Technorati /tags/ pages are a good example imo of a site that seems to be gaining SERP positions thanks to massive one-way links (also doesn't hurt that many of these links are from reputable/authority/relevant sites):

52,309 posts currently tagged "seo" (assuming a high percentage of them pointing to technorati with rel="tag" links). Result?

http://www.google.com/search?q=seo&s...en-US:official

9th out of 11 million results.

If you look at the top of their page:

"http://www.technorati.com/tags/SEO"

you'll see:

<meta name="robots" content="index,nofollow" />

(Not a scientific test).

John Battelle recently released an interview with Matt Cutts regarding wc3 deciding to use nofollow on all their $1000 links.

question:

W3C Schools is listing its supporters' websites on Page Rank 9 and PR7 pages in exchange for donations, $1000 a pop in cash or trade (http://www.w3.org/Consortium/sup). Speculation on this is buzzing because though W3C is a well respected educational resource many SEO blackhats endorse similar tactics. Does Google consider link selling a type of webspam against Google's TOS? And if so, should we expect to see some kind of a censure on W3C? Or how does it differ from what Google considers webspam?

You can read Matt Cutts' answer here:
http://battellemedia.com/archives/002917.php

[DJilla]

Originally Posted by Halfdeck

As a submitter, I wouldn't like it either. I'm not advocating the idea; just pondering it. Not something I'd try with a new LL with just a handful of submitters.

I've been thinking just the same. I'm not going to do it because I'm that smaller LL with submitters that I don't want to freak out. But, I'm pretty sure that once what we all believe is confirmed beyond a shadow of a doubt (ie straight recips don't bring PR or listings anymore like they used to) that this is the way to go. I've been seeing similar thoughts at a whole bunch of boards that basically agree.

Originally Posted by Halfdeck

However, LL aren't meant to be cheap directories, PageRank boosters, or reciprocal linking networks. LL is a source of traffic, plain and simple.
Slapping a nofollow on your category page links *may* result in higher SERP positions, because your incoming links from free sites are all one-way.

If this is proven to be true then the submitter benefits from more traffic.

[virgohippy]

Originally Posted by DJilla

If this is proven to be true then the submitter benefits from more traffic.

Only problem is making 'em believe it.

I know one big reason I jumped into freesites so strong was because of the SE benefits.

[Carrie]

It's gotten to the point where the "SE benefits" (incoming links, quicker spider access coming to your sites) far outweigh the traffic benefits when submitting to LLs. Most of the submitters I know *only* do it now for SE purposes, any sale the site might possibly make is just a bonus.

Used to be you could submit a freesite and make 3 sales the first week. Now you're lucky if you make 3 sales in the 6 months that it's up (and then some LL owners rip it down). The traffic ain't what it used to be... hence the LL owners trading with TGPs, increasing required number of pictures, making pictures bigger, etc.

If you take away the SE benefits then there really isn't much reason left to submit anymore, unless you like working your ass off for the possibility of an odd once-in-a-blue-moon signup. The time would be better spent doing other things.

[DJilla]

Originally Posted by Carrie

It's gotten to the point where the "SE benefits" (incoming links, quicker spider access coming to your sites) far outweigh the traffic benefits when submitting to LLs. Most of the submitters I know *only* do it now for SE purposes, any sale the site might possibly make is just a bonus.

That's an interesting take and I certainly felt the same way about the SE benefits when I got into it. I'm not sure if your referring specifically to just getting your site "spidered" more often in which case, ya you are probably right. But for PR benefit and getting specifc FS listed I think the marketplace has shown that both are pretty low liklihoods. Hence maybe the move towards submitting to LL's you know are high in traffic or aren't going to disappear next month (like mine). But, I don't think you can deny that the spiders will come as a result of your FS listing and is a good reason for expecting an LL to not use the nofollow tag. Perhaps the best and only place for a nofollow tag is strictly on sponsor galleries.

Originally Posted by Carrie

Used to be you could submit a freesite and make 3 sales the first week. Now you're lucky if you make 3 sales in the 6 months that it's up (and then some LL owners rip it down). The traffic ain't what it used to be... hence the LL owners trading with TGPs, increasing required number of pictures, making pictures bigger, etc.

My guess is that lower sales is due to more competition.

[Carrie]

Originally Posted by DJilla

My guess is that lower sales is due to more competition.

Probably a good part of it. I'd love to know if the # of submitters has grown, shrunk, or remained pretty steady over the past 4 years or so. That would be interesting.

I think another factor in less sales might be the sheer number of sites that the larger LLs have in their archives. The ones that have been around a few years have a few hundred thousand sites in there, most likely. I was never up for the "remove the listing after 3/6 months" (because sometimes the SEs don't even notice you for that long or they sandbox you for a while to see if you're gonna stick around), but I'm starting to think that dumping sites that are over a year old might help the submitters make more sales. There would still be a ton of sites to keep the bookmarkers around looking at the LLs ads, so maybe it wouldn't negatively affect the LL owners to do this. Would love your take on it D.