|
|
|
Thread Tools | Search this Thread | Rate Thread | Display Modes |
2006-09-16, 02:28 AM | #1 |
I'm the only guy in the world who has to wake up to have a nightmare
Join Date: Feb 2004
Location: London, United Kingdom
Posts: 1,895
|
robots txt to prevent crawling of freesites
OK,
I have a guy submitting that on his root robots.txt has the following: User-agent: * Disallow: /gall/ Disallow: /gall1/ Disallow: /gall2/ Disallow: /gall3/ Disallow: /gall4/ Disallow: /gall5/ Disallow: /gall6/ Disallow: /gall7/ Disallow: /gall8/ Disallow: /gall9/ Disallow: /gall10/ Disallow: /gall11/ Disallow: /gall12/ Disallow: /gall13/ Disallow: /cgi-bin/ Disallow: /img/ domain: soccerwank.com (also sexcarrot.com with different directory names his freesites are in) On the freesites themselves, in the head, is the meta: meta name="robots" content="index, follow" I was running a link checker and was getting flags with this message: "The link was not checked due to robots exclusion rules. Check the link manually." Hence me looking at the root robots file. Seems very fishy to me, and this is titled 'possible cheaters' but I can't fathom whether this is an honest mistake, as obviously his freesites aren't going to get pickjed up by the SES, or just a way to glean traffic from LLs. |
2006-09-16, 02:32 AM | #2 |
Are you sure you're an accredited and honored pornographer?
Join Date: Aug 2003
Posts: 68
|
Also looks like a way to turn recip links (A->B->A) into one-way links from the LLs to his domains. (One-way links being more valuable.)
|
2006-09-16, 03:02 AM | #3 | |
on vacation
|
Quote:
he's a member of the board, maybe we'll hear something. |
|
2006-09-16, 12:29 PM | #4 |
There's Xanax in my thurible!
|
This isn't the only forum member doing this. Like Jel I'm not sure it's exactly cheating so I would like to hear more opinions on this.
|
2006-09-16, 12:39 PM | #5 |
on vacation
|
yes me too. I wouldn't consider it cheating, really - it's not breaking any rules, but it's misleading.
|
2006-09-18, 03:18 AM | #6 | |
Madness is like gravity. All it takes is a little... push.
Join Date: Feb 2006
Location: California
Posts: 1,679
|
Quote:
Malicious intent falls under the unspoken rule of, "I don't like your business practices, therefore, I don't want to do business with you." In this situation, I probably wouldn't send a rejection email, or even ask what's up. I'd just silently make their sites dissappear with a quick click of the delete button. Jel, thanks for bringing this issue up. As if I didn't already have enough to check for...
__________________
~Warm and Fuzzy. |
|
2006-09-20, 06:34 AM | #7 |
Are you sure you're an accredited and honored pornographer?
Join Date: Aug 2003
Posts: 68
|
Methinks folks don't look in this section often enough.
I think I'll send TT a note and tell him to look here; I like him and it surprises me that this kind of thing would be done on purpose. Maybe he's got a good explanation for it. Either way, it certainly isn't in anyone's LL rules that it can't be done, so...? Weird situation. |
2006-09-20, 07:28 AM | #8 |
They have the Internet on computers, now?
Join Date: Feb 2006
Posts: 141
|
Hi.
I am the owner of both soccerwank.com , sexcarrot.com and pornogata.com wich have the same robots.txt files and all the domains. When i first started out building galleries and freesites, i was told to create a robots.txt file like that to prevent google from crawling thousands of duplicate galleryfiles and hundred of duplicated freesitefiles. I am linking to the freesites on my mainsite, but offcourse that would not benefit all the LL i am submitting to. I willl change the robots.txt files on all my domain asap, and prevent from crawling only the galleryfolders instead of the freesite directory folders. I am sorry i have braught up the issue, cuz i was really not aware of it, i just followed some friends good advice. |
2006-09-20, 07:31 AM | #9 |
They have the Internet on computers, now?
Join Date: Feb 2006
Posts: 141
|
And please dont see my as a possible cheater.. i have never even once tried to cheat fellow webmasters with intention.
|
2006-09-20, 07:36 AM | #10 |
They have the Internet on computers, now?
Join Date: Feb 2006
Posts: 141
|
I have now removed the disallow to all directories at these 3 domains.
However, i do know most galleribuilders that have a HUB site are doing the same, but that is not me, so just wanted to tell that my directories are open for spiders now. |
2006-09-20, 07:40 AM | #11 | |
on vacation
|
Quote:
That's why I wanted to see what this was all about...I've never seen anything in your sites suggesting you cheat in any way, and I don't think you are a cheater either picXX. Thanks for coming by and clearing it up |
|
2006-09-20, 07:46 AM | #12 | ||
They have the Internet on computers, now?
Join Date: Feb 2006
Posts: 141
|
Quote:
Thanks to Carrie allso, that braught this thread to my attention. Virgohippy: Quote:
|
||
2006-09-20, 08:00 AM | #13 |
Searching for Jimmy Hoffa
Join Date: Jan 2005
Location: Long Island, NY
Posts: 771
|
Unfortunately picxx you were a victim of bad advice. I'd suggest you no longer listen to the person that gave you this advice in the first place
There's a lot of bad information tossed around on boards, etc, and you really have to be careful on who you listen to. This board is a good place to post questions about LL as you won't get steered wrong as you'll be getting the info straight from the horses (owners) mouth. |
2006-09-20, 03:27 PM | #14 | |
Madness is like gravity. All it takes is a little... push.
Join Date: Feb 2006
Location: California
Posts: 1,679
|
Quote:
With my next reincarnation, I'll make it a point to include a "if you don't see your site listed within a couple weeks, and you don't recieve a declined reason, contact me here..." But I still refuse to get my hands dirty!
__________________
~Warm and Fuzzy. |
|
2006-09-21, 03:08 AM | #15 | |
I'm the only guy in the world who has to wake up to have a nightmare
Join Date: Feb 2004
Location: London, United Kingdom
Posts: 1,895
|
Quote:
|
|
2006-09-22, 11:37 AM | #16 |
You can now put whatever you want in this space :)
|
I know LL owners like to get all the backlinks they can get their hands on (so would I), but you're forgetting one thing:
Duplicate content. Getting linkbacks from supplemental pages is not going to do anyone any good. EDIT: Not to mention low quality backlinks from free sites aren't going to make or break your ranking on Google (though MSN probably eats them up). In a few years, who knows, Google may ignore them altogether. One way to look at recips is advertising your LL via increasing brand awareness. Approach them as means of inflating your SE position -- and you're in violation of Google guidelines. Preventing duplicate content is a legitimate reason for disallowing mirrors, however. A large percentage of supps under a domain may negatively impact the entire domain.
__________________
Success is going from failure to failure without a loss of enthusiasm. Last edited by Halfdeck; 2006-09-22 at 11:59 AM.. |
2006-09-22, 12:11 PM | #17 | |
on vacation
|
Quote:
BUT make sure your meta tags don't say anything different, that's all. now I know this was an honest mistake but it's a good opportunity for everyone to make sure they check the little details |
|
2006-09-22, 04:02 PM | #18 | |
Searching for Jimmy Hoffa
Join Date: Jan 2005
Location: Long Island, NY
Posts: 771
|
Quote:
|
|
2006-09-22, 05:56 PM | #19 | |
There's Xanax in my thurible!
|
Quote:
Therefore whenever my bot ran across his domains all of his free-sites would get flagged and pulled as unavailable and I'd have to re-add them manually. This was incredibly annoying. I actually did this for a while but finally got tired of it and just left his sites in an error/delisted status. He no longer submits to me, but I still see his name pop-up from time to time. |
|
2006-09-22, 08:17 PM | #20 | ||
Madness is like gravity. All it takes is a little... push.
Join Date: Feb 2006
Location: California
Posts: 1,679
|
Quote:
Linking to a number of freesites on a domain with tons of dissallowed pages? or Linking to a number of freesites which may or may not be flagged for spam? Seems to me most submitters aren't able to produce and submit more than a small handful of mirrors anyway. Quote:
Not only that, but the occasional bot check of existing listings must surely make regular check ups yet even more fun.
__________________
~Warm and Fuzzy. |
||
2006-09-22, 08:28 PM | #21 | |||||
You can now put whatever you want in this space :)
|
Quote:
Quote:
Quote:
In that thread, I wrote: Quote:
Quote:
__________________
Success is going from failure to failure without a loss of enthusiasm. |
|||||
2006-09-22, 08:48 PM | #22 | |
Madness is like gravity. All it takes is a little... push.
Join Date: Feb 2006
Location: California
Posts: 1,679
|
Quote:
__________________
~Warm and Fuzzy. |
|
2006-09-22, 08:59 PM | #23 | |
You can now put whatever you want in this space :)
|
Quote:
Mirror sites and robots.txt disallow both lead to your LL likely getting no link juice from recips whatsoever. If you want decent backlinks, you might think about accepting only unique free sites. Even then, if you're linking to each other, chances are the link is completely ignored by Google.
__________________
Success is going from failure to failure without a loss of enthusiasm. |
|
2006-09-22, 10:04 PM | #24 | |
Madness is like gravity. All it takes is a little... push.
Join Date: Feb 2006
Location: California
Posts: 1,679
|
Quote:
But I see your point. In my own experiments I've noticed that backlinks from unique pages with backlinks from other unique pages gives a much higher return than backlinks from non-unique pages... well, from google at least.
__________________
~Warm and Fuzzy. |
|
2006-09-23, 02:56 AM | #25 | |
Searching for Jimmy Hoffa
Join Date: Jan 2005
Location: Long Island, NY
Posts: 771
|
Quote:
Now, I personally always submitted to a small LL grouping to remove the duplicate page penalty, but if there were more LLs that interested me, I would now do as I stated above. As for whether it's good advice or not to use the robot.txt...if this is setting off people's scripts, if it's too much hassle for them to review your freesites, reviewers will do exactly do as preacher and jel did...they'll most likely not even bother reviewing or listing your sites. So, there's a certain futility in following the robot.txt advice if it prevents you from easily getting listed at the LLs you're submitting to. As a submitter you have the options of either submitting to very few LL's (that was my choice when I was regularly submitting), Change the pages enough to avoid the duplicate page penalty (not hard to do when you're working with a template system), or use the robot.txt and not get listed on a number of sites you're submitting to. Out of those options the most sane and easy one is to submit to a very small high quality group of linklists that you know will list you regularly. You posted something Linkster said and maybe he can stop in and help me out on this point as I will gladly defer to his expertise in this area because I know he knows wayyyy more about this topic than I ever will. In that post you quoted, Linkster says that linksforsex went to a single recip, but a few months ago he went back to the category specific recips and that's still the case today. Now, why would a Link List switch back to category recips? As far as I can tell category recips have mainly been put in place for SEO? I'm not sure what benefit category specific recips would have other then SEO. If LLs are using category specific recips, it seems they're doing so for SEO, if that's the case then it would behoove all involved to make google as happy as they can and take effort to remove the duplicate page penalty, but at the same time not negate the category recips that so many LL's use by using a robot.txt to block the engine from searching those pages. Essentially you are breaking a Link Lists rules because you're completely negating any benefit the Link List owner was trying to get by having category specific recips. |
|
|
|