|
2005-09-07, 11:24 AM | #1 |
I saw weird stuff in that place last night. Weird, strange, sick, twisted, eerie, godless, evil stuff. And I want in
|
Fast 404 error checker for Link Lists
does anyone know where to get the subject?
It must check links for 404/redirect... Php one so I am able to cronjob it The one I have fails from time to time. |
2005-09-08, 01:58 PM | #2 |
I saw weird stuff in that place last night. Weird, strange, sick, twisted, eerie, godless, evil stuff. And I want in
|
Please guys, I know you're hiding something
|
2005-09-08, 06:39 PM | #3 |
Aw, Dad, you've done a lot of great things, but you're a very old man, and old people are useless
Join Date: Jun 2005
Posts: 23
|
Do you have the links to be checked in a dbase or does it just have to check from the site?
Regards, Thomas
__________________
Please Re-Read The Rules For Sig Files |
2005-09-09, 01:53 AM | #4 |
I saw weird stuff in that place last night. Weird, strange, sick, twisted, eerie, godless, evil stuff. And I want in
|
it doesn matter..I think from db will be faster..so..But will be happy with bot also
|
2005-09-30, 11:58 AM | #5 |
Trying is the first step towards failure
|
I don't know any PHP script which will do this 'out-of-the-box'
Do you have any coding experience? If so, check: http://www.php.net/manual/nl/ref.curl.php If not, feel free to contact me
__________________
Submit galleries/links | Trade Hardlnks | Free Forum Hosting | Build your own search engine | Sponsor scorecard |
2005-10-01, 04:16 AM | #6 |
Just because I don't care doesn't mean I don't understand!
Join Date: Apr 2004
Location: Spaceship Earth
Posts: 91
|
|
2005-10-01, 12:43 PM | #7 |
I saw weird stuff in that place last night. Weird, strange, sick, twisted, eerie, godless, evil stuff. And I want in
|
thanks so much guys!
Mr. Stiff I can php..but confused what to do with curl? how it can help me out? 2Joneze...thanks for the link! |
2005-10-02, 04:24 AM | #8 |
Trying is the first step towards failure
|
Hi,
Curl is a good program for getting webpages, headers, etc. It's installed on most (good) hosting servers. Here's how I use it: - Column 'lastspider' on my gallery table - Query table, getting URL's not spidered the last xxx days/hours/weeks/whatever - Use curl extension to connect to URL. - You can choose only to download headers, which is much faster than downloading the full page - Check header respons (must be 200). If it's 404 -> page not found, 301 or 302 -> redirect) - Update your table!
__________________
Submit galleries/links | Trade Hardlnks | Free Forum Hosting | Build your own search engine | Sponsor scorecard |
2005-10-03, 11:58 AM | #9 | |
No offence Apu, but when they were handing out religions you must have been out taking a whizz
|
Quote:
__________________
Please Re-Read The Rules For Sig Files |
|
2005-10-03, 02:03 PM | #10 |
I saw weird stuff in that place last night. Weird, strange, sick, twisted, eerie, godless, evil stuff. And I want in
|
I have checker..it does not use curl() and it fails me..gives invalid results most of the times.
I dont like scripts that are zend since I use to optimise script myself..making it unique. Thanks guys, this thread should be usefull for these who dont know about it. |
2005-10-07, 08:15 AM | #11 |
I saw weird stuff in that place last night. Weird, strange, sick, twisted, eerie, godless, evil stuff. And I want in
|
mr stiff, its a good idea.."researching" curl right now..
$ch = curl_init(); curl_setopt($ch, CURLOPT_URL, "http://www.sortlinks.com"); curl_setopt($ch, CURLOPT_HEADER, 0); curl_setopt($ch, CURLOPT_CURLOPT_REFERER, $host); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"); curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_0); $t = curl_exec($ch); echo $t; I was only able get full page like php file() could you please let me know how to get header and get so called spider response? |
2005-10-10, 06:51 PM | #12 |
Aw, Dad, you've done a lot of great things, but you're a very old man, and old people are useless
Join Date: Jun 2005
Posts: 23
|
code = curl_easy_setopt(http_headconn, CURLOPT_NOBODY, 1);
__________________
Please Re-Read The Rules For Sig Files |
2005-10-11, 11:24 AM | #13 |
I saw weird stuff in that place last night. Weird, strange, sick, twisted, eerie, godless, evil stuff. And I want in
|
Fatal error: Call to undefined function: curl_easy_setopt()
|
2005-10-14, 12:45 PM | #14 |
With $10,000, we'd be millionaires! We could buy all kinds of useful things like ... love!
|
On my domains I block all known offline browsers, email harvesters, download managers, etc.
Curl is one of those that I block... because I don't want anyone 'mirroring' my content. I've tried using scripts to clean out the 404s and redirects, but nothing is 100% accurate. Even manual checking isn't perfect, as you could check at the time the server is going thru a reset for whatever reason. You should use (and trust) whichever you find the most satisfactory for you... or a combination of 2 or 3. Just my 2c worth.
__________________
Playboy Webmasters - The name says it all! $35 per signup or 60% revshare. |
2005-10-14, 07:33 PM | #15 |
I saw weird stuff in that place last night. Weird, strange, sick, twisted, eerie, godless, evil stuff. And I want in
|
oast, how do you manage to dist. good bots from bad ones?
|
2005-10-14, 08:13 PM | #16 |
With $10,000, we'd be millionaires! We could buy all kinds of useful things like ... love!
|
Thru the User Agent string that (nearly) all programs use to identify themselves.
I use htaccess then to forbid (or redirect) the 'bad boys'.
__________________
Playboy Webmasters - The name says it all! $35 per signup or 60% revshare. |
2005-10-14, 08:30 PM | #17 | |
With $10,000, we'd be millionaires! We could buy all kinds of useful things like ... love!
|
A small extract from the filtering lines of my .htaccess file looks like this:
Quote:
mod_rewrite can be a very powerful tool if used correctly. AFAIK Apache is the only server it is available on, but as a large number of hosting companies prefer Apache, you should be OK
__________________
Playboy Webmasters - The name says it all! $35 per signup or 60% revshare. |
|
2005-10-14, 08:41 PM | #18 | |
With $10,000, we'd be millionaires! We could buy all kinds of useful things like ... love!
|
Quote:
Honestly Bill, I was naive at the time. I don't do things like that any more |blowkiss|
__________________
Playboy Webmasters - The name says it all! $35 per signup or 60% revshare. |
|
2005-10-20, 02:59 AM | #19 | |
Trying is the first step towards failure
|
Quote:
Definatly leave the line 'curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)");' This will fool those webmasters checking for Curl
__________________
Submit galleries/links | Trade Hardlnks | Free Forum Hosting | Build your own search engine | Sponsor scorecard |
|
2005-10-20, 03:27 AM | #20 |
I saw weird stuff in that place last night. Weird, strange, sick, twisted, eerie, godless, evil stuff. And I want in
|
thanks ost and Mr. Stiff - good job
|
|
|