Greenguy's Board


Go Back   Greenguy's Board > General Business Knowledge
Register FAQ Calendar Search Today's Posts Mark Forums Read

Reply
 
Thread Tools Search this Thread Rate Thread Display Modes
Old 2013-10-04, 10:52 PM   #1
Bill
Selling porn allows me to stay in a constant state of Bliss - ain't that a trip!
 
Join Date: Apr 2003
Posts: 3,914
Did everyone else with a colo-mojo account just have all database servers vanish, or am I lucky?

So, about an hour ago all my database servers on every subaccount I have on the new mojo-colo just went dead - anybody else?

I never get concerned about a database server being down for 5 min, but an hour starts to make me nervous. Just never happened before.
Bill is offline   Reply With Quote
Old 2013-10-05, 07:10 AM   #2
Cleo
Subversive filth of the hedonistic decadent West
 
Cleo's Avatar
 
Join Date: Mar 2003
Location: Southeast Florida
Posts: 27,936
Just checked a few of mine and everything seems fine.

I do nightly backups of all my databases that are then backed up to a Mac in my home just in case something like this happens though.
__________________
Free Rides on Uber and Lyft
Uber Car: uberTzTerri
Lyft Car: TZ896289
Cleo is offline   Reply With Quote
Old 2013-10-05, 08:16 AM   #3
Bill
Selling porn allows me to stay in a constant state of Bliss - ain't that a trip!
 
Join Date: Apr 2003
Posts: 3,914
Well, it's been an insane night for me. I have one site that I do for a local organization that is having it's big yearly fundraiser today, and the site is dead dead dead. I havre told godaddy to move the nameservers another server I have, but godaddy is being pokey about updating the namservers, they have not resolved to the temp emergency page yet.

(removed the url - not smart to leave a nonadult url up on an adult board. I always intended to delete it just as soon as a few had seen it. especially you Cleo, I know you are interested in these things,a nd up in the bleak quiet mornings, lol.)

Most of my wordpress has come back, with quite a few errors and plugins failed still, but all my drupal sites are absolutely trashed, like that one whose link I pasted.

I explained to a few other customers who I am working on things for that the serves are down, luckily their sites are not mission critical right now, they can handle it.

I have a mix of good backups and shitty backups, I have never had trouble like this before, even at the worst hosts I ever had I could always get them to restore from a few days back, I guess that isn't the case now with mojo-colo? I just don't know.

Chris just fucking spoiled me. I have to remember that the relationship between host and builder is more adversarial than I have gotten used to.

Last edited by Bill; 2013-10-05 at 08:50 AM..
Bill is offline   Reply With Quote
Old 2013-10-05, 08:21 AM   #4
Bill
Selling porn allows me to stay in a constant state of Bliss - ain't that a trip!
 
Join Date: Apr 2003
Posts: 3,914
What is truly weird is that about three hours into the database server crash, the ftp servers died too. So I could not even ftp into the server and throw up an emergency html page with event info and maps onto the domain.

How the fuck do ftp servers die three hours after the database servers?

The techs tried to answer at first, and did restore part of the database servers, but they are half corrupted. Then silence for hours. A short while ago a replay saying sorry, they are stuck, and they have to go into the datacenter and fix hardware.
Bill is offline   Reply With Quote
Old 2013-10-05, 08:23 AM   #5
Cleo
Subversive filth of the hedonistic decadent West
 
Cleo's Avatar
 
Join Date: Mar 2003
Location: Southeast Florida
Posts: 27,936
Sounds like you may have lost a drive.
__________________
Free Rides on Uber and Lyft
Uber Car: uberTzTerri
Lyft Car: TZ896289
Cleo is offline   Reply With Quote
Old 2013-10-05, 08:28 AM   #6
Bill
Selling porn allows me to stay in a constant state of Bliss - ain't that a trip!
 
Join Date: Apr 2003
Posts: 3,914
Also, to be fair, I manage a certain number of sites that are on ordinary commercial hosts. I use midphase a lot for this kind of thing, just because I have been with them a while.

At midphase, the only techs left on the weekends, from Friday about 3 to Monday about 8am, are lower level with low power accounts. they just can't do all that much for real problems.

One Friday a module corrupted the database server for a site with about 9000 pages. the backups I had would have sufficed, but, I would have lost 20 or so pages. BUT, the low level techs could not, for some reason, restart the crashed database, so my backups were useless anyway.

Finally on sunday morning a tech came in with the power to give me a database on another server, and I pointed the drupal at the new sever with the old backup and had the site alive.

the original database server and their backups tho took a further two days to reactivate, and then walk back thru two days of corrupted restores to hit one that was sound.

That was a weird experience. This is heaven compared to that. Consumer accounts - they are fine until they SUCK, lol.
Bill is offline   Reply With Quote
Old 2013-10-05, 08:33 AM   #7
Bill
Selling porn allows me to stay in a constant state of Bliss - ain't that a trip!
 
Join Date: Apr 2003
Posts: 3,914
Quote:
Originally Posted by Cleo View Post
Sounds like you may have lost a drive.
yeah, I know. once the guy mentioned hardware, it all began to come together in a dark picture.

Some of the database backups must still exist tho, because like I said, they did, about 2.75 hours in, start up something, all my wordpress came back, just throwing plugin errors. The content seemed just as it had been hours before.

I actually figured if the wordpress is back the drupal would be fine, because drupal is tough as nails and will run even with badly corrupted tables- but every drupal site is just a tragedy.

I'm sure the fact that the wordpress works but the drupal doesn't tells me something significant, but I don't know what it is.
Bill is offline   Reply With Quote
Old 2013-10-05, 08:37 AM   #8
Cleo
Subversive filth of the hedonistic decadent West
 
Cleo's Avatar
 
Join Date: Mar 2003
Location: Southeast Florida
Posts: 27,936
Back when Colo was Chris's host my server lost a drive. It had been acting up for about a week and Chris drove down to see what was going on. He decided to put a new drive in my box and when he restarted it the old drive would not spin up. Turned out that two of the directories in the backups where corrupted as they had backed up my drive which had become corrupted. The two directories that were corrupted were the directory with all my databases and a directory that contained my main domain used for doing promo.

The only thing that saved my ass was my local backups on a Mac in my home and the only reason that I had a non corrupted backup was that I do seven different backups, one for each day of the week.
__________________
Free Rides on Uber and Lyft
Uber Car: uberTzTerri
Lyft Car: TZ896289
Cleo is offline   Reply With Quote
Old 2013-10-05, 08:41 AM   #9
Bill
Selling porn allows me to stay in a constant state of Bliss - ain't that a trip!
 
Join Date: Apr 2003
Posts: 3,914
Holy shit, it just came back. Now I have to scramble a bit, I already told godaddy to change nameservers.
Bill is offline   Reply With Quote
Old 2013-10-05, 08:48 AM   #10
Bill
Selling porn allows me to stay in a constant state of Bliss - ain't that a trip!
 
Join Date: Apr 2003
Posts: 3,914
Cool, just a few minutes ago everything came back completely, all the wordpress is good, no more plugin errors at all that I can see, the drupal is perfect as far as I can tell.

That's one of the differences between a commercial consumer host like midphase, and these kinds of hosts. yes, this was slower than I expected, for me the worst outage ever since I switched all to colo years ago, but still, at a certain point they get the fucking job done.

Still, I have a task ahead of me - scheduling consistent backups of everything. (I do always use backup plugins and modules for client sites, just not my own, and testbed sites and whatnot).
Bill is offline   Reply With Quote
Old 2013-10-05, 09:03 AM   #11
Cleo
Subversive filth of the hedonistic decadent West
 
Cleo's Avatar
 
Join Date: Mar 2003
Location: Southeast Florida
Posts: 27,936
Good News!

Yeah backups of backups of backups is what I always try to do.

I actually need to pickup another 4 terabyte drive so that I can start storing an offsite backup again.
__________________
Free Rides on Uber and Lyft
Uber Car: uberTzTerri
Lyft Car: TZ896289
Cleo is offline   Reply With Quote
Old 2013-10-06, 02:04 AM   #12
javbucks
You can now put whatever you want in this space :)
 
javbucks's Avatar
 
Join Date: Oct 2011
Posts: 544
Quote:
Originally Posted by Cleo View Post
Back when Colo was Chris's host my server lost a drive. It had been acting up for about a week and Chris drove down to see what was going on. He decided to put a new drive in my box and when he restarted it the old drive would not spin up. Turned out that two of the directories in the backups where corrupted as they had backed up my drive which had become corrupted. The two directories that were corrupted were the directory with all my databases and a directory that contained my main domain used for doing promo.

The only thing that saved my ass was my local backups on a Mac in my home and the only reason that I had a non corrupted backup was that I do seven different backups, one for each day of the week.


Backups are smart!
javbucks is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -4. The time now is 07:19 AM.


Mark Read
Powered by vBulletin® Version 3.8.1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
© Greenguy Marketing Inc