Did everyone else with a colo-mojo account just have all database servers vanish, or am I lucky?
 

So, about an hour ago all my database servers on every subaccount I have on the new mojo-colo just went dead - anybody else?

I never get concerned about a database server being down for 5 min, but an hour starts to make me nervous. Just never happened before.

Just checked a few of mine and everything seems fine.

I do nightly backups of all my databases that are then backed up to a Mac in my home just in case something like this happens though.

Well, it's been an insane night for me. I have one site that I do for a local organization that is having it's big yearly fundraiser today, and the site is dead dead dead. I havre told godaddy to move the nameservers another server I have, but godaddy is being pokey about updating the namservers, they have not resolved to the temp emergency page yet.

(removed the url - not smart to leave a nonadult url up on an adult board. I always intended to delete it just as soon as a few had seen it. especially you Cleo, I know you are interested in these things,a nd up in the bleak quiet mornings, lol.)

Most of my wordpress has come back, with quite a few errors and plugins failed still, but all my drupal sites are absolutely trashed, like that one whose link I pasted.

I explained to a few other customers who I am working on things for that the serves are down, luckily their sites are not mission critical right now, they can handle it.

I have a mix of good backups and shitty backups, I have never had trouble like this before, even at the worst hosts I ever had I could always get them to restore from a few days back, I guess that isn't the case now with mojo-colo? I just don't know.

Chris just fucking spoiled me. I have to remember that the relationship between host and builder is more adversarial than I have gotten used to.

What is truly weird is that about three hours into the database server crash, the ftp servers died too. So I could not even ftp into the server and throw up an emergency html page with event info and maps onto the domain.

How the fuck do ftp servers die three hours after the database servers?

The techs tried to answer at first, and did restore part of the database servers, but they are half corrupted. Then silence for hours. A short while ago a replay saying sorry, they are stuck, and they have to go into the datacenter and fix hardware.

Sounds like you may have lost a drive. :(

Also, to be fair, I manage a certain number of sites that are on ordinary commercial hosts. I use midphase a lot for this kind of thing, just because I have been with them a while.

At midphase, the only techs left on the weekends, from Friday about 3 to Monday about 8am, are lower level with low power accounts. they just can't do all that much for real problems.

One Friday a module corrupted the database server for a site with about 9000 pages. the backups I had would have sufficed, but, I would have lost 20 or so pages. BUT, the low level techs could not, for some reason, restart the crashed database, so my backups were useless anyway.

Finally on sunday morning a tech came in with the power to give me a database on another server, and I pointed the drupal at the new sever with the old backup and had the site alive.

the original database server and their backups tho took a further two days to reactivate, and then walk back thru two days of corrupted restores to hit one that was sound.

That was a weird experience. This is heaven compared to that. Consumer accounts - they are fine until they SUCK, lol.

yeah, I know. once the guy mentioned hardware, it all began to come together in a dark picture.

Some of the database backups must still exist tho, because like I said, they did, about 2.75 hours in, start up something, all my wordpress came back, just throwing plugin errors. The content seemed just as it had been hours before.

I actually figured if the wordpress is back the drupal would be fine, because drupal is tough as nails and will run even with badly corrupted tables- but every drupal site is just a tragedy.

I'm sure the fact that the wordpress works but the drupal doesn't tells me something significant, but I don't know what it is.

Back when Colo was Chris's host my server lost a drive. It had been acting up for about a week and Chris drove down to see what was going on. He decided to put a new drive in my box and when he restarted it the old drive would not spin up. Turned out that two of the directories in the backups where corrupted as they had backed up my drive which had become corrupted. The two directories that were corrupted were the directory with all my databases and a directory that contained my main domain used for doing promo.

The only thing that saved my ass was my local backups on a Mac in my home and the only reason that I had a non corrupted backup was that I do seven different backups, one for each day of the week.

Holy shit, it just came back. Now I have to scramble a bit, I already told godaddy to change nameservers.

Cool, just a few minutes ago everything came back completely, all the wordpress is good, no more plugin errors at all that I can see, the drupal is perfect as far as I can tell.

That's one of the differences between a commercial consumer host like midphase, and these kinds of hosts. yes, this was slower than I expected, for me the worst outage ever since I switched all to colo years ago, but still, at a certain point they get the fucking job done.

Still, I have a task ahead of me - scheduling consistent backups of everything. (I do always use backup plugins and modules for client sites, just not my own, and testbed sites and whatnot).

Good News!

Yeah backups of backups of backups is what I always try to do.

I actually need to pickup another 4 terabyte drive so that I can start storing an offsite backup again.

|fonz|

Backups are smart!