Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Why Mirroring Is Not a Backup Solution

kdawson posted more than 5 years ago | from the pointed-lesson dept.

Data Storage 711

Craig writes "Journalspace.com has fallen and can't get up. The post on their site describes how their entire database was overwritten through either some inconceivable OS or application bug, or more likely a malicious act. Regardless of how the data was lost, their undoing appears to have been that they treated drive mirroring as a backup and have now paid the ultimate price for not having point-in-time backups of the data that was their business." The site had been in business since 2002 and had an Alexa page rank of 106,881. Quantcast said they had 14,000 monthly visitors recently. No word on how many thousands of bloggers' entire output has evaporated.

cancel ×

711 comments

Sorry! There are no comments related to the filter you selected.

DUH! (5, Insightful)

Anonymous Coward | more than 5 years ago | (#26301311)

DUH!

Re:DUH! (1)

frovingslosh (582462) | more than 5 years ago | (#26301845)

Let us consider the advice of the great philosophers Mr. T and Nelson on this subject.

Re:DUH! (5, Funny)

djupedal (584558) | more than 5 years ago | (#26301965)

As if millions of voices suddenly cried out in terror, and were suddenly silenced.

Again a frost post to a red story (5, Funny)

Anonymous Coward | more than 5 years ago | (#26301313)

While this mirrors previous comments, it's not really a backup solution.

Re:Again a frost post to a red story (1)

moranar (632206) | more than 5 years ago | (#26301915)

Now this is what I call First Post. Well done!

This was caused by malicious interference (-1, Flamebait)

Anonymous Coward | more than 5 years ago | (#26301327)

How long will we let Italians use the internet unhindered?

"The clemency which treats with tyranny is barbarous" --Robespierre

Oh Come on! (0)

Anonymous Coward | more than 5 years ago | (#26301331)

Who was the IT person there? It's been obvious for years and years that mirroring is a crap solution for backup! I.D.I.O.T.S!

When is backing up *not* an option? (5, Interesting)

wandazulu (265281) | more than 5 years ago | (#26301333)

Mirroring, RAID, grid, whatever. At some point, you want your data safe and secure on something not physically attached to any power source.

Re:When is backing up *not* an option? (5, Insightful)

Anonymous Coward | more than 5 years ago | (#26301465)

Incremental backups to tape every night, full backup at the weekend. Tapes must be stored off-site at a proper storage location. Got lots of data and a small backup window? Get a faster tape drive and a tape robot. It costs money, but you data costs more.

This is at a minimum people. Come on!

Re:When is backing up *not* an option? (2, Informative)

z_gringo (452163) | more than 5 years ago | (#26301749)

nightly dumps of the database and rsync of the data directories to servers in different locations should be adequate. If you have lots of data, I don't see how tapes are really going to do the daily backup jobs.

backing up nightly to a large mirrored NAS and a periodic copy to a removable device seems like a good way to go these days. I haven't used tapes for years.

Re:When is backing up *not* an option? (0, Troll)

Nutria (679911) | more than 5 years ago | (#26301909)

If you have lots of data, I don't see how tapes are really going to do the daily backup jobs.

More tape drives, idiot, and parallel backups. Mainframes and OpenVMS have been doing it that way for decades.

To many shops think HA==DR (4, Informative)

uncledrax (112438) | more than 5 years ago | (#26301521)

It's more an issue that some people think that HA == DR.. which obviously this story reminds us that it is not the same thing.

Mirroring / RAID == HA.. if one of your HDDs let the smoke out, you still don't incur downtime. If you have a hot-spare, you're even better.. all it does it let you have alittle time to correct the
issue (ie: "It can wait until morning").

Also, one other very important thing.. mirroring doesn't prevent/restore data corruption. If you're mirroring your rm -rf (as pointed out by Corsec67 below), your RAID will happy do what it does.. and span your command to all your disks.... Congrats, you just successfully gave yourself HA to your disk erasing! :]

Backups are DR.. If your RAID croaks.. your SOL if you don't off-machine backups. If you accidently nuke your disks with an rm or something, you can still go back and restore data.. sure you'll likely loose -some- data, but -some- is better then all in this case.

Re:To many shops think HA==DR (1)

Anonymous Coward | more than 5 years ago | (#26301997)

Not to be a pain, but I can figure out that DR is 'Data Recovery'. Is HA 'Hardware Availability'? It makes sense that way, but it took me a moment to figure it out (not easy to google those two things, no?)

Re:When is backing up *not* an option? (0)

Anonymous Coward | more than 5 years ago | (#26301829)

Not attached to *any power source*? Surely you must be joking, Mr. Wandazulu!

stunned silence (0)

Anonymous Coward | more than 5 years ago | (#26301335)

Slashdot is stunned into silence.

Re:stunned silence (5, Funny)

conureman (748753) | more than 5 years ago | (#26301525)

I am experiencing a strange phenomenon. The jaw-drop reflex has been popping my mouth open for several minutes and won't stop. If I focus I can close it, but then it pops open again. wow.

Dear Every Corporate Tool in the Universe: (5, Insightful)

yttrstein (891553) | more than 5 years ago | (#26301337)

And that's why your IT department actually needs funding. Sleep tight.

Re:Dear Every Corporate Tool in the Universe: (5, Insightful)

TubeSteak (669689) | more than 5 years ago | (#26301537)

And that's why your IT department actually needs funding. Sleep tight.

They've had the site live for 6 years.
This wasn't a lack of funding, it was just sheer stupidity.

6 years and nobody ever thought it'd be a good idea to back everything up to dvd or an external hard drive. HTML compresses really well in case they didn't know.

Re:Dear Every Corporate Tool in the Universe: (4, Insightful)

yttrstein (891553) | more than 5 years ago | (#26301569)

Stupidity IS a lack of funding. Pay the salary of someone smart enough to handle your data correctly if you have no interest in becoming smart yourself. Simple.

Re:Dear Every Corporate Tool in the Universe: (5, Insightful)

Kjella (173770) | more than 5 years ago | (#26301999)

Being too stupid to recognize your own shortcomings is also a form of stupidity. Or hubris, whichever is more appropriate.

Re:Dear Every Corporate Tool in the Universe: (2, Insightful)

Vanders (110092) | more than 5 years ago | (#26301619)

Apparently they only had the two drives: one mirroring the other. They had one single drives worth of data. Just think how little data that really is, for a moment. They could have bought a second hand tape drive from eBay and a box of new tapes for a couple of hundred dollars and had a complete backup solution going in one afternoon.

For want of a nail, the shoe was lost.
For want of a shoe, the horse was lost.
For want of a horse,the rider was lost.
For want of a rider, the battle was lost.
For want of a battle, the kingdom was lost,
And all for the want of a horseshoe nail.

Re:Dear Every Corporate Tool in the Universe: (4, Insightful)

mrchaotica (681592) | more than 5 years ago | (#26301777)

Hell, they could have spent $50 on a USB hard drive (i.e., half-assed it) and been better off!

Re:Dear Every Corporate Tool in the Universe: (3, Insightful)

Volante3192 (953645) | more than 5 years ago | (#26301623)

Never underestimate the beancounter's desire to save every cent possible. If your site's working perfectly fine, well, what's the point of having backups? Seriously, I see this happen all the time with small businesses. "Oh, it's never failed before, why do we need backups?" Then the server implodes.

Course, they then get pissed at us for not preventing it, but what do they expect us to do, shell out for a tape drive with our own cash? I think not.

Re:Dear Every Corporate Tool in the Universe: (1)

IMarvinTPA (104941) | more than 5 years ago | (#26301759)

Do the backups, then charge them back-costs on the service they didn't think they needed, but now do. If not, let them play with the disk recovery companies, etc.

IMarv

Re:Dear Every Corporate Tool in the Universe: (5, Funny)

slugtastic (1437569) | more than 5 years ago | (#26301791)

I'm more surprised that the site lived for 6 years without back-up. That's pretty hardcore.

Re:Dear Every Corporate Tool in the Universe: (5, Funny)

modmans2ndcoming (929661) | more than 5 years ago | (#26301857)

Screw that!! IT Departments are cost centers and have absolutely no benefit to the bottom line of a company... none at all... nope.

rm -rf / (5, Informative)

corsec67 (627446) | more than 5 years ago | (#26301341)

rm -rf /

That is one reason why mirroring isn't a backup, and why backups should ideally be off-line.

Re:rm -rf / (5, Funny)

Piranhaa (672441) | more than 5 years ago | (#26301941)

C:\>rm -rf /
'rm' is not recognized as an internal or external command,
operable program or batch file.

Everything's still running here...

Ouch (3, Informative)

scubamage (727538) | more than 5 years ago | (#26301357)

We do data hosting, and I can't imagine how catastrophic that would be. Jebus. Let this be an ultimate example of why numerous backups are needed. Always. Without question.

Re:Ouch (4, Insightful)

conureman (748753) | more than 5 years ago | (#26301577)

Or even one, stale, backup.

Re:Ouch (3, Insightful)

jabithew (1340853) | more than 5 years ago | (#26301879)

This story put the fear of god into me. The first thing I did since reading it is to back up the website I admin (for my dad) locally. I'd always assumed our host would have good backup, but that seems naÃve now.

Excellent! (5, Funny)

GravityStar (1209738) | more than 5 years ago | (#26301361)

Excellent! We can use their demise as yet another cautionary tale.

Re:Excellent! (5, Funny)

El Torico (732160) | more than 5 years ago | (#26301629)

Excellent! We can use their demise as yet another cautionary tale.

Ironically, it's more useful than the entire collection of blogs that they stored.

Re:Excellent! (1)

Volante3192 (953645) | more than 5 years ago | (#26301733)

With all these cautionary tales, you'd think someone out there would catch on...

Re:Excellent! (1)

Rearden82 (923468) | more than 5 years ago | (#26301843)

Exactly! These kinds of examples are ideal for demonstrating to PHB types that data loss isn't just some kind of theoretical once-in-a-billion-years kind of thing, and that backups actually serve a purpose.

I also find it amusing that they could apparently afford to blow money on nice shiny Mac servers, yet a tape drive (hell, even a $99 external hard drive or two from Best Buy) was out of the question.

Mirroring is not intended as a data backup (3, Informative)

zaibazu (976612) | more than 5 years ago | (#26301367)

It is an inexpensive protection against a total harddisc failure, but effective at this part. A software going rogue or a user deleting the wrong files can't be helped by it.

That's what backups are for (5, Interesting)

MBCook (132727) | more than 5 years ago | (#26301369)

It's really unfortunate that this happened. If they had simply had a backup snapshot of the DB they could have restored it. RAID only saves you from disk failures. It doesn't work on OS/user failures.

Unfortunately this is the kind of thing you tend to learn from experience (either yours or someone else). It's very easy to think "RAID 1 = disks are safe".

Just like a database cluster wouldn't have saved them. A clustering database can save you from load, or you can swap servers if a disk goes bad. But when someone issues "DELETE * FROM..." the other cluster nodes start to happily run the same thing and now you have 2 (or 3 or 10 or...) empty database boxes.

I hope those bloggers had a backup of some sort of their own.

Re:That's what backups are for (1)

LWATCDR (28044) | more than 5 years ago | (#26301589)

No it really isn't unfortunate this happened. It was going to happen sooner or later. The bad thing is that it didn't happen when there was less data to loose.
      What where they thinking?

Re:That's what backups are for (5, Insightful)

MBCook (132727) | more than 5 years ago | (#26301765)

My guess (and this is a guess, I'd never heard of the site before yesterday) is that this is some guy who started his own little site and it got bigger and bigger. Basically he never designed the backup, the system was just slowly pieced bigger and bigger until it got to it's current state.

The comments in the messages from the site's operator about the cost of the drive recover and thinking both drives just died at once indicate to me that this site was basically a hobby for him and he isn't experienced as an admin.

Re:That's what backups are for (1)

sdpuppy (898535) | more than 5 years ago | (#26301939)

some guy who started his own little site and it got bigger and bigger. Basically he never designed the backup, the system was just slowly pieced bigger and bigger until it got to it's current state

Would someone with mod points give this fellow a +5 ? I would say that your post describes is a very likely scenario.

Re:That's what backups are for (1)

sdpuppy (898535) | more than 5 years ago | (#26301841)

The bad thing is that it didn't happen when there was less data to loose.

Absolutely - they really needed to tighten up their data much earlier!

But grammatical pet peeves aside, you're absolutely right:

It was going to happen sooner or later.

I mean really - even if you don't know any better and you just spend a minute thinking about it, it so obvious that you can run into a situation like this with mirroring - even with just the thought: "what is an important file is deleted? " - never mind the whole shebang.

Oops - reminds me - better back up my har@^#&* .

Re:That's what backups are for (1)

Boron55 (1060136) | more than 5 years ago | (#26301671)

The right query would be "DELETE FROM..." (no star)

You just failed the database module exam. ;)

Seriously, I do this mistake too, from time to time.

Re:That's what backups are for (1)

SoloHan (1223138) | more than 5 years ago | (#26301821)

Argh, I just logged in to say the right one was DELETE FROM.... But as long as I am already logged on, I have to say that I once saw an SQL-like DBMS accepting the star.

Re:That's what backups are for (2, Interesting)

ergo98 (9391) | more than 5 years ago | (#26301691)

Unfortunately this is the kind of thing you tend to learn from experience

Even a moment of thought would have made it abundantly clear that this was not a backup situation, and it certainly should not require loss to pound it into someone's head.

They were clearly way over their heads if they thought this protected them from anything other than a single drive failure. More likely they were entirely aware of the risk, but decided to wing it anyways.

Filed away under "Shoulda known better". (0)

Anonymous Coward | more than 5 years ago | (#26301381)

Sad, very sad.

Noobs. No, really. (1)

urbanriot (924981) | more than 5 years ago | (#26301389)

Even the greenest IT employee knows that mirroring is to protect against hard drive failure and not software corruption. Obviously someone felt they knew better than people that actually know better, or someone didn't consult the right people. This is the end result. Tape, USB keys, disc backup... there's so many debatable methods of backing up that there's no excuse for this one.

Re:Noobs. No, really. (4, Informative)

emag (4640) | more than 5 years ago | (#26301695)

Even the greenest IT employee knows that mirroring is to protect against hard drive failure and not software corruption.

I only wish that were true. I've given up arguing with friends about this, who insist that their mirrors are good enough backups. I just stare at colleagues who think such, especially those who SHOULD know better. And I *know* coworkers are doing this @ work, too, and I'm just waiting for about 50TB of data to suddenly go missing...

Re:Noobs. No, really. (1)

etnoy (664495) | more than 5 years ago | (#26301931)

"There are two types of people in the world. Those who take backups, and those who have never had a disk crash/fs corruption/IDE driver bug/whatever"

El Oh El (3, Insightful)

greymond (539980) | more than 5 years ago | (#26301393)

That's all I can say at this. I'm really surprised that with all the users they had, they are so quick to say "everything is gone and we're giving up" instead of just starting over and maybe implementing protocol that would make sure this doesn't happen again.

Re:El Oh El (5, Insightful)

kurtmckee (870398) | more than 5 years ago | (#26301575)

I'm really surprised that with all the users they had, they are so quick to say "everything is gone and we're giving up"

Considering how complete and unrecoverable the loss is, they have no idea who their users are. The accounts would have to be recreated from scratch, but who would try? Their users have no reason to ever trust them again. Journalspace would have a difficult time wooing back their original users, and no new user would seriously consider using them.

Bowing out is the only recourse, but I'm glad they're considering releasing their source code.

Re:El Oh El (1)

Kjella (173770) | more than 5 years ago | (#26301627)

That's all I can say at this. I'm really surprised that with all the users they had, they are so quick to say "everything is gone and we're giving up" instead of just starting over and maybe implementing protocol that would make sure this doesn't happen again.

With such a backup policy, is it certain they even have the system? Clearly they don't have a development server with a recent database copy. Sounds to me like they're totally screwed and by the time they got something new together all their users would be gone and all of them would loudly warn anyone else from coming back. I think it shows a good degree of self-insight to simply say tha this screwup is beyond fixing. Even reforming under a new name would be better than the stigma of this disaster.

Re:El Oh El (3, Funny)

spuke4000 (587845) | more than 5 years ago | (#26301933)

Indeed. Everyone knows that when you drive your company into the ground through incompetence you don't give up! You go to Washington to get your bail out. That's the American way.

Bizarre (1)

Courageous (228506) | more than 5 years ago | (#26301395)

It's bizarre that anyone would ever, under any circumstances, consider a "mirror" to be a backup. Mirrors automatically replicate errors, including the human variety.

Point in time snapshots might be a sort of lazy man's backup, but even then, consider the possibility of fire or disaster, and not having some sort of second location is just plain foolhardy.

C//

Mirroring is *NEVER* a "backup" (1)

sisukapalli1 (471175) | more than 5 years ago | (#26301397)

Mirroring is more of a safeguard for hardware failure. It does not replace backup. Both serve entirely different purposes.

If they had at least some online rsync'ed backups from at least a day/week/month/etc., that would be a backup.

The site *didn't* have any backups, and they pay the price.

S

Thank you (2, Funny)

ari_j (90255) | more than 5 years ago | (#26301445)

This is fascinating and altogether newsworthy. I had never before thought of this. I am very pleased, indeed, that kdawson engaged his most finely-honed editorial faculties to post this article to the front page, as it is not only stunning and fascinating in substance but also rather eloquently written.

It's so easy even a caveman can do it. (1)

GPLDAN (732269) | more than 5 years ago | (#26301467)

With the proliferation of snapshot technology (for instance Parallels on the Mac contains the ability to snapshot the VM and restore any snapshot) plus the ability to back the VM up to a USB2 disk that you can just plugin and then pull and put in a safe place - having point in time backups of servers has become so trivial and easy that it is unbelieveable that anybody would run a buisiness without it.

A leading SAN/NAS vendor (no commercials here) has a solution for SMB that includes a disk shelf and controller integrated together with 4TB of storage for about $14k. Maybe closer to $20k if you license the Linux and Windows backup agents to the SAN. Really - ANYBODY can do this.

If you're really THAT hard up for operating cash, get a 2U server with an integrated tape drive at the very very very least.

Inconceivable? (3, Funny)

LSD-OBS (183415) | more than 5 years ago | (#26301477)

I do not think it means what you think it means.

How hard is it to remember: (4, Insightful)

computersareevil (244846) | more than 5 years ago | (#26301497)

Mirroring: High availability
Backups: High reliability

The rules of backups (5, Informative)

Anonymous Coward | more than 5 years ago | (#26301511)

The rules of backups:

1. Backup all your data
2. Backup frequently
3. Take some backups off-site
4. Keep some old backups
5. Test your backups
6. Secure your backups
7. Perform integrity checking

Re:The rules of backups (0)

Anonymous Coward | more than 5 years ago | (#26301597)

8. ???
9. Profit !

Re:The rules of backups (2, Funny)

Anonymous Coward | more than 5 years ago | (#26301633)

1. Do not talk about backups
2. DO NOT TALK ABOUT BACKUPS

Only 2 drives? (4, Insightful)

lalena (1221394) | more than 5 years ago | (#26301527)

Maybe I could understand that there might be issues with backing up live databases, and they didn't want to deal with it. Still not an excuse.
BUT, according to the site "the server which held the journalspace data had two large drives in a RAID configuration". Only TWO drives.
All they had to do was pull one of the drives, replace it, and lock up the original off site. In a couple of hours the drives would have been mirrored again.

To the HR department (5, Funny)

squeegee_boy (319210) | more than 5 years ago | (#26301533)

Important note: don't hire the IT dude with Journalspace.com on his resume.

Re:To the HR department (3, Insightful)

DaveV1.0 (203135) | more than 5 years ago | (#26301583)

The only problem with that idea is that it may not have been the IT guy's decision to save money by not having a true backup system. I have seen companies skimp on backup systems because they thought their RAID system was enough.

Re:To the HR department (2, Insightful)

ergo98 (9391) | more than 5 years ago | (#26301731)

A better backup solution needn't cost much, or even anything. Simply FTPing to your own home machine on occasion would have been a millionfold improvement (given the popularity metrics, I don't think this was like a staffed operation or anything. Just a guy or two)

A Great Disturbance ... (0)

Anonymous Coward | more than 5 years ago | (#26301541)

This morning I felt a great disturbance in the Blogosphere, as if tens of voices suddenly cried out in terror.

OS X Server (-1, Troll)

Anonymous Coward | more than 5 years ago | (#26301561)

Their announcement blames either malicious intend or OS failure. They where running OS X. I can only start to imagine the posts on slashdot had this been a Windows server! ;)

Virtual Machines for Backup happiness (1)

MosesJones (55544) | more than 5 years ago | (#26301591)

One thing that I've been switching to recently has been backing up not just the disk or data but creating a full virtual machine backup of the server. Space wise this can be a big hit so incremental data backups are done daily, with a full VM hit once a month alongside the full data dump. Now I'm shifting to doing a daily VM in addition which gives me the last close of play.

The reason for this is restore time, if it takes a few days to restore then its a right pain (or for some companies fatal) but a VM restore I can fire up on temporary kit in a matter of an hour or less and give a downgraded service while we patch up the full servers.

A lesson for admins, and users too (4, Insightful)

gzipped_tar (1151931) | more than 5 years ago | (#26301595)

No doubt this incident is the result of the admin's fault. He's been confusing mirroring and backup and carried on the mistake until it's too late, as pointed out in other comments.

Now what about a user's angle? The morale is you can never think your data is safer when it's "in the cloud". If you value your blog and your readers, you *should* save a copy of your work as well as the readers' info, *locally*, somewhere you have control over.

There's no place like $HOME.

Re:A lesson for admins, and users too (4, Insightful)

djmurdoch (306849) | more than 5 years ago | (#26301781)

And a corollary to the parent's good advice: if you can't easily get a complete copy of your work, find another host. Manual one-by-one downloads don't cut it.

Re: A lesson for admins, and users too (1)

lalena (1221394) | more than 5 years ago | (#26301863)

If I had mod points, I would mod this up.
No one thinks about the user's responsibility to backup their data on the web.

I assume that almost no one backs up their blog entries, but how many people would lose all of their friend's email addresses if they lost their email account. How about their family pictures hosted online.

Even if someone did have a popular blog and backed up all of their data, they have still lost any fans, links, page rank... that they had accumulated.

Really!? (1)

k3v1n (262210) | more than 5 years ago | (#26301679)

Really hard to imagine something like this, especially when database backups are SO EASY. mysqldump, gzip, scp to a different machine or upload to S3. gzip'ed database dumps take up so little space!

Re:Really!? (1)

daveime (1253762) | more than 5 years ago | (#26301849)

Unless you fancy killing your production server for 45 minutes while he dumps out a 1.5GB table with numerous indexes (thus locking the table for any writes), I'd suggest this policy.

Replicate the database onto a second server, and on that server, do all of the above. Believe me, it saves a lot of pain and disk grinding on your production server.

Hoax (0, Troll)

jamesl (106902) | more than 5 years ago | (#26301683)

This is so ignorant that it must be a hoax.

Google Cache to the rescue? (0)

Anonymous Coward | more than 5 years ago | (#26301689)

Perhaps the people at Google can help out?

Professionalism (1, Informative)

theskipper (461997) | more than 5 years ago | (#26301699)

From TFL:

The data server had only one purpose: maintaining the journalspace database. There were no other web sites or processes running on the server, and it would be impossible for a software bug in journalspace to overwrite the drives, sector by sector.

The list of potential causes for this disaster is a short one. It includes a catastrophic failure by the operating system (OS X Server, in case you're interested), or a deliberate effort. A disgruntled member of the Lagomorphics team sabotaged some key servers several months ago after he was caught stealing from the company; as awful as the thought is, we can't rule out the possibility of additional sabotage.

First, it's somewhat lame/unprofessional to list "sabotage" as a possibility. Even if it's the strongest possibility. Adding the OSX comment and that a bug in their code is impossible is even lamer.

More importantly, if the key servers were sabotaged months ago, the first thing that I'd want to do is make a full image stored in multiple offsite locations. Ignorance of the RAID/backup issue is one thing, but knowing that the sabateur could have sprinkled the db with crap is even scarier.

Smells like there's more to the story than this. Or not.

Re:Professionalism (1)

jav1231 (539129) | more than 5 years ago | (#26301985)

Exactly. "Technical incompetence or ingnorance about security. We're not sure which." Okay so which makes you look better?

No Archive.org either (5, Informative)

computersareevil (244846) | more than 5 years ago | (#26301709)

They also purposely blocked archive.org via a robots.txt exclusion, so the bloggers can't use that to try and recover some of their blogs.

Re:No Archive.org either (1)

ds_job (896062) | more than 5 years ago | (#26301871)

I was going to post that they had enough nous to get a robots.txt but not to actually maintain their system. I wouldn't want that company on my CV even if I was the janitor. This is going to be a millstone for anyone who was involved in this going back from day 2 (I can just about forgive them for being stupid on day 1)

Re:No Archive.org either (1)

42forty-two42 (532340) | more than 5 years ago | (#26301901)

They blocked every spider except for a short list of whitelisted ones. There are still some random pages in google's cache if you search site:journalspace.com

There is a denial going on (5, Insightful)

hwyhobo (1420503) | more than 5 years ago | (#26301715)

In today's world where primary storage and protection storage are well-defined, and where entire industry grew around it (examples: NetApp, Data Domain), one is hard-pressed to understand the reason for such a debacle. The reading of the note referred to in the article [journalspace.com] leads me to believe, unfortunately, that Journalspace's IT department did not understand the difference.

It is sometimes considered a bad form to say something bad about fellow techies. We prefer to look for 'outside' causes. Still, to learn and avoid the same problems in the future, one has to admit his mistakes first. This paragraph from the Journalspace's page:

The value of such a setup is that if one drive fails, the server keeps running, using the remaining drive. Since the remaining drive has a copy of the data on the other drive, the data is intact. The administrator simply replaces the drive that's gone bad, and the server is back to operating with two redundant drives.

makes me believe there is a denial going on.

Someone needs to be FIRED (4, Funny)

spitek (942062) | more than 5 years ago | (#26301729)

You pay your infrastructure people to maintain business, continuity I mean the tittle of this post made me go, "Really, no shit" That's like systems admin 101! If the admin was aware then the manager that didn't listen needs to be fired. If the manager listened and they are just run by retards then they got what they deserve. You'd think 17,000 visitors a month would be worth enough to do it right, in add revenue alone. The cost of a consumer machine running linux with a few TB's of SATA space - $1200 How much the company paid to have a system's admin play video games all day - $50,000 The cost of a 17,000 vistor a month site going down because they had no data base backups - Priceless.

Re:Someone needs to be FIRED (2, Funny)

macbuzz01 (1074795) | more than 5 years ago | (#26301881)

I think they are all fired now.

Wow. Like seeing an amazing car wreak on video. (1)

1_brown_mouse (160511) | more than 5 years ago | (#26301799)

I don't know how you can run a business without protecting your bread and butter.

You have insurance on your car. Hopefully some health insurance.

How about some Database Insurance? Hell, $150 for a simple 1TB USB HD?

Can you be that incompetent? Why yes, I guess you can be.

Just give up? (1, Troll)

girlintraining (1395911) | more than 5 years ago | (#26301883)

Slashdot must be populated mostly by engineers and programmers that work on the software side of things, because nobody's considered mentioning a data recovery service, instead giving the "You're doooooooomed! You should have paid your IT people more" line. How very kind everyone here is of other people's technical mistakes. Because none of us have ever seen a bunch of dot files in a directory and typed "rm -rf .*" and then cried after or screwed up some production server with a "minor change"...

In other news... (1)

talon77 (410766) | more than 5 years ago | (#26301897)

The sky is blue, the earth is round, and U2 sucks.

Thanks.

Source code may be released! (1)

nih (411096) | more than 5 years ago | (#26301911)

We're considering releasing the journalspace source code to the open source community

I can't wai

Mirroring (5, Insightful)

jav1231 (539129) | more than 5 years ago | (#26301917)

See mirroring is like...well a mirror. If you stand before one and stick a fork in your eye your mirror-image does the same. In real time. Analogies are there for a reason.

Even rsync mirroring would be better (0)

Anonymous Coward | more than 5 years ago | (#26301921)

Blimey. When I saw "mirroring" in the title, I thought it was going to talk about rsync. But raid mirroring being your entire backup strategy? That's special.

Look at the bright side (0)

Anonymous Coward | more than 5 years ago | (#26301923)

The first real backup they do is going to run really, really fast.

Running Mac OS server, seriously?! (0)

Anonymous Coward | more than 5 years ago | (#26301943)

Is this a joke? Did I read that correctly? The server was running on Mac OS X server? No wonder they are dead now! Only a Macretard could make such decisions.

Data recovery services? (0)

Anonymous Coward | more than 5 years ago | (#26301945)

They should still be able to send the drives to a data recovery service and recover a lot, if not all, of the data that was overwritten, right?

Liberal Bloggers (0, Troll)

Nom du Keyboard (633989) | more than 5 years ago | (#26301949)

If it's only liberal bloggers content...

If you can't be a good example... (1, Funny)

Anonymous Coward | more than 5 years ago | (#26301957)

"If you can't be a good example, you'll just have to be a horrible warning."

Catherine Aird
Quoted in the Book Practical UNIX & Internet Security

RAID has never been a backup solution (1)

David_Hart (1184661) | more than 5 years ago | (#26301969)

RAID is used for three purposes, to speed up access to data, to create a volume larger than any one disk, and to mitigate against disk failure. It has never been, nor ever will be a backup solution on its own. A backup solution involves making a copy of the data to independent storage, be it tape, disk, etc. Ideally a copy will also be sent offsite in case of fire, etc. In addition, the restore process needs to be tested on a seperate restoration box to make sure that the backup process and the media are working correctly.

Personally, I am not sure why this is a story on Slashdot as everyone here should have at least this basic understanding of how to protect their data.

David

I Can't belive it.... (1)

s0litaire (1205168) | more than 5 years ago | (#26301971)

all these posts and not ONE OS X / Mac-Fanboy joke!! not even a "They did have a backup solution: they prayed to their God Jobs every night to protect the data" Or "Apple servers are to cool to corrupt!" Come on guys! i know it's Friday night but you can't still be hungover from New Years... :D:D

The simplest things (1)

kylegordon (159137) | more than 5 years ago | (#26301979)

Many years ago, I had a minor disk failure. However, it resulted in the loss of the root directory inode. Everything within the directory (I think it was mounted as /var/) vanished... and mirroring would never have saved it.

sjmurdoch [cam.ac.uk] very kindly wrote python and debugfs magic that recovered about 95% of the structure and files, but it was a lesson learned against using mirroring as a form of backups...

So, what was it worth in dollar terms? (1)

jcr (53032) | more than 5 years ago | (#26301983)

Anyone have an estimate of how much equity in this business just vanished?

-jcr

Sabotage (1)

phorm (591458) | more than 5 years ago | (#26301995)

A disgruntled member of the Lagomorphics team sabotaged some key servers several months ago after he was caught stealing from the company; as awful as the thought is, we can't rule out the possibility of additional sabotage.

Seems to be quite possible that a previous admin did this. In this case, the only real backup would be something disconnected (and tested). Risk factors are otherwise still high:

a) RAID: Data overwrite, controller failure, PC failure,multiple disk failure, data corruption/RAM/drive issues, deliberate erasure, out-of-space related errors

b) RAID+backup disk: Controller failure, major hardware malfunction (power surge, explosion).

c) On-site sync: Site-wide catastrophe (explosion, flood,electrical surge, etc), network issues (usually temporary)

d) Off-site sync: Deliberate sabotage (privilaged user), which could be somewhat offset using a pull-based backup rather than a push (backup server having very limited access and none from the server being backed up). Can also be redundant servers in different locations

e) Permanent storage media: Even this could go bad if the tapes/disks/etc get damaged or demagnetized etc. Tape backups can also be slow'ish (incremental-differential backups a bit faster to do)

There is no silver bullet, but relying on just RAID ignores a huge number of potential failures, ESPECIALLY after existing issues with attempted sabotage. No as it was the database that was nuked and not necessarily the webserver/etc, I'd be interested in knowing what logs were in place and if there traces of sabotage. A login at an odd time, but a weird account, or by a script set to execute on say "01-01-2009 00:00:01" would be a good start.

Wonder if a data-recovery firm might be able to get something bad on a DB if it's just been erased. Might be easier than if it were actually overwritten with bad data...

I can only say Gentoo-wiki.com (0)

Anonymous Coward | more than 5 years ago | (#26302011)

Gentoo-wiki.com is a awesome site. However, the guy who runs it either has no backup plan or really a lot of bad luck. I think it is at least the third time the main site says "Gentoo-Wiki recently had it's database lost; this is the rewrite of the site.".

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>