Beta

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Tech Magazine Loses June Issue, No Backup

CmdrTaco posted more than 6 years ago | from the happens-to-everyone dept.

Data Storage 245

Gareth writes "Business 2.0, a magazine published by Time, has been warning their readers against the hazards of not taking backups of computer files. So much so that in an article published by them in 2003, they 'likened backups to flossing — everyone knows it's important, but few devote enough thought or energy to it.' Last week, Business 2.0 got caught forgetting to floss as the magazine's editorial system crashed, wiping out all the work that had been done for its June issue. The backup server failed to back up."

cancel ×

245 comments

Sorry! There are no comments related to the filter you selected.

After the swearing stopped. (5, Funny)

AltGrendel (175092) | more than 6 years ago | (#18955113)

The first words from management were "You're kidding me, right?"

Then the swearing started again.

Re:After the swearing stopped. (1)

xzvf (924443) | more than 6 years ago | (#18955297)

That's what I said when Byte went out of business and instead of refunding my subscription they sent me "Business 2.0". Multiple letters latter, they stopped sending the piece of crap magizine, but I never got my refund. It was only like $10, but it was mine.

this happened to me too! (1)

yagu (721525) | more than 6 years ago | (#18956135)

That's what I said when Byte went out of business and instead of refunding my subscription they sent me "Business 2.0". Multiple letters latter, they stopped sending the piece of crap magizine, but I never got my refund. It was only like $10, but it was mine.

OMG, this happened to me too! Instead of Business 2.0, they switched me to PC Magazine, one of the biggest and most annoying MS shills. Not only had I lost a cherished subscription, but it had been replaced with the very antithesis of what I had subscribed to. And!, I had a three year paid subscription.... I tried unsuccessfully to get a refund, and eventually just resorted to tossing the new PC Magazine issue when it arrived. What a ripoff!

Re:After the swearing stopped. (1)

sjwest (948274) | more than 6 years ago | (#18955411)

We do backups, but i dont floss. Quite never saw the point of flossing. - perhaps they should have flossed less.

Re:After the swearing stopped. (0, Offtopic)

maxume (22995) | more than 6 years ago | (#18956203)

Off topic, but read up on the surgery for periodontitis(that's what comes after gingivitis. Basically, flossing be good for your gums).

Re:After the swearing stopped. (5, Insightful)

Anonymous Coward | more than 6 years ago | (#18955463)

Actually that is what you get for having Geek Squad as your outsourced IT staff.

honestly, they CANT have competent IT. The FIRST thing you do in the morning is check the backups.

I have a HP sdat jukebox here and I STILL check the backup logs to make sure the backup and verify succeeded last night. if they dont I mirror the important files right away and then run a manual backup to not lose the last 24 hours of backup.

I hope that Business 2.0 learned that paying top $$$ for competent IT is a good idea and they should run a article about it.

Re:After the swearing stopped. (1)

TrippTDF (513419) | more than 6 years ago | (#18955511)

Agreed- this is just a no brainer.

What does it say about a tech magazine that can't even handle the basics of technology?

Re:After the swearing stopped. (4, Insightful)

IdleTime (561841) | more than 6 years ago | (#18955925)

I am not surprised.

There is not a week going by without me getting an issue from one of out regular analysts with question about how the customer can salvage their data because they don't have a backup. My standard answer is that we may be able to save some data, but it's going to cost a lot of $$$. And I also say: "When you don't have a backup, you have either deemed that you can easily recreate the data or that they are not important for the company"

And these are not mom&pop companies but big multi million/billion dollar companies.

Re:After the swearing stopped. (4, Insightful)

Auntie Virus (772950) | more than 6 years ago | (#18955991)

I have a HP sdat jukebox here and I STILL check the backup logs
HP DAT? You'd better do more than check the logs. A test restore (if your users don't already test for you by deleting files) at least a few times a week might save your butt one day. Actually DAT or not, test restores are a must. Logs lie.

Re:After the swearing stopped. (1)

Nutria (679911) | more than 6 years ago | (#18956167)

Actually DAT or not, test restores are a must. Logs lie.

If the logs are lying to you, then that's a prime indication that you're using the wrong software.

Re:After the swearing stopped. (1)

JATMON (995758) | more than 6 years ago | (#18956367)

It is also a very good practice to test the recovery of the backups on a regular basis. Just because the logs say that the backups ran successfully does not necessarily mean that the restore will work.

Re:After the swearing stopped. (1)

Nutria (679911) | more than 6 years ago | (#18956667)

It is also a very good practice to test the recovery of the backups on a regular basis. Just because the logs say that the backups ran successfully does not necessarily mean that the restore will work.

Sure, if you have the capacity.

And don't reply "disks are cheap", because there's more to a disk subsystem than buying a few 500GB drives from NewEgg.

Re:After the swearing stopped. (0, Redundant)

Hoi Polloi (522990) | more than 6 years ago | (#18956445)

At what point did Nelson Muntz [wikipedia.org] stick his head in the room and say "Ha Ha!"?

With this much free advertising (5, Funny)

Anonymous Coward | more than 6 years ago | (#18955129)

who needs a magazine?

Re:With this much free advertising (0)

Anonymous Coward | more than 6 years ago | (#18955201)

My God! It's full of advertising!

*zap*

Oblig. (0, Redundant)

cheese-cube (910830) | more than 6 years ago | (#18955131)

Ha Ha!

We've all been there. Don't be too pious, here. (2, Insightful)

ScentCone (795499) | more than 6 years ago | (#18955137)

Maybe not so bad as losing your entire monthly product, per se... but it does happen. I'll bet their accounting, HR, and other back office systems are probably fine. This stuff is always ugliest at the department server level in smaller operations. I'll bet they get some good Mea Culpa 2.0 editorials out of it, though.

Re:We've all been there. Don't be too pious, here. (1)

Thyamine (531612) | more than 6 years ago | (#18955567)

I agree (about not being pious about it). No matter how often I tell clients to backup files/software, or we sell them solutions to do it, it never fails that people lose important data. Including myself at home. You always assume things are running fine, or think 'hey, I'll just backup this weekend'. Then, system crash, power outage, etc, etc.

It happens, it will continue to happen, not much else to say really.

Re:We've all been there. Don't be too pious, here. (1)

Silver Sloth (770927) | more than 6 years ago | (#18955761)

Maybe not so bad as losing your entire monthly product, per se... but it does happen
If it happens to me then I'm looking for a new job. Honestly, losing 24hrs worth of data is the worst that should happen. Well, OK, maybe 48hrs if you happen to be unlucky enough to have a crash just before the backup after a failed backup (if you see what I mean) And if the data is that important then a suitable RAIDed disk array will sort things out. (we use mirrored RAID5 arrays in separate data centres but I work for a bank)

No, I'm not being pious, I would guess that the loss in revenue from a lost edition would cover the costs of a sufficiently resilient system that data loss would be next to impossible. In a business situation loss of data is nearly always unacceptable.

Re:We've all been there. Don't be too pious, here. (1)

daeg (828071) | more than 6 years ago | (#18956009)

There are many, many industries where losing any data is unacceptable. Banking and health care (which I work in) being the front runners in my book. I can't imagine losing even 15 minutes' worth of patient data, or accidentally deleting test results, or losing even a single piece of a record. Not only is it potentially dangerous (for instance, losing the allergy information for a patient), it's bad service. Who wants to get their blood re-drawn because of an IT problem?

Nelson Muntz (5, Funny)

erroneous (158367) | more than 6 years ago | (#18955139)

Some stories should just come with Nelson Muntz [wikipedia.org] sound files embedded.

Ha-ha!

Re:Nelson Muntz (1)

Kierthos (225954) | more than 6 years ago | (#18955289)

Heck, I'm wondering why there's not a haha tag on this already.

Re:Nelson Muntz (0)

Anonymous Coward | more than 6 years ago | (#18956545)

Heck, I'm wondering why there's not a haha tag on this already.
I think the haha tag got blacklisted from the front page because it was showing up on so many stories.

err... (3, Insightful)

cosmocain (1060326) | more than 6 years ago | (#18955141)

HAHA!

*coughs

TFA:

Business 2.0 never had to rely on their backup software until that day, which is why they probably did not realize that it was either obsolete or dysfunctional.


sorry, their MAIN problem is not in any way a dysfunctional backup system. ever heard of verifying backuped data?

Re:err... (5, Informative)

Lumpy (12016) | more than 6 years ago | (#18955531)

hell with that. ever heard of competent IT staff? why has their CTO not been fired yet?

honestly though, talking management into backup solutions is like pulling teeth, then they blame you for not having it in place when the failure does happen.

Last place I worked at we were using 4 year old DLT tapes because management was too stupid and cheap to buy new ones.

"we will buy new when those fail" is what we were told.

Re:err... (3, Informative)

dal20402 (895630) | more than 6 years ago | (#18955669)

/grabs hammer...

*bang* *bang* *bang*

Oops, it looks like a couple of those DLT drives are running into problems. We need replacements. Did you see what happened to Business 2.0?

Re:err... (1)

metamatic (202216) | more than 6 years ago | (#18956495)

No, it's worse than that. He said the *tapes* are 4 years old. They've been using the same *tapes* for backup for 4 years.

That's like playing Russian Roulette with your data. Once a tape gets that old you might as well not bother backing up to it at all.

Re:err... (4, Funny)

morgan_greywolf (835522) | more than 6 years ago | (#18955579)

ever heard of verifying backuped data?


Errr...uhh....umm...'verifying'? Uh, I'll be right back!

Re:err... (-1)

Anonymous Coward | more than 6 years ago | (#18955609)

Curse you merciful Poseidon!

Re:err... (5, Funny)

radtea (464814) | more than 6 years ago | (#18955791)

sorry, their MAIN problem is not in any way a dysfunctional backup system. ever heard of verifying backuped data?

I'm sure they've heard of it, in a conversation that went something like this:

IT Guy: We need a system for verifying our backups.

Suit: How come? Don't the backups work?

IT Guy: We need to be sure that if there is a failure, the backups will be ok.

Suit: But they're just copies, aren't they? I copy files all the time and it never goes wrong.

IT Guy: This is a little more complicated than that.

Suit: How hard can it be?

IT Guy: Well, I was thinking we might need to hire a part-timer just to take care of backups and verification.

Suit: But we've never had a failure! Sounds like empire building to me. I know that's what I'd be doing in your position. Nice try. We'll keep the backup system the way it is, thanks.

IT Guy: But..!

Suit: Moving on to the next item on the agenda... ok, Executive Bonuses!

Re:err... (0)

Anonymous Coward | more than 6 years ago | (#18955953)

Verifying? That's simply not an option for many people. With our current Windows garbage backup software, we can't because the backup software can only be installed on the production systems since it has a serial number and a required Internet registration process (think something like Windows XP or many of the newer pieces of Microsoft software). That also means in order to be allowed to use the restoration software we have to get Windows going and have a working Internet connection to be allowed to register the software. We can't test a restore to one of the systems with the backup software because it doesn't understand the concept of restoring to a different location. So, we have almost $200k worth of software, drives, and tapes that we can't verify. That's the way the Windows world works. Of course the IT manager is very happy about the situation becase the constant Windows problems means that he and his many employees stay busy. Since he now has four dozen employees to manage our 15 servers, he makes a lot of money.

We used to use SunOS and tar. Of course it was possible to verify backup tapes done with tar onto a production system when you delete the leading / on the restore and make the destination dir something other than the production dir. We could also restore to development systems because tar is freely available. It's been over a decade since we switched from SunOS, but I still really miss it. At that time we had more servers, and I was the only IT employee. Now there's 49 people managing fewer servers. We do more with our servers and have more employees, but the amount of manpower required to keep Windows running is just ridiculous.

Re:err... (1)

Princeofcups (150855) | more than 6 years ago | (#18956263)

>sorry, their MAIN problem is not in any way a dysfunctional backup system. ever heard of verifying backuped data?

"Yes, we guarantee 100% that your data is being backed up. Look at all those tapes going offsite. You need a recovery? I dunno. We never tested that."

jfs

Nelson Says (-1, Redundant)

riffzifnab (449869) | more than 6 years ago | (#18955149)

I think I speak for most of us when I say:

Ha ha

They probably still have most of it (4, Insightful)

ZachPruckowski (918562) | more than 6 years ago | (#18955153)

I imagine that they still can resemble a lot of it from other files - they should still have all the layout pieces for one, and all the authors ought to have at least rough drafts of their stories on their personal computers. The deadline's screwed, but they can probably get it out a few weeks late (or in July, depending on how often they normally publish).

Re:They probably still have most of it (1)

chiskop (926270) | more than 6 years ago | (#18955227)

From the article:

Luckily for them copy-edited text of the lost issue has been mailed to the lawyers to get legal clearance. The page layouts had to be totally redone from scratch.

Re:They probably still have most of it (1)

Threni (635302) | more than 6 years ago | (#18955245)

Yeah, great, that's the content - now how about the advertising? That's where they make their money.

Re:They probably still have most of it (3, Funny)

Red Flayer (890720) | more than 6 years ago | (#18955479)

Yeah, great, that's the content - now how about the advertising? That's where they make their money.
Editorial department server content was lost. Advertising content is normally handled by the production department.

I think we can all relax and rest assured that the June issue of Business 2.0 will have all its intended advertising.

Why isn't this a default (1, Interesting)

Anonymous Coward | more than 6 years ago | (#18955155)

Why isn't it a default for an OS to ask where the backups should go when it is installed? Backups are like any other kind of security ... necessary. You shouldn't have to hunt for the instructions on how to back-up, they should be in your face.

Re:Why isn't this a default (3, Informative)

metamatic (202216) | more than 6 years ago | (#18956533)

Why isn't it a default for an OS to ask where the backups should go when it is installed?

Wait for OS X 10.5 and "Time Machine".

What we're missing. (0)

Anonymous Coward | more than 6 years ago | (#18955157)

HOW TO TAKE YOUR IDEA FROM CONCEPT TO $30 MILLION - OVERNIGHT

Oh God! I'm sure we're missing soooo much!

HAHAHA (2, Funny)

Anonymous Coward | more than 6 years ago | (#18955183)

HAHAHA

Rag (2, Funny)

BigDumbAnimal (532071) | more than 6 years ago | (#18955195)

No June issue?

That's OK, nobody reads Business 2.0 anyway.

Re:Rag (1)

Fred_A (10934) | more than 6 years ago | (#18955675)

Read ? Well, that's not a problem if you need something to read. But what am I going to line the litter box with ??

Re:Rag (1)

Jim Hall (2985) | more than 6 years ago | (#18955831)

That's OK, nobody reads Business 2.0 anyway.

Well, certainly not anymore. :-)

Re:Rag (5, Insightful)

UbuntuDupe (970646) | more than 6 years ago | (#18956227)

nobody reads Business 2.0 anyway.

I wish. I wish people didn't read Time, either (the publisher), but they do. Time's writing style is the dumbed down, try-to-be-hip crap I wouldn't have gotten away with in sixth grade. Seriously. Like I said before [slashdot.org] , to understand why its writing is like fingernails on a blackboard for me, consider how the same information would be conveyed by two sources:

8-year-old: "6 divided by 3 is 2."

Time magazine: "Okay, imagine you've got a half-dozen widgets, churned out of the ol' Widget Factory on Fifth and Main. Now, say you've gotta divvy 'em up into little chunklets -- a doable three, let's say -- and each chunklet has the same number that math professor Gregory Beckens at Overinflated Ego University calls a 'quotient'. The so-called 'quotient' in this case? Dos."

Based on how that post got modded, I'm not alone in this.

What was the nature of the crash? (2, Insightful)

91degrees (207121) | more than 6 years ago | (#18955197)

It seems unlikely to have crashed in such a way that a data recovery specialist would be unableto get most of the data back.

But whatever the case - there is a useful lesson here. Make sure your backups are backing something up.

Re:What was the nature of the crash? (1)

EveryNickIsTaken (1054794) | more than 6 years ago | (#18955225)

Sounds like the editor dropped his iBook.

Re:What was the nature of the crash? (3, Insightful)

Chris whatever (980992) | more than 6 years ago | (#18955349)

Hum!!! Unless you are there looking at the data being backed up there is no way unless you get notification from your system that it has completed.

usually that is the case but it has happened when one of my backup failed one night and someone needed a file restores from the previous day, if that company never checked it's backup or never configure some kind of noticaition upon failiure or success then they are very lame

The laws of the universe (2, Funny)

pseudosero (1037784) | more than 6 years ago | (#18955231)

Hold true, once again.

Re:The laws of the universe (1)

Architect_sasyr (938685) | more than 6 years ago | (#18955381)

Never underestimate the power of human stupidity eh...

This happens all too often, it's funny because it was Business 2.0, but would we all be laughing if it was the main Ubuntu repository (multi-site distributions are conveniently ignored for the purpose of this post)? Or [insert OS/code here]? I know I'd be pretty pissed if this happened to my kit.

Let this be a lesson to you all... the laws of the universe DO hold true... and every man or woman can be struck down and proven wrong.

Re:The laws of the universe (0)

Anonymous Coward | more than 6 years ago | (#18955509)

We'd all be laughing our asses off if it were the main Microsoft Windows repository though.

High profile SNAFUs (5, Insightful)

Rob T Firefly (844560) | more than 6 years ago | (#18955323)

This reminds me of the recent uproar over a car crash involving the New Jersey governor. He was critically injured because he wasn't wearing his seatbelt, and people freaked, asking what sort of role model he could possibly be. I argued that he was an awesome role model, because sometimes people need to see a mistake end badly for someone else before they'll do what's necessary to protect themselves from making the same mistake. Seeing a high-profile magazine get hit like this can do the same for backup slackers the world over.

I don't know about you people, but after reading this (and giving it the "haha" tag) I'm going home and catching up on a couple of backups I've been slacking off on for a while.

Mistakes (1)

catdriver (885089) | more than 6 years ago | (#18955571)

Mistakes [despair.com]

"It could be that the purpose of your life is only to serve as a warning to others."

Didn't you read the article? (1)

funkdancer (582069) | more than 6 years ago | (#18955697)

How about doing a restore practice run whilst at it? Just kidding. Sort of. ;)

Re:Didn't you read the article? (1)

Nutria (679911) | more than 6 years ago | (#18956553)

How about doing a restore practice run whilst at it? Just kidding. Sort of. ;)

You pony up the $300,000 (Large Systems cost a heck of a lot more than Windows servers) for a QA system and I'd leap at the chance to test the backups and procedures.

Re:High profile SNAFUs (1)

CrackedButter (646746) | more than 6 years ago | (#18956243)

... and I'm going home to floss!

Re:High profile SNAFUs (1)

SQL Error (16383) | more than 6 years ago | (#18956253)

I argued that he was an awesome role model, because sometimes people need to see a mistake end badly for someone else before they'll do what's necessary to protect themselves from making the same mistake.
And then he was caught speeding [nytimes.com] on the way home from hospital.

Re:High profile SNAFUs (5, Insightful)

Hoi Polloi (522990) | more than 6 years ago | (#18956521)

I wouldn't use the term "role model" for things like that. I'd say "examples" is the better word. The governor was an example of what NOT to do.

Re:High profile SNAFUs (1)

Rob T Firefly (844560) | more than 6 years ago | (#18956649)

Very good point, that's a much better way to put it.

Pr0n still on the way right? (2, Funny)

minotaurcomputing (775084) | more than 6 years ago | (#18955329)

As long as those magazines that come in the smarmy black plastic covers still arrive I can't complain.
-m

What software? (1)

reed (19777) | more than 6 years ago | (#18955377)

Anyone know what "editorial software system" (CMS) they use?

How does this actually happen? (4, Interesting)

Orange Crush (934731) | more than 6 years ago | (#18955397)

There aren't a lot of ways for a machine to "crash" that loses all its data. Even a lightning-fried hard drive can have its platters removed by a data recovery lab and many files can be pulled off. A mechanical failure doesn't grind the platters into sand. As a network server it really should have a RAID too. So how exactly can "the server crash" so spectacularly that the RAID, backups, and widely available data recovery services all fail? Did the building blow up?

Re:How does this actually happen? (2, Informative)

BVis (267028) | more than 6 years ago | (#18955523)

The data "should" be recoverable, and the network server "should" have a RAID. I bet you anything that someone in IT asked for money for the RAID and it was denied, since lots of people with budget control think RAID is bug spray.

Backups and fault-tolerant hardware cost money. You can talk about potential losses and risks until you're blue in the face, until it *actually* costs the company money, nobody will listen. What's going to happen here more than likely is the person who asked for the RAID will get fired, as they're probably the same person in charge of the backups. This will also provide a scapegoat for that person's manager, since obviously if they got fired for it there need be no further repercussions or changes in behavior.

The only way they deserve to get fired is if they didn't advocate as hard as possible for enough backup hardware/software to allow for verification of backed up data and recovery in case of a mechanical hard drive failure. If they did, and were denied, then they did everything they could. (Which doesn't mean they won't get fired, it's just less deserved at that point. However, the thought there is that if they didn't want to get fired for incompetence, they should have tried to become a manager...)

Re:How does this actually happen? (3, Insightful)

Rob T Firefly (844560) | more than 6 years ago | (#18955715)

IANA publisher, but I would also imagine that in such a deadline-intensive business, data from a fried disk is about as good as lost. Sure, they can send their drives off to data recovery labs who could slowly recover an uncertain portion of the data for a pantload of money, but by the time that's done it'll be time for the next issue anyway. I'd guess it would be a lot quicker and cheaper to write off the disks and salvage what they can from everyone's local copies of the data.

Re:How does this actually happen? (1)

WillAdams (45638) | more than 6 years ago | (#18956163)

Yep.

When you're under a deadline gun, sometimes re-creation is faster than:

  - logging a job ticket w/ the help desk
  - waiting for it to work its way up the queue
  - waiting for IT to figure out which file to restore
  - waiting for the file to be restored

William
(who keeps Quark set to save 10 revisions of each file that he's working on to a backup partition (this can be a couple of GBs of data for some projects) and at a previous job where he was administering an NT Server had everyone's copy of Quark set to make these backup files in a folder on the server which was cleared out after the Friday night backup)

Re:How does this actually happen? (2, Insightful)

igb (28052) | more than 6 years ago | (#18955959)

`` I bet you anything that someone in IT asked for money for the RAID''

Perhaps, although my experience is that IT people are incredibly bad at framing business cases in terms more compelling than my daughter's request for a mobile phone for her birthday: a few vague reasons, followed by a sulk when asked for specifics.

I keep 20TB on RAID5, and replicate it daily to a RAID5 array that has no components or software above the spindle level in common (Solaris/EMC and Pillar Data). The data we really care about is on RAID 0+1, in some cases with three-way mirroring. We take it out to tape, in case the filesystem pukes over all the copies or the RAID controller decides to go bonkers. We're about to put ten miles between the two file servers. At no point have I had much pushback from management over the money, once the risks and rewards are explained. Too often, IT people convince themselves that some Dilbert-esque stereotype of a manager is going to say no, and therefore make their case in a passive-agressive style that will make anyone say no.

ian

RAID =! BACKUP (2, Informative)

RMH101 (636144) | more than 6 years ago | (#18956155)

If they deleted the files in error, then the RAID would faithfully mirror that deletion across all physical disks...

Re:How does this actually happen? (4, Informative)

Paulrothrock (685079) | more than 6 years ago | (#18955735)

A mechanical failure doesn't grind the platters into sand.

Doesn't it? [ufl.edu]

I think I know how! Or at least why ... (0, Flamebait)

Bearpaw (13080) | more than 6 years ago | (#18955833)

Maybe some White House officials had secondary email accounts on the system.

Re:How does this actually happen? (1)

SuseLover (996311) | more than 6 years ago | (#18955923)

The primary purpose of RAID is only protection against HARDWARE failure along and enhancing performance. Backups are the only way to protect data from; user error, program errors, O/S errors (file system corruption), and virus infections/hackers, etc.

I always get worried when a scheduled backup fails and I make sure it is corrected ASAP and do regular restores to verify the integrity of data.

Re:How does this actually happen? (2, Funny)

DebateG (1001165) | more than 6 years ago | (#18956555)

I once lived with a roommate who got home early one day:

Me: You're home early; not enough work to do?
Roommate: No, the server burned out
Me: Oh, that's no big deal; you just wait for them to get replacement parts and then you get back to it
Roommate: No, seriously, it's burned out. The air conditioning unit failed, the entire server room heated up to the point of spontaneous combustion and the entire server room caught fire

Lesson learned, keep your backups somewhere far, far away from the servers.

At this exact moment across the world (3, Funny)

Timberwolf0122 (872207) | more than 6 years ago | (#18955441)

Tens of IT managers are getting Hundreds of IT minions to check Thousands of backup tapes and befor a senior manager walks in.

Re:At this exact moment across the world (1)

doobie22 (970556) | more than 6 years ago | (#18955967)

This popped into my head too :)

Distributed backup (0)

Anonymous Coward | more than 6 years ago | (#18955457)

They should have just installed p2p software and had backup with multiple off-site copies. (Okay, maybe some copyright and security issues, but one fix at a time.)

Wrong problem (5, Insightful)

mseeger (40923) | more than 6 years ago | (#18955507)

Hi,

the problem was, as always, not the backup. I've rarely seen problems resulting from the backup process. The troublesome process is the restore. Or as a friend put it once:

Nobody wants backups, what everybody wants is a restore.

In my twenty years of IT i've seen several companies making backups like a well oiled machine. The backup process was well documented and everyone was trained to a degree, they could do it with their eyes closed. But everything fell apart in the critical moment, because all they had planned was making the backup. Nobody ever imagined or tried a restore on the grand scale. So they ended up with a big stack of tapes with unuseable data.

Backup is the mean, not the goal.

Regards, Martin

Re:Wrong problem (0)

Anonymous Coward | more than 6 years ago | (#18955657)

I see the backup/restore process as risky itself because small errors have major consequences. Secondly the time required for a fix may cost as much as the loss itself. Would mirrored drives be a more effective solution? I want my backup ready to run.

Re:Wrong problem (3, Interesting)

mseeger (40923) | more than 6 years ago | (#18955915)

> Would mirrored drives be a more effective solution?

Yes and No:

  • Mirrored drives are a good protection against drive failures and (usually) offer an easy restore process. If you mirror a drive and put the copy away (e.g. into a safe) this is a real and widely used backup method. As always you should at least try once to boot the system while removing the primary disk. Somtimes RAID controllers have some irks too.
  • This method usually depends on the availability of a certain hardware, if you cannot get a new mainboard or raid controller of the same type, the mirrored disk contains data you may have trouble getting at. You may ignore this issue, if you have the same hardware at a safe location again.
Regards, Martin

Re:Wrong problem (1)

Hoi Polloi (522990) | more than 6 years ago | (#18956625)

That is protection against hardware failure. It doesn't protect against user data loss or recovering old versions. Say someone wants to get an old directory back that was purposefully deleted a while back?

Re:Wrong problem (1)

swm (171547) | more than 6 years ago | (#18955713)

I've been in quite a few shops where the backups worked better than the restores.

Re:Wrong problem (5, Interesting)

RetroGeek (206522) | more than 6 years ago | (#18955725)

The troublesome process is the restore.

I heard a story about a LAN admin who was doing backups every night. The tapes would go into a safe, then would go offsite, then be used again.

Everything worked well(?) until they needed to do a restore. The tape in the safe was corrupt. The tape at the offsite storage was corrupt. No tape was good.

It seems that the LAN admin made tea every morning. The electric kettle sat on top of the steel safe.

So the backup tape was placed into the safe, then the kettle was started, magnetizing the safe, and erasing the tape.

Not ONCE did anyone try to do a test restore to prove the system.

Re:Wrong problem (1)

RetroGeek (206522) | more than 6 years ago | (#18955845)

I forgot to mention that the LAN adim tested the backup each morning with a sample file restore. Which is why they thought that the backups were valid.

Moral? Test the entire backup loop.

Re:Wrong problem (3, Informative)

ByteSlicer (735276) | more than 6 years ago | (#18956179)

I highly doubt that the kettle could demagnetize the tape in the safe, due to the Farraday shielding [wikipedia.org] . Even if the kettle was on top of the tape (outside the safe), the generated magnetic field would not be strong enough (although the heat would probably melt the tape).

Nice story, though. Reminds me of the sysadmin in my first company who automatically back-upped our server every day. Only problem was: the proces put a copy of the backup on a drive that was being back-upped. You can imagine what happened after a few weeks (it failed, disk full). He only noticed a few months later when we asked him to restore some files.

Re:Wrong problem (1)

lgbarker (698397) | more than 6 years ago | (#18956577)

A previous employer's ops center did this. Stacked old mainframe-style reel tapes under a phone with a solenoid-driven bell. Phone rang, erased the tape. Took a couple of cycles to figure that they needed better tape storage than the Ops manager's desk.

Re:Wrong problem (1)

ILikeRed (141848) | more than 6 years ago | (#18956431)

That is why backups must be created in an open format.

I don't care what the front end is so much as I care if I can restore onto any system with a minimum of fuss. And as ugly and whatever else you want to call tar, it's format is beautiful and simple, accessible with a wide variety of different front ends, and you have to look really hard to find a system that will not be able to read the format. Even WinZip can read and pull files out of the tar format. I prefer not to have a proprietary back up program - but I absolutely insist on the tar format for the actual archive.

RAID1? (1)

burnin1965 (535071) | more than 6 years ago | (#18955569)

And the server wasn't setup with a RAID1 at least for the partitions that hold critical data?

Backup alone is not enough, in some cases there needs to be multiple levels of protection.

Re:RAID1? (1)

Backup To The Web (1096353) | more than 6 years ago | (#18955701)

Agreed. A good backup and DR plan requires multiple bullets. RAID stops the front line of things, then local backups (tape, etc). Additionally, online backup is becoming a much more viable solution. Not only could they recover their data, their writers could have retention copies of multiple versions of their articles to rebuild the issue (or even just fix their own accidental deletion.)

Re:RAID1? (1)

DTemp (1086779) | more than 6 years ago | (#18956305)

There's a difference between FAULT PROTECTION and BACKUP.

If you don't know the difference, I suggest you learn.

Sitting on your bum with only a RAID-1 protecting your data is only smart if you hate the company you work for and want to stick it to 'em.

Caught out. (1)

auroran (10711) | more than 6 years ago | (#18955583)

The International Herald Tribune http://www.iht.com/articles/2007/05/01/news/magazi ne.php [iht.com] article about it makes a good comment. Business 2.0 has drawn attention for skewering firms, including, occasionally, other magazines, in its annual list of the "101 Dumbest Moments in Business." I wonder if will include itself in the upcoming one? They should also have remembered the old saying "There are two types of computer users, those who have lost data and those who are about to..."

Coming soon ... (2, Funny)

Stavr0 (35032) | more than 6 years ago | (#18955659)

The first issue of Business 3.0.

Got a (1)

kennylogins (1092227) | more than 6 years ago | (#18955755)

backup of that 2003 article?

The MPAA and RIAA... (0)

Anonymous Coward | more than 6 years ago | (#18955769)

...may some day come running to me thanking me profusely.

I'm just providing them with free off-site backup, that's all.

Practicing what they preach (2, Funny)

GuyfromTrinidad (1074909) | more than 6 years ago | (#18955875)

Since the parent company, Time named "you" the person of the year they were simply following "you" and not doing regular backups

Backups are usually no problems. (1)

Opportunist (166417) | more than 6 years ago | (#18956021)

The problem comes with the need of a restore.

There are invariably 2 possible situations that can happen when a restore is in order:

Either the backups simply don't work 'cause they've been using the same media for 5+ years and nobody ever bothered checking the error messages (or because there were no errors produced in the first place 'cause the backup was done without a verify).

Or the last person who knew how to do a restore with those tapes was fired a year ago and the others just followed the backup procedure which has been documented down to the most trivial steps, but there isn't a single piece of information to be found how a restore would work. The reason for this is simply that, since that techie has to get a vacation sooner or later, too, even the secretary has to be able to do a backup (thus the documentation), but the techie alone is good enough to know the restore procedure.

This again is due to:
a) Him being too lazy to write down the procedure (hell, without his boss requiring him to write down the backup specs he wouldn't have done that either) and
b) His boss having not the foggiest idea that restores could be a tad bit more complicated than "doing a backup in reverse".

Re:Backups are usually no problems. (1)

raind (174356) | more than 6 years ago | (#18956657)

Amen to that, being the lone admin can be painful.

Link to original article (2, Informative)

hargettp (74445) | more than 6 years ago | (#18956027)

Huh, I guess I wasn't paying attention to when Slashdot turned into Digg, even though I read both. Here's a link [iht.com] to the original article, rather than what might be a splog. Especially since the article text was copied verbatim.

always test you (1)

geekoid (135745) | more than 6 years ago | (#18956081)

recovery systems and methods.
Having a backup you can'r recover from is useless.

I wonder if they run DR on a regular basis. (2, Informative)

TomTraynor (82129) | more than 6 years ago | (#18956435)

I know that where I work we run a Disaster Recovery (DR) exercise every time we do either a hardware or software upgrade to prove that everything still works. If nothing has changed one is done every second year. We actually pull our off-site production tapes and restore to a new machine that is not in the same city where the current production machine resides. It may be over-kill for them, but, a test of that for them (or any other business) would be a fruitful exercise in that they prove that the backups are good and they can restore from a given point and carry on with a minimal loss of work.

For one of our server apps we actually have two laptops configured with all of the required software and we do restore production data from backups on a regular basis as we use that for our system testing on projects. This happens several times a year so we know that the backup and restore procedures truly work. It is also very cool walking in to the client site, plug in the laptop and show them that in an emergency they have a working machine very quickly. Not as fast as a server, but, it gets them a working machine until the replacement server arrives.

Check out their website: It's 'Dumbtastic!' (2, Funny)

rahimobius (1087399) | more than 6 years ago | (#18956487)

One of the 'Specials' on the Business 2.0 [cnn.com] website:
101 Dumbest Moments in Business [cnn.com]

See the video, test your Dumbest knowledge, and let us know what you think was the year's most boneheaded moves. (more)
Quiz: Test your Dumbest knowledge
videoBusiness: It's 'Dumbtastic!'
I think they might want to revise their list. I'm sure I would like to :)
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?
or Connect with...

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>