Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

So You've Lost a $38 Billion File

Zonk posted more than 7 years ago | from the makes-you-feel-better-about-yourself dept.

Data Storage 511

smooth wombat writes "Imagine you're reformatting a hard drive so you can do a clean install but then realize that you have also reformatted the back up hard drive. No problem. You reach for your back up tapes only to find out that the information on the tapes is unreadable. Now imagine the information that is lost was worth $38 billion. This scenario is apparently what happened in July to the Alaska Department of Revenue. From the article: 'Nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone: some 800,000 electronic images that had been painstakingly scanned into the system months earlier, the 2006 paper applications that people had either mailed in or filed over the counter, and supporting documentation such as birth certificates and proof of residence.' Using the 300 cardboard boxes containing all the information, staff worked overtime for several months to rescan everything at an additional cost of $200,000."

cancel ×

511 comments

Sorry! There are no comments related to the filter you selected.

Time for... (5, Funny)

Short Circuit (52384) | more than 7 years ago | (#18414365)

Seppuku [wikipedia.org] ?

Re:Time for... (1)

voice_of_all_reason (926702) | more than 7 years ago | (#18414497)

He surely wouldn't want for a Second.

Re:Time for... (2)

Philip K Dickhead (906971) | more than 7 years ago | (#18414781)

Data recovery.

Geez guys! They can find the Pr0n you "deleted". I guess there needs to be more significant motivation than $38 Bil USD.

Re:Time for... (1)

thc69 (98798) | more than 7 years ago | (#18414565)

No, time for Google [wikipedia.org] Seppuku [livejournal.com] .

Re:Time for... (0)

Anonymous Coward | more than 7 years ago | (#18414573)

Uhmm ... government agency, so very little accountability. In fact, worker probably got a performance award for bringing the issue of poor backups to light.

Re:Time for... (4, Funny)

Rakshasa Taisab (244699) | more than 7 years ago | (#18414993)

Thanks for alleviating my ignorance [wikipedia.org] .

Tapes? (1)

DogDude (805747) | more than 7 years ago | (#18414399)

Just curious... do most people still use *tape* for backup? Personally, I use multiple hard drives and a DVD burner on a daily basis.

Re:Tapes? (4, Informative)

MindStalker (22827) | more than 7 years ago | (#18414453)

Yea, tape is pretty common. DVD burners simply aren't rated for backups as some burned DVDs don't have a very long shelf life. Now sounds like some screwed up in purchasing cheap tapes as well. Oh no.

BTW article is silly, the file isn't worth $38 billion $200K at best because thats the cost of rescanning everything. Would be interesting to see an accounting record of how much recreating all the documents would cost had they not had a hard copy.

Re:Tapes? (1)

Short Circuit (52384) | more than 7 years ago | (#18414695)

BTW article is silly, the file isn't worth $38 billion $200K at best because thats the cost of rescanning everything. Would be interesting to see an accounting record of how much recreating all the documents would cost had they not had a hard copy.
You're assuming the physical documents still exist. The purpose in scanning the documents may have been to cut down on physical storage requirements.

Re:Tapes? (4, Insightful)

Red Flayer (890720) | more than 7 years ago | (#18414713)

Well, the summary states that the files were rescanned at a cost of $200,000 -- so it sure sounds like the hard copies were preserved.

Re:Tapes? (1)

Short Circuit (52384) | more than 7 years ago | (#18414835)

Ah. I missed the "re" part of "rescanned". I thought that was the cost of the initial digitization.

Re:Tapes? (-1, Redundant)

dextromulous (627459) | more than 7 years ago | (#18414459)

Don't underestimate the bandwidth of an Alaska full of tapes going down the interstate.

Re:Tapes? (1)

Mipoti Gusundar (1028156) | more than 7 years ago | (#18414791)

Best bandwidth per buck is crate of papers going to India!

Re:Tapes? (1)

stoolpigeon (454276) | more than 7 years ago | (#18414467)

yeah they do - when it is more than a couple boxes in ma's basement.

Re:Tapes? (3, Insightful)

teflaime (738532) | more than 7 years ago | (#18414493)

Most ENTERPRISES still have tape at some level as part of a comprehensive disaster recovery plan. Tape is easy to offsite, fairly reliable overall and still have comprehensive support available in all platforms. Most INDIVIDUALS don't do backups at all.

Re:Tapes? (4, Insightful)

Itninja (937614) | more than 7 years ago | (#18414721)

Easy to offsite???

I work for an IT organization and we pay a company called Iron Mountain $100's monthly to schlep our boxes and boxes of backup tapes to their offsite storage facility.

And remember there is a difference between making 'backups' (store my important files somewhere else so I can get them in case of a system failure) and preparing for 'disaster recovery' (store everyones files somewhere else so we can rebuild the entire infrastructure in case the building burns to the ground).

Re:Tapes? (0)

Anonymous Coward | more than 7 years ago | (#18414505)

in corporate environments where an off-site strategy is needed, tape backups are pretty much still in use..unfortunately since they are a pain to deal with.

although lately i have seen more disk based backups with multiple rotating external disks.

Re:Tapes? (1)

Itninja (937614) | more than 7 years ago | (#18414513)

Indeed. For super-critical, but not super-massive, data like this especially. I use a couple of external 250GB hard drives to back up my data fast. Then I take one of the HD's home with me for off-site storage. The best part is, there's never a 'restore' period! Just plug the external into the PC and a 'removable device' appears in 'my computer'.

Of course, if you need to store terabytes upon terabytes of data, tape may still by the only option....

Re:Tapes? (0)

Anonymous Coward | more than 7 years ago | (#18414515)

so how long does it take to burn a terabyte or 2 to dvds?
most 'people' never used tape anyway. corporation and government systems do.

Re:Tapes? (5, Informative)

greginnj (891863) | more than 7 years ago | (#18414549)

Really? For what volume of data? For people with 100s of GB of transactional data, tape robots are pretty much the only option, or you'll be spending your whole day swapping DVDs. OTOH, it sounds like this was relatively static data (since it could be re-entered from paper), so maybe a DVD version would have been an appropriate measure as well. There's also a lesson here that you should frequently do test restores from backup tapes.

Data Volume (2, Informative)

tempest69 (572798) | more than 7 years ago | (#18415009)

800,000 images * 300 kb /image (decent scan after compression) = 800k * 300kb = 800 MB * 300 = 2.4GB * 100= 240 Gigs..

Id skip on the DVD backup, sounds like a mistake waiting to happen. Backing this up to a network drive over Gig-E is still going to be a mess, but it should be a few hours of slacktime.. (yes in theory you could manage 240 gigs in roughly 35 minutes over gig-E, but you couldnt pull off enough seeks in that time via the hard drive (800k seeks * 8 ms/seek= 6400s ~= 106 minutes).

Storm

Re:Tapes? (1, Funny)

Anonymous Coward | more than 7 years ago | (#18414555)

Hmm...this should translate to:

Just curious...do most people still think differently then I do? Because I do something one way, so should everyone else. Did I mention I am also unable to empathize?

--AC

Re:Tapes? (1)

geekoid (135745) | more than 7 years ago | (#18414613)

Tape works just fine, espcially for large amounts of data.
They maynot of been maintaining their tapes, or they were using low quality equipment, or it was just one of those things.

For the illogical 'anti-tape' crowd, I will point uot that they did have a problem with the hard drive as well.

Re:Tapes? (0, Offtopic)

Anonymous Coward | more than 7 years ago | (#18415003)

May not "of" been? What the fuck?

May not HAVE been, you idiot.

Re:Tapes? (2, Insightful)

Detritus (11846) | more than 7 years ago | (#18414623)

Tape works great if you are willing to spend big bucks for top-class hardware. Unfortunately, most people try to get by with the cheap stuff, which is very unreliable. Try to explain to a manager why you need a $50K tape system to backup a $10K server. Computers have gotten very cheap, high-quality tape transports haven't.

Re:Tapes? (1)

varmittang (849469) | more than 7 years ago | (#18414673)

Yeah, when you have a good 300+ gigs of data that needs to be backed up daily, me sitting there burning DVDs are not the way its going to get done.

Re:Tapes? (4, Insightful)

daeg (828071) | more than 7 years ago | (#18414675)

Hm. Tapes with a proven shelf life of many, many years, or DVDs where a single scratch can render 4GB of data worthless. I wonder which enterprises (or governments) should chose?

Re:Tapes? (1)

An dochasac (591582) | more than 7 years ago | (#18414767)

Tapes are still widely used in the industry. It scares me how many companies may be choosing the convenience of DVD over the stability of tape. I've seen digital 8mm digital (exabyte) tapes survive unharmed 10 years in an attic where temperatures ranged from 15F-150F and yet I've seen CDs and DVDs without a scratch on them mysteriously stop working after less than 2 years. As for hard drives, they can be made part of a viable storage technology, but the drives themselves are notoriously unreliable. I'd use RAID5/RAID6/Z with some sort of parity checking filesystem (e.g. ZFS on BSD or OpenSolaris).

Re:Tapes? (0)

Anonymous Coward | more than 7 years ago | (#18414829)

The operative word here is "personally"

When you can manage to fit over 2-3 terabytes of daily backups onto some sort of DVD cycle, let me know.

DVDs, or even blue-monkey-super-HD++-extra-super-sized-VDs or whatever, simply don't cut it.

Most places now do some sort of hierarchical backup, with spool drives and phased migration to tape, but tape is still king for streaming speed, retention, data integrity, and most importantly capacity, for removable backup media.

Now please excuse me while I try to figure out what to do when our LTO III robots won't cut it anymore either.

Re:Tapes? (1)

Ryan Amos (16972) | more than 7 years ago | (#18414847)

Yep. Hard drives and DVDs are backup media, tape is disaster recovery media. DVDs are not durable enough nor do they really hold enough (tape has much better density; 400GB per LTO3 tape compared to 9GB per DL-DVD.) Tape is still the undisputed king for long-term, high capacity backups.

For home use, online backups are fine, but for even small businesses, volumes of data approaching 1TB are not uncommon.

re: tape backup (1)

King_TJ (85913) | more than 7 years ago | (#18414927)

Where I work, we finally phased out our tape backup drives last year in our server room. A product called the "CRU Dataport" is a very nice removable, hot-swap hard drive frame and carrier assembly. You just install their frame in an external 5.25" drive bay, and buy a set of carriers for it. Install suitable SATA drive drives in each carrier, and switch them out nightly just like backup tapes. Backup software like "Backup Exec" can still be used, but it will treat each cartridge just like it did the backup tapes.

Tape still has a few advantages though. For one thing, they're less fragile than a hard drive. They're also less valuable to the average employee than a SATA hard drive, so there's less worry of one disappearing if you give it to someone to take off-site regularly.

But all in all, we like the switch. Backups complete in less time, and it's faster restoring selected files. (No rewinding or tape re tensioning needed, etc.)

Re:Tapes? (1)

91degrees (207121) | more than 7 years ago | (#18414975)

Yup. You can get 160GB onto a tape, each tape costs a lot less than a disk drive and is more reliable. But they are becoming less common. The price difference is falling and the drives are expensive.

Sounds more like a $200,000 file (1, Insightful)

Anonymous Coward | more than 7 years ago | (#18414415)

But maybe that's just me, someone who opted not to work in government after studying political science.

Redo the work? (5, Insightful)

Anonymous Coward | more than 7 years ago | (#18414433)

For that kind of money, I'd probably just send the HD to data recovery specialists.

Re:Redo the work? (4, Insightful)

omeomi (675045) | more than 7 years ago | (#18414587)

For that kind of money, I'd probably just send the HD to data recovery specialists.

Well, this is the government. They probably didn't have a budget for data recovery, but they did have a budget for scanning documents...the actual dollar amounts of each probably matter very little ;-)

Re:Redo the work? (2, Interesting)

bilbobob (1036984) | more than 7 years ago | (#18414891)

HD Recovery specialists? They could have could have bought something like GetBackData for NTFS and saved themselves $199921 in recovery costs. As far as I'm aware, reformatting your HD is one of the least successful methods of permanently destroying your data (even if you mean too).

Re:Redo the work? (1)

sumdumass (711423) | more than 7 years ago | (#18415013)

Do you really think that is appropriate? I would have brought the recovery team here instead of sending it away. You know how important packages get lost in the mail. It would suck to lose this and have it be your last change at getting something out of it.

I sent a drive away in the past to find 3 pictures and it cost somewhere around $2500 for then entire drive (they wouldn't do just the three pictures). The pictures were worth 40% of 2 mill to my customer. Quite a bit less then 38 billion but I still had a currier deliver it to the front door and hand it to a live person.

It was worth every penny. So unless they are using some odd format that the drive specialist cannot work with, It should should have been there before this article was thought of.

The Senator (3, Funny)

jeevesbond (1066726) | more than 7 years ago | (#18414437)

Senator Ted Stevens remarked that they should have sent it in an Internet, apparently tubes are much more reliable than tape.

Re:The Senator (2, Funny)

Donniedarkness (895066) | more than 7 years ago | (#18414703)

They DID send it in an internet, but we were clogging up the tubes with "youtubes" and "myspaces" that it didn't reach them until AFTER they finished copying it!

And this is why... (3, Interesting)

bobetov (448774) | more than 7 years ago | (#18414439)

...print will never be dead.

Re:And this is why... (3, Funny)

imboboage0 (876812) | more than 7 years ago | (#18414559)

Re:And this is why... (0)

Anonymous Coward | more than 7 years ago | (#18414745)

A Hard Drive may burn just as much as paper does. :P

$38 billion? (4, Insightful)

pipatron (966506) | more than 7 years ago | (#18414441)

How did they figure these files were worth $38 billion when it only cost $200000 to create them from scratch?

Re:$38 billion? (3, Informative)

stoolpigeon (454276) | more than 7 years ago | (#18414511)

right - the account is worth 38 billion - the file was apparently worth about 200 grand in labor. of course it didn't cost that much to make the first time, as it was done over a longer period without all the o.t.

Re:$38 billion? (3, Funny)

192939495969798999 (58312) | more than 7 years ago | (#18414547)

The method for arriving at that figure was also tragically lost. A team of monkeys recreated the figure in 3 minutes with a number pad at a cost of $45.

Re:$38 billion? (4, Insightful)

SKiRgE (411560) | more than 7 years ago | (#18414591)

The value is based on what the data represents, not the material labor in re-scanning them. loose example, I could spen $30 on painting supplies and create a $100,000 masterpiece (well not by me, but, anyway...)

Re:$38 billion? (1)

mwvdlee (775178) | more than 7 years ago | (#18414711)

Presumably you could not recreate that same identical masterpiece if the original were destroyed. In this case they could.

Easy... (0, Redundant)

Registered Coward v2 (447531) | more than 7 years ago | (#18414607)


How did they figure these files were worth $38 billion when it only cost $200000 to create them from scratch?


RIAA / MPAA mathematicians

Re:Easy... LOL (1)

Maxo-Texas (864189) | more than 7 years ago | (#18414667)

And, I have ot say those numbers were smelly. Not sure where they pulled them from.. but...

The whole thing is a joke... (1, Interesting)

Rocketship Underpant (804162) | more than 7 years ago | (#18414657)

No matter what they might have lost, it wasn't worth $38 billion. The state gross product of Alaska is only $33 billion, and their tax revenue will only be a fraction of that. And it's not like they lost any of that either, just some files.

What, another hyperbole-filled, wildly inaccurate Slashdot post? Inconceivable.

Re:The whole thing is a joke... (1)

Red Flayer (890720) | more than 7 years ago | (#18414903)

Hmm. Did you RTFA?

The fund is worth $38 billion. Dividend payouts from the fund last year were over $650 million, about $1,107 per payout, roughly 600,000 payouts.

This fund has nothing to do with tax revenue or annual gross product, it comes from oil revenues over a period of decades. The point of the fund is so the gov't doesn't have to pay out of a current account, and so doesn't NEED to depend on annual tax revenues.

Re:The whole thing is a joke... (2, Informative)

Xzzy (111297) | more than 7 years ago | (#18414967)

The Alaska Permanent Fund is not tax funded at all. Technically, it's not even part of the state government.

At the simplest level, it's saved up money from the oil boom the state had in the 70's that the permanent fund corporation invests, saves, and takes care to insure it's always going to be there. Once a year it calculates earnings, subtracts operating and inflations costs, and hands out the remainder to qualifying Alaska residents. Usually it's in the area of $1000, but can fluctuate quite a bit.

They passed $30 billion last year, the news story would indicate it's gone up a bit since then. ;)

Re:$38 billion? (1)

mikael (484) | more than 7 years ago | (#18414693)

Because that is the sum of all the financial transactions that were legally documented. Suppose every pae of those 800,000 pages is a single tax return worth on average $5000 to the state. Then you get a rather large number: 800,000 pages * $5000 = $40,000,000,000

Re:$38 billion? (1)

illegalcortex (1007791) | more than 7 years ago | (#18414909)

Yes, but as has been pointed out numerous times, the file itself is not worth $38 billion as was stated. If I write a check for $100 and it gets destroyed, I can just write you another check. That means the check itself isn't worth $100. It's only worth how much it costs to get that check printed and reissued.

Re:$38 billion? (0)

Anonymous Coward | more than 7 years ago | (#18414717)

Do you think the headline, "State of Alaska Makes $200,000 Mistake" is going to draw a lot of readers?

Re:$38 billion? (1)

mwvdlee (775178) | more than 7 years ago | (#18414783)

Actually, the value of those files is what somebody else would pay for it.

I can spend $200,000 on labor producing a large pile of shit (for that kind of money, it'd be very large), but it wouldn't be worth shit (well, actually, that'd be exactly what it's worth, but you get the point).

Re:$38 billion? (1)

evilviper (135110) | more than 7 years ago | (#18414827)

How did they figure these files were worth $38 billion when it only cost $200000 to create them from scratch?

Yeah, it's far too hard to read the HEADLINE and/or the SECOND SENTENCE of the article, which both explain EXACTLY where the figure comes from...

I'm sure it was much faster to post a comment to /. and check every few hours until someone to posts an answer.

Investment VS Opportune Cost (1)

RingDev (879105) | more than 7 years ago | (#18414849)

Lets say you have a small software development company. You have 6 employees that cost you on average of $90k per year each (including taxes, 401k match, health care, salary, office space, etc...) Now lets say that it takes you 2 years to build your application. Over those two years your team spent 20% of the time in meetings, 15% of the time goofing off, 10% of the time debugging, 20% of the time designing/planning, 5% of the time training, and the rest (25%) actually making progress on code.

Your investment for that final product is roughly $1.08 million dollars.

Now, imagine the day before you are sending the code off to the press to go gold, you lose everything. Luckily, you retain all of your staff, and they are all very familiar with the project. At this point, they can start working on recoding the entire project. No more meetings, no more design decisions, no more planning, no more training, and less goofing off. You can now re-create that final product in about 35% of the total time.

Your opportune cost to replace the final product is roughly $378,000.

So yes, the file could have absolutely been worth $38 Billion, yet only cost $200,000 to recreate.

-Rick

Re:$38 billion? (1)

Mateo_LeFou (859634) | more than 7 years ago | (#18414929)

File it right next to the $18 billion (or whatever this week'd ass-originated number is) in "losses" that the RIAA/MPAA attribute to piracy.

Oh, and the $60 billion lost to "web video piracy" as per
http://www.havocscope.com/Counterfeit/webvideos.ht m [havocscope.com]

Yea, I wonder... (1)

mr_3ntropy (969223) | more than 7 years ago | (#18414941)

..Alaska.

A thought that crosses my mind... (2, Insightful)

physicsboy500 (645835) | more than 7 years ago | (#18414499)

If you're going to back up a file (actually a set of files) that is worth that much, wouldn't it be smart to go a bit further than keeping a backup copy on magnetic media?! Maybe in more than one place too?!

Re:A thought that crosses my mind... (1)

Volante3192 (953645) | more than 7 years ago | (#18414683)

Hard drives and backup tapes are both magnetic, and they technically did keep a backup on a format that wasn't magnetic: the original hard copies.

Note from the article (and the summary) the IT guy formatted both the original and the backup hard drives. Then there was the bad tape. So that's 2 sources. What surprises me is it wasn't on another tape somewhere.

I'll do it! (5, Funny)

mwvdlee (775178) | more than 7 years ago | (#18414509)

So the information is still available in 300 boxes and it would cost about $200,000 to scan and recreate the $38 billion file again?
I'll do it for $1 billion.

Re:I'll do it! (1)

Misch (158807) | more than 7 years ago | (#18414661)

Halliburton? Didn't we tell you to get off slashdot?

Haha (-1, Troll)

Anonymous Coward | more than 7 years ago | (#18414521)

LOL OWNED!

maxtor? (4, Funny)

ElephanTS (624421) | more than 7 years ago | (#18414525)

As their IT consultant I stand by my use of Maxtor drives.

Oblig. (-1, Troll)

Anonymous Coward | more than 7 years ago | (#18414527)

HA HA!

Actual Cost?? (0, Redundant)

brunes69 (86786) | more than 7 years ago | (#18414541)

Using the 300 cardboard boxes containing all the information, staff worked overtime for several months to rescan everything at an additional cost of $200,000."

Err.... if it only cost $200,000 to replace the data, where the hell does the $38 Billion figure come from?

Re:Actual Cost?? (0)

phasm42 (588479) | more than 7 years ago | (#18414729)

Dumbfuck article summary (did anyone bother to proof it even once??)

The fund represented in the lost files was worth $38 billion. It'd be like if I had a QuickBooks file on my $38 billion account, and I erase the file.

Re:Actual Cost?? (1)

lymond01 (314120) | more than 7 years ago | (#18414749)

Well, let's say it's all of Alaska's tax income data for the 2006 year. Of course it's not impossible to recreate the data, but since it's basically electronic money, you could consider it "lost" if only temporarily.

Where did the numbers come from? (0)

Anonymous Coward | more than 7 years ago | (#18414561)

The information was worth $38 billion, but they rescanned it at a cost of only $200K?

Sounds like RIAA accounting to me.

Data recovery? (1)

Upphew (676261) | more than 7 years ago | (#18414577)

Wtf?!? They call to ms and dell, but not to people who actually recover data? My first call would been to http://www.ibas.com/ [ibas.com]

ps. I don't work at ibas

I don't get it. (1)

byteherder (722785) | more than 7 years ago | (#18414605)

"Imagine you're reformatting a hard drive so you can do a clean install but then realize that you have also reformatted the back up hard drive. No problem. You reach for your back up tapes only to find out that the information on the tapes is unreadable..."

Ok, so you go to the primary hard drive and make another backup.

Another question, doesn't anyone test their backup systems?

Re:I don't get it. (1)

geekoid (135745) | more than 7 years ago | (#18414659)

"...e that you have also reformatted the back up hard drive."

That implies that the original and the back up had been reformatted.

Re:I don't get it. (1)

Detritus (11846) | more than 7 years ago | (#18414813)

Another question, doesn't anyone test their backup systems?

I've often been in the situation where I couldn't properly test the backup system because management decided to save money by not buying any spare hardware.

"Hey, we bought you a tape drive and a box of tapes, quit complaining!"

Backups are the devil (5, Insightful)

Maxo-Texas (864189) | more than 7 years ago | (#18414621)

Because no one ever restores them regularly to test them.

I was at a company years ago and argued for both a ton more backups than they were making and for a test restore. They were not in the mood to do either. After about nine months, for some unknown reason they had to restore a file.

And the backup tape was unreadable. The next good backup was 17 days older.

After that we got $30 bucks of backup tapes every week and we had a 7 day rotation with the 7th day going in the vault. And we did regular test restores once a quarter.

You should REGULARLY test your backups.
You should have LOTS of backups.

Re:Backups are the devil (1)

mwvdlee (775178) | more than 7 years ago | (#18414953)

Isn't it possible to have tapes automatically verify data? Sure, it may take a while longer for the process to finish, but atleast you'll know the process was actually worth the wait.

Re:Backups are the devil (1)

Hoi Polloi (522990) | more than 7 years ago | (#18414961)

Like an old boss of mine once said, "You don't have a backup if you've never done a restore."

Re:Backups are the devil (1)

Cedric Tsui (890887) | more than 7 years ago | (#18414973)

Hmmm... And the thing is, $30 bucks a week is a negligible cost. A mostly automated test once every 3 months is a negligible cost. An extra vault is a negligible cost.

Where to start... (1)

dtjohnson (102237) | more than 7 years ago | (#18414625)

1) No surprise reading this:

Over the next few days, as the department, the division and consultants from Microsoft Corp. and Dell Inc. labored to retrieve the data, it became obvious the worst-case scenario was at hand.

2) Always make sure the backup really works.

3) Better procedures are needed if a single tech can reformat both hard drives in the same session

4) Much better hardware and software are needed for data worth $38 billion.

5) Paper backups are a good last resort and as a check on data integrity

Worth how much? (1, Informative)

Anonymous Coward | more than 7 years ago | (#18414629)

Sounds like the file was worth $200,000. The account was worth $38B.

Ask your CEO (0)

Anonymous Coward | more than 7 years ago | (#18414647)

I'd be fired and leave with $30 Million. Then I'd move to the Cayman Island, having a sunbath, and asking myself between two cocktails how hard my former employees and my former state have been working to clean my mess. Well maybe I won't even ask cause I don't give a shit about it.

I'd hate to be the tech support guy... (4, Funny)

condour75 (452029) | more than 7 years ago | (#18414669)

And it's not in the recycle bin? Ok, let's not panic. Click start, go to find, choose files and folders...

Damn! (4, Insightful)

Penguinisto (415985) | more than 7 years ago | (#18414681)

That has got to rank up there w/ the all-time worst 'oh-shit' moments in that poor bastard's career (if it still exists). I wonder what the sysadmin was thinking, storing data on the same partition as the OS. No sane production environment rig that I know of would (or at least should) have that. It may be a Windows thing, but on most servers I've dinked with, the OS sat on a pair of RAID disks by itself, and all the data sat on the monster pile of disks on their own logical RAID drive (at least RAID 5... 5+0 w/ a hot spare preferred).

That, or you'd think they'd at least have that kind of stuff stored on more than one server if it were that valuable?

/P

This should be... (1, Insightful)

Anonymous Coward | more than 7 years ago | (#18414707)

This should actually be a best-practice kind of situation really. Accidents happen. Both hard disks got formatted. The tapes were shit. BUT, we had a paper trail. It was painstaking, but it was recoverable. EXCELLENT! This is exactly why banks have paper records of everything, and why they pay a LOT of money to have them properly stored.

That said... the excerpt is a bit misleading. The data was worth $38 billion. They didn't lose $38 billion. They managed to get it back in shape for $200,000, which is not pennies, but probably well worth the effort.

Alaskan Pipeline (2, Insightful)

Doc Ruby (173196) | more than 7 years ago | (#18414723)

I bet it cost a lot less than $200K to bribe the government officials (probably with a few bottles of wine) not to check whether they were protecting their $38B investment with more than $45K worth of IT staff.

Value = cost of loss, != $38B (1)

davidwr (791652) | more than 7 years ago | (#18414727)

The cost of the loss was the $200,000 to rescan the documents plus the cost of not having those documents online for part of the past 9 months, plus any related costs.

Think of it this way:
If you lose a $38 billion dollar check in a fire, but it only costs you $100 to get the bank to re-issue the check, your loss is $100 plus a few days' interest. A few million dollars in interest is a far cry from $38 billion.

If they'd shredded the boxes they'd be in real trouble.

Re:Value = cost of loss, != $38B (1)

rubycodez (864176) | more than 7 years ago | (#18414857)

that would have been a better ending to the story, "they then went back to the original documents in cardboard boxes, but alas their auditing firm used ex-Arthur Andersen employees and they had all been shredded."

Data recovery? (4, Interesting)

evilviper (135110) | more than 7 years ago | (#18414777)

With hard drives, data doesn't just go away. Sure, it may not be recoverable with simple "undelete" software, but data recovery experts will charge far less than $200,000 to pull important files off of a wiped hard drive.

The same goes for tapes. There is no mention in the article of why they were "unreadable" what level of damage there was to the data, etc.

We all make mistakes, but 3 layers of backup data storage all failing suggests a horrifically poor system in-place. Not JUST "very bad," that's hard to believe, without some massive natural disaster causing it.

I could be a douche and say it has never happened (1)

PrimeWaveZ (513534) | more than 7 years ago | (#18414789)

But I'll admit that about 6 years ago, when moving from one old laptop to another, I accidentally erased all of the data from my main and my backup Jaz disks before ensuring that every piece of data from 6th grade through my first year of college was backed up onto new media. It wasn't that I didn't have a good strategy for backup: it was adequate for my needs at the time. It's not that I didn't know I shouldn't format disks with important stuff on them: I figured I had transferred it already. At age 18, I didn't have the money for recovery services, nor did I have much of a printed record of anything.

What happens when you fuck up that big? Take it like a man, and live with the shitty consequences. Know that there's nobody to blame but yourself, but learn from that catastrophe so it never happens again.

Re:I could be a douche and say it has never happen (4, Insightful)

Overzeetop (214511) | more than 7 years ago | (#18415021)

What happens when you fuck up that big?
If you the CEO of a major corporation, you get a severance check that looks like a phone number. If you're the IT plebe, you get to look for a new job.

Not to bad (3, Informative)

geekoid (135745) | more than 7 years ago | (#18414797)

That they cuold get recovery for only 200K.

I know that many companies would not be able to recover information lost in that manner.

I worked for a company that had not had a back up, at ALL for 4 years. All there business was lectronic. If the system had crashed there company would die. I spent 6 mopnths trying to them to pay for a back up system. FInally the provided a tape drive thawas 5 years old and completly inadequate... I decided to go elsewhere.

Something else worth mentioning.. (0, Troll)

ka24 (80379) | more than 7 years ago | (#18414811)

Why were the drives being formatted anyway? You don't need to do such things with some OSes.

Re:Something else worth mentioning.. (2, Funny)

Detritus (11846) | more than 7 years ago | (#18414997)

Please report to your nearest Microsoft customer reeducation camp.

Misinformed (0)

Anonymous Coward | more than 7 years ago | (#18414831)

It reads: Now imagine wiping out a disk drive containing information for an account worth $38 billion (29 billion).

Thats "For an account worth"....not...."files worth"

$200k for recovering $38BN - very cheap (0)

Anonymous Coward | more than 7 years ago | (#18414841)

What is the big deal, $200K is nothing compared to $38BN. Besides, the accident created (temporary) jobs.

To err is human... (1)

Slipgrid (938571) | more than 7 years ago | (#18414883)

To err is human, to really f-up requires root.

From , "nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone." [msn.com]

Really? Why is the that the oil money payouts or the military contract accounts are the only ones that ever get deleted? The IRS is using the same database that they've been using for the past fifty plus years, but they never seem to have that problem.

Remember kids... (1)

foo fighter (151863) | more than 7 years ago | (#18414919)

It's not about the backup, it's about the restore.

If you aren't regulary testing your recovery capabilities, your nightly backups are masturbation. It may make you feel good for a bit, but it's not satisfying.

I lost 75 trillion dollars! (4, Funny)

140Mandak262Jamuna (970587) | more than 7 years ago | (#18414957)

I launched notepad and wrote 75 trillion dollars and saved the file. And, because I was feeling very extravagent, I deleted the file. I am rich. I can afford to lose 75 trillion dollars without batting and eyelid and am man enough to brag the info to the whole world.

Come on guys, it took only 200,000$ to create the data. It probably had records of payments totalling 38 billion dollars. But what they lost was 200,000$ not 38 billion dollars.

Pshh (1)

RMelon (1068520) | more than 7 years ago | (#18414981)

Alaskans....
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>