Beta

Slashdot: News for Nerds

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Ask Slashdot: Best Offline Storage Method For Large Archives?

Soulskill posted about 3 years ago | from the stone-tablets-and-a-sharp-chisel dept.

Data Storage 397

An anonymous reader writes "I have a collection of large projects (Indesign files with associated images), which are typically 40GB to 60GB each. In this current climate, what is the 'best' method of archiving these? Spinny magnets? Solid state drives? USB? Tape? Blu-ray? All have pros and cons and price considerations. If I remove the price issue (my data is important to me), does this change the choice?"

cancel ×

397 comments

Go with tried and true (0)

Pikoro (844299) | about 3 years ago | (#36819388)

Put it all on 5 1/4" floppies :)

Re:Go with tried and true (0)

kimvette (919543) | about 3 years ago | (#36819486)

Single sided, single density, right?

Re:Go with tried and true (1)

ls671 (1122017) | about 3 years ago | (#36819638)

Right, 1.2 MB used to be less reliable for me and the guy said that his data was important to him.

Re:Go with tried and true (1)

bennomatic (691188) | about 3 years ago | (#36819698)

What is this "MB" you speak of? My 1541 drive stores 170KB per floppy. That should surely be enough for anyone.

Re:Go with tried and true (0)

Frosty Piss (770223) | about 3 years ago | (#36819512)

Put it all on 5 1/4" floppies :)

I don't get it. Sure, I realize that it would take a HUGE number of floppies to accomplish this, but I'm a bit lost as to why that's "funny" as in the "smiley face" emoticon. Can you explain?

Re:Go with tried and true (1)

Pikoro (844299) | about 3 years ago | (#36819642)

Can you imagine how long it would take to toss even a single gig of data onto 360k floppy disk? The funny part would be how much money you would waste by sitting in front of a computer switching disks every 60 seconds or so and then writing labels for them all, and sticking them on (straight of course or you have to carefully peel it off and put on a new one).

Lesse.. wolfram alpha says just over 4 weeks (assuming an 8 hour work day) http://www.wolframalpha.com/input/?i=60gib+%2F+360kib+*+60+seconds [wolframalpha.com]

For a total of 174,763 floppies. That would be a stack of floppies (http://www.wolframalpha.com/input/?i=174763+*+2mm) ~350 meters tall, or just taller (1.2x) than the Eiffel Tower.

Might be easier to just buy a bunch of TB hard disks and put them in a Raid 5 configuration (with hot spares of course) and be done with it.

Re:G.one (1)

xiayou (2316372) | about 3 years ago | (#36819900)

Re:G.one (1)

kakarote (2294232) | about 3 years ago | (#36819924)

ahm.!! ahm.!! somebody already said that You can always rebuild it if you have to from proper printouts. Get them professionally published to archive quality. Then to save your actual digital stuff, rotational disk archiving. No digital storage medium will be 100% effective so use a Raid array and plan to upgrade it every 3-4 years to a new array of better/bigger drives. Even then though, your absolute best best is that first part, then put them into archival storage in multiple locations.

Re:G.one (1)

xeon13 (2268514) | about 3 years ago | (#36819936)

well i think This kind of storage as I recall can do up to several terabytes (maybe up to 50TB?).

Re:Go with tried and true (1)

MichaelKristopeit423 (2018892) | about 3 years ago | (#36819834)

Put it all on 5 1/4" floppies :)

I don't get it. Sure, I realize that it would take a HUGE number of floppies to accomplish this, but I'm a bit lost as to why that's "funny" as in the "smiley face" emoticon. Can you explain?

you're "quoting" your "subconscious"??? who said it was "funny"?

you're an idiot.

could it not make someone generally happy to reminisce that we've come a long way, baby :)

cower in my shadow some more behind your chosen urine based pseudonym, feeb.

you're completely pathetic.

:)

Re:Go with tried and true (1)

Pikoro (844299) | about 3 years ago | (#36819946)

Uh oh. MichaelKristopeit is on my side? I don't know how I should feel about that.

Re:Go with tried and true (1)

davester666 (731373) | about 3 years ago | (#36819518)

Papertape is the way to go. Not susceptible to magnetism at all. And it'll be easy to reinvent a way of reading it when the 4th Reich comes to power, as there will always be some way to tell a computer "yes, I can see a light, or no, I cannot see a light".

Re:Go with tried and true (0)

cjcela (1539859) | about 3 years ago | (#36819546)

Nah. You should actually print it, at least with a font size of 12 points. And then, laminate each page, for extra protection!

Re:Go with tried and true (0)

Anonymous Coward | about 3 years ago | (#36819572)

Put it all on 5 1/4" floppies :)

What compression program do you use that can make 40 GB fit on 5 floppies?

Personally, I'd store it on /dev/null. There's plenty of space there, it's way faster, and you can easily backup it to a usb stick and carry it with you.

Re:Go with tried and true (1)

cheeks5965 (1682996) | about 3 years ago | (#36819746)

wow, those would be five really tiny floppies.

Re:Go with tried and true (1)

cvtan (752695) | about 3 years ago | (#36819588)

You are kidding right? Use 8-in floppies that can store 180k per disk.

Re:Go with tried and true (1)

UNIX_Meister (461634) | about 3 years ago | (#36819790)

Don't forget to punch the floppy so you can use the back side!

Re:Go with tried and true (1)

KieranC (1807174) | about 3 years ago | (#36819640)

Print it

Re:Go with tried and true (0)

Anonymous Coward | about 3 years ago | (#36819656)

That might work, if you mean use the 5.25" floppies as coasters to stop the hard disks from getting scratched up on the desk...

Re:Go with tried and true (1)

toastar (573882) | about 3 years ago | (#36819822)

No Cost consideration? 2 Servers with Raid 7 Arrays/Servers using enterprise drives of course. LTO 5 Stacker With Weekly Iron Mountain Pickups. This is the same setup I would suggest for my clients for Seismic Data. I'd be happy to set it up for you for $125k. Depending on your opinions and general level paranoia, we can discuss online backups, the pricey part always seems to be fat pipe.

Hard drives (2)

serkit (2358056) | about 3 years ago | (#36819396)

Definitely hard drives

Re:Hard drives (2)

JMJimmy (2036122) | about 3 years ago | (#36819944)

Put it in the cloud! *waves arms like it's something mystical*

Seriously though, there is no great solution. Burned discs separate over time, there's not enough data on SSDs yet but it's not looking promising, platter drives are susceptible to radiation, tape to magnetic fields and degradation. HDD in triplicate, replace every 7-10 years is the "best" method right now. So despite being modded down, serkit is right. Hard drives.

Multiple copies (0)

Anonymous Cowar (1608865) | about 3 years ago | (#36819400)

If you're worried about long term reliability, try a raid-1 array of a spinning drive mirrored against an SSD. Make monthly backups to optical. That way, if your SSD fails, you still have two other options. This would probably be the most affordable method us mere mortals could have to hope to store long term data with a pretty good reliability. Unless you can get your hands on a second hand tape drive and some 500g tapes.

Re:Multiple copies (0)

Anonymous Coward | about 3 years ago | (#36819682)

Unless you buy extremely good archival grade discs, optical media is the worst suggestion. Have you ever tried doing an md5 check on a 5yr old dvd-r? Next, with 40-60GB files, thats what right now...$100 for every two files if you mirror them to ssd? Why not just buy 20 3TB hdd's, from different batches of course, and run an uber raid-1 mirrored across all 20. That'd be cheaper than buying the amount of ssd's required to back up just one single 3TB hdd.

Rotational media (4, Interesting)

BWJones (18351) | about 3 years ago | (#36819402)

For this project [utah.edu] , we have multiple multi-terabyte (5-18 terabyte) datasets that need backup. We have online and offline strategies and the offline strategy is simply multiple, redundant copies on hard drives stored in static proof containers onsite and off site.

Hard drives are *very* cheap all things considered, are easy to store, take up very little physical space and if things go badly, restoring from them is faster than just about any other method. For datasets in the GB range, its a no-brainer to go with hard disks.

Re:Rotational media (0)

Anonymous Coward | about 3 years ago | (#36819516)

Yes, with highly fault tolerant, such raid 6. and a _good_ storage controller such as a xyratex.

Re:Rotational media (3, Informative)

lucm (889690) | about 3 years ago | (#36819740)

> and a _good_ storage controller such as a xyratex.

I would rather run Windows Home Server on a RAID-0 of IBM DeathStars installed in a HP Pavilion than deal with a Xyratex.

Re:Rotational media (5, Insightful)

Mad Merlin (837387) | about 3 years ago | (#36819522)

I concur on this point, online storage really makes the most sense. Cheap, high performance (for sequential read/write) and easily expandable. You can get a single machine with dozens of SATA drives in it (including the drives) for way under 5 figures. When drives fail, they're simple to replace, and every couple years, migrate the whole thing to newer (faster, bigger) drives. Mirror your data unless you don't care about it. RAID 1/10 for really small datasets (2-4 drives), RAID 6 for moderate size datasets (5-10 drives) and RAID 60 for anything bigger.

A very important note to keep in mind... stay away from hardware RAID! When your controller dies, so does all your data, unless you have an identical spare controller card (buy it up front, they won't exist in a couple years). The same goes for fake RAID (ie, software RAID driven by the BIOS), but s/controller card/motherboard/g;. Pure software RAID (ie, using mdadm) is a safe bet.

Re:Rotational media (-1)

Anonymous Coward | about 3 years ago | (#36819608)

A very important note to keep in mind... stay away from hardware RAID! When your controller dies, so does all your data, unless you have an identical spare controller card

The xyratex mentioned above are a very good solution, and they do offer dual controllers, a fraction of the cost of a SAN and all the features.

Re:Rotational media (2)

Local ID10T (790134) | about 3 years ago | (#36819672)

A very important note to keep in mind... stay away from hardware RAID! When your controller dies, so does all your data, unless you have an identical spare controller card (buy it up front, they won't exist in a couple years). .

I have to disagree... Adaptec raid card -they stand the test of time. Hell i can still buy a 2940 controller card if I want one...new!

Re:Rotational media (1)

Spazmania (174582) | about 3 years ago | (#36819960)

I've used a dozen different brands and models of hardware raid cards. The only ones that haven't lost my data are the defunct Mylex DAC960 and HP's Smartarray controllers.

Had a problem with the mail server recently where a malfunctioning drive kept killing the scsi bus, causing the Smartarray controller to see every drive except that one as faulty. Once I figured out the bad drive, recovered without losing the data. Had similar problems using Adaptec and LSI cards a few years ago. Data gone. Maybe the current generation of Adaptec and LSI cards is better, but the ones I trust are the HP Smararray.

Linux software raid is also superb for reliability if you can tolerate the performance of software raid. Which for backups you generally can.

And for the record, an Adaptec 2940 is not a raid controller. It's a plain jane SCSI card -- something which unlike RAID controllers Adaptec did very well.

For the OP - go with hard disks. Use different brands and models and always use more than one. Some production runs are defective; I had a 10 disk raid 5 once where 6 went bad, all from the same production run. Killed me because they went bad faster than I replaced them in the raid 5. Putting copies of the file on at least two different brands and models will save you from that. Use different sizes too... if you put one copy on a 2TB seagate drive, put the other copy on a 1TB WD drive.

Keep some at home, some at work, some in the bank safe deposit box.

Re:Rotational media (1)

lucm (889690) | about 3 years ago | (#36819796)

> stay away from hardware RAID! When your controller dies, so does all your data, unless you have an identical spare controller card

I totally agree. On some systems even a broken RAID-1 cannot be recovered using a different controller.

It's a good thing that more and more filesystems are going beyond RAID (such as zfs or btrfs) and even storage unit are moving to block replication.

Re:Rotational media (2)

juventasone (517959) | about 3 years ago | (#36819622)

Even on a small scale this makes sense. The easiest is 2.5" external drives ($100 for 1TB). This avoids the mess of power adapters. If you need significantly more storage, you may want to consider a dock ($50) and internal desktop drives ($80 for 2TB). Consider this: you can buy from anywhere a USB adapter that will plug into a 20+ year old drive and any OS will mount it. Wish I could say the same about all my removable media...

Traditionally the way to do this is with tape. As you replace the drive (and you will), your tape capacity increases, but it will be read-compatible with your old tapes. The investment is huge, but it makes it very easy to replicate, take off-site, archive, etc.

Re:Rotational media (1)

lucm (889690) | about 3 years ago | (#36819754)

I agree with that, unless the budget is insanely low hard disks are a no-brainer. They are also easier to encrypt than tapes and offer a much higher rate of reuse.

Re:Rotational media (1)

mlts (1038732) | about 3 years ago | (#36819848)

Just make sure your encryption program is available, perhaps as freeware, or use a utility that is widespread and easily gotten. I've seen fancy encryption programs for drives that would be useless if there ever was a restore needed just because the license keys likely wouldn't work.

For encryption, I'd go with LUKS, or TrueCrypt on a disk level. For file encryption, gpg is solid, perhaps tar, bzip2, and gpg, although you might need a utility for error detection/correction to repair any data lost due to bad sectors.

The hardware is one piece of the puzzle. Having the right software to pull data off is the other. This is what is nice about hard disks -- assuming a common filesystem, a hard disk that has USB access today likely will be readable by most machines 10 or 20 years from now either directly, or via an adapter. Tape can be universal, but oftentimes a tape archive may not be readable because it was created in some wierd format (like the DOS backup programs that backed up to QIC drives in the '90s, all incompatible with each other.)

Tape (2)

sirsnork (530512) | about 3 years ago | (#36819408)

You probably need to define "best". How long do you really want to keep them for, and in what sort of environment.

Traditionally the answer is tape, and probably will be in your case too for files of that size. Optical isn't proven enough (at least for the sizes your're talking about) to be trusted, and HDD's need to be run up fairly regularly to keep working.

Re:Tape (1)

uvajed_ekil (914487) | about 3 years ago | (#36819592)

Why not punch cards? Huh huh... so clever!

Re:Tape (0)

Anonymous Coward | about 3 years ago | (#36819852)

Tape isn't obsolete. Not by a very long shot. I earn a very good living by managing backups that are stored to tape; if disk really were cheaper/better, companies would have abandoned tape a long time ago. (Hint: they haven't.)

If you're talking large quantities (hundreds of terabytes) of data that needs to be accessed relatively infrequently (in the sense that you need to work with a particular file, or set of files, for a period of time, and then move on to the next file set, leaving the one you just finished with languished, untouched, for a few months or years), you cannot beat tape. Hard disks have to be kept powered on, and there are issues with spinning them up and down on that, if I remember rightly, reduce the lifespan. With tape, you only need a couple of tape drives powered up; the raw storage sits there quietly, needing nothing more than physical space.

To the OP: you really need to define your usage patterns. How many of these files are we talking about? (ie: how much total data?) Presumably, by "archive" you mean "move it off my working space, onto something where it won't be touched for an extended period of time" - in which case, I'd buy an LTO drive (LTO2 and LTO3 are reasonable for consumers; LTO5 if you can afford it, and can front it with some very fast drives so the data spools down at streaming speeds) or two, and possibly a tape library to go with it. I'd only bother with external hard drives if money was very tight, or the quantities of data small enough that it would fit on a relatively low (5-10) number of tapes.

Re:Tape (1)

afidel (530433) | about 3 years ago | (#36819654)

For that small of files the most reliable inexpensive method is probably a small LTO2 library with each backup going to multiple tapes. $20 per tape with each capable of holding quite a few projects (200GB native). I just checked ebay and there are a ton of 2x LTO2 libraries for ~$500.

Re:Tape (1)

Anonymous Coward | about 3 years ago | (#36819670)


and HDD's need to be run up fairly regularly to keep working.

Citation needed.

Re:Tape (1)

Local ID10T (790134) | about 3 years ago | (#36819732)

Stage it to a HD for ease of access, back up the HD to tape for long term storage.

Dont forget to test your backups! Just because your software "confirms" that it wrote the data does not mean you can restore the data.

Large removable disk on the low end, tape highend (4, Informative)

mlts (1038732) | about 3 years ago | (#36819414)

BD-R disks are an idea, and relatively inexpensive, but your best bang per buck would be large removable disks in the 2-3 TB range. The reason I state "disks" plural is for obvious reasons.

I would also use a program like WinRAR with a recovery record, or one of the PAR utilities used for USENET to store your files in. This way, you can tell if there was file corruption, and have a good chance of recovering from it.

For serious stuff where money is less of an issue, I'd consider a LTO-5 tape drive and multiple tapes. Tapes tend to last longer than HDDs because they have very few moving parts.

Don't forget to see about copying your archives to new media every couple years. It isn't uncommon to be able to pop a 10+ year old tape or HDD in and pull off the contents... but it isn't uncommon either to find the HDD clicking, or the tape full of hard errors.

BitTorrent hash check (1)

Anonymous Coward | about 3 years ago | (#36819948)

I would also use a program like WinRAR with a recovery record, or one of the PAR utilities used for USENET to store your files in. This way, you can tell if there was file corruption, and have a good chance of recovering from it.

If all you want is a by-the-megabyte file corruption check, go with BitTorrent. Create a .torrent of each project directory. You can fill the tracker field with some bogus server name, say http://127.0.0.1./ [0.0.1] The beauty of a BitTorrent hash file is that you can pinpoint exactly where a file error occurs in a file, give and take a megabyte or whatever the file chunk you set for your BitTorrent file. This is unlike a ordinary md5 or sha1 check sum where all you know is whether a file is corrupted or not

Plain old SATA drives (0)

Anonymous Coward | about 3 years ago | (#36819424)

Use SATA drives, possibly in hot-pluggable trays. Treat them right, store them right, spin them up periodically, and use a filesystem (like ZFS) that can do data integrity checking. And as others have said, if it's important, mirror it (RAID 1, etc).

Re:Plain old SATA drives (1)

GrantRobertson (973370) | about 3 years ago | (#36819954)

This is exactly what I was going to recommend.

A lot of people assume that if one is going to store data on a hard drive then that drive must be powered up all the time. All reliability figures are based upon the assumption that the drives will be powered up and in use for most of a working day. However, if you only power up a drive when you need to store or retrieve data - data that is written only for archival purposes - then the drive could last a life-time.

In my system I use an external, dual-drive, eSATA connected, setup. (I like this one [startech.com] .) I only turn on the drive when I need to transfer files to it. When I don't need a drive in the dock, I put it in an anti-static bag with a desiccant packet (just as they came from the manufacturer), squeeze the whole thing into a slightly modified old VHS case (I cut out the things that go into the reel holes in the tape), and put it on the shelf - labeled, of course.

I prefer the dual dock so I can simply do a full-drive copy to make backups of my archive disks. At full eSATA speeds it doesn't take nearly as long or take up nearly as much real-world space as tape, and it is less expensive as well.

VHS (0)

commo1 (709770) | about 3 years ago | (#36819428)

Re:VHS (0)

Anonymous Coward | about 3 years ago | (#36819448)

You're an idiot.

At 200 K/s an arvid system will take 27 hours to write a 20GB file, and take 10 180-minute tapes. Why not just convert to ASCII and print out hardcopy?

Re:VHS (2)

ZorinLynx (31751) | about 3 years ago | (#36819468)

Whoosh!

Re:VHS (1)

lucm (889690) | about 3 years ago | (#36819806)

I'll wait until they have it on Blu-Ray or at least Tivo

HDD + Tape (0)

Anonymous Coward | about 3 years ago | (#36819440)

Get a bunch of 3tb HDD's and put them in a raid 6 then you are protected against multiple drive failures and so monthly/ bi-monthly tape backups

Spinning HD (0)

Anonymous Coward | about 3 years ago | (#36819464)

Multiple spinning drives regularly upgraded. I have 1.9 TB of my photography that I try to keep archived. I maintain HDs of different brands at work, another set at home and another set at the parents 100 miles away. Copy and duplicate these regularly. Right now integrating 300GB of photos into my system and its a PITA, but worth it. Anything short of the Ruskies frying every HD in America with an EMP burst has me sleeping soundly.

And though I would love to, the thought of burning some 500 archival DVDs or ~100 Blu-Rays has me staying away from optical media (this is what I used to use, and I currently have close to a 1000 CDs and DVDs of photos and projects sitting in my storage unit)

same goes for the half TB of music and movies, but my photos are more precious, therefore receive the extra attention.

NAS storage (0)

Anonymous Coward | about 3 years ago | (#36819466)

Hard disk NAS storage would work best. Spinning disk has been around long enough to make it reliable and cheap. for 250$ you can get a really good NAS setup with 2 to 3 TB.

Re:NAS storage (1)

perpenso (1613749) | about 3 years ago | (#36819584)

Hard disk NAS storage would work best. Spinning disk has been around long enough to make it reliable and cheap. for 250$ you can get a really good NAS setup with 2 to 3 TB.

And some NAS devices have multiple drives and can be configured for RAID.

No such thing (0)

Orgasmatron (8103) | about 3 years ago | (#36819470)

There is no such thing as offline storage any more, except as a transient backup of spinning media.

RAID it and spool a copy out to tape.

Bare Drives via Hot Pluggable Trayless SATA (3, Informative)

bynick (1038382) | about 3 years ago | (#36819472)

Screw tape... you pay $2,000 USD for the drive, $50+ per tape for a couple of hundred gigs. Go with bare drive external: Install a trayless SATA bay for 3.5" hard drives... this will run you $12. Buy some bare SATA drives.. these run $50 for 1TB and are available up to 3TB. I buy bare drive hard cases for about $3 each. My Intel ICH10R on-board RAID controller supports hot-swap -- so in effect it's a big 3.5" floppy.. that's right. If your tape drive breaks, you're out another two grand. This is far less expensive, faster, higher density, and random access. In addition, you can boot from it. Want RAID0? Install two trayless SATA bays for a total of $24 and back up in pairs.

Re:Bare Drives via Hot Pluggable Trayless SATA (0)

Anonymous Coward | about 3 years ago | (#36819566)

I really want to do this, really bad, but you're way smarter than me.

If you could provide newegg links to the right items to buy for this, you'd be my hero.

Parts List for This Method (1)

bynick (1038382) | about 3 years ago | (#36819676)

Trayless SATA - http://www.newegg.com/Product/Product.aspx?Item=N82E16817998041 [newegg.com] - This isn't the exact brand I used, but this is the style. Do some comparison shopping. The case I use for each drive is the ADIDT HS-1 for 3.5" HD. I bought them off ebay for about half of newegg's price. I couldn't find them listed at the moment on ebay, but there are plenty of hits on the web.. hit google and you'll see the pics.. assorted colors. They're stackable too and have spaces for labels. This is a strong case -- it takes two fingers for me to open the snap. I also print numeric tags with a label maker to stick on the drive for identification in the corner. 1 TB hard drives - http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&N=100007603%20600003269&IsNodeId=1&name=1TB%20and%20higher [newegg.com] - my pick is the Western Digital Green drives.. read up on their soft seek technology which made them the quietest drive at the time I researched them. They come in consumer and RAID versions. The consumer version works well for both applications and costs less. For the cost savings using this method, you can double up in drives which is a given for storing any data -- always have at minimum two copies. Because they're just plain drives, you won't need special hardware to read them if your PC is destroyed by natural disaster or stolen. Store one set off-site... safe-deposit box works good. Encryption is a plus http://www.truecrypt.org/ [truecrypt.org] .

Re:Bare Drives via Hot Pluggable Trayless SATA (0)

Anonymous Coward | about 3 years ago | (#36819780)

a slashdot reader that can't handle dealing with hot swap hard drives? wtf?

Re:Bare Drives via Hot Pluggable Trayless SATA (1)

bynick (1038382) | about 3 years ago | (#36819930)

Coward.

Re:Bare Drives via Hot Pluggable Trayless SATA (1)

lucm (889690) | about 3 years ago | (#36819880)

> Screw tape... you pay $2,000 USD for the drive, $50+ per tape for a couple of hundred gigs

What brand of hardware are your buying? Rolex? A LTO-5 cartridge is about 60$ and unless you format it with a very, very big cluster size it will store 1.5TB, not 200GB.

> I buy bare drive hard cases for about $3 each

Ok, I see. For tapes you shop at Rolex but for your "neat setup" you browse eBay until you find one of those Hong-Kong power sellers that ship on the Pacific Princess Express Line. Makes sense.

Re:Bare Drives via Hot Pluggable Trayless SATA (1)

bynick (1038382) | about 3 years ago | (#36819922)

Rolex - tehe.

Drobo... (0)

Anonymous Coward | about 3 years ago | (#36819476)

http://www.drobo.com/

Re:Drobo... (1)

sco08y (615665) | about 3 years ago | (#36819942)

Only makes sense if you're doing small scale stuff, and (because you don't understand depreciation) will want to hang on to old drives.

Drobo is very flexible, but horribly slow.

External SSD hard drive (1)

Billly Gates (198444) | about 3 years ago | (#36819484)

Make sure it is hot pluggable with USB (if one exists yet) as both IDE and Scsi have changed many times with incompatible adapters and cables with different plugs. Odds are they will change again and be unreadable in a couple of years.

Dvd's have rot in which the metallic thin sheet peels off. They say it is based on UV light damage but I found a Gentoo cd under a dark bed in a blinded room from 2005 that is rotting away as we speak. So BluRay discs are out of the question.

Another slashdotter mentioned an external hard drive but magnetic interference from the Earth would erase it like my old audio tapes within a decade or two.

Whatever you choose make sure it is external as USB and Firewire like to remain backwards compatible and this makes it easy to share between machines. Something solid state is the best way to make sure the data remains secure. Or find an internet provider where you can upload it too if you do not mind paying per month or year.

Re:External SSD hard drive (0)

Anonymous Coward | about 3 years ago | (#36819728)

I have old hdd's from 93-96 that worked flawlessly when I spun them up for shits n giggles a month back. I agree that optical media is the worst solution. Almost every disc, which I carefully stored after burning at 4x and md5sum'ing, now fails that same md5 less than 5yr later.

The practices are the same... (2)

fuzzyfuzzyfungus (1223518) | about 3 years ago | (#36819494)

Depends on price: HDDs are crazy cheap, for the capacity; but untrustworthy. However, thanks to the cheapness, redundancy, preferably in multiple locations, periodic testing/copying to newer disks/etc. is fairly affordable. Make sure that you have(either manually, at the utility level, or at the FS level, hashes/checksums) and hope for the best. LTOs are rather more durable, having fewer moving parts in the storage media; but the cost of entry is substantially higher. All the same principles apply, though.

There are no truly reliable storage mechanisms for large quantities of digital data, only storage mechanisms cheap enough that you can duplicate your way to reliability.

SATA drives (0)

kimvette (919543) | about 3 years ago | (#36819504)

Buy three SATA drives and make multiple copies, and store them separately. Hard drives are dirt cheap now.

Tape. (0)

Anonymous Coward | about 3 years ago | (#36819506)

You're talking about archiving, which I assume means permanent to long term (10 years or more) storage. I wouldn't trust flash, writable optical media, or rotational media for that time frame. This is what tapes were built for. Say LTO3 will let you put 10 projects on a tape, and new tapes are ~$30 each. The initial outlay isn't super-cheap ($600-1k for the drive), but you'll make up for it in the long run.

Online Storage (1)

esten (1024885) | about 3 years ago | (#36819510)

Why not go with an online storage solution such as Amazon S3 and let them be your backup and not worry about doing it yourself. I know that you can ship them a hard drive so you don't need to spend time uploading data.

Re:Online Storage (2)

lucm (889690) | about 3 years ago | (#36819932)

The title choosen by the author of the original post: "Best Offline Storage Method For Large Archives?"
Your answer: "Why not go with an online storage solution such as Amazon S3"

I suspect that one of you is off-topic, but I also wanted to say that S3 is really a great service and quite cheap.

Laptop drives (0)

Fordiman (689627) | about 3 years ago | (#36819514)

Oh, no. I've been in this argument before.

Ok, so Seagate sells a USB 3.0 SATA drive where the adaptor can be removed and used for other drives. Get one of those, and when if fills up, just start buying 1TB laptop SATA drives. They're small, cheap-ish, and more impact-resistant than their desktop counterparts.

2.5" (Laptop) Drives are Durable (1)

bynick (1038382) | about 3 years ago | (#36819692)

I'll second the 2.5" (laptop) hard drives -- I have seen them take a fall to the floor while powered up and survived. They are extremely durable. My pick is the WD Passport (run the toolkit to remove the virtual driver / backup disk and its as close as you can get to a 'plain drive' these days without the need for drivers or other junk / bells / whistles). It's good in USB 2.0 or 3.0. 3.0 is backwards compatible with 2.0. The 2.0 is $10 less expensive and the cable is a bit lighter. A USB 3.0 cable is comparable to Ethernet in feel.

3 hdds (1)

after.fallout.34t98e (1908288) | about 3 years ago | (#36819534)

1: Current online storage
2: online backups (live, hourly, daily, whatever...): a backup drive ready to take the place of the online storage at any time
3: offline backups

every month:
2 becomes 3
1 becomes 2
either 3 becomes 1 or new drive becomes 1

Tape/climate control (3, Informative)

triffid_98 (899609) | about 3 years ago | (#36819538)

You can't argue with Tape. It's been proven to last since the 1960's if kept in a climate controlled space (dry/cool). Just make sure to keep a spare tape drive handy (just ask NASA), because spare parts for 40 year old tape drives are surprisingly difficult to locate.

Optical isn't even close, assuming you're talking burned discs. Taiyo Yuden claims a 70 year shelf life, but they have only been around for what, 8 years tops?

Hard drives are an option if you've built a redundant array, but even with that you're still going to be out of luck if you burn up your raid controller.

Re:Tape/climate control (1)

hitmark (640295) | about 3 years ago | (#36819688)

Just wish that tape drives where a bit more available in prices ranges outside of the business stuff. Then again, i guess that is where optical come in. Burn two copies, and store them cool and dry. In the end, any media that allow the separation of rw hardware and the actual storage object without requiring a clean room is more reliable imo. This because rw hardware failure can then be recovered from.

Why settle for one type of media? (0)

perpenso (1613749) | about 3 years ago | (#36819548)

In this current climate, what is the 'best' method of archiving these? Spinny magnets? Solid state drives? USB? Tape? Blu-ray?

If the data is important why settle for one type of media? At least external HD and tape, maybe external SSD too. Move to newer media periodically.

My QIC-80 tapes from the 90s are probably unreadable by now. However their contents were moved from tape to CD to DVD over the years. My backups just sort of accumulate and grow over the years. Its practical for me to do this since I religiously keep things in /src and /doc hierarchies and only back these up. I don't bother with operating systems and applications, they can simply be reinstalled.

Re:Why settle for one type of media? (1)

triffid_98 (899609) | about 3 years ago | (#36819710)

You'd be surprised. Just for grins I restored my (circa 1989) QIC-80 tapes a year ago. No problems at all.

Optical, I've had both DVD and CD bitrot, even on the old Kodak 'gold' discs.

...That said, your point is totally valid. Multiple archive copies is the safe way to go. If you want to be even more secure, go with PAR. PAR or RAR recovery records will tell you when chunks are corrupt and can allow you to recover an uncorrupted copy even if both archives are damaged.

Punch cards (1)

JulianDraak (1918564) | about 3 years ago | (#36819558)

Punch cards made out of steel. As long as they don't deteriorate, you're good to go.

Re:Punch cards (1)

Onlyodin (1137597) | about 3 years ago | (#36819748)

How about Punch cards made out of titanium?

Magneto-optical or, cautiously, tape (1)

mentil (1748130) | about 3 years ago | (#36819560)

Last I looked into this, the best format in terms of reliability was magneto-optical. It heats up the disc with a laser before the magnetic bits are able to be manipulated, so it's unlikely to be corrupted by only magnetic interference or only light/heat. You can get a 9GB rewritable disc or 30GB write-once for ~$50.

There's also tape, which has massive capacities, but every anecdote about tape I've heard ends in "but the tape backup was partially/completely corrupted". Make two tape backups from the source (not copied from one tape to the other) using different technologies from different manufacturers if you absolutely must use tape. Keep one copy on-site for a few days if you'd normally ship backups to a storage center, so that you may not be required to recall the truck if the server explodes and the on-hand copy works.

Re:Magneto-optical or, cautiously, tape (1)

afidel (530433) | about 3 years ago | (#36819738)

Dude, I've put 6400+ LTO 1-4 tapes through 4 different libraries over the last 5 years and had exactly 3 failures to read, two of those cartridges were dropped and so the failure was not unexpected, the third had a manufacturing flaw where an extra piece of tape somehow got into the cartridge. We verify every tape after writing using a different drive in the library and do restores almost daily so it's not like I'm blindly trusting the backup reports. I've restored data from old DLT IV carts that were almost 15 years old without issue. People having problems with tape reliability were almost always using Traven, DAT/DSS, or QIC tape systems, enterprise grade DLT or LTO tapes (or IBM or StorageTek's proprietary formats) have always been reliable.

No Price Ceiling? (1)

krotscheck (132706) | about 3 years ago | (#36819578)

Carve the files onto titanium plates and store them in an underground bunker somewhere with little seismic activity.

Multiple backups (1)

uvajed_ekil (914487) | about 3 years ago | (#36819580)

If you have more than a few of these projects, SSDs are not yet a good choice for backups. You say price is no issue, but I doubt you'd want to buy 50 SSDs at their current prices. I'd suggest a few "spinny magnets," perhaps in an array if you need more apce than a couple of terabytes. Pros: low cost per terabyte, reasonable transfer speed, decent reliability, easy to implement as a tried and true technology. And of course for added safety, mirror the drive/drives to a second set. USB (if you are refering to removable thumb drives) would not be your best choice, though tape might be worth considering, especially as a secondary backup. I know nothing about Blu-ray, so I won't comment, though the capacity of the disks is a little low, isn't it?. Personally, I'd go with redundant spinny magnets to prevent having your collection on multiple removable drives/discs/whaetevers that can be lost.

Hook up a redundant raid array, or two arrays, put them in a safe place, forget it. Tapes or a portable HD array to be taken off-site to guard destruction against fires, tsunamis, tornados, hurricanes, Godzilla, and bombs. How much data you have, and how frequently you will add to the collection, are factors that need to be considered but aren't mentioned here. My suggestion assumes that you will backup frequently and have a lot of your 40-60GB projects. Less data or less need to back it up might steer you towards something else.

unlimited money (0)

Anonymous Coward | about 3 years ago | (#36819590)

if money is not an object, dump it all on standard HDD and burn discs for backup. i dont know how long you plan to keep it but relevant quote
"How long will a Blu-ray Disc last?
It is expected to last 30 years or more when stored at room temperature. The optimum temperature is 68F, and the optimum relative humidity is 40%."
from relevant source http://www.tapestockonline.com/sotdfubldibd.html

discs and lots of them (0)

Anonymous Coward | about 3 years ago | (#36819606)

It doesn't really matter what you choose to backup to. Just make sure you have multiple copies (and are stored in multiple locations) and also look at having some kind of corruption method built in that you can use should you be unable to get the data back (see http://en.wikipedia.org/wiki/Parchive). Also if your going the cd/dvd/blueray method make sure you verify the burns in a separate drive than you did the burn in (different manufacture also helps, md5sums of files is fine)? And make sure you store the media is a cool dry spot that doesn't see the light of day (http://en.wikipedia.org/wiki/Humidor ??) It will take a little extra time but it's well worth it if your data is indeed important.. using a offsite backup service is the easy way if you can schedule how often you will want data offsite

Proven for over 100 years: (0)

Anonymous Coward | about 3 years ago | (#36819620)

Put your important work on paper.

It's been proven to last long and can contain quite a lot of information.

It Doesn't Matter (1)

rally2xs (1093023) | about 3 years ago | (#36819626)

what you do, it'll all be lost in the next giant solar flare that gets shot at earth, or the next EMP attack. Nothing will survive in the way of computer equipment to be able to read it.

Drawn patterns (0)

Anonymous Coward | about 3 years ago | (#36819632)

Forgot what it is called, but it is where multi-colored lines and patterns are drawn out on a sheet of paper/fabric/other material. This kind of storage as I recall can do up to several terabytes (maybe up to 50TB?).

The answer is the same as the last time (1)

BitZtream (692029) | about 3 years ago | (#36819644)

The real answer is ... hire someone that knows what they are doing, as by asking the question you clearly don't.

Yes, thats a shitty answer that you're not going to like, but its the right answer.

The longer answer is ... you back it up int he same place you back all your important data up. Which could be done any number of ways.

Spinning platters is fine if you maintain them, as is every other method of data access under the sun.

Stop trying to stick the data in some sort of long term storage and just keep all your data active, as YOU MOVE to new storage mediums, you move ALL your data with you at the same time. So you are always using current technology and worrying about pulling those bits off something that is hard to find in 10 years won't be an issue because you'll not be using something hard to find in 10 years, you'll be using whatever is popular in 10 years.

This is really easy to accomplish.

You have server A and server B. You work on server A, its close to you, has redundant storage and fast access ... and automatically syncs to server B, which is also full of redundant storage and several thousand miles away from you for disaster recovery purposes.

You neglect the most important question... (2)

stox (131684) | about 3 years ago | (#36819652)

How long?

What is good for a decade may not be good for a century, and vice versa.

For millenium+ archives, nothing beats punch cards.

Redundancy by method (1)

Onlyodin (1137597) | about 3 years ago | (#36819662)

The key to data protection is risk mitigation. Depending on how important your data is, you should probably consider employing multiple methods of protection, such as a Disk or SSD based copy with a Tape or Optical based copy.

Personally, I'd keep a near-online copy by means of an External Drive or NAS device which can be powered down if necessary, but if you want to go further you could lock that in a fire-resistant safe/filing cabinet, but you should definitely have another copy offsite somewhere.

You could even use an online storage provider? Let them worry about maintaining the hardware? But you still need a second (offline, offsite) copy, imho.

Obvious answer (0)

Anonymous Coward | about 3 years ago | (#36819668)

Multiple copies

Multiple formats

Multiple locations

If your data is truly worth something to you, this is the only and best approach. LTO-5, BD-R, and RAID are not bad ideas.

Data storage vs transmission (0)

Anonymous Coward | about 3 years ago | (#36819684)

Spinny magnets? Solid state drives? USB? Tape? Blu-ray?

One of these things is not like the others. USB is not a way to store your data: it's a way to transfer it.

On the original topic, though: since you said that price isn't an object, whatever you go with, get enough of it to store multiple copies. Your best storage-certainty per dollar is probably from hard drives for small volumes, and tape for large volumes (since you need to pay for the tape drive).

Torvalds quote (3, Interesting)

bmo (77928) | about 3 years ago | (#36819690)

"Only wimps use tape backup: _real_ men just upload their important stuff on ftp, and let the rest of the world mirror it" - Linus Torvalds

--
BMO

Dedup or Tape (3, Interesting)

lucm (889690) | about 3 years ago | (#36819696)

If price is not an issue, a great solution is to go with a data-deduplication device (such as EMC DataDomain or IBM Protectier). If you were to host one unit in your basement and the other in coloc environment far from your home, you could setup replication and have a very reliable archive. Coloc of a 1U device can be quite cheap, I have one of them for which I pay less than 100$ a month.

If you have a smaller budget, then the best cost-benefit is still found on tape, and it can even work in case of network disruption. Like Andrew Tannenbaum said: "Never underestimate the bandwidth of a station wagon full of tapes". A single LTO-5 tape is very cheap (50-60$) and can store 1.5TB (can easily double that with dedup).

There are other interesting technologies out there, such as MAID, which you can use as a VTL with a good backup software to maintain a reliable archive, however cheap disks are cheap and in a MAID configuration they might not last as long as typical disks because of the on/off behavior.

Proper printouts (0)

Anonymous Coward | about 3 years ago | (#36819798)

You can always rebuild it if you have to from proper printouts. Get them professionally published to archive quality. Then to save your actual digital stuff, rotational disk archiving. No digital storage medium will be 100% effective so use a Raid array and plan to upgrade it every 3-4 years to a new array of better/bigger drives. Even then though, your absolute best best is that first part, then put them into archival storage in multiple locations.

NAS+2 online stores (1)

keith_nt4 (612247) | about 3 years ago | (#36819850)

I guess this isn't a very popular suggestion. And you seemed to imply you wanted a local archive for your data, something you do yourself.

I would just a large iSCSI NAS. 2TB Drives are really cheap these days. FreeNAS even lets you flag a drive as a hotspare so you don't have to as much about failures.

Then back this NAS up to at least two online storage services. Make sure they're not both the same thing on the back end (like amazon's S3). Actually Carbonite personal can't distinguish iSCSI from a local drive and is unlimited storage for personal use. I'm sure that violates some terms some where but technically it's possible. Pick another high capacity online service for redundancy.

Also, encrypt the data locally *before* it's uploaded (it's just a good idea).

You didn't say how much total data you have to archive or how fast if at all it is growing nor how often you would need to access it. I have seen amateurs making 40TB storage servers from component parts. Honestly I can't think of a reason to go with anything other large capacity drives. I assume 2TB drives don't have any where to go but down in price.

Why bash the OP? (1)

bynick (1038382) | about 3 years ago | (#36819914)

Why are the OPs in-thread questions hidden? And what's up with bashing him for asking questions? I took the time to answer him/her twice. To the person that did that you make me sick. I hope he/she can find the reply since it met the same fate otherwise this site is a waste of bandwidth.

This is not an advertisement (1)

symbolset (646467) | about 3 years ago | (#36819950)

It does look like one though, I admit.

Some time ago I read about the BackBlaze box here on slashdot [slashdot.org] . Essentially it's a 4U server chassis design that holds 45 LFF SATA drives and a server motherboard, plus the requisite connector bits and power and so on. BackBlaze is a storage provider that offers some online storage service and they designed the chassis to do high-density storage and hired a company, Protocase, to build it. BackBlaze doesn't sell servers, or server designs. They designed it because they needed it and shared it in the hope others would give back design improvements.

BackBlaze open-sourced the design and authorized Protocase to sell it. I learned about this when I followed up on the story with Protocase because I'm in the server trade and the storage density was intriguing. We went back and forth but I never bought the thing.

Purely by coincidence I got an email from Protocase just today. They're selling the thing now as a fully built server with everything you need (motherboard, processor, PSU, expanders, drive controllers, etc) -- except drives now for $5395.00 (1-4 units) and $4995.00 (5-9 units). Their website won't sell it, you have to contact lpodgursky@protocase.com via email for how to buy this because they're not geeks like us - they bend sheet metal for a living. At the time of the slashdot story this would store 67TB, but nowadays it's twice that. 3TB drives now cost $120, which would be $10,800 roughly for 135TB raw or probably 110TB usable - which puts it at $100 per served terabyte. Some folks would consider that a bargain. You'll want the 10Gbps links as that much volume will be link constrained for volume migrations. For storage density that's 1.35PB (raw) per rack, which is about as good as it gets right now. Bring cash or AmEx because Protocase is a tiny company and can't offer terms for new customers.

Of course for stuff that's commercially valuable that much data would cost a lot to recreate. I would probably want two of these at least, and store multiple copies on each one. Advances in HDD density should take care of expansion needs and migration needs if your data is currently less than 50TB. For software look into OpenFiler, which is free to use and has commercial support available.

This is not an advertisement. I don't work for any of these people. I don't care if you buy this thing. But if it was my money and my data and it was worth $50K or more... I'd buy several of these and find some geographically diverse locations to put them and devise a strategy for replicating and migrating my data as the hardware grew stale.

So as long as I'm posting this... to totally sexy this up with automatically tiered storage for performance I'd add a couple Fusion-IO IODrive Octals per unit with Fusion-IO's directCache software to front this storage with 10TB of SSD cache per 135TB of slow SATA disk. That should get you up to over 1M 512B iops per node if you've stepped up to Infiniband QDR to handle the bandwidth. And I don't work with them either. This last bit will cost several times all of the rest of it. Probably layer lustre file system on top of that for large volume needs. If you need less volume, look into drobo.

I've already gone overlength for this post, so I may as well go completely nuts. So here's some of Lewis Carroll's "Alice in Wonderland":

* "Just the place for a Snark!" the Bellman cried,
As he landed his crew with care;
Supporting each man on the top of the tide
By a finger entwined in his hair.
o Fit the First : The Landing

* "Just the place for a Snark! I have said it twice:
That alone should encourage the crew.
Just the place for a Snark! I have said it thrice:
What I tell you three times is true."

"Three times" is a good rule for data. If you put data in three disparate places it's less likely to be lost. Alice in Wonderland is a great reference manual for just about everything. The Reverend Dodgson was a wise man.

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Create a Slashdot Account

Loading...