Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
Data Storage Hardware

Bulk Data Storage For The Common Man? 483

Posted by timothy from the the-common-man-with-a-tb-a-month dept.
Vigyaan writes "Lately, I have been looking into different bulk data storage options available to a common man. My work depends on generating, storing and analyzing a large amount of data -- averaging about 1 TB per month. I would like to have a storage system which is automated, fast, reliable and most importantly does not cost the price of an eye. Right now, I have a 4 node Linux cluster with 10 large hard disks (total capacity 1.6 TB); data storage roughly costs about $0.60/GB (excluding the cost of PC hardware). But long term storage is painful -- DVDs cost about $0.10-$0.15/GB but takes too much human time and leaving data on hard disks makes me nervous because of possible failures. RAID is a possibility, but it increases the cost significantly. I was wondering, if Slashdot readers have any recommendations for a cheap automated way to store and retrieve data."
This discussion has been archived. No new comments can be posted.

Bulk Data Storage For The Common Man? More

Bulk Data Storage For The Common Man?

Comments Filter:

  • Finally a use for my 1GB Gmail invites... (Score:4, Funny)

    by anakin357 ( 69114 ) on Monday July 05, 2004 @06:55PM (#9616501) Homepage
    I'll send you a couple.

  • Hard disks (Score:5, Informative)

    by ConsumedByTV ( 243497 ) on Monday July 05, 2004 @06:56PM (#9616508) Homepage
    You're always going to get a better rate with Hard drives but you're going to be prone to failure.

    If you buy them in bulk you can save.

    Burning DVDs is going to take you forever and drive you nuts.

    Find a hotswappable set of drives and use that for your offline backups. Use a raid for your current backups.
    • My personal option for smaller data sets would be hard disks too. Specifically, hard disks in external fast-usb or firewire enclosures, so they really become hot pluggable.

      A friend tells me that 20GB tapes are cheap. With a simple tape changing robot that might be manageable, just put in a new tape container every day...

      • Re:Hard disks (Score:4, Informative)

        by littlerubberfeet ( 453565 ) on Monday July 05, 2004 @07:18PM (#9616672)
        hard disks are good.

        If you want one of those nifty things with robotic arms and whatnot, plan on spending upwards of $3500. The AIT Automated Tape Library goes for that much and holds only 15 tapes. Plan on spending tens of thousands for something like Ampex's DIS 914 for 30 Terabytes.

        Your friend is right: tapes or cheap. The equipment needed to support them is expensive, slow and error prone. It gets cost effective once you have enough money for a new Porsche though...

      • Re:Hard disks (Score:3, Interesting)

        by ConsumedByTV ( 243497 )
        I personally use a firewire enclosure, it's fast, it's hotpluggable and it's easy to swap the internal disk.

      • Re:Hard disks (Score:3, Informative)

        by silas_moeckel ( 234313 )
        If your going to just plug in backup and swap try the USB 2.0 to IDE backup boxes pretty much its a power brick and an US to IDE chipset in a plastic case with a 40 pin IDE connector on it. You plug in the drive and your good to go. No cases or hot swap caddies to deal with. And 5400 RPM drives dont get hot to the touch sitting on the desk. It's not pretty but if your just running backup keep on buying $100 IDE disks (generaly best cost per GB)

      • Re:Hard disks (Score:4, Informative)

        by eric76 ( 679787 ) on Monday July 05, 2004 @09:14PM (#9617351)
        Tapes can be pretty dependable, but you need a better quality tape system than that typically sold for PC backups. The 20 GB tapes are just not that dependable.

        If I had the money, at a minimum I'd get a tape drive that could handle the 200 GB (uncompressed) tapes. Something like IBM's LTO Gen-2 Tape Library. That should run a bit less than $6,000.

        For that matter, if I won the lottery, my first purchase would probably be a top of the line tape backup system instead of a the usual new car.

        Since I can't afford it, I use DVDs and CDs for backups. They are a pain in the neck and are not that dependable, but I keep backups up to a year on DVD+RW so if one fails, hopefully the others will have the data.

        Instead of writing directly to the DVD writer, I write the backups to disk and then copy the backup sets to the DVDs.

        I also keep a complete current backup of nearly everything important on a seperate computer.

    • 4*400Go Sata on Raid 5 (Score:5, Informative)

      by da5idnetlimit.com ( 410908 ) on Monday July 05, 2004 @07:23PM (#9616695) Journal
      depending on the value of your data, you should try having a nice 4*400Go SATA in raid 5 *2, possibly using a distributed file system for redundancy...

      Not the cheapest, but fast, simple and saves you the unholy pleasure of having 2-3 DLT boxes to archive/cycle each month...

      You already have a linux cluster, so implementing a distributed file system, or even simply a nightly incremental mirror to the target server if you can afford losing one day work/computation...

      It would help if you told us what sort of data you work with... from databases and to automated telescope tracking system, both need large amount of storage, but you won't need the same system array for each...

      I seem to remember a /. story [slashdot.org] on a rackable Petabyte storage system [archive.org]

      You don't need to go to the Petabyte capacuty but you will find some interesting comments on filesystems, disk virtualisation, 1U rack providers and so on....so a 1 Terabyte rack server is definetly possible...

      Good luck...

    • Re:Hard disks (Score:5, Interesting)

      by DetrimentalFiend ( 233753 ) * on Monday July 05, 2004 @07:40PM (#9616784)
      We're dealing with storage issues right now at work, and what we're doing is buying a server with 8x250 GB SATA drives. We then run the drives in raid 5, so we have 1.75TB of storage space (unformatted). Including computer costs, it's running us about $2.50 per GB, but it's a very beefy 3u server. For backup, we're currently backing up to tape. That costs us under $0.50 per GB with ultrium tapes. For some of our data, we've been backing up to DVD's, but we've pretty much given up on that. In the long run, it's not worth it.

    • Re:Hard disks (Score:3, Informative)

      by eric76 ( 679787 )
      Use RAID to increase your on-line availability.

      RAID does not a backup system make. You still need backups.

      For increased on-line availability, how about a good distribued file system with several servers? And, of course, back everything up anyway.

  • Waiting for ... (Score:3, Interesting)

    by Entropy ( 6967 ) on Monday July 05, 2004 @06:56PM (#9616511)
    Blu ray based dvd burners.

    Those will be sweet =)

  • good luck (Score:5, Funny)

    by Madcapjack ( 635982 ) on Monday July 05, 2004 @06:56PM (#9616512)
    PRINTSCREEN should do the trick.

    • Re:good luck (Score:2, Funny)

      by mlk ( 18543 )
      please dont joke about that, not too long ago I received an email, please help me open this word document. Complete with a BMP, natrally I assumed this was the error message. Nope, it was a "print screen" of its Icon. And yes, while I had said picture up, the user did try double clicking.

      Poor luser.

      • Re:good luck (Score:3, Funny)

        by schtum ( 166052 )
        Translation:

        Please don't joke about that. Not too long ago, I received an email asking for help opening a Word document. Attached was a bitmap image which I naturally assumed was an error message. Instead, it was a screen-capture of the document's icon! The user was double-clicking on the image!

        So I shot her.

  • Wirewire drives? (Score:5, Interesting)

    by NanoGator ( 522640 ) on Monday July 05, 2004 @06:57PM (#9616522) Homepage Journal
    For long term storage, how do you feel about firewire drives? Maybe not as cheap as you'd like, but you can get them in >160 gig flavors, plus you can hook them up to just about anything. Once you do the backup, which'd be a simple copy and paste, you can just unplug the drive and store it in a safe or something.

    Again, I'm not sure if that's as cheap as you'd want, but that's a solution I came up with for a similar problem. My company's going to be 3D rendering some stuff that could end up eating 50 megabytes a frame. (Extra data is stored for future refinement... I can go into detail if I've piqued anybody's curiosity.) We can't afford to lose this data, so the Firewire drive approach is what we're considering right now.

    • Re:Wirewire drives? (Score:4, Informative)

      by Rik van Riel ( 4968 ) on Monday July 05, 2004 @07:02PM (#9616557) Homepage
      For long term storage, how do you feel about firewire drives? Maybe not as cheap as you'd like,

      Oh, but they are cheap. Just buy a large IDE disk and a $30 firewire/fast-usb enclosure.

      I'm just not sure about the "long term", though. I have no idea what the shelf life of a hard disk is.

    • Re:Wirewire drives? (Score:5, Informative)

      by littlerubberfeet ( 453565 ) on Monday July 05, 2004 @07:11PM (#9616619)
      Lemme address the firewire thing: I work in a sound studio, and we generate about 5-8 gigs of data a month, mostly music for TV. This isn't a huge amount, but we rely on multiple sets of Firewire drives for backup and then internal hard drives for current projects. This means we have all 400 or so projects at our fingertips. Given how fast we do things, this is important.

      Lacie makes their 1 terabyte firewire (943 gigabyte formatted) drive. I we get them for $1,080 a drive (Macmall matched Provantage's price). This is more then the article author spends now per gig, but these drives have done quite well in the studio. You can find cheaper firewire though.

      We are at the point where hard drives give the best bang for the buck. The only fault of firewire is that my bosses have burned several bridges. ground yourself before unplugging the drives. The bridges were cheap though. In any case, hard drives are probably the most failsafe and cost effective solution, with firewire being the easiest interface to use those drives with.
    • I'd love to see a Firewire hub that could act as a hardware RAID controller. A program on the computer would enable management of the RAID controller, and once formatted, the logical volumes would be presented to the host computer as standard disk volumes, eliminating the need for any special drivers on the host computer, as well as enabling the entire array to be portable to other platforms.

      How expensive could something like this really be? $300-400 at most, I'd have to guess considering what most plac
      • > I'd love to see a Firewire hub that could act as a hardware RAID controller.

        Firewire drives can be daisychained, and in fact OS X allows you to set up software RAID on multiple firewire drives attached to the system. You can't move them to another system and get access, but that's about the only limitation that I've found and it's more than decent for local high-density storage..

  • Personally I prefer something in a blonde (Score:4, Insightful)

    by kfg ( 145172 ) on Monday July 05, 2004 @06:58PM (#9616525)
    I was wondering, if Slashdot readers have any recommendations for a cheap automated way to store and retrieve data."

    Although the good ones don't come cheap. I guess this another case of "pick any two."

    KFG
    • mods, this is not off-topic.

      KFG meant to say "You can have fast, good, or cheap. Pick two."

      It's an old software design maxim that applies suprisingly well to this subject.


      • KFG meant to say "You can have fast, good, or cheap. Pick two."

        It's an old software design maxim that applies suprisingly well to this subject.

        ...and to many things, particularly if you replace "fast" with "convenient". Just for kicks, think about it.

        Food? Check. Clothing? Check. Beer? Check. Housing construction? Check.

        Pretty much anything that involves the exchange of money for goods and services follows this maxim.

  • 1TB a month?!? (Score:5, Funny)

    by stinkydog ( 191778 ) <sd@stCOWrangedog.net minus herbivore> on Monday July 05, 2004 @06:59PM (#9616529) Homepage
    Short of launching his own space probe, the only way for this guy to consume a TB a month of storage is a serious porn habit. Just post your 'content' on Edonkey and it will be available when you 'need' it. You likely only watch them once anyway.

    SD

    • Re:1TB a month?!? (Score:5, Funny)

      by grasshoppa ( 657393 ) on Monday July 05, 2004 @07:07PM (#9616593) Homepage
      Using this method, I have achieved my life long dream of tapeless ( well, everything-less ) backups.

      I simply make a tar.bz2 file with all my important files, filter it through gpg, then post it on edonkey, usually titled, "Olsen twins getting it on", and then usually the date.

      Viola, instant backup that is available to me whereever I may go.

  • Cheap solution (Score:4, Insightful)

    by codeguru73 ( 689454 ) on Monday July 05, 2004 @06:59PM (#9616530)
    Buy some inexpensive IDE drives with high storage capacity and use a software raid solution. What kind of budget do you have anyway?

  • Bulk storage? (Score:3, Funny)

    by mikers ( 137971 ) on Monday July 05, 2004 @07:00PM (#9616534)
    I got a couple of drawers of old floppy disks. $10 takes 'em. Plenty of bulk.

    The Sony "lifetime" warranty may still be good on them too!

  • age old problem... (Score:5, Insightful)

    by Lumpy ( 12016 ) on Monday July 05, 2004 @07:00PM (#9616538) Homepage
    Ahh the large amount of data that has X value versus a storage solution...

    If your data is worth $20,000.00 then a $2000.00 solution is dirt cheap.

    what is your data worth? that is where you need to start and then look at the 10-30% of the data's value to start looking at how must to spend on it's storage.

    If 1 month's data was lost forever, how much money would it cost the company? that is your actual $ amount that you should be shopping at.

    and that is how I got the company to buy a $20,000.00 1000 tape DLT jukebox.

    my data is worth over $100,000 a month and is much lower than yours is size.

    That is where you need to start. Justify your storage costs by figureing out what it is worth to begin with.

  • Well... (Score:4, Interesting)

    by stonecypher ( 118140 ) <stonecypher@noSpam.gmail.com> on Monday July 05, 2004 @07:00PM (#9616541) Homepage Journal
    Depending on your budget, the appropriate thing to do may be to get an automated DVD burning system to do scheduled incremental backups in duplicate. We used to do that with CDs at an ISP I used to work at. It's unfortunately difficult to search for while not getting people pirating movies, but this is the first thing I found on Google [ymi.com]; doubtless there's better out there.

  • Tape? (Score:2, Interesting)

    by darkjedi521 ( 744526 )
    I haven't seen anyone mention magnetic tape yet. I'm sure it has its drawbacks too, but considering its still widely used for backup purposes in a commericial environment, it can't be too bad. Especially depending on how much a cartridge can hold. Its not the cheapest, but it might be something to look into.
    • This company [certance.com] came up after a quick search.
      I think everyone imagines a room full of those spinning reel-to-reel things when you mention tape. It's archaic, but viable today!

  • Buy an older tape drive (Score:5, Informative)

    by Apparition-X ( 617975 ) on Monday July 05, 2004 @07:01PM (#9616548)
    Look for an LTO gen 1 or SDLT220/320 on ebay, with a SCSI connection (some of them are fibre, and I assume you don't want to go there!). Don't forget to pick up some tapes. In general, this sounds like it would work if you plan on doing this for a while, and can leverage the initial investment over months or years.

    Capacities are (for the cost of a sub $50 tape):
    - LTO1: 100 GB uncompressed
    - LTO2: 200 GB uncompressed
    - SDLT220: 110 GB uncompressed
    - SDLT320: 160 GB uncompressed

    If your data is particularly ammenable to compression (i.e. database data) you could easily get 3 or 4 to 1 compression with these drives without sinking your CPU utilization.

  • Oh, I see... (Score:4, Funny)

    by spinkham ( 56603 ) on Monday July 05, 2004 @07:01PM (#9616550)
    You want it fast, cheap, reliable, easy, and now, eh? Good luck with that.... Sounds like a request from the PHB...

  • 1 TB/Month (Score:2, Insightful)

    by nikonius ( 17796 )
    It does not sound like your needs are anywhere near that of the 'common man'. You sound more like a power user to me. Somethimes you have to pay for heavy-duty storage as the cost of doing business.
  • Aren't Write-Once Read-Many solutions ideal for this kind of thing?

    If you're not looking for permanent backups, the per-media cost may be prohibitive though.

  • Cheap and Big (Score:3, Insightful)

    by guamman ( 527778 ) on Monday July 05, 2004 @07:02PM (#9616556)
    Tape Drives - Probably the cheapest way to store large amounts of information. The only drawback is that they aren't fast. However, If your harddrives are large enough to hold the data you are currently working on and tapes are used exclusively for backup then a speed problem shouldn't be . . . a problem.

  • eh (Score:2, Interesting)

    by maxbang ( 598632 )

    If you don't have the patience for DVD backups (neither do I), then you're pretty much stuck with RAID. So buck up, spend the extra cash, and setup a storage box or two on the network with one or two terabytes in each. I have a branch of my network setup on gigabit, one box has 250 GB of storage on RAID 1 across two 250 GB (this one's for video projects), the other has 160 GB in RAID 0 (my learning system). Works fine and easy as hell to setup. If I need to add storage I can either add some drives or just a

  • compression (Score:3, Informative)

    by Suppafly ( 179830 ) <slashdot@sup p a f l y .net> on Monday July 05, 2004 @07:05PM (#9616575)
    First off, if you aren't already compressing that data, start. You may be able to cut the size down dramatically using compression.

    Then backup using tapes just like every other place that has to do backups. Generally do full backups once a week and incremental ones nightly or whatever is necessary based on the data you are working with.

  • spongedrive is best (Score:5, Funny)

    by cubyrop ( 647235 ) on Monday July 05, 2004 @07:06PM (#9616580)
    i am responsible for providing storage solutions for a mid-sized content creation company which, through version archiving, accumulates near 1-200 GB per day. they require access to their media backups on a rolling basis, so tapes are not an option.

    i have found that a Teutonium cluster of 6.5 TB Spongedrives (either Cray or SecreTech are fine) fits the bill nicely. housed in a 15-unit rack server, the amoeba-shaped drives utilize BioLas technology to store data on 6-dimensional Moebius Cilia for a slick seek time of 0.00 ms.

    a cluster costs about $45,000 USD but the price should come down in 2004 Q4 when SecreTech launches their new 40-platter blackholium SCSI's.

  • Drawbacks, what are you willing to put up with? (Score:5, Informative)

    by Anonymous Coward on Monday July 05, 2004 @07:06PM (#9616581)
    All forms of media/backups have their own drawbacks... but some aren't as bad as others, and the others often are more accessable.

    Tape: Tapes break, they wear, they have dropouts, take a while to back everything up, can't always access files if you just want to restore something (Different methods vary, folks)... but ultimately, it's cheap when you use DAT because they're a common media. Swap the tapes twice as often (and throw old ones out) if you're paranoid about tape related failures.

    Hard Drive: Most common form of backup I see now, mainly for the 1:1 size factor. Yeah, drives fail, too. Sometimes you have a pretty good warning when this is going to happen, sometimes you don't. (My 13GB Maxtor and 40GB IBM Deathstar drives both went *pfft* on reboot.) Get enough of them at once, you could swap out the logic boards if one does fry out. Ultimately, RAID or just simple 1:1 mirroring is probably the most efficient and easy method. Accessing bits and pieces is also easiest under this method. I personally just use an external USB2 case with a 120GB drive in it. Everything I want to back up goes on that drive, and then eventually... DVDRs. I turn off the drive when I don't need it... hopefully prolonging the life of it when I need it most.

    DVDR: Not anymore. If we had these new-fangled DVDR discs (+ or -) say... when 2 to 6GB drives were common.... sure... But in addition to hard drives, recovering selective files is easy under this method too... Unless you use a backup program that crunches everything together on the disc in some spanning format. Burn times can be tedious... but it's not bad if you consider the overall amount of data you're putting on the disc. Cheaper than quality-brand name CDRs, though, in terms of price per mega/gigabyte. Only an idiot would trust $0.01-per-disc spindles for long-term backups. Even the longevity of DVDR has yet to be seen...

    CDR: I'm not going to bother.

    Network: Well, still relies on hard drives and other components... but good if you don't want to saddle one room with a ton of boxes. Simply for space and efficiency... external drive is probably better anyway.

    Old fashioned method: Print everything out and keep it in a filing cabinet somewhere. You could always OCRA the stuff later. ;-)

  • LTO Ultrium 2 Tape Drive (Score:4, Informative)

    by jeffgeno ( 737363 ) on Monday July 05, 2004 @07:07PM (#9616587)
    The drive will run about $4000, but the tapes are only around $0.20/GB assuming a 1.5:1 compression ratio. And keeping that assumption, 1 TB of data should only take 3 200 GB native tapes per month, so swapping wouldn't be so bad with the single tape drive. An autoloading library would be significantly more expensive, but if you really need automation, that's the way to go.
  • The best idea I could give you is to just create a sister system, where you mirror all your data. Not cheap, but cheaper than getting a pro-grade solution.

    The reason you won't find such things on the cheap is because the average person with a PC doesn't even know what a GB is. He simply goes into the store, the sly salesman says "oh, what do you need it for," and then says "well 60-80 gb should be all you ever need."

    Now, contrast that to me - my friends shit when they hear I have a 250 gb drive and a 12

    • I have a dual MP2400 with 4 x 120GB WD 1200JB drives. I have a single XP2800 machine with 4 x 120GB WD 1200JB drives, 2 x 200GB Maxtor 6Y200P0, and 2 x maxtor 7Y250P0 drives. I have a dual Xeon 2.8Ghz machine with 4 x 120GB Maxtor 6Y120M0 drives. That accounts for all my regularly used machines. I guess I'm not a common man. :-) Not to brag...

      I have to disagree with the sister system though. For most geeks like you and I a sister system would be fairly adequate. It would be better with an occasiona

  • Look at what the rest of the corporate world uses for large scale storage management. It is still ruled by Tape drives.

    I don't know how much an eye goes for at the moment, but if you can spring for a Super DLT drive you'll get up 320GB (Compressed) for each tape.

    It all comes down to the Quality:Cost:Time triangle.

  • Easy (Score:5, Funny)

    by Pedrito ( 94783 ) on Monday July 05, 2004 @07:10PM (#9616609)
    I use bioneural gel packs at a cost of $0.04 per teraquad. What is this hard drive of which you speak?
  • Well it isn't going to happen, you -HAVE- to drop change for what you want, as a back-up solution. There really isn't any way around that.
    There are many plausible suggestions though that won't break the bank totally. One of course is raid as has been mentioned and will be a few times I imagine. But you may also wish to look into hot swappable solutions.
    USB 1.1/2.0, Firewire and SATA are all relatively cheap storage solutions if you shop around (Pricewatch [pricewatch.com] is a good place if you are willing.). You can con
  • Buy a tape drive.

    If you have money burning a hole in your pocket, but a tape changer, so you don't have to change the tapes.

  • Hijack Cassini (Score:5, Funny)

    by Anonymous Coward on Monday July 05, 2004 @07:13PM (#9616635)
    ... and program it as a repeater.

    It's about 90 minutes away, so at 250 Kbps that's over one terabit in storage on the way out there, and another terabit on the way back.

    Worst-case access latency is about three hours, though. Maybe the hard disks are a better idea.

    If you send your probe^H^H^H^H^H repeater to Alpha Centauri, you'll get more than 20,000 times the storage capacity.

  • Use those HDDs! (Score:2, Interesting)

    by jsm008us ( 774007 )
    You can get cheap computers from the trash, donations, bulk, etc. You can use that cluster to mirror your data once or twice. I don't know what data you have, but if you have the same data on more than 1 different hard drive, you can be rest assured it will be fine. Or you can just print it all!

    The stockmarket is backed up to three (or more?) seperate locations. Look into NVRAM (e.g. flash media) or a cluster with all those hard drives linked together, with a constant backup. With the builtin IDE controlle

    • Re:Use those HDDs! (Score:5, Funny)

      by Cecil ( 37810 ) on Monday July 05, 2004 @07:53PM (#9616863) Homepage
      I'm just curious, do you have any idea how much data 1 Terabyte is? Are you suggesting that he PRINT it?

      Let's say for the sake of argument that all 256 bytes can be printed as a visibly distinguishable character, or that he's got 1TB of plaintext. Also assume you can fit 10,000 characters on a 8 1/2 by 11 page.

      You can fit 10^4 bytes per page, and you need to print 10^12 bytes (I know, it's actually 2^120, but that needlessly complicates the math, so shush)

      That means you will need 10^12 bytes / 10^4 bytes/page = 10^8 pages.

      One hundred million pages. Assuming he has a good laser printer with infinite toner, let's say he can print 60 ppm or one page per second. It would take one hundred million seconds to print the data, which is 1157 days, or a little over 3 years.

      Given that he generates 1TB per month, I think this backup plan would probably become the top agenda item of most of the anti-deforestation groups out there.

      • Re:Use those HDDs! (Score:4, Interesting)

        by deranged unix nut ( 20524 ) on Monday July 05, 2004 @08:50PM (#9617199) Homepage
        Depending on how you do it, you can get a lot more than the density that you assume. Check out www.paperdisk.com [paperdisk.com].

        That said, this method would still be more than twice as expensive as storing data on hard drives, would still require a million pages, but would take a little under 2 weeks to print.

        It still doesn't seem like a feasible option.

        The up-side is that, if stored properly, the data would likely be safe potentially for many hundreds of years.

  • Do what Google does (Score:5, Insightful)

    by glinden ( 56181 ) * on Monday July 05, 2004 @07:15PM (#9616654) Homepage Journal
    Build yourself a cluster [computer.org] of cheap boxes with cheap IDE disks and replicate your data across them. Because the data is replicated across your cluster, no need for backups or RAID.

    • Bad idea (Score:5, Informative)

      by Anonymous Coward on Monday July 05, 2004 @07:47PM (#9616825)
      Google stores data for fast access, not for reliable storage. They don't care if they lose a few hundred gigs when a handful of disks die, they'll just re-spider it in a few days when the Googlebot hits the sites which were lost. Their solution is NOT optimized for reliable storage and it's not suited in the slightest to this guy's problem.

      • Re:Bad idea (Score:4, Informative)

        by Anonymous Coward on Monday July 05, 2004 @09:21PM (#9617392)
        this is incorrect. GFS (Google File System) has many systems with the same data on each node. These nodes have 3 copies of each data slice. If one server fails then the other two mirrors re-copy the data.. If two fail then the server mirrors the data to ensure it is never lost.

        google does not want ANY data to be lost. The have many mirrors of all data.

    • Re:Do what Google does (Score:4, Insightful)

      by swb ( 14022 ) on Monday July 05, 2004 @07:50PM (#9616846)
      It's a great idea, but one of the problems is what happens when your data goes bad before you realize it and it gets replicated. Then you want what you had yesterday, and that means tape.

      You can solve this by ensuring some kind of in-process backup (like a SQL maintenance schedule, where it replicates itself), but then you're loading your replication process with a bunch of data that doesn't really need to be online, it needs to be in a vault someplace.

      Besides, Sarbannes-Oxley and the IRS want you to keep backups 5+ years anyway, so this replication-only model is only good for data whose internal integrity isn't meaningful to anyone but the owner.

  • Vital information left out (Score:4, Insightful)

    by dfghjk ( 711126 ) on Monday July 05, 2004 @07:16PM (#9616656)
    How many months at 1TB/month do you require access to online? After you are done with data can you discard it or do you need it archived? What is the cost of losing your data set at any given time? In what manner do you expect to access it (read/write mixture and sizes plus aggregate throughput and number of client connections). The answers to these questions could cause the cost of a solution to vary but a couple orders of magnitude.

  • options options, what is your time and data worth? (Score:5, Insightful)

    by segfaultcoredump ( 226031 ) on Monday July 05, 2004 @07:17PM (#9616660)
    Lets see.... hard Drives are running about $0.50 per GB, DVD's are running about $0.06 per GB (100 pack, "house brand", not something I'd put my data on but this is slashdot, and there are idiots out there who think that it is a good idea), and tapes are also running about $0.20 -> 0.50 per GB (for the DLT/AIT/LTO type, the ones that have enough capacity to not drive you nuts)

    So, you can put your data on 4-5 HD's, 10 tapes or 232 DVD's per month. The Cost of doing so will be about $500 per month for the tapes or HD's and $50 for the DVD's (assuming your time cost $0)

    At work, we had a need to keep a few TB of data online permanently, so we purchased a few NexSAN [nexsan.com] ATABeast's. At $50,000 for 10TB of usable storage ($5/GB), they may be a bit out of your price range. The advantage is that you can hold almost a years worth of data and it is protected by RAID5. It also makes management a lot easier, since it is very difficult to mount 42 300G drives in a single chassis (and it takes only 4U of rack space).

    On the low end, NexSAN has the ATABoy2 or ATABaby (2TB or 1TB) for the $8-$15K range. This will let you hold a months worth of data

    On the high end, You have EMC disk arrays (Think upwards or $20+/GB for the 'cheap' stuff from them.

    Overall, if you have 1TB per month, you need to either a) get a grant to fund your work, b) hire somebody to swap DVD's for you or b) seriously rethink your data generation.

    Any of the "cheap" storage methods have serious drawbacks, and the low cost ones are, well, not so low cost if $15,000 sounds like a lot of money to you.

    otherwise, good luck

  • If its volume you want (Score:5, Informative)

    by TheUncleBob ( 791234 ) on Monday July 05, 2004 @07:17PM (#9616663) Homepage
    If you are more interested in volume than speed, then the emphasis should be on the 'ID' part of RAID. Inexpensive Disks. If you used 160GB Drives, which appear to have the best bang for your buck at the moment, and put 6 (yes 6!) in a pc. Just use any old cheap pc (I use 200-400Mhz PII)

    Run the disks RAID 5 and you will get about 800GB of storage for $600 . Now get two cheap ata100 cards so you have a total of 6 channels, and mount each drive as a master on each channel. Build a 2gb root partition on the first disk (mirror it if you want) and then set the rest of the space up as a huge raid 5 array.

    Et Voila cheap, big server. To archive data, turn off pc, and throw into attic :-)

  • CD Changer (Score:2, Interesting)

    by andrebsd ( 685491 )
    Well, I have a cd changer for computers made by NSM... It's scsi (comes with a 2x reader origionaly) so all you gotta do is find a scsi dvd burner (or a long enough ide cable and convert it, since the motors are all powered by a com port anyway) and replace thd drive, (or like in my case, a cd-rw - had the drive for a while, so at the time a dvd burner would have cost to much) then you have 100 dvd's you can burn data to automatically, and when those are full just swap them out for new ones.

    Now the problem

  • DLT is the way to go (Score:3, Informative)

    by pastpolls ( 585509 ) on Monday July 05, 2004 @07:23PM (#9616696)
    I actually use a DLT with autoloader I got off ebay for under $200. I then bought a lot of used DLT tapes (100) and use them to backup my Video and DVD projects. It is great because when I fill my offline storage (about 1TB) I just fire up the backup software and get the old DLT going overnight. It is done by morning and the shelf life for those tapes is about 20 years.

  • Ultrium (Score:3, Informative)

    by 7vEn_T_7vEn ( 794241 ) on Monday July 05, 2004 @07:24PM (#9616698)
    I'm not sure what your budget is but if your like me you want something that complies to standards so it will be around, is cheap and effective. For this I would have to recommend an Ultrium tape backup drive. The drive is standards based (google it) and the tapes are dirt cheap a 200/400 gb tape pulls up for $55. If you figure (hardware compression) 250gb of storage per tape then it will cost just $.22/gigabyte. The problem is that the drive itself is listing for about $2600, not exactly cheap but it's guaranteed to be backwards compatible with future lto standards and the media is as cheap as you could possible ask for. One more thought, look into an LTO Gen 1 solution (100/200) for a cheap drive, cost per gigabyte is roughly the same, it will just take more swapping.

  • Consider Online Backup (Score:2, Interesting)

    by jp10558 ( 748604 )
    One company that provides massive online backup and storage at reasonable prices is Streamload [streamload.com]. You might want to check them out.

  • Only on slashdot (Score:5, Funny)

    by gexen ( 123248 ) on Monday July 05, 2004 @07:29PM (#9616728)
    Only on Slashdot would they start talking about huge storage arrays and title it "for the common man"

  • carousel for everyman (Score:4, Informative)

    by Doc Ruby ( 173196 ) on Monday July 05, 2004 @07:29PM (#9616729) Homepage Journal
    A DVD-R jukebox [dvdchanger.com] can give you 200 DVDs at once. That's $3600 (drive/changer) [nextag.com] + $268 (1000 DVD-Rs) [pricewatch.com], for (1000*4.7GB) 4.7TB@$4000, or $1.18:GB. That's almost double your HD cost, but you'd need at least another host PC, and multiple controllers for the 16HD RAID, which is probably another $1000. And another $268 buys you another 4-5 months storage, so by next April you're down to $0.14:GB; in a year you're at $0.12:GB. A shelf of 200-disc "CD" books will hold your archives, 1 book per carousel for "fast" retrieval. Backup all your DVDs offsite at $0.27:GB. As DVD-R prices fall over time, you're probably looking at something like $0.05:GB, probably less than even plummeting HD prices. And the DVDs (especially with the cheap backups) are much more reliable, especially over 10 years, than the HDs. If you are looking at 10 year archive, at $80:month in DVDs, for 29% more money you can add a second host PC/changer set, left in their boxes, in case the original PC/changers fail.

  • Um, 1TB a month in IDE drives is cheap . . . (Score:3, Informative)

    by millisa ( 151093 ) on Monday July 05, 2004 @07:32PM (#9616743)
    I would hope that if you are working with a TB of data, the value of that data is pretty high . . .

    Promise SX6000 = $255.95. (6) 200GB IDE drives in a Raid 5 = $624.95

    If you had a separate boot drive from the SX6000, you could just bring the system down for a couple hour maintenance once a month and slam all the drives out and put fresh ones in.

    Just keep buying new 200GB drives anymore and shelf the old ones (or if its *really* valuable and your home firesafe isn't enough, pay Iron mountain or someone to keep it).

    There aren't hidden labor costs outside of those two hours it takes to setup a new array every month (DVD's are about 60 bucks a month for a TB, with a hundred or so for a drive (which *will* need to be replaced occasionally if you are burning that much) but you'll spend hours and hours just dealing with the swap outs and breaking up your data . .. )

    If you don't have to keep the TB of data after a month or three, then your price gets even cheaper after you invest in your initial hard drive media sets . . . and you can put all the drives in hot swap chassis to further minimize your time dealing with the issue.

    Of course this is all moot if your 1TB of data isn't valuable enough to invest 600 a month in . . .

  • DVD Autoloader (Score:3, Insightful)

    by SuperJason ( 726019 ) on Monday July 05, 2004 @07:33PM (#9616751) Homepage
    Explain this to me, I can buy a 200 disc cd changer for $100 bucks, but the same thing with a burner (cd/dvd) runs thousands of dollars. Isn't there any company out there that can do it cheaper?

    Heck, I remember a slashdot article about a guy who built one out of WOOD!

    This would be a great solution for short term recovery storage. Just keep a stack of CD's or DVD's ready, and it will load them in and burn them all automatically.

    On a site note, it would be great for converting a 400 disc cd collection into MP3's.

  • The simple economics of your "work" (Score:4, Insightful)

    by swb ( 14022 ) on Monday July 05, 2004 @07:45PM (#9616811)
    If your "work" (as in food, housing and income) requires this kind of storage, you should be charging the kind of money that can make the ecomomics of such data storage actually viable. I'm assuming that some of the really high-end storage devices from EMC, Hitachi, et al could handle your data generation/replication/backup needs effortlessly.

    If that's too expensive (and it usually is), you can kludge your own system using low-end stuff from Hpaq/IBM/Dell's x86-server-oriented product lines. LTO1 drives are pretty cheap and we've found them to be very reliable over the past 3+ years, as well as offering 100 gig native per tape.

    If even that's too expensive, then I seriously think you need to re-think the economics of your work situation. If your work doesn't cover your capital costs, you're not charging enough. If the work and data are business valuable enough, cutting your storage bill to the bone by building Linux clusters crammed with IDE HDDs is just a bad business decision.

    If this is just your hobby-type work, then you need a cheaper hobby, like heroin addiction or something affordable. Physical space and electricity aren't cheap enough in a metropolitan area to burn through 1TB of storage per month, let along reliable data storage.

  • You're NOT using RAID??!!!! (Score:3, Insightful)

    by swordgeek ( 112599 ) on Monday July 05, 2004 @08:55PM (#9617230) Journal
    Let me get this straight: You have a four-node cluster, you have 1.6TB of online storage, and you need some sort of permanence; and you're not using RAID of any form?

    This is utter insanity! Without RAID, your only hope of safety is in your backups--which you're only asking about now!

    RAID your data ASAP, and then start looking for backup systems. Take a look at some of the DLT4000 replacements.

  • Another perspective (Score:3, Interesting)

    by v1 ( 525388 ) on Monday July 05, 2004 @08:56PM (#9617235) Homepage Journal

    Just tossing out another point of view - similar but different than some of the others previously discussed. First off, examine the data you are keeping - do you really need that much? Nowadays it's common to be able to acquire data faster than it can be processed, and if you never stop gathering data, well, you never will catch up, only fall farther and farther behind.

    If you DO need this data, and you are going to need it for awhile, (year or more) I'd recommend cheap HDs. They also have an advantage of being easily catalogged, and are untouchable when compared with access time of tapes. Don't go raid5 though, this is not "catastrophy-proof". (flood, fire, tornado, etc) For catastrophy protection, mirror your drives. When you have them loaded up with data, pull the FW cables and swap drives in the enclosures with fresh empty drives. Label them well, and then take each half of the mirror to DIFFERENT LOCATIONS. It's OK to keep one set on-site, but the other set must be somewhere else, preferably in another zip code. This will allow you near instant access to your data (since it's onsite), will protect your data from mechanical failure (through mirroring) and will protect you against catastrophy. (you WILL need to acquire new firewire boxes etc if your office gets leveled... don't forget this detail in general - the data is of no value if you lack the equipment (tape drives etc) to read it back in with) I know you can get compression and fit more on a tape etc by using archiving software, but it may be worth the extra cost to obey the KISS rule and just simply drag and drop the data to the formatted HDs. This will make data recovery MUCH SIMPLER, and if there are errors on the HD when you need to recover, this will insure you can actually recover most/all of the information. Archive streams and tapes are notorious for losing 100% of the data that follows a corruption point in the stream.

    Once you know you no longer need a specific set, drop it back into the pool of usable drives. Buy them by the case, it's much cheaper this way. It also is advisable to buy the same make/model every time you have to get more drives, even if there are newer, larger, cheaper models out, because having all the same drives means one less complication to worry about in times of crisis.

  • Xraid (Score:3, Insightful)

    by Phrack ( 9361 ) on Monday July 05, 2004 @09:42PM (#9617500)
    I haven't tried it myself, but Apple's Xraid appears to be gaining in popularity as a reasonably priced bulk data storage solution. It reportedly works with Linux, Windows, Netware and, of course, Macs.

    If that doesn't suit ya, and it's bulk storage without necessarily speed you're looking for, check into the ATABoy line from Nexsan.

    • Re:Xraid (Score:3, Funny)

      by Lukey Boy ( 16717 )
      I do believe that this is the first time I've ever seen the word Apple used in the same sentence as "reasonably priced".

  • What are your near- and long-term requirements (Score:5, Informative)

    by TBone ( 5692 ) on Monday July 05, 2004 @10:50PM (#9617842) Homepage

    I looked through some of the answers here, and as near as I can tell, you've got a bunch of home hobbyists telling you how to back up your home computers. Perhaps all your needs entail is a computer with an external IDE drive array and 4-10 200G SATA drives in it. But from your initial post, it's not clear what you need your offline storage _for_.

    First of all, you mention that you generate and use 1G of data a month. What happens at the end of that month? Does all of the data become useless? Is some of it carried through? Is it useful for historical processing for some time after it's not "live" any more? The disposition of that offline data is important; you can't determine how you can most effectively back up your data until you know what you need to do with that data once it's backed up.

    Since no one cares about backing up old data that they never use any more, I'm going to assume you need this data in some form in the future. I'm also assuming that your data ages out completely every month.

    Realistically, you have two options: Large redundant disk arrays, or tape. Various factors give credence to one or the other.

    First of all, get off of the SATA hacks, and realize you're going to need to go to SCSI, whether you end up with disk or tape. You're backing up data, you're going ot want it to be reliably written out, and SCSI is the de facto standard for backup architecture. Yes, you pay more for it, but there's a reason for it: the SCSI equipment I manage at work fails a fraction of the percentage of time that the various IDE/ATA systems fail. While SATA is marketed as a consumer technology, it will never meet the rigors of being a reliable backup methodology.

    • Media Cost: Tape wins over disk here. LTO tape is running, at a quick check, for about $75 retail for 200/100G tapes. Even assuming only reasonable compression, you're looking at 150G for $75 bucks. And that is single-cart pricing; tape pricing quickly drops if you're ordering in bulk (typically in packs of 10, then at the 3-packs level, then more, check with your preferred media vendor)
    • Hardware Cost: Disk wins, but it's a double-edged sword - every disk you own has electrical and mechanical failure chances. The more disks you have, the more likely you are to lose one of them. The more you're storing on disk, the more you open yourself to a catastrophic failure of those disks themselves. High-end fast tape drives and libraries are expensive, but they just _work_. You plug them in, load your preferred tape management software (hell, run mtx for that matter), and start backing stuff up. No formatting, settings up arrays, hot-swap schedules, anything like that. But you pay through the nose for it - expect to spend into the $10K range for a large-scale tape storage solution that you could match (in short-term storage duration) for a couple of thousand dollars for a disk-based solution.
    • Hosting Space: Try to store 10TB of disk, and you'll need an air conditioner in that room just to cool down the disk cabinet and controllers. 10TB of tape just sits there though; you can store 4TB of tape online in a small 3U (about 6 inches) tape library - that's 24 tapes, and such libraries typically also support two drives. Go to 5-6U, and you can get 4 drives and over 50 tapes. If those were 200GB LTO tapes, you'd be looking at up to 10TB of storage available online, or easily offline and off-siteable. In addition, tape is easily expandable. Need more storage space? Buy another tape. No new hardware needed, no power concerns, just drop it in the drive or library and go.
    • Speed: Disk definitely has an edge. Set up an decent SCSI RAID5 array (real hardware raid across multiple disks on separate physical controllers, not this playtime software 0+1 homebrew IDE raid crap) and watch your write speeds triple. If you need to back up that 1 TB overnight, you don't have much of a choice but to go to disk in some form. But again, you pay a price for it. The speed you save in the

  • How about abusing physics law's? (Score:3, Interesting)

    by OlivierB ( 709839 ) on Tuesday July 06, 2004 @02:34AM (#9619041)
    I remember reading one day about some research somebody did about abusing the network capacity to store data. Basically he would send mail to himself via a third party smtp server. Of course he would tell his destination server to ignore his messages until a set date, then refuse the messages which would then be bounced back to his originatin email acocunt. By having a roll on that he achieved some pretty amazing storage for FREE! with ultra reliable ISP grade mail backup. Now aplly to same principle to space! Saw you have a server on Mars. You could transmit to Mars the data in full before MArs even started receiving the Data. When Mars would receive the data it would immediately send it back, not even waiting for the message to be completely received. Thus the data would not use any storage on mars either. At this point you have achieved media less storage. And have abused the network capacity of Space. Talk about the geek factor in that!. I don't really wan't to model this network's capacity but everybody here understands that it is a function of the transmission rate, celerity, distance with "relay" server. Of course there is an amount of data for which you will start needing some sort of storage on both servers. This will noly happen if the data has time to do a return trip to and from the relay server in less time than one can transmit the data in full. Improve the transmission rate and your network "memory capacity" multiplies.

  • Work Demands should be realistic. (Score:3)

    by Linus Sixpack ( 709619 ) on Tuesday July 06, 2004 @06:50AM (#9619854) Journal
    My work demands 1TB a month ....

    It sounds like you need a good cost benifit analysis and an idea of a budged.

    First RAID your existing data.

    Second Replicate any working solution you have now identically for next month and backup hardware.

    Have a serious talk with work as to what is expensive and what you can afford. What happens if a data set is lost? How much damage\cost would that incur? I would look int AIT drives from Sony.

    It sounds like you are in a frame of mind where you see everything as expensive. This will heavily influence your decision. Walk through a data disaster scenario with your backers and examine your costs in that light.

    ls

Slashdot Top Deals

Remember, UNIX spelled backwards is XINU. -- Mt.

Close