Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Ask Slashdot: Practical Bitrot Detection For Backups?

timothy posted about 10 months ago | from the error-detected-goodbye dept.

Data Storage 321

An anonymous reader writes "There is a lot of advice about backing up data, but it seems to boil down to distributing it to several places (other local or network drives, off-site drives, in the cloud, etc.). We have hundreds of thousands of family pictures and videos we're trying to save using this advice. But in some sparse searching of our archives, we're seeing bitrot destroying our memories. With the quantity of data (~2 TB at present), it's not really practical for us to examine every one of these periodically so we can manually restore them from a different copy. We'd love it if the filesystem could detect this and try correcting first, and if it couldn't correct the problem, it could trigger the restoration. But that only seems to be an option for RAID type systems, where the drives are colocated. Is there a combination of tools that can automatically detect these failures and restore the data from other remote copies without us having to manually examine each image/video and restore them by hand? (It might also be reasonable to ask for the ability to detect a backup drive with enough errors that it needs replacing altogether.)"

cancel ×

321 comments

Sorry! There are no comments related to the filter you selected.

PAR2 (5, Informative)

Anonymous Coward | about 10 months ago | (#45651923)

Re:PAR2 (1)

Anonymous Coward | about 10 months ago | (#45652269)

dvdisaster also uses Reed Solomon Codes, but lets you burn the data to a CD/DVD/BD in a single image:

http://dvdisaster.net/en/index.html

Re: PAR2 (0)

Anonymous Coward | about 10 months ago | (#45652765)

Well, I'm glad one person remembers optical media and its lack of this side effect. I usually archive stuff, such as photos and work. Its easy to use a marker to write a date range on the actual disc.

Doing so on-the-fly prevents the need for doing this in bulk.

ZFS filesystem (5, Informative)

Anonymous Coward | about 10 months ago | (#45651939)

One single cmd will do that,

zpool scrub

Re:ZFS filesystem (1)

ravenswood1000 (543817) | about 10 months ago | (#45651953)

Yep, ZFS

BTRFS filesystem (0)

Anonymous Coward | about 10 months ago | (#45652151)

Or, similarly for BTRFS:

btrfs scrub start /btrfs

Re:BTRFS filesystem (4, Informative)

mlts (1038732) | about 10 months ago | (#45652681)

I'll be the heretic here, but on Windows 8.1 and Windows Server 2012 R2, there is a feature called Storage Spaces. It works similar to ZFS where you toss drives into a pool, then create a volume that is either simple, mirror, or with parity, and Windows does the rest. If a volume needs more space, toss some more drives in the pool.

To boot, it even offers autotiering so data can be stored on a SSD that is frequently used, or remain on the HDDs if it isn't. Deduplication is handled on the filesystem level [1].

No, this isn't a replacement for a SAN with RAID 6 and real-time deduplication, but it does get Windows at least in the same ballgame as Oracle with ZFS.

[1]: Not active deduplication. The data is initially stored duplicated, but a background task finds identical blocks and adds pointers. Of course, the made from scratch filesystem, ReFS (which has the ability to check for bit rot on reads like ZFS), doesn't have this, so one is still stuck with NTFS for this feature.

Re:ZFS filesystem (5, Informative)

vecctor (935163) | about 10 months ago | (#45652283)

Agreed, ZFS does exactly this, though without the remote file retrieval portion.

To elaborate:

http://en.wikipedia.org/wiki/ZFS#ZFS_data_integrity [wikipedia.org]

End-to-end file system checksumming is built in, but by itself this will only tell you the files are corrupt. To get the automatic correction, you also need to use one of the RAID-Z modes (multiple drives in a software raid). OP said they wanted to avoid that, but for this kind of data I think it should be done. Having both RAID and an offsite copy is the best course.

You could combine it with some scripts inside a storage appliance (or old PC) using something like Nas4Free (http://www.nas4free.org/), but I'm not sure what it has "out of the box" for doing something like the remote file retrieval. What it would give is the drive health checks that OP was talking about; this can be done with both S.M.A.R.T. info and emailing error reports every time the system does a scrub of the data (which can be scheduled).

Building something like this may cost a bit more than for just an external drive, but for this kind of irreplaceable data it is worth it. A small atom server board with 3-4 drives attached would be plenty, would take minimal power, and would allow access to the data from anywhere (for automated offsite backup pushes, viewing files from other devices in the house, etc).

I run a nas4free box at home with RAID-Z3 and have been very happy with the capabilities. In this configuration you can lose 3 drives completely and not lose any data.

Re:ZFS filesystem (5, Informative)

Guspaz (556486) | about 10 months ago | (#45652421)

You don't need raidz or multiple drives to get protection against corrupt blocks with ZFS. It supports ditto blocks, which basically just means mirrored copies of blocks. It tries to keep ditto blocks as far apart from eachother on the disk as possible.

By default, ZFS only uses ditto blocks for important filesystem metadata (the more important the data, the more copies). But you can tell it that you want to use ditto blocks on user data too. All you do is set the "copies" property:

# zfs set copies=2 tank

Re:ZFS filesystem (2, Informative)

Mike Kirk (58190) | about 10 months ago | (#45652591)

I'm another fan of backups to disks stitched together with ZFS. In the last year I've had two cases where "zfs scrub" started to report and correct errors in files one to two months in advance of a physical hard drive failure (I have it scheduled to run weekly). Eventually the drives faulted and were replaced, but I had plenty of warning, and RAIDZ2 kept everything humming along perfectly while I sourced replacements.

For offsite backups I currently rotate offline HDD's, but I should move to Cloud storage. Give a bit of my surplus space and bandwidth to someone like Symform, and in turn they give me a free little slice of the Cloud to have TrueCrypt archives mirrored into. Win-win!

Excellent question (0)

Anonymous Coward | about 10 months ago | (#45651949)

I really hope this discussion provides good answers, with practical solutions for Windows, IOS, and Linux... I think that this is the sort of thing that everyone could really use!

Are there cloud storage providers that can do this for the above example of an approx. 2 TB data set, and provide complete security?

Re:Excellent question (2)

sandytaru (1158959) | about 10 months ago | (#45652081)

There are, but you'll be paying a lot of $$$ for that kind of storage in the cloud. I get 4GB for free from DropBox. SkyDrive from Microsoft will set you back $1000/month for 2TB - DropBox is about twice that much. It's not really practical for media files.

A much better solution would be archival quality Blue-Rays. They can hold 25 GB apiece and they're supposed to last 100 years, but they really just need to last long enough until a new, even denser storage media comes along.

Re:Excellent question (5, Informative)

SirMasterboy (872152) | about 10 months ago | (#45652173)

Not all cloud storage is expensive. It's only $4 a month for unlimited backups to CrashPlan.

They also do checksums and versioning and can be set to never remove deleted files from the backup.

I have 12.8TB backed up to them and it's been working great.

Other than that, ZFS can't be beat. I use that as well.

Re:Excellent question (1)

yakatz (1176317) | about 10 months ago | (#45652737)

Second shoutout for Crashplan! I have eight computers backing up to one account with "unlimited" storage and versioning.

Re:Excellent question (1)

mlts (1038732) | about 10 months ago | (#45652739)

I'm curious how that is doable. Even Amazon Glacier would be about $10.24 per terabyte stored per month, so I'd be looking at about $130/month for that much info.

I am not passing judgement... just have not heard much about CrashPlan, good/bad other than a quick search on it.

Re:Excellent question (1)

SirMasterboy (872152) | about 10 months ago | (#45652833)

Well, BackBlaze is another similar backup company who is far more public about their costs and operations. I think they have said their customer break-even point is around 3-4TB. So if most customers have far less than that, then a few can have far more and it all works out.

http://www.wired.com/insights/wp-content/uploads/2011/10/backblaze-cost.png [wired.com]

Re:Excellent question (1)

heypete (60671) | about 10 months ago | (#45652923)

Users that utilize large amounts of storage are relatively uncommon and are subsidized, in part, by users who utilize less storage. If everyone used terabytes of storage at $4/month, that wouldn't really be sustainable.

Although just a personal anecdote, I've used CrashPlan for ~4 years now (with 11 computers belonging to various family members all backing up to their service with a total of around 500GB being stored with them). Zero complaints. It's done everything I expected, always worked, and never had issues. When I had a laptop stolen and purchased a replacement, I was able to restore all the files from CrashPlan in about a day or two of downloading. I highly recommend it.

Re:Excellent question (3, Interesting)

mlts (1038732) | about 10 months ago | (#45652511)

In reality, Dropbox, Skydrive, and other cloud services should be treated as a type of media, just like BD-ROMs, tape, SDD, HDD, and even hard copy.

The trick is to use different media to protect against different things. My Blu-Ray disks protect an archive against tampering or CryptoLocker (barring a hack that flashes the BD burner's ROM to allow the laser to overwrite written sectors.) However, they have to be maintained in a good environment with a good indexing system. My files stashed on Dropbox bring me accessibility virtually anywhere... but malware that erases files could wipe that volume out in no time.

Similar with external HDDs. Those are great for dealing with a complete bare metal restore, but provide little to no protection against malware. Tape, OTOH, is expensive for the drive and requires a fast computer, but once the read-only tab is flipped or the WORM session is closed, the data is there until the tape is physically destroyed.

Of course, there is not just media... there are backup programs. This is why I use the KISS principle when it comes to backups. I use an archiving utility to break up a large backup into segments (with recovery segments to allow the archive to be repaired should media go bad), then burn the segments onto optical media.

I've found that using a backup utility can work well... until one has to restore, the company is out of business, and one can't find the CD key or serial number so the software will install. One major program I used for years worked excellently... then just refused to support new optical drives (as in ignoring them completely.) So, unless I can find a DVD drive on its antiquated hardware list on eBay, all my backups are inaccessible. I was lucky enough to find that and copy the data to a HDD, but using the lowest common denominator is a good thing.

Backups are the often neglected underbelly of the IT world. While storage, security, availability and other technologies have advanced significantly, backups on the non-enterprise level are still languishing behind in almost every way possible. It was only a few years ago that encryption became standard with backup utilities [1].

[1]: With encryption comes key management, and some backup programs make that easy, some make it incredibly hard.

Re:Excellent question (-1, Troll)

drussell (132373) | about 10 months ago | (#45652149)

Excellent question

No, it's not! This is supposed to be a nerd site for people with at least SOME kind of technical ability (or at least it USED to be...) You're telling me you don't even know how to store your files?!! Do us all a favor and please stop visiting slashdot. Thank you!

Re:Excellent question (0)

Anonymous Coward | about 10 months ago | (#45652295)

He has already expressed knowledge regarding how to STORE his files. He is asking for expert help on how to store them BETTER. Read his post again, and offer some constructive advise, or go back to your basement.

Re:Excellent question (1)

drussell (132373) | about 10 months ago | (#45652747)

Actually, that was a reply to THIS post, not the original question posted by timothy...

I really hope this discussion provides good answers, with practical solutions for Windows, IOS, and Linux... I think that this is the sort of thing that everyone could really use!

Are there cloud storage providers that can do this for the above example of an approx. 2 TB data set, and provide complete security?

I still think questions about basic data integrity, checksums, parity, ECC on disks etc. should be completely unnecessary and most certainly already be second nature to the slashdot crowd, but I guess I'm just living in the past.

Thanks for immediately jumping down my throat, though ;)

Re:Excellent question (1)

lxs (131946) | about 10 months ago | (#45652525)

So if someone doesn't have your level of expertise on a single isolated topic you automatically dismiss this person as unworthy of your company?
This is why people don't like you.

Re:Excellent question (0)

Anonymous Coward | about 10 months ago | (#45652657)

So if someone doesn't have your level of expertise on a single isolated topic you automatically dismiss this person as unworthy of your company?
This is why people don't like you.

In fairness the real context of the question is "what is the cheapest way for me to safely store my files without doing any work" since a basic solution involving local hard drives, md5sum, and diff _should_ be obvious to any slashdotter, or else they should GTFO. The way the submitter worded it makes it sound like they genuinely don't know, and the reality is they either know and want an easier/cheaper way, or are dumb and should GTFO. We really should be demanding more from approved submissions.

Re:Excellent question (1)

drussell (132373) | about 10 months ago | (#45652899)

Thank you. A thoughtful, concise Anonymous post... You've just restored some of my faith in the AC. ;)

Re:Excellent question (1)

drussell (132373) | about 10 months ago | (#45652791)

So if someone doesn't have your level of expertise on a single isolated topic you automatically dismiss this person as unworthy of your company?

The Anonymous Cowards? Yes.

Please continue the technical discussion. Sorry for the noise.

Re:Excellent question (3, Insightful)

lgw (121541) | about 10 months ago | (#45652265)

Bitrot is a myth in modern times. Floppies and cheap-ass tape drives from the 90s had this problem, but anything reasonably modern (GMR) will read what you wrote until mechanical failure.

The key therefore is to verify as you write. Usually, verifying a sample of a few GB will let you know if everything went OK. DO your backups with checksums of some sort. A modern tape drive and backup software will do that automatically, and let you schedule a verify automatically as part of backups (2 TB? That's 1 tape - might want to consider that), though ideally you should verify a tape on a different drive than the one you wrote it on.

For disk-based backups, local or cloud, I strongly recommend archiving to a format with checksums (RAR etc) over some sort of raw file copy. Especially for anything going over the network: RAR a volume/file set locally first, then upload, then test the archive.

If you have a superstitious fear of bitrot, you can always do some random sampling of archive integrity, and keep multiple historical copies of files just in case (e.g., don't just delete backup N-1 when you do backup N, do a rotation scheme).

Re:Excellent question (1)

QuietLagoon (813062) | about 10 months ago | (#45652555)

Bitrot is a myth in modern times.

You state this without any substantiation as if it were a fact.

Re:Excellent question (1)

linear a (584575) | about 10 months ago | (#45652889)

Bitrot is a myth in modern times.

You state this without any substantiation as if it were a fact.

And I'll counter the above. The last bitrot event I had to deal with - on current server grade (Windoze, tho) hardware was waaaay back last Friday.

Re:Excellent question (5, Interesting)

rabtech (223758) | about 10 months ago | (#45652571)

Bitrot is a myth in modern times. Floppies and cheap-ass tape drives from the 90s had this problem, but anything reasonably modern (GMR) will read what you wrote until mechanical failure.

This isn't just wrong, it's laughably wrong. ZFS has proven that a wide variety of chipset bugs, firmware bugs, actual mechanical failure, etc are still present and actively corrupting our data. It applies to HDDs and flash. Worse, this corruption in most cases appears randomly over time so your proposal to verify the written data immediately is useless.

Prior to the widespread deployment of this new generation of check-summing filesystems, I made the same faulty assumption you made: that data isn't subject to bit rot and will reproduce what was written.

ZFS or BTRFS will disabuse you of these notions very quickly. (Be sure to turn on idle scrubbing).

It also appears that the error rate is roughly constant but storage densities are increasing, so the bit errors per GB stored per month are increasing as well.

Microsoft needs to move ReFS down to consumer euro ducts ASAP. BTRFS needs to become the Linux default FS. Apple needs to get with the program already and adopt a modern filesystem.

Re:Excellent question (2, Insightful)

Anonymous Coward | about 10 months ago | (#45652637)

it doesn't seem that way... http://forums.freenas.org/threads/ecc-vs-non-ecc-ram-and-zfs.15449/

Re:Excellent question (1)

DigiShaman (671371) | about 10 months ago | (#45652803)

CDRs suffer nasty bitrot. Usually most CDs made in the past 10 years. I suppose you could have vacuum sealed them, but how many people knew to do that?!! You can get medical grade gold disks, but those you have to special order (not found in your local computer store).

One of my clients geoscience data projects archived in CDRs. It's only when they went to pull them did they discover the bitrot problem. We used Nero DiskSpeed to performa surface scan. You can see entire segments where goes green (good), transitions into yellow (correctible), to red (damaged unreadable) and the back out to yellow and green again. It's the material that oxidizes. Since then, they pulled all data they could back onto disk and tape. God only knows how long that will last too.

Re:Excellent question (0)

Anonymous Coward | about 10 months ago | (#45652823)

GMR refers to the technology of the read head. It has nothing to do with the quality of the long term storage of the media. Perhaps you could educate yourself?

Re:Excellent question (1)

fnj (64210) | about 10 months ago | (#45652855)

Bitrot is a myth in modern times. Floppies and cheap-ass tape drives from the 90s had this problem, but anything reasonably modern (GMR) will read what you wrote until mechanical failure.

Oh, really? Is that why drive manufacturers specify a non-recoverable read error rate - typically on the order of 1 bit per 100 terabits [wdc.com] ? Let's see now. A single 4TB drive contains 32 terabits of data. So if you have three of them, either in a RAID or separately, and you try to read the entire contents, you can expect an average of one bit to be rotted permanently and lost forever. Or that bad bit could happen a lot earlier. Conceivably the first bit you try to read. Or the one millionth. And that is not considered a failed drive. You can't magically guard against these by verifying the recorded data one time, either a nominal portion or even in its entirety.

RAR's checksums will only detect errors that happen to occur when you test read the RAR archive. They won't repair it, and testing OK is no guarantee that it won't have an error the next time you read it. PAR2, on the other hand, does provide for repair.

ZFS can at least detect, and optionally repair (if you use the redundancy options) these isolated bad bits, without the necessity for any special file metadata like PAR2. Of course, there's nothing to say you can't use both ZFS and PAR2.

Re:Excellent question (2)

clickclickdrone (964164) | about 10 months ago | (#45652313)

Are there cloud storage providers that can do this for the above example of an approx. 2 TB data set, and provide complete security?

Cloud and complete security together is an oxymoron.

Re:Excellent question (1)

heypete (60671) | about 10 months ago | (#45652419)

It depends on your storage needs. For things that you need to regularly access, Amazon S3 will cost you about $175/month for 2TB storage plus transfer fees, but is readily accessible at any time.

Amazon Glacier would only cost you $20/month for that amount of storage, but has various limitations on retrieval time (~4 hour minimum) and higher costs if you need to retrieve more data in a shorter amount of time. As the name suggests, it's designed for "cold storage".

Both offer extremely high degrees of reliability.

Checksums? (1)

nine-times (778537) | about 10 months ago | (#45651979)

I don't know if there's a better solution, but you could store checksums of each archived file, and then periodically check the file against its checksum. It'd be a bit resource intensive to do, but it should work. I think some advanced filesystems can do automatic checksums (e.g. ZFS, BTRFS), but those may not be an option, and I'm not entirely sure how it works in practice.

Re:Checksums? (2)

QuietLagoon (813062) | about 10 months ago | (#45652177)

I use checksums to check for bitrot.

.
Once a week, I use openssl to calculate a checksum for each file; and I write that checksum, along with the path/filename, to a file. The next week, I do the same thing, and I compare (diff) the prior checksum file with the current checksum file.

With about a terabyte of data, I've not seen any bitrot yet.

Long term, I plan to move to ZFS, as the server's disk capacity will be rising significantly.

Re:Checksums? (0)

Anonymous Coward | about 10 months ago | (#45652431)

You are assuming you started with good files. In the submitter's case, he started with some good files, some unknown number of bad files, etc. So this would just confirm that the bad file hasn't gotten worse. He wants to find the bad files and fix them as well.

Re:Checksums? (1)

QuietLagoon (813062) | about 10 months ago | (#45652523)

You are assuming you started with good files.

No assumption on my part. I did start with good files. :)

In the submitter's case, he started with some good files, some unknown number of bad files, etc.

That's not how I read the comment. From the OP:

With the quantity of data (~2 TB at present), it's not really practical for us to examine every one of these periodically so we can manually restore them from a different copy.

That sound to me as if he wants to check the files from time to time and locate ones that have gone bad.

Re:Checksums? (2, Interesting)

Anonymous Coward | about 10 months ago | (#45652247)

Periodically checking them is the important part that no one seems to want to do.

A few years back we had a massive system failure and once we recovered the underlying problems and began recovery we found that most of the server image backup tapes for 6 months+ could not be loaded. The ops guys took a severe beating for it.

You think this stuff will never happen but it always does. We had triple redundancy with our own power backups but even that wasn't on a regular test cycle. Some maintenance guy left the switch open between floors for some reno job over a year prior and while the generators were running the power didn't make it to infrastructure.... it was as if hundreds of UPSs screamed at once and were silenced when failover didn't happen.

You really can't beat Murphy's Law, but with regular testing you can soften the effects.

Re:Checksums? (5, Informative)

Waffle Iron (339739) | about 10 months ago | (#45652393)

I never archive any significant amount of data without first running this script at the top:

find -type f -not -name md5sum.txt -print0|xargs -0 md5sum >> md5sum.txt

It's always good to run md5sum --check right after copying or burning the data. In the past, at least a couple of percent of all the DVDs that I've burned had some kind of immediate data error

(A while back, I rescanned a couple of hundred old DVDs that I burned ranging up to 10 years old, and I didn't find a single additional data error. I think that a lot of cases where people report that DVDs deteriorate over time, they never had good data on them in the first place and only discover it later.)

Re:Checksums? (0)

Anonymous Coward | about 10 months ago | (#45652667)

Or maybe you just didn't buy the absolute cheapest DVDs available. Also the quality of the burner may play a role.

That's what RAID is /for/ (0)

Anonymous Coward | about 10 months ago | (#45651989)

It seems to me that you've already identified an easy solution: RAID. A simple mirror of 2x 2-4TB drives is pretty cheap these days, so it would seem to be an ideal solution for one of your copies. Keep one "live" copy on your normal desktop, one backup on an off-site RAID, and if you feel like it, another copy on a cloud service or other media--Tape backups aren't sexy, but they're pretty cheap and very effective at long-term cold-storage. BluRay disks aren't terribly expensive anymore, though the jury is still out on their long term (decade+) durability.

Re:That's what RAID is /for/ (1)

Anonymous Coward | about 10 months ago | (#45652201)

I don't think you understand what RAID is or what it does.

That's what some RAID levels _could_ be for (1)

Sloppy (14984) | about 10 months ago | (#45652547)

A two-disk RAID1, or a RAID5, theoretically ought to be able to detect when there's corruption, but shouldn't be able to correct it. If you've got two different data values, you don't know which one is right.

But it occurs to me: RAID6 (or three-or-more disk RAID1) really ought to be able to correct. Imagine a three-disk RAID1: if two disks say a byte is 03 and one disk says 02, then 03 is probably right. RAID6, similarly, has enough information to be able to do the kinds of repairs that you could do with par2.

It'd be cool to find out this is already in the kernel's md device. Probably not so yet, though. ?

Re: That's what RAID is /for/ (0)

Anonymous Coward | about 10 months ago | (#45652611)

There's cheaper storage for archiving and different requirements.

Perforce (0)

Anonymous Coward | about 10 months ago | (#45651999)

You can install Perforce which is a CM system like SVN etc. but it's very good w/large binary data. You can have it run a verify command nightly (or as often as you like) and it will compare the MD5SUM of ever version of every file (which was computed when that version of the file was commited to the CM system) with the current MD5SUM and let you know which ones have changed (bitrot).

If you don't CM your data, you can just do an MD5SUM recursively and store it off, then periodically repeat the procedure and diff the 2.

If you like GUIs Beyond Compare is an excellent program and it does snapshots (CRCs of directory trees) and then lets you compare the snapshot with an updated / recomputed version.

uhuh (-1)

Anonymous Coward | about 10 months ago | (#45652009)

Sounds like an NSA sys admin doesn't know how to do his job. Well, just run "dd if=/dev/null of={file to check}" on all your files to verify they're good. It will also correct any errors!

Re:uhuh (2, Informative)

Anonymous Coward | about 10 months ago | (#45652215)

Warning for all UNIX newbies: that command will reset the file to 0 bytes. Just that you know.

(I've seen some cases when a rookie is setting up a Linux system and people jokingly throw him these "rm -rf /" commands and the poor guy actually ends up wrecking his system.)

Re:uhuh (0)

Anonymous Coward | about 10 months ago | (#45652299)

Boy you must be a real hit at parties.

Re:uhuh (1)

CanHasDIY (1672858) | about 10 months ago | (#45652305)

Warning for all UNIX newbies: that command will reset the file to 0 bytes. Just that you know.

(I've seen some cases when a rookie is setting up a Linux system and people jokingly throw him these "rm -rf /" commands and the poor guy actually ends up wrecking his system.)

I think the general consensus is that if you're stupid enough to run a command you got from SomeRandomInternetAsshole420 without verifying what it will do first, you deserve to have your system wiped.

Re:uhuh (2)

Sarten-X (1102295) | about 10 months ago | (#45652401)

And yet, one of FLOSS's selling points is our great community support...

Re:uhuh (1)

Behrooz Amoozad (2831361) | about 10 months ago | (#45652643)

WARNING: DO NOT RUN ANY COMMAND IN THE PARENT, THIS COMMENT OR ANY OF THE SIBLING COMMENTS.
You really suck at being an asshole too, the right command for destroying files and being innocently obfuscated is:
dd if=/dev/zero|pv|dd bs=1024 count=$(ls -s 'filename'|awk '{print $1}' of='filename'|openssl sha1

SpinRite (-1)

Anonymous Coward | about 10 months ago | (#45652029)

try recovering the photos with SpinRite -- https://www.grc.com/sr/spinrite.htm

How are you getting bitrot? (1)

drussell (132373) | about 10 months ago | (#45652033)

If your physical media is dying, you'll get hardware errors so restore from a(nother) backup and replace the media.

If your files are being corrupted, what kind of crappy filesystem are you using to store these precious memories?!!

Re:How are you getting bitrot? (1)

Anonymous Coward | about 10 months ago | (#45652255)

Cosmic rays, magnetic data corruption. If you do not re-write the bits they decay.

Re:How are you getting bitrot? (0)

Anonymous Coward | about 10 months ago | (#45652351)

Hard disks use very strong error correcting codes. The odds of randomly flipped bits resulting in undetected corruption are virtually nil. If bad data is stored on a disk, it's far more likely that bad data was written to the disk in the first place (OS bugs, or random corruption since most people use cheap non-ECC memory).

Re:How are you getting bitrot? (0)

Anonymous Coward | about 10 months ago | (#45652703)

I don't think using expensive non-ECC memory will help much.

Re:How are you getting bitrot? (0)

Anonymous Coward | about 10 months ago | (#45652407)

I'd like to know how this is happening also. I have many TBs of data on a variety of mediums and I cannot remember ever seeing a corrupt file over the past twenty years. I also have a Baby Croc account with many TBs and never had a problem. I still use recovery, though (par and rar with recovery record). If you cannot periodically examine family photos and videos, why do you have them? I have shitloads of family photos and I look at them all the time. That's why I took the photos.

Pars (0, Insightful)

Anonymous Coward | about 10 months ago | (#45652051)

Should have parred your data to begin with.

Look to the past (0)

Anonymous Coward | about 10 months ago | (#45652059)

Tape. LTO is error-correcting and extremely stable and reliable.

It it expensive? Yes, for small datasets - for large datasets it can be much cheaper.
Is it a pain in the ass? Yes, but what's your data really worth?

Re:Look to the past (2)

Venotar (233363) | about 10 months ago | (#45652553)

The tapes may be stable (I'm suspicious of that claim: their temperature tolerances aren't as high as modern hard drives, they actually care about dust, and I would expect them to be more susceptible to magnetic interference); but the tape drives are not. Over time drive heads become misaligned. They continue to write fine and can read what they write; but sufficient misalignment prevents other drives of the same type from reading the tape. That tape then becomes only as useful as the drive that wrote it. Lose the drive, you lose the use of the data on the tape. Unless you test reading the tape in a different drive than it was written from (while the writing drive is still available for pulling the data out), this condition's effectively undetectable until you actually need the data.

There's a reason so many shops have moved to disk based backups. Tape simply isn't reliable. Tape is cheap; but definitely NOT reliable.

Re:Look to the past (0)

Anonymous Coward | about 10 months ago | (#45652701)

For values of "large" much, much greater than the size of a home movie/music/photo collection. OP could build 2 more servers full of 2TB consumer hard drives for the price of an LTO-5 drive.

Re:Look to the past (1)

mlts (1038732) | about 10 months ago | (#45652779)

I just wish LTO drives were cheaper. Otherwise, they would be ideal for backups because they support encryption on the drives themselves. All LTO-4 tapes and newer support this, so any LTO-4 drive given the right key can decrypt another drive's tape.

Of course, WORM media is always nice, especially with malware being a constant threat.

Rsync & ZFS (0)

Anonymous Coward | about 10 months ago | (#45652061)

rsync --checksum for the remote copies.

ZFS is a good filesystem for bitrot protection, you don't want to propagate the errors.

ZFS (4, Interesting)

Electricity Likes Me (1098643) | about 10 months ago | (#45652075)

ZFS without RAID will still detect corrupt files, and more importantly tell you exactly which files are corrupt. So a distributed group of ZFS drives could be used to rebuild a complete backup by only copying uncorrupt files from each.

You still need redundancy, but you can get away without the RAID in each case.

geez what brand of drives are you using (0)

Anonymous Coward | about 10 months ago | (#45652083)

Windows 8.1 and 2012 support this if you setup a storage space in a mirror (could be 2 standard external usb disks) and format with the resilient file system (ReFS) rather than NTFS, it will do background scans that correct for bitrot, it's new and not well proven but that's microsoft's claim anyway

Rewritten for /. (0)

Anonymous Coward | about 10 months ago | (#45652111)

[seen in the margins of a book on data backup owned by someone claiming to be Fermat reincarnated]

"I have found an elegant solution to the problem of self-healing distributed backups which are neither co-located nor in constantly aware of each other's state. The details are too long to fit in this space."

Re: Rewritten for /. (1)

techprophet (1281752) | about 10 months ago | (#45652205)

And thus the saga of that damned Frenchman continues

Who cares? (-1)

Anonymous Coward | about 10 months ago | (#45652135)

More articles about comic book movies and video games please.

It has been done (0)

Anonymous Coward | about 10 months ago | (#45652179)

The functionality isn't new. Large robotic tape libraries would pull out tapes periodically and verify health of the media, copying to new if unwell.

About 5 years ago the RAID chip vendor folks were touting RAID 6, as required for sets of 1+ TB drives as the potential for experiencing a read fault on recovery of a failed RAID 5 becomes much more likely as drive volumes increase.

So first line of defense RAID storage with health monitoring, then are backups ( offsite as well )

Par2 and Reed-Solomon (1)

mpol (719243) | about 10 months ago | (#45652185)

Bitrot does happen.
When a disk has a bad block and detects that, it will try to read the data from it and put it on a block from the reserve-pool. However, the data might be bad and corrupt, so you lose data.
Disks do have a Reed-Solomon (aka par-files) index, so it can repair some damage, but it doesn't always succeed.

Anyway, what I do for important things, is have par2 blocks that go along with the data. All my photo-archives have par2 files attached to them.

I reckon you could even automate it. To have a script that traverses all directories and tries to repair the data if it's broken. If it fails, you get notified.

zfs or btrfs (1, Interesting)

Anonymous Coward | about 10 months ago | (#45652191)

First off, make sure you have a separate backup storage volume that doesn't get touched by normal applications and which keeps history. Backup doesn't protect you very much if accidental deletes or application bugs corrupt all your copies within one backup cycle. Use an appropriate backup tool to manage this, where appropriateness depends on your skill and willingness to tinker. You could use something as simple as an rsync --link-dest job, or rsync --inplace in combination with filesystem snapshots, or some backup suite that will store history in its own format.

For bit-rot protection of the stored backup data, make a backup volume using zfs or btrfs with at least two disks in a mirroring configuration (where the filesystem manages the duplicate data, not a separate raid layer). Set it to periodically scrub itself, perhaps weekly. It will validate checksums on individual file extents. If one copy of a file extent cannot be read successfully, it will rewrite it using the other valid mirror. This rewrite will allow the disk's block remapping to relocate a bad block and keep going. The ability to validate checksums is the value add beyond normal raid, where the typical raid system only notices a problem when the disk starts reporting errors.

Monitor overall disk health and preemptively replace drives that start to show many errors, just as with regular raid. Some people consider the first block remapping event to be a failure sign, but you may replace a lot of disks this way. Others will wait to see if it starts having many such events within days or weeks before considering the disk bad.

Splendid (0)

aaaaaaargh! (1150173) | about 10 months ago | (#45652233)

We have hundreds of thousands of family pictures and videos we're trying to save

Yes, you've got to save them! Your children will be so thankful for countless extended family diashow evenings!

"Look, here is little Tim vomiting when he was 12 years old! How sweet! -- Another vomiting picture. -- Another one. -- I'll skip the next 11 images, still 12,371 to go after all..."

Re:Splendid (0)

Anonymous Coward | about 10 months ago | (#45652347)

"If you can't say something nice, don't say nothing at all."
Who are you to belittle the things which others enjoy?

Re:Splendid (0)

Anonymous Coward | about 10 months ago | (#45652391)

If you are ever able to procreate, you will understand the desire to preserve your photos.

Re:Splendid (1)

F. Lynx Pardinus (2804961) | about 10 months ago | (#45652641)

I understood him to be commenting on the number, not the existence, of the photos. I'm the designated archivist for the family's (7 members in 2 households) photos. At last check , I have about 20k photos in the archive. It's hard to imagine having "hundreds of thousands" without having enormous amounts of redundant or irrelevant photos, which is what the parent post is poking fun of.

Re:Splendid (0)

Anonymous Coward | about 10 months ago | (#45652721)

If you are ever able to procreate, you will understand the desire to preserve your photos.

I'd think even a person with ancestors would grasp this, but the GP has proved me wrong. I suspect we're dealing not with naivete or lack of experience, but a more fundamental personality/mental health issue.

Re:Splendid (1)

Sloppy (14984) | about 10 months ago | (#45652633)

You really gotta be careful with that attitude. The photos seem worthless at the time you take them, and most of them remain worthless forever. Most of them. Then you see that old picture of when your now-grown-up dog used to be a cute little puppy, and awww!!!

parity archives (0)

Anonymous Coward | about 10 months ago | (#45652235)

Add 20% par2 files.

freenas with zfs partition (0)

Anonymous Coward | about 10 months ago | (#45652237)

zfs will detect and correct the bitrot. freenas is probably the easier solution for providing that file system to a household.

Rar (1)

rava (618424) | about 10 months ago | (#45652281)

I'm glad you're bringing this up. I haven't seen any backup software that addresses bitrot. And bitrot does happen, I lost a few pics to it. What I do: I have a monthly script that makes a RAR archive from my pictures directory. RAR checks file integrity but also has "recovery" options that allow you to recover files from a damaged archive (to a point)

For single drives (0)

Anonymous Coward | about 10 months ago | (#45652287)

For single disk setups use ZFS with copies=2. You will lose half of your storage, but you will gain error correction. With the default copies=1, you get error detection, but not correction.

A paranoid setup (4, Interesting)

brokenin2 (103006) | about 10 months ago | (#45652297)

If you really want hassle free and safe, it would be expensive, but this is what I would do:

ZFS for the main storage - Either using double parity via ZFS or on a raid 6 via hardware raid.

Second location - Same setup, but maybe with a little more space

Use rsync between them using the --backup switch so that any changes get put into a different folder.

What you get:

Pretty disaster tolerant
Easy to maintain/manage
A clear list of any files that may have been changed for *any* reason (Cryptolocker anyone?)
Upgradable - just change drives
Expense - You can build it for about $1800 per machine or $3600 total if you go full-on hardware raid. That would give you about 4TB storage after parity (4 2TB drives - $800, Raid Card - $500, basic server with room in the case - $500)

What you don't get: Lost baby pictures/videos. I've been there, and I'd pay a lot more than this to get them back at this point, and my wife would pay a lot more than I would..

Your current setup is going to be time consuming, and you're going to lose things here and there anyway.. If you just try to do the same thing but make it a little better, you're still going to have the same situation, just not as bad. In this setup you have to have like 5 catastrophic failures to lose anything, sometimes even more..

Re:A paranoid setup (1)

Minwee (522556) | about 10 months ago | (#45652461)

Expense - You can build it for about $1800 per machine or $3600 total if you go full-on hardware raid. That would give you about 4TB storage after parity (4 2TB drives - $800, Raid Card - $500, basic server with room in the case - $500)

Either use a RAID controller or use ZFS. It's not a good idea to use both at the same time.

Re:A paranoid setup (1)

brokenin2 (103006) | about 10 months ago | (#45652601)

I've used them together. Seems to work just fine.. Just don't let ZFS know that there's more than 1 drive. You can't have them both trying to manage the redundant storage.

ZFS has some great features besides it's redundant storage. You can get them from other filesystems too though I suppose, but I like snapshots built into the filesystem. It *is* overkill to have the filesystem doing checksums and the raid card detecting errors as well, but that's why this is the paranoia setup... Not really looking for the performance king..

ZFS certainly isn't necessary though, if you've got hardware raid.

WinRAR... (1)

mlts (1038732) | about 10 months ago | (#45652301)

WinRAR isn't perfect, but it works on a number of platforms, be is OS X, Windows, Linux, or BSD. This provides not just CRC checking, but one can add recovery records for being able to repair damage. If storing data on a number of volumes (like optical media), one can make recovery volumes as well, so only four CDs out of a five CD set are needed to get everything back.

It isn't as easy as ZFS, but it does work fairly well for long term archiving, and one can tell if the archive has been damaged years to decades down the road.

BTRFS or ZFS (1)

mcelrath (8027) | about 10 months ago | (#45652345)

BTRFS and ZFS both do checksumming and can detect bit-rot. If you create a RAID array with them (using their native RAID capabilities) they can automatically correct it too. Using rsync and unison I once found a file with a nice track of modified bytes in it -- spinning rust makes a great cosmic ray or nuclear recoil detector. Or maybe the cosmic ray hit the RAM and it got written to disk. So, use ECC RAM.

But "bit-rot" occurs far less frequently than this: I find is that on a semi-regular basis my entire filesystem gets trashed (about once every year or three). This happened to me just last week...my RAID1 BTRFS partitions (both of them) got trashed because one of my memory modules went bad. In the past I've had power supplies go bad causing this, or brown outs, and in other cases I never identified the cause. I've seen this happen across ext3, jfs, xfs, and btrfs so it's (probably) not the file system's fault. In such cases, fsck will often make the problem worse. (Use LVM and its "snapshot" feature to perform fsck on a snapshot without destroying the original). You'd think these advanced filesystems would have a way to rewind to a working copy (for instance in BTRFS -- mount a previous "generation") but this seems to not be the case.

Anyway, btrfs guys, your recovery tools could be a lot better. The COW enables some pretty fancy recovery techniques that you guys don't seem to be doing yet. If you've got a great btrfs or zfs recovery technique, please reply and tell us.

Re:BTRFS or ZFS (0)

Anonymous Coward | about 10 months ago | (#45652589)

btrfs/zfs send to another machine (prevention) is the best approach, as the OS will always be a single source of failure even in the best systems (as well as fire/flood/lighting...).

for btrfs recovery there is https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg27613.html and https://btrfs.wiki.kernel.org/index.php/Restore

RAID + redundancy (1)

sl4shd0rk (755837) | about 10 months ago | (#45652437)

There's really no way around it. Storage media is not permanent. You can store your important stuff on RAID but keep the array backed-up often. RAID is there to keep a disk*N failure from borking your production storage and that's it. If you can afford cloud storage, encrypt your array contents (encfs is good) and mirror the contents with rsnapshot [rsnapshot.org] or rsync [samba.org] to amazon, dropbox, a friends raid array, whatever. SATA drives are cheap enough to keep a couple sitting around to just plug in and mirror to every weekend but you'll probably find a friend's cable modem and rsync+ssh a very handy alternative (hint: check out --bwlimit option) when run from cron.

Freenas (ZFS based) or BTRFS (1)

hibble (1824054) | about 10 months ago | (#45652537)

"We'd love it if the file-system could detect this and try correcting first, and if it couldn't correct the problem, it could trigger the restoration. But that only seems to be an option for RAID type systems, where the drives are colocated."

If you have ~2TB of irreplaceable memories set up a NAS with a RAID array. whilst bit-rot can be detected it can only correct itself if the file system knows what the bits should have been. To this end BTRFS and my recommendation ZFS can be set to say scan all data 1 a week/month etc and using the redundant data in the RAID array correct the 'Bit-Rot'.

I have a intel atom board in a old case with 4 drives(2x 500GB mirror and 2x 1TB mirror). I have FREENAS on this it is powered on every night by wake on lan. Backs up any new data and gets shut down. once a week it backs up new data then runs the command 'zfs scrub' this checks for bit-rot or inconsistencies in the file-system and corrects them if any are found.(can email you a warning if you want as well). This way if any files get damaged on a home pc/ laptop etc.. any user can turn on the NAS and recover there files from the shared folder.

1 point of warning ZFS is RAM hungry so 4GB is the minimum. something to keep in mind when ebaying for a old pc to use. others will also point out that file transfers are ~20-30MB/s with a low powered atom so use something with more grunt if its to be a 24/7 NAS.

Long term Data Preservation (0)

Anonymous Coward | about 10 months ago | (#45652543)

This is a big deal in digital movie preservation. There will be a cloud solution based on Swift open source available in the next couple of months.

use a torrent (0)

Anonymous Coward | about 10 months ago | (#45652595)

make a torrent of your stuff, spread the copies around, use a private tracker, force a recheck on any file you think is corrupt and let the swam do it's thing.

Just get a carbonite account (1)

gravis777 (123605) | about 10 months ago | (#45652597)

I have been going through this issue myself. In a single weekend of photo and video taking, I can easily fill up a 16 gig memory card, sometimes a 32 gig. About 10 years ago I lost about two years worth of pictures due to bitrot (ie my primary failed, and the backup DVD-Rs were unreadable after only a year - I was able to recover only a handfull of photos using disc-recovery software). Since then, I kept at least three backups, and reburning discs every couple of years. But if I can fill up two BD-Rs in a weekend, and given the high price of media, that wasn't an option. Extra harddrives?

I finally realized the best way was just to get a Carbonite account. They are about $70 a year for unlimited encrypted storage space (if you are really anal, I guess you could always put things into TrueCrypt encrypted file containers and upload them). The worst part is how long it takes to do a backup on a residental broadband line (it would also suck if your ISP has data caps). It has taken me about 2 weeks to do half a terrabyte.

The deal is, the peace of mind that comes from this is huge, and it is cheaper than buying another harddrive.

Yes, I know that is not the question you asked, but I feel like it is a much more practical alternative. I mean, as I continue backing stuff up, I am sure I will pass a terrabyte. How much are you going to pay for discs, for harddrives? Then trying to keep them safe and secure, and having to worry about bitrot?

Seriously, I've lost family pictures and videos before even though I had backups, and it sucked. Do yourself a favor and get a cloud backup. Yeah, it may take a while to do your backups and restorations, but it is worth it.

Re:Just get a carbonite account (1)

Anonymous Coward | about 10 months ago | (#45652769)

Regarding bitrot as above commenters -- how can you be sure that your cloud provider is not suffering from bitrot on your stored files? PAR2 files help, but a cloud provider that would checksum the files you uploaded with a checksum file you provide (and not just say "Yep, everything is fine, nothing to see here.") would ease my mind somewhat.
Assuming, of course, the cloud provider is using a modern ZFS or BTRFS to store your cloud data.

Re:Just get a carbonite account (0)

Anonymous Coward | about 10 months ago | (#45652821)

This is also a good time to pitch in a vote for CrashPlan. It does both cloud backup *and* backup to other computers running CrashPlan, so you can pretty easily manage a multi-site backup of your system. The program is cross-platform, so the other systems can be Win/Mac/Linux.

My only complaint is the lack of documentation of the on-disk storage format, so you have to rely on the CrashPlan app to read your data. I use a second (but not as robust) backup method for this reason, but I have had no complaints about CrashPlan.

There are pills which can help you (-1)

Anonymous Coward | about 10 months ago | (#45652617)

Obsessing about "hundreds of thousands of family pictures" is at once
comical and tragic.

The truth is, once the people in the pictures are dead, few if any people
on earth will care even slightly about the pictures or who was in those pictures.

Quit being so goddamned anal and enjoy life while you are still ALIVE.

How about Stone? (1)

Fussen (753791) | about 10 months ago | (#45652639)

M-DISC:
DVD format presently, BLU-RAY format in the future. Someday an electronic eye will just be able to look at the disc surface and see it all in one snapshot.

They aim for 1000 years. I expect 100. It may be reasonable. Just keep drives around.

http://www.mdisc.com/proving-ground/ [mdisc.com]

M-Disc (0)

Anonymous Coward | about 10 months ago | (#45652683)

Obviously. Since your data isn't constantly changing, and is videos and photos, M-Disc is the ideal solution.

Bacula (1)

dshk (838175) | about 10 months ago | (#45652835)

It might be an overkill, but the open source backup software Bacula has a verify task, which you can schedule to run regularly. It can compare the contents of files to thir saved state in backup volumes, or it can compare the MD5 or SHA1 hashes which were saved in the previous run. I assume other backup software has similar features.
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?