Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

One Developer's Experience With Real Life Bitrot Under HFS+

timothy posted about 3 months ago | from the so-really-it's-both-plus-and-minus dept.

Bug 396

New submitter jackjeff (955699) writes with an excerpt from developer Aymeric Barthe about data loss suffered under Apple's venerable HFS+ filesystem. HFS+ lost a total of 28 files over the course of 6 years. Most of the corrupted files are completely unreadable. The JPEGs typically decode partially, up to the point of failure. The raw .CR2 files usually turn out to be totally unreadable: either completely black or having a large color overlay on significant portions of the photo. Most of these shots are not so important, but a handful of them are. One of the CR2 files in particular, is a very good picture of my son when he was a baby. I printed and framed that photo, so I am glad that I did not lose the original. (Barthe acknowledges that data loss and corruption certainly aren't limited to HFS+; "bitrot is actually a problem shared by most popular filesystems. Including NTFS and ext4." I wish I'd lost only 28 files over the years.)

cancel ×

396 comments

Sorry! There are no comments related to the filter you selected.

I've also had this happen with HFS+ (4, Informative)

carlhaagen (1021273) | about 3 months ago | (#47235971)

An old partition of some 20000 files, most of them 10 years or older, in where I found 7 or 8 files - coincidentally jpg images as well - that were corrupted. It struck me as nothing other than filesystem corruption as the drive was and still is working just fine.

Re:I've also had this happen with HFS+ (4, Insightful)

istartedi (132515) | about 3 months ago | (#47236091)

coincidentally jpg images as well

Well, JPGs are usually lossy and thus compressed. Flipping one bit in a compressed image file is likely to have severe consequences. OTOH, you could coXrupt a fewYentire byteZ in an uncompressed text file and it would still be readable. I suspect your drives also had a few "typos" that you didn't notice because of that.

How about the "next gen file system" ? (0)

Anonymous Coward | about 3 months ago | (#47238155)

I googled a little bit and came up with this link -

http://arstechnica.com/information-technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/

It talked about bitrot and the next-gen filesystem and according to it, the so-called "next gen" filesystem would detect bitrot

I do not know if it's the case, so I am posting the link here for others to discuss this matter further

Re: How about the "next gen file system" ? (0)

Anonymous Coward | about 3 months ago | (#47240635)

This has definite repercussions for anyone doing any type of digital archive repository. Librarians have known about this for a while.

Re: How about the "next gen file system" ? (1)

Anonymous Coward | about 3 months ago | (#47242347)

ZFS definitely can detect bit rot! It's designed to do so.

Re:I've also had this happen with HFS+ (1)

thegarbz (1787294) | about 3 months ago | (#47239511)

coincidentally jpg images as well

Well, JPGs are usually lossy and thus compressed. Flipping one bit in a compressed image file is likely to have severe consequences.

That depends entirely on how the compression technique works. With JPEGs flipping a bit here or there shouldn't actually have too much effect on the overall image but may affect an 8x8 block of pixels.

The problem with unreadable JPEGs usually comes from how decoders handle the errors, or rather can't handle the errors. Windows Picture Viewer is the worst often simply flat out failing to render anything at all. Most other viewers will render down to the flip bit then render the rest either garbage or grey. There are however tools to fix the resulting data.

Re:I've also had this happen with HFS+ (1)

Desty (2749557) | about 3 months ago | (#47240723)

JPEG is quite robust to corruption, and even PNG's lossless compression seems to be tolerant of a few stray bytes. However, encrypted files would probably be badly damaged by this sort of corruption.

This is the kind of situation where some form of transparent, redundant error-recovery system is extremely important. I'm sure that in the medium term future (after everyone is using SSDs and the cost/capacity ratio falls much further) some kind of RAID setup will be the norm and these kinds of problems will become vanishingly unlikely.

Re:I've also had this happen with HFS+ (1)

doccus (2020662) | about 3 months ago | (#47242563)

The problem with unreadable JPEGs usually comes from how decoders handle the errors, or rather can't handle the errors. Windows Picture Viewer is the worst often simply flat out failing to render anything at all. Most other viewers will render down to the flip bit then render the rest either garbage or grey. There are however tools to fix the resulting data.

I suspect Windows Picture Viewer isn't so terribly much an issue on HFS+

Re:I've also had this happen with HFS+ (1)

thegarbz (1787294) | about 3 months ago | (#47244123)

I suspect that NTFS is no better than HFS+ at dealing with Bitrot.

Re:I've also had this happen with HFS+ (0)

Anonymous Coward | about 3 months ago | (#47236953)

If you have the original files and the corrupted files, you might be able to find out what has gone wrong with those files and write a file repair program. It would seem simple to be able to flip bits one at a time until the correct image was decoded.

Re: I've also had this happen with HFS+ (0)

Anonymous Coward | about 3 months ago | (#47238517)

If you have the original files then just use those. No point flipping the corrupt files back to original state.

Re:I've also had this happen with HFS+ (3, Insightful)

Jane Q. Public (1010737) | about 3 months ago | (#47238303)

I agree with istaredi.

Ultimately, it isn't a "failure" of HFS+ when your files get corrupted. It was (definitely) a hardware failure. It's just that HFS+ didn't catch the error when it happened.

Granted, HFS+ is due for an update. That's something I've said myself many times. But blaming it when something goes wrong is like blaming your Honda Civic for smashing your head in when you roll it. It wasn't designed with a roll cage. You knew that but you bought it anyway, and decided to hotdog.

Checksums also have performance and storage costs. So there are several different ways to look at it. One thing I strongly suggest is keeping records of your drive's S.M.A.R.T. status, and comparing them from time to time. And encourage Apple to update their FS, rather than blaming it for something it didn't cause, or for not doing something it wasn't designed to do.

Re:I've also had this happen with HFS+ (1)

Anonymous Coward | about 3 months ago | (#47240621)

...blaming it when something goes wrong is like blaming your Honda Civic for smashing your head in when you roll it. It wasn't designed with a roll cage. You knew that but you bought it anyway, and decided to hotdog. ...I strongly suggest is keeping records of your drive's S.M.A.R.T. status

In this case (and every case, really) "hotdogging" means "reading files." Reading files should not be an extreme sport for a filesystem. The drive's S.M.A.R.T. status was "everything is just dandy." This is just thermal turbulence and cosmic rays hitting the drive platter and HFS+ failing to notice or recover. Everything about this is utterly mundane, but it's stupid and shouldn't happen in 2014. We have half a dozen modern filesystems that know better, and whatever cost there is to error recovery they've made up for it in many ways.

You're right that you can't blame HFS+ for doing exactly what it's designed to do. I don't read anyone's writings so far as blame for HFS+ itself causing harm. But, bitrot is real, and a filesystem that was initially designed for computing and storage constraints 25 years ago is a bad tool for today. Apple has had pressure to change the filesystem for a long time. Articles like this do exactly what you suggest—renew pressure on Apple to fix it.

Re:I've also had this happen with HFS+ (0)

Anonymous Coward | about 3 months ago | (#47240627)

Only if you have no choice but you use a Honda Civic to drive on the roads in your area.

Long have I lusted for the ability to use ZFS for my install drive on Mac, or NTFS, or ANYTHING other than HFS+, because journaling is not a substitute for file recovery and repair and HFS has actually been a lot worse of me in my experience.

I also suspect OSX itself of corrupting a number fo drives formatted with exFAT tot he point of total partition loss. The same goes for drives formatted NTFS and running Paragon NTFS.

Re: I've also had this happen with HFS+ (0)

Anonymous Coward | about 3 months ago | (#47242391)

You can have a ads drive on a mac. Software exists to allow this, as long as you aren't booting from it!
If you get bit rot like this, you'd actually be okay if you had a backup. Most backup systems only compare actual data if the size and date mod are different. So your backup would have a functioning copy.

The basics still apply.

Re:I've also had this happen with HFS+ (0)

Anonymous Coward | about 3 months ago | (#47240945)

can you explain how a bit gets flipped on a hard drive just like that? op said that the drive works just fine still, so let's assume he test it properly to reach the conclusion. how does a bit or two flip on the platter?

Re:I've also had this happen with HFS+ (0)

Anonymous Coward | about 3 months ago | (#47244311)

It's randomly caused over time by ambient radiation.

Re: I've also had this happen with HFS+ (0)

Anonymous Coward | about 3 months ago | (#47242875)

BitRot has been a problem for millennia - just look at any book or scroll that has been copied from an older version.

Legacy file systems should be illegal (1)

Anonymous Coward | about 3 months ago | (#47235977)

We know how to build good file systems. We have done it for years with ZFS and now Btrfs. Sticking to legacy file systems which are prone to corruption is simply not acceptable. It is about time that legislative authorities makes it illegal for Apple and other negligent vendors to ship file systems that are essentially faulty by design. A noticeable fine per corrupted file would be appropriate, with possibility of prison time upon recurring incidents.

Re:Legacy file systems should be illegal (2, Informative)

Anonymous Coward | about 3 months ago | (#47236003)

The problem is, neither ZFS or Btrfs would have stopped an arbitrary bit inside an arbitrary file from becoming corrupt if the disk failed to write it or read it correctly. Only multiple disks and redundancy would have solved that.

Re:Legacy file systems should be illegal (5, Insightful)

kthreadd (1558445) | about 3 months ago | (#47236031)

At least you would know that the file was corrupted, so that you could restore it from a good backup.

Re: Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47236199)

In sure those backups won't suffer bit rot either.

But seriously if you do make backups, do an incremental and then a full every so often. A full backup every month, incremental every day and retain backups for a full year. So every year you have 12 full sets and then simply overwrite/delete the oldest set as you go.

Re: Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47236279)

In sure those backups won't suffer bit rot either.

But seriously if you do make backups, do an incremental and then a full every so often. A full backup every month, incremental every day and retain backups for a full year. So every year you have 12 full sets and then simply overwrite/delete the oldest set as you go.

Arrgh. It ain't that simple. I hope to hell you're not backing up a filesystem used to hold a large PostgreSQL database like that. Your incrementals will be effectively as large as the full backups. Imagine how long it'd take you to restore near the end of a month. When your full backups are 100 GB and your "incremental" backups are 99 GB, might as well take a full backup every night.

FWIW, 100 GB backups are SMALL.

Re: Legacy file systems should be illegal (5, Interesting)

aix tom (902140) | about 3 months ago | (#47236375)

A database is something special

I basically make a "full backup" of my Oracle DBs once a week, and a "incremental backup" in the form of DB change logs every five minutes. (that is, the change logs are pushed "off site" every five minutes, of course they are being written locally continuously with every change.

The thing with backups, though, is not only to make them often but to also *check* them often. With my DBs there is a handy tool where I can check the backup files for "flipped bits" because there are also checksums in the DB files.

For my "private backups to DVD/BR" I only fill them up to ~70%, and fill the rest of the disk with checksum data with dvdisaster. [dvdisaster.net] , for other "online backups" I create PAR2 files that I also store. With those parity files I can check "are all bits still OK?" now and then, and repair the damage when/if bits start to rot in the backup. In the 10 years I do this, with ~150 DVDs and ~20BRs so far I had 2 DVDs that became "glitchy", but because of the checksum data I was able to repair the ISO and re-burn them.

Basically, IF you go through the trouble of setting up an automated backup system either with software or with your own scripts, It doesn't add much work to also add verification/checksum data to the backup. And that goes a long way into preventing data loss due to bit rot.

Re: Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47238571)

And that (the personal stuff, not the corporate) makes you perhaps one person in 1000 who does take those precautions.

It's not too much to ask that something saved to permanent storage, be "permanent". The OS and filesystem needs to take the lead in responsibility and do so by default and automatically. Otherwise it won't happen. Not for the average user.

Heck, we're lucky if the average home user has a backup. Now that's something we can reasonably ask regular people to do, but even then, know that many or most won't. And knowing the failure rate on available backups, we can plan for that too.

Re: Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47242883)

It may be 'not too much to ask' but OTOH you get what you paid for.

Re: Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47236589)

Dude, most backups on systems based on a DB have always been that way. I assume you are new to this and just found that out or I don't think you would have even mentioned it. The type of backup you are doing on that DB can vary the size based on the recovery model you are using for those specific DB's (do you point in time or not etc). Additionally, intelligent backup software can get those effective incremental sizes down significantly with compression and block level dedupe. Your effective incremental file size may be large but the actual space taken up by the backup on its file system is much smaller, sometimes 90%+ smaller.

Re: Legacy file systems should be illegal (1)

wagnerrp (1305589) | about 3 months ago | (#47236513)

Actually, no, they won't. The chances of bitrot occurring in the same location on both your primary store and your backup, such that neither had viable data to recover, is astronomically low.

Re: Legacy file systems should be illegal (3, Informative)

O('_')O_Bush (1162487) | about 3 months ago | (#47237023)

Vs the chances you back up already corrupted files and don't notice until you've aged off the good versions.

Re: Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47242403)

Because most backup software doesn't just check size and date mod when deciding to do a checksum? The backup would be fine.

Re:Legacy file systems should be illegal (-1, Troll)

stenvar (2789879) | about 3 months ago | (#47236945)

Maybe flaky Sun hardware needed the file system to perform such checks; high quality disk drives test for data integrity in the controller. For ZFS to check is redundant and probably does more harm than good.

Re: Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47237173)

This. A bitrotted sector will come up as "uncorrectable" on any IDE or SATA drive, be unreadable rather than returning garbage, and get remapped on next write.

Re: Legacy file systems should be illegal (1)

Guspaz (556486) | about 3 months ago | (#47237201)

So? If ZFS or Btrfs has redundancy, an unreadable sector gets corrected just the same as a garbage sector.

Re: Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47238663)

More nonsense. ZFS can't magically recover data. To recover data, it needs extra storage, just like RAID.

Re: Legacy file systems should be illegal (1)

Guspaz (556486) | about 3 months ago | (#47239237)

Where did I claim otherwise? You must have failed to read the part where I said "If ZFS or Btrfs has redundancy".

Re:Legacy file systems should be illegal (3, Interesting)

Chandon Seldon (43083) | about 3 months ago | (#47236035)

Btrfs (at least) can store multiple copies on one disk and use a checksum to identify the good copy to read. Obviously more disks is better, but...

Re:Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47236059)

You mean... just like HFS+ can?

For all that HFS+ is a crufty old tacked on, hacked about mess, it actually has a surprising number of modern features you would expect to find in ZFS and Btrfs like file systems, for example, compression, deduplication, and redundancy support.

I wouldn't be at all surprised if copy on write semantics was added at some point soon too.

Re:Legacy file systems should be illegal (1)

HuguesT (84078) | about 3 months ago | (#47243303)

I'm sorry, citation needed. It does have compression and encryption, and you *can* make a RAID 0 or 1 with it, so it has some (weak) redundancy support, but I have never heard about it having deduplication. ZFS is the only filesystem I know that has deduplication.

Re:Legacy file systems should be illegal (1)

cmurf (2833651) | about 3 months ago | (#47243431)

HFS+ doesn't support deduplication or redundancy, A copy of a file is not what's meant by redundancy, nor is a hard link what's meant by deduplication.

Re:Legacy file systems should be illegal (4, Informative)

mgmartin (580921) | about 3 months ago | (#47236083)

As does zfs: man zfs
copies=1 | 2 | 3 Controls the number of copies of data stored for this dataset. These copies are in addition to any redundancy provided by the pool, for example, mirroring or RAID-Z. The copies are stored on different disks, if possible. The space used by multiple copies is charged to the associated file and dataset, changing the used property and counting against quotas and reservations. Changing this property only affects newly-written data. Therefore, set this property at file system creation time by using the -o copies=N option.

Re:Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47236507)

Btrfs (at least) can store multiple copies on one disk and use a checksum to identify the good copy to read. Obviously more disks is better, but...

The problem with Btrfs is that it is GPL and thus can't be used commercially.

Re:Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47236941)

Untrue - it can be used commercially, as long as other file systems can be plugged into it that are not GPL (depending upon the the version of the GPL) and you provide all aspects that that particular version of the GPL requires. None of my code links to GPL code, although GPL code may be usable with some of it via adapters or the fact that it uses an open API.

Re:Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47237219)

The problem with Btrfs is that it is GPL and thus can't be appropriated and made into closed-source derivative works to be sold commercially.

There, fixed that for you. HTH, HAND.

Re:Legacy file systems should be illegal (2)

gweihir (88907) | about 3 months ago | (#47236673)

Indeed. And only regular checking can detect it (ling SMART self-test and RAID consistency check every 7-14 days). The OP simply is naive and did not bother to find out how to properly archive data.

Re:Legacy file systems should be illegal (1)

qubezz (520511) | about 3 months ago | (#47242871)

The problem is, neither ZFS or Btrfs would have stopped an arbitrary bit inside an arbitrary file from becoming corrupt....

I think you should have a look at this 10 year old blog post: https://blogs.oracle.com/elowe... [oracle.com]
ZFS can use single and double-parity (like RAID5 with two parity drives, but no failure if power is pulled during writing). In addition, it has bit scrubbing where all data is verified regularly.

Re:Legacy file systems should be illegal (2)

CauseBy (3029989) | about 3 months ago | (#47245953)

I don't know if any popular filesystems do so, but there are ways to write data to disk such that flipped bits can be detected and corrected. I remember studying this in comps sci class way back in the Clinton/Bush transition era. So multiple disks and redundancy is one good solution but another is a filesystem that can recover lost bits.

If you need to store 64 bits of data you can imagine laying them out in an 8x8 square. Now, widen the square to 9x9 and write a checksum/evenness bit in the extra column to the right, plus the extra row on the bottom. So, if 8 bits sum to checksum 5, then your evenness bit is a 1 making 5+1 an even number. If the checksum is 4, your evenness bit is a 0.

Now, if one of the bits in the 8x8 square is flipped accidentally, then you can detect it and fix it by looking at your extra row-and-column of evenness data. If one bit in the 8x8 grid gets flipped, then the evenness bits in one row and one column will be wrong, reliably indicating the bad bit.

As a bonus, you have one extra bit in the bottom right corner which can act as a final checksum on the evenness bits. That allows you to detect if your checksums are valid. If only one bit in the whole 9x9 grid is flipped, you should be able to detect it and correct it no matter where it is.

I don't know much about real-world filesystems so I don't know if that is a common procedure or not.

Re:Legacy file systems should be illegal (5, Insightful)

jbolden (176878) | about 3 months ago | (#47236045)

Yes absolutely great idea! Rather than having technical decisions being made at tech conferences and among developers, system administrators and analysts we should move that authority over to legislature. Because we all know we are going to see a far better weighing of the costs and benefits of various technology choices by the legislature than by technology marketplace.

Apple used HFS+ because it worked to successfully migrate people from Mac OS9, it supported a unix / MacOS hybrid. They continue to use it because it has been good enough and many of the more robust filesystems were pretty heavyweight. I'd like something like BTFS too. But I don't think the people who disagree with me should be jailed.

Re:Legacy file systems should be illegal (1)

Anonymous Coward | about 3 months ago | (#47236163)

The op's position is obviously absurd, but seriously how well is this tech conference decision thing going? We have like what, just a small number of even FOSS operating systems have modern file systems. I wouldn't mind Apple being one of the larger operating systems vendors being kicked in the butt on this. Even Microsoft deserves its share of shame on this.

Re:Legacy file systems should be illegal (1)

RealityGone (883555) | about 3 months ago | (#47236605)

Actually, Microsoft does have a next-gen file system now with Server 2012. It may not be on par with Btrfs or Zfs, yet, and it is really only meant for a file server (it requires multiple disks for some features). But at least they're making movement in the right direction.

ReFS : http://en.wikipedia.org/wiki/R... [wikipedia.org]

Re:Legacy file systems should be illegal (1)

mlts (1038732) | about 3 months ago | (#47236921)

Microsoft has two technologies in Windows Server 2012: Storage Spaces (which is LVM level), and ReFS. Both when used together can detect bit rot, but IIRC, only when the Storage Space volume is set to mirroring, nor parity.

This is similar with ZFS. RAID-Z will detect bit rot, but won't fix it. RAID-1, RAID-Z2, and RAID-Z3 will detect and fix bit rot on a scrub. One can also use copies or ditto blocks.

Linux, there isn't much either way. I have no clue if LVM2 + btrfs will do anything about bit rot, assuming it has the ability to repair it from a mirror or a RAID 6 volume. This seems to be one of those "ask four people, get five answers" type of items.

If I were setting up a file server or backend RAID, I'd probably will go with Linux and ZFS (from the zfsonlinux projects.) The / and /boot filesystems wouldn't be able to be placed on ZFS, but almost everything else can. With a RAID-Z2 pool, this will go far in detecting and handling bit rot.

Re: Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47237093)

You can install your / on and even /boot on ZFS.

Look for a video called "Installing Gentoo Linux on Native ZFS" on YouTube.

Re:Legacy file systems should be illegal (1)

Drinking Bleach (975757) | about 3 months ago | (#47237797)

I have both / and /boot on ZFS on Linux.

Re:Legacy file systems should be illegal (1)

KevReedUK (1066760) | about 3 months ago | (#47238693)

Microsoft has two technologies in Windows Server 2012: Storage Spaces (which is LVM level), and ReFS. Both when used together can detect bit rot, but IIRC, only when the Storage Space volume is set to mirroring, nor parity.

Used to be true. In 2012R2, you can use integrity streams on parity spaces too, and get the corruption prevention there.

Re:Legacy file systems should be illegal (1)

Electricity Likes Me (1098643) | about 3 months ago | (#47242203)

RAID-Z will absolutely detect and fix bit-rot.

RAID-Z is single disk parity redundancy aka RAID-5 (but using CoW to plug the write-hole).

A regular, single-disk ZFS device will detect but not be able to fix bit-rot, but that's not any type of RAID.

Re:Legacy file systems should be illegal (1)

KevReedUK (1066760) | about 3 months ago | (#47238683)

On reading up on ReFS, I am of the opinion that it is a step in the right direction, but has been released before it's ready. The version included with Server 2012 (and subsequent versions so far) doesn't include a whole raft of technologies that are (and have been for a looong time) present in NTFS.

Don't get me wrong, I'm no NTFS fanboy, but the majority of the features that they failed to include are ones that are practically indispensable in a range of settings. Whilst I could concur that, with the advent of (compared to a few years ago) cheap storage, NTFS compression has essentially had its day, other features are not so easy to do without. If you're a company that uses (out of necessity) legacy software with a need for 8.3 filenames, you can't use ReFS to host your data. If you need to use EFS, you're SOL. And to design a file-system with such a limited feature-set and release it, then say that it is ideal for using in a file server situation when it doesn't even support QUOTAS? Yes, I know, you can get third party COTS packages to handle that, but why bother if it's already in NTFS?

Frankly, one of the most laughable things to me is that they release this new file system designed to heal itself, but leave out so many features that THEIR OWN OS can't even be installed on it (no hard-links, for a start).

I do, however, go back to my original sentence.... ReFS is a step in the right direction, but is essentially useless in many (most) scenarios until it gets the features that we have largely come to rely on. IMHO, despite MS's claims to the contrary, it's not even file-share ready. The only environment I would even consider this to have a place in its current incarnation is in a tiny-business or home server environment in those frightening (but thankfully rare) cases where hardware RAID is out of the question. Even then, however, it could only be used to store the file shares. No chance of putting your Exchange/Sharepoint/SQL on it. But why bother, when you can just as easily use NTFS?

Re:Legacy file systems should be illegal (1)

jbolden (176878) | about 3 months ago | (#47237417)

but seriously how well is this tech conference decision thing going?

I think pretty well. The technical community has managed to advance and keep up a good rate of advancement, disrupting itself over and over and over again to produce superior products.

Re:Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47236711)

HFS+ was innovated in the 8.1 days, before OSX was in the roadmap. Future modifications may not have been backwards compatible, but would not change the format name.

Re:Legacy file systems should be illegal (1)

kthreadd (1558445) | about 3 months ago | (#47236751)

HFS+ was just an extension to HFS, which goes back to the System 2 days. HFS suffered from a number of limitations which made in unsuitable on volumes larger than 2 GB.

Re:Legacy file systems should be illegal (2)

NJRoadfan (1254248) | about 3 months ago | (#47236847)

Think of HFS+ as the equivalent of FAT32 for Macs. Its basically the old file system with support for larger drives and files. Apple latter tacked on journaling in OS X 10.3. I'm surprised Apple didn't push for a replacement file system after the switchover to Intel CPUs.

Re:Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47237001)

Apple has continued to develop HFS+, and it now sports pretty much every feature you'd expect from a modern file system. Their CoreStorage system looks a lot like it's the beginnings of a pool based storage system like zfs, but it's growing organically rather than being dropped in in one big step.

Re:Legacy file systems should be illegal (1)

Em Adespoton (792954) | about 3 months ago | (#47237459)

Think of HFS+ as the equivalent of FAT32 for Macs. Its basically the old file system with support for larger drives and files. Apple latter tacked on journaling in OS X 10.3. I'm surprised Apple didn't push for a replacement file system after the switchover to Intel CPUs.

https://en.wikipedia.org/wiki/... [wikipedia.org]

It's a bit more complicated than that. First off, HFS+ is actually more like a combination of NTFS and exFAT; they got the basic design down at an early stage so that it would be extensible. This means that they didn't fall into the FAT32 trap. Interestingly, although people refer to it as HFS+, the internals refer to the current incarnation as HFSX.

Journaling was added to HFS+ with OS X 10.2.2. This moved the internal name from HFS+ to HFSJ.
HFSX was added with 10.3, and introduced optional case sensitivity and made the volume wrapper optional.
With 10.4, full ACL support was added to HFSX.
10.5 added hardlinking.
10.6 added optional compression -- which could be where some of the issues being discussed in TFA are from. I used to use STACKER back in the day, until some bad bit flips caused massive data corruption -- I've avoided compressed dynamic storage ever since, until 10.6.

There have been rumours for years of Apple adopting ZFS, and at one point the DP releases of the OS even had it available -- but it has never rolled out into OS X itself.

Instead of tackling "bitrot" head-on, Apple seems to have taken the "make backups easy" approach. This works to some degree, but since the backups use hardlinking, you really only have two copies of the data -- the one on your main drive, and the one on your backup drive. This makes cycling your backup drives even more important than it already was.

Re:Legacy file systems should be illegal (2)

Guy Harris (3803) | about 3 months ago | (#47237747)

10.5 added hardlinking.

Are you certain? The ln command, when run without -s, would return an error if you used it under Tiger or earlier?

Or are you referring to hardlinking to directories, which was something UNIX traditionally supported, but which required root permissions (as it was used by the mkdir command to create the . and .. directories), and which was removed at one point (4.2BSD, as that added the mkdir() system call, making the ability of link() to link to a directory unnecessary?), and added back in 10.5 with the introduction of Time Machine, so that it could be used in backup trees as a very hacky form of de-duplication (each backup tree is a complete copy of the file system being backed up, but if there's an older copy of an unchanged file or a directory everything under which is unchanged, the "copy" is done by making a hard link rather than by copying the file to the backup disk).

Instead of tackling "bitrot" head-on, Apple seems to have taken the "make backups easy" approach. This works to some degree, but since the backups use hardlinking, you really only have two copies of the data -- the one on your main drive, and the one on your backup drive. This makes cycling your backup drives even more important than it already was.

That's what happens with any backup scheme that does incremental backups - if a file hasn't changed, a copy isn't made.

Re:Legacy file systems should be illegal (1)

cmurf (2833651) | about 3 months ago | (#47243387)

On both 10.8 and 10.9 computers, Disk Utility's default format option is "Mac OS Extended (Journaled)" and this translates into a signature on disk of:
00000400 48 2b 00 04 80 00 20 00 48 46 53 4a 00 00 00 75 |H+.... .HFSJ...u|

From opensource.apple.com hfs_format.h [apple.com] says this is version 4.

When choosing either of the case sensitive options, I get:
00000400 48 58 00 05 80 00 20 00 48 46 53 4a 00 00 00 75 |HX.... .HFSJ...u|

If journaling is disabled they become (respectively):
00000400 48 2b 00 04 80 00 00 00 31 30 2e 30 00 00 00 75 |H+......10.0...u|
00000400 48 58 00 05 80 00 00 00 31 30 2e 30 00 00 00 75 |HX......10.0...u|
So internally it's H+ or HX, and both can be HFSJ. For whatever reason, by default we still get version 4 (H+).

Re: Legacy file systems should be illegal (3, Informative)

maccodemonkey (1438585) | about 3 months ago | (#47237569)

They did try and replace the file system around the time of the Intel switch. Got killed by licensing problems.
http://appleinsider.com/articl... [appleinsider.com]

Re: Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47237885)

Wrong, it was a serious rewrite, adding long filename, unicode support, 64 bits, multiple streams per file, unix permission, and probably more. Journaling was added later I thinkk.

Re:Legacy file systems should be illegal (-1)

Anonymous Coward | about 3 months ago | (#47236067)

That would mean using a modern OS. People made a fuss about Windows XP going EOL. You do the math.

Re: Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47236143)

Not sure if stupid or something else.

Changing the default file system doesn't require a different OS.

Re: Legacy file systems should be illegal (3, Informative)

the_B0fh (208483) | about 3 months ago | (#47236357)

Not if your OS is tied intimately to your filesystem. Linux might not, because a large number of things are abstracted out, but FreeBSD depends on its file system, Solaris took a very long time/effort before it could boot off ZFS. Forget about moving Windows off NTFS. Apple actually did some work on putting it onto ZFS, maybe they will continue.

Re: Legacy file systems should be illegal (1)

Anonymous Coward | about 3 months ago | (#47236597)

FreeBSD 10 has an "install to ZFS root" option now. It's been possible for a while to do it manually.

Re: Legacy file systems should be illegal (2)

jbolden (176878) | about 3 months ago | (#47237499)

The developer involved left Apple, went off to found his own company. Completed the work and then got acquired by Oracle. http://getgreenbytes.com/solut... [getgreenbytes.com]
So it is in some sense done. The question is whether Apple is going to buy an Oracle product or Oracle will sell or ...

Re: Legacy file systems should be illegal (1)

Guy Harris (3803) | about 3 months ago | (#47237769)

Forget about moving Windows off NTFS.

Microsoft haven't [microsoft.com] . I guess they realized that software actually used alternate data streams, so they had to add them back to ReFS, although only "up to 128K for both Windows 8.1 and Windows Server 2012 R2", so they're more like "big extended attributes" than full alternate data streams.

Re:Legacy file systems should be illegal (4, Interesting)

peragrin (659227) | about 3 months ago | (#47236113)

This is something everyone forgets.
It takes decades to build long term reliable file systems.

ZFS, BTFS, are less than a decade old.
Windows runs on NTFS Version something. NTFS was started in what year?
HFS, and then HFS + was built in what year?
How long has Microsoft been promising WinFS?

File systems change but only slowly. This is good. you need a good long track record to convince people they won't lose files every ten years due to random malfunctions.

Re:Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47236263)

And then there was ReiserFS, which was really gifted at deleting the data, pretending complete innocence, burying the disk drive and pretending it had run off to Russia, and trying to hide the blood soaked screwdrivers in the woods to recover them when the police aren't looking.

            http://en.wikipedia.org/wiki/Hans_Reiser

Re:Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47236277)

"But he is one of us!"

Haha

Re:Legacy file systems should be illegal (1)

CaptnZilog (33073) | about 3 months ago | (#47236979)

I'll wait for MansonFS, it has a longer history and doesn't pretend to be innocent.

Re: Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47238445)

But you have to admit it had some killer features!

Re: Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47244317)

For crying out loud, the Hans Reiser jokes have been done to death already!

Re:Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47236287)

Yep. IIRC one of the early OSXs(10.1? 10.2?) offered the option to installing using ZFS, so I tried it on an older machine and it just broke so much of OS X although I can't recall at this point IF it was the OS or 3rd party stuff...

I was disappointed and had meant to try it out again later(probably was 10.2 then because...) it was not too long after this that I was getting severely tires of $130 yearly "upgrades", elimination of so many options to make at least powermacs cheaper(e.g. stripping ram, drives, etc. which would leave you with a good price as apple was charging insanely HIGH prices for RAM & hdds & GPUs back then usually anywhere from 2.5-10x market rate), and motorola et. al. were unable to cough up decent processor upgrades. So, it was back primarily to the land of unix/windows/*BSD for me.

Even my older powerbooks once 10.2 got so quickly deprecated ended up running Yellow Dog linux and got snappier again...

Re:Legacy file systems should be illegal (1)

the_B0fh (208483) | about 3 months ago | (#47236373)

Bullshit. I was running up through 10.4 or 10.5 on a PowerMac G4 450mhz. It was more responsive (I didn't say faster) and usable than the dual 1.4Ghz Pentium 3 I had.

Re:Legacy file systems should be illegal (1)

Anonymous Coward | about 3 months ago | (#47236423)

I had an iBook G4 1.2 GHz and 10.5 was the most sluggish thing I've ever seen. I later downgraded back to 10.4 until I retired the hardware.

Re:Legacy file systems should be illegal (1)

nine-times (778537) | about 3 months ago | (#47236371)

How long has Microsoft been promising WinFS?

I thought Microsoft gave up on WinFS. Are they still promising it?

Re:Legacy file systems should be illegal (1)

jbolden (176878) | about 3 months ago | (#47237503)

No they aren't. You are correct in 2006 they announced that the WinFS team was being moved under the SQLServer group and in 2008 they completed a less ambitious product to allow SQL Server to store and access arbitrary files efficiently.

Re:Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47236655)

the first widely used ntfs came in windows nt 3.1, so yeah. it's 21 years ago. They've had a lot of time to test it, compared to hfs+

Re:Legacy file systems should be illegal (1)

NJRoadfan (1254248) | about 3 months ago | (#47236877)

NTFS has received a few non-backward compatible revisions over the years. The last big update was with Windows 2000. What was evil about it was Win2k would silently upgrade the file system on mounted drives, rendering them unreadable on machines running older versions of Windows NT (NT4 SP4 was the first to add NTFS5 read/write support).

Re:Legacy file systems should be illegal (1)

steelfood (895457) | about 3 months ago | (#47236857)

This is good. you need a good long track record to convince people they won't lose files every ten years due to random malfunctions.

Or every few months due to unhandled edge cases.

Writing reliable code is hard. Writing reliable code for generic uses is even harder. Filesystems are at the farthest end of both spectrums: it needs to be most generic and most reliable at the same time.

Re:Legacy file systems should be illegal (1)

Electricity Likes Me (1098643) | about 3 months ago | (#47242211)

10 years is an absurd timeframe.

You can't leave any type of storage media inactivated for that long and expect it to survive. The only "safe" storage is the type you constantly re-verify.

Re:Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47236289)

You realize it's physically inherently impossible to ensure 100% reliable bit transmission, yes? You can push arbitrarily close though.

Re:Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47239059)

The broken bits can always be mended with error correction.

Re:Legacy file systems should be illegal (2)

gweihir (88907) | about 3 months ago | (#47236659)

Bullshit. Anybody doing competent archiving will either use professional archive-grade tape or spinning disks in RAID that gets checked frequently and with a second copy on a geographically independent location. I do that and my loss statistics for the last 14 years is exactly zero. I do have to replace a disk about once every 2 years in the 3-way RAID1 I use as primary archive though. This RAID runs with full SMART self-test every 7 days and RAID consistency check (full data compare) every 14 days. Expecting your data to not rot if you do not maintain is is just plain incompetent.

There used to be one consumer-affordable medium that was archival-grade as well: MOD. I used them for about 10 years and never lost a single bit. Then it became to hard to get a replacement drive, and I moved to the spinning disk solution with additional off-site copies. It seems the consumer does not actually care about long-term archival or at the very least is far to stingy to pay anything for it. Otherwise MOD would have lived on. And do not even get me started about trash like "archival grade" writable DVD/CD. They are not.

Re:Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47236819)

It is about time that legislative authorities makes it illegal for Apple and other negligent vendors to ship file systems that are essentially faulty by design.

"For every complex problem, there exists a solution that is simple, neat, and wrong." I can't say I'm surprised at this authoritarian approach ("there oughta be a law!") to making companies responsible for their customers' data. You should read the fine print on your end-user license agreement. Apple is not responsible for your data, you are. Back up your shit.

Re:Legacy file systems should be illegal (1)

Jane Q. Public (1010737) | about 3 months ago | (#47238357)

It is about time that legislative authorities makes it illegal for Apple and other negligent vendors to ship file systems that are essentially faulty by design.

This is exactly like saying "It should be illegal to sell a Toyota Prius without 4-point harnesses, ABS, and a roll cage."

You knew what it was when you bought it. It is inappropriate to try to make the entire world "safe" via legislation. It doesn't work that way and you'd probably hurt more people than you help.

You aren't a little kid who needs to be forced to wear a bicycle helmet. Or for that matter if it's YOUR kid, you shouldn't need a law that forces you to make her wear a helmet. That's your job.

Re: Legacy file systems should be illegal (0)

Anonymous Coward | about 3 months ago | (#47238625)

Years? Btrfs is still in active development. I've seen both ZFS and btrfs both have corrupting issues. No file system is impervious.

What I hear from this guy... (0)

Anonymous Coward | about 3 months ago | (#47235979)

1. All filesystems and/or storage hardware are imperfect - and what do you mean by "integrity check"?! This thing has ROUNDED CORNERS;

2. I don't back up my data adequately;

3. For some reason, 1 is more important than 2.

Re:What I hear from this guy... (1)

gweihir (88907) | about 3 months ago | (#47236757)

Yes, he did it to himself by making assumptions he liked and zero verification whether they hold up in the real world. Now he blames others for his stupidity.

Backup? (3, Insightful)

graphius (907855) | about 3 months ago | (#47235983)

shouldn't you have backups?

Re:Backup? (4, Insightful)

kthreadd (1558445) | about 3 months ago | (#47236023)

The problem with bit rot is that backups doesn't help. The corrupted file go into the backup and eventually replace the good copy depending on retention policy. You need a file system which uses checksums on all data block so that it can detect a corrupted block after reading it, flag the file as corrupted so that you can restore it from a good backup.

Re:Backup? (5, Insightful)

dgatwood (11270) | about 3 months ago | (#47236139)

Depends on the backup methodology. If your backup works the way Apple's backups do, e.g. only modified files get pushed into a giant tree of hard links, then there's a good chance the corrupted data won't ever make it into a backup, because the modification wasn't explicit. Of course, the downside is that if the file never gets modified, you only have one copy of it, so if the backup gets corrupted, you have no backup.

So yes, in an ideal world, the right answer is proper block checksumming. It's a shame that neither of the two main consumer operating systems currently supports automatic checksumming in the default filesystem.

Re:Backup? (1, Informative)

Antique Geekmeister (740220) | about 3 months ago | (#47236285)

The bitrot will change the checksums and cause the files to show up as modified.

Moreover, what will you do about a reported bitrotted file unless you have genuine archival backups somewhere else?

Re:Backup? (0)

Anonymous Coward | about 3 months ago | (#47236345)

The bitrot will change the checksums and cause the files to show up as modified.

not really -- unless the bitrot changes the file's modifcation time it won't be detected as changed. the backup doesn't do a checksum of the file to compare to the backup... that would require reading all the data on the drive for every backup and would take more than a day for a daily backup.

Re:Backup? (2)

nine-times (778537) | about 3 months ago | (#47236381)

If I remember correctly, that's not how Apple's current backup system works. Every time a file gets written to, there's a log someplace that records that the file was modified. Next time Time Machine runs, it backs up the files in that log. If the OS didn't actually modify the file, it won't get backed up.

I may be wrong, but that's how I understood it.

Re:Backup? (2, Informative)

Anonymous Coward | about 3 months ago | (#47236663)

Macosx Time Machine works by listening to filesystem events except for the first backup where everything is copied over as is. Bit rot doesn't get transferred until you overwrite the file, time by which it should have been obvious something was fishy or the bitrot was negligible and you didn't notice yourself. There are also situations where Time Machine itself says "this backup is fishy, regenerate from scratch?". Happened last week, but only after a failed drive had to be replaced which caused a 150GB backup.

Re:Backup? (1)

m00sh (2538182) | about 3 months ago | (#47237987)

The bitrot will change the checksums and cause the files to show up as modified.

Moreover, what will you do about a reported bitrotted file unless you have genuine archival backups somewhere else?

Bitrot happens when the error correcting can't catch the error because there is a say 1e-10 chance of a bunch of bits flipping and the error correcting not catching it. Yes, add checksum and what not over it and you push the error chance to say 1e-20 but there is still chance of bitrot. There is still a pattern of bit flips that will bypass the checksum as well.

You can always make the error correction stronger to make the error probability as small as possible but over years and gigabytes later, there will be one bit that will be flipped. However, you will lose performance by going with such a strong error correction/checksum system.

Re:Backup? (1)

Smurf (7981) | about 3 months ago | (#47238855)

The parent post is assuming that the user is using Time Machine for the backups. In that case, the checksums are usually not verified (as nine-times said in his reply).

Nevertheless, in some cases Time Machine will perform a "deep" scan, for example if you have not backed up for a long time or if you upgrade your computer's drive. In that case, the corrupted file would be identified as a "change" and would be backed up again, just as you said.

Nevertheless, take into account that the corrupted file is not replacing the original in the backup. Both copies are left there so once you discover the corruption you can use Time Machine to navigate to a backup that is old enough and allow you to recover the file.

Re:Backup? (1)

dgatwood (11270) | about 3 months ago | (#47240219)

Even during a deep traversal, AFAIK, it is just using modification information in the filesystem, not reading the entire file. Otherwise, a deep traversal would take as long as a full backup (days) instead of a tiny fraction of that time.

Now if the length of the file changes because of filesystem corruption, that's another matter.

Re:Backup? (1)

Smurf (7981) | about 3 months ago | (#47240467)

Very good point!

Still the gist of my comment remains: the old, uncorrupted copy of the corrupted file is kept in Time Machine even if the corrupted file ever gets into the backup. Having access to all older versions of your files is what Time Machine is all about!

Differential backup (1)

phorm (591458) | about 3 months ago | (#47249293)

That should be OK for differential backups (depending on how often you make a "full"). If a file changes by 1 bit, that'll affect a byte. A differential will be written, but it hardly takes any space and you can still roll back.

Make a good reason to *check* backups every now and then though.

Re:Backup? (1)

NJRoadfan (1254248) | about 3 months ago | (#47236905)

Its a shame that none of the major SOHO NAS vendors (Synology, Drobo, etc) use check summing file systems. They seem to be sticking with things like ext4 instead of ZFS.

Re:Backup? (1)

Bing Tsher E (943915) | about 3 months ago | (#47236955)

only modified files get pushed into a giant tree of hard links,

Wouldn't that mean that files modified by a corrupt filesystem would be the first candidates for backing up?

Re:Backup? (1)

dgatwood (11270) | about 3 months ago | (#47237971)

Only files modified by a vnode write operation.

Re:Backup? (0)

Anonymous Coward | about 3 months ago | (#47237425)

That's what the parity drive is for. Set up a RAID5, real or LVM and have a checksum for every bit. I have 3TB of video on a 10 year old Dell Poweredge, yes that kind of video. I've had to replace both the motherboard and power supply on it due to age, but NEVER have I had a file that was *just* corrupted.

It's a shame that people don't use the tools available to them in all major operating systems. (FTFY)

Re:Backup? (1)

dargaud (518470) | about 3 months ago | (#47237817)

I've never understood why, when you save a file, a checksum isn't computed at the same time and stored among the metadata. Then you can have a command that operates on a file, a directory or the entire filesystem (in that case when there's low disk activity) to verify that checksum. It would be easy and useful, no ?

Re:Backup? (1)

Jeremi (14640) | about 3 months ago | (#47239067)

I've never understood why, when you save a file, a checksum isn't computed at the same time and stored among the metadata. [...] It would be easy and useful, no ?

It would be, and applications that just write out a file to disk can and do implement exactly that (although I think many of them save the checksum in the file's data-stream itself, rather than as filesystem-dependent metadata).

Implementing checksumming at the filesystem level is a good deal trickier, because the filesystem has to support more than just one-and-done writing out of new files. It has to allow programs to do things like mmap() files into a region of memory and keep the file-on-disk synchronized with the mapped region of RAM whenever the CPU writes data to that RAM; and it has to allow programs to seek around and overwrite arbitrary subsections of an existing file from multiple threads simultaneously, etc.

I think it is possible for a filesystem to maintain/update checksums even in the face of all of that, and it may even possible to do it efficiently -- but I think it is also sufficiently difficult that most filesystems implementers don't bother to try (especially since even minor errors in the checksumming mechanism will present themselves to the user as messages that his files have been corrupted -- and whether the files actually are corrupted or not, that will make the user very unhappy).

Re: Backup? (0)

Anonymous Coward | about 3 months ago | (#47239101)

This is why you do block checksums,

Re:Backup? (0)

Anonymous Coward | about 3 months ago | (#47236973)

You need a file system which uses checksums on all data block so that it can detect a corrupted block after reading it, flag the file as corrupted so that you can restore it from a good backup.

Data blocks already have checksums on the drive itself. If you want additional, backup-related integrity checks, they should go into the backup system. And backup archives archives are already protected with checksums and error correction.

Putting this stuff into a file system (like ZFS does) is just plain stupid.

Re:Backup? (1)

Anonymous Coward | about 3 months ago | (#47237085)

Are you stupid or just plain retarded.

ZFS is the most robust, least susceptible to bitrot disk/volume/filesystem manager out there.

It can detect problems due to simple power supply issues, bad sectors, etc and fix them before the data is committed.

It not only checksums the data as written, but does it at every layer where the data transitions through.

Disk surface to disk cache, disk cache to system memory, and back - every place there's a transition, it does a checksum, correcting any errors along the way.

Aside from having total media failure, you will not lose a single bit of data with ZFS.

Re:Backup? (0)

Anonymous Coward | about 3 months ago | (#47237127)

Surely you would only copy the file to the backup ONCE, unless it is changed? (And not changed by being corrupted). Sounds like the author of the article doesn't back things up. How much would it have cost to back up all of his photos to a DVDR?

Re:Backup? (1)

jopsen (885607) | about 3 months ago | (#47237131)

You need a file system which uses checksums on all data block so that it can detect a corrupted block after reading it, flag the file as corrupted so that you can restore it from a good backup.

When I decide to archive a lot of files I put them in a tarball and generate par2 files... That way a single bitflip or two will be okay :)

Re:Backup? (1)

Marillion (33728) | about 3 months ago | (#47237285)

I'm a fan of computing par2 repair blocks at a 15%. Every so often run a par2verify.

I beg you pardon, but (0)

Anonymous Coward | about 3 months ago | (#47237287)

The problem with bit rot is that backups doesn't help. The corrupted file go into the backup and eventually replace the good copy depending on retention policy. You need a file system which uses checksums on all data block so that it can detect a corrupted block after reading it, flag the file as corrupted so that you can restore it from a good backup.

Sounds like written by someone who is confused what backups are for and who hasn't heard of making archive copies. Backups are for recovering sudden catastrophe or minor mishaps (recover recent file loss), Archive copies are, well, digging archived copies of files looooooooooong after their creation.

You should have multiple separate archive copies to different kind of media, preferably ro-media and store at least one copy offsite.

Archive copy can be a full copy clone saved perpetuity or made with a dedicated archiving software which provides additional features and book keeping.

Re:Backup? (0)

Anonymous Coward | about 3 months ago | (#47236049)

From how far back to you propose the backup will be retrieved? Even tape media kept in perfect conditions may be storing the degraded copy of the file if the rot occurred before the backup was taken.

The critical point to remember is that bitrot occurs without being noticed. It could be a single bit that flips or an entire block of a file, but the file systems still says "all is happy here... move along!"

Backup tools do not detect this event. File systems like ZFS and BTRFS work to prevent bitrot, but there are debates as to if they actually do that job.

Re: Backup? (0)

Anonymous Coward | about 3 months ago | (#47239115)

Zfs does checksum on read so it will detect bitrot. It also does scheduled scrubs, so if you don't read the file for a long time, zfs still will.

Re:Backup? (2)

ZosX (517789) | about 3 months ago | (#47236057)

This is a good idea, but not a solution. Often you have no idea that the file is bad until after the fact, in this case years later. I've had mp3 collections get glitches here and there after a few copies from various drives. If you have no idea the data is bad in the first place, your backup of the data isn't going to be any better. I would say that all of my photography I've collected over the years has stayed readable somehow. I do check in lightroom every once in a while, but I wouldn't be shocked to find a random unreadable file. Not good really, but there's probably not much I can do other than make sure that my files are verifiable.

Re:Backup? (1)

ColdWetDog (752185) | about 3 months ago | (#47236101)

I have close to 4 terabytes of photography and video stored (not that kind of photography and video). I, too, have seen occasional unreadable files, typically in JPEGS but also an occasional TIFF file. Any compressed container (like a JPEG) is going to be more susceptible to this issue thus JPEGs aren't a great storage format. Video files are harder to figure - a corrupted bit could easily get overlooked.

I've never actually lost a picture that I was interested in - I always have more than one copy of the image on the disk - a TIFF and a RAW file typically. Yes, it would be nice if the file system didn't do that. No, I don't think I would believe anybodies claim that that would indeed happen. Further, it's always a risk - benefit calculation. You can spend a lot more money getting near perfect replication but I don't think many people are willing to have a system with ECC memory throughout the chain.

Re:Backup? (1)

wagnerrp (1305589) | about 3 months ago | (#47236559)

Video files are harder to figure - a corrupted bit could easily get overlooked.

Again, it depends on whether it is compressed or not. A corrupted bit in video with only interframe compression will look just like a damaged JPEG. You may have an unreadable frame, or may have a corrupted macroblock or two in that frame. A corrupted bit in video with intraframe compression will smear that corrupted frame or macroblock for potentially several seconds until you hit the next I-frame to flush the image.

You can spend a lot more money getting near perfect replication but I don't think many people are willing to have a system with ECC memory throughout the chain.

The common solution to this issue is software, not hardware. You have your filesystem compute and store checksums at the block level, and you give your filesystem access to redundancy, either through redundant copies on disk, or multiple parity disks. When your filesystem reads the data, it checks it against the checksum, and if needed, recomputes the data from the redundant storage. That said, you do still need ECC memory on the CPU doing those calculations for it to be reliable.

Re:Backup? (1)

ColdWetDog (752185) | about 3 months ago | (#47237635)

True, but what I'm getting at is that a couple of dropouts in multi terrabyte data sets doesn't bother me (YMMV, of course). The current system seems 'good enough' for what I'm doing and I imagine I am in the majority as far as home / SOHO class PCs are concerned. If you're running an enterprise data shop, you have other priorities and options, but I can see why Apple in particular, hasn't moved off of HFS+ so far.

Re:Backup? (0)

Anonymous Coward | about 3 months ago | (#47237911)

But what happens if the bit-rot happens to the checksum rather than the file?

Re:Backup? (1)

wagnerrp (1305589) | about 3 months ago | (#47238527)

Then you store three copies of the checksum, and compare.

Re:Backup? (0)

Anonymous Coward | about 3 months ago | (#47284189)

So, RAID then? :)

Re:Backup? (1)

wagnerrp (1305589) | about 3 months ago | (#47285261)

No, just duplication. Most filesystems have some degree of metadata duplication anyway, for redundancy and performance.

Re:Backup? (1)

stinerman (812158) | about 3 months ago | (#47238559)

Totally read "4 terabytes of pornography". I was debating where to make a toast to you or send you a drum of Jergens.

Re: Backup? (0)

Anonymous Coward | about 3 months ago | (#47239121)

Ram is cheap. Ecc is about 12 percent more expensive. It is still cheap. The general public simply doesn't know why they would want it and/or does not care.

Re:Backup? (2)

rnturn (11092) | about 3 months ago | (#47236225)

Even if you did have backups how could you even begin to know which saveset to restore from? You could have been backing up a corrupted file for a lo-o-ong time.

Friends wonder why I still purchase physical books and CDs. This is why. I'll have to come up with a simple 2-3 sentence explanation of the problem the OP was describing for when they ask next time. I've had MP3 files made from my CD collection mysteriously become corrupted over time. No problem, I can just re-rip/convert/etc. but losing the original digital version of your newborn would be heartbreaking. Make several copies to reduce the odds of losing it. Make a good print using archival paper and inks and keep in away from light in a safe deposit box so it could be rescanned should the digital file become corrupted. Of course, one can go overboard as not every photo is worth that kind of effort but it appears we might be starting to see, first-hand, the problems described in Bergeron's "Dark Ages II". Even worse what if this [consortiuminfo.org] were to happen? (So don't even bring up the "cloud", OK?)

Re:Backup? (2)

gweihir (88907) | about 3 months ago | (#47236769)

You should have:
1. backups
2. redundancy
3. regular integrity checks of your data

Or alternatively, you should have been using an archival grade medium, like archival tape or (historically now unfortunately) MOD.

What the OP did is just plain incompetent and stupid and if he had spent 15 minutes to find out how to properly archive data, he would now not be in this fix. Instead he made assumptions without understanding or verification against the real world now blames others for his failure. Pathetic. Dunning-Kruger effect at work.

Re:Backup? (1)

radarskiy (2874255) | about 3 months ago | (#47238569)

How do you distinguish between intentional and unintentional changes? How much storage overhead do you need to keep all changes so that you can roll back any unintentional change?

Re:Backup? (1)

gweihir (88907) | about 3 months ago | (#47242305)

You are talking about a different problem. This discussion is about flat files that do not get changed. You are talking about database-sores and the like.

Re:Backup? (1)

radarskiy (2874255) | about 3 months ago | (#47242889)

"You are talking about a different problem"
Confiming the discussion to one specific subset on one specific person's storage needs would make it a pretty useless public discussion.

"You are talking about database-sores and the like."
No I am not. Stop lying.

Re:Backup? (1)

gweihir (88907) | about 3 months ago | (#47243243)

What kind of fucked-up jerk are you? Have you even read the OPs article? It is about long-term flat-file storage in a file-system. It us not about journaled, reversible changes to data and that is a completely different problem with completely different requirements and approaches.

Re:Backup? (1)

radarskiy (2874255) | about 3 months ago | (#47243933)

"It us not about journaled, reversible changes to data and that is a completely different problem with completely different requirements and approaches."
That is not what I am talking about. Stop lying.

Re:Backup? (1)

gweihir (88907) | about 3 months ago | (#47247551)

Well, obviously you are not talking about anything relevant here. Go away.

Re:Backup? (1)

sh00z (206503) | about 3 months ago | (#47283951)

How do you distinguish between intentional and unintentional changes? How much storage overhead do you need to keep all changes so that you can roll back any unintentional change?

I'm only a rocket scientist, not a CS person, but it seems intuitive to me. If it's an intentional change, then the new version will have a later last-modified date than the back-up. If the hash of the back-up copy made at the time it was written matches a hash calculated at the time of the second back-up, then the integrity of the back-up is confirmed. If the active copy has not been intentionally changed in the interval, and its hash no longer matches the other two, then the active should be discarded in favor of the back-up. I'm sure you can do the mental figuring for the equivalent to detect if the back-up rather than the active has bitrotted.

The challenge arrives if you have *both* intentional and unintentional changes to the same file between successive back-ups. To exclude unintentional changes, you would have to do hashes every time you save/compile/whatever,as well as keep a keystroke log of the edits. Then, you would have to execute the exact same change process on the back-up copy, repeating the process described above. It would be incredibly resource-intensive (essentially having a 'bot duplicate the work you have performed between back-ups), but it would sure be thorough.

Lord, I HOPE this is not an original idea. If I just invented it, and somebody tries to patent it later, you're in for a world of hurt.

Not if you're the IRS... (1)

Nova Express (100383) | about 3 months ago | (#47236975)

...and you want to prevent those nosy congressmen pawing through your emails looking for felonies...

Re:Backup? (0)

Anonymous Coward | about 3 months ago | (#47239313)

Or maybe even "frequent backups" where the backed up file contains errors just as well as the file on the suspect disk.

It's the contents of the files... (0)

Anonymous Coward | about 3 months ago | (#47235987)

It's the contents of the files that are corrupted. This has nothing to do with HFS+, and everything to do with a lack of redundancy, and a bad hard drive.

Re:It's the contents of the files... (1)

kthreadd (1558445) | about 3 months ago | (#47236037)

The point is that there are good file systems that can detect when the storage unit fails, give you an alert and allow you to restore the file from a good backup. Without this feature the corrupted file will just get backed up like any other file and eventually replace the good backup.

Re:It's the contents of the files... (1)

gweihir (88907) | about 3 months ago | (#47236779)

No, there are not. There are data-archival systems that can do this though. This is not a filesystem-layer problem at all.

Re:It's the contents of the files... (1)

Guspaz (556486) | about 3 months ago | (#47237237)

Yes, there are. There are filesystems that do per-block checksums. If data corruption occurs, it knows about it as soon as it tries to read the block. If it has no redundancy, ZFS will tell you which file is corrupt and suggest restoring it from backup.

Re:It's the contents of the files... (1)

gweihir (88907) | about 3 months ago | (#47237625)

You have no clue what you are talking about. Simple per-block checksums on the HDDs are already doing that. This is not a filesystem issue. This is also not a subject topic where clueless idiots like you can contribute anything.

Re:It's the contents of the files... (1)

Marsell (16980) | about 3 months ago | (#47239179)

The irony of your calling someone else clueless...

Drives do indeed have checksums on their blocks. That does not prevent them from sometimes feeding you back garbage anyway -- see misdirected and phantom reads and writes. Since ZFS uses a self-validating merkle tree, whereas disk checksums live in the same block as the data, ZFS is largely immune to this problem.

If you've worked with disks any length of time, as in actually trying to write a robust filesystem, you'd know that disks sometimes lie. They usually work but every now and then they do the most ridiculous things, due to mechanical, electrical or firmware problems. That's why filesystems like ZFS were created (what, you thought Sun spent man-decades of expert time on it for giggles?). kthreadd is correct.

Please just stay away from storage. The topic is much more complicated than you make it out to be.

Re:It's the contents of the files... (0)

Chandon Seldon (43083) | about 3 months ago | (#47236053)

It turns out that you can have this problem even with a RAID. I'm running RAID-1 with 3 disks for my long term storage, and I need to move it from ext4 to btrfs at some point to avoid the failure case where it selects a bitrotted copy to read from. It would be nice if the RAID layer were smart enough to use the matching two of three, but that would make reads slower...

Re:It's the contents of the files... (1)

wagnerrp (1305589) | about 3 months ago | (#47236569)

Sufficiently advanced RAID implementations will carry checksums of those blocks for exactly that purpose.

Re:It's the contents of the files... (1)

gweihir (88907) | about 3 months ago | (#47236809)

Bullshit. That is not a RAID-layer task. And the disks do that themselves just fine. Historically, there were actually RAID implementations that did what you describe, but they were scrapped due to various problems. Doing this in RAID is the wrong approach.

Re:It's the contents of the files... (1)

Guspaz (556486) | about 3 months ago | (#47237249)

Nope, wagnerrp is correct. raidz does exactly what he describes, and your claim that raidz was "scrapped due to various problems" is incorrect.

Re:It's the contents of the files... (0)

gweihir (88907) | about 3 months ago | (#47237629)

You are completely unaware of storage system history. And you are just as arrogant as you are clueless. Pathetic.

Re:It's the contents of the files... (1)

Guspaz (556486) | about 3 months ago | (#47239239)

Uh huh. As somebody who uses raidz, and has a decent high-level idea of how it works, I'm going to say you're full of shit.

Re:It's the contents of the files... (1)

gweihir (88907) | about 3 months ago | (#47239409)

Still pathetic. And no, you have absolutely no clue. Do you even know what an ECC is and how low the probability of it not detecting an error is for HDDs? And while you are looking that up, look up the Dunning-Kruger effect as well.

Re:It's the contents of the files... (1)

Guspaz (556486) | about 3 months ago | (#47239601)

I don't know why all your posts are so focused on the drive's internal checksums not detecting errors. As you say, that's very rare. A far more common occurrence, and one that I've seen many times, is a drive detecting corrupt data and being unable to correct it. At that point, it's up to the filesystem to use whatever redundancy you've provided (be it duplication or parity) to recover the lost data. The error correction on a drive can't do squat if a block is sufficiently corrupt.

You act as if the drive missing corruption is the problem. It's not.

Re:It's the contents of the files... (1)

gweihir (88907) | about 3 months ago | (#47242265)

My claim is exactly the other way round. The claim by others was that extra error detection on RAID layer was needed. It is not.

"Sufficiently advanced RAID implementations will carry checksums of those blocks for exactly that purpose." is wrong. That is all I am saying. RAID does not carry block checksums because they are not needed. RAID may carry redundancy in several different forms, but redundancy (even ECC) is not "checksums".

What people here seem to completely miss is that filesystem-level data checksums are not there to detect corruption on the disk. The disk does that just fine. They are there to detect data corruption due to corruption in the path from main memory to the disk and back from it.

Re:It's the contents of the files... (1)

the_B0fh (208483) | about 3 months ago | (#47236715)

ZFS raid does that.

Re:It's the contents of the files... (1)

gweihir (88907) | about 3 months ago | (#47236797)

That does not happen unless you fail to verify the data when placing it on that raid. For bit-rot detection on the disks, the disk-internal is more than enough. WHile the manufacturers state "1 uncorrectable sector in 10^15" read, it is more like "1 in 10^30" undetected faulty sector. And of course, any sane RAID setup includes a full disk data consistency check every 14 days or so. If you place defective data on the RAID, the RAID can do nothing for you.

I would also really recommend to read up on RAID and disk technology, you do not seem to understand how things work.

Re:It's the contents of the files... (1)

cmurf (2833651) | about 3 months ago | (#47243475)

OK so you're saying that manufacturer's are wrong by f'n 15 orders of magnitude when they say "less than 1 uncorrectable error for every 10^14 bits read". That's such an immense amount of error that you're basically accusing them of being incompetent, possibly even of fraud. Next you're also proposing that consumer hard drives have a bit error rate 13 orders of magnitude less than that claimed by LTO tape manufacturers. You're like the drunk guy running into walls, tripping over himself, shouting and pissing himself, while bitching about everyone else have craptastic balance and smelling like alcohol and urine and talking way too loudly.

Bitrot not the fault of filesystem (5, Insightful)

Gaygirlie (1657131) | about 3 months ago | (#47236005)

Bitrot isn't the fault of the filesystem unless something is badly buggy. It's the fault of the underlying storage-device itself. Attacking HFS+ for something like that is just silly. Now, with that said there are filesystems out there that can guard against bitrot, most notably Btrfs and ZFS. Both Btrfs and ZFS can be used just like a regular filesystem where no parity-information or duplicate copies are saved and in such a case there is no safety against bitrot, but once you enable parity they can silently heal any affected files without issues. The downside? Saving parity consumes a lot more HDD-space, and that's why it's not done by default by most filesystems.

Re:Bitrot not the fault of filesystem (0)

Anonymous Coward | about 3 months ago | (#47236065)

"Attacking HFS+ for something like that is just silly." TFP says they're certainly not limited to HFS. The point was it did occur under HFS also, it's not immune.

Re:Bitrot not the fault of filesystem (1)

jbolden (176878) | about 3 months ago | (#47236071)

There is a 3rd possibility. As the size of the dataset increases you can construct a more complex error correcting code on that dataset with loss of spacing being 1/n. Note that's essentially saving information about the decoding and then the coded information, sort of like how compression works. Which for most files would be essentially free. And of course you could combine with this compression by default which might very well result in a net savings. But then you pick up computational complexity. With extra CPUs though having a CPU (or hardware in the drive) dedicated to handling that isn't unreasonable.

Re:Bitrot not the fault of filesystem (0)

Anonymous Coward | about 3 months ago | (#47236131)

And of course you could combine with this compression by default which might very well result in a net savings.

Uhmmm, all common formats for large files are already compressed pretty effectively, so no you could add compression and get net savings.

reading checksum + n blocks is SLOW (1)

raymorris (2726007) | about 3 months ago | (#47236169)

It's not a matter of CPU load. Suppose you have one checksum block for every eight data blocks. In order to verify the checksum on read, you have to read the checksum block and all eight data blocks. So you have to read a total of nine blocks instead of one. Reading from the disk is one if the slowest operations in a computer, so ddoing it nine times instead of one slows things down considerably.

Re:reading checksum + n blocks is SLOW (2)

jbolden (176878) | about 3 months ago | (#47236349)

You don't have checksum blocks in the space efficient method. Rather in the computational way I'm talking about it is a transformation. You might have something like every 6354 bits becomes 6311 bits after the complex transformation. It doesn't slow down the read but you have to do math.

Re:Bitrot not the fault of filesystem (0)

Gaygirlie (1657131) | about 3 months ago | (#47236417)

Note that's essentially saving information about the decoding and then the coded information, sort of like how compression works. Which for most files would be essentially free.

No, you are talking out of your ass there. First of all, if the system worked like you explain it then having the decode - block itself get corrupted would render every single file relying on it invalid, so you'd still end up having to maintain at least a second copy of the decode - block and checksums for them both, but you'd still have two points of breakage that, if ever corrupted, would still render everything corrupted. That's really shitty design. On a similar note, maintaining such a decode - block is far from being "free" -- try compressing e.g. a 2-hour movie and you'll notice that most likely it only just got slightly bigger, not smaller. Then do the same while you enabled recovery mode, ie. the compression system writes a second decode - block in the file, and the file will certainly get even bigger. Go on, try it, you'll see.

Re:Bitrot not the fault of filesystem (0)

Anonymous Coward | about 3 months ago | (#47237039)

No you're wrong. What he's describing is fairly common and actually there's a very popular system on Usenet called PAR based upon the concept.

Think of it this way, you have four blocks:

{Data: {1, 2, 3, 4}, checksum: 10}
{Data: {5, 6, 7, 8}, checksum: 26}
{Data: {1, 2, 7, 8}, checksum: 18}
{Data: {5, 6, 3, 4}, checksum: 18}

Now you add to this a data recovery block. The data recovery block looks like this:

{Data: {12, 16, 20, 24}, checksum: 72}

You've just increased the size of the data by 20%. Now, watch what happens if we lose data:

{Data: {1, A, 3, 4}, checksum: 10}
{Data: {5, 6, 7, 8}, checksum: 26}
{Data: {1, 2, 7, B}, checksum: 18}
{Data: {5, 6, 3, C}, checksum: 18}

In this case A can be derived exclusively from the data recovery block, it's 16-6-2-6=2. This value can be validated using the checksum of the block.

B and C can each be derived from the checksum of the block and the end result validated using the data recovery block:

B = 18-7-2-1 = 8,
C = 18-3-6-5 = 4

Does 8 + 4 + B + C (= 8 + 4 + 8 + 4) = 24? Why yes it does!

Obviously you can (and would) add additional checksums to the entire system just to double triple check that your calculations have integrity. But the system works. We lost 3 out of 16 of our data points and were able to recover them by looking at the additional nine checksums.

Note this is a simplified system. Proper systems use CRCs ;-)

Re:Bitrot not the fault of filesystem (1)

jbolden (176878) | about 3 months ago | (#47237447)

BTW this anon is absolutely correct. Worth modding up.

Re:Bitrot not the fault of filesystem (0)

Anonymous Coward | about 3 months ago | (#47239031)

absolutely correct

You really like hyperbole, don'tcha?

You've just increased the size of the data by 20%.

25% increase. You're suggesting that 1.2 * old_size == new_size. old_size is 20 (5 per block--4 data and 1 checksum--and 4 blocks). 1.2 * 20 = 24. You've added one block, or 5, so new_size is actually 25, not 24. Thus, it should be 1.25 * old_size == new size. 1.25 * 20 = 25.

</pedantry>

Re:Bitrot not the fault of filesystem (0)

Anonymous Coward | about 3 months ago | (#47245121)

(original A/C here) You're right, but bear in mind that plugging in real numbers usually results in there being much less overhead. If we were using a 16x16 block, instead of 4x4, the increase would be a mere 6.25%. When we get to 512x512...

Re:Bitrot not the fault of filesystem (1)

jbolden (176878) | about 3 months ago | (#47237445)

No, you are talking out of your ass there.

The fact you don't know about something doesn't mean the other person is talking out of their ass.

First of all, if the system worked like you explain it then having the decode - block itself get corrupted would render every single file relying on it invalid, so you'd still end up having to maintain at least a second copy of the decode - block and checksums for them both, but you'd still have two points of breakage that, if ever corrupted, would still render everything corrupted. That's really shitty design.

Well first off the decode matrix is derivable. It wouldn't necessarily be a block. Moreover a 6000x6000 matrix is 4.5m you can have to copies of it and it wouldn't have any impact on storage.

. . Then do the same while you enabled recovery mode, ie. the compression system writes a second decode - block in the file, and the file will certainly get even bigger. Go on, try it, you'll see.

Of course it gets bigger with error correct. But using a complex code not by very much. As the check matrix gets larger (i.e. more computationally complex) the cost goes to zero. The cost in terms of storage is slightly worse than 1 -> 1+1/n where n is the number of bits in the encoding.

Re: Bitrot not the fault of filesystem (4, Insightful)

jmitchel!jmitchel.co (254506) | about 3 months ago | (#47236073)

Even with just checksums, knowing that there is corruption means knowing to restore from backups. And in the consumer space most people have plenty of space to keep parity if it comes to that.

Re:Bitrot not the fault of filesystem (0)

Anonymous Coward | about 3 months ago | (#47236295)

In fact google has run into this problem. Harddrives do checksums, but they still found bits flipped that gave the checksum and therefor not detected as an error.
google uses a lot of harddrives.

Re:Bitrot not the fault of filesystem (0)

Anonymous Coward | about 3 months ago | (#47236363)

Which is why there is Raid 5 with double parity...

And when done in hardware, there is no delay in reading the data, nor in the correction. Raid management can be used to detect "prefailures" where the bit error rate gets too large... and requests a replacement before actual failure.

Re:Bitrot not the fault of filesystem (1)

wagnerrp (1305589) | about 3 months ago | (#47236573)

Isn't that RAID6?

Re:Bitrot not the fault of filesystem (1)

bwwatr (3520289) | about 3 months ago | (#47246795)

Not really. Hardware RAID5 uses a parity disk to allow sectors to be read when an unrecoverable read error (URE) occurs on one of the member disks. RAID6 will allow unrecoverable errors to happen on two member disks. But in cases where the member disk doesn't encounter a read error, but instead happily reads back a block of data with a flipped bit, RAID isn't going to help you. ZFS/Btrfs would have helped you though.

Re:Bitrot not the fault of filesystem (0)

Anonymous Coward | about 3 months ago | (#47236369)

More likely it is bad RAM, and not using ECC RAM. Or flaky power supply. Maybe in the process of copying the file or saving it the data buffer got corrupted.

The hard disk (or CD or DVD, OP does not mention at all what type of physical storage he is using) has built-in error correction so the data shouldn't be easily corrupted.

Re:Bitrot not the fault of filesystem (1)

Gaygirlie (1657131) | about 3 months ago | (#47236389)

More likely it is bad RAM, and not using ECC RAM. Or flaky power supply. Maybe in the process of copying the file or saving it the data buffer got corrupted.

The hard disk (or CD or DVD, OP does not mention at all what type of physical storage he is using) has built-in error correction so the data shouldn't be easily corrupted.

Mmmmno. If he had bad RAM he would be having a lot more issues with the system than just 28 broken files over 6 years. And no, HDDs do not have built-in error correction, they have checksums -- those things are not the same thing.

Re:Bitrot not the fault of filesystem (1)

fnj (64210) | about 3 months ago | (#47236517)

And no, HDDs do not have built-in error correction, they have checksums -- those things are not the same thing.

Sorry to inform you that your knowledge on this subject is not perfectly correct and inclusive. Hard drives use per-sector ECC. ECC stands for Error Correction Code. The very term tells you its function is to do precisely what you say is not done. Here is one tutorial [pcguide.com] . This stuff is pretty basic and widely known.

"When a sector is written to the hard disk, the appropriate ECC codes are generated and stored in the bits reserved for them. When the sector is read back, the user data read, combined with the ECC bits, can tell the controller if any errors occurred during the read. Errors that can be corrected using the redundant information are corrected before passing the data to the rest of the system. The system can also tell when there is too much damage to the data to correct, and will issue an error notification in that event. The sophisticated firmware present in all modern drives uses ECC as part of its overall error management protocols. This is all done "on the fly" with no intervention from the user required, and no slowdown in performance even when errors are encountered and must be corrected."

Re:Bitrot not the fault of filesystem (1)

Electricity Likes Me (1098643) | about 3 months ago | (#47236591)

Hard disks use ECC to allow the disk to reach the capacities it does. It is not designed for anything other then making the hard disk perform well. It doesn't protect you against hard disks which write incorrect data to start with, or have faulty cables etc.

Re:Bitrot not the fault of filesystem (0)

Anonymous Coward | about 3 months ago | (#47236741)

Typical hard disks are specified to have an uncorrectable bit error rate of 1 in 10TB, and uncorrectable does not mean undetectable, so the chance of getting wrong data back is still much much lower. In practice you either get a read error or you get your data back. The third possibility, getting wrong data back, is exceedingly unlikely. 28 undetected bit errors in just 100GB of data certainly isn't the hard disk's fault. You may have gotten used to silent corruption from using USB flash drives, but hard disks really are not that shoddy.

Re:Bitrot not the fault of filesystem (1)

laird (2705) | about 3 months ago | (#47238499)

To correct slightly - ECC isn't really about disk capacity. It's there because magnetic media isn't perfect, so even on a well written block there's a percentage chance of a read having a bit mis-read, and even on a failed read there's some percentage of good data read. The ECC lets the drive controller correct for those errors, so the vast majority of errors are corrected by the controller. A really smart controller or driver (GCC Technologies had this 20 years ago, perhaps they all do now) pays attention to ECC errors and re-writes blocks that have errors so that any marginal writes are rewritten with the user's data is protected before the block degrades to an unrecoverable failure, automatically. And if a read is so bad that ECC fails, you can re-try the read until you get a good enough read for ECC to recover the data, which almost always works, then rewrite it. If you do that, it's very, very hard to lose data on magnetic media, because it's nearly impossible for a block to go completely bad with no warning.

That being said, with the huge volumes of data that people use now, even a very rare percentage multiplied by that huge pile of data means they'll lose data. If you really care about that, you need to store your data on two physically separate devices, so a physical failure of one can't affect the other. This is expensive-ish, but that's the cost of protecting data. So an offsite backup is really the only solid option. Anything in your house can be affected by a fire, power spike, etc., so if you really care about the data, get CrashPlan or something like that.

Re:Bitrot not the fault of filesystem (1)

fnj (64210) | about 3 months ago | (#47238859)

Hard disks use ECC to allow the disk to reach the capacities it does. It is not designed for anything other then making the hard disk perform well. It doesn't protect you against hard disks which write incorrect data to start with, or have faulty cables etc.

So you don't think it's worth providing recovery from some error cases just because you can't protect against every single case?

If you don't mind my asking, why would you claim something that is patently untrue? ECC detects and corrects ALL instances of a hard drive writing single bad bits per sector. So it's clearly "protecting you" against a great many instances of "incorrect data" being written by the drive.

Incidentally, the SATA protocol incorporates 32-bit CRCs in all packets flowing in either direction. This will pick up an extremely high percentage of errors arising from faulty cables and bad receiver and transmitter circuits at the interface. For reads, the host does not use data flagged with CRC errors. For writes, the drive does not write data flagged with CRC errors. This feature has saved my ass more than once by preventing the writing of corrupt data when I had problems with my cables.

Re:Bitrot not the fault of filesystem (1)

Electricity Likes Me (1098643) | about 3 months ago | (#47239519)

What part of my comment sounds like I'm saying it's not worth doing error recovery?

Re:Bitrot not the fault of filesystem (1)

fnj (64210) | about 3 months ago | (#47239683)

The part where you claim "It [ECC] doesn't protect you against hard disks which write incorrect data to start with, or have faulty cables etc."

Re:Bitrot not the fault of filesystem (1)

Electricity Likes Me (1098643) | about 3 months ago | (#47242187)

And again, which part of that seems like I'm saying it's not worth doing error recovery at all, given that I'm saying the exact opposite.

Re:Bitrot not the fault of filesystem (1)

gweihir (88907) | about 3 months ago | (#47236817)

Actually, bit-rot is the fault of the user that a) selected a non-archival grade storage system for his archiving and b) failed to verify the data was actually written correctly to the archival system.

This is user stupidity, plain and simple. If this person had spent 15 minutes finding out how to properly archive data, he would not have lost anything.

FEC: Par2 and RSbep (0)

Anonymous Coward | about 3 months ago | (#47236845)

The only way to avoid bitrot is with forward error correction such as par2 (pypar2 is a good GUI) and the rsbep.

Re:Bitrot not the fault of filesystem (1)

steelfood (895457) | about 3 months ago | (#47236927)

It's amusing you're putting the blame for imperfection on the known imperfect nature of physical systems.

I'd prefer to blame management of the software product for not pushing for more reliable software, which is by nature supposed to be perfect. Note that in some cases, management and lead developers are basically the same people, but in Apple's case, I'm certain they can afford to hire real managers to make these kinds of important decisions.

Re:Bitrot not the fault of filesystem (0)

Anonymous Coward | about 3 months ago | (#47237103)

No, you don't need parity to detect bitrot, it makes it easier, but is not 100% necessary, as plenty of bitrot occurs in the hardware between the system memory and the platter, and zfs detects/corrects this automatically.

You can also do mirroring, and yes, you "lose" more space, but improve performance greatly, and get the same or better protection than you do with parity.

I won't touch btrfs, mostly because I'm still waiting for linux kernel based operating systems to grow up and work correctly (still way too buggy for the things I do), so I still use Solaris, which, in my not so humble opinion, is still the best UNIX variant available today.

Good backups aren't enough (1)

jmitchel!jmitchel.co (254506) | about 3 months ago | (#47236043)

Good backups aren't enough. If the filesystem isn't flagging corruption as it happens, the backup software will happily back up your corrupted data over and over until the last backup which has the valid file in it has expired or become unrecoverable itself.

Re:Good backups aren't enough (1)

drew_92123 (213321) | about 3 months ago | (#47236097)

I copy all of my non-changing files to a a special directory and have the backup app I use compare them to another copy and alert me of any changes. At any one time I have 3-4 copies of my important files on separate disks PLUS my backups. Cuz fuck losing files! ;-)

Re:Good backups aren't enough (1)

gweihir (88907) | about 3 months ago | (#47236833)

That is complete BS! The disks certainly flag any data that has gone as bad as bad. Undetected read errors do not happen unless the disk electronics is dying. And for that you have redundancy.

Re:Good backups aren't enough (0)

Anonymous Coward | about 3 months ago | (#47237315)

Holly crap, i just have a feeling of deja vu of deja vu of this conversation.

ZFS, Apple! (2)

grub (11606) | about 3 months ago | (#47236047)


This is why Apple should resurrect its ZFS project. Overnight they would be the largest ZFS vendor to match with being the largest UNIX vendor.

Re:ZFS, Apple! (0)

Anonymous Coward | about 3 months ago | (#47236081)

They would also be sued pretty quickly by Oracle. Clearly not an option.

Re:ZFS, Apple! (2)

grahamsaa (1287732) | about 3 months ago | (#47236167)

I'm not sure this is true. Other vendors like iXsystems already sell products that ship with ZFS. As I understand it, ZFS is BSD licensed. While Oracle distributes its own version of ZFS that may (or may not) include proprietary features, the open sourced version is freely distributable. The only reason it's packaged as a userland utility for Linux is that the BSD license isn't compatible with the kernel's GPL license. Apple's kernel is definitely not GPL, so this isn't a problem for them.

One problem might be that using ZFS without ECC memory can result in data loss, and ECC memory is more expensive (and not compatible with most consumer oriented processors that Intel makes). This would increase the cost of Apple hardware and could (possibly) be a hurdle, as Intel doesn't want to support ECC memory on their consumer oriented processors (as this could hurt sales of more expensive server-oriented processors. But Apple is a large enough vendor that they could probably negotiate something with Intel that could be workable.

That said, I don't know many Apple users that know what ZFS is, and it doesn't seem like there are many people clamoring for it. It would be a great addition to OSX though.

Re:ZFS, Apple! (1)

kthreadd (1558445) | about 3 months ago | (#47236205)

ZFS does not require ECC memory more than any other file system. I have no idea where you got that from.

Re:ZFS, Apple! (1)

grahamsaa (1287732) | about 3 months ago | (#47236271)

Of course it doesn't, and I never said that. But your chances of data corruption if you use ZFS without ECC are somewhat greater, and potentially much more catastrophic. A web search for 'ZFS without ECC' will point you to a number of horror stores. Basically, ZFS always trusts what's in memory, so if what's in memory differs from what's on disk, the contents on disk get overwritten. If this discrepancy is due to bit rot, that's great -- you've just saved your data. But if it's due to a memory error, your system proactively corrupts your data. Considering that most non ECC DIMMs have a couple errors a year, you will very likely lose data if you run ZFS on a system without ECC.

Of course, ECC doesn't fix everything, but it should halt your system if your RAM has an uncorrectable error, which is better than corrupting your files on disk.

Re:ZFS, Apple! (1)

kthreadd (1558445) | about 3 months ago | (#47236391)

I see what you mean now, but I must say that I really don't agree with these non-ECC horror stories. You have much bigger problems if you have memory corruption.

Re:ZFS, Apple! (1)

nabsltd (1313397) | about 3 months ago | (#47236633)

You have much bigger problems if you have memory corruption.

If you don't use ECC memory, you will have memory corruption. Even if you do use ECC memory, you might have corruption, and it might even go unnoticed, but the odds are far less likely.

"Corruption" in this sense doesn't mean that whole DIMMs are broken...it just means that one bit has changed in a way that the user/OS/CPU didn't want it to. In many cases, this can be completely harmless (e.g., graphical data used in-memory only has one bit wrong...you might not even notice a color shift if it's the LSB), a little annoying (e.g., unexpected program termination), or very annoying (e.g., BSOD). But, if this happens in memory used for disk write buffers, then you get the issue that the GP had you Google for.

Re:ZFS, Apple! (1)

kthreadd (1558445) | about 3 months ago | (#47236771)

Then it's much simpler. This ECC issue has absolutely nothing to do with ZFS. You should use ECC RAM if you are doing any form of disk IO no matter which file system you're using, or you are under the risk of data loss.

Only idiots use non-ECC RAM (0)

Anonymous Coward | about 3 months ago | (#47238345)

You are correct. All computer memory should be ECC. Do not buy non-ECC RAM as this marks you as reckless and shortsighted.

If your ECC RAM is failing you will see warnings in OS logs while your system still operates correctly and stores your data safely. This is your opportunity to replace the RAM.

If you do not have ECC RAM, then failing memory will corrupt an unknown amount of data before you work out that your problems are RAM related.

Manufacturers of non-ECC RAM should be sued. Their products are not "fit for purpose".

Re:ZFS, Apple! (1)

Jane Q. Public (1010737) | about 3 months ago | (#47238723)

Mod this one up. If memory is corrupted, disk can become "corrupted", but only because it's a copy of the actual contents of memory.

Fault-tolerant memory and fault-tolerant file systems may have similarities but they are separate issues. If either one becomes corrupted it can "corrupt" the other, but only because one is a copy of the other.

Re:ZFS, Apple! (1)

nabsltd (1313397) | about 3 months ago | (#47253185)

You should use ECC RAM if you are doing any form of disk IO no matter which file system you're using, or you are under the risk of data loss.

I agree. Unfortunately, no Intel desktop CPU/chipset supports ECC, and many AMD desktop chipsets are castrated by the board manufacturer to not allow ECC.

When RAM was slow and 4GB was huge and expensive, this wasn't as big a deal, but now that 8GB is the reasonable starting point and 32GB is quite affordable, Intel especially needs to step up and add ECC support to their desktop CPUs/chipsets.

Re:ZFS, Apple! (0)

Anonymous Coward | about 3 months ago | (#47236677)

Indeed. I had a faulty memory chip installed. Alone itself it didn't fail. The previous one didn't fail either. But when both were working at the same time it would freak out. Manifested itself as static in mp3 files being played. Quite noticeable.

Re:ZFS, Apple! (2)

Kaenneth (82978) | about 3 months ago | (#47237529)

Back when I did tech support for a lightweight Mac database product, they didn't use Parity (much less ECC) RAM.

I had a customer call in because students were continually getting corrupted databases on their assignments.

over the course of several phone calls, we narrowed it down to only happening in 1 of 3 labs.

After excluding anything high-energy (like a physics lab) in the building, I got the customer to reveal that they were constructing a new building next door, and the construction power tools were running off the same circuits as the computer lab...

They got the construction workers to use a different source of power, and the corruption problems disappeared.

Re:ZFS, Apple! (1)

sjames (1099) | about 3 months ago | (#47237837)

All file systems trust what's in memory (they have to, anything that would test it lives in memory too!). So the point about ECC RAM and ZFS is that since it is doing it's part to prevent bitrot (unlike most file systems), adding ECC RAM to the picture makes bitrot practically non-existent.

If a block has just been fetched from disk and the checksum fails, it will not write that back to disk, since it knows the data is bad. If the buffer for a write is corrupt, no filesystem can know that.

Re: ZFS, Apple! (0)

Anonymous Coward | about 3 months ago | (#47236171)

Oracle and Apple are allies. Apple is a huge Oracle customer. Steve Jobs and Larry Ellison were BFFs.

Re: ZFS, Apple! (0)

Anonymous Coward | about 3 months ago | (#47236215)

And how exactly did that end up for Steve Jobs?

Re:ZFS, Apple! (2)

Calibax (151875) | about 3 months ago | (#47236213)

No they would not be sued by anyone.

Sun open sourced ZFS under a permissive license. Oracle close sourced it again. However, a number of companies are supporting derivatives of the open source version.

ZFS is available for a number of operating systems today. A non-inclusive list:
FreeBSD from iXsystems
Linux from Lawrence Livermore National Laboratory and also Pogo Linux
SmartOS from Joyent
OmniOS from Omniti
Osv from CloudOS

In addition a number of companies are using ZFS in their products:
CloudScaling
DDRdrive
datto
Delphix
GE Healthcare
Great Lakes SAN
Losytec
High-Availability
HybridCluster
Nexenta Systems
OSNEXUS
RackTop
Spectra Logic
Storiant
Syneto
WHEEL Systems
Zetavault

ZFS can detect and correct silent corruption when configured to do so. I have a NAS that has 24 TB of raw storage, 16 TB of useable storage, running under OmniOS. I have well over 10 million files on the NAS (it is used as a backup for 8 systems) - I haven't lost a file in 4 years and I don't expect to lose any.

Re:ZFS, Apple! (0)

Anonymous Coward | about 3 months ago | (#47236237)

You forget that there's this thing called patents. None of those no-names on the list is really worth suing for patent infringement, but Apple is.

Re:ZFS, Apple! (1)

Anonymous Coward | about 3 months ago | (#47236273)

NetApp sued Sun over ZFS saying ZFS infringed their patents. Sun (later Oracle) countersued. Both suits were settled without any money flowing either way.

ZFS is considered safe to use without threat of legal action.

Do you seriously think that dozens of companies would use it in their businesses if there was a risk of being sued out of existence by Oracle?

Re:ZFS, Apple! (1)

Calibax (151875) | about 3 months ago | (#47236301)

I would hesitate to call GE Healthcare a small company. I doubt that Lawrence Livermore National Labs would be considered small as it's part of the government. Joyent is the company that supports node.js.

Anyone can sue anybody about anything, but winning is different matter. ZFS is considered safe from a legal point of view.

Re:ZFS, Apple! (0)

Anonymous Coward | about 3 months ago | (#47237065)

GE Healthcare employes over 46,000 people.

Nexenta Systems claims many petabytes under management.

CloudScaling built the ZFS-backed cloud for KT (Korea Telecom), a very large, once-state-run, multi-billion dollar telecommunications company.

And LLNL is a federal research facility.

All of these represent strategic targets for lawsuit, either from a pure money perspective (such as KT), or from a strategic perspective (such as Nexenta Systems, or on a completely different front, LLNL).

Many of these have their own patent portfolios. It is not a foregone conclusion that NetApp would win such a lawsuit. It wasn't winning the lawsuit against Sun, before the Oracle buy made it go away anyway.

Re:ZFS, Apple! (1)

sribe (304414) | about 3 months ago | (#47236275)

Sun open sourced ZFS under a permissive license.

And NetApp claims that Sun & Oracle violated their patents.

Re:ZFS, Apple! (0)

Anonymous Coward | about 3 months ago | (#47236317)

But NetApp's claim was settled without any money changing hands.

Re:ZFS, Apple! (1)

fnj (64210) | about 3 months ago | (#47236601)

They would also be sued pretty quickly by Oracle. Clearly not an option.

Your conclusion is a bit hasty and unwarranted. I am not going to tell you that Oracle CANNOT sue anyone for any trumped-up reason, but ZFS is licensed under the Common Development and Distribution License (CDDL) and is open source [rtt-law.com] . For linux, there is an issue with how CDDL plays with GPL, so no distro has yet bundled ZFS with linux. Linux users, however, can themselves pick up "ZFS on Linux" [zfsonlinux.org] and install it themselves without violating either the CDDL or GPL.

But OSX is not GPL. Other systems that are not GPL bundle ZFS, and are not sued. For example, FreeBSD comes with ZFS, and there are a number of other systems, such as FreeNAS, PS-BSD, illumos and nexenta.

See OpenZFS [open-zfs.org] .

Re:ZFS, Apple! (1)

ColdWetDog (752185) | about 3 months ago | (#47236111)

I'm curious why it's been ignored or deprecated or whatever Apple did to it. They have the resources to throw at a project like that. Presumably there was some calculation somewhere along the line that didn't make sense. Not that Apple is much for telling us things like that, but it would be fun to know.

Re:ZFS, Apple! (0)

Anonymous Coward | about 3 months ago | (#47236145)

I think there were licensing issues (http://gizmodo.com/5389520/licensing-issues-at-heart-of-apples-decision-to-kill-snow-leopard-zfs-plans).

Re:ZFS, Apple! (1)

kthreadd (1558445) | about 3 months ago | (#47236181)

So how is FreeBSD able to license ZFS by simply importing it into the source tree and Apple is not?

Re:ZFS, Apple! (0)

Anonymous Coward | about 3 months ago | (#47236491)

Because BSD and (essentially) any license play nice together. The same cannot be said about the proprietary license Apple uses on its code, and the CDDL (ZFS license).

Re:ZFS, Apple! (0)

Anonymous Coward | about 3 months ago | (#47240677)

It's not a matter of licensing. As the GP said, "[t]hings like pulling an external drive out during mid write could corrupt an entire ZFS volume." This is a very common risk for Mac users; especially owners of MacBook Airs with limited SSD storage capacity. FreeBSD is expect to run on workstations and servers with user/admins who are rightfully paranoid about improper unmounts and know how to recover from them if they do happen for uncontrollable reasons, and know that recovery is going to take a lot of time. That's not acceptable for "the Mac experience."

ZFS is wonderful, and I'd love to see ZFS or BtrFS on OS X, but if an unclean unmount can trash a partition then it's a really bad fit for Mac. It already confuses Windows users that you need to manually unmount external volumes of OS X. Corrupting a volume because you forgot to do so would inspire a KILL-IT-WITH-FIRE-type reaction in new Mac users.

Re:ZFS, Apple! (0)

Anonymous Coward | about 3 months ago | (#47236365)

Don Brady, the engineer who championed the use of ZFS at Apple, left the company. Nobody else was interested in heading up the project so it was cancelled. At the time there was also a threat of ZFS infringing on NetApp patents which also played into the decision. That's not an issue now.

Re:ZFS, Apple! (2)

jbolden (176878) | about 3 months ago | (#47237651)

Apple did announce why the project failed. ZFS on consumer grade hardware with consumer interactions was too dangerous. Things like pulling an external drive out during mid write could corrupt an entire ZFS volume. Apple simply couldn't get ZFS to work under the conditions their systems need it to. They had to backout completely and come up with a plan-B. The developer who worked on this left Apple and now produces a better ZFS for OSX. That company got bought by Oracle so Oracle owns it now.

Re:ZFS, Apple! (0)

Anonymous Coward | about 3 months ago | (#47240003)

Citation please.

I'm pretty certain there was no such announcement because it is not true that ZFS is dangerous in any context. With the built-in protections, ZFS works as well on consumer grade hardware as enterprise grade. In fact, consumer grade drives are preferred because of cost.

Re:ZFS, Apple! (1)

laird (2705) | about 3 months ago | (#47238501)

ZFS was a Sun project, and they've effectively killed it. Apple might have been looking at ZFS, but they never made it a part of their OS.

Shame, as it's a really nice filesystem. My previous file server was ZFS, and it was a delight. But it's kinda picky about what hardware it'll run on, and the old file server (dual Xeon) was just too power hungry to keep running at home...

Re:ZFS, Apple! (0)

Anonymous Coward | about 3 months ago | (#47242659)

ZFS is alive and well and continues to be developed by Oracle. It is used in several Oracle products and also internally. However, the Oracle version is closed source.

Fortunately, Sun open sourced it under the CDDL license and that version is being enhanced by the OpenZFS consortium. At present ZFS is available for FreeBSD, Linux and OSX.

Isn't Samsung the largest UNIX vendor? *grin* (1, Informative)

sirwired (27582) | about 3 months ago | (#47236127)

Due to their commanding smartphone marketshare, along with millions of devices with embedded Linux shipped every year, wouldn't Samsung be the largest UNIX vendor?

Oh? What's that? You weren't counting embedded Linux and I'm a pedantic #$(*#$&@!!!. Can't argue with that!

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

jo_ham (604554) | about 3 months ago | (#47236151)

Now there's a can of worms. I think the question "Is Linux really Unix?" is a guaranteed heat-generator.

Re:Isn't Samsung the largest UNIX vendor? *grin* (0)

Anonymous Coward | about 3 months ago | (#47236209)

Close enough

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

Antique Geekmeister (740220) | about 3 months ago | (#47236419)

If you follow the specifications, there's no need for heat. No Linux variant has been certified according to the POSIX standards for UNIX, and most variants have subtle ways in which they diverge from the POSIX standards, at least subtly. Wikipedia has a good note on this at http://en.wikipedia.org/wiki/S... [wikipedia.org]

Personally, I've found each UNIX to each have some rather strange distinctions from the other UNIX's, and using the GNU software base and the Linux based software packages to assure compatibility among the different UNIX variants.

                         

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

Bing Tsher E (943915) | about 3 months ago | (#47237003)

UNIX is a brand name, and POSIX is a standard to meet. Just like an appliance might be UL Certified, an OS might be POSIX compliant.

The trademark of the UNIX brand name is owned by a whole separate group.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

jbolden (176878) | about 3 months ago | (#47237655)

Given Linux's intellectual and usage dominance I'd say that the old Open Systems approach clearly no longer works. A standard that excludes Linux is not a standard. So I'm coming down that POSIX / Open Group should not be the definition of UNIX.

Re:Isn't Samsung the largest UNIX vendor? *grin* (0)

jo_ham (604554) | about 3 months ago | (#47237763)

Given Linux's intellectual and usage dominance I'd say that the old Open Systems approach clearly no longer works. A standard that excludes Linux is not a standard. So I'm coming down that POSIX / Open Group should not be the definition of UNIX.

Just because you wish it really hard doesn't make it so.

Like it or not, "UNIX" has a specific meaning, both in terms of branding and adhering to a defined standard. You can't just decide to claim that standard as your own even if you don't meet it.

"Any standard that excludes Linux is not a standard" is just an absurdly arrogant and silly thing to say.

Since you're totally ok with claiming that the Open Group no longer gets to define what UNIX is because you say so, I'll also take claim of what GNU means, since any standard that excludes Apple and Microsoft is not a standard. OS X and Windows will now be called GNU/OS X and GNU Windows. Seems fair to me.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

jbolden (176878) | about 3 months ago | (#47238543)

UNIX" has a specific meaning, both in terms of branding and adhering to a defined standard.

I disagree. I don't think UNIX is a brand. I think it is a cultural movement that led to a variety of products of which the Open Software movement of the 1990s was a part.

since any standard that excludes Apple and Microsoft is not a standard.

I'd agree with that providing you mean "personal computing standard" or "desktop standard" or whatever. Yes, absolutely. That was precisely my position with Internet Explorer any standard Microsoft didn't buy into isn't a standard.

Re:Isn't Samsung the largest UNIX vendor? *grin* (0)

jo_ham (604554) | about 3 months ago | (#47238585)

UNIX is literally a brand. Like I said, you can't just wish really hard that it's not. It is a registered trademark of the Open Group.

You can disagree as much as you like, but it doesn't alter actual, verifiable facts.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

jbolden (176878) | about 3 months ago | (#47238615)

UNIX is a registered trademark. UNIX as an entity pre-existed that trademark. UNIX is used as a word in ways that aren't associated with the Open Group. They've been fighting real hard to assert that UNIX isn't a generic term because if it is a generic term they lose their trademark. The fact that they've had to fight its use as a generic term is something even the Open Group agrees to. It is you who is ignoring verifiable facts.

Re:Isn't Samsung the largest UNIX vendor? *grin* (0)

jo_ham (604554) | about 3 months ago | (#47239551)

UNIX is a registered trademark.

That's pretty much thread over right there. Again, you can complain and scrunch your eyes up and wish real hard, but you're trying to twist facts to suit your argument. Remember, this started with you saying that "any standard that doesn't include Linux is not a standard" in a laughably arrogant and stupid statement, and now you're trying to claim Unix for your own to support your silly statement.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

sl149q (1537343) | about 3 months ago | (#47237861)

Is it more important that Linux be considered to be POSIX or for POSIX to figure out how to accomodate LInux?

POSIX and LINUX (1)

jbolden (176878) | about 3 months ago | (#47238553)

Well IMHO I'd say more important for The Open Group (POSIX) to figure out what role they should play in a world where we don't have a variety of mostly coequal Unixes. A rather a highly fragmented family of Unixes in Linux including Android, a very popular desktop Unix that violates most of the Unix norms in spirit in OSX, and the only remaining big box Unix (AIX) is more aimed at bring over cool features from mainframe. None of them really care about running each other's software. So really the question is what role should the The Open Group play in such a world?

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

Jane Q. Public (1010737) | about 3 months ago | (#47238653)

Given Linux's intellectual and usage dominance I'd say that the old Open Systems approach clearly no longer works. A standard that excludes Linux is not a standard. So I'm coming down that POSIX / Open Group should not be the definition of UNIX.

Yes, but you should clarify that "Open Groups" is the name of a group, and doesn't come close to representing all of Open Source or Open Software. You know that, I know that, but not everybody knows that.

Having said that, I think we are basically agreed. The existing POSIX should be re-labeled just POS, and we should all just move on. I am all for open technical standards, but if they aren't changing with the times, then the times will move along without them.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

jbolden (176878) | about 3 months ago | (#47239165)

Glad you agree on the main point.

I don't know that the Open Group even claims to represent Open Source at all. They represented Open Standards which was an earlier movement about interoperability between commercial vendors. They claim to represent interests of "customers" not "users" as per Open Source. They claim to work with various "suppliers" not "developers", etc...

Re:Isn't Samsung the largest UNIX vendor? *grin* (0)

Anonymous Coward | about 3 months ago | (#47239925)

A standard that excludes Linux is not a standard. So I'm coming down that POSIX / Open Group should not be the definition of UNIX.

You're going at it backwards. Open Group can't exactly say "this isn't the standard anymore, because Linux doesn't meet it." They're not going to change the definition of "UNIX" based on what you or anyone else thinks it should be.

No, their job is to define the standard and it's Linux's job to meet it if they decide to. There no way to do it the other way around.

Besides, there's already an adequate term covering the broader definition you're talking about for the user-centered movement, and systems compatible with, or inspired by UNIX - they're called "UNIX-like."

I wouldn't worry too much about whether Linux or anything else is "real UNIX," though - almost nothing is anymore, ever since AT&T sold the dang thing and it's changed hands and mutated ever since.

Re:Isn't Samsung the largest UNIX vendor? *grin* (0)

Anonymous Coward | about 3 months ago | (#47247927)

A standard that excludes Linux is not a standard. So I'm coming down that POSIX / Open Group should not be the definition of UNIX.

Unlike Unix, the Linux kernel's API is changed on a regular basis as a political statement and to encourage manufactures to add their code to the source tree. I wouldn't call an unstable API a standard in any sense of the word.

Re:Isn't Samsung the largest UNIX vendor? *grin* (0)

Anonymous Coward | about 3 months ago | (#47251837)

Unlike Unix, the Linux kernel's API is changed on a regular basis as a political statement and to encourage manufactures to add their code to the source tree.

This seems to me like evidence that Linus was wrong in his argument with Tanenbaum over modular vs. monolithic.

Stability and consistency are good design features. If Linux's kernel design is better, then why are they constantly changing the kernel??

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

sjames (1099) | about 3 months ago | (#47237733)

The heat comes from the reason it isn't certified. Due to the costs involved, nobody has tried it. Same deal for *BSD.

When most people ask if it is Unix, they mean would a person familiar with Unix have any problems with tricks and traps or not. That is a much more subtle question, but also a much more important one.

I have used and admined several and find that in spite of certification, each has it's own quirks. Linux and BSD don't seem to have any more than the others. So while calling Linux or *BSD Unix might violate a trademark, they are, for practical purposes, Unix systems in all but name.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

Jane Q. Public (1010737) | about 3 months ago | (#47238673)

When most people ask if it is Unix, they mean would a person familiar with Unix have any problems with tricks and traps or not. That is a much more subtle question, but also a much more important one.

I think a better question would be: can I take my code for X, natively compile it on Y, and expect it to run?

Although it is built on BSD, the majority of C code written for Linux will compile and run in OS X. At least, just about everything I've tried has. I haven't tried to do FPS games or anything. But "work" code, sure. Despite differences here and there, it's a *nix system and works like one. Even X11 stuff runs fine (like Gimp for example), even though OS X has its own proprietary GUI.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

Jane Q. Public (1010737) | about 3 months ago | (#47238635)

No Linux variant has been certified according to the POSIX standards for UNIX, and most variants have subtle ways in which they diverge from the POSIX standards, at least subtly.

Haha. Now you've opened a completely different can of worms. For just one example: why should POSIX matter much these days?

BSD, for example, can essentially (though perhaps not completely technically) be called "Linux with extensions". (They deny it but their own description [freebsd.org] pretty much gives no technical differences except to say that Linux binaries won't run... and without further explanation that could simply be a compiler dead-man. The only specific difference they point out is licensing.) And the only real reason BSD isn't POSIX-compliant is because they have no interest in paying the fees.

Take OS X for example... it's built on BSD yet it IS POSIX-compliant. Because they wanted to be and paid the cert fees. Big deal.

If OS X and even Windows can be made POSIX compliant (they can [stackoverflow.com] ), then just about anything could be made POSIX compliant if the owners wanted to bother. They just don't want to.

So now that the waters are thoroughly muddied, I'll muddy them further by saying: today, if you're not an Enterprise shop... you should ask yourself whether you really have any reason to give a shit.

Re:Isn't Samsung the largest UNIX vendor? *grin* (0)

Anonymous Coward | about 3 months ago | (#47247893)

BSD, for example, can essentially (though perhaps not completely technically) be called "Linux with extensions".

Wouldn't it be more accurate to call Linux a subset copy of BSD?

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

Bing Tsher E (943915) | about 3 months ago | (#47236995)

But is Mach UNIX? I don't mean 'POSIX compliant' because Windows NT 4.0 is POSIX compliant.

I have several UNIX license plates. They are officially licensed and sold (or were sold) by The Open Group.

Saying that Apple pimps off UNIX to produce their closed-source candyland binaries isn't really 'The UNIX Way' no matter how much it's one of Apple's new 'Altivec Unit' bullshit bullet points.

Re:Isn't Samsung the largest UNIX vendor? *grin* (0)

Anonymous Coward | about 3 months ago | (#47237645)

> But is Mach UNIX?

It was at the time Steve Jobs started NeXT. They even paid AT&T for a Unix license, which was required to use Mach:

https://www.cs.cmu.edu/afs/cs/project/mach/public/FAQ/license.info [cmu.edu]

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

jo_ham (604554) | about 3 months ago | (#47237779)

It is - they bought a UNIX licence back in the NeXT days I believe.

OS X is POSIX compliant and ostensibly the core pieces of NeXT.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

Guy Harris (3803) | about 3 months ago | (#47237925)

But is Mach UNIX? I don't mean 'POSIX compliant' because Windows NT 4.0 is POSIX compliant.

If Mach is "the Mach kernel", I don't think it offers UNIX APIs, but at least two OSes based on Mach have passed the Single UNIX Specification test suite (which NT 4.0 hasn't, and which even Interix^Wthe Subsystem for Unix-based Applications hasn't).

Re:Isn't Samsung the largest UNIX vendor? *grin* (0)

Anonymous Coward | about 3 months ago | (#47240045)

But is Mach UNIX? I don't mean 'POSIX compliant' because Windows NT 4.0 is POSIX compliant.

That's a good question, since NeXSTEP was based on Mach and it was a licensed UNIX variant. Also Mac OS X which actually doesn't use a Mach microkernel, but incorporates Mach code into its kernel, is a certified UNIX.

I think the practical answer is, "it's as much UNIX as any other BSD variant is," which kind of means "it used to be UNIX before they removed all AT&T code," turning what was a modified AT&T UNIX into a UNIX compatible clone... confusing? Yes!

Mach was an operating system as well as a kernel, since it was created as an experiment to see if they could replace the BSD kernel with a microkernel, separating concerns between simplified kernel abstractions and a more domain-specific compatibility layer.

AFAIK Carnegie-Mellon never created any other environments for Mach besides the BSD-compatible one, but others proved it possible by using Mach in other projects. Mac OS X is the only surviving one that uses any Mach code, I believe.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

Bill_the_Engineer (772575) | about 3 months ago | (#47244811)

I think the actual question being "Is Android on Linux really Unix?" should cause very little heat since most people wouldn't care or think "No" is pretty obvious.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

jo_ham (604554) | about 3 months ago | (#47247787)

I think the actual question being "Is Android on Linux really Unix?" should cause very little heat since most people wouldn't care or think "No" is pretty obvious.

Tell that to the guy in a parallel thread to this one who thinks that because "any standard that doesn't include Linux is not a standard" and that because he doesn't think that the Open Group deserves to control what is and isn't Unix that he is taking the term for himself and declaring that it means whatever he wants it to mean.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

Bill_the_Engineer (772575) | about 3 months ago | (#47247947)

Well some folks are fanatics.

Re:Isn't Samsung the largest UNIX vendor? *grin* (0)

BasilBrush (643681) | about 3 months ago | (#47236825)

You do know that Linux stands for "Linux Is Not UniX, don't you?

I'm afraid you weren't nearly pedantic enough.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

Bing Tsher E (943915) | about 3 months ago | (#47237017)

Linux stands for 'we didn't allow Linus to call it Freax.' Nothing more.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

jbolden (176878) | about 3 months ago | (#47237659)

Not it doesn't. LINUX stands for "Linus' Minix".

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

Jane Q. Public (1010737) | about 3 months ago | (#47238697)

Not it doesn't. LINUX stands for "Linus' Minix".

Unfortunately the etymology of the word is lost in obscurity. Even Linus' own word on the matter is no longer trustworthy.

The fact that around the same time, self-referencing acronyms ("Gnu is Not Unix", for example) were very popular and it is likely you won't convince many people even if you're right, unless you have an unimpeachable source.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

jones_supa (887896) | about 3 months ago | (#47239045)

Lost in the obscurity? Just to get the facts straight, back in the day Linux was initially called "Freax". Ari Lemmke provided some FTP space for Linus and he tongue-in-cheek created a directory called "linux". There is nothing ambiguous about the background of the name.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

Jane Q. Public (1010737) | about 3 months ago | (#47239093)

I don't say this often, because this is /., not Wikipedia. But for the same reason I mentioned before: [citation needed].

Everything you say may be true. But it contradicts many things other people have been saying, for a very long time.

So an assertion that "it is called that because" needs some evidence. I didn't make a claim, GP did. I didn't say anybody was lying, I just said other people (for a very long time now) have said other things.

So get off MY butt and produce evidence, or shut up.

Re:Isn't Samsung the largest UNIX vendor? *grin* (0)

Anonymous Coward | about 3 months ago | (#47239149)

I don't say this often, because this is /., not Wikipedia. But for the same reason I mentioned before: [citation needed].

How about Just for Fun [wikipedia.org] by Linus Torvalds and David Diamond?

Re:Isn't Samsung the largest UNIX vendor? *grin* (0)

Anonymous Coward | about 3 months ago | (#47240145)

From Rebel Code [google.com] :

"[Lemmke] had this small area on ftp.funet.fi ... and he said that 'hey, I'm putting a directory aside for you.' So he created the /pub/os/linux directory," Linus recalls

"Linux was my working name," Linus continues, "so in that sense he didn't really name it, but I never wanted to release it as Linux." He says he was afraid that "if I actually used it as the official one people would think that I am egomaniac, and wouldn't take it seriously."

"I chose this very bad name: Freax -- free + freak + x. Sick, I know," Linus acknowledges. "Luckily, this Ari Lemmke didn't like it at all, so he used this working name instead. And after that he never changed it."

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

jbolden (176878) | about 3 months ago | (#47239173)

Small correction. Ari created a directory called "LINIX", the more unixy "LINUX" was one more stage.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

jones_supa (887896) | about 3 months ago | (#47239177)

Ah, interesting. Didn't know that part. :)

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

jbolden (176878) | about 3 months ago | (#47243071)

Yep. It makes more sense for Linus' Minix = LINIX. I've often wondered if the pronunciation issue "Lin-ix" vs. "Lin-ux" comes from LINIX / LINUX name that even though LINIX didn't last long it lasted long enough to get into the oral culture.

Re:Isn't Samsung the largest UNIX vendor? *grin* (0)

Anonymous Coward | about 3 months ago | (#47239229)

Small correction. Ari created a directory called "LINIX", the more unixy "LINUX" was one more stage.

I'm calling bullshit on that one. AFAIK it was never "Linix"

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

BasilBrush (643681) | about 3 months ago | (#47241941)

More Unixy? LINIX is closer to UNIX than LINUX.

Back in the 70s there used to be a UK porn magazine called "FLICK. It was called that because if you pick the font and kerning right, that L and I begin to look like a U.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

jbolden (176878) | about 3 months ago | (#47243077)

LINIX = Linus' Minix
Linux = a Unix variant (the letters for Unix are in there)
 

No, it doesn't. (1)

sirwired (27582) | about 3 months ago | (#47238453)

GNU stands for GNU's Not UNIX.

Linux is just "Linus with an 'x' at the end to make the name look UNIX-y"

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

Karlt1 (231423) | about 3 months ago | (#47237081)

Due to their commanding smartphone marketshare, along with millions of devices with embedded Linux shipped every year, wouldn't Samsung be the largest UNIX vendor?

Oh? What's that? You weren't counting embedded Linux and I'm a pedantic #$(*#$&@!!!. Can't argue with that!

Mac OS X is Unix -- it's been certified as Unix by the group that holds the copyright to the term. Every version of OS X from 10.5 - 10.9 except for 10.7 has been certified unix.

http://www.opengroup.org/openb... [opengroup.org]

Linux is Unix like.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

Jane Q. Public (1010737) | about 3 months ago | (#47238715)

Mac OS X is Unix -- it's been certified as Unix by the group that holds the copyright to the term. Every version of OS X from 10.5 - 10.9 except for 10.7 has been certified unix.

No. OS X meets the "Open Group" standard called POSIX, which means it is "sufficiently" Unix-like... to meet that standard.

That is all it means. It doesn't mean "OS X is Unix". If anything, it is more like Linux than Unix, but it isn't quite either one.

There are versions of Linux that could also be POSIX-compliant if they wanted to make a minor tweak or two, and pay the certification fees. They don't want to bother. It's that simple.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

Karlt1 (231423) | about 3 months ago | (#47238803)

No. OS X meets the "Open Group" standard called POSIX, which means it is "sufficiently" Unix-like... to meet that standard.

That is all it means. It doesn't mean "OS X is Unix". If anything, it is more like Linux than Unix, but it isn't quite either one.

No. Did you read the link? The Open Group certified OS X as meeting all of the requirements to be certified as " Unix".

POSIX compliance is different.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

Jane Q. Public (1010737) | about 3 months ago | (#47238831)

No. Did you read the link? The Open Group certified OS X as meeting all of the requirements to be certified as " Unix".

POSIX compliance is different.

NO, it isn't! POSIX *IS* the "Single Unix Specification"! They are the same things!

POSIX certification does NOT mean the OS "is Unix"!!!

Re:Isn't Samsung the largest UNIX vendor? *grin* (0)

Anonymous Coward | about 3 months ago | (#47240243)

It's more complicated than that. SCO was the last entity that really "sold UNIX," so there isn't really any UNIX anymore-- there are only products that Open Group certifies to use the UNIX trademark.

Also, SUS isn't the same as POSIX, but the core includes what POSIX used to be. POSIX compliance (or what it is today) isn't sufficient to pass SUS.

Re:Isn't Samsung the largest UNIX vendor? *grin* (1)

grub (11606) | about 3 months ago | (#47237905)

Linux isn't UNIX. Apple is the largest UNIX vendor in the world.

Re:Isn't Samsung the largest UNIX vendor? *grin* (0)

Anonymous Coward | about 3 months ago | (#47244791)

Embedded Linux technically doesn't count as Unix. It would be more accurate to call it a bootstrap and hardware abstraction layer, since the Dalvik based OS is actually doing all the interactions.

Besides the obvious of not being certified, you have the fact that most smartphones do not have all of the user land tools specified installed by default.

Re:ZFS, Apple! (0)

Anonymous Coward | about 3 months ago | (#47238185)

There is a currently maintained implementation of ifs available for OSX at o3x.org.

Under heavy development, but the current public release works really well.

-ac

Re:ZFS, Apple! (0)

Anonymous Coward | about 3 months ago | (#47238197)

Damn autocorrect - ZFS!

28 files in 6 years is a hardware defect (1)

Anonymous Coward | about 3 months ago | (#47236085)

Sure, a modern filesystem should be designed to catch and possibly work around bit errors, but in the end, hardware which causes that many bit errors is defective and needs to be fixed or replaced. RAM would be my first suspect if there aren't any error messages in SMART or disk related entries in system logs. If the RAM is defective, can you really blame the filesystem? What if the files got corrupted in RAM while you were working on them?

Re:28 files in 6 years is a hardware defect (1)

kthreadd (1558445) | about 3 months ago | (#47236093)

How could the RAM be responsible for damaging a file between the time it was written to disk and when it was read from disk?

Re:28 files in 6 years is a hardware defect (1)

Qzukk (229616) | about 3 months ago | (#47236175)

The RAM is responsible for damaging the file while it sits in a buffer waiting to be written to the disk in the first place.

Re:28 files in 6 years is a hardware defect (1)

washu_k (1628007) | about 3 months ago | (#47236177)

Bad RAM could have corrupted the file as it was being written to disk. The file is corrupted all along, but not the disk/filesystem's fault

Or the file could have been corrupted in RAM on read, and would actually be fine if read on a working machine.

Or the disk has been replaced in those 6 years and the file was corrupted during the copy because of bad RAM

There are lots of possibilities for the file to get corrupted that don't involve the disk or filesystem.

Re:28 files in 6 years is a hardware defect (1)

v1 (525388) | about 3 months ago | (#47236185)

I see bad RAM cause two problems. First, when you are copying a file or editing it, and it gets saved, if the data was corrupted while it was in memory, it can become damaged when writing it. It doesn't have to affect the part of the file you were working with. If you were adding to the end of a long text document, page 2 could get damaged when you hit Save.

Second problem, more common in my experience, is directory corruption due to bad ram. When a machine would come in with a trashed directory, we used to just fix it and return it. But sometimes they'd come back again in a similar state. I'd run a memory test and find/replace a bad stick before repairing it again. Later I just got in the habit of running a short ram test anytime there was unusual directory damage. I found it in about 1 in 10 of the cases I checked. Those checks were only run in cases of severe or unusual damage though. Directory damage takes out files wholesale, and can affect data that never entered the computer, and not due to any hardware failure in the storage.

For the record, I manage over 20tb of data here, and to date I've lost two files. One was a blonde moment with RM on a file that wasn't backed up. (I had NO idea that RM followed symlinks!) The other was a failed slice in a mirror that cost me a singe document. That's over a span of over 20 years. If you've lost over 20 files in the last 10 years, you're doing something (or more probably several somethings) wrong.

Re:28 files in 6 years is a hardware defect (1)

Antique Geekmeister (740220) | about 3 months ago | (#47236435)

"rm" doesn't follow symlinks. However, if you have a symlink that is a directory, and hit "tab" to complete the link's name, it will put a dangling "/" on the link name. _That_ is referencing the directory from effectively "inside" the actual target directory.

I've had several conversations with colleagues over why just hitting 'tab for completion' can be hazardous. This is one of the particular cases.

Re:28 files in 6 years is a hardware defect (1)

kthreadd (1558445) | about 3 months ago | (#47236449)

That depends on your shell. Bash works that way, but zsh does not; at least not by default as far as I know.

Re:28 files in 6 years is a hardware defect (0)

Anonymous Coward | about 3 months ago | (#47236193)

The bit errors are in the live set. He kept the roughly 100GB of pictures on his computer, where over the years they most likely got moved from disk to disk or partition to partition, possibly even from a previous computer to the current one (his photos date from as far as eight years back). He doesn't really say. The chance of at least 28 bit errors in a set of 100GB going undetected by the error correction inside a hard disk is nil. The disk would have thrown plenty of read errors instead of handing over faulty data. Hard disks specify uncorrectable bit error rates of about 1 per 10TB, and the error correction is designed to still detect almost all of these errors. That leaves the connection to the computer or the computer itself as the source of the bit rot. From experience I can say that RAM isn't nearly as reliable as people would hope it to be.

Re:28 files in 6 years is a hardware defect (1)

gweihir (88907) | about 3 months ago | (#47236861)

This is not a filesystem-layer issue at all. And I agree, if the disk did not detect defective sectors, then this was not bit-rot on them. While the rate for uncorrectable sectors stated by disk manufacturers is something like 1 in 10^15, the rate for undetected bad sectors is so low it will not happen, unless the disk electronics that calculates the checksums is dying.

And people that do not verify the files they put into an archive have not understood the first thing about archiving data or making backups. Yes, stupidity will cause you to lose data. It is not a technology problem, it is a problem of people making assumptions without verifying them.

Detachment (1)

PopeRatzo (965947) | about 3 months ago | (#47236117)

The solution is to not become too attached to data. It's all ephemeral anyway, in the grand scheme of things.

Re:Detachment (1)

grasshoppa (657393) | about 3 months ago | (#47236149)

Well, in the "grand scheme of things", so are we.

Me? I get rather attached to the source file I've been working on for the past 6 months.

Re:Detachment (1)

PopeRatzo (965947) | about 3 months ago | (#47238891)

You'll never find enlightenment as long as you remain invested in earthly things such as "source files".

Learn to let go, grasshoppa. This is the way to enlightenment.

Now go practice your kung fu and let me finish this polish sausage.

Re:Detachment (0)

Anonymous Coward | about 3 months ago | (#47303607)

Re:Detachment (2)

NormalVisual (565491) | about 3 months ago | (#47236161)

Yeah, tell that to the IRS when you go to pull your records during an audit... ;-)

Re:Detachment (1)

Anonymous Coward | about 3 months ago | (#47236231)

Tell them you stored all of your records in e-mail messages. They'll understand the loss.

Re:Detachment (1)

93 Escort Wagon (326346) | about 3 months ago | (#47237279)

Tell them you stored all of your records in e-mail messages. They'll understand the loss.

What loss? They can ask the NSA for their copies.

Re:Detachment (0)

Anonymous Coward | about 3 months ago | (#47238659)

And the Republicans have arranged it so you will be put in prison if you can't produce your records indefinitely. They know it is ridiculous, but since it will screw over minorities because those hateful old white men force us to move more often, they support it. That is the way of their kind.

Re:Detachment (0)

Anonymous Coward | about 3 months ago | (#47303655)

So far what I lost... (4, Interesting)

cpct0 (558171) | about 3 months ago | (#47236133)

Bitrot is not usually the issue for most files. Sometimes, but it's rare. What I lost is a mayhem repository of hardware and software and human failure. Thanks for backup, life :)

On Bitrot:

- MP3s and M4As I had that suddenly started to stutter and jump around. You play the music and it starts to skip. Luckily I have backups (read on for why I have multiple backups of everything :) ) so when I find them, I just revert to the backup.
- Images having bad sectors like everyone else. Once or twice here or there.

- A few CDs due to CD degradation. That includes one that I really wish I'd still have, as it was a backup of something I lost. However, the CD takes hours to read, and then eventually either balks up or not for the directory. I won't tell you about actually trying to copy the files, especially with normal timeouts in modern OSes or the hardware pieces or whatnot.

Not Bitrot:

- Two RAID Mirror hard drives, as they were both the same company, and purchased at the same time (same batch), in the same condition, they both balked at approximately the same time, not leaving me time to transfer data back.

- An internal hard drive, as I was making backups to CDs (at that time). For some kind of reason I still cannot explain, the software thought my hard drive was both the source and the destination !!!! Computer froze completely after a minute or two, then I tried rebooting to no avail, and my partition block was now containing a 700mb CD image, quarter full with my stuff. I still don't know how that's possible, but hey, it did. Since I was actualy making my first CD at the time and it was my first backup in a year, I lost countless good files, many I gave up upon (especially my 90's favorite music video sources ripped from the original betacam tapes in 4:2:2 by myself).

- A full bulk of HDs on Mac when I tried putting the journal to another internal SSD drive. I have dozens of HDDs, and I thought it'd go faster to use that nifty "journal on another drive" option. It did work well, although it was hell to initialize, as I had to create a partition for each HDD, then convert them to journaled partitions. Worked awesomely, very quick, very efficient. One day after weeks of usage, I had to hard close the computer and its HDD. When they remounted, they all remounted in the wrong order, somehow using the bad partition order. So imagine you have perfectly healthy HDDs but thinking they have to use another HDDs journal. Mayhem! Most drives thought they were other ones, so my music HDD became my photos HDD RAID, my system HDD thought it was the backup HDD, but just what was in the journal. It took me weeks sporting DiskWarrrior and Data Rescue in order to get 99% of my files back (I'm looking at you, DiskWarrior as a 32 bit app not supporting my 9TB photo drive) with a combinaison of the original drive files and the backup drive files. Took months to rebuild the Aperture database from that.

- All my pictures from when I met my wife to our first travels. I had them in a computer, I made a copy for sure. But I cannot find any of that anywhere. Nowhere to be found, no matter where I look. Since that time, many computers happened, so I don't know where it could've been sent. But I'm really sad to have lost these

- Did a paid photoshoot for an unique event. Took 4 32GB cards worth of priceless pictures. Once done with a card, I was sifting through the pictures with my camera and noticed it had issues reading the card. I removed it immediately. When at home, I put the card in my computer, it had all the troubles in the world reading it (but was able to do so), I was (barely) able to import its contents to Aperture (4-5 pictures didn't make the cut, a few dozens had glitches). It would then (dramatically, as it somehow have its last breath after relinquishing its precious data) not read or mount anywhere, not even being recognized as a card by the readers. Childs, use new cards regularly for your gigs :)

- A RAID array breaking, and the company nowhere to be found after all these years, and the discs not being able to be read elsewhere.

- Countless HDDs breaking in various ways (including a cat throwing a vase full of water onto an open laptop ... yeah ... love my cats sometimes), all without consequences as I have daily backups of everything I own, and monthly offsites.

Re:So far what I lost... (0)

Anonymous Coward | about 3 months ago | (#47236341)

A few CDs due to CD degradation. That includes one that I really wish I'd still have, as it was a backup of something I lost. However, the CD takes hours to read, and then eventually either balks up or not for the directory. I won't tell you about actually trying to copy the files, especially with normal timeouts in modern OSes or the hardware pieces or whatnot.

I've had some success with recovering very scratched CDs that wouldn't work using GNU Ddrescue, and I imagine that it'd work similarly on a CD with bitrot. If it worked, you'd end up with an ISO image of the CD, albeit with a likely chance of some corrupted areas, which you could then either mount or re-burn.

Re:So far what I lost... (1)

Electricity Likes Me (1098643) | about 3 months ago | (#47236611)

Scratched CDs can be recovered by polishing them with Brasso.

The trick is it polishes the scratches out flat so they don't mess with reflections of the reader.

Re:So far what I lost... (1)

steelfood (895457) | about 3 months ago | (#47236971)

I've had pressed CDs that began developing holes in the reflective layer starting from the outside edge. Those defects are irrecoverable.

Scratched CDs and DVDs are not as big of a deal these days as they used to be. Really good drives can handle them without losing or corrupting data, though the drive would slow down over the scratched areas, so if the disc was bad, the read speeds would drop to almost nothing. There used to be a really good site that did incredibly detailed reviews of optical drives, including taking black sharpies to media. Not sure if it's around anymore.

But honestly, who uses optical media anymore? Especially after the HD-DVD/Blu-ray debacle, I think everyone's turned off by optical media and prefers streaming now.

Re:So far what I lost... (0)

Anonymous Coward | about 3 months ago | (#47239305)

Bzzzzzt. Wrong. Lightly scratched CDs can be recovered.

Re: So far what I lost... (1)

cpct0 (558171) | about 3 months ago | (#47236753)

Yeah, with scratches it's a valid assumption. However, in my case it was cheap CDs with inks that degraded, so the reflectivity of the data itself was degraded, the drive was ultimately unable to retrieve data on most sectors, or it was able after dozen of reads over the same block of data, until the data got its green flag from the recovery algorithm embedded in Data-CD format specs.

A scratch is localized. CD dye degradation is global. But thanks for the idea.

Re:So far what I lost... (0)

Anonymous Coward | about 3 months ago | (#47236989)

I've seen the problem of stuttery videos with gaming PC's and stereoscopic video. I'm not sure if it was the CPU's (Intel vs. AMD) or whether it was that the video file was fragmented across the file system. But the same file on the same file system would just freeze and lose sound while being played on one system, and play smoothly on another:

http://www.3dtv.at/movies/Skydiving_en.aspx

And the story is? (3, Insightful)

Immerman (2627577) | about 3 months ago | (#47236137)

Bitrot. It's a thing. It's been a thing since at least the very first tape drive - hell it was a thing with punch cards (when it might well have involved actual rot). While the mechanism changes, every single consumer-level data-storage system in the history of computing has suffered from it. It's a physical phenomena independent from file system, and impossible to defend against in software unless it transparently invokes the one and only defense: redundant data storage. Preferably in the form of multiple redundant backups.

So what is the point of this article?

Re:And the story is? (0)

Anonymous Coward | about 3 months ago | (#47236189)

Anecdotal statistics.

Re:And the story is? (0)

Anonymous Coward | about 3 months ago | (#47239319)

Bitrot is the slowly creeping problem where backups don't necessarily help. Of course the backups might have less bitrotten but if the backups were down frequently, probably not. That is the scary part of the bitrot, the fear of actually and permanently losing data. People don't usually do backup comparison runs if they don't suspect there is a problem with their data.

Re:And the story is? (1)

Immerman (2627577) | about 3 months ago | (#47240291)

That's why you need multiple redundant backups - bitrot *will* hit all of them, but it's extremely unlikely to hit them all in the same spot, so redundant backups stored in an error-detecting format will allow you to reconstruct a single good copy.

Clickbait generating shit (1)

Torp (199297) | about 3 months ago | (#47236179)

The real article would be titled "file systems with no data redundancy and no checksums are vulnerable to bitrot".
That covers about any file system with the lone exception of ZFS when ran on a raid, maybe btrfs? and i guess some mainframe stuff.

Re:Clickbait generating shit (1)

gweihir (88907) | about 3 months ago | (#47236885)

Not even that. As the case does not seem to involve unreadable sectors, the corruption did likely not happen on disk. So the title should more be "people that are to stupid to verify their backups are readable and correct may lose data". He may also have copied his data around without making sure the copy matches the original. Stupid.

article is suspect, summary is worse (4, Informative)

sribe (304414) | about 3 months ago | (#47236183)

In a footnote he admits that the corruption was caused by hardware issues, not HFS+ bugs, and of course the summary ignores that completely.

So, for that, let me counter his anecdote with my own anecdote: I have an HFS+ volume with a collection of over 3,000,000 files on it. This collection started in 2004, approximately 50 people access thousands of files on it per day, and occasionally after upgrades or problems it gets a full byte-to-byte comparison to one of three warm standbys. No corruption found, ever.

Re:article is suspect, summary is worse (0)

Anonymous Coward | about 3 months ago | (#47236305)

Dude, this is slashdot post ponies. Any submission that bashes Apple is clickbait, and they'll post cuz they need the ad revenue or something.

Re:article is suspect, summary is worse (0)

Anonymous Coward | about 3 months ago | (#47236537)

Apple products are billed as premium, and often come with premium costs. You should be glad that someone out there is willing to shame them into actually fixing their long-standing problems, because it seems like most of their users are complacent faith-based drones with a persecution complex that kicks in whenever their mothership's flaws are pointed out.

Re:article is suspect, summary is worse (1)

jones_supa (887896) | about 3 months ago | (#47239113)

Agree. Apple is in the market of creating premium products and thus its creations should also be scrutinized rigorously.

Re:article is suspect, summary is worse (0)

Anonymous Coward | about 3 months ago | (#47239431)

Dude, this is slashdot post ponies. Any submission that bashes Apple is clickbait, and they'll post cuz they need the ad revenue or something.

It's been years, but it was still a better site redesign [imgur.com] than beta

Re:article is suspect, summary is worse (0)

Anonymous Coward | about 3 months ago | (#47241887)

One wonders if there is a larger MLP audience than slashdot audience, and if the site owners care, why they haven't redesigned the site to target that audience.

Re:article is suspect, summary is worse (0)

Anonymous Coward | about 3 months ago | (#47236503)

If the hardware is failing and the filesystem is neither detecting nor compensating for it, then the filesystem has failed as well. Of course there is no magic filesystem that will guarantee everything for you, but simply hand-waving the problem as "the hardware's fault" is not something you want to do because if your filesystem sucks then backups will likely be corrupted over time as well.

Re:article is suspect, summary is worse (1)

pla (258480) | about 3 months ago | (#47236515)

In a footnote he admits that the corruption was caused by hardware issues, not HFS+ bugs, and of course the summary ignores that completely.

The summary doesn't claim HFS caused the bitrot, you read that into it. The summary merely points out that HFS doesn't reliably detect and correct flaws in the underlying storage media (as does NTFS, as does almost every filesystem widely used).

More importantly, while merely detecting this issue may not incur too much overhead, correcting it requires some fairly large degree of redundancy. So although plenty of people have mentioned assorted alternative FSs that may (or may not) actually address the problem, doing so still requires wasting some not-insignificant percent of your disk space (a mere 10% of a 4TB drive would hold a whopping 90 full single-layer DVDs).

Joe Sixpack doesn't even get why 4TB doesn't equal 4 TiB; you expect him to understand the concept of parity striping to deal with cosmic rays randomly flipping bits on his platters? Try explaining that one to the public, and next time you visit Grandma, you can expect to find her PC dead because she wrapped it in tinfoil, including the ventilation fans.

Re:article is suspect, summary is worse (1)

sribe (304414) | about 3 months ago | (#47236721)

The summary doesn't claim HFS caused the bitrot, you read that into it.

The summary's first sentence ends: "about data loss suffered under Apple's venerable HFS+ filesystem" and shortly thereafter it continues with: "HFS+ lost a total of 28 files over the course of 6 years." So the chosen wording most certainly does imply that HFS is at fault. One has to click the link to the article, then read all the way through the frickin' footnotes before one encounters anything to explicitly disavow that implication.

Re:article is suspect, summary is worse (1)

pla (258480) | about 3 months ago | (#47236815)

Yes, it does - But those all hold true. The corruption did occur under HFS+, and HFS+ did "lose" some portion of those files. It didn't, however, cause the corruption or the loss of those files. You have read attribution into a scenario where none exists.

In fairness, I will agree with you that TFS (and to a lesser degree, TFA) doesn't clearly discriminate between "HFS-induced damage" and "cosmic-ray-induced damage". But they both knew the root cause, and it wouldn't have made sense to blame it on HFS+ unless they did so as an outright deception. Do you claim they did so intentionally, or just that their wording leaves room for interpretation? If the latter, I will agree with you. If the former, I don't know what else to say except that I don't agree.

Re:article is suspect, summary is worse (1)

laird (2705) | about 3 months ago | (#47238511)

The issue is that the headline and summary have HFS all over the place, and even say that "HFS corrupted files", when HFS wasn't relevant to the corruption - no standard filesystem protects you completely from disk drives' blocks going bad.

That being said, some of the high end SAN/NAS systems do have controls like forcing all blocks on the device to be read and (if needed) rewritten periodically, which would refresh the data and prevent "bit rot". But that's not done by the filesystem, either - it's a layer between the filesystem and the disk drives.

Re:article is suspect, summary is worse (0)

Anonymous Coward | about 3 months ago | (#47237547)

Just curious, but why would you do byte-for-byte comparison rather than verification through a hashing algorithm? Generate checksums using one of the many options available (md5sum should be plenty good, but you can go for sha256sum or sha512 sum if you're really paranoid), and check your archives against the results periodically. Much faster than byte-for-byte compares, and if you ever find two files which truly differ but produce the same checksum, be sure to let the researchers know -- right after you go buy Powerball and Mega Millions tickets.

Re:article is suspect, summary is worse (0)

Anonymous Coward | about 3 months ago | (#47238097)

No corruption FOUND does not equate no corruption EVER.

O! I read that as DICK rot (0)

Anonymous Coward | about 3 months ago | (#47236207)

Now THAT is serious.
Now that IS serious.
Now that is SERIOUS.
NOW that is serious.

Clueless article (4, Informative)

alexhs (877055) | about 3 months ago | (#47236227)

People talking about "bit rot" usually have no clue, and this guy is no exception.

It's extremely unlikely that a file would become silently corrupted on disk. Block devices include per-block checksums, and you either have a read error (maybe he has) or the data read is the same as the data previously written. As far as I know, ZFS doesn't help to recover data from read errors. You would need RAID and / or backups.

Main memory is the weakest link. That's why my next computer will have ECC memory. So, when you copy the file (or otherwise defragment or modify the file, etc), you read a good copy, some bit flips in RAM, and you write back corrupted data. Your disk receives the corrupted data, happily computes a checksum, therefore ensuring you can read back your corrupted data faithfully. That's where ZFS helps. Using checksumming scripts is a good idea, and I do it myself. But I don't have auto-defrag on Linux, so I'm safer : when I detect a corrupted copy, I still have the original.

ext2 was introduced in 1993, and so was NTFS. ext4 is just ext2 updated (ext was a different beast). If anything, HFS+ is more modern, not that it makes a difference. All of them are updated. By the way, I noticed recently that Mac OS X resource forks sometimes contain a CRC32. I noticed it in a file coming from Mavericks.

Re:Clueless article (1)

gweihir (88907) | about 3 months ago | (#47236911)

I agree. Silent corruption on disk data can basically only happen with a defective bit in the disk's RAM. Even then, it is exceedingly unlikely and the defective bit will also make sectors that are fine show up as defective.

As to main RAM corruption, you do not need to use ECC. You just need to verify what you put on disk, flushing OS disk caches before.

Re:Clueless article (1)

swilver (617741) | about 3 months ago | (#47241179)

Just use ECC RAM. The price difference is tiny.

Of course you can verify, but very few people do that, and it certainly is not something most filesystems do for performance reasons.

Re:Clueless article (1)

gweihir (88907) | about 3 months ago | (#47242301)

ECC RAM does not help you against most sources of corruption. Defective controllers, bus drivers, etc. all are completely untouched by ECC RAM. On the other hand, verifying data helps against all of them. If you do not verify, then you will have undetected bad data on disk sooner or later.

Re:Clueless article (2)

rabtech (223758) | about 3 months ago | (#47237115)

People talking about "bit rot" usually have no clue, and this guy is no exception.

It's extremely unlikely that a file would become silently corrupted on disk. Block devices include per-block checksums, and you either have a read error (maybe he has) or the data read is the same as the data previously written. As far as I know, ZFS doesn't help to recover data from read errors. You would need RAID and / or backups.

I'm afraid it is you who is clueless. Up until ZFS started gaining traction, we all had the luxury of assuming the storage chain was reliable (RAM, SATA controller, cables, drive firmware, read/write heads, oxide layers, etc). Or at least we would know something went wrong.

But it was found that in the actual real world, these systems all silently corrupt data from time to time. The problem is much worse as the volume of data grows because the error rates are basically unchanged, meaning what was once expected to be a random bit flip that would strike one user out of a million once per year is now something that strikes every single user multiple times per year.

I'm not talking theory or what *should* happen. I'm talking about actual real world experience with check summing filesystems that demonstrate, beyond any doubt, that bit rot happens and happens far more frequently than most people believe. Actual experience with ZFS proves that disks can and **will** read back out different bits than what was written silently with no block read errors.

Further, you're increadibly ignorant of now ZFS or BTRFS deal with redundancy. You can setup to mirror blocks, in some cases on a per-file or directory basis, providing protection against corrupting. A background scrubber scans the disk when idle cycles are available and detects and repair corrupting from the available good blocks, or log an error if there are no good mirrors or parity blocks available.

With our new knowledge and experience it is no longer sufficient to cross our fingers and hope for the best. We cannot trust filesystems or the underlying hardware, we must verify.

Re:Clueless article (0)

Anonymous Coward | about 3 months ago | (#47238701)

Actual experience with ZFS proves that disks can and **will** read back out different bits than what was written silently with no block read errors.

And how do you know that it's not your installation of ZFS that's causing the problem?

Re:Clueless article (1)

eWarz (610883) | about 3 months ago | (#47239311)

As someone who has 20 mb hard drives from decades ago...I find this whole bit rot thing a bit hard to swallow. While i'm not saying it's not possible...it's not as likely as people claim. What has more likely happened is the amount of improper power ons/offs/resets damaged the data as it was being written to disk to begin with.

Re:Clueless article (1)

swilver (617741) | about 3 months ago | (#47241197)

...and I suppose this silent corruption was verified by reading it into main memory?

My own simple tests (copy 1 TB of data from one place to another) on ECC and non-ECC systems showed quite clearly where the culprit was. Bit error rates of 1 bit/100 GB with the non-ECC system showed the problem clearly.

Re:Clueless article (0)

Anonymous Coward | about 3 months ago | (#47237199)

Main memory is the weakest link. That's why my next computer will have ECC memory.

We promise that to ourselves, but our lust for consumer grade paradise and cheap prices is too great. I have wondered about those i3-taking, ECC-supporting server boards if the error checking still works with the consumer processors. AMD with ECC on computer memoirs have been the more affordable solution so far. ECC memory itself costs practically the same than the regular heat-sinked variety.

Re:Clueless article (1)

toddestan (632714) | about 3 months ago | (#47242995)

I have wondered about those i3-taking, ECC-supporting server boards if the error checking still works with the consumer processors.

Since the memory controller is part of the CPU you can't just drop in a regular consumer processor and get ECC this way. You're stuck with whatever models that Intel decides to turn on the ECC bit for, which is pretty much the Xeons and a few oddball embedded versions.

Re:Clueless article (1)

Anonymous Coward | about 3 months ago | (#47237675)

As far as I know, ZFS doesn't help to recover data from read errors. You would need RAID and / or backups.

Data point: ZFS can recover data from read errors, as long as you're using some form of data redundancy as part of a vdev or pool, i.e. a mirror or raidzX.

For example, say your pool consists of 2 mirrored devices (disks): if the read from one of those fails (effectively returns EIO), ZFS can and will read from the other device in attempt to get the data it wants. If it's successful, it logs a read error for that device (see "zpool status") and informs you, in layman's terms, "hey, I got a read error from this device, but I was able to read it from the other device so valid data was given to the userland app that did a read(), but you should probably do something about that device". The userland application doing the read() never sees any of this go on -- it's happening at the ZFS and kernel layer.

The same methodology applies for raidzX (through parity and other methodologies), ditto with if you have multiple vdevs consisting of redundancy methods (think RAID-10).

If you're using ZFS with a single device (i.e. no redundancy) then ZFS will inform you of the read error but cannot "auto-correct" it -- meaning the underlying userland application syscall gets EIO.

You already covered checksumming so I won't go into that, but checksumming does not guarantee you can recover data -- only that you can detect things like silent corruption or "bit rot".

P.S. I have never seen "bit rot" where the magnetic media on a hard disk has quietly gone bad on any system I've used ZFS on (this would show up as a random checksum error in "zpool status"). I'm more inclined to believe what others have reported as "bit rot" is more likely filesystem corruption through software means (bugs), or unexpected power loss on the system (this does wonders to a filesystem; journalling doesn't recover your data, it just ensures your filesystem is usable after-the-fact). I do data recovery (software-based) as a hobby and spend a lot of time reading about and tinkering with actual ATA protocol.

Re:Clueless article (1)

dargaud (518470) | about 3 months ago | (#47237871)

I had a home server / workstation with ECC, but had to convert to a laptop: no ECC anywhere except maybe a few milspec models that cost the price of a car and have the specs of a watch...

Clueless article (1)

Anonymous Coward | about 3 months ago | (#47242179)

Just some minor corrections in your text.

ZFS always detects bit rot. ZFS can always repair bit rot if it is configured to do so; if some redundancy is used. For instance, raid or mirror. BUT! You can also configure ZFS to double (triple) every data on a single disk, so ZFS can repair data using only a single disk. BTRFS can also do this, and it does by partitioning the disk into two partitions, building a mirror on a single disk. This is extremely cumbersome, ZFS does not do that. You dont need to repartition or anything with ZFS, just specify "copies=2". Done.

BTW, there is no research if BTRFS is safe, on the the other hand, there are several research projects showing that ZFS is safe, read the research papers on the ZFS wikipedia article.

I read some guy speculating in a storage solution that should repair the corrupted data block from a given checksum, by trying different valid data blocks fulfilling the checksum. He googled this and it turned that someone already tried that. Guess which solution? Yep, ZFS. But that solution was omitted because it took to much time. Pretty cool anyway.

HFS reliability (0)

Anonymous Coward | about 3 months ago | (#47236303)

Anyone who owned a Mac since the 80s remembers having to use Norton Disk Doctor and later DiskWarrior at least once per month to repair the filesystem. Entire folders could go randomly missing each time you booted up your Mac, and if you accidentally lost power to your hard drive, the use of one of those was mandatory.

Re:HFS reliability (3, Insightful)

sribe (304414) | about 3 months ago | (#47236385)

Anyone who owned a Mac since the 80s remembers having to use Norton Disk Doctor and later DiskWarrior at least once per month to repair the filesystem. Entire folders could go randomly missing each time you booted up your Mac, and if you accidentally lost power to your hard drive, the use of one of those was mandatory.

Oh yes. I remember those days well. Journaled HFS+ fixed that, and for about the last decade the only times I have encountered a corrupted file system on a Mac, that discovery was followed shortly by total failure of the hard disk.

So, what was your fucking point?

Re:HFS reliability (1)

jones_supa (887896) | about 3 months ago | (#47239123)

So, what was your fucking point?

He was just thinking back the ole times.

Re:HFS reliability (1)

Smurf (7981) | about 3 months ago | (#47236395)

Anyone who owned a Mac since the 80s remembers having to use Norton Disk Doctor and later DiskWarrior at least once per month to repair the filesystem. Entire folders could go randomly missing each time you booted up your Mac, and if you accidentally lost power to your hard drive, the use of one of those was mandatory.

No, not "anyone who owned a Mac since the 80s...". My first Mac was a Mac Plus bought in 1987 (IIRC), and I have never used those tools nor experienced the problems you mention.

Re:HFS reliability (1)

laird (2705) | about 3 months ago | (#47238533)

+1 this. The only time I've ever had Mac filesystem problems was when there were unexpected power loss. When you lose power while writing to the drive, bad things can happen. But I've not seen even that since Mac OS 7 or so. :-)

Re:HFS reliability (0)

Anonymous Coward | about 3 months ago | (#47239339)

Anyone who owned a Mac since the 80s remembers having to use Norton Disk Doctor and later DiskWarrior at least once per month to repair the filesystem. Entire folders could go randomly missing each time you booted up your Mac, and if you accidentally lost power to your hard drive, the use of one of those was mandatory.

No, not "anyone who owned a Mac since the 80s...". My first Mac was a Mac Plus bought in 1987 (IIRC), and I have never used those tools nor experienced the problems you mention.

Correct. You actually had to power on the machines in order to experience said issue. Simply owning Macs and never powering them up would imply you avoided the problem. Common misunderstanding.

BTW, I'm a different AC.

Seriously, dude, my first mac was a IIcx in 1989. Data loss in non-journaling filesystems like HFS is damn common. Next you will be telling us you have never seen a bomb dialog system error or a sad mac icon in these past 25 years. Power loss and system crashes can and did fuck HFS volumes. Norton Disk Doctor was great, and got progressively worse over time (like everything else released by Symantec), until by the mid 90's I really hesitated to run it because it often caused more problems/data loss than it fixed. DiskWarrior, on the other hand, is amazing and everyone should own a copy (they are still releasing new versions, OS X compatible)... it was revolutionary back on Mac OS 9 and has continued to shine.

But what do I know? My first full time job was as an Apple Certified Technician at an Apple Authorized Service Center back in 1998.

Re:HFS reliability (1)

Smurf (7981) | about 3 months ago | (#47240451)

All the Macs I've owned have always been my main personal computer, and the first couple were my only computer at the time. I did everything on them: schoolwork, gaming, stuff for my dad's office and for others, etc. Looking back, I believe I spent way more time with them than I should have.

Did I experience system crashes with the dreaded bomb box? Yes, plenty of them. Did I experience sad Macs? Yes, occasionally. (I believe it was supposed to appear on hardware failure, but after restarting the computers continued to hum along for years). I never owned (nor pirated) a copy of Norton Disk Doctor, although I did see it running on other people's computers.

It's not my fault that my experience differs from yours.

Re:HFS reliability (0)

Anonymous Coward | about 3 months ago | (#47244121)

I guarantee your filesystem and file data was far more trashed by the non-journaled HFS filesystem than you allege it was.

If your file gets corrupted by system crashes while the OS was writing to the disk, and your HFS has no file sector checksum, how would you know if some random files were corrupted or lost parts? It seems, barring some self-implemented file verification regimen against known-good backups, you wouldn't know about the corruption unless it was gross enough to crash the system, cause an application to puke on the file, or show you mangled text, etc.

As I pointed out, by the mid 90's, I would no longer use Norton Disk Doctor on any drive that didn't already have data loss unrecoverable by any other tool. Symantec fucked it up, and NDD started causing more data loss than it fixed. Tech Tool was okay. Diskwarrior is awesome, and if your alleged charmed life comes to an end one day, think of DiskWarrior first. It will actually allow you to recover the whole drive without writing anything to it, which is an important first step if your backups aren't up to the moment when your corruption strikes, then it will let you write the fixes back to the disk later to complete the repair. In this sense, it's even safer/superior to the built-in Disk Utility... I have had Disk First Aid cack a drive before, but it's impossible for DiskWarrior to cause that problem before you can grab a backup.

Re: HFS reliability (0)

Anonymous Coward | about 3 months ago | (#47236397)

As someone who bought a mac 128k in May 1984 (educational discount $1000) and owned only Macs at home - I virtually never had to do disk repairs - probably on the order of one per decade. Certainly not close to even yearly.

Re: HFS reliability (0)

Anonymous Coward | about 3 months ago | (#47239343)

I guarantee you that your filesystem and files were far more trashed than you believe they were.

How often did you run Disk First Aid when nothing was apparently wrong? Also, don't forget it wouldn't verify your boot drive. It was a pain to test, and your non-journaling HFS would happily trash files (or filesystem data) that were being written when a system bomb dialog popped up.

Don't pretend that was a once in a decade event.

Re:HFS reliability (1)

Etcetera (14711) | about 3 months ago | (#47237355)

Anyone who owned a Mac since the 80s remembers having to use Norton Disk Doctor and later DiskWarrior at least once per month to repair the filesystem. Entire folders could go randomly missing each time you booted up your Mac, and if you accidentally lost power to your hard drive, the use of one of those was mandatory.

I think you're confusing generic Disk Repair with rebuilding the Desktop File...

Unless your drives were seriously damaged (floppies thrown in a backpack were always a bad idea no matter where you were), missing icons and whatnot were at the disk catalog level (used by Finder), not the HFS level. Command-Option on disk insert would fix it for me.

In the event of a power outage or something similar, it was always advisable to run Disk First Aid (and later versions System 7.5+ or Mac OS 8.1 maybe?) would run it automatically for you in the event of an unsafe shutdown, but that's just morally equivalent to running an fsck.

Re:HFS reliability (1)

doccus (2020662) | about 3 months ago | (#47243683)

Stuff disappears on me all the time ;-)

Btrfs (1)

Flammon (4726) | about 3 months ago | (#47236331)

I've slowly been moving all my systems to Btrfs from least important to most important and have had no problems so far.

Re:Btrfs (1)

wisnoskij (1206448) | about 3 months ago | (#47236641)

Btrfs "pronounced "Butterface"" - Wikipedia
Lol.
Strangely that acronym could also stand for BiT Rot Free System which is pretty ironic, I guess.

Re:Btrfs (1)

gweihir (88907) | about 3 months ago | (#47236917)

You are moving data you care about to a new and not well-aged filesystem? Then you will get all the data-loss you deserve.

Re:Btrfs (1)

Flammon (4726) | about 3 months ago | (#47237035)

There are varying degrees of stability and I felt that after 7 years of development, official inclusion into the Linux kernel, Facebook deployment and the default fs on OpenSUSE that it's good enough for my laptop, workstation and a few other systems. Having that said, I've not migrated by backup drives yet, they're still on XFS. It may be a while until I migrate those. http://www.phoronix.com/scan.p... [phoronix.com]

Re:Btrfs (1)

gweihir (88907) | about 3 months ago | (#47237641)

Well, if your date is worthless, then by all means, go ahead. Here is a hint though: How long an FS has been in development is completely immaterial. What matters is how long it has been stable.

So answer me this... (1)

trparky (846769) | about 3 months ago | (#47236399)

Some people are talking about the fact that bitrot could happen as a result of bad RAM. Are you talking about bad system RAM or the RAM onboard the HDD's controller board?

If it was indeed bad system RAM, wouldn't bad system RAM cause a random BSOD (Windows) or Kernel Panic (Linux)? With how much RAM we use these days it's very likely we're going to be using all of the storage capacity of each of the DIMMs that we have in our systems.

Myself I have 16 GBs of RAM in my Windows machine and at any moment in time I'm using at the very least 40% of the RAM in the system with spikes up to at least 60% depending upon what I'm doing at the time. So with that said, the possibility of kernel memory structures being corrupted at some point while using memory (in even less used DIMMs in your system) I figure is going to happen. I'm not sure how the memory in the DIMMs are being used though. Is it being used sequentially? (DIMM 0, chip 1... 2... 3... 4, DIMM 1, chip 1... 2... 3...4, etc.) Or is the data thrown about randomly on the DIMMs?

Myself, if I had a random BSOD just happen I'd be running MemTest86+ in a hot second to test my system RAM and be asking to Corsair (the company that made my DIMMs) for an RMA.

So if does indeed turn out to be bad system RAM that causes this, I guess that it's a good idea not to be buying cheap RAM to begin with. Myself, I've never had a problem with Corsair Vengeance RAM modules so I will continue to buy that line of Corsair memory.

It's all about ERROR rates (2)

bussdriver (620565) | about 3 months ago | (#47236581)

RAM may have a low error rate much better than HDDs or SDs. That does not mean that you won't have errors even if you have a good brand and treat it well. Bit-level errors can and do happen all the time without us knowing; other times it happens in the wrong place and we notice (but think it is something else) it isn't until it gets really bad that we notice.

Example, say your RAM has a 1% bit loss rate (ignore that is insanely high) well if 90% of your data is not touchy code but data, the odds are that you may not notice 1 bit getting flipped that often. Then you have the fact that RAM could maintain that error rate over decades of smaller faster RAM but now you are storing MORE data and cycling it MORE than was possible on the older computers. So, if you had 1 bit error every gigabyte of throughput on a slow 1Mhz computer with 1MB of RAM it would take a long time for that 1% bit flip to happen (and if you noticed you'd still not likely blame the RAM) -- but today pumping though in seconds what that old machine would take a year; the error would occur quite often. SAME problem with storage but with an additional problem in that they still have the same lifespan requirements - RAM can be refreshed can checked.

Something else to be considered, the error correction schemes being used today are being pushed by the demand for higher density storage. Your HD isn't doing huffman or any of those old simple bit recovery schemes they've moved beyond that long ago to the next gen stuff from what your 56k modem was doing to fight phone line noise. They could make it better... but you would be giving up significant storage space. Perhaps somebody with a good marketing scheme and enough upset consumers could get you to pay MORE for less storage space... I know I would buy into it.

Essentially, we are at a point where HDDs expect you to scrub them for errors every year to avoid the bit rot... which is what I now do... haven't detected an error in years... however, the block level checksums the HDD uses has false positive error rate (just like CRC16 does) and the odds of a false positive may be poor--- again, we are working in the trillions now-- up near it's limitations (I'm assuming whatever they use now scaled... but it may not have which is why more people are talking about these issues. We know it's unlikely industry has adapted to the trends evenly over the decades... it's likely become a minior problem before they are forced to change devices to a newer proprietary checksum and error correction scheme. )

Do serious work? use ECC RAM. I'm still waiting for some low power AM1 motherboard that supports ECC so I can build a ZFS server... the AM1 chip supports ECC but no motherboards do.

Re:It's all about ERROR rates (1)

trparky (846769) | about 3 months ago | (#47236639)

I have noticed that a lot of OEMs (Dell, HP, Apple, etc.) use a no-name brand of RAM in many of their systems that they build. If you look at them, especially the CAS latency stats, you'll notice that many of the RAM chips found in most pre-made computers are absolutely pitiful (to say the least).

So with that being said, who knows if this no-name RAM that is installed in many pre-made computers that many people buy is of any real quality. I'm guessing... no. So, with that said perhaps that odds of bitrot happening on pre-made machines is going to be higher than that of systems that have better quality of system RAM installed in them.

Re:It's all about ERROR rates (1)

gweihir (88907) | about 3 months ago | (#47236737)

ECC is not what you need for reliable data archiving. What you need is independent checksums and you need to actually compare them to the data on disk. If you store an MD5 or SHA1 hash with all files, corruption from RAM, buses and the like will not go undetected. The way things go today though, most people do not even verify a backup. No surprise they lose data, incompetence and laziness comes at a price. Of course, you should make sure your RAM runs stable, but I have not had a single ECC corrected bit in running with ECC for several years on several machines including two servers, so I decided to drop it.

Re:It's all about ERROR rates (1)

laird (2705) | about 3 months ago | (#47238523)

I used to work in supercomputing, and with terabytes of RAM and Petabytes of data I/O, you *bed* everything had ECC and parity bits every step of the way (yeah, extra parity bits in the RAM). And cosmic rays really do flip bits in RAM from time to time, and when you're running a $10M machine running a 2 month computation, you really do care about being able to detect the error, restore the machine's state to the previous snapshot, and keep running.

It's amusing to me that consumers now have enough data that this stuff starts affecting them. It'll be interesting to see if consumers start paying extra for the reliability.

Re:It's all about ERROR rates (1)

gweihir (88907) | about 3 months ago | (#47242313)

And these machines have ECC on all buses, controllers, etc. Then it makes sense. Consumer-grade ECC is only on the memory itself. That is not enough and you need to do data-verification.

ok you asked for it (1)

bussdriver (620565) | about 3 months ago | (#47240551)

I dug up the study.
"End-to-end Data Integrity for File Systems: A ZFS Case Study"
Zhang, Rajimwale, Arpaci-Dusseau

Cosmic rays do happen; odds go up as elevation increases. I would guess location also matters.
other looking provided this gem:
Google reports that more than 8% of every DIMM gets error, each year. Google found that the error rates were several magnitudes larger than small scale studies showed.

Re:ok you asked for it (1)

gweihir (88907) | about 3 months ago | (#47242317)

I am not disputing that. But RAM is not the only significant source of bit-errors.

Re:It's all about ERROR rates (1)

bussdriver (620565) | about 3 months ago | (#47240647)

Note that I did mention vendors expect us to scrub our storage data to correct errors to catch and repair losses... I don't remember being told... but then I've not read any paperwork that came with a HDD in a long time.

I talked about ECC RAM because it is a similar problem. We are raising our demands on the tech so the reliability level (includes associated techniques) has to increase to meet our higher demands. The fact we are noticing this more to me indicates either that we have more discussion of the problem or that the reliability has not been scaling at the pace of our increased demand on the technology.

I do remember experiencing LESS bit rot in my storage in the past; I had less data in the past... but I also accessed that smaller data set more.

When I had 5.25" floppy disks I had errors happen and I noticed them... likely all of them. The impact was huge when data was low and code was high. Plus, I didn't have much storage to keep track of so I was able to spot it. Today, I have more data storage than I have time to review it.

The natural entropy of magnetic storage does far more harm to modern data densities than the old floppies. One would expect more data is required to correct errors at the same integrity levels because of the densities involved today and the nature of the physics involved is different (more chaotic) than in the past... but we expect to use a % of the newly discovered storage so integrity would go down.

Re:So answer me this... (0)

Anonymous Coward | about 3 months ago | (#47236651)

Unlike bit flips from radiation, RAM defects aren't randomly spread over the entire address space. Often the defect is only in a few bits or even in just one bit, and then it isn't necessarily something simple, like a stuck bit (always 0 or 1). I once owned a DIMM with just one defective bit which failed just one of Memtest's patterns, and then only about 50% of the time. That DIMM caused file corruption similar to that described in the story. The machine was rock solid otherwise. Apparently the OS never used that part of the physical address space for vital OS structures.

Re:So answer me this... (1)

trparky (846769) | about 3 months ago | (#47236735)

If I were in your shoes, if that module failed a MemTest (even just one pass) then that module will be getting replaced with an RMA from the RAM manufacturer. I don't care if the system is stable, if that module failed... it's getting replaced.

Re:So answer me this... (0)

Anonymous Coward | about 3 months ago | (#47236805)

Yes, obviously that DIMM was retired once it was found to be faulty. How many users memtest their RAM though (and do more than a single pass)? More than 1 in 1000? I doubt it. The point is, bad RAM exists, and it doesn't necessarily cause system instability. You can go years without noticing RAM defects and the spurious file corruption they cause unless you are aware of the problem and take the extra steps and do the work to regularly check data integrity.

Re:So answer me this... (1)

jones_supa (887896) | about 3 months ago | (#47239129)

Unlike bit flips from radiation, RAM defects aren't randomly spread over the entire address space. Often the defect is only in a few bits or even in just one bit, and then it isn't necessarily something simple, like a stuck bit (always 0 or 1). I once owned a DIMM with just one defective bit which failed just one of Memtest's patterns, and then only about 50% of the time. That DIMM caused file corruption similar to that described in the story. The machine was rock solid otherwise. Apparently the OS never used that part of the physical address space for vital OS structures.

As a nifty little trick, if you know the exact memory address of that bit, you can use the Linux kernel "badram" boot parameter to exclude that location. :)

Re:So answer me this... (1)

fnj (64210) | about 3 months ago | (#47236657)

If it was indeed bad system RAM, wouldn't bad system RAM cause a random BSOD (Windows) or Kernel Panic (Linux)?

Likely so, but if we are talking about errors that only show up in 28 file-reads out of millions of file-reads, there is no reason to believe that you would be bound to see such a panic during the period in question.

BTW, bad RAM anywhere in the chain from disk drive to CPU - main system RAM, CPU cache RAM, hard drive cache RAM, controller RAM, etc - could cause such a panic, since most data travels all the way through such a chain. I am rather awestruck at how reliable the millions to billions of transistors in that chain actually are.

Re:So answer me this... (1)

silas_moeckel (234313) | about 3 months ago | (#47237717)

It need not be bad ram. If your not running ECC from top to bottom a stray bit of radiation etc will flip a bit every now and then. ECC lets this be detected and corrected. This can be an issue with the whole chain and is a tradeoff calculating ecc means added latency, requires buffers which in themselves give more places to have a bit flip.

how is this a file system problem? (1)

stenvar (2789879) | about 3 months ago | (#47236529)

This sounds like actual disk errors. File systems can't do much about them, you really need something like a RAID.

Re:how is this a file system problem? (1)

gweihir (88907) | about 3 months ago | (#47236745)

The OP did it wrong due to stupidity or laziness and now he is blaming others like an immature, petulant child would do.

Re:how is this a file system problem? (2)

kthreadd (1558445) | about 3 months ago | (#47236791)

The file system can do quite a bit if it actually does consistency checks on the data when reading it. ZFS does this and will alert you if the contents of a file has changed after it was last written, allowing you to restore a good copy from backup and verify that it is still valid.

Re:how is this a file system problem? (-1)

Anonymous Coward | about 3 months ago | (#47236887)

ZFS only does consistency checks, it can't recover the data for you. So it doesn't actually fix this problem.

It's also an insanely stupid thing to dob in a file system, since both disk drives and many file formats do the same thing. Do not want.

Those that are incompetent will lose their data (1)

gweihir (88907) | about 3 months ago | (#47236727)

There are only two options for reliable data archiving: 1. Spinning disks with redundancy and regular checks 2. Archival grade tape. There used to be MOD as well, but as nobody cared enough to buy it, development stalled and then died. The OP simply was naive and stupid and did not bother to find out how to archive data properly. It is well-known how to do it and has been for a long time. I have not lost a single bit that I care about. Of course, I have a 3-way RAID1 with regular SMART and RAID consistency checks. I have off-site backups that are made with full or at least crypto-hash comparison to the original. I have lost plenty of bits that were not on RAID and I have to replace a disk in that RAID1 about every 1-2 years because of read errors, but none of that is surprising.

In short: The OP is lamenting his own stupidity and he is not even aware of it. Dunning-Kruger effect at work.

And BTW, before I forget: SSDs have worse properties for archiving that spinning disks. As people are generally stupid, I expect the "problem" of bit-rot will get worse. At least as long as people are too lazy to find out how to do things properly or are unwilling to spend the money that doing things right takes.

Re:Those that are incompetent will lose their data (1)

WheezyJoe (1168567) | about 3 months ago | (#47237059)

There are only two options for reliable data archiving: 1. Spinning disks with redundancy and regular checks 2. Archival grade tape. There used to be MOD as well, but as nobody cared enough to buy it, development stalled and then died.

Any experience with M-discs [wikipedia.org] as archival media? Newer cd and dvd burners [amazon.com] are compatible with them, but do they deliver?

Re:Those that are incompetent will lose their data (1)

gweihir (88907) | about 3 months ago | (#47237605)

No. Consumer-grate trash. The absence of a cartridge is already enough to see that. Also, anybody claiming "1000 years data lifetime" must be lying, as the best accelerated aging models can give you 60-80 years predictability but not more.

Re: Incompetent -- Learning Archival Strategies (1)

BoRegardless (721219) | about 3 months ago | (#47237117)

What is the best overview doc/book out there for covering backup-archiving options?

I want to be more conversant with the subject before starting work with a FileMaker Pro DB consultant.

I will be doing a mission critical but small database, so data storage size won't be an issue as far as existing 1-4TB HDDs go & RAID arrays. Losing a day's or even an hour's data entry is not an option.

Re: Incompetent -- Learning Archival Strategies (2)

FaxeTheCat (1394763) | about 3 months ago | (#47237481)

Losing a day's or even an hour's data entry is not an option.

If you have that kind of requirements (less than an hour lost data), then you are not looking for just backup/archive. You are looking for a fully redundant storage system.
In addition to the backup system, of course.

For reading, check up on backupentral.com, Symantec.com (Backup Exec/Netbackup) emc.com (Avamar, networker).
I once managed a Filemaker database server (v5), and it has a built in featuer to copy the database files for backup. Real simple. Cannot remember if the database had to be taken offline, as we had users only during normal working hours, but these days that should NOT be a requirement.

Re: Incompetent -- Learning Archival Strategies (1)

gweihir (88907) | about 3 months ago | (#47237617)

Simple: It is a "Datasheet" covering an "archival grade medium". If you do not know that, you have absolutely no business working on any kind of "mission critical" storage, as you are simply incompetent with regard to that subject.

Re: Incompetent -- Learning Archival Strategies (2)

WheezyJoe (1168567) | about 3 months ago | (#47237759)

Simple: It is a "Datasheet" covering an "archival grade medium". If you do not know that, you have absolutely no business working on any kind of "mission critical" storage, as you are simply incompetent with regard to that subject.

Easy, there, big fella. Posting a link to a datasheet would have sufficed. Ain't right to call a man incompetent for asking a question. Truly, an incompetent is one who don't never ask the question assuming he already knows. Credit is due for seeking to learn something.

Re: Incompetent -- Learning Archival Strategies (0)

Anonymous Coward | about 3 months ago | (#47237857)

Easy, there, big fella. Posting a link to a datasheet would have sufficed. Ain't right to call a man incompetent for asking a question.

You don't understand, it's gweihir's JOB here to insult other people and call them incompetent!

He has one job, he's gonna do it...

Re: Incompetent -- Learning Archival Strategies (1)

gweihir (88907) | about 3 months ago | (#47242289)

And it is AC's job to spread lies. So take all he says with a grain of salt.

Re: Incompetent -- Learning Archival Strategies (1)

BoRegardless (721219) | about 3 months ago | (#47239003)

Just for clarity, I'm not going to run the backup/archival system as an FMPro consultant will do that.

I need to get some more background, so I have knowledge of where the tradeoffs are. I know this is done all the time, but I'm sure there are still choices to be made.

Re: Incompetent -- Learning Archival Strategies (1)

gweihir (88907) | about 3 months ago | (#47242281)

You need to find out the details of the backup/archival system being used. There is no way around that. It cannot be modeled as an opaque component, you need to understand the whole stack.

Re: Incompetent -- Learning Archival Strategies (1)

gweihir (88907) | about 3 months ago | (#47242273)

"Incompetent" is a state, not an insult. And no, I cannot post any datasheets as I do not know what kind of equipment will be used.

Give me a break... 28 files in 6 years? (0)

Anonymous Coward | about 3 months ago | (#47237251)

If this guy had had that many printed photographs sitting around in physical form farm more than 28 of them would have been destroyed in 6 years of non-professional storage. By any historical criteria this is an unbelievably successful archival medium!

As a simple preventative... (0)

Anonymous Coward | about 3 months ago | (#47237755)

IIUC, the main reason for the problem is that the magnetic value of some bits weakens to the point that it cannot be correctly read. Assuming that the actual magnetic coating is not damaged, couldn't we avoid this problem simply by having a low-level low-priority task running that would read/write each block (without actually updating the various file dates)? By reading and then rewriting, the bit values would be reinforced. The task would simply cycle through all blocks on the disk and then start over.

For SSDs, if bitrot is even a problem therein, you would probably want the task to run somewhat infrequently, say once a month?

Single Disk Parity (1)

randallman (605329) | about 3 months ago | (#47238791)

Does any file systems support single disc parity?

Set a parity ratio depending on risk vs. space loss tolerance. Say it is 1000. You can lose any of 1000 bytes in a parity group and recover while only giving up .1% of your disk space to parity.

Been using HFS since the 80's (0)

Anonymous Coward | about 3 months ago | (#47238805)

ALL filesystems and HD's suffer from file corruption over time. It's not even the filesystems fault, so I'm a little confused why this guy is blaming HFS+. Maybe he didn't want to blame anything else? All media, whether is it CD's Magnetic HD's, flash, etc. will suffer this fate in the end. The best thing the end user can do is scan the HD and files for problems regularly.

Long story short, my PM 9600 still boots just fine, and that's from '97 using HFS+. I've also been using the HFS filesystem since the late 80's (I think), and I've found it to be one of the BEST file systems I have ever had to use on any computer.

If someone wants to come up with a sort of ECC file system, than that would be awesome. But until someone does, then you're just going to have to deal with losing a few bits here and there.

I thought bitrot.. (1)

doccus (2020662) | about 3 months ago | (#47243675)

was what happened to Microsoft coder's socks.. I have that on good authority from someone at Apple..
Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?