Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Ask Slashdot: Keeping Digital Media After Imaging?

Unknown Lamer posted about a year ago | from the digitial-frisbees-for-all dept.

Media 122

New submitter rogue_archivist writes "I'm an archivist at a mid-sized university archives, trying to develop a policy for archiving computer files ('born-digital records' in archival parlance). Currently old floppy disks, CDs, and the occasional hard drive are added to our network storage. Then the physical media is separated from archival paper documents and placed into storage. My question for all you slashdotters out there is: should these disks be imaged and then the physical copies discarded? Is there any benefit for keeping around physical copies of storage media long since rendered obsolete?"

Sorry! There are no comments related to the filter you selected.

Is there any benefit? (0)

Anonymous Coward | about a year ago | (#44365121)

No. Image and discard.

Image-Discard-and back-up the image (0)

Anonymous Coward | about a year ago | (#44365379)

Image->Discard->and back-up the image

Don't forget the last step, make back-ups of your digital copies.
My family pics reside on >6 hard drives including one in a safety deposit box.

Re:Image-Discard-and back-up the image (3, Insightful)

doti (966971) | about a year ago | (#44365843)

image->backup->check image and backup->discard

Re:Image-Discard-and back-up the image (3, Informative)

wonkey_monkey (2592601) | about a year ago | (#44368355)

image->backup->check image and backup->discard->sign in triplicate->sent in->send back->query->lose->find->subject to public inquiry->lose again->bury in soft peat for three months and recycle as firelighters

Re:Is there any benefit? (4, Insightful)

cusco (717999) | about a year ago | (#44366011)

As an archivist, I would think you might want to:

1) keep multiple copies of each type of media, preferably from different manufacturers, all written with identical data
2) Separately, a copy of the data contained on the media

Occasionally check the media to see at what rate their integrity is decaying. As readers for the media become increasingly difficult to encounter develop alternative methods to read the data, checking it against your reference copy. Eventually someone is going to appear on your doorstep with something like the Pioneer spacecraft data tapes or the Nixon Oval Office recordings, and if you can pull the data off it you'll be the hero of the day.

Re:Is there any benefit? (4, Interesting)

cusco (717999) | about a year ago | (#44366177)

Just to reply to myself . . .

Target headquarters in Minneapolis gets VCR tapes from security systems all over the country, and they work with the FBI to read them and export the video. Security equipment manufacturers are notorious for using proprietary equipment or file formats to limit interoperability with the competition's systems, and they apparently have a lab that specializes in decoding them to extract the usable data.

Don't image, copy raw files, keep old media (2)

drnb (2434720) | about a year ago | (#44366399)

Re:Is there any benefit?

No. Image and discard.

Untrue. Backups get lost, go bad, or otherwise screwed up. At a previous employer old directories no longer accessed got backed up to tape to free up server space. We used a lot of storage at this company. On multiple occasions some years old files were needed. About half the time we would be told that these files were no longer recoverable by IT. After one of these backup failures I recalled that I had made a backup DVD of one old project we were trying to get a copy of. I went to our archivist and she found the DVD, it was readable.

Save the media. Buy a USB based floppy drive. It makes sense to copy the files on these legacy format to a server for future use but keeping the originals around as a backup is a good idea.

Do not image the media. These image formats may fall out of favor and not be recognizable in the future. Look at the various problems NASA has had with some of its old tapes using formats no longer supported. Create a folder on a server for a particular piece of media and just copy and verify the files from the legacy media to the folder on the server.

DVDs only live for 7 years (2, Insightful)

elabs (2539572) | about a year ago | (#44365129)

For born-analog content, always keep the original physical copy. You never know when you will need to rescan at a higher quality or when you will discover errors in your digital copy. DVDs are not born analog. In fact the only have a shelf life of around 7 years. You need to get everything off DVDs and make several digital copies of it. You should keep the DVDs as long as possible but eventually you will not be able to read them anymore. Make sure your digital copies of the DVDs are error-free because there will come a time when you cannot go back to the DVDs.

Re:DVDs only live for 7 years (2)

Russ1642 (1087959) | about a year ago | (#44365409)

You're thinking of burned DVDs. Most professional video DVDs are stamped.

Re:DVDs only live for 7 years (2)

Em Adespoton (792954) | about a year ago | (#44365451)

You never know ... when you will discover errors in your digital copy. DVDs are not born analog. In fact the only have a shelf life of around 7 years. You need to get everything off DVDs and make several digital copies of it. You should keep the DVDs as long as possible but eventually you will not be able to read them anymore. Make sure your digital copies of the DVDs are error-free because there will come a time when you cannot go back to the DVDs.

Hmm... DVDs only live for 7 years, eh? In an archive?

I was just watching a DVD last night that I bought in 2000; it still works fine, with no scratches or degradation. I was also pulling data off a DVD-R the other day that I recorded in 2003. This DID have a slight bit of degradation, so maybe there's an issue here. Never had a problem with properly stored pressed DVDs though.

For that matter, I've still got 5 1/4" floppy disks that have readable data on them from 198, and Audio CDs from 1990. Got rid of all my cassette tapes though; both the digital and analog ones degraded really quickly with use.

Re:DVDs only live for 7 years (1)

Russ1642 (1087959) | about a year ago | (#44365523)

Well we all know that floppy's from 198 were made of stone so of course they're going to last.

Re:DVDs only live for 7 years (0)

Anonymous Coward | about a year ago | (#44366049)

I'd love to see the stack of digital cassette tapes though...

Re:DVDs only live for 7 years (0)

Anonymous Coward | about a year ago | (#44367065)

They look like standard audio cassette tapes (at least in some incarnations [wikipedia.org] ).

Re:DVDs only live for 7 years (1)

unitron (5733) | about a year ago | (#44368493)

By 198 I think they were already up to parchment and sheepskin, although papyrus hadn't totally fallen out of favor at that point in some parts of the world.

Re:DVDs only live for 7 years (1)

SuricouRaven (1897204) | about a year ago | (#44365847)

The lifespan of a burned DVD is highly unpredictable. Some can last decades - but don't count on it.

Re:DVDs only live for 7 years (0)

Anonymous Coward | about a year ago | (#44366091)

And I have or had commercial DVDs from 1994 that were beginning to show visible cracks and had become unreadable in 2001 even though they had been stored in dark, cool places normally considered ideal for preserving them.

Lesson: Your mileage may vary.

Re:DVDs only live for 7 years (0)

Anonymous Coward | about a year ago | (#44367081)

Can I borrow the time machine you used to get DVDs in 1994? They were invented in 1995. They became commercially available in Japan in 1996, and in the US in 1997.

Re:DVDs only live for 7 years (1)

Em Adespoton (792954) | about a year ago | (#44367745)

Can I borrow the time machine you used to get DVDs in 1994? They were invented in 1995. They became commercially available in Japan in 1996, and in the US in 1997.

Indeed... I figure the cracks were due to the stress of said time travel. When I started collecting DVDs in 2000, video rental stores hadn't even heard of them, and I had to order them online.

Re:DVDs only live for 7 years (1)

WillKemp (1338605) | about a year ago | (#44368777)

Not disputing what you say, but video disks were around in the early 80s. When i was working (as a games programmer) for Thorn EMI in London in 1982, they were working on interactive video disks. These were 10 inch disks as far as i remember.

Re:DVDs only live for 7 years (1)

ganjadude (952775) | about a year ago | (#44367553)

about 1/2 of my C64 and C128 5 1/4 floppies are still working in the C128 (the 2 64s i have no longer boot, i think a cap popped) the other half are unreadable, at least by the handful of 5 1/4 drives I own. Some of these disks are almost 30 years old. Kept in temp controlled room for the past 13 years (when i aquired them from a school). I cant say ive tested all disks (i have over 500 pounds in weight, just in floppy disks and sleves) but I have to admit that some disks that I know worked in the past are not workng as of 6 months ago when I had the urge to set everything back up. But the point im getting at is that there is no way DVDs, DVD-rs or even DVD-rws have a 7 year life.

Re:DVDs only live for 7 years (1)

Em Adespoton (792954) | about a year ago | (#44367653)

about 1/2 of my C64 and C128 5 1/4 floppies are still working in the C128 (the 2 64s i have no longer boot, i think a cap popped) the other half are unreadable, at least by the handful of 5 1/4 drives I own. Some of these disks are almost 30 years old. Kept in temp controlled room for the past 13 years (when i aquired them from a school). I cant say ive tested all disks (i have over 500 pounds in weight, just in floppy disks and sleves) but I have to admit that some disks that I know worked in the past are not workng as of 6 months ago when I had the urge to set everything back up. But the point im getting at is that there is no way DVDs, DVD-rs or even DVD-rws have a 7 year life.

I was just watching a DVD last night that I bought in 2000; it still works fine, with no scratches or degradation.

Commercially pressed DVDs are a different beast than the ones you write yourself. The foil is etched and pressed into the plastic instead of inks being hit by lasers after assembly. The ones you write yourself seem to have about a 7-10 year life if you treat them well.

Re:DVDs only live for 7 years (3, Insightful)

PRMan (959735) | about a year ago | (#44365669)

Not this tired argument again. I have burned CDs from 1995 that still work perfectly fine. Sure, they "estimated" that they would only last 7 years. Guess they were wrong, since unless I can see physical scratches or other damage, 99% of my discs from my life still work perfectly. The only ones that didn't last and had no physical damage were a cheap brand I got where the dye turned cloudy, but that happened within the first 2 years.

Re:DVDs only live for 7 years (1)

cusco (717999) | about a year ago | (#44366039)

Don't leave them laying in the sun. I was able to say the same thing until I left a burned CD from the late '90s on the window sill a couple of years ago. Four of the five CD players that I put it in couldn't read it at all, and the fifth could only make sense of about half of it.

Re:DVDs only live for 7 years (5, Interesting)

Miamicanes (730264) | about a year ago | (#44366163)

> I have burned CDs from 1995 that still work perfectly fine.

If they're redbook audio CDs, and your definition of "work perfectly fine" is "I can stick the disk in, hit play, it spins up at 1X, music comes out, and the player doesn't totally gag", you might be right. Now try ripping the disc using software that can monitor the realtime bit error rate. You'll probably be *horrified* to see how high it is.

Redbook audio CDs are very robust, even when their bits are rotting all over the place. They were designed in an era when hardware couldn't do much in realtime, so they bent over backwards to make sure they had a "plan B" to make sure the show would go on after the disc got scratched, dirty, or whatever else happened to it. They were designed so the audio data is interleaved in a way that when a read error occurs, the left and right channels get merged for 1 sample. A redbook audio CD has to be nearly *destroyed* (cracked, melted, fried, whatever) before it literally won't play, as long as the player is able to find the lead-in and sync up to the spiral track.

It'll start to sound "rough" and lose channel separation, but things have to be pretty bad before it will LITERALLY stop playing. At least, as long as the player itself is faithfully following the original redbook audio specs, and isn't trying to realtime-rip the audio to a ram buffer and play it back from there (which is what some, if not most, new optical-disc media players do TODAY). I have plenty of CDs that new players choke on and refuse to even try playing, but yet my 25 year old antique CD player that cost something outrageous like $600 or $800 when new, can play just fine. Apparently, it's because first-generation CD players were precision hardware that could blindly track a CD spiral as long as the disc itself was 100% within spec, whereas new players depend upon realtime error-analysis to stumble and wobble around, and make up for the fact that discs no longer spin precisely, and worm-gear optical assemblies no longer track with precision measured in microns.

That said, my experience has ALSO been that CD-R discs manufactured in THIS century are less likely to rot and become unplayable in new drives, but are more likely to have major problems with old players. The old players were precision hardware, and assumed the discs themselves were manufactured to precision specs. The first-gen CD-R media had dye that deteriorated over time, but their spiral tracks were spot-on, just like pressed discs. As drives got better at handling sloppy tracking, the discs themselves became sloppier.

Net effect: first-gen redbook audio CD-R media is likely to play with acceptable audio quality on an old CD player from the 80s or early 90s, but be unplayable on many modern drives & be un-rippable on most drives (some will allow you to spin down to 1X & emulate the playback mode of a legacy player if you're running a sophisticated ripping app). Newer discs that are still old will probably skip and have problems playing on an old player, but might still be equally bad on a new one. When today's bargain-bin CD-R media is 10 years old, it will probably be unplayable on anything, the same way my old VHS tapes from the 80s still play fine, but VHS tapes recorded after ~1998 are largely unplayable on anything I can find.

TLDR point: the storage life of "last-gen" CD-R media is likely to be better than first-gen CD-R media was at the same age, but enormously WORSE than that of the best "turn of the century" CD-R media (the golden era when quality standards were still high, and the worst faults of the first-gen media were addressed. Any box of CD-R media you buy TODAY is probably shit of the worst kind. The best media you can buy TODAY for long storage life? Non-LTH BD-R single-layer discs. But MAKE SURE they aren't LTH... most manufacturers don't go out of their way to scream, "These discs are LTH garbage!"

Re:DVDs only live for 7 years (2)

vux984 (928602) | about a year ago | (#44366299)

You should keep the DVDs as long as possible

Bottom line, if you have N digital copies then what is the benefit of keeping the original DVD over one N+1 digital copies of the DVD?

Near as I can tell. Zero benefit. And massively increased storage requirements. So make one extra digital archive and discard them. Better still donate them! to public libraries? independent / private archivists? You don't have to "destroy" them -- which is surely about as counter-instinctual as it gets for an archivist. :)

eventually you will not be able to read them anymore

Odds are that if there were errors reading from it today, you won't get a better copy from that disc 50 years from now. Better to make copies from 2 different discs or exchange back ups from another center. 2 different rips of the same disc is better than 2 copies of a rip from the same disc in terms of ever being able to restore missing information from a rip.

Re:DVDs only live for 7 years (1)

vux984 (928602) | about a year ago | (#44366321)

er... "2 different rips of the same disc"

I actually meant rips of the same title from different discs.

Re:DVDs only live for 7 years (1)

Miamicanes (730264) | about a year ago | (#44367941)

> For born-analog content, always keep the original physical copy.

And if, for whatever reason, you CAN'T retain the original copy, oversample the bejesus out of it, store it with the most lossless compression you can, and sample it multiple times, in multiple ways, with multiple scanners/digitizers/capture cards.

Case in point: videotape. If your goal is to merely capture a copy to casually watch years later, and you don't mind having a digital copy that's demonstrably worse than the analog original, just about any cheap one-click capture card will do. BUT... if you want to stop the clock and capture a copy that doesn't visibly look worse than the original, compress it to MPEG-2 it at a MINIMUM of 6,000kbps VBR, at 720x480 (or 540, if it's PAL). Yes, we all know that VHS is low-res. The problem is, it's also noisy, and noise doesn't compress well. If a 720x480 video encoded with VBR and the maximum DVD-compliant bitrate available looks as good as the original VHS, consider yourself LUCKY. Regardless, never downsample the vertical resolution. Horizontal resolution with VHS is open to debate, but your vertical resolution is known and definite -- 480 (525) if it's NTSC (or 60hz PAL-J), and 540 (576) if it's PAL.

Now, we get into my #1 beef with Blu-Ray -- its despicable lack of compliant encoding options for 480i60 and 540i50 source, so there's really no way to profoundly oversample VHS, yet retain casual "stick in the disc and hit play" usability that can be watched directly in a Blu-Ray player, but someday used for restoration efforts.

If you REALLY want to capture a videotape and preserve every possible nuance for future restoration efforts before the tape degrades further, horizontal oversampling of VHS is mandatory. If Blu-Ray supported a hypothetical 1280x480 resolution with 4:4:4 chroma (remember, with VHS, your vertical resolution is your "good" resolution... it's 480 or 540. It's your horizontal color resolution that completely sucks. Chroma-encoding variants that split color between 2 scanlines will ALWAYS visibly-degrade VHS captures. Don't do it. )

In a perfect world, there would be a near-line archival video format designed for videotape captures that directly sampled the full-bandwidth FM signal coming off the tape (preferably, at 3 slightly different trackings... spot-on, plus one slightly "off" each way). A few times, I've tried to work out the math and figure out what kind of insane sample rate would be necessary to try magnetically sampling the tape straight-on, with multiple overlapping rows of microscopic read heads, and sufficiently-high density, to literally map the flux of every flake of oxidized metal on the tape and allow future offline reconstruction by digitally simulating VHS's diagonal read path, so that even if the original tape were destroyed, a future restoration could be performed against the digital copy. I have a hunch that something like this is actually do-able now (and probably would have been do-able a few years ago, had multi-terabyte hard drives been affordable), but I've never even read speculation online about such an archival device.

The point is, if your goal is to someday do color restoration, resolution-enhancement, or whatever, taking advantage of analog's tendency to keep surprising us with ghostly hints of lost detail that we'd always assumed were gone, you need a profoundly-oversampled copy with basically NO lossy compression (or oversampling so profound, you can downsample away the artifacts). Why do you need so much oversampling depth and detail? True hardcore restoration and after-the-fact re-synthesis is going to depend upon what are known as "higher-order artifacts", and those are precisely what get mangled by any kind of lossy compression. Remember, lossy compression is designed to throw away bits in areas where your eyes won't notice detail. An algorithm looking for subtle field-ripples as evidence that a pixel nearby is supposed to be intensely red instead of brownish-orange won't see them if some earlier compression algorithm threw away the rippled fuzzy noise and averaged the pixel out to brownish-orange.

The moral of the story: it takes more than mere framerate and pixel count to make Nyquist happy. Nyquist is a MINIMUM, not a LIMIT. If you're scanning movie film, and your final video doesn't look like you're viewing the film through a microscope and can clearly see every last crystal of grain, your resolution isn't likely to be high enough for future archival restoration efforts.

Re:DVDs only live for 7 years (0)

Anonymous Coward | about a year ago | (#44368187)

480/525 is correct for NTSC, but PAL has never been 540/576. It's 576 visible lines, 625 total. As such I suspect you've never actually used or been anywhere near it.

The contents, not the container (4, Insightful)

innocent_white_lamb (151825) | about a year ago | (#44365131)

Your interest is in the contents, not the container. Therefore, once you have a known-good copy of the data, you're all set.

Remember to keep a few of the old tapes/drives/whatever for the museum display, of course.

Re:The contents, not the container (2)

Rob the Bold (788862) | about a year ago | (#44365545)

Your interest is in the contents, not the container. Therefore, once you have a known-good copy of the data, you're all set.

Remember to keep a few of the old tapes/drives/whatever for the museum display, of course.

You might be interested in the "container" if it was itself interesting for some reason. A floppy disk owned by a particular person and labelled in their hand comes to mind as an example. Maybe something was crammed in the disk envelope with the disk. If it's of interest, you'd probably want to keep this piece of ephemera with the original item that contained it if it's safe to both items to do so.

OTOH . . . I would think that this would be avery unusual case.

Re:The contents, not the container (1)

Genda (560240) | about a year ago | (#44366003)

Indeed... an original floppy disk with a note penned by the hand of Galileo himself, might have real historical value ;-)

Re:The contents, not the container (1)

rk (6314) | about a year ago | (#44366181)

Nah, I have lots of those. Would you be interested in some? $9.99 each, but I'm running a special: 3 for $20!

Re:The contents, not the container (1)

skids (119237) | about a year ago | (#44367897)

Other things to consider are whether, on rewriteable media, the media may contain shadows of deleted data that may be of historical interest, and even on write-once media, whether the software you are using to copy it is copying everything it can, if there might be, e.g. stenography in a redbook CD Q channel.

I expect such concerns would only be relevent to certain special cases.

Re:The contents, not the container (1)

doti (966971) | about a year ago | (#44365865)

once you have a known-good copy of the data AND made a backup of that.

ALWAYS have at least two separated copies, peferably more.

Re:The contents, not the container (2)

NoobixCube (1133473) | about a year ago | (#44366847)

Responsibility for backups should be handled by the Department of Redundancy Department.

In my archivist job (4, Insightful)

Sparticus789 (2625955) | about a year ago | (#44365137)

I work on a team which does archiving. We have multiple layers of data storage. First, we keep all copies of media in a library. The media is imaged and stored on a SAN. The SAN is backed up to an off-site NAS. And once a year, we copy the data to hard drives and ship the drives to another site across the country. If you have the capability, put the originals in an archival storage area. I have never known a single archivist to get rid of anything, so you must be new to this community.

As an FYI, there is no such thing as obsolete media, as evidence by this project [loc.gov] . And trust me, you can usually find a way to image most old media formats.

Re:In my archivist job (2)

rogue_archivist (2994163) | about a year ago | (#44365349)

You make a good point about no medium being truly obsolete. As long as there's enough funding, that is. Also, archivists get rid of things all the time through deaccessioning or weeding, and with physical storage space always being at a premium, it prompted my question.

Re:In my archivist job (1)

Em Adespoton (792954) | about a year ago | (#44365381)

Just remember that imaging the bits is only half the equation -- you're also going to want to document the file format unless you really want someone to have to reverse engineer those .abx documents from 1985 from scratch every time they want to make sense of the contents.

Re:In my archivist job (1)

Spazmania (174582) | about a year ago | (#44365471)

Floppy disks go bad pretty quickly. Few of yours disks from 1993 still work. Tough to find any working disks from 1983. Unless there's something inherent to the disk itself (the "original" software with the artwork and sleeve) there's not a whole lot of point in keeping it after securing the data.

And God help you with tapes.

Hard disks have better longevity. If you can find a working PC-AT with a working MFM controller you can probably still boot that 40 meg drive from 1988. But... why? You can fit thousands of those on a modern thumb drive. Physical storage costs money. You can spend it better places than storing obsolete hard drives.

Re:In my archivist job (0)

Anonymous Coward | about a year ago | (#44365995)

Are you from an Eastern Bloc nation? Reasonably stored floppy disks last decades. I am a vintage computer collector and coming across unreadable disks is the exception, not the rule. I have literally hundreds of disks from 1983 or earlier and they all work fine and thousands from the later 80's through early 90's.

My original question was not tongue in cheek. Soviet-era floppies really only could be counted on to work for months to a couple years. That was not an inherent issue with the floppy disk though. I could understand that someone from that time and place who never experienced Western floppies could have that misconception to this day.

Re:In my archivist job (0)

Anonymous Coward | about a year ago | (#44366697)

I could understand that someone from that time and place who never experienced Western floppies could have that misconception to this day.

western floppies.

Re:In my archivist job (1)

Sparticus789 (2625955) | about a year ago | (#44367957)

You are just completely wrong. I have boxes and boxes full of software from pre-1993 that is readable and that I am able to collect data off of. Just the other day, I booted an Apple SE and read 20 different 3.5 inch floppy disks.

Re:In my archivist job (0)

Anonymous Coward | about a year ago | (#44367389)

There is a *true* digital archiving system under development.

Read the "BACKGROUND ART" section of US Patent 8,085,304.

image, don't discard. (1)

drabbih (820707) | about a year ago | (#44365143)

Old media will become obsolete and degrade ofer time. It is best to copy to modern media. The files should be stored based on their SHA hash code, so that duplicates need not be stored. You can't have too many copies.

Re:image, don't discard. (2)

mlts (1038732) | about a year ago | (#44365535)

What would be ideal is a file format that stores data with some error correction, so if a block got corrupted on older media, the corruption wouldn't just be detectable, but possibly correctable.

It isn't really "archival grade", but I've used the WinRAR utility for this. Archives made in 1999-2000 with error correction are still readable, check-able, and repairable, and can be moved from old CD-R to DVD to Blu-Ray, possibly to whatever the next generation of optical media will be. In fact, multi-volume archives that might have one CD or DVD go bad in a set are recoverable because I usually had one recovery volume for every four others, which might add 20% more disks to a set, but it seemed to be a fair compromise for restoring.

Analog media like photos? Keep. Who knows if there might be a better scanning technique to find more information from a photograph, similar to how one finds info about paintings.

Digital media? At least make a hash file that goes with the stored data at the minimum so corruption can be detected as the items pass to different storage media over time.

Re:image, don't discard. (1)

SuricouRaven (1897204) | about a year ago | (#44365753)

PAR2 files.

That's exactly what you need. I used to stick 200MB of par2 data onto every DVD-R I burned - if a file was found damaged years later, that was almost always enough to recover the lost blocks.

Re:image, don't discard. (1)

JesseMcDonald (536341) | about a year ago | (#44366051)

PAR2 files.

That will help if it's merely a bit of file contents which were lost, but what do you do if the error is in the filesystem metadata? You need to be able to access the filesystem to read the PAR2 files, along with the rest of the disc's content. If you lose an inode or superblock then a PAR2 file would be out of reach, even if you can recover the remaining blocks from the disk image.

Is there a way to add PAR2 data to a raw disc image while still allowing the disc to be read with standard tools?

Re:image, don't discard. (0)

Anonymous Coward | about a year ago | (#44366593)

dvdisaster

Re:image, don't discard. (1)

SuricouRaven (1897204) | about a year ago | (#44368043)

There's no need to keep the PAR2 files on the same media, but even if they are you can identify them by their magic bytes and recover them even if the filesystem is unuseable, so long as they are not fragmented. And once that's done, you can in turn use the par2 files list of slice hashes to conduct a brute-force search of the raw device for matching data. I wrote a program to do just that, many years ago.

Re: image, don't discard. (1)

nbritton (823086) | about a year ago | (#44366863)

Do we have a digital archival container format? I.e. a tar archive with built-in error correction to identify missing bits, lets say with a SHA hash, and a method to recompute the lost bits, lets say with an XOR operation? Do we have anything like that?

If you have a good checksum hash and known file size, but the data is corrupted, you could semi-randomly replace bits in a brute force fashion until you produced anther file with the same hash. The possiblity of reconstructing the bits in an alternate fashion that would produce a hash collision with the original and yield useable data is zero for all intents and purposes. If you segmented the file and computed hashes for each segment, you could do distributed parity XOR / Reed-Solomon calculations to recompute any bad segments.

I'd like to see a container format like this, given enough CPU cycles you should be able to recover just about anything, much like an infinite number of monkeys can produce the works of Shakespeare.

What is 'Obsolete,' Anyway? (1)

CanHasDIY (1672858) | about a year ago | (#44365163)

OK, keep in mind that I'm being rather abstract here:

What makes a thing obsolete? That it isn't a commonly used item anymore, or that its usefulness has become non-existent?

Take, for example, the carrier pigeon - once considered 'obsolete' due to the invention of telecommunications equipment, I can see the medium coming back into vogue in wake of the new knowledge that governments the world over are monitoring our every word over the aforementioned modern channels. Today, you can't send a message along electronic media without it being intercepted, somewhere, by someone other than the intended recipient; however, you can tie a coded message to a bird's leg and be reasonably confident in the message reaching it's intended recipient without interception and decoding (international and relay flights notwithstanding).

Thus, that which was obsolete becomes useful again, bringing us back to the initial philosophical quandary: What makes a thing obsolete, anyway?

Re:What is 'Obsolete,' Anyway? (1)

bonehead (6382) | about a year ago | (#44365341)

OK, that's all well and good, but what scenario do you propose that would make the 5 1/4" floppy disk a useful tool again?

Re:What is 'Obsolete,' Anyway? (4, Funny)

CanHasDIY (1672858) | about a year ago | (#44365395)

OK, that's all well and good, but what scenario do you propose that would make the 5 1/4" floppy disk a useful tool again?

Wobbly tables.

Re:What is 'Obsolete,' Anyway? (1)

bonehead (6382) | about a year ago | (#44365547)

That's what parking tickets are for.

Re:What is 'Obsolete,' Anyway? (1)

CanHasDIY (1672858) | about a year ago | (#44365657)

That's what parking tickets are for.

Hmm.. shuriken, perhaps?

Some neat ideas here [crookedbrains.net]

Re:What is 'Obsolete,' Anyway? (0)

Anonymous Coward | about a year ago | (#44365789)

Not if you like to be barefoot... or your child does... or your dog...

Re:What is 'Obsolete,' Anyway? (1)

lightBearer (2692183) | about a year ago | (#44365357)

In addition to the point about carrier pigeons, even modern technology is re-embracing what others might consider to be obsolete.

See: IPoAC [ietf.org]

Re:What is 'Obsolete,' Anyway? (1)

CanHasDIY (1672858) | about a year ago | (#44365663)

Ah, yes, a classic.

That one ranks up with the internet-enabled toaster.

Re:What is 'Obsolete,' Anyway? (0)

operagost (62405) | about a year ago | (#44365573)

however, you can tie a coded message to a bird's leg and be reasonably confident in the message reaching it's intended recipient without interception and decoding (international and relay flights notwithstanding).

And drones... anyone for squab?

Re:What is 'Obsolete,' Anyway? (1)

sabinelr (1061112) | about a year ago | (#44365689)

Used to be you could find piles of old asphalt roofing lying around and you could make flying saucers with it. Demands much more proficiency than Frisbees. 5-1/4 floppies are equally good for this, with the added menace of the sharp corners. 3-1/2 floppies might work too. MFM drives are good targets on the shooting range, and you have the added benefit of erasing sensitive data.

Take a picture. (0)

Anonymous Coward | about a year ago | (#44365199)

Scan the container for labels and nostalgia. Keep a few samples. Hope you have solid backup policies, and you test your backups. Otherwise, well....

Another major problem is reading the original file format later. Or even that some media (forth floppies) come without an actual file system. Archivists have been working on that too. So instead of (or, in addition to) asking a bunch of nerds, see what your fellow professionals have been able to come up with.

Also, "media" is plural, thanks.

if you don't care about the content (4, Insightful)

gl4ss (559668) | about a year ago | (#44365201)

..or can check all of the content to be perfectly read, then yeah, sure, no loss in destroying the originals.

however.. if you have the space, why destroy? another issue is sw where you in theory might have to prove ownership of a legit copy or the originals might have some other curiosity value. another thing with paper records is that if you destroy the old ones, what was stopping you from introducing new data like a record for your uncles graduation from said university and with you having destroyed the paper records no way to go check them.

so my question is, is it really that expensive to store them, just for posterity's sake? even then you could just destroy them via sloppy storage rather than intentionally burning energy for destroying them..

Re:if you don't care about the content (2)

TubeSteak (669689) | about a year ago | (#44365721)

so my question is, is it really that expensive to store them, just for posterity's sake? even then you could just destroy them via sloppy storage rather than intentionally burning energy for destroying them..

There's no practical difference between an item you can't find and an item that's been destroyed.

So in reality, you don't even have to destroy the stored items, just go ahead and lose the manifest.

I don't get it. (0)

Anonymous Coward | about a year ago | (#44365219)

Then the physical media is separated from archival paper documents and placed into storage

What does that mean?

Those DVDs you burn are stored and the paper is ....what?

If you really MUST archive stuff, then store it in multiple media - paper, DVD, original, etc ...

For example, if I were archiving Da Vinci's paintings, I'd keep the original, photograph the original in the highest def digital camera I can get, and photograph it in film - preferably slide film because then you don't have to worry about the second layer. (analog sucks for archive, btw.), and have the most talented copy artist ever dupe it.

Re:I don't get it. (1)

walshy007 (906710) | about a year ago | (#44365849)

Why does analog suck for archiving? sure you can't just get a hash of the data and tell at a moments notice whether it is exactly as it was, however you also can't store a hard disk in a vault for 70 years and have a high expectation of it working.

Why not? (1)

jedidiah (1196) | about a year ago | (#44365221)

Unless you are paying Manhattan real estate prices, why not keep the originals? They serve as another backup. They will likely not be too much of a burden. Most "obsolete" media is still perfectly usable and may be so for quite some time.

There is simply no need to rush into destroying something you already have and can serve as an alternate form of backup.

Originals always have some value in being the definitive version of something.

For display along side the files... (1)

HockeyPuck (141947) | about a year ago | (#44365247)

It's always interesting to see the files and what they were kept on. Floppy disks, whether 3, 5 or 9in variety. Old tape reels, large disk platters... "This file took up 3 of these..." or..

An entire windowing system (macos) PLUS MS-Word fit on two floppy disks.

My phone currently has more storage than the enterprise datacenter that I used to work at in the 80s. And it was a LARGE datacenter...

The Medium Can Hold Secrets (5, Interesting)

TechyImmigrant (175943) | about a year ago | (#44365281)

If you had a 1979 copy of Wizardry on an Apple ][ floppy disk, you could images the contents. But if you wrote them back to a disk and tried to run it, it would fail.

This is because as a means of copy protection, Wizardry used track arcing. Part of a track was written on a track. Another partial track was written half a head-step away. The timing of the writes was synchronized so the partial tracks didn't overwrite. Anyone doing a naive read and write, or even a not-so-naive scan of the half tracks would fail, because they would get the timing of the writes necessary to prevent collision and to meet the consistency checks in the program.

Obviously people reverse engineered this and wrote adaptive copy programs that you could direct to do the right thing, but how is an archivist going to know that?

If you can get this level of deviousness on a primitive floppy disk, I imagine that there is plenty of deviousness to go around on other formats.

Keep the media.

Re:The Medium Can Hold Secrets (1)

rogue_archivist (2994163) | about a year ago | (#44365747)

That kind of creative copy-protection is infamous, but for largely institutional or personal records, most people wouldn't have gone to that level of trouble just to obfuscate data. If they had, then we may miss out on some hidden data, but I don't think that argument alone can justify keeping around any type of storage medium that may have a timing trick, hidden filesystem, or other form of protection. At the same time, folks have developed functional technology [kryoflux.com] to record timing information as well as bitstreams and sector information when creating a disk image, so if there were the type of tricks you describe they could be reverse-engineered later.

MOD THAT UP! Re:The Medium Can Hold Secrets (2)

Ungrounded Lightning (62228) | about a year ago | (#44365813)

I have no mod points at the moment. But that's a VERY important point: A straight copy may not be good enough, due to outside-the-standards copy protection schemes.

Other floppy-based commercial games used a number of other techniques.

(One, for instance, had track 3 deliberately corrupted, by scratching the medium with a pin. No error on reading it - or writing and re-reading it - and the game would load, erase the disk, and play. This let the person who made the copy think he had a good copy - when in fact he had a blank disk. Let's see you make a good archival copy of THAT. B-b )

You get the same thing on other media as well - even analog. (Example: Macrovision, which plays with the sync and saturation levels, so that analog TVs intended for over-the-air reception (usually) correct the distortion as if it were a fading signal, while videotape machines copy the "fading" picture and regenerate a non-fading sync, so the copy isn't corrected when viewed.)

One of the several copy protection schemes for DVDs includes hidden modulation in sync information, decoded by the drive's hardware and detected by its firmware, so you can make a perfect copy of the bits and it still won't play.

Wikipedia has a long list of such copy-protection schemes, any of which would make archival copying difficult to impossible (without special equipment that would expose you to arrest and federal prosecution if you possessed it).

Re:The Medium Can Hold Secrets (0)

Anonymous Coward | about a year ago | (#44366219)

Ahhh. Those were the days. Wizardry had one of the toughest copy protection schemes out there. Sadly, even my 3.5" disks from the late 90's were unreadable about 5 years ago. If I still had my 5" disks I doubt there's be much left.

If you can get this level of deviousness on a primitive floppy disk, I imagine that there is plenty of deviousness to go around on other formats.

The primitiveness of Apple's floppy drive was the key to much of the copy protection used. Everything was software controlled. The track servos as you mentioned, but also the bit encoding/decoding was done entirely in software. Later systems with better hardware abstraction are much easier to copy.

I doubt copy protection or strange encoding plays a roll in this Ask Slashdot.

Re:The Medium Can Hold Secrets (1)

tibit (1762298) | about a year ago | (#44367885)

Other formats are usually readable by a device with fairly closed firmware that is designed to spec. Nobody gives you a CD-ROM drive with access to head positioning servo loop and access to the raw bitstream after clock recovery. Yet this is almost what you has on Apple ][ - that's why the floppy controller hardware was so simple (a couple stock TTL chips, maybe a PAL or two). The magic was in the software (firmware). Same goes for a modern CD-ROM drive, but the firmware is not really easily amenable to hacking, and you can't really replace it as a matter of normal operation of the equipment. On Apple ][ it was trivial, since the firmware ran on the main CPU. On a CD-ROM, or really any other media access device, there's a dedicated CPU - these days it's most often a proprietary SoC with documentation available under an NDA only if you commit to a certain purchase volume. Never mind that the amount of code is orders of magnitude beyond what was routine in Apple ][ days.

ask the duraspace.org mailing list (1)

Anonymous Coward | about a year ago | (#44365287)

I implemented that archive system and I got great support, knowledge and experiance from their community : https://wiki.duraspace.org/display/DSPACE/Discussion

digitize your Gutenberg Bible & toss the origi (0)

Anonymous Coward | about a year ago | (#44365345)

In the same way a Gutenberg Bible has something a modern reprint can never match...

loading an 8" floppy into a drive and waiting several minutes to access a text file has something a file on a NAS can never match.

Not all old documents should be preserved in their original format if they are duplicated elsewhere, but a representative sample of each generation should be kept for posterity. Of course, idiots will damage those 8 inch floppies over the years. So, when in doubt, save more than you will ever need.

Readable disks are far more useful than museum-relics that can be displayed but not used.

I am constantly amazed at how well (very) old computers work. My Grid Pad 1 (early laptop) boots just fine.

Re:digitize your Gutenberg Bible & toss the or (0)

Anonymous Coward | about a year ago | (#44365479)

"loading an 8" floppy into a drive and waiting several minutes to access a text file has something a file on a NAS can never match."
"when in doubt, save more than you will ever need."
"Readable disks are far more useful than museum-relics that can be displayed but not used."

Fuck Off

I've dealt with imaging thousands of floppies and they are all shit and deserve to be in the landfill. The same goes for mag tape.
The sooner the shelves are cleared of these, the better.
The DATA is what you care about, not the carrier.

Re:digitize your Gutenberg Bible & toss the or (0)

Anonymous Coward | about a year ago | (#44365927)

loading an 8" floppy into a drive and waiting several minutes to access a text file has something a file on a NAS can never match.

You obviously don't know how shitty my NAS is... :-(

From one archivist to another.... (1)

Anonymous Coward | about a year ago | (#44365439)

Get rid of the darn things. Make sure you have the proper emulation and other tools you need, be sure to reformat, but absolutely get rid of the disks. They will fail (magnetic impulses cannot be captured forever) and you will be left with a goodly-sized stock of unreadable media (in fact, IIRC, latest NDSA suggestions are to remove all files from optical media ASAP). Save yourself the trouble and the expense and dump them from the start.

They won't last (1)

nine-times (778537) | about a year ago | (#44365465)

I don't know what your goals and requirements are, but I wouldn't bet on old floppies, CDs, or even hard drives lasting for very long. There's an essential problem with old physical media in that the readers are becoming more scarce. You may have a lot of floppies, but how easy is it to find a floppy drive? It's not always easy to find adapters for old IDE or SCSI formats as newer interfaces have been developed. Personally, I don't expect CD/DVD drives to be around in 10 years.

But beyond that, there's an even bigger issue: media goes bad. Of course, how quickly it goes bad depends on quite a few things, including how it was manufactured, and how it's stored. Even if you store a bunch of CDs and floppies under good conditions, I'd expect at least 10% to go bad within 6 years. I'm completely pulling that number out of my ass and I have no science to back me up, but my point is, this stuff is not reliable. I think my 10% number is too low, even, but I'm trying to make sure I don't exaggerate.

Re:They won't last (1)

PRMan (959735) | about a year ago | (#44365883)

I have never had a CD "go bad" except for one cheap batch I got where the dye turned cloudy within a year. Every other CD-R in my possession still reads perfectly unless it is obviously damaged.

Re:They won't last (1)

jedidiah (1196) | about a year ago | (#44365889)

Then save your appetite for destruction for when the original media is genuinely unreadable. There is no need to hasten that which is only perceived as inevitable.

Trash it once it actually is trash.

Don't bother until then.

I've stored CDs in pretty harsh ways and managed to get far more than 6 years of shelf life out of them. They aren't nearly that fragile. Some are subject to manufacturing defects but that's another sort of problem and it's hardly universal.

Re:They won't last (2)

hedwards (940851) | about a year ago | (#44365969)

I disagree, keeping the original only makes sense when the original is in a stable format and you have plenty of room.

I've been dealing with this problem on a much smaller scale, and if you aren't extremely careful it can be hard to keep track of which disks you're keeping because you can, and which ones you're keeping because you have to.

Dump it to disk, verify the contents, back it up and chuck the original media. In the long term, 1 CDROM is going to last better than 400 or so floppies will.

Now, if you're dealing with paper, those tend to be incredibly durable provided decent paper and ink was used, those you're generally best keeping if you're archiving and have space.

Re:They won't last (0)

Anonymous Coward | about a year ago | (#44367901)

This attitude is what leads to hoarding. It's creating a whole raft of problems, while solving none.

the data is what you care about - arguing that "we must keep the disks as well!" is like saying "we must have all the fragments of marble that were chipped off the block of marble that Michelangelo carved David out of!"

Don't confuse the useless crap with the content.

If you are 100% you have everything off of it. (1)

WarlockD (623872) | about a year ago | (#44365475)

I was recently did the same thing. I had about 2 old OnStream 30 GiG tapes and a hand full of old QIC-80's. Not even mention the CD-R pile in my room.

During the years I never had the space to just extract everything and sort though it all. Not to mention I would move backup data from tape, to CD, back to tape so I have copy's of the same things all over the spectrum. I have recently started consolidating it all, finding an old OnStream tape drive and old QIC floppy drives to restore everything to a single drive, get rid of all the duplicates and save the important stuff on archived DVD media and/or "the cloud" It was a nightmare but now I don't have to worry about trying to get hold of a bankrupt tape drive company's hardware in another 10 years.

I will then delete the tapes and burn them.

If it was hand labeled by a professor he liked or someone famous like Bill Gates I could see that. But there is no other reason to keep it around once it's contents have been properly indexed and stored. The only exception is when you need the obsolete media to be used in another obsolete computer. AKA making a disk for a cC64.

Let me put it this way. What do you think will happen in 10 years, when someone else finds that box of media. Even if he was told that it was all indexed and stored, he might question it and do it all over again "just to be sure" :P

Re:If you are 100% you have everything off of it. (1)

bluefoxlucid (723572) | about a year ago | (#44365513)

He thinks in 10 years he'll have a Hoarder's Crisis.

Re:If you are 100% you have everything off of it. (1)

hedwards (940851) | about a year ago | (#44366001)

I'm in a similar situation, just not to that extent. One of the tricks is to categorize things and decide roughly where things belong before you start. Then move things to the correct place, verify the copy and back up. After that, I generally destroy the original, especially if it's in a weird format. (And yes, I consider 3.5" floppies to be a weird format, and really anything other than CDROM or DVDROM at this point)

The Medium Can Have As Much Value... (0)

Anonymous Coward | about a year ago | (#44365577)

...as the contents.

Depending on what the contents are, and your reasons for keeping it, the medium may be just as valuable. Or, said another way, the contents may lose their value when divorced from the medium.

For example, I'm thinking specifically of old copies of MacOS. The primary reason for keeping old versions of MacOS around would be to boot old Macs. If you discard the medium, you'll never boot that hardware again.

And before you jump and say "I'll just write another copy", it's nowhere near that simple. Original Mac drives spun the discs at variable speeds, while PC drives spun at a fixed rate. You cannot write a Mac floppy disk with a "modern" commodity floppy drive. If you can get your hands on an image (the contents), it will still require multiple generations of Mac hardware and software to backtrack far enough to write a usable physical copy.

Of course, a fair rebuttal here is "Why would you want to boot an original Mac?", and for that I have no other answer besides "Nostalgia".

Re:The Medium Can Have As Much Value... (1)

hedwards (940851) | about a year ago | (#44366021)

That's good in theory, but it's unlikely that your version of MacOS from the '80s is still going to work. Floppies are terrible in terms of reliability over long periods of time.

image, verify replicate job done (1)

maliqua (1316471) | about a year ago | (#44365639)

see title

the app store ideas and apples lack of ports is ba (1)

Joe_Dragon (2206452) | about a year ago | (#44366407)

the app store ideas and apples lack of ports is bad for archiving.

We may get to the point where the app store sand boxing makes it so that an archiving app can't put the files in an place where other apps can see it. and we may have hard time reading an outside data source as well.

Keep software and hardware to read it (1)

jfdavis668 (1414919) | about a year ago | (#44366413)

Though it is not my primary business, I offer my services when people have difficulties accessing archives. I am often surprised when people come to me to rescue data off floppies, both 3 1/2 and 5 1/4. These are mostly legal documents or contracts. Why people keep floppies but not drives to read them is beyond me. I have a 3 1/2 USB drive I keep for routine work. I saved a Pentium 90 with a 5 1/4 inch at home I use for those rare occasions. The other problem is reading the data. I have Office 97 on the Pentium to convert and read old formats. I have another machine with Windows and Office 2000 to bring documents to somewhat modern formats. I helped one company update their union contract. They had it printed as a small book, and had been giving it out for 20 years. When they finally ran out of copies, they wanted to incorporate the changes over the years and reprint it. They handed me a pile of floppies. Each chapter was a separate document, which I finally figured out were in Wordstar for DOS format. Luckily, there is an Office 97 converter for that. The lesson is, without software to read it, your archives are useless to keep. Save old versions of your software, and if necessary, hardware to run it.

Re:Keep software and hardware to read it (1)

tibit (1762298) | about a year ago | (#44368031)

Wordstar is pretty much a text file. All you need to do is mask out the upper bit. You'll have some control characters left, but those are easy to remove. That's what I remember from 2.5 decades ago...

The Basics (4, Informative)

westlake (615356) | about a year ago | (#44366539)

I'm an archivist at a mid-sized university archives, trying to develop a policy for archiving computer files ('born-digital records' in archival parlance).

Get Your Bits Off (Old Storage Media) [loc.gov]

Demystifying Born Digital Reports [oclc.org]

Working Draft of the Levels of Digital Preservation Chart [loc.gov]

Keep a sample (2)

Peter (Professor) Fo (956906) | about a year ago | (#44366845)

The actuality of bit-rot in media is uncertain. Many documents 500 years old are readable-ish if you have the skills and accept that some parts may have decayed. That tells us a lot about te exact media people used way back then.

The trouble with digital records is this:-
Searchability is a requirement (even though we don't expect that with written records). The reason is that there is so much of it when compared with the sparse records of times past. So you need a 'good' copy for data analysis and some original media to inform historians of the future how we looked upon the information, or what 'ordinary people' or 'ordinary businesses' had at their disposal.

DVDs are inherently flaky (1)

gatortom (2808629) | about a year ago | (#44366851)

DVDs, even commercially stamped, can suffer from bit rot. Optical disk technology is inherently flaky. Use multiple HD backups and make sure you have offsite storage.

Don't forget the physical data (2)

Tim99 (984437) | about a year ago | (#44366919)

One problem about a purely digital archive copy of an original media item is that the information that has been physically written onto the item (handwriting, or a printed label, or artwork on a DVD etc) may not be archived. In some cases this information has little value - On the other hand, it may have great historical significance, like a handwritten note on a DVD by someone who is later awarded a Nobel Prize. You as an archivist do not necessarily know what will be important, and you cannot retrieve this after the item has been destroyed.

A compromise that may be acceptable is that you digitise photo-images of the appearance of the item and ensure that this information always remains associated with the digitised content.

Re:Don't forget the physical data (1)

magic maverick (2615475) | about a year ago | (#44368401)

This. I logged in to say this.

An archivist should keep the original as much as possible. Otherwise, what's the point? Would it be acceptable to photocopy a letter and than discard the original because it's too old? No. You photocopy (or actually non-destructively scan these days), and then you keep the original in a climate controlled environment. People can work off the copy, but sooner or later someone will want to look at the original.

At a minimum you take high-resolution photos of the media and record those photos (and the type of media, etc.) along with the copy of that you are putting into your fancy database.

What's the point of archiving anything? It's too keep it for posterity.

Geeze, what do they teach archivists these days...

Disclaimer: I'm not an archivist, but I have done a lot of study and work in the digital sector (including archiving).

Archiving Digital Data (0)

Anonymous Coward | about a year ago | (#44367119)

The real question is how long do you need this data to be archived.
If it more than a few year, print is out, and make a mico-fiche of it.
From where I stand, digital data only has a life of a few years.

Keep them or pay the price. (0)

Anonymous Coward | about a year ago | (#44367209)

First, you may have legal issues. You may not be allowed to copy the data but you may also someday prove that you did buy the 'data' and show proof of
the original. This is a big fear for anyone that has had the Internet Nazis, err The Media Industry come after them.

Print out the best prints and store the prints (1)

jools33 (252092) | about a year ago | (#44368385)

As a photographer myself I would recommend to print out the best choice prints and store them physically, as photographic prints still has the best record for preservation when compared to any / all types of digital media. By all means take running copies of all your data, on and offsite backup. A physical copy of the best prints though is likely to be preserved longer.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?