Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

27 Billion Gigabytes to be Archived by 2010

timothy posted more than 6 years ago | from the if-not-sooner dept.

Data Storage 178

Lucas123 writes "According to a Computerworld survey of IT managers, data storage projects are the No. 2 project priority for corporations in 2008, up from No. 4 in 2007. IT teams are looking into clustered architectures and centralized storage-area networks as one way to control capacity growth, shifting away from big-iron storage and custom applications. The reason for the data avalanche? Archive data. In the private sector alone electronic archives will take up 27,000 petabytes (27 billion gigabytes) by 2010. E-mail growth accounts for much of that figure."

cancel ×

178 comments

Sorry! There are no comments related to the filter you selected.

We have the prefixes, why not use them? (5, Informative)

Valacosa (863657) | more than 6 years ago | (#21877100)

In other words, 27 Exabytes?

Note to science and tech journalists: please stop stringing together "millions" and "billions" in an attempt to make the numbers seem large, impressive, and incomprehensible. Scientific notation and SI exist for a reason.

Re:We have the prefixes, why not use them? (1, Insightful)

N Nomad (1198231) | more than 6 years ago | (#21877290)

Does it bother you that much that these journalists want to make it easier for the general public to understand how big data storage they are talking about? please, get off your high horse, nerd. Find something better to do with yourself.

Re:We have the prefixes, why not use them? (5, Insightful)

thomasdz (178114) | more than 6 years ago | (#21877336)

Yeah, but before the 1985 "Back to the Future" movie came out, how many "general public" people knew the prefix "Giga"? That's when I started hearing regular people start to use it.
We gotta start using the prefixes before they start to become common. I'd rather see "27 Exabytes" followed by a parenthetical comment saying (27 Billion GigaBytes)

Re:We have the prefixes, why not use them? (1)

redalien (711170) | more than 6 years ago | (#21877398)

I'd settle for knowing if they mean 27 exabytes or 27 exbibytes

Re:We have the prefixes, why not use them? (5, Insightful)

mdwh2 (535323) | more than 6 years ago | (#21877864)

Yes, but in Back to the Future, there wasn't a real need to explain how large "giga" really was, it was just there as a scientific-sounding buzzword. So whilst using the term in this article might have made people become familiar with the word, they wouldn't have any idea what size it actually meant.

People didn't become familiar with Gigabyte because of Back to the Future anyway, they are familiar with it because that's what they now buy hard drives and ipods in. When they are sold in Exabytes, you'll see the term used in journalism too.

Re:We have the prefixes, why not use them? (1)

dasmoo (1052358) | more than 6 years ago | (#21878608)

I'm fairly sure he was saying 1.21 Jigawatts anyway

Re:We have the prefixes, why not use them? (1)

callmetheraven (711291) | more than 6 years ago | (#21877892)

Only we all thought they were "Jigawatts," a word that sounds politically incorrect but is actually meaningless.

Re:We have the prefixes, why not use them? (0)

Anonymous Coward | more than 6 years ago | (#21877352)

You must be new here.

Re:We have the prefixes, why not use them? (1, Informative)

Anonymous Coward | more than 6 years ago | (#21877462)

I'm no nerd (well I guess I'm a wannabe nerd since I'm reading Slashdot) and true, I wouldn't have known how much an exabyte is. But a billion is such a large number that I can't really comprehend that either. I agree with the op though that exabyte should have been used.

kilobyte
megabyte
gigabyte
terabyte
petabyte
exabyte

Seeing it like that, when you can relate it to the other ones, makes it easier to understand than "a billion, gajillion, fafillion bytes!"

will someone think of the kids! (5, Funny)

metamorfoza (995978) | more than 6 years ago | (#21877930)

Does it bother you that much that these journalists want to make it easier for the general public to understand how big data storage they are talking about?

I agree. However, I would go even further and instead of using geekish bytes and bits we should use something like 400 billions of mp3s. You know, so that myspace user out there can understand TFA. They clearly have interest in this sort of news.

Re:will someone think of the kids! (1)

fellip_nectar (777092) | more than 6 years ago | (#21878466)

Or, for the Slashdot reader, how many billion choked chickens of pr0n it equates to.

Re:We have the prefixes, why not use them? (0)

Anonymous Coward | more than 6 years ago | (#21878264)

Then why not just say 20615843020800 Floppy disks, or 42409734214.21 CD's

That would even be easier for the general public to understand.

Even easier, about 29686813949952 pornographic images.

Re:We have the prefixes, why not use them? (0, Troll)

Anonymous Coward | more than 6 years ago | (#21878392)

Slashdot isn't the general public, jack ass.

Re:We have the prefixes, why not use them? (4, Funny)

mincognito (839071) | more than 6 years ago | (#21877292)

Note to science and tech journalists: please stop stringing together "millions" and "billions" in an attempt to make the numbers seem large, impressive, and incomprehensible. Scientific notation and SI exist for a reason.
Exactly! For the thousandth time, let's cut out the exaggerated and sensational writing Slashdot! If I had a dollar for every sensational headline I've read here, not to mention the gazillion overstated comments I read here per day, I'd be a billionaire by now!

Re:We have the prefixes, why not use them? (5, Funny)

Anonymous Coward | more than 6 years ago | (#21877396)

No, you'd only be a thousand millionaire.

Re:We have the prefixes, why not use them? (0)

Anonymous Coward | more than 6 years ago | (#21877688)

Only?! He could withdraw ten million $100 bills from the bank!

Re:We have the prefixes, why not use them? (0)

Anonymous Coward | more than 6 years ago | (#21877796)

Exactly! For the thousandth time, let's cut out the exaggerated and sensational writing Slashdot! If I had a dollar for every sensational headline I've read here, not to mention the gazillion overstated comments I read here per day, I'd be a billionaire by now!

Twice over, no doubt.

Re:We have the prefixes, why not use them? (1)

platykurtic (1210910) | more than 6 years ago | (#21877964)

Are you sure you wouldn't be a million-thousandaire?

Re:We have the prefixes, why not use them? (1)

ILuvRamen (1026668) | more than 6 years ago | (#21878364)

but it sounds so much cooler. In fact, they should have counted it in bytes. I believe that would be 27 fuckingbigillion bytes. Now that sounds impressive!

Re:We have the prefixes, why not use them? (0, Redundant)

failedlogic (627314) | more than 6 years ago | (#21877304)

I've read reports that journalists have a million billion* words in their vocabularies. Exabyte seems to be one of the few missing.

*Sorry, had to. ;)

Re:We have the prefixes, why not use them? (4, Insightful)

phoebusQ (539940) | more than 6 years ago | (#21877308)

SI does exist for a reason: to allow for short, precise, descriptive, standardized measurements. However, the point of the numbers in this article is to show how absurdly large this amount of data really is. This isn't a scientific paper, it's a piece of journalism. In that case, there's nothing wrong with using numbers that aren't completely reduced to demonstrate scale.

nibbles! (1)

garlicbready (846542) | more than 6 years ago | (#21878074)

personally I prefer nibbles (4bits each or 1/2 a byte used with old parallel ports)
to make the numbers look bigger
working under the assumption of 1024 to the power of 6

2,305,843,009,213,693,952 nibbles of information
now that's a lot a chewin

Re:nibbles! (2, Funny)

broggyr (924379) | more than 6 years ago | (#21878752)

I thought half of a byte was called a nybble...

Re:We have the prefixes, why not use them? (1)

Yez70 (924200) | more than 6 years ago | (#21878388)

Journalists must write at an eighth grade level or the majority of their readers would not be able to understand them. Of course, for an arrogant intellect, such as yourself, maybe you should just stop reading so you can be happy.

a helpful reference page for large numbers (4, Interesting)

HappyEngineer (888000) | more than 6 years ago | (#21878534)

Here is my helpful reference page for big numbers [g42.org] . I love big numbers. I'm actually working on a site right now which will help people to visualize big numbers. I can't give out the url yet because it'll be another month or two before it's ready to be seen. But, it'll have many fun options like Cow Stacking and Hamster Canyon.

Cow stacking is where you select cow as the animal and from earth to moon as the place and you'll see a graphic of cows being stacked to the moon and the number of cows which would be required to complete that stack.

Hamster Canyon will be where you select a hamster and the Grand Canyon and you'll see a picture of the Grand Canyon filled with hamsters and a number that indicates the total number of hamsters required to fill the canyon.

Re:We have the prefixes, why not use them? (0)

Anonymous Coward | more than 6 years ago | (#21878880)

In other words, 27 Exabytes?

Note to science and tech journalists: please stop stringing together "millions" and "billions" in an attempt to make the numbers seem large, impressive, and incomprehensible. Scientific notation and SI exist for a reason.

Holy screaming shit -- that's more than one-quarter pornobyte.

So, in other words... (5, Interesting)

thesymbolicfrog (907527) | more than 6 years ago | (#21877102)

From the summary:
"E-mail growth accounts for much of that figure."

We're archiving spam?

Re:So, in other words... (4, Insightful)

4D6963 (933028) | more than 6 years ago | (#21877182)

We're archiving spam?

Which raises a question I find interesting, do we check for redundancy when archiving mails, in a way so that we can save a hell of a lot of space on spam (and other legitimate automated messages), since spam is by definition essentially the same message sent to a number of persons. Also, couldn't correlating stored mails for redundancy allow for better spam identification (although it would be no silver bullet since legitimate automated messages are often redundant).

Re:So, in other words... (2, Insightful)

Smordnys s'regrepsA (1160895) | more than 6 years ago | (#21877364)

Good Spamers uses multiple methods of fooling spam scans.

~They use pictures of text, instead of text, so it takes more effort to filter based on content.

~They use random text at the bottom of their message to give the filter something to read.

~They generate random noise to superimpose over the picture. Every batch has a different noise layer.


I'm sure they do more [IANASB - spam bot - so I wouldn't know the details] but the slight differences between what WE would perceive as the same message foil both the spam filters and your plan of reducing redundancy. If you find a way to implement your idea, please release it as FOSS! I'm sure you could get a Nobel Peace prize out of it, or at least some free (as in beer) drinks! :)

Re:So, in other words... (1)

4D6963 (933028) | more than 6 years ago | (#21877464)

OK so basically you're dismissing my entire idea (which was part a question, I mean why wouldn't it be done to a certain extent already?) just because some an unknown (by you and me) ratio of the spam data isn't redundant.

That would be kind of like saying "Why bother with implementing compressed file systems! Most people fill their disks with file that can't be significantly compressed anyways!". Sure, but you've still got millions of copies of the exact same Nigerian scams out there which are stored without any redundancy check, or so I presume.

Re:So, in other words... (2, Insightful)

Smordnys s'regrepsA (1160895) | more than 6 years ago | (#21877658)

I'm simply saying, the same thing that stops spam from being blocked in the first place stops your idea from coming to fruition. Millions of almost, but not quite, the exact same Nigerian scams are sent/stored without us having the ability of accurately checking for redundancy. With ~95% of all email being spam, you could make millions if you developed a program/process for correctly identifying multiple emails that are almost, but not quite, the exact same email CORRECTLY as spam, instead of let's say... a forwarded quiz with answers about yourself that is almost, but not exactly, the same email as the original quiz with your friend's answers (or, insert your example here). Do that, and you not only have found a way to check for redundancy in email storage, you have found a way to stop the redundancy (or, ~95% of the redundancy) from happening in the first place (I'm sorry, the lameness filter has kicked in, please stop attempting to send spam through this email address).

So, no, I'm not rejecting your idea outright. I'm saying that by the time it is possible, it won't be AS needed.

Re:So, in other words... (1)

4D6963 (933028) | more than 6 years ago | (#21877798)

I see, but my idea is more focused on solving the storage problem, and to get around the "95% redundancy" problem my idea was based on cutting messages into blocks depending on whether they're redundant or unique, as described here [slashdot.org] .

Re:So, in other words... (5, Interesting)

goodtim (458647) | more than 6 years ago | (#21877456)

Actually, I have a partial answer to this question. As a sysadmin for a Novell GroupWise email system, I can tell you that the actually message data for duplicate incoming messages (such as spam that is sent to many people at the same time) are only stored on disk once. Some sort of "pointer" is used to reference the messages to the individual users mailboxe's. Check out the docs [novell.com] if you are interested.

That said with about 1400 users (spread across multiple postoffices), we have probably about 400gb of email data. We are able to keep it low, by having a 120 day retention policy. After that point, email can be archived locally, otherwise its deleted. Independant of that, and to comply with regulations and disaster recovery scenarios, email data is backed up and replicated offsite using disk-to-disk backup (eVault [evault.com] in case anyone is interested).

This gives us the ability to archive email for up to 27 years or something like that (with relatively low storage costs because the disk-to-disk is incremental, storing changes at the per-block level).

As for Microsoft Exchange, I have not the slightest clue how data is stored.

Re:So, in other words... (1)

multipartmixed (163409) | more than 6 years ago | (#21877620)

I suspect that the behaviour you're describing is only for the case when multiple deliveries occur via a single SMTP transaction (i.e. multiple RCPT TO commands before DATA) rather than the general case of messages-which-happen-to-be-identical, which is what the OP was positing.

Either that, or when the sending system sends the same message in multiple transactions (i.e. poor mailer, or a mailer interrupted by a 452 response code) and the messages have the same Message ID header.0

That said, the original poster makes an assumption that identical-looking messages are likely to be indistinguishable, which they in fact are not, unless generated by a non-compliant mailer and probably get received by a non-compliant mailer. Message ID must vary from message-to-message, and the Date and Received-By: headers are extremely likely to vary from message-to-message.

So, the OP then faces a HUGE search problem which will only "hit" when the sending MTA, and probably the receiving MTA, are non-comformant. This is unlikely to occur with any great frequency, making that search heuristic non-productive. He'd get better lucky archiving large message fragments as some huffman-coding variant (and surely much better could be done with a little thought).

Re:So, in other words... (1)

4D6963 (933028) | more than 6 years ago | (#21877778)

That said, the original poster makes an assumption that identical-looking messages are likely to be indistinguishable

No, I make the assumption that identical-looking messages have most of their data in common, and that this common data, even if only a chunk of the message starting and stopping at an arbitrary point, could be stroed efficiently.

That means cutting messages into blocks, if it is found that some part has something in common with another one, to store common blocks of data all in one place. This way, a personalised message with only a few words varying from a copy of it to another would get all of its redundant data stores only once.

Here's how it would be stored, once messages would be correlated and that all the similarities would be identified, messages would be cut into blocks, depending on whether these blocks are unique or redundant. Every block would be given an ID, and stored using that ID, and mails would be stored as a list of these IDs.

Re:So, in other words... (1)

TapeCutter (624760) | more than 6 years ago | (#21877990)

"That means cutting messages into blocks, if it is found that some part has something in common with another one, to store common blocks of data all in one place."

Substitute "words" for "blocks" and you will find you have invented a dictonary.

Re:So, in other words... (1)

4D6963 (933028) | more than 6 years ago | (#21878126)

Substitute "words" for "blocks" and you will find you have invented a dictonary.

Duh, of course by blocks I mean blocks of a significant threshold size. You're just nitpicking ;-)

Re:So, in other words... (2, Informative)

TapeCutter (624760) | more than 6 years ago | (#21878828)

"You're just nitpicking ;-)"

Ummm, no. I have CS degree and 20yrs experience. What you are talking about is the attacking the problem of redundant information [wikipedia.org] by comparing blocks, this has already been 'solved'. ;)

Re:So, in other words... (1)

damn_registrars (1103043) | more than 6 years ago | (#21878860)

we check for redundancy when archiving mails, in a way so that we can save a hell of a lot of space on spam

I could see that helping if the same spam is sent to the clients on your network, but it doesn't account for all the subsequent iterations of the spam.

YMMV, but I see a lot of spam carrying highly varied introductory garbage (to attempt to fool spam filtering software, of course). Some of my email accounts easily receive 10x as much spam as legitimate email, which would make a redundancy check difficult to apply.

But if it works for you, then more power to you.

Re:So, in other words... (1)

webmaster404 (1148909) | more than 6 years ago | (#21877188)

Exactly. E-mail use is declining in non-company use to IM and text messaging. Due to spam and other factors I would highly disagree that E-mail will grow that much. With cross-platform IM clients such as Pidgin, the OS is no problem for IM and in young people both IMs and text messaging have made E-mail needless.

Re:So, in other words... (0)

Anonymous Coward | more than 6 years ago | (#21877424)

U yoots myt tink IM n txtmsgng mak emil needlesh, buht ish jst duh saim ting.

DOIK!

Re:So, in other words... (1)

wizardforce (1005805) | more than 6 years ago | (#21877224)

homeland security.

Re:So, in other words... (0)

Anonymous Coward | more than 6 years ago | (#21877228)

That's a shitload of v1agr7 and c1ali5. It will keep future digital archaeologists up all night (if it's longer than 4 hours they should see their holographic doctor).

Re:So, in other words... (2, Interesting)

LoudMusic (199347) | more than 6 years ago | (#21878698)

From the summary:
"E-mail growth accounts for much of that figure."

We're archiving spam?
No, we have associates using their email as a file storage device - sending documents to eachother through email rather than just sending an email that says "Your 38MB file is on the file server in /X/here/where/there/document.type".

E-mail growth... (5, Funny)

Urger (817972) | more than 6 years ago | (#21877108)

E-mail growth accounts for much of that figure.

They should have that looked at. A good dermatologist could remove it.

Distributed Storage (3, Informative)

Anonymous Coward | more than 6 years ago | (#21877110)

Some big projects are generating too many data that they have problems to deal with all that.
For example the Folding@home is implementing a distributed storage mechanism for their data and we'll likely have a new @home project soon - Storage@home.
http://en.wikipedia.org/wiki/Storage@home [wikipedia.org]
http://www.stanford.edu/~beberg/Storage@home2007.pdf [stanford.edu]
http://folding.stanford.edu/English/Papers#ntoc7 [stanford.edu]

Re:Distributed Storage (0, Flamebait)

danwat1234 (942579) | more than 6 years ago | (#21877506)

Storage@home!? That's hilarious. I thought that was a joke at first... Sounds like a good alternative to spending more money on the project just to store data.

How Much do We Need to Store? (4, Insightful)

Zordak (123132) | more than 6 years ago | (#21877124)

E-mail is the biggest burden on the storage space, and so much of that is garbage (I'm not even talking about spam---most "legitimate" e-mail is garbage). I wonder if there would be appreciable negative repercussions to deleting most of it. It seems like as often as not, all you get from archived e-mails is well-documented and discoverable "smoking guns" when you get sued. What if we just stored less of it? Would it be that bad? How likely is it that you're going to need some random Word document from 1998? Not criticizing---I'd really like to know.

Re:How Much do We Need to Store? (2, Insightful)

Naturalis Philosopho (1160697) | more than 6 years ago | (#21877302)

In the U.S., it's the law that a company must retain all electronic documents just in case they do ever have to go to court, for whatever reason. IMO, this is one of those very poorly thought out laws as 1) how do you punish a company for contempt when they can't hand over their e-mails because of 2) almost nobody currently archives all of their e-mails. Also, how do you prove that you've not deleted any? Plus, how does anybody ever sort through them all during discovery? I pity that law clerk.

Re:How Much do We Need to Store? (1)

kestasjk (933987) | more than 6 years ago | (#21877962)

This already happened when MS lost a bunch of e-mail relating to the IE case, didn't it?

Re:How Much do We Need to Store? (1)

phoebusQ (539940) | more than 6 years ago | (#21877320)

In the US (and I'm sure other places as well), companies are required to archive electronic data.

Re:How Much do We Need to Store? (1)

Bluehorn (34947) | more than 6 years ago | (#21877842)

This reminds me of my data loss night mare back in 2004. While I was still a student, I lost both the disk of my work station and shortly afterwards the server. Of course, I had a backup of really important data, which did not include email archives at that time (silly me).

I was bothered that I had lost some ten thousand emails due to that double disk failure.

Actually, I never remembered that accident again until I read that slashdot story just now... Seems like no important data was lost.

Anyway, my backups now include email ;)

Re:How Much do We Need to Store? (1)

ZorbaTHut (126196) | more than 6 years ago | (#21878068)

Every once in a while I need to dig out an ancient email from my email repository. I don't have any way of knowing which one ahead of time - sometimes it's something obviously important, sometimes it turns out to be something incredibly unimportant (one of my friends deleted an important Livejournal entry once accidentally, but I'd responded to the entry with a mostly-unimportant comment and Livejournal emails me with the entire entry text when I do that. Surprise! It's important!)

On top of that, the sheer effort involved in figuring out which emails are important and which aren't simply isn't worth it. I've got around 400mb of email, containing at least 50,000 individual messages - it's cheaper, in terms of time and effort, to keep it all.

Re:How Much do We Need to Store? (1)

kent_eh (543303) | more than 6 years ago | (#21878714)

I seem to recall several recent articles about new data retention laws requiring companies to do just that - store potentiality incriminating e-mails for absurdly long periods of time.

So, to answer your question:

What if we just stored less of it?

You might get fined or jailed.

Re:How Much do We Need to Store? (1)

Zordak (123132) | more than 6 years ago | (#21878766)

I know of no such law. I know that the Federal Rules of Civil Procedure require litigants to produce archived data, and I know that litigants can be sanctioned for destroying data in bad faith. The Rules also provide a safe harbor for data destroyed in good faith in accordance with a reasonable data retention policy. So what's reasonable? What is the real probability that a business will have non-litigation problems?

duh...users store their files in their email! (4, Informative)

Maskirovka (255712) | more than 6 years ago | (#21877132)

article summary:

Users in a lot of places use their email as a document management system. This is somewhat effective on an individual basis, but in large organizations shared documents get duplicated dozens or even hundreds of times as each user has their own copy. In the next few years products like Sharepoint will alleviate some of that, though storage is cheap enough that it may not be worth the cost to both reeducate users and build the infrastructure for it. A SAN can hold real a lot of word documents and PDFs after all...

Practical Internet Groupware (1)

mangu (126918) | more than 6 years ago | (#21877206)

Users in a lot of places use their email as a document management system. This is somewhat effective on an individual basis, but in large organizations shared documents get duplicated dozens or even hundreds of times

That's exactly the message of this book [oreilly.com] . Email, although widely used, is neither practical nor effective as a means of divulging information in a company. And duplication of information is the lesser problem.


For instance, suppose someone leaves the company, either permanently or in a vacation, and somebody else takes over a job. How do you transfer the relevant information to the substitute? Forward several dozens of emails and hope it makes sense? What if Alice forwards an email to Bob but not to Charlie, how do you make sure everybody in the project has access to all the relevant information?


Email and http are widely used because they are widely available, but neither of them is a very good solution for information handling.

Re:duh...users store their files in their email! (1)

webmaster404 (1148909) | more than 6 years ago | (#21877236)

I don't get it. Most large companies have servers that store documents and such, along with that, most computers have 40 gig- 120 gig hard drives and drives up to 1 TB or so can be bought for cheap. How are we running out of space in a large company? And why "archive" E-mail thats stored on the computer AND an E-mail server?

Re:duh...users store their files in their email! (1)

Slippy. (42536) | more than 6 years ago | (#21877556)

Ah, the simple questions. A civic is nice, and maybe you've souped it up to pull a little trailer, but that doesn't mean a tractor-trailer is going to be cheap too.

Unreliable, slow desktop storage is cheap. Reliable, redundant, fast, networked storage - not cheap.

You can dump a TB on your local disk. Now copy it. Boy-howdy, that's a looooonnnngggg wait. Now let 1000 people all do this at the same time.

  - redundant storage,
  - fast bus,
  - redundant controllers,
  - redundant locations maybe,
  - redundant power,
  - cooling,
  - redundant wiring,
  - and expert management ('cause somebody has to be blamed!)

*All this* is still expensive. Storage prices (hard drives) drop quickly, but not as fast as usage. And consumer drives - not so reliable in large storage arrays.

Re:duh...users store their files in their email! (1)

hjf (703092) | more than 6 years ago | (#21878166)

zfs.

Re:duh...users store their files in their email! (1)

leenks (906881) | more than 6 years ago | (#21877636)

Go and work for a large company and find out. You can't use the hard drive in a workstation to store anything other than applications - the machine will (out of necessity) be a standard image that will get blasted from time to time with updates, or when something breaks on the Windows install.

For enterprise storage, hard drives are not cheap. Yes, you can buy domestic IDE drives for cheap, but check the prices on SAS or "enterprise grade" storage. A large company will have potentially petabytes of data - backups for that amount of data aren't cheap, let alone archiving.

Emails are "archived" because most companies age off old emails. Any sensible company will archive emails that users delete (look at Enron as an example of why you'd want to do this).

Re:duh...users store their files in their email! (4, Insightful)

Znork (31774) | more than 6 years ago | (#21877416)

Better article summary:

Storage vendors want to sell expensive solutions to gullible execs, pay analysts to produce credible-sounding FUD scenarios.

"monthly e-mail traffic at more than 30 million messages, vs. 17 million just one year ago."

Like, wow. In the meantime 500GB disks cost the same or less than 250GB disks did a year ago.

"The university settled on an IBM storage infrastructure that will afford the institution 350TB of capacity"

350TB? 350 disks? Half that in a year and a quarter in 2? That's not really a huge amount of storage. Anymore. It's an amount of storage I could go order from my friendly online computer store and get delivered tomorrow.

The fact is, corporate storage isnt driving the market anymore, the consumer market is. Most people I know have more storage in their home PC than the average server requires. Companies want to save video? Consumers want their PVR's to save the cable-tv stream.

Re:duh...users store their files in their email! (3, Insightful)

leenks (906881) | more than 6 years ago | (#21877668)

More like 1000 or 2000 disks, not 350. 1TB drives haven't really hit the enterprise yet. The biggest SAS drives in use are still 300GB.

 

Re:duh...users store their files in their email! (1)

Torque (49173) | more than 6 years ago | (#21878850)

Shockingly, this is one area that Exchange does a reasonable job. Since we know the behavior is "send files via email", you want an email system that doesn't croak under that kind of load. Exchange, with single-instance storage, actually gets this right.

If I send, via Exchange, the same email to 30 users, with an attachment to it, that email (and attachment) are stored once. With any other mail system I get 30 copies of it. THAT is a huge improvement.

(Zimbra may actually do single-instance storage, but I haven't done investigation enough yet to be sure)

shocking? (0)

Anonymous Coward | more than 6 years ago | (#21877150)

hmm, I can believe this. I ran an e-mail server for the last company I worked for, and it was amazing how fast space got taken up just due to residual e-mails.

Since I'm the type to do the same thing, I can't be critical, so I left no quotas.

2010 (5, Funny)

Anonymous Coward | more than 6 years ago | (#21877156)

All these archives are yours except Europa. ATTEMPT NO WRITINGS THERE.

Re:2010 (1)

Atele (1177559) | more than 6 years ago | (#21878334)

But in 2061 it's okay.

corporate email storage (0)

Anonymous Coward | more than 6 years ago | (#21877164)

With corporations getting sued and having their own emails used against them in court, shouldn't they be destroying old email, not saving it?

Use standard units people understand. (3, Funny)

jd (1658) | more than 6 years ago | (#21877190)

Things like Libraries of Congress, Libraries of Alexandria, Spams per Square Inch. You know, the units that people have become familiar with. Besides which, are they power-two gigagytes or SI gigabytes? Also, how much bandwidth is needed to shift all that data? In the standard Imperial units of Clay Tablets per German Juggernaut per unit of French motorway, naturally.

Surprising . . . (3, Insightful)

cashman73 (855518) | more than 6 years ago | (#21877194)

That 90% of that 27,000 petabyte figure isn't for archiving p0rn,... Although I guess, from the corporate IT perspective, they're not worried about backing up p0rn, since most people probably don't do that at work.

But it is mostly email they're talking about here, and I bet a HUGE part of this archiving is:

  1. spam
  2. Email forwards that have been sent 1,000 times that still have all the original message headers attached
  3. Non-business-related multimedia emails sent by administrative assistants using the company's email and time to send and receive cutesy messages from/to their family & friends
  4. Business-related powerpoint and multimedia emails by non-techie PHBs that don't know how to transfer such files via FTP, and who are too damn lazy to use a thumbdrive

Yep! Solve problems 1-3, and you'd vastly decrease the amount of email that you have to archive! I won't complain about #4, since I actually value my job, but it would be nice if more PHBs knew more about tech,...

Re:Surprising . . . (2, Insightful)

houghi (78078) | more than 6 years ago | (#21877626)

About 4. I do not understand management where I am.

I make several excel files every week for reporting. They are located on a shared drive. Only extra data is added every monday, yet instead of puting a link to the files, or the directory, management wants me to send them by email every week to several people.

Utterly stupid, if you ask me.

Re:Surprising . . . (1)

ZorbaTHut (126196) | more than 6 years ago | (#21878526)

The directory is backed up and version-controlled, right?

Because if not, that might be an (admittedly crummy) attempt at a backup system.

For Fucks sake (2, Insightful)

Colin Smith (2679) | more than 6 years ago | (#21877256)

Just delete the crap.

 

Re:For Fucks sake (1)

Smordnys s'regrepsA (1160895) | more than 6 years ago | (#21877388)

Old, but little known axiom - "One Man's Crap is Another Man's...Fetish?"

Yep (1)

krunchyfrog (786414) | more than 6 years ago | (#21877282)

That's my pr0n collection allright.

This is starting to be Manditory (3, Funny)

Smordnys s'regrepsA (1160895) | more than 6 years ago | (#21877418)

Only 27,000 petabytes? n00b!

My pr0n collection takes at least 3 Internets* to store, archived.


*(sorry, forgot the conversion rate for Libraries of Congress)

30 million emails? (1)

cashman73 (855518) | more than 6 years ago | (#21877288)

30 million emails went through the pitt.edu email servers last year, and my account there didn't get squat during the christmas break! I wonder where all the email is going? Although the university is closed anyway, so that might have something to do with it,...

I suppose if I was crazy enough, I'd post my address here on slashdot to see if we can slashdot Pitt's email servers,... maybe we can turn 30 million messages into 60 million messages. On second thought, I don't want 30 million messages,... ;-)

Re:30 million emails? (1)

JustNiz (692889) | more than 6 years ago | (#21878186)

It sounds like someone might be using your servers for sending/forwarding spam. Your system might be telling us all how we can "improve ur p3n1s size" or "help Dr. mbongo from Burkina-faso move $99999999 into your account".

Wow, welfare for programmers... (1)

tjstork (137384) | more than 6 years ago | (#21877332)

So now, SOX and new discovery rules have created welfare for programmers. What value is all of this e-mail? The bulk of it is worthless and the cost of this is a huge drain on the economy. How many disk drives does it take to store 27 Ebs, and how many people will it take to manage it all?

Re:Wow, welfare for programmers... (3, Interesting)

phoebusQ (539940) | more than 6 years ago | (#21877496)

How do you figure that storage needs driving the increase in disk capacities and creating jobs is "a huge drain on the economy"?

And what do data-archiving rules have to do with welfare for programmers? Maybe for disk manufacturing firms or data admins, but programmers?

Re:Wow, welfare for programmers... (1)

tjstork (137384) | more than 6 years ago | (#21877690)

How do you figure that storage needs driving the increase in disk capacities and creating jobs is "a huge drain on the economy"?

We wouldn't need to store the data except for government intervention. So, instead of companies investing in their actual products, such as making better cars and airplanes, they are investing in something that adds no value to the product whatsover. The result is a transferrance of wealth to computer people but without any consumer benefit. In other words, its welfare for computer people.

Re:Wow, welfare for programmers... (1)

phoebusQ (539940) | more than 6 years ago | (#21877766)

I disagree.

We don't have a complete enough picture of the effects of data storage requirements. First, they may have some economic benefits. Second, it seems unlikely that the costs are so massive that they have any serious impact on bottom-line product development. Third, welfare would imply that there was no productive benefit caused by these "computer people", which we know is untrue.

how much is surveillance data? (2, Interesting)

petes_PoV (912422) | more than 6 years ago | (#21877342)

E-mail growth accounts for much of that figure

And a great deal of video archive from CCTV as well I expect.
The question that arises is how would you index all this?

Why, the answer is simple: (0)

Anonymous Coward | more than 6 years ago | (#21878680)

Google.

Moving away from Big Iron? (2, Funny)

HockeyPuck (141947) | more than 6 years ago | (#21877484)

FTFA:

Mounting interest in these approaches highlights a pronounced shift away from "big-iron storage" - traditional storage arrays typically composed of custom application-specific integrated circuits, RAID controllers, and fixed-disk and cache-scalability ceilings.
Now TFA goes on to say customers are turning towards Network Appliance as a company that uses COTS parts and software. They use an intel CPU and FC/GigE adapters from other vendors, but I wouldn't call them 100% COTS. It's not like it's a generic PC built from FRYS with JBOD on the back.

NetApp is a great company and makes a great product aimed for a specific market segment: Fileservices (NFS/CIFS). I don't see many customers tossing out the EMC DMX, HDS Tagmastore or IBM Shark for a FC enabled netapp array. I also don't see a lot of FICON shops asking netapp to support FICON.

Now the phase storage mgmt is entering is the 'good enough' phase. Does my organization need the current generation of "high end" arrays? Maybe not. The current generation of midrange with its better or cheaper $/GB and increasingly parallel featureset to the highend arrays, is starting to looking more attractive to many customers.

Re:Moving away from Big Iron? (2, Insightful)

phoebusQ (539940) | more than 6 years ago | (#21877512)

FTFA, RAID, TFA, COTS, CPU, FC, GigE, FRYS, JBOD, CFS, CIFS, EMC, DMX, HDS, IBM, FC, FICON... 17+ acronyms in one post...that's pretty impressive. Do you kiss your mother with that mouth? :)

Re:Moving away from Big Iron? (3, Funny)

HockeyPuck (141947) | more than 6 years ago | (#21877644)

FRYS isn't an acronym... :)

and yes I do.

Re:Moving away from Big Iron? (1)

DMUTPeregrine (612791) | more than 6 years ago | (#21877924)

From the F*cking article, Redundant Array of Inexpensive Disks, The F*cking article, Commercial Off-The-Shelf, Central Processing Unit, Fiber Channel, Gigabit Ethernet, Fry's Electronics (not an acronym), Just a Bunch Of Disks, Caching File System, Common Internet File System, Electro-Magnetic Compatibility, DataMining Extensions, Hierarchical Data System, International Business Machines, Fiber Channel (again), FIber CONnectivity. Didn't Read The F*cking Article (RTFA) yet, so some acronyms with more than 1 definition may be using the wrong one....

Re:Moving away from Big Iron? (3, Funny)

HockeyPuck (141947) | more than 6 years ago | (#21878104)

We're talking storage (sorry DASD) here... It's all about...

Hooking up a pair of EMC DMX's (or IBM ESSes, or HDS USPs) over a pair of OC48s for SRDF/PPRC/USR unless you are a zOS shop, then you could run XRC. Since this is a BC/DR plan, we'll run it over FCIP protected by IPSec over a DWDM leased line, which must be protected by a UPSR/BLSR, otherwise in the event of a link failure, the R1s will split from the R2s.

Then you're SOL.

Re:Moving away from Big Iron? (1)

Awod (956596) | more than 6 years ago | (#21878758)

Every morning when I emerge from the basemen.............. erm, go visit her for the holidays..

time to buy EMC stock! (1)

mgranit11 (862145) | more than 6 years ago | (#21877734)

Thats what i will be doing!

and 26.5 exabytes are porn (1)

plasmacutter (901737) | more than 6 years ago | (#21877784)

and 26.7 exabytes are dedicated to porn storage!

thank you! i'll be here all week!

hey was that rotten fruit! HEY! SECURITY!

Redundant Data (2, Interesting)

tm8992 (919320) | more than 6 years ago | (#21877932)

I wonder how much of this data is really redundant--copies of other data. How many emails can really be unique? How many employees download the same video a hundred times on the company's server? As network speeds increase, it will be less necessary for multiple users to store the same thing (think streaming those videos), so could this really be an exaggeration of future storage requirements? Could a better system be designed to minimize redundancy?

If designers still optimized their images (0)

Anonymous Coward | more than 6 years ago | (#21877970)

If designers still optimized their images down from 50k to 15k instead of flirting with the design hotties and smearing poop on other peoples keyboards this might not be a problem.

it must be Seagate (0)

Anonymous Coward | more than 6 years ago | (#21878288)

Shit, is it all Pr0n ? :-P

So that's about 20 billion gigabytes of data... (2, Insightful)

LordHuggington (1210226) | more than 6 years ago | (#21878380)

that will be lost or stolen as company employees fail to properly encrypt back-ups, leave laptops in their car while running in for a latte or some such? Seriously, though, the article says storage is corporations' number 2 concern. What's number one from this survey? Is it security?

Single Instance Storage (0)

Anonymous Coward | more than 6 years ago | (#21878474)

Microsoft implemented something called Single Instance Storage (SIS) with Windows 2000 and 2003 (http://research.microsoft.com/sn/Farsite/WSS2000.pdf).

If it weren't quite so cryptic to implement and use it would probably help reduce some of the problem.

Foreigners in their own country (0)

Anonymous Coward | more than 6 years ago | (#21878656)

First of all, almost all elements used to build these laptops are belonging to somewhere else.

The components are possibly Chinese. the ideas are possible brewed from open source (a good concept, but a salad in the end, for this very same argument explained in here). Many of the "teaching contents" ain't local (with this, you know, local is "Local" in every place is a different culture-animals-religions-traditions-dictators-martyrs-heroes-ECONOMY).

After all, you start trying to give people a better education, but in the process, transform them in aliens, individuals separated from them own reality, and context (i think their context is being abused, since centuries, and robed, and being utilized, and being the last defecating end of giga-planet monopolies-mafias).

So, what happens if you "create" a "global" child in that medium?? usually chaos (think some of those lands are in chaos at this moment), and the necessity of "global people" to rescue them. (finally you generate a Trojan, more chaos, and local monsters that defend their land from the foreigners (attacks they think)).

So, in the end, OLPC, can do, to its maximum extent, provide a "transparent structure", to which, every land would fill with their history, and what they got in their blood.

BUT, HOW.

how can you override the material from which the laptop is made?, necessary evil some will say?

Most directly, people in the countries DON'T need, (nor needed in the past), computers.

They need peace.

They need the time and space to learn from their elders, to heal, and to cultivate the land. to learn from it, to recall what is which this land produces, and how you should take care of it.

All that, is not in a computer (although you can document it, its not advised), is in their will.

Introducing a big factory, the marked economy, in this lands that CANT TAKE THEM, that dont have that history.

Or SENDING THEM WEAPONS, WONT HELP, them achieve the reconciliation, the healing, or the sustainable growing their own land needs.

Even complaining and cursing, saying they ain't good people to do business with, is NOT WHAT THEY NEED. I mean, that does only harm.

In the end, interventionism, generates a monster.

But.. why is the aggressive reaction occurring in this lands? why is people "hunting" each other there?
Is it because of interventionism and the aliens "global culture" generates? (read: we are all living in america)
Is it because of the big factories emplaced in this poor lands? (poor in currency)
Is this because of the social strait stairs that the big factories/market economy generates to be able to "participate" in this economy?
Is this because of the intervention of mafia/monopoly interfering them to consolidate and consume those lands/people?
is this because they are CONSUMING PRODUCTS THAT ARE NOT FROM THEIR OWN LAND? (which generates another type of alien).

In part, those are stuff negroponte didnt took into account, when tracing his plan.

and are stuff market economy will never think about. If they would think of that, they couldnt destroy and colonize new lands. (read some resentment there).

For you C64 users. (1)

Joe U (443617) | more than 6 years ago | (#21878772)

That's at least 662,257,761,200,000,000,000 nybbles! (roughly) You may need extra floppies.
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>