Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

'Digital Universe' To Add 1.8 Zettabyte In 2011

CmdrTaco posted more than 3 years ago | from the less-than-half-of-it-is-porn dept.

Data Storage 60

1sockchuck writes "More than 1.8 zettabytes of information will be created and stored in 2011, according to the fifth IDC Digital Universe study. A key challenge is managing this data deluge (typified by the Large Hadron Collider at CERN, which generates 1 petabyte of data per second)."

cancel ×

60 comments

Sorry! There are no comments related to the filter you selected.

LHC data is _not_ stored in the digital universe (3, Insightful)

rbrausse (1319883) | more than 3 years ago | (#36595738)

the experiments may generate the PB per second but most of the data is rejected before it hits any storage system...

Re:LHC data is _not_ stored in the digital univers (3, Insightful)

Infiniti2000 (1720222) | more than 3 years ago | (#36595880)

Indeed, only about 25PB are stored every year from the LHC.

Re:LHC data is _not_ stored in the digital univers (1)

AliasMarlowe (1042386) | more than 3 years ago | (#36596362)

Indeed, only about 25PB are stored every year from the LHC.

No. They store all of it, but mostly in /dev/null

Re:LHC data is _not_ stored in the digital univers (1)

Joce640k (829181) | more than 3 years ago | (#36595936)

Shouldn't it be "Pebibytes"? We're supposed to be geeks.

Re:LHC data is _not_ stored in the digital univers (1)

Daimanta (1140543) | more than 3 years ago | (#36595986)

Shouldn't it be "Pebibytes"? We're supposed to be geeks.

Yes, but we're not morons.

Re:LHC data is _not_ stored in the digital univers (0)

Anonymous Coward | more than 3 years ago | (#36596280)

And as geeks we understand that the English language isn't governed by a committee of Swiss engineers.

Re:LHC data is _not_ stored in the digital univers (1)

AliasMarlowe (1042386) | more than 3 years ago | (#36598038)

And as geeks we understand that the English language isn't governed by a committee of Swiss engineers.

Unfortunately, they're English engineers...

Re:LHC data is _not_ stored in the digital univers (1)

GrumpySteen (1250194) | more than 3 years ago | (#36606282)

I suppose we should be grateful that data isn't measured in petahogsheads.

Re:LHC data is _not_ stored in the digital univers (0)

Anonymous Coward | more than 3 years ago | (#36596444)

Geeks didn't change the standards to the new format. Sales people trying to make right the shit they've fed us for years did, most notably from storage vendors.

Re:LHC data is _not_ stored in the digital univers (1)

the_other_chewey (1119125) | more than 3 years ago | (#36605160)

Yes, and as geeks we understand that the quoted number is an approximate
one and the data generated are not inherenly binary, so there is no need
for either the precision of "exactly one PiB" or the context of "this is binary".

In fact, the decimal prefix is the much more sensible one to use here.

1.8 Petabyte per second... (1)

LordofEntropy (250334) | more than 3 years ago | (#36595752)

Wow that's a lot of data. Can't wait to see more of the results published.

Re:1.8 Petabyte per second... (1)

cashman73 (855518) | more than 3 years ago | (#36595932)

How many libraries of congress is that?

Re:1.8 Petabyte per second... (0)

Anonymous Coward | more than 3 years ago | (#36596000)

Over 9,000!

Re:1.8 Petabyte per second... (1)

daem0n1x (748565) | more than 3 years ago | (#36597744)

And 99,9999% of it is completely useless.

Re:1.8 Petabyte per second... (1)

perryizgr8 (1370173) | more than 3 years ago | (#36619072)

its a decimal, not a comma. why do so many people make the same stupid mistake?

Re:1.8 Petabyte per second... (1)

daem0n1x (748565) | more than 3 years ago | (#36620340)

Stupid is to think English is the only language in the world.

Re:1.8 Petabyte per second... (1)

perryizgr8 (1370173) | more than 3 years ago | (#36620832)

decimal notation changes with language?!? that's a new one. even if it does, the rest of your comment WAS in english.

Re:1.8 Petabyte per second... (1)

daem0n1x (748565) | more than 3 years ago | (#36621748)

decimal notation changes with language?!?

Yes it does.

that's a new one.

Only for you

even if it does, the rest of your comment WAS in english.

The decimal separator on my keypad is comma. If you weren't so swift in calling me stupid, I would have explained it earlier.

Re:1.8 Petabyte per second... (1)

perryizgr8 (1370173) | more than 3 years ago | (#36622788)

ok, my bad.

Hellagood (1)

slipnslidemaster (516759) | more than 3 years ago | (#36595808)

That's 1.8 x 10^-6 hellabytes for those of you keeping track.

Re:Hellagood (1)

BrokenHalo (565198) | more than 3 years ago | (#36596470)

That unit is as good as any. We're rapidly running out of prefixes here, and we still need formal definitions for units such as bucketload, shitload, shitton[ne] and so forth.

Re:Hellagood (1)

slipnslidemaster (516759) | more than 3 years ago | (#36597088)

The key to getting a prefix established...is to just start using it.  The rest of the world and ultimately standardizing bodies will adopt it eventually.

http://www.facebook.com/pages/The-Official-Petition-to-Establish-Hella-as-the-SI-Prefix-for-1027/277479937276?ref=search&sid=1050298974.3729275378..1

Re:Hellagood (1)

BrokenHalo (565198) | more than 3 years ago | (#36597194)

Fair enough, but I doubt if you're going to win any converts by insisting on posting your comments with horrible monospaced fonts just to grab attention... :-}

Question about slashdot (-1)

Anonymous Coward | more than 3 years ago | (#36595822)

Why does slashdot often display a green tag with 'loading' at the bottom forever? And why if I middleclick on a link in a slashdot comment, which should open in a new tab, does the link not open in a new tab and ./ jumps to a random part on the page?

This is very annoying! It started a few weeks ago.

n00b (0)

Anonymous Coward | more than 3 years ago | (#36595900)

Welcome to the internet.

Re:Question about slashdot (1)

perryizgr8 (1370173) | more than 3 years ago | (#36619062)

and why in hell does a /. tab not close immediately but does some shit for a few miiliseconds before closing?

Well that's inconvenient... (0)

Anonymous Coward | more than 3 years ago | (#36595826)

We're going to have two make *two* different ZFS filesystems to hold it all

Re:Well that's inconvenient... (1)

tom17 (659054) | more than 3 years ago | (#36597878)

Why two? For redundancy?

Free Dating Websites (1)

Newsbd (2248922) | more than 3 years ago | (#36595868)

I was very pleased to find this web-site.I wanted to thanks for your time for this wonderful read!! I definitely enjoying every little bit of it and I have you bookmarked to check out new stuff you blog post. Find us "How to Date Online With Girl" http://www.freedatingbd.blogspot.com/ [blogspot.com]

Re:Free Dating Websites (1)

L4t3r4lu5 (1216702) | more than 3 years ago | (#36596170)

rel=nofollow dude. Your linkwhoring is pointless here.

Can I have that in LoCs (4, Funny)

Bob the Super Hamste (1152367) | more than 3 years ago | (#36595944)

Can we get that in a proper measurement like Libraries of Congress.

Re:Can I have that in LoCs (1)

Phurge (1112105) | more than 3 years ago | (#36597100)

Using 1 LoC = 20 TB, then 1.8 ZB = 96,636,764 LoCs

Or as wolfram alpha says 1.8 ZB = 144,000 x estimated information content of all human knowledge.

Re:Can I have that in LoCs (2)

Phurge (1112105) | more than 3 years ago | (#36597160)

oops 1.8 ZB = 144 x estimated information content of all human knowledge.

Re:Can I have that in LoCs (1)

Cylix (55374) | more than 3 years ago | (#36597502)

Or as woflram and heart says, 0 = 144,000 x 0 (estimated worth of human knowledge).

Re:Can I have that in LoCs (0)

Anonymous Coward | more than 3 years ago | (#36597932)

..1.8 x 10^-6 hellabytes

let me put it this way.
If each Kilobyte was a leaf, it would be the Forest Moon of Endor.

How much is redundant (2)

Bob the Super Hamste (1152367) | more than 3 years ago | (#36596038)

I wonder how much of that data is redundant. I know that for one of my side projects I have "redundant" data that I got from the Minnesota DNR, various MN counties, the state legislature, and the federal gov. Even after it had been preprocessed and trimmed down so it only has what I care about it is still around 12GB of vector data which is about 1/3 the original size.

Re:How much is redundant (0)

Anonymous Coward | more than 3 years ago | (#36602730)

all of it? after all we know the algorithm for generating pi..

bah (1)

PJ6 (1151747) | more than 3 years ago | (#36596052)

that's just Netflix.

57.5 billion 32GB iPads required to store this :D (1)

Mightee (2221294) | more than 3 years ago | (#36596054)

to store this amount of data, you need 57.5 billion 32GB iPads which will cost around $34.4 trillion — and that's equivalent to Gross Domestic Product (GDP) of United States, Japan, China, Germany, France, the United Kingdom and Italy combined. :D

Re:57.5 billion 32GB iPads required to store this (0)

Anonymous Coward | more than 3 years ago | (#36596544)

Because iPads are the most efficient way of storing large amounts of data.

Re:57.5 billion 32GB iPads required to store this (1)

networkBoy (774728) | more than 3 years ago | (#36598928)

You are right, of course.
That'll be 1,500,000,000,000 double density 8 inch floppies worth of data. Much more efficient, and at the last price I paid for them, (about 12 years ago now) they were $12.00 each (not per box, each)) so that is $18,000,000,000,000, or roughly 52.325581395348837209302325581395% of the cost of the iPads.
-nB

large size (0)

Anonymous Coward | more than 3 years ago | (#36596078)

Don't worry, the large size won't be an issue. You can put it in a ZIP and then put that into another ZIP and so on. Pretty soon it gets down to USB stick size and if you keep going you can reduce it to 1 byte if needed.

Re:large size (2)

L4t3r4lu5 (1216702) | more than 3 years ago | (#36596230)

Don't worry, the large size won't be an issue. You can put it in a ZIP and then put that into another ZIP and so on.

That's just stupid. They use the same compression algorithm!

Put it in a ZIP in a TAR in a RAR in a 7z in an ACE in a bZip in a CAB in a dmg in a a ARJ, and finally save it as a GIF. You can't use JPEG as it's lossy.

Re:large size (1)

Noughmad (1044096) | more than 3 years ago | (#36597460)

No, man. It's just ZIP's all the way down.

Created, not stored (0)

Anonymous Coward | more than 3 years ago | (#36596088)

This might be the amount of data being created this year, but there will be many times this amount of data actually being stored. Remember, every repressive government in the world (USA, Britain, China, etc.) will be storing their own copies (or mandating it be done for them by ISPs).

Great (1)

Dunbal (464142) | more than 3 years ago | (#36596172)

More space to be filled by Russian mining bots. Oh wait which universe is this?

Large Hadron Collider data anomaly? (3, Interesting)

L4t3r4lu5 (1216702) | more than 3 years ago | (#36596312)

So it generates 1PB of data per second, yet from the article "[T]he data comes from the four machines on the LHC in which the collisions are monitored â" Alice, Atlas, CMS and LHCb â" which send back 320MB, 100MB, 220MB and 500MB"

That's a few orders of magnitude short of 1 Petabyte, folks. Where are these numbers coming from?

Re:Large Hadron Collider data anomaly? (0)

Anonymous Coward | more than 3 years ago | (#36596546)

Those numbers add up to the 25PB/year that they store. The other 99.9999% (really) is filtered our at the detectors before being "sent back".

Re:Large Hadron Collider data anomaly? (1)

L4t3r4lu5 (1216702) | more than 3 years ago | (#36596670)

Those numbers add up to the 25PB/year that they store. The other 99.9999% (really) is filtered our at the detectors before being "sent back".

Ah, thanks for that. I was under the impression that the computer centre would be discarding the unused data, but if the detectors are smart enough to do it then all the better!

Re:Large Hadron Collider data anomaly? (1)

Eivind (15695) | more than 3 years ago | (#36596598)

One is the raw-amount, simply the number of sensors multiplied with the frequency each sample at multiplied with the size of each sample.

But the article itself say they filter and store only the interesting stuff, which is, as we can see from the later numbers, a triflingly small fraction of the entirety.

A camera that can do full-HD at 30fps captures 186MB/s afterall, but it does not follow that a facility with 3 such security-cameras need to store however many petabytes that becomes, in order to have a recording of the burglars.

Re:Large Hadron Collider data anomaly? (1)

AJH16 (940784) | more than 3 years ago | (#36596972)

As eivind pointed out, the PB a second is raw data. The LHC utilizes 2 layers of in hardware filtering and another layer of software filtering (as I recall from a while back at least) in order to trim the data down to a quasi-reasonable data stream that can be effectively stored and analysed.

Re:Large Hadron Collider data anomaly? (2)

MartinSchou (1360093) | more than 3 years ago | (#36597040)

If you drive 60 miles/hour for 30 seconds, you haven't driven 60 miles. One is a measure of speed, the other is distance.

Same with this. 1 PB/s is speed. 1,140 MB is the amount of data. All it really means is that these 1,140 MB are generated (and possibly collected) in 1.06 microseconds.

Wow, great compression at the LHC! (1)

brian0918 (638904) | more than 3 years ago | (#36596440)

(typified by the Large Hadron Collider at CERN, which generates 1 petabyte of data per second)

So, the LHC produces 1 petabyte per second, and given that there are 30+ million seconds in the year, that means the LHC produces 30+ zettabytes a year. Clearly there is a problem with your typification.

Bogus (1)

vlm (69642) | more than 3 years ago | (#36596594)

I call bogus on this.

10e21 / 10e10 = 10e11 bytes/living human being.

The global GDP and global hard drive manufacturers simply cannot support a 100 GB hard drive per person per year... Cheapest option per byte is probably 1 TB drive for every 10 people. My basement therefore balances against a small African village, but there's plenty of small African villages, and only one me.

Even if all the ACTIVE /.-er types have a basement like mine, and they do not, there are simply not enough of us. And on a global GDP basis a tenth of a TB hard drive is way too expensive per person, that would put data storage at roughly "rice consumption" levels. And the rich are only getting richer while the poor get poorer for some decades now, so don't try the "world is getting richer thus can afford it" argument.

Re:Bogus (0)

Anonymous Coward | more than 3 years ago | (#36596958)

Yes African villages can't afford a 100GB hdd, but the average user in richer countries can afford a 1TB disk per year easily? I think the price is now around £30. Divide the world population by 10 and I'd imagine that'd be the richer percentage.

Re:Bogus (1)

stevelinton (4044) | more than 3 years ago | (#36603352)

Do the sums. 1 ZB = 10^9 TB. a TB had drive costs c US$50, probably less in quantity, so information storage is US$50 billion/year industry.
Doesn't seem implausible to be honest.

Re:Bogus (1)

weffew... (954080) | more than 3 years ago | (#36604170)

Most of CERN's data isn't on hard drives anyway - it's got the biggest tape system in Europe that I know of (I do high end storage for a living).

A photo, for funsies: http://www.flickr.com/photos/naezmi/3309812634/ [flickr.com]

C

great... (0)

Anonymous Coward | more than 3 years ago | (#36596600)

...so now I'm going to have to get *another* zettabyte hard drive :(

LHC data will not be information for years to come (1)

recharged95 (782975) | more than 3 years ago | (#36600476)

And most of it will be junk data.

Data != Information. And then there's Metcalfe's law applied to information value.

wow (0)

Anonymous Coward | more than 3 years ago | (#36601532)

That's a lot of porn...

Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?