Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Aussie Scientists Build a Cluster To Map the Sky

Soulskill posted more than 5 years ago | from the thinking-big dept.

Space 58

Tri writes "Scientists at the Siding Spring Observatory have built a new system to map and record over 1 billion objects in the southern hemisphere sky. They collect 700 GB of data every night, which they then crunch down using some perl scripts and make available to other scientists through a web interface backed on Postgresql. 'Unsurprisingly, the Southern Sky Survey will result in a large volume of raw data — about 470 terabytes ... when complete. ... the bulk of the analysis of the SkyMapper data will be done on a brand new, next generation Sun supercomputer kitted out with 12,000 cores. Due to be fully online by December, the supercomputer will offer a tenfold increase in performance over the facility's current set up of two SGI machines, each with just under 3500 cores in total.'"

cancel ×

58 comments

Sorry! There are no comments related to the filter you selected.

I wonder (2, Interesting)

downix (84795) | more than 5 years ago | (#28361415)

I wonder which CPU the supercomputer will be using. Could be Opterons, or SPARC. I could easily imagine 12000 out of a SPARC Niagra or SPARC VIIfx (8 cores per-die) and would use less wattage than the same number of cores in Opteron. Plus, if they're doing dual or quad-precision, the SPARCs will hold their own.

Re:I wonder (2, Informative)

evol262 (721773) | more than 5 years ago | (#28361697)

They're x6275 blade modules, meaning dual quad core Xeons with a max of 96GB of RAM.

I like the CMT SPARCs as much as anybody else, but they're frankly not competitive for this sort of workload (massive compression).

Re:I wonder (1)

downix (84795) | more than 5 years ago | (#28361937)

That's why I mentioned the dual or quad precision, the only math-area that the SPARCs are still pretty competitive in.

Re:I wonder (1)

evol262 (721773) | more than 5 years ago | (#28362357)

SPARC VII, yes. The Nehalem Xeons are about on par with T2s at substantially lower cost.

Re:I wonder (1)

downix (84795) | more than 5 years ago | (#28364849)

You know, I tried to verify this, but I could not get any Nehalem Xeon setups able to match the T2 setups. Even remotely capable of matching one. The closest I got was an Apple XServe, w/ 3GB of RAM vs the T5130 w/ 4GB of RAM, and the XServe came up only $2k less than the SPARC while also consuming more electicity. But this is comparing with an Apple, so we all know the markups there. Once there are more Nehalem's out there we'll get a better idea as to cost comparison.

Re:I wonder (1)

evol262 (721773) | more than 5 years ago | (#28365281)

I'm going to say you didn't look hard enough. Published SPEC results have a Sun Fire x4270/x2270 walking all over the T5240/T5120 for 1/3rd of the cost or less (with 6GB of RAM).

They cannot, of course, run nearly as many threads in hardware, but for brute-forcing compression, it's sort of a moot point.

Re:I wonder (1)

downix (84795) | more than 5 years ago | (#28367017)

A ha, the Intel® Xeon® Processor 5500 Series is the Nehalem, did not have the model number handy in my search. TYVM for putting in the missing pieces.

Re:I wonder (1)

evol262 (721773) | more than 5 years ago | (#28371809)

I don't pretend to understand Intel's modeling scheme here. A small bump (55xx), with some of them having HyperThreading, some not, some have QPI, some not, etc. It's not exactly intuitive, and I wouldn't have known to look there either if we weren't using some of them.

Google (0)

Anonymous Coward | more than 5 years ago | (#28361431)

Google has already mapped the moon, mars, earth, working on the oceans, why not just leave the sky to them? they have a pretty good sucess rate

Re:Google (1)

KasperMeerts (1305097) | more than 5 years ago | (#28361719)

Done. [google.com]

Google never really mapped it though, they just offer old material in a user-friendly way.

Re:Google (1)

floateyedumpi (187299) | more than 5 years ago | (#28361813)

Google is definitely jumping into this game in a big way with their planned involvement in LSST [lsst.org] . Key point:

Over 30 thousand gigabytes (30TB) of images will be generated every night during the decade -long LSST sky survey.

So basically (4, Funny)

Norsefire (1494323) | more than 5 years ago | (#28361451)

it's a skynet?

Re:So basically (0)

Random2 (1412773) | more than 5 years ago | (#28361491)

No, it's skynet, mate!

Re:So basically (1)

sdpuppy (898535) | more than 5 years ago | (#28364209)

Certainly. They're uploading to the Sun, no?

They're also uploading to SGI since the Sun only has one core.

ducks

Two Words (2, Funny)

ehaggis (879721) | more than 5 years ago | (#28361477)

Beowulf Cluster

Re:Two Words (0)

Anonymous Coward | more than 5 years ago | (#28361517)

No, no. Crysis and Vista. Wait, that's 3 words. Oh, well.

Re:Two Words Beowolf Cluster... (1)

davidsyes (765062) | more than 5 years ago | (#28363443)

Can be bay like wolves and sing Men At Work's "Land Down Under"?

http://www.youtube.com/watch?v=DNT7uZf7lew [youtube.com]

But, change the song to Chips At Work's "Skies Up Over", hehehehehe....

The Math! It hurts the brain (2, Insightful)

dmomo (256005) | more than 5 years ago | (#28361607)

next generation Sun supercomputer kitted out with 12,000 cores ... will offer a tenfold increase in performance over the .. two SGI machines, each with just under 3500 cores in total

How is that 10x faster? I imagine because the new v. old cores are not equally comparable. In that case, why talk number of cores at all?

Re:The Math! It hurts the brain (2, Funny)

kburk (1352309) | more than 5 years ago | (#28361721)

ya, but can it run crysis?

Re:The Math! It hurts the brain (2, Informative)

evol262 (721773) | more than 5 years ago | (#28361725)

They're moving from 1.6Ghz single core Itaniums to dual quad core Xeon blades. I suspect they're talking about cores to emphasize the density gain, and because people like huge numbers.

Re:The Math! It hurts the brain (5, Funny)

kenp2002 (545495) | more than 5 years ago | (#28361777)

"Old Man Rant"

Why do I cring every time I hear people use terms like Tenfold and order of magnatitude....

From what I gather the whole 10 Fold, 3 Fold, was more about the progressive thickness of cloth in relation to the number of folds back in the war when we made planes out of canvas.

1mm thick material when increase 3 fold is

1 -> 2 -> 4 -> 8 mm thick. Ten fold would then be 512 mm thick...

Why are we talking about folding stuff? Where are the protients... WHa? I DON'T WANT TO TAKE THE PILLS! WHO ARE YOU PEOPLE! LEAVE ME ALONE!

I'M NOT DOING ANY LAUNDRY! ;)

Re:The Math! It hurts the brain (0)

Anonymous Coward | more than 5 years ago | (#28362149)

From what I gather the whole 10 Fold, 3 Fold, was more about the progressive thickness of cloth in relation to the number of folds back in the war when we made planes out of canvas.

tenfold [merriam-webster.com]

Re:The Math! It hurts the brain (0)

Anonymous Coward | more than 5 years ago | (#28364153)

Maybe we should use expressions like "a 10dB increase in cores" or "a 3dB increase in memory"? How about the current chips features are -1.7dB those of the old chips?

Whatever...I'm going back to trying to get 120.827dBm into my flux capacitor.

Re:The Math! It hurts the brain (0)

Anonymous Coward | more than 5 years ago | (#28364397)

Couple of items. First, the term is much older. Old enough that we're not certain when it starts, so a number of traditional industries are candidates for the origin once we wrote things down. Second, not canvas, but doped linen for fabric-covered aircraft of the Great War. (There were also plywood and metal covered aircraft at the time.) Sometimes cotton, then more recently dacron, and around the time I was getting out of it (80s), there was a sort-of thin fiberglass-like covering being introduced. Maybe called 'razorback'? I keep meaning to check to see how that one worked out for longevity, and for fuss during inevitable renewal.

"Canvas" _is_ a fairly broad term, but the stuff used for aircraft isn't the same sort of weight as what's used for canoes and paintings, and should not be included. The term to use is "fabric", when not being specific about which fabric.

Re:The Math! It hurts the brain (2, Interesting)

Xtifr (1323) | more than 5 years ago | (#28364731)

From what I gather the whole 10 Fold, 3 Fold, was more about the progressive thickness of cloth in relation to the number of folds

That's an...interesting theory, but I can't find anything to support it. My M-W dictionary says the phrase goes back to the 12th century (so it has nothing to do with making planes of any type), and clearly states that "tenfold" means ten times [merriam-webster.com] , so your suggestion that it "really" means 2^10 is simply false. My own guess is that this ancient phrase has more to do with "in the fold" (where you find sheep, or perhaps wolves) than with cloth, but I can't prove that either.

Re:The Math! It hurts the brain (1)

kenp2002 (545495) | more than 5 years ago | (#28364783)

you so totally missed the joke...

Re:The Math! It hurts the brain (1)

Xtifr (1323) | more than 5 years ago | (#28368247)

Ah well, wasn't the first time, and certainly won't be the last. :)

Re:The Math! It hurts the brain (0)

Anonymous Coward | more than 5 years ago | (#28365487)

1 -> 2 -> 4 -> 8 mm thick. Ten fold would then be 512 mm thick...

2^10=1024. It's a pretty useful fact actually.

Re:The Math! It hurts the brain (1)

Plagued by Penguins (11693) | more than 5 years ago | (#28372269)

it's ~10x faster on specfp_rate than the itanium2 altix.
specfp turns out to be a good representation of the codes we typically run...

Marvel of efficiency (3, Funny)

nightfire-unique (253895) | more than 5 years ago | (#28361615)

Sun servers running perl you say?

I'd say that has the price:performance ratio of a Rolls Royce. And that's not a good thing. :)

Re:Marvel of efficiency (1)

nizo (81281) | more than 5 years ago | (#28361713)

Maybe they need that kind of power to run the database? (Don't hurt me!)

Seriously though, can I have their crummy old machines when they are done with them?

Re:Marvel of efficiency (0)

Anonymous Coward | more than 5 years ago | (#28361745)

You ever try running a regex on a 470 terabyte string?

Re:Marvel of efficiency (0)

Anonymous Coward | more than 5 years ago | (#28361863)

PERL's interpreted -- wouldn't there be a significant speed increase if they switched to something compiled? Like C? I'm sure that's now always that case, but just curious...

Re:Marvel of efficiency (1)

ivarneli (4238) | more than 5 years ago | (#28361949)

I think it's unlikely that Perl is being used for any of the heavy computations. I would guess that Perl is being used as a glue language that interfaces with the database and programs written in lower-level languages to do the heavy lifting.

Perl power!!! (2, Insightful)

Murpster (1274988) | more than 5 years ago | (#28361639)

Good to see the coolest language around being put to a(nother) cool use.

Re:Perl power!!! (1)

reginaldo (1412879) | more than 5 years ago | (#28365837)

Yes! Perl is so well suited for ETL processing, it makes me happy when people use it.

At my current job I want to take our Informatica server and throw it out the window. That way I can use Perl.

Hmmm... (3, Funny)

kenp2002 (545495) | more than 5 years ago | (#28361673)

With that much data we might finally have enough information to generate a singular point of reference in space and time so we can retrofit a poorly designed all stainless steel car and travel back in time 200 years and not find out selves drifting in the middle of nowhere since 200 years ago, relative to some unknown non-moving reference point, our planet, solar system, and galaxy is probably no where near where it was 200 years ago!

FLUX CAPACITOR FTW!

MMMMmmm.... (2, Funny)

amstrad (60839) | more than 5 years ago | (#28361677)

Imagine a open/globular cluster of those....

Re:MMMMmmm.... (1)

spaceyhackerlady (462530) | more than 5 years ago | (#28362569)

But does it run Linux?

Joking aside, this sounds like a wonderful project, both on the astronomy side and on the technology side. I can't wait to see what they find out with it.

...laura

Re:MMMMmmm.... (1)

MaxwellEdison (1368785) | more than 5 years ago | (#28364311)

Joking aside, this sounds like a wonderful project, both on the astronomy side and on the technology side. I can't wait to see what they find out with it.

My God! It's full of stars!

Okay...now I am willing to put joking aside.

Re:MMMMmmm.... (2, Funny)

CyberLord Seven (525173) | more than 5 years ago | (#28365317)

Someone used to have a wonderful sig:

My dog! It's full of rats!

2001: A Dyslexic Odyssey

No, I don't know the name. I almost never look at names, just sigs.

Re:MMMMmmm.... (1)

spaceyhackerlady (462530) | more than 5 years ago | (#28365419)

Sounds like the old joke about the agnostic dyslexic insomniac.

Who used to lie awake at night wondering if there really was a Dog.

...laura

So... (1)

Ltap (1572175) | more than 5 years ago | (#28361699)

It's the Skynet? In all seriousness, it sounds awesome and it's good to know that they're using the right tools for the job (perl to organize, postgreSQL to manage large amounts of data).

standard GPU board beats an SGI super these days (1, Informative)

peter303 (12292) | more than 5 years ago | (#28362191)

SGI stopped making supers many years ago.

Re:standard GPU board beats an SGI super these day (3, Informative)

evol262 (721773) | more than 5 years ago | (#28362229)

The Altix line begs to differ. They're not using Onyxes.

PostgreSQL capacity? (1)

johannesg (664142) | more than 5 years ago | (#28362503)

How do they store so much data into PostgreSQL? I thought it had a limit of 32TB per table, so are they using some sort of table partitioning scheme?

Re:PostgreSQL capacity? (1)

TooMuchToDo (882796) | more than 5 years ago | (#28362561)

They're probably only storing the refined data in PostgreSQL, with the raw data being stored as flat files on disk and tape. This is how we do it with 300TB of disk and 17PB of tape.

Per usual - I'm confused (1)

earlymon (1116185) | more than 5 years ago | (#28362673)

Sounds like a great step forward - and I want to be on record saying that I'm pulling for them.

However - I'm confused, per usual. As Oracle now owns Sun, what's to say that the camel won't stick his nose into the tent, seeing a database opportunity (read: marketing opportunity) and try to pressure, cajole, coerce, or otherwise influence them to drop a working PostgreSQL in favor of an all-Oracle (Sun) solution? I'm not saying that one is better than the other - I'm just concerned that a political motive on the supplier's part can have ripple effects to the tech guys making it happen.

I freely admit my confusion, so this may be a big non-issue. I just don't know.

Sounds like a job for... (1)

DerekLyons (302214) | more than 5 years ago | (#28362999)

Sounds like a job for BOINC [berkeley.edu] .

Yes, but (1)

KiwiCanuck (1075767) | more than 5 years ago | (#28363107)

"next generation Sun supercomputer kitted out with 12,000 cores." Can it run Crysis with full eye-candy?

Supercomputer power (0)

binkzz (779594) | more than 5 years ago | (#28363407)

"Sun supercomputer kitted out with 12,000 cores." " They collect 700 GB of data every night, which they then crunch down using some perl scripts "

Doesn't that mean it could be run from a home computer running c code instead?

Google Universe? (0)

Anonymous Coward | more than 5 years ago | (#28364053)

Question - when can i cruise a model of the galaxy from my laptop? I wanna see some of the cosmos up close before i die. I think thats what everyone who's not a super geek wants to get out of this (and the geeks want it too I'm sure).

Re:Google Universe? (1)

rlseaman (1420667) | more than 5 years ago | (#28364467)

I wanna see some of the cosmos up close before i die.

Look at the WorldWide Telescope from Microsoft Research: http://www.worldwidetelescope.org/ [worldwidetelescope.org] and Sky from Google: http://www.google.com/sky/about.html [google.com]

These fill somewhat different niches and both provide compiled and web-based clients.

Almost good enough (1)

SnarfQuest (469614) | more than 5 years ago | (#28364669)

It sounds like it almost has enough power to run Windows Vista with options turned on. Too bad you can only run three programs at a time though.

Imagine a beowulf cluster of these...

Perl - One of the best for ETLs (1, Interesting)

Anonymous Coward | more than 5 years ago | (#28366237)

Back in 2004 I calculated contract interests for Columbia Pictures in PERL the data 2.4 Terabytes(took a day using 24 crappy pentium IVs). This doesn't surprise me check this out

look for duplicated words in a line
        perl -0777 -ne 'print "$.: doubled $_\n" while /\b(\w+)\b\s+\b\1\b/gi' foo.txt

or to cheat in scrabble in Unix

input: tolkien

perl -lne'print if /t/ && /o/ && /l/ && /k/ && /i/ && /e/ && /n/ && length($_)==8' /usr/share/dict/words

output:
knotlike
townlike

Just try writing that in C,
Perl is actually quite fast if you use a bunch of one-liners -

western NSW??? (1)

gfim (452121) | more than 5 years ago | (#28368919)

The article describes Coonabarabran as being the "central-west NSW location". Now, Coonabarabran certainly isn't very far west [google.com] in NSW. But, then I realised that they mean population-wise. There aren't too many people west of there!!!

spare core cycles (0)

Anonymous Coward | more than 5 years ago | (#28369613)

...so are they going to use the 12000 cores spare CPU cycles for F@H or SETI....lol

Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>