Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Supercomputer Becomes Massive Router For Global Radio Telescope

Soulskill posted about a year ago | from the go-big-or-go-home dept.

Supercomputing 60

Nerval's Lobster writes "Astrophysicists at MIT and the Pawsey supercomputing center in Western Australia have discovered a whole new role for supercomputers working on big-data science projects: They've figured out how to turn a supercomputer into a router. (Make that a really, really big router.) The supercomputer in this case is a Cray Cascade system with a top performance of 0.3 petaflops — to be expanded to 1.2 petaflops in 2014 — running on a combination of Intel Ivy Bridge, Haswell and MIC processors. The machine, which is still being installed at the Pawsey Centre in Kensington, Western Australia and isn't scheduled to become operational until later this summer, had to go to work early after researchers switched on the world's most sensitive radio telescope June 9. The Murchison Widefield Array is a 2,000-antenna radio telescope located at the Murchison Radio-astronomy Observatory (MRO) in Western Australia, built with the backing of universities in the U.S., Australia, India and New Zealand. Though it is the most powerful radio telescope in the world right now, it is only one-third of the Square Kilometer Array — a spread of low-frequency antennas that will be spread across a kilometer of territory in Australia and Southern Africa. It will be 50 times as sensitive as any other radio telescope and 10,000 times as quick to survey a patch of sky. By comparison, the Murchison Widefield Array is a tiny little thing stuck out as far in the middle of nowhere as Australian authorities could find to keep it as far away from terrestrial interference as possible. Tiny or not, the MWA can look farther into the past of the universe than any other human instrument to date. What it has found so far is data — lots and lots of data. More than 400 megabytes of data per second come from the array to the Murchison observatory, before being streamed across 500 miles of Australia's National Broadband Network to the Pawsey Centre, which gets rid of most of it as quickly as possible."

Sorry! There are no comments related to the filter you selected.

400 Mb per seconds (1)

vikingpower (768921) | about a year ago | (#44368217)

Is that much ? If it is structured, and if the processing of it requires taking the structure into account - well hell yes, then that is humongous.

Re:400 Mb per seconds (2, Informative)

Anonymous Coward | about a year ago | (#44368287)

Most of it is noise you can throw away quickly. After that point it gets more and more difficult to choose so you need balance processing+storage+bandwidth
CERN ran into similar problems but at least they had a part of the science done on-site. (a week in geneva is way better than a week in the middle of the fucking desert)
Space people have kind of the opposite problem, since they have very limited on site storage/processing power and limitations in bandwidth/telemetry and they cant just dump more computers to solve the problem (rad hard electronics are not cheap and weight is counted in million$ up there). Usually the end result is bitter sacrifices of valuable data and bitter fights in the community on whose instrument will get to send back stuff.

Re:400 Mb per seconds (4, Informative)

the_other_chewey (1119125) | about a year ago | (#44368347)

Most of it is noise you can throw away quickly.

In the case of the Square Kilometer Array (named for its total collection area by the way,
not because it is "spread across a kilometer of territory", whatever that's supposed to mean),
none of it is noise.

The SKA relies heavily on processing everything, using advanced phased-array
and other "inverse beam-forming" techniques to look at multiple targets in multiple
frequency ranges at once (the final design will have continuous coverage from
70 MHz to 30 GHz!).

This is only possible with centralised processing, so none of the antenna sites can throw
anything away: They don't know what will be important.

Re:400 Mb per seconds (1)

Immerman (2627577) | about a year ago | (#44369907)

>This is only possible with centralised processing, so none of the antenna sites can throw
anything away: They don't know what will be important.

Even more than that, *all* of it is potentially important. As I understand it phased arrays pretty much require the whole signal from all the antennas to get the benefit of having the antennas at all, it's not until *after* the signals are combined and processed that you can weed out the data you're not interested in. In fact based on 20s of reading on phased arrays I get the impression that the multi-directionality ability, etc. is likely determined by the transformation function you use to combine the raw data, so the same raw data can create "images" of various directions.

Re:400 Mb per seconds (1)

Rich0 (548339) | about a year ago | (#44370153)

Traditionally phased array was done by feeding the raw signals to a central point and then the "processing" is analog (circuits, not algorithms). The output is a single signal that contains the desired "image" which then goes into an A/D. At any time you only can look at the data in one way, since the raw data is not captured (raw being the data from each individual antenna).

That works great for a radar on a ship where the antennas are all next to each other and where you can just rapidly steer back and forth, or where you are only tracking a single point and just don't want to use servos to do it. When the antennas are spread across a large area then you can't just run antenna feeds directly into a central box due to signal loss (even with amplification). It also doesn't work if you want to capture wideband data and look at all directions simultaneously after the fact. For that you need to digitize every antenna feed, and you also need an absolute time reference (which means lots of atomic clocks unless the sites are close enough to share a reference).

Re:400 Mb per seconds (0)

Anonymous Coward | about a year ago | (#44368491)

a week in geneva is way better than a week in the middle of the fucking desert

Speak for yourself...some of us find interest no matter where we are.

Fuck off you small-minded ignoramus.

Re:400 Mb per seconds (1)

war4peace (1628283) | about a year ago | (#44369101)

" before being streamed across 500 miles of Australia's National Broadband Network to the Pawsey Centre, which gets rid of most of it as quickly as possible."

I imagine a bunch of Indian and Chinese people pressing Shift+Delete randomly on files. Their target: 90% resolution rate on incoming data :)

Re:400 Mb per seconds (1)

NixieBunny (859050) | about a year ago | (#44368405)

That's not much data by radio astronomy standards. The typical millimeter-wave VLBI experiment records data about 10 times that fast in aggregate, onto hard disk drives that are shipped to a central correlator facility.

Re:400 Mb per seconds (1)

vikingpower (768921) | about a year ago | (#44368511)

Shipping hard disk drives ? Aha. Now that is a way to increase bandwidth, remember the old adagium "Never underestimate the bandwidth of a Boeing 747 full of DVDs" ;-)

Re:400 Mb per seconds (1)

war4peace (1628283) | about a year ago | (#44369151)

As a matter of fact...
OK, I may be too pedantic, but a 747 full of DVDs is just large storage, wildly different from large bandwidth.
When you stream something, the data is immediately ready for processing as it comes (provided it's structured with that goal in mind). On the other hand, a 747 full of DVDs is data that must be read before it's ready for processing, and the average DVD read speed is more or less 100 Mbps, maybe a bit more than that but not by much. Throw time spent writing those DVDs into the mix and you'll get a shitty bandwidth, if you really want to go as far as calculating a bandwidth equivalent.

Let's assume fly time is zero, just for kicks. Now for a DVD it takes you 10 minutes to write it (at high speeds) and 10 minutes to read it, that's 20 minutes per DVD. Say you can write/read 100 DVDs at the same time, that's roughly 430 GB every 20 minutes, that's roughly equivalent to a bandwidth of 367 MB/second. That's provided all DVDs are readable and you have tens of people you allocate to this project.

Re:400 Mb per seconds (0)

Anonymous Coward | about a year ago | (#44369549)

When your project is located in the middle of nowhere and the only bandwidth available is satellite (or crappy satellite a couple hours of the day in the case of some high data rate projects at the South Pole), 100 Mb/s would be an amazing deal. No need to make any assumptions about zero fly time, that is just latency, which all communication methods have to some varying degree. Projects like IceCube send data in on tape once a year, although filtered results are available via satellite. A lot of the analysis waits to be done once a year anyway, as the statistics are useless doing it one day at a time and a long observing period is needed.

Re:400 Mb per seconds (1)

Immerman (2627577) | about a year ago | (#44369965)

So use SSDs instead. The point being though that I can get maybe 1GB/s with a high-speed data link, or umpteen PB/s with a truck full of storage media (I first heard the maxim as "...a station wagon full of floppy disks".

As for bandwidth reading the data, sure you'd need a lot of connections to get anywhere near that. Heck, you'd need a lot of computers to process data that quickly - a single PC with dual channel DDR2-800 RAM has a maximum data throughput (no processing, just reading it from memory) of only 800M lines/second * 64bits/line * 2 interfaces = 12.8GB/s

Re:400 Mb per seconds (1)

epiphani (254981) | about a year ago | (#44368929)

It's not. It's really not much at all. For $150k, you could build a hadoop cluster that would happily accept the data stream, process it, and make it available for consumption. If you just want to store it, you don't even need that much.

That's a waste of a Cray. Well, a Cray is a waste of money these days anyway.

Re:400 Mb per seconds (0)

Anonymous Coward | about a year ago | (#44369631)

But you're just making a big assumption about how much computation and what kind of computation actually needs to be done on the data. It looks like a lot of their real time processing involves calibration and cross-correlation computations, which require considerable crosstalk between the different pieces of data. As is, they are already using a lot of custom hardware instead of general purpose computers to do a lot of the processing fast enough.

Re:400 Mb per seconds (1)

epiphani (254981) | about a year ago | (#44370007)

Not really. The real-time components (aka correlation) are basically just straight up FFTs. Custom hardware in correlators might make sense (and probably does at scale), but through ASICs or FPGAs. They're not doing that (...yet). Throw a GPU or two into each node, and you'd get far more FLOPS than you would with a cray. This work is mostly embarrassingly parallel, so throwing money into cray's is a total waste of time.

Re:400 Mb per seconds (0)

Anonymous Coward | about a year ago | (#44370979)

Correlation is just simply a multiplication of all the signals from a single antenna pair called a baseline that results in a combined signal from a single beam or part of the sky. An interferometric telescope like the MWA has lots of baselines that mimic the collecting are of a very large dish without having to build a dish. The MWA is in fact using a combination of FPGA's and GPU's to form and correlate the signal from the array. It's my understanding its all very parallel in nature. The Pawsey part is simply being used as an archive for the data which is also used to push data to other archives.

Re:400 Mb per seconds (0)

Anonymous Coward | about a year ago | (#44371225)

The correlators are done in specific hardware and not general purpose computers (using FPGAs in this case I think). But the real-time computer has to handle calibration against over 100 calibration sources and apply a ionosphere model to correct the data for correlations. Data is binned into 8 second segments, so all of the calculations have to be done in 8 seconds. Except they already have done extensive work on creating GPU algorithms for this, and have a setup with 64 GPU nodes to do such calculations. It wasn't just about speed, but they had total power limits too due to that being on site in the middle of nowhere.

Of course this is all irrelevant to the Cray computer discussed above, which is not the on-site real time processing computer, it is instead a large computer installation in a suburb of Perth. This computer is not dedicated to processing of MWA data 24/7, and like most supercomputer centers, computing time is shared among several projects. You can't replace this with just a small cluster, as much as any other large time share computer that is actually being used can't be replaced by a smaller cluster.

Re:400 Mb per seconds (1)

TheTrueScotsman (1191887) | about a year ago | (#44369071)

It's actually 400 MB. You need to get this sort of thing right if you're planning a career in the tech industry when you finish school.

Re:400 Mb per seconds (1)

vikingpower (768921) | about a year ago | (#44372745)

Thanks, mate. Must have been in the wrong career for 19 years, then. Glad you are not my not-a-single-typo-forgiving-boss ;-)

What a bad summary. (2, Insightful)

Anonymous Coward | about a year ago | (#44368241)

A lot of waffling that tells me nothing about the premise. Why did they do it, why did they need to, what made that thing uniquely suitable so nothing else would do?

HEY EDITORS. DO YOUR JOB ALREADY, DAMMIT. STOP WASTING MY TIME.

400MB/s (4, Funny)

Thanshin (1188877) | about a year ago | (#44368245)

More than 400 megabytes of data per second come from the array to the Murchison observatory, before being streamed across 500 miles of Australia's National Broadband Network to the Pawsey Centre

They forgot to mention the step where the 400 MB go to the NSA to be checked for signs of extra terrestrial terrorism.

Re:400MB/s (0)

Anonymous Coward | about a year ago | (#44368497)

oh for mod points...or an account for that matter, but hey this is slashdot, accounts are for karma whores....

oops! ;oP

Two, actually! (1)

Impy the Impiuos Imp (442658) | about a year ago | (#44368253)

What it has found so far is data — lots and lots of data. More than 400 megabytes of data per second come from the array

Well, I knew someone on this planet actually needed gigabit Internet if we looked hard enough.

Re:Two, actually! (1)

rex.clts (2791393) | about a year ago | (#44368279)

Note that GigaBIT Ethernet tops out at ~119 MegaBYTEs per second. You're going to need a ~3.3 Gbps link, not including overhead.

What's the FPS on 400mb/s? What resolution? (0)

Anonymous Coward | about a year ago | (#44368261)

Lose excess weight early... less supercomputing needed to route your ass around space...

Summer? (5, Informative)

mjwx (966435) | about a year ago | (#44368281)

I live in Western Australia and it's winter here.

Later "this summer" doesn't start until December.

500 miles

For those of us who dont use archaic measurements, it's 800 KM from the city of Perth, which makes it 800 KM from the closest city. If anyone is interested, here's the google maps link [google.com.au] and it's distance to Perth, Western Australia. [google.com.au] . There's literally nothing out there, picking up an AM radio station is difficult, making it the perfect place for a telescope.

If you truly want to get lost, you need to go somewhere like Murchison, no-one will find you. Of course just about everything there is trying to kill you, from King Brown snakes to Land Sharks and Koala Drop Bears.

Re:Summer? (3, Informative)

Javaman59 (524434) | about a year ago | (#44368425)

I live in Western Australia and it's winter here.

I live in South Australia, and it's winter here, too.

Later "this summer" doesn't start until December.

I would say it does, because using seasons as a unit of time is a distinctly Northern hemisphere convention. In my observation, American's and Canadians are the main users of it (more than the British).

I often get confused talking to an American when they talk about doing something "in the summer", and it's not so much that they have a different summer, but that I'm not used to measuring time like this. (We only use it for things that are specifically related to the weather, such as sports).

In Australia we wouldn't say "later this winter", we'd just say "around August/September".

Re:Summer? (1)

Swampash (1131503) | about a year ago | (#44368435)

Yeah, "product scheduled for release this autumn", wtf does that mean?

Still, the USA uses Imperial measurements so it's not exactly hip to, you know, measurements that people can actually understand.

Re:Summer? (0)

Anonymous Coward | about a year ago | (#44369585)

88 percent of the world's population live in the northern hemisphere. So most people have summer at about the same time as Americans. I can hardly blame them for not taking a small minority into account for intentionally vague time descriptors.

Perhaps they should make sure to convert dates into the Jewish calendar if someone from Israel might overhear them?

Re:Summer? (1)

ebno-10db (1459097) | about a year ago | (#44372051)

the USA uses Imperial measurements so it's not exactly hip to, you know, measurements that people can actually understand.

Don't they teach arithmetic in your country?

Re:Summer? (1)

Artea (2527062) | about a year ago | (#44368507)

Thats because we only have "summer" and "wet summer", it kind of makes the measurement somewhat vague.

..Ok fine, so it did hit 8 degrees this morning, but it's 18-20 during the day which would be considered a warm spring day for some parts of the US.

Re:Summer? (1)

fast turtle (1118037) | about a year ago | (#44369517)

18-20 isn't a warm spring day. Hell I consider anything less then 25-35 to be a cold day during the spring as we routinely hit 40-45 during this time of year.

Of course, if we even hit 8 during the period Dec-March we're suffering a heat wave as it's usually closer to -8 here during that period and the funny thing is, I'm only 300Km from Los Angeles.

Re:Summer? (1)

Anonymous Coward | about a year ago | (#44370415)

In my observation, American's and Canadians are the main users of it (more than the British).

Not true. We, Canadians, use the following seasonal measurements to indicate the time of the year: Almost winter, winter, still winter, and construction.

Re:Summer? (1)

ebno-10db (1459097) | about a year ago | (#44371871)

In my observation, American's and Canadians are the main users of it (more than the British).

Not true. We, Canadians, use the following seasonal measurements to indicate the time of the year: Almost winter, winter, still winter, and construction.

On the plus side there is very little risk of heat stroke.

Re:Summer? (1)

ebno-10db (1459097) | about a year ago | (#44371915)

I live in Western Australia and it's winter here.

It's currently the middle of the night in Perth, and still 12C. Tomorrow's high is forecast to be 20C. That is not winter.

Re:Summer? (1)

mjwx (966435) | about a year ago | (#44376133)

I live in Western Australia and it's winter here.

It's currently the middle of the night in Perth, and still 12C. Tomorrow's high is forecast to be 20C. That is not winter.

Yes it is.

Summer in Perth is 40 Deg C.

I would use this for (1)

Davo Batty (2855025) | about a year ago | (#44368309)

an intergalatic radio station to beam "24 hour rap at full volume" which should scare off any aliens.

Misleading summary and first article (5, Informative)

amaurea (2900163) | about a year ago | (#44368361)

The Square Kilometer Array will have a *collecting area* of one square kilometer. That means that if you add up the area of all the detectors, you get one square kilometer. Since there is some distance between each detector, the SKA will cover a ground area *much* larger than a square kilometer.

Part of the SKA will be built in the MRO-area in Australia. But it is far from finished - construction won't begin in earnest until 2016 I think. So the most powerful radio telescope in the world is not at MRO now. It is LOFAR in Europe.

Re:Misleading summary and first article (5, Informative)

ogre7299 (229737) | about a year ago | (#44368415)

The article also washes over the fact that there are different telescopes for different parts of the radio spectrum. The MWA and LOFAR are the most powerful in the MHz regime, but the VLA is still the most powerful between 1 to 50 GHz, and ALMA is the most powerful from 85 and 700 GHz.

Re:Misleading summary and first article (5, Informative)

amaurea (2900163) | about a year ago | (#44368475)

Right. And then there are the issues of resolution and survey area. Planck covers the same frequency range as ALMA, but measures the whole sky in total intensity and polarization, for example, and is much better at measuring the CMB than ALMA. So the term "powerful" is an over-simplification.

Re:Misleading summary and first article (0)

Anonymous Coward | about a year ago | (#44368789)

the VLA is still the most powerful between 1 to 50 GHz

In mapping speed, perhaps, but I think that Arecibo's huge collecting area gives it better sensitivity, and it has a few receivers in that range...

Farther into the past? (-1, Troll)

Swampash (1131503) | about a year ago | (#44368441)

Than any other instrument to date?

Newsflash atheist, there's this instrument called THE BIBLE. It looks back as far as it's possible to look. 6000 years, right to the beginning.

Re:Farther into the past? (0)

Anonymous Coward | about a year ago | (#44370159)

It only looks as far back as the beginning of Earth and humans. The aliens that terraformed the planet and engineered humans (modeled after their own DNA) existed much longer before that.

Get your head out of your ass, you ignorant Christian. Did you not READ Genesis? It's all described there.

Petaflops (2)

Nedmud (157169) | about a year ago | (#44368563)

Well it sure can do a lot of floating point operations per second; how does that help for networking applications exactly?

Re:Petaflops (1)

White Flame (1074973) | about a year ago | (#44368615)

Ditto. Also, in many "big data" projects, FLOPS is of little use anyway. There is a ton of textual processing and predicate matches to be done in the rest of the world. With ARM entering the HPC space, hopefully more broadly meaningful integer & IO ops numbers will be bandied about rather than just this laser-focus on vector floats.

Re:Petaflops (0)

Anonymous Coward | about a year ago | (#44369371)

Non-ditto. For this sort of HPC application, FLOPS are of utmost importance. You can talk about textual processing all you want, for those in the field FLOPS are still the priority.

Re:Petaflops (0)

Anonymous Coward | about a year ago | (#44371129)

I/O is important to capture that data and FLOPS are equality important when processing so much data.

Getting rid of data? (1)

Celarent Darii (1561999) | about a year ago | (#44368599)

From the article:

before being streamed across 500 miles of Australia's National Broadband Network to the Pawsey Centre, which gets rid of most of it as quickly as possible.

Get rid of data? Don't you mean routing the data to its destination? And you would hope the Pawsey Centre actually DID something with the data and not just get rid of it.

Re:Getting rid of data? (0)

Anonymous Coward | about a year ago | (#44370295)

From the article:

before being streamed across 500 miles of Australia's National Broadband Network to the Pawsey Centre, which gets rid of most of it as quickly as possible.

Get rid of data? Don't you mean routing the data to its destination? And you would hope the Pawsey Centre actually DID something with the data and not just get rid of it.

No, I think OP really did mean that the Pawsey Centere throws away most of the 400 MB/s that they receive from the array. Think of it as a form of lossy compression, like MP3. They know an awful lot about the data they're collecting and they can quickly decide what parts are important or interesting, and which are noise that can be safely removed. I don't know what portion of the time the array is on and collecting data, but 400 MB/s would add up fast to a storage headache.

Re:Getting rid of data? (0)

Anonymous Coward | about a year ago | (#44371069)

Every single bit that is produced from the array is sent over the NBN link to Pawsey where it is archived in long term storage. The array can saturate the link quite easily and can be run up to 10Gb/s.

Lots of data? (0)

Anonymous Coward | about a year ago | (#44368975)

Send it to the NSA for them to sort out.

Wrong technology (0)

Anonymous Coward | about a year ago | (#44369221)

Unfortunately alien civilizations abandoned electromagnetic waves a million years ago. They only toyed with them for about 200 earth-years.Shortly after they had toyed with smoke signals. Build your entanglement antenna and they will tell you all about it because their communications are all around everywhere RIGHT NOW! They can also send you the best pictures of the universe. Don't waste your time playing with the wrong technology.

Routing? (2)

mc1138 (718275) | about a year ago | (#44369561)

So... anyone actually know more about the "routing" part of this. All I saw was that they turned it into a "really big router" whatever that means, and then talk about the array. I'm assuming they're using the super computer to actually make the decisions of who is getting what data in real time, and sending it to the correct place, but they don't really talk about that at all. Anyone have a better link?

Re:Routing? (0)

Anonymous Coward | about a year ago | (#44371045)

The routing of data means exactly that, it pushes data to other facilities that want the same data. The routing of data is only a small part of what the Pawsey center does. It has other capabilities that allows data to be processed and archived.

Re:Routing? (2)

ebno-10db (1459097) | about a year ago | (#44371797)

Good ol' Wikipedia has a decent description of the overall system: http://en.wikipedia.org/wiki/Murchison_Widefield_Array [wikipedia.org]

An educated guess is describing it as a router is ridiculous. It's more like intelligently combining the M incoming data streams (beam forming) so that the data can be shipped at a lower bandwidth to N universities (each of which may be using a different combination of incoming data and hence looking at a different beam).

One of the nice things about phased array (electronically steered) antennas is that you can simultaneously receive signals from N "virtual antennas" (usually called beams in the business), each of which may be pointed in a different direction and have a different beam width, frequency and bandwidth. You create those N virtual antennas by combining the input signals from the M physical antennas in N different ways. The combined signals are of much lower bandwidth than the incoming signals. Hence you could have people at university A looking at one place in the sky, the people at university B simultaneously looking at a different place in the sky, and have both of them receiving real-time signals.

Keep an axe handy.... (1)

TimO_Florida (2894381) | about a year ago | (#44372667)

If it renames itself Colossus and starts looking for routes to Guardian, CUT THE LINES!

Crazy Americans (0)

Anonymous Coward | about a year ago | (#44376577)

Science aside, I found it hilarious that in the article they refer to the location as being in "Western Australia’s Mid West". Calling some deserted place 800KM north of Perth as the "mid west" sounds like a crazy Americanised geographical term.

Not the National Boardband Network (1)

gdtau (1053676) | about a year ago | (#44378353)

It doesn't use the NBN. That's an optical access network for residential housing and small business with an access rate of 100Mbps. It uses AARNet -- Australia's Academic and Research Network -- which has installed multiple 100Gbps links across Australia for this project.
Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?