×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

The Problem With the Top500 Supercomputer List

Soulskill posted more than 3 years ago | from the nobody-cares-about-the-bottom-490-or-so dept.

Supercomputing 175

angry tapir writes "The Top500 list of supercomputers is dutifully watched by high-performance computing participants and observers, even as they vocally doubt its fidelity to excellence. Many question the use of a single metric — Linpack — to rank the performance of something as mind-bogglingly complex as a supercomputer. During a panel at the SC2010 conference this week in New Orleans, one high-performance-computing vendor executive joked about stringing together 100,000 Android smartphones to get the largest Linpack number, thereby revealing the 'stupidity' of Linpack. While grumbling about Linpack is nothing new, the discontent was pronounced this year as more systems, such as the Tianhe-1A, used GPUs to boost Linpack ratings, in effect gaming the Top500 list." Fortunately, Sandia National Laboratories is heading an effort to develop a new set of benchmarks. In other supercomputer news, it turns out the Windows-based cluster that lost out to Linux stumbled because of a bug in Microsoft's software package. Several readers have also pointed out that IBM's Blue Gene/Q has taken the top spot in the Green500 for energy efficient supercomputing, while a team of students built the third-place system.

cancel ×
This is a preview of your comment

No Comment Title Entered

Anonymous Coward 1 minute ago

No Comment Entered

175 comments

Quelle surprise! (4, Insightful)

Elbart (1233584) | more than 3 years ago | (#34283128)

Now that the Chinese are ahead, there's suddenly a problem with the list/benchmark.

Re:Quelle surprise! (1, Insightful)

Daishiman (698845) | more than 3 years ago | (#34283212)

Agreed. It seems like the issue is "big enough" only now that other people are catching up.

Re:Quelle surprise! (1)

Z00L00K (682162) | more than 3 years ago | (#34284056)

And any number presented only has a meaning if it's applicable to the problem you have.

If you have a problem that scales well to TPC-C it may be useless to go by Linpack numbers to find the solution you are after. And the other way around.

Re:Quelle surprise! (4, Insightful)

natet (158905) | more than 3 years ago | (#34284360)

Agreed. It seems like the issue is "big enough" only now that other people are catching up.

I call bullsh*t on this comment. Around 8 years ago, the top computer on the list was a Japanese machine, and it rode atop the list for 3 years straight. Those of us who have worked in high performance computing have known for years that the top 500 list was a load of crap. It's something to write a press release about so that the people that give us money to build the big computers feel like their money is well spent. I worked on a top 5 computer at one time, but our focus was always the science that we wanted to do on the computer. Running the linpack benchmark for the top 500 list was an afterthought (though it was a pleasant surprise to score as well as we did).

Re:Quelle surprise! (0)

Anonymous Coward | more than 3 years ago | (#34283266)

Aye. This never was a problem in the decades that the USA were #1.

The benchmark is fine the way it is, really. Supercomputers are used for number-crunching; number-crunching, no matter which field you're in, pretty much always means dealing with huge matrices (doing BLAS/LAPACK stuff, basically). And that, surprise, is exactly what LINPACK measures: and it's not just raw processing speed that factors into it but also things like interconnect speed and so on.

But yeah, if it hadn't been for China making a splash, nobody would've said a thing. And the following bit is also telling: "[China] used GPUs to boost Linpack ratings, in effect gaming the Top500 list". So, in effect, they cheated because they added more computing power to a machine intended for number-crunching? Geez.

Re:Quelle surprise! (4, Insightful)

Macman408 (1308925) | more than 3 years ago | (#34283674)

+1.

There's nothing that's inherently "cheating" about using GPUs in a supercomputer. If your problem maps well to the hardware they have (and many large scientific and engineering workloads do), they can provide a huge speedup at a relatively low cost and relatively high performance per watt. After all, a GPU floating point throughput can be around 20 times faster than on a CPU; they're designed to do many things all at once (high throughput, high latency), while a CPU is designed to do one thing really really fast (lower throughput, lower latency). Recently, with multicore CPUs, the extra cores add performance very similarly to how a GPU would. If having a GPU is cheating, I'd surmise that having a multi-core CPU is cheating too.

It is true that LINPACK doesn't measure everything - it doesn't put a heavy stress on the interconnect, for example. Though if your problem is compute bound, you'd probably do well to find a way to minimize interconnect use to begin with. In any case, LINPACK measures *something* - it's a place to start comparing speeds, not the absolute truth of who will always be the fastest.

Besides, what's so important about the Top500? It gives somebody bragging rights for 6 months, or, if you're very lucky, a year or two, before something bigger comes along and squishes you. Not to mention, there are many supercomputers not on the list. If the NSA builds the world's largest supercomputer, they're probably not going to brag about how much compute power they have. They prefer the mystery, so outsiders have no idea what is within the realm of possibilities for them. I was at Cray once, and was told that they sometimes sell supercomputers into such secretive areas. The government (or whoever) will send a few guys to get trained about the computer, then it gets loaded onto trucks, and Cray never hears a thing about the computer ever again. No support calls, no upgrades, no idea where it even went to or what it's used for.

Re:Quelle surprise! (1)

NotBornYesterday (1093817) | more than 3 years ago | (#34283322)

Meh. Most of the computing world has confused speed for power for decades now, whether on the desktop, in the datacenter, or in most benchmarks. Any attempt to better quantify performance can only be a good thing. However, I share your skepticism of the timing.

Re:Quelle surprise! (1)

Beardo the Bearded (321478) | more than 3 years ago | (#34283628)

It's not just this. These benchmarks are all just games, and people are powergaming the tests. Sure, the Chinese blew away this one specific test, but how powerful is that machine? IE9 cheated on SunSpider [slashdot.org] and got better results and better ad copy.

What they have to do is come up with a new benchmarks each year. Have it based roughly on the old test so you know what's going to be tested but have different questions. Otherwise it'll be just like my classes at engineering school. Some students memorized last year's tests and got artificially inflated grades. They couldn't figure out that e.g. 3.8V on the meter was, in fact, 4V when you add in the resistor tolerance.

My point is that unless you are actually testing ability rather than specifics at one rote task you end up with a generation of machines that "need propsal got approved pls can you will write me paper?"

Re:Quelle surprise! (1)

mcgrew (92797) | more than 3 years ago | (#34284406)

Most of the computing world has confused speed for power for decades now

How is anything other than speed and storage "power" to a computer?

The actual benchmark does stress interconnects (4, Informative)

Animats (122034) | more than 3 years ago | (#34283572)

Yes, noticed that.

Here's the actual benchmark used for Top500: [netlib.org] "HPL - A Portable Implementation of the High-Performance Linpack Benchmark for Distributed-Memory Computers". It solves linear equations spread across a cluster. The clustered machines have to communicate at a high rate, using MPI 1.1 message passing, to run this program. See this discussion of how the algorithm is parallelized. [netlib.org] You can't run this on a set of machines that don't talk much, like "Folding@home" or like cryptanalysis problems.

Linpack is a reasonable approximation of computational fluid dynamics and structural analysis performance. Those are problems that are broken up into cells, with communication between machines about what's happening at the cell boundaries. Those are also the problems for which governments spend money on supercomputers. (The private market for supercomputers is very small.)

So, quit whining. China built the biggest one. Why not? They have more cash right now.

Re:Quelle surprise! (5, Insightful)

inhuman_4 (1294516) | more than 3 years ago | (#34284084)

The Linpack complaining has been going on for years. I remember this coming up with the NEC earth simulator, and other ASIC based systems.

Here are some interesting numbers:
AMD Radeon HD 4870X2 ~2.4 teraFLOPS
Intel Core i7 980 XE ~107.55 gigaFLOPS

According to this the AMD is 20x faster then the Intel; and this is true, but only in some cases. If what is need is graphic processing the AMD will crush the Intel. But if you need anything else (I am ignoring GPGPU for simplification) the AMD doesn't just lose, it doesn't run. This is a problem for all ASIC based systems, GPU ones are just the newest to come out.

So this new Chinese supercomputer (and other ASIC based supercomputers) score very high in Linpack because the ASICs are designed to be good at this type of task. This makes for a non-general purpose, but very cost effective solution.

But this then means that a supercomputer that cannot use GPUs for its intended task, score very low because they are general purpose machines. Because the Top500 is based on one benchmark (Linpack) you end up with a car to pickup-truck comparison; sure the car is faster, but what about towing capacity?

The end result is the supercomputer analog of the megahertz-myth, people like bigger numbers. A high score proves that is it faster at somethings, but not that it is faster in general.

Why is being on the the Top500 important? (1)

bogaboga (793279) | more than 3 years ago | (#34283136)

I have always wondered why being on the Top500 list of supercomputers that important for those on the list. I will be better served by being told the advantage(s) or edge those who've been on that list have gotten since they got onto the list. Thanks.

Re:Why is being on the the Top500 important? (1)

marcosdumay (620877) | more than 3 years ago | (#34283286)

The advantage is that, contrary to the arguments of TFA, the test is very representative of scientific and engeneering problems. That way, if you want to be at the top at the available computing power, you'll very probably want to be at the top 500 list.

Re:Why is being on the the Top500 important? (1)

natet (158905) | more than 3 years ago | (#34284446)

The advantage is that, contrary to the arguments of TFA, the test is very representative of scientific and engeneering problems. That way, if you want to be at the top at the available computing power, you'll very probably want to be at the top 500 list.

Not necessarily true. It is representative of a CLASS of scientific and engineering problems. If the science that you want to run involves heavy use of vectors, then you want a computer that would be high on the top 500 list. Derivatives and integrals? Not as much. Problems that require a high degree of interaction between nodes? Get a computer with a faster interconnect. It all depends on the science you intend to do with that computer. The NEC Earth Simulator (mentioned in another thread) would do poorly for chemical models compared to another machine. Climate models wouldn't run as well on a cluster of Dells as it would on a Vector based box like the Earth Simulator.

Re:Why is being on the the Top500 important? (2, Insightful)

SnarfQuest (469614) | more than 3 years ago | (#34283428)

I have always wondered why being on the Top500 list of supercomputers that important for those on the list.

I always choose my supercomputers from the Bottom500 list.

I will be better served by being told the advantage(s) or edge those who've been on that list have gotten since they got onto the list. Thanks.

At the price level these things cost, you can probably list your own requirements instead of accepting the vendors.

If you are purchasing a SuperComputer, you are looking for something to do raw number crunching. You aren't worrying about how well it will run MicroSoft Word, or how many frames/second you'll get out of doom.

Re:Why is being on the the Top500 important? (0)

Anonymous Coward | more than 3 years ago | (#34283780)

I agree.

None of those computers have enough hardware to run Vista.

Re:Why is being on the the Top500 important? (1)

tverbeek (457094) | more than 3 years ago | (#34284388)

It's like trying to rank the most "intelligent" people by IQ. OK, you can score them on that test, and you can sort that list of scores, but what does it tell you when there are so many other kinds of intelligence, ad so many other ways to measure it?

LINPACK isn't so bad (1, Interesting)

Anonymous Coward | more than 3 years ago | (#34283142)

How exactly would one perform a sparse matrix product on 100,000 Android cell phones? Linpack isn't great, but it isn't terrible, either. Many supercomputing processor cycles go into large matrix operations--FFT, products, SVD/PCA/Eigenvectors. These can be run in parallel, but they don't break down into trivially parallel operations and the machines on the list really do perform well on them.

A bigger issue is that the rules allow operators to essentially optimize for the specific problem sizes and input, and that the machine doesn't have to be running stably or in the same configuration for all of the runs. It is OK if your machine is overclocked and melts one minute after it executes the benchmark. That isn't useful. The Green 500 is a better metric because it considers the power cost of the computations--that's what most supercomputing centers care about--but even it needs much more stringent rules.

Re:LINPACK isn't so bad (2, Interesting)

KenSeymour (81018) | more than 3 years ago | (#34283342)

I wonder why they don't use EISPACK?

That is for solving Eigen systems.

I remember in the early 1980's writing a program to check my linear algebra homework using Fortran and EISPACK.

This is why I love the fact that Bender likes to drink "Old Fortran" malt liquor.

I have to admit I don't know much about benchmarking but I remember using LINPACK and EISPACK on the VAX and later the Cray YMP.

I Don't See A Problem (2, Insightful)

sexconker (1179573) | more than 3 years ago | (#34283148)

I don't see a problem with using GPUs.
They do lots of parallel shit really fast.

It's no different than slapping on a math coprocessor, or adding a block of hardware to accelerate common encryption/decryption functions.

Re:I Don't See A Problem (0)

Anonymous Coward | more than 3 years ago | (#34283252)

Except for that whole error checking thing. Maybe we should just give that a pass.

Re:I Don't See A Problem (0)

Anonymous Coward | more than 3 years ago | (#34283792)

The Tesla [nvidia.com] series from NVidia, which the Chinese used, has ECC.

Missing the Point (4, Insightful)

Lev13than (581686) | more than 3 years ago | (#34283178)

As the article alludes, the big problem with ranking supercomputers via Linpack is that it doesn't advance supercomputer design. The net result is a pissing match over scalability, where winning is dependent upon who can cram the most cores into a single room. The real innovatiors should be recognized for their efforts to reduce space, power and cost, or finding new algorithms to crunch the numbers in more efficient or useful ways.

Re:Missing the Point (1)

Rob Kaper (5960) | more than 3 years ago | (#34283306)

Why shouldn't advancements in distributed efforts count as supercomputer design? Space, power and cost reduction are all there, cloud computing is not only about scalability but definitely also about efficiency.

Re:Missing the Point (1)

TheLink (130905) | more than 3 years ago | (#34283346)

Meh, did the Top500 list actually advance supercomputer design in the first place?

I thought it has always been a mostly pointless pissing match "where winning is dependent upon who can cram the most cores into a single room" just like you stated.

Or probably more accurately - "who has the most money to waste/spend on building a supercomputer".

Re:Missing the Point (1)

idontgno (624372) | more than 3 years ago | (#34283952)

Now, hold on. There is a positive impact on the technology. It's just diffuse.

Score high on Top500. Work that into your marketing. Make better sales. Profit!!

OK, no technology so far. Bear with me.

Divert a tiny portion of your Profit!!, after executive compensation and high-priced lobbying and election-buying, after dividends, after management bonuses, after unwise merger and acquisition activity... After all that, divert a small portion of the Profit!! into R&D. The technology advances... a tiny bit. But the journey of 1,000 miles starts with the first baby-step.

See? Top500 definitely advances technology, for sufficiently small values of "advances".

Re:Missing the Point (1)

phantomfive (622387) | more than 3 years ago | (#34283510)

Normally I would agree with that kind of thing, but super-computers are custom deals. The people who buy them aren't going into Costco some afternoon to pick up a new desktop and choosing the thing with the highest number, they are paying a lot of cash for these things, and they better do some deep analysis to make sure it does what they want. You don't just go up and buy a top 500 computer.

Linpack is for the rest of us, like me, who don't particularly care about super-computing, but like to look at shiny things. Whatever else is bad about a computer, doing petaflops is pretty sweet. Even if it is just a bunch of android phones hooked together. Actually that would be even sweeter. Please someone do it.

You missed the point too, btw (3, Informative)

elsurexiste (1758620) | more than 3 years ago | (#34283532)

You are allowed to use hardware-specific features and change the algorithm for this benchmark. That way, any optimization is used and innovation, as you call it, emerges. Besides, scalability *is* the most desired quality for a supercomputer that doesn't aim for space, power and cost... like the ones most likely to be in TOP500. You have Green500 for the other things you mentioned.

Re:Missing the Point (1)

Professeur Shadoko (230027) | more than 3 years ago | (#34283550)

> The real innovatiors should be recognized for their efforts to reduce space, power and cost

Trust me, to build a computer for the top of the top500, you have to be pretty good at these. Well, at least space and power. With over 100,000 cores, without an efficient (dense) packaging you'll need a room way too big. And a dense packaging leads to all kind of heat issues, so you have to be power-efficient.

Re:Missing the Point (1)

blair1q (305137) | more than 3 years ago | (#34283992)

Cost too. Otherwise each of these would have simply been specified to be 10 times the size it is. But cost and purpose come into play, so the total capability of the system gets limited.

That said, 100,000 smartphones would cost about $40 million. Which isn't unreasonable. And it would indeed outperform all of these machines on LINPACK tasks. If I were selling smartphones, I'd be putting that in my ad copy today.

Re:Missing the Point (4, Insightful)

timeOday (582209) | more than 3 years ago | (#34283732)

The real innovatiors should be recognized for their efforts to reduce space, power and cost, or finding new algorithms to crunch the numbers in more efficient or useful ways.

And NHRA should start awarding drag-racing championships on fuel efficiency rather than quarter-mile times.

Look, the Top500 is about performance, as in speed. There are other metrics for flops/watt or flops/dollar, or whatever. If those were the lists that managed to draw competitors and eyeballs, then nobody would care about Top500 and we wouldn't have to quibble about whether Linpack is a representative benchmark of what it claims to measure: speed.

Re:Missing the Point (1)

BitZtream (692029) | more than 3 years ago | (#34283802)

Sure, sounds good. You should create a set of benchmarks for those things.

Linpack was created for what people actually want however, a way to measure their super computing dicks.

Even if you come up with something new, its still going to turn in to a way to measure their penis, its human nature, survival of the fittest and all that. You're just creating artificial measurements we naturally turn into tests to determine who is better than the other. Long long ago, the winner was the guy who knew how to use a club the best, now we don't use clubs, but evolution won't stop regardless of how hard we humans try to halt our own progress in the name of civilization.

Re:Missing the Point (1)

PingPongBoy (303994) | more than 3 years ago | (#34284016)

big problem with ranking supercomputers via Linpack is that it doesn't advance supercomputer design. The net result is a pissing match over scalability, where winning is dependent upon who can cram the most cores into a single room. The real innovatiors should be recognized for their efforts to reduce space, power and cost, or finding new algorithms to crunch the numbers in more efficient or useful ways

Don't know about you but the sudden urgency to enter Linpack pissing matches has been suppressed since the operation. The need for speed continues unabated. The common way of thinking has it ranking well on Linpack is merely a side effect of having built a better mousetrap, errr supercomputer.

As much as I applaud innovation, measuring this innovation ought to involve algorithms that already run on many platforms, not new algorithms. That way, comparing Apples to PCs won't raise eyebrows. A Linpack test by itself may be insufficient so bring on the other programs.

New Benchmark (2, Funny)

Monkeedude1212 (1560403) | more than 3 years ago | (#34283188)

int i = 0;
while(i infinite)
{
i++;
}

---

Whatever computer finishes first is clearly the fastest supercomputer.

Re:New Benchmark (2, Informative)

Monkeedude1212 (1560403) | more than 3 years ago | (#34283198)

Right. (Less than symbol didn't show up because I didn't choose plain text! Derr)

Re:New Benchmark (0)

Anonymous Coward | more than 3 years ago | (#34284100)

Use < and > (except without the &).

Slashdot's parser sucks. I've wanted to use ≥ or ≤ many times (yeah, it only does ASCII, I know).

Re:New Benchmark (1)

Zero__Kelvin (151819) | more than 3 years ago | (#34284138)

"Right. (Less than symbol didn't show up because I didn't choose plain text! Derr)"

Actually, it didn't show up because you didn't preview it.

Re:New Benchmark (2, Funny)

atmtarzy (1267802) | more than 3 years ago | (#34283270)

We all know Linux is great... it does infinite loops in 5 seconds.

- Linus Torvalds about the superiority of Linux on the Amterdam Linux Symposium

Re:New Benchmark (1, Interesting)

Anonymous Coward | more than 3 years ago | (#34283290)

Whatever computer finishes first is clearly the fastest supercomputer

Or has the representation of "infinite" that is the lowest.

Also, I have seen cases where compiler optimization is smart enough to remove the entire loop if there are no side effects to incrementing i, and it's not used outside the loop.

Re:New Benchmark (4, Interesting)

KarmaMB84 (743001) | more than 3 years ago | (#34283362)

Also, I have seen cases where compiler optimization is smart enough to remove the entire loop if there are no side effects to incrementing i, and it's not used outside the loop.

Most compilers should be doing this. Hell, even IE9 is supposed to do it for JavaScript now. It gets great scores on SunSpider because of it (the JIT can throw away entire tests).

Re:New Benchmark (1)

TheLink (130905) | more than 3 years ago | (#34283622)

There are side effects. My CPU gets hotter than if it was idle. This might be an intentional side effect.

I've used this "side effect" to get some ants out of my notebook PC. Nope I didn't see any food in the notebook PC when I opened it up to "debug" it, saw very many ants though...

Re:New Benchmark (0)

Anonymous Coward | more than 3 years ago | (#34283354)

um you do realize that an int type will overflow in prety short order on any realatively modern machien right?

Re:New Benchmark (0)

Anonymous Coward | more than 3 years ago | (#34283374)

C fail. If you want an infinite loop in C, don't even bother with the comparison. Just have the while conditional always be true. Like so....

int i = 0;
while(1)
        i++;

You don't need the braces in while either for a single line.

Re:New Benchmark (0)

Anonymous Coward | more than 3 years ago | (#34283470)

C fail yourself.
If you want an infinite loop in any language, don't even bother with variables.
Like so....

int main(){
while(1);
}

Re:New Benchmark (1)

Monkeedude1212 (1560403) | more than 3 years ago | (#34283620)

It's not an infinite loop. It must end when they reach infinite. (Which is possible, because Buzz light year makes it there and further)

Re:New Benchmark (1)

elsurexiste (1758620) | more than 3 years ago | (#34283560)

int i = 0; while(i infinite) { i++; }

---

Whatever computer finishes first is clearly the fastest supercomputer.

Just wait for the RAM to collapse. ;)

Re:New Benchmark (1)

dkleinsc (563838) | more than 3 years ago | (#34283706)

I've got something a bit better here:

while(1) {
        system.exec('yes');
}

That way, instead of 1 infinite process doing nothing, you spin up an infinite number of infinite processes doing basically nothing.

Re:New Benchmark (1)

icebraining (1313345) | more than 3 years ago | (#34284420)

WTF is system.exec?

If you mean system(), it'll just launch one and block. If you mean exec*(), they'll launch one replacing the current process.

This message brought to you by ... (-1, Flamebait)

jvillain (546827) | more than 3 years ago | (#34283230)

This messages brought to you by Microsoft. If we can't compete we'll FUD. If FUD doesn't work we litigate. If litigation doesn't work we get the laws changed.

Re:This message brought to you by ... (0, Flamebait)

Yvan256 (722131) | more than 3 years ago | (#34283298)

And if changing the laws don't work, chairs will be delivered to you in person by our chairman.

Re:This message brought to you by ... (0)

Anonymous Coward | more than 3 years ago | (#34284160)

Too bad Microsoft isn't the one bitching, and too bad Windows beat NeckBeard Linux in performance on smaller loads and just failed on the larger load due to a (fixed) bug.

The REAL benchmark (1)

Bobfrankly1 (1043848) | more than 3 years ago | (#34283292)

If it can't handle a Starcraft 2, 8 player, full army of all zerglings rush without choking with max settings at a 1080p resolution, it's not a supercomputer.

Mind-bogglingly complex (5, Funny)

Yvan256 (722131) | more than 3 years ago | (#34283294)

The guide has this to say about supercomputers: "Supercomputers," it says, "are big. Really big. You just won't believe how vastly, hugely, mindbogglingly big they are. I mean, you may think your SGI Challenge DM is big, but that's just peanuts to supercomputers, listen..."

Simple fix (1)

ChrisMaple (607946) | more than 3 years ago | (#34283328)

Redefine all variables in LINPACK to be higher precision than available from any graphics processor. 128 bit floats, for example. (Requires writing a library to handle the new floats, obviously.)

Re:Simple fix (1)

Professeur Shadoko (230027) | more than 3 years ago | (#34283450)

It would hurt both the GPU and the CPU performance. Maybe not in the same way, but still.

Plus the article complaining about the use of GPUs is stupid. GPUs can be and are used to solve real scientific problems. They are not as easy to efficiently use as the CPUs, but they are almost as versatile.

Re:Simple fix (1)

TD-Linux (1295697) | more than 3 years ago | (#34283676)

Since when are GPU's not a valid CPU design? In fact, many of the top 500 resemble a GPU much more than your favorite old x86. IBM's Blue Gene is made up of a huge pile of fairly low clocked (
The part of GPU's that is general purpose is pretty much the same thing - a big pile of vector processors and a fat memory pipe.

Re:Simple fix (1)

TD-Linux (1295697) | more than 3 years ago | (#34283698)

Excuse me, slashdot ate my comment, but to finish the second sentence:
IBM's Blue Gene is made up of a huge pile of fairly low clocked (less than 2ghz) dual core powerpc chips optimized for vectorized floating point operations.

Re:Simple fix (1)

afidel (530433) | more than 3 years ago | (#34283710)

Uh, why? If the vector units on GPU's are now good for doing real scientific workloads why would you want to nerf the test? Remember the big NEC earth simulator that was on the top of the list for so long, it was nothing but a bunch of vector processors. If they can get information in and out of the GPU fast enough and have enough interconnect bandwidth with low enough latency then it's useful for doing most supercomputer type workloads.

Good to hear (5, Informative)

Sycraft-fu (314770) | more than 3 years ago | (#34283350)

The Top500 has the problem in that many of the systems on there aren't super computers, they are clusters. Now clusters are all well and good. There's lots of shit clusters do well, and if your application is one of them then by all means build and use a cluster. However they aren't supercomputers. What makes supercomputers "super" is their unified memory. A real supercomputer has high speed interconnects that allow direct memory access (non-uniform with respect to time but still) by CPUs to all the memory in the system. This is needed in situation where you have calculations that are highly interdependent, like particle physics simulations.

So while you might find a $10,000,000 cluster gives you similar performance to a $50,000,000 supercomputer on Linpack, or other benchmark that is very distributed and doesn't rely on a lot of inter-node communication, you would find it falls flat when given certain tasks.

If we want to have a cluster rating as well that's cool, but a supercomputer benchmark should be better focused on the tasks that make owning an actual supercomputer worth it. They are out there, that's why people continue to buy them.

Re:Good to hear (1)

afidel (530433) | more than 3 years ago | (#34283824)

Interconnect topology definitely does affect Linpack performance, perhaps not to the degree that it affects some other hard cases, but it is definitely a factor and shows up in the ratio of Rpeak to Rmax. Even so a 2D mesh of Infiniband or 10GbE connections is sufficient for all but the most chatty of problems.

Re:Good to hear (3, Insightful)

timeOday (582209) | more than 3 years ago | (#34283854)

OK, so you think only algorithms requiring uniform memory access are valid benchmarks. How uniform does it have to be? Real world problems do have structure, they do have locality, and an architecture that fails to exploit that is going to lose out to those that do.

Sure, your point is taken, otherwise you could say "my benchmarks is IOPS" and "my computer is every computer in the world" and win. But Linpack is not that; you can't score well without a fast interconnect, because what it measures is a set of computations that are actually useful. (Which is why the quip about a beowulf cluster of Android smartphones is stupid... because it couldn't actually be done. Go ahead and try to get on Top500 with a seti@home-type setup.)

Re:Good to hear (1, Informative)

Anonymous Coward | more than 3 years ago | (#34284396)

What you're saying sounds informed but is utter nonsense for real world applications, and really needs to be modded down. It's something I'd expect only of someone who sold single machine supercomputers, or someone utterly misinformed.

A real supercomputer has high speed interconnects that allow direct memory access (non-uniform with respect to time but still) by CPUs to all the memory in the system. This is needed in situation where you have calculations that are highly interdependent, like particle physics simulations.

What year did you last look at clusters? The high speed interconnects used now are sufficient for particle simulations; I know, I've written and on one of those clusters near the top of the list. Contrary to your comment, NUMA has pretty much become the standard for high end supercomputers whereas shared memory supercomputers are relatively rare. This is for several reasons, not least of which are: top-ranked supercomputers are rarely dedicated to a single customer, single customers of them rarely have unchanging resource needs, and that customers of supercomputing facilities don't want to rewrite their algorithms if at all possible and therefore stick with the standards they know (e.g., OpenMP or MPI). Finally, even many single-machine supercomputers are NUMA because after a certain point, DMA across the entire machine stops making sense for almost any application/use case.

100.000 ? (1)

Professeur Shadoko (230027) | more than 3 years ago | (#34283352)

Even with an hypothetical hyper-fast network, 100.000 android phones won't get you anywhere near the top of the list.

Heck, even 100 000 Nehalem (core i7) cores won't get you in the top 5.

So, android phones ? You'll need millions of them.

Re:100.000 ? (1)

Kvasio (127200) | more than 3 years ago | (#34284020)

Even with an hypothetical hyper-fast network, 100.000 android phones won't get you anywhere near the top of the list.

Heck, even 100 000 Nehalem (core i7) cores won't get you in the top 5.

So, android phones ? You'll need millions of them.

But does it run Linux? Can You Imagine a Beowulf Cluster of These?

How is the usage of GPUs 'gaming the system'? (0)

Anonymous Coward | more than 3 years ago | (#34283376)

The Tianhe system utilizes GPUs to increase its general purpose computing power. The system is not designed to only perform embarrassingly parallel computations -- it is going to be used for actual scientific research. Not only that, but the system beat out the old #1 in raw petaflops by a significant margin while using 40% less power. This efficiency gain is huge within the HPC community. Also note that the student built third place green500 system utilizes GPUs to achieve its efficiency (5 of the top 12 on the list actually use nvidia GPUs).

The Broader Point (1)

Salamander (33735) | more than 3 years ago | (#34283378)

...is that *no* single-figure-of-merit benchmark is going to be worth anything. Sandia's "Graph 500" Johnny-come-lately isn't going to be any better than Linpack that way, and will just skew the results towards a different not-generally-useful architecture. A far better idea has been around for over five years: the HPC Challenge [utk.edu] benchmark. It consists of seven different tests (Linpack is just one) which stress different aspects of system design. Anybody who knows anything about building big systems would identify some mix of these tests that best approximates their own workload, use that as a starting point for looking at likely alternatives, and then remember that it's just a starting point. The only benchmark that really matters is the one that you run yourself on your own application, but that can be a very expensive and time-consuming exercise so these lists can be a good way to figure out which systems deserve that more extended analysis. Linpack, on the other hand, isn't even useful for that. What's sad is that some people either didn't know (which says something about how we train engineers) or didn't care until a Chinese system found its way to the top (which says something even worse).

Heard at Microsoft headquarters: (3, Insightful)

pushing-robot (1037830) | more than 3 years ago | (#34283384)

Good news, everyone! Our supercomputer OS only lost because it's buggy!

Re:Heard at Microsoft headquarters: (2, Interesting)

MightyMartian (840721) | more than 3 years ago | (#34283468)

Good news, everyone! Our supercomputer OS only lost because it's buggy!

Leela: How is that good news, Professor?

Professor Farnsworth: I still charge enough per seat to be feared.

Re:Heard at Microsoft headquarters: (0)

Anonymous Coward | more than 3 years ago | (#34283762)

So... ...no different than any other software in the company. OK, thanks for that bit of information.

Well, there's a non-notable point! (4, Insightful)

icannotthinkofaname (1480543) | more than 3 years ago | (#34283392)

In other supercomputer news, it turns out the Windows-based cluster that lost out to Linux stumbled because of a bug in Microsoft's software package.

As it should. That's not news; that's how the game is played. If your software is buggy, and those bugs drag your performance far enough down, you don't deserve a top500 spot.

If they fix their software, rerun the test, and perform better than Linux, then they will have won that battle (the battle for the top500 spot, not the battle for market share) fair and square.

Re:Well, there's a non-notable point! (1)

anss123 (985305) | more than 3 years ago | (#34283690)

In this case it was the benchmark software that was buggy, not the OS. The interesting bit, as noted by the article, was: Why did Windows perform better on small workloads compared to Linux?

Re:Well, there's a non-notable point! (2, Informative)

gerrywastaken (1675920) | more than 3 years ago | (#34284506)

In this case it was the benchmark software that was buggy, not the OS.

Yeah that's right, the LINPACK benchmark software that Microsoft strangely got to rewrite themselves. Yep that's apparently where the bug was.

I wonder why MS was given the task of rewriting the benchmark in the first place. I guess it will always be a mystery... oh hold on, no wait... "It should be noted that Microsoft has helped fund the Tokyo Institute of Technology's supercomputing programs." Guess it helps to read the sentences they stick near the end of unrelated paragraphs at the end of the articles.

Re:Well, there's a non-notable point! (1)

BitZtream (692029) | more than 3 years ago | (#34283840)

True, except when the bug is in the software thats doing the rating, not the software thats being tested.

Read the article, Windows wasn't buggy (in relation to this situation), the benchmark software was.

HPCC (1)

Meeni (1815694) | more than 3 years ago | (#34283400)

The HPC challenge has been available for a long time now. It has never got real attention because people do want a single metric to rank computers and make a classification. If you really want to know if your computer is fit for a particular purpose, you can. Don't blame the top500 for providing what the people want to see. As a side note, I quite don't see why using accelerators would be "gaming" the top500. This is a very stupid statement, accelerators have a wide range of applications for real science, it is not about getting the biggest number in HPL. This message written from the premises of the SC meeting.

Of course Benchmarks are dumb! (0)

Anonymous Coward | more than 3 years ago | (#34283494)

It's like taking qualifying for a race and assuming that's going to be how your winners line up.

It may, by some sheer chance work out that way, but more often than not, even if nobody crashes or has another mechanical problem, there is a lot that can go on to change matters. Fuel consumption, tire wear, changes to the track, they all have an impact.

Even the winner of the race doesn't necessarily have to have been the best performer throughout the race. Stranger things have happened.

What else besides Linpack?! (0)

Anonymous Coward | more than 3 years ago | (#34283548)

One could argue that there should be more "breadth" in the test, but Linpack scores are like IQ scores: Both are just one particular interpretation on what it means to be weak/strong in a particular field. In other words, we *know* that there is really no one single measure of how either should be judged. Specialized processors which may not fare too well on Linpack may do a great job at other tasks, etc.

So why Linpack? I would argue that large scale linear algebra is still the bread and butter of supercomputing. Yes, you can use "supercomputers" for things other than, say, crunching linear(ized) differential equtions to model nuclear explosions, but that wouldn't be the primary purpose of most of these systems.

Its a Political Issue... (0)

Anonymous Coward | more than 3 years ago | (#34283558)

There have been a number of attempts to come up with a replacement for LINPACK, some of which have gained traction in the industry. These deal with the fact that computer performance is multi-dimensional. Some deal with different technical metrics (memory bandwidth, disk bandwidth, memory latency), and others by measuring performance with applications or application kernels.

The problem, though, is political, not technical. Its a lot easier for a decision maker - often without a technical background - to deal with a single number and ranking, than it is to understand and deal with the complexities and tradeoffs that come with the more accurate multidimensional solutions. And since many supercomputer purchases fall into the political realm (in terms of congressional earmarks to pay for them, or national programs to improve "competitiveness"), you need something that your congressman or premier can handle easy. Its sad to see how many supercomputer procurements include LINPACK, and that some even ask for an estimate of where the new system will end up in the TOP500 list.

It'd be nice if this weren't the case, but I don't see anything that is likely to change it....

New benchmark (1)

asvravi (1236558) | more than 3 years ago | (#34283580)

Superpack = number of years taken to compile a full Gentoo branch * number of hours taken to render slashdot home page

How about utilization metrics? (1)

sottitron (923868) | more than 3 years ago | (#34283718)

Speed is one thing, but how about normalizing the list by how well its owners are utilizing those transistors?

Bigger and Bigger Computers (0)

Anonymous Coward | more than 3 years ago | (#34283742)

....but are we really doing anything constructive with them at all?

It kinda seems like its just the US and China partaking in some international dick size contest, while the real winners are the companies given beefy government contracts to build the computers in both countries: IBM et. al.

Chinese gaming the system, News at 11. (0)

Anonymous Coward | more than 3 years ago | (#34283848)

You have to understand these cultures.

Most other places, exams are a measure of how well you know the stuff. atleast they try to be. You are a certified something, which requires XXX years of practical experience and YY years of study. So once you have the experience and years of study, you take the exam, clear it, and therefore would be considered whatevet it signifies.

In India and China, exams are seen as **The Goal**. Never mind the experience. Never mind the study. Just pass the exam before you have EVEN one day fo practical experience. Mug up the question bank. cheat in the exam. bribe the examiner. Whatever. Just pass the exam, get the credential, and then, expect to be the same as the ppl with the required experience etc.

This is how they try to measure up to the west.

So, got ot make the numbers with this super computing measure. Say there's a way to tweak linkpack to do it. Just do it and make the numbers. Get the numbers, publish it. Does not matter if your computer cannot do anything else. it passed the exam!

(For India - Does not matter if your software developer has not coded a line till now - He has passed the Java certification exams, and therefore must be good!.)

And so it goes. Nothing new here.

Why Green is so important (1)

stox (131684) | more than 3 years ago | (#34283888)

The single largest expense, over the lifetime of a supercomputer, is power consumption.

KVM? (1)

knotprawn (1935752) | more than 3 years ago | (#34284128)

To quote the original article "Because Tsubame uses a KVM hypervisor and various cloud-like provisioning tools, it can run both Windows and Linux at the same time on different nodes, and offer users various types of processing configurations." As one of the commentors on the original article pointed out, KVM is virtual machine software designed for linux, so is this benchmark comparing the performance of linux and windows virtual machines (running on a linux host), rather than comparing the performance of linux and windows directly? Or is this comment relevant only in the sense that Tsubame is currently running KVM and totally irrelevant in the context of the testing performed?

Bummer for you, Sandia (1)

OSPolicy (1154923) | more than 3 years ago | (#34284184)

>Fortunately, Sandia National Laboratories is heading an effort to develop a new set of benchmarks...

Bummer for you, Sandia. NASA already did that with the NAS Parallel Benchmarks. Here's a hint: you're funded by the US Government (just like NASA), and NPB died when the Japanese started kick US butt on NPB.

My supercomputer can beat up your supercomputer! (0)

Anonymous Coward | more than 3 years ago | (#34284230)

Seriously? This is what is keeping folks awake at night?

This is just as pointless as People Magazine's "500 Sexiest People List" -- but with a lot less cleavage. //TB

Benmarks. (1)

Icegryphon (715550) | more than 3 years ago | (#34284322)

>But the Benchmark is in FLOPs which uses FLUs.(Like ALUs)
So if they have problems then the only ones to blame are the Designers of the computer.
A FLOP, is a FLOP, is a FLOP.

The possible problems are:
A. The aren't using the Math Libraries correctly.
B. They aren't using the CPU's or GPU's FLU correctly.
C. They configured the system bad. (I.e. Networking, etc)

100,000 smartphones (1, Interesting)

Anonymous Coward | more than 3 years ago | (#34284330)

If this phone cluster actually did outperform the fastest supercomputers, why would that make the benchmark stupid?

I mean, the idea of using 100,000 smartphones might be stupid when examined pragmatically, but I don't see how that affects the validity of the performance.

And nobody, so far, is claiming that using GPUs is an inherently stupid idea for any reason, so that should have no bearing on the Tianhe's victory.

When a foreign computer wins, the benchmark needs to be changed? Now that is gaming the system, American style.

I don't believe it (0)

Anonymous Coward | more than 3 years ago | (#34284364)

" one high-performance-computing vendor executive joked about stringing together 100,000 Android smartphones to get the largest Linpack number"

Would take a few hundred thousand.

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Sign up for Slashdot Newsletters
Create a Slashdot Account

Loading...