Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Open Source Solution Breaks World Sorting Records

Soulskill posted more than 5 years ago | from the out-of-sorts dept.

Supercomputing 139

allenw writes "In a recent blog post, Yahoo's grid computing team announced that Apache Hadoop was used to break the current world sorting records in the annual GraySort contest. It topped the 'Gray' and 'Minute' sorts in the general purpose (Daytona) category. They sorted 1TB in 62 seconds, and 1PB in 16.25 hours. Apache Hadoop is the only open source software to ever win the competition. It also won the Terasort competition last year."

cancel ×

139 comments

Sorry! There are no comments related to the filter you selected.

Overlords (3, Funny)

Narpak (961733) | more than 5 years ago | (#27978823)

I for one welcome our new datasorting overlords!

Re:Overlords (4, Funny)

Jurily (900488) | more than 5 years ago | (#27978859)

I for one welcome our new datasorting overlords!

With a name like Apache Hadoop, I wouldn't be surprised if they came from Star Wars.

Re:Overlords (3, Informative)

rackserverdeals (1503561) | more than 5 years ago | (#27979351)

I wouldn't be surprised if they came from Star Wars.

Actually, it came from Google. Sorta.

Apache Hadoop is an implementation of MapReduce that Google uses in their search engine. I believe the details were found in a paper Google released on it's implementation of MapReduce.

Re:Overlords (3, Informative)

daemonburrito (1026186) | more than 5 years ago | (#27979653)

"MapReduce: Simplified Data Processing on Large Clusters [google.com] ." Jeffrey Dean and Sanjay Ghemawat, OSDI '04.

They wrote about it in Beautiful Code, too (great book). MapReduce isn't complex, in fact the name comes from a feature that a lot of functional languages provide (yeah, I know, it's not exactly the same thing).

There are many implementations of it. The wikipedia article is pretty informative: http://en.wikipedia.org/wiki/MapReduce [wikipedia.org] . I didn't know about "BashReduce"... Heh.

Re:Overlords (3, Funny)

davester666 (731373) | more than 5 years ago | (#27980425)

Fastest implementation of BubbleSort EVER!

Re:Overlords (1)

Briareos (21163) | more than 5 years ago | (#27980211)

With a name like Apache Hadoop, I wouldn't be surprised if they came from Star Wars.

Apache Hadouken probably would have packed even more of a punch... :D

np: DJ Walkman - Milk Und Herring (Milk Und Herring)

Re:Overlords - Trivia (5, Informative)

e9th (652576) | more than 5 years ago | (#27980393)

Hadoop's name (and mascot) came from Doug [the project leader] Cutting's son's yellow stuffed elephant toy.

Re:Overlords (5, Funny)

ModMeFlamebait (781879) | more than 5 years ago | (#27980665)

datasorting for I new one our overlords! welcome

I'm sure that I can rock their scores (1, Funny)

Anonymous Coward | more than 5 years ago | (#27978827)

Just give me a few minutes to patch together a bubblesort from my highschool Pascal class. I'll show them record speed!

Re:I'm sure that I can rock their scores (5, Funny)

Thinboy00 (1190815) | more than 5 years ago | (#27978951)

My sort [wikipedia.org] will totally beat yours!

Re:I'm sure that I can rock their scores (2, Funny)

Midnight Thunder (17205) | more than 5 years ago | (#27978971)

Bogosort: for when you have you are paid by the hour, but aren't penalised for being late.

Re:I'm sure that I can rock their scores (3, Funny)

Anonymous Coward | more than 5 years ago | (#27980095)

Bogosort: for when you have you are paid by the hour, but aren't penalised for being late.

with my luck, bogosort would get it right the first time.

Re:I'm sure that I can rock their scores (3, Funny)

bmajik (96670) | more than 5 years ago | (#27979173)

I've asked lots of interview candidates to implement randomSort. They've never heard of it, so then I describe the algorithm.

Watching their eyes go wide is the highlight of the interview, typically.

Occasionally some person who has overcome their interview nervousness will, with eager honesty, try to implore to me that this is not a very good sort algorithm, and that much better ones are taught in universities these days.

Good Times.

Re:I'm sure that I can rock their scores (1)

Midnight Thunder (17205) | more than 5 years ago | (#27979573)

I've asked lots of interview candidates to implement randomSort. They've never heard of it, so then I describe the algorithm.

Did you change roles from developer to HR?

Re:I'm sure that I can rock their scores (3, Funny)

Anpheus (908711) | more than 5 years ago | (#27980135)

No, he clearly changed roles from developer to Evil HR. He's probably directly subservient to Catbert.

Re:I'm sure that I can rock their scores (1)

mwvdlee (775178) | more than 5 years ago | (#27980703)

In what way exactly does this help weed out the bad from the good candidates.

Any candidate that actually tries and succesfully implements the algorithm is someone you DON'T want on your team.
Any candidate that runs of screaming is one you DO want, but they're already gone.

Re:I'm sure that I can rock their scores (1)

creimer (824291) | more than 5 years ago | (#27979383)

Just give me enough money to remember my Logo sort algorithm from grade school. The turtle will always be the fastest! :P

Is it settled? (4, Funny)

Jah-Wren Ryel (80510) | more than 5 years ago | (#27978987)

So, it appears they have finally sorted out whether open source beats proprietary.

Re:Is it settled? (1)

x78 (1099371) | more than 5 years ago | (#27979217)

I wonder if they _sorted_ it themselves..

Re:Is it settled? (1)

marcosdumay (620877) | more than 5 years ago | (#27979641)

They did, but they took a lot more time than the FOSS people.

Re:Is it settled? (1)

nacturation (646836) | more than 5 years ago | (#27980457)

So... open sorts wins?

When's it going to be 1.0? (3, Insightful)

AlexBirch (1137019) | more than 5 years ago | (#27979011)

If it's winning competitions at 0.20, when will they release it?

Re:When's it going to be 1.0? (1)

SunTzuWarmaster (930093) | more than 5 years ago | (#27979277)

Software is done when it's good and ready!

Re:When's it going to be 1.0? (1)

AlexBirch (1137019) | more than 5 years ago | (#27979311)

It'd be nice to put it into a production app...

Re:When's it going to be 1.0? (5, Informative)

Anonymous Coward | more than 5 years ago | (#27979283)

It's 0.20 but it's stable and production ready already. I use it with HBase [apache.org] and it scales awesomely.

Re:When's it going to be 1.0? (1)

AlexBirch (1137019) | more than 5 years ago | (#27979859)

Isn't 1.0 production for most software jargon?

Re:When's it going to be 1.0? (0)

Anonymous Coward | more than 5 years ago | (#27980155)

Just like beta is only for pre-production software used by a very limited set of users...oh wait.

Re:When's it going to be 1.0? (2, Insightful)

BikeHelmet (1437881) | more than 5 years ago | (#27980637)

Isn't 1.0 production for most software jargon?

Nah, that's 6.0

MS DOS 6.0
IE 6.0
Visual Studio 6.0

I doubt anybody would want to use an earlier version than that!

Re:When's it going to be 1.0? (1)

lostguru (987112) | more than 5 years ago | (#27980801)

I doubt anyone would want to use those versions either

I'm only wish more programs were open source (1, Interesting)

blahplusplus (757119) | more than 5 years ago | (#27979035)

... truth be told, a lot of good engineering could happen if many of peoples favorite commercial applications could have the souce distributed with them, a lot of old games for instance coudl be updated and maintained.

I think what holds the progress of open source back is interesting projects that exist that people want to work on but are locked away under corporate lock and key.

Re:I'm only wish more programs were open source (1)

Bert64 (520050) | more than 5 years ago | (#27979263)

People maintaining old games is now what the companies that produced those games want... They would rather sell you new games, or sell the old ones to you again..
Corporations will always put their own interests first, and those interests will often be detrimental to everyone else.

Corporations employee people (0)

Anonymous Coward | more than 5 years ago | (#27980889)

and it's not detrimental to the people that work at those companies to protect the corporations intellectual property.

Great! It's open source! (1)

erroneus (253617) | more than 5 years ago | (#27979039)

But has anyone patented it yet? Patents trump copyright after all.

Re:Great! It's open source! (1)

buchner.johannes (1139593) | more than 5 years ago | (#27979157)

You can't patent Apache 2.0 licensed stuff.
Also, you can't patent software*.

*in Europe

Re:Great! It's open source! (4, Interesting)

berend botje (1401731) | more than 5 years ago | (#27979221)

Also, you can't patent software in Europe

Not yet, but they are working on it. They tried to snuck it through by hiding it in the amendments of an agricultural bill. Luckily Poland kept watch and rose a stink about it.

It's not over. There is too much money to be gained for that.

Re:Great! It's open source! (2, Interesting)

haruchai (17472) | more than 5 years ago | (#27979475)

Why isn't this illegal - adding unrelated legislation to a ? Is there anywhere in the world why this practice is not permitted, or better yet, prosecuted?

Re:Great! It's open source! (1)

newcastlejon (1483695) | more than 5 years ago | (#27979513)

Why isn't this illegal - adding unrelated legislation to a ? Is there anywhere in the world why this practice is not permitted, or better yet, prosecuted?

I never heard of it happening here in the UK, as far as I knew only the US did. Shows how little I knew.

Re:Great! It's open source! (3, Informative)

jimicus (737525) | more than 5 years ago | (#27979657)

Here in the UK, the patent office has been issuing software patents for some time in "anticipation" of them becoming legal at some point in the future.

No, I don't understand that either.

Re:Great! It's open source! (1)

newcastlejon (1483695) | more than 5 years ago | (#27979813)

GP was referring to tacking on legislation to an unrelated bill, i.e. patent legislation on an agricultural bill. It's my understanding that this is sometimes used in the US to block a bill by means of appending something that no fool will vote in.

Re:Great! It's open source! (1)

psycho12345 (1134609) | more than 5 years ago | (#27980109)

This is indeed the case (for killing bills). The nastier version is tacking some random crap on to the annual budget, and using the excuse of getting the budget passed to ram it through even though the bill alone wouldn't even get to a vote by itself.

Re:Great! It's open source! (1)

franki.macha (1444319) | more than 5 years ago | (#27979711)

That depends how you define unrelated, but I think that the Anti-terrorism, Crime and Security Act of 2001 [guardian.co.uk] is a perfect example of the fact that the name of a law is chosen to try and make sure that it gets passed.

Re:Great! It's open source! (4, Interesting)

Halo1 (136547) | more than 5 years ago | (#27979757)

Why isn't this illegal - adding unrelated legislation to a ? Is there anywhere in the world why this practice is not permitted, or better yet, prosecuted?

The GP is confusing a bunch of things. First, the Council of Ministers threw out all limiting amendments from the European Parliament and reached an Political Agreement on a shoddy text through backdoor maneuvering by Germany and the European Commission [google.com] . That text would have turned the European Patent Office's practice of granting software patents into EU legislation.

A Political Agreement has no juridical nor legislative value, but it has never happened that a political agreement was later on annulled and that negotiations were reopened. So also in this case, even though the German, Dutch, Spanish and Danish parliaments afterwards passed motions asking to reopen the discussions, the Council's bureaucrats did not want to do that because it "would undermine the efficiency of the decision making process".

Anyway, once you have a Political Agreement (which is reached by the representatives of the ministries responsible for the matter at hand) and nobody "wants" to discuss it anymore, the agreement can be placed as an "A item" on any EU Council of Ministers meeting, since it only needs rubber stamping in that case. In the case of the Software Patents Directive, it appeared several times as an A item on the agenda of an Agriculture and Fisheries meeting (which is presumably where the GP's confusion stems from).

In principle, there would have been nothing wrong with that, but in this case there was no actual political agreement, and in particular Poland was very unhappy with the way it had been treated. So 4 times in a row, Poland either had this "A item" removed from the agenda (sometimes at the last minute, because the responsible Polish minister had to be informed that they were again trying to get it through at a meeting he had no business with), or turned it into a "B item", which means that it can't be rubber stamped but that they first have to talk a bit about it (which nobody wanted to do).

In the end it still did get approved, but that whole circus helped with in convincing the EU Parliament to table a resolution asking the Commission to restart the directive's process [ffii.org] , and when the Commission refused to later on squarely reject it [ffii.org] .

You can find some more of my thoughts on the Council's behaviour here [ffii.org] .

Re:Great! It's open source! (1)

rackserverdeals (1503561) | more than 5 years ago | (#27979467)

But has anyone patented it yet? Patents trump copyright after all.

There are a number of patent applications related to MapReduce from Google and Yahoo.

What data? (1, Insightful)

Tinctorius (1529849) | more than 5 years ago | (#27979047)

They sorted 1TB in 62 seconds, and 1PB in 16.25 hours.

This doesn't say anything if we don't know what kind of records were supposed to be sorted.

100 bytes, 10 byte keys. (5, Informative)

eddy (18759) | more than 5 years ago | (#27979115)

Probably why the second sentence in the article is "All of the sort benchmarks measure the time to sort different numbers of 100 byte records. The first 10 bytes of each record is the key and the rest is the value."

Re:What data? (2, Insightful)

Antisyzygy (1495469) | more than 5 years ago | (#27979119)

Things can be sorted by any of their properties. What is important is this software sorted data objects this quickly regardless of what property they were being ordered by. It beats all of the other sorting algorithms.

Re:What data? (4, Funny)

rackserverdeals (1503561) | more than 5 years ago | (#27979299)

They sorted 1TB in 62 seconds, and 1PB in 16.25 hours.

This doesn't say anything if we don't know what kind of records were supposed to be sorted.

It's amazing what you can learn if you actually RTFA.

All of the sort benchmarks measure the time to sort different numbers of 100 byte records.

If that's not good enough for you, post your email address and maybe someone will be kind enough to send you the 100TB and 1PB data files they used.

READ THE MOTHERFUCKING ARTICLE YOU STUPID MORON (0, Troll)

Anonymous Coward | more than 5 years ago | (#27980537)

READ THE GODDAMNED FUCKING ARTICLE YOU STUPID MOTHERFUCKING LAZY COCKSUCKING PIECE OF SHIT.

Really, dude... it's not that hard to put in a modicum of effort that will pay dividends in terms of you not looking like a totally clueless fucking moron.

Cool (1)

bigdaddy25fb (1166129) | more than 5 years ago | (#27979083)

Gonna pass this on to my boss, hopefully now we can move off of our terrible, terrible proprietary sorting software...Good to see open source breaking inroads in so many areas!!!

Re:Cool (1)

PiSkyHi (1049584) | more than 5 years ago | (#27980343)

Maybe your boss could write to google and let them know just how special sorting is, I'm sure they'd love to hear it.

They won the "Who has the most moneys" award. (5, Insightful)

nathan.fulton (1160807) | more than 5 years ago | (#27979099)

...this cluster had nearly 4 times the number of nodes as the previous records. This competition was testing who had more nodes working together the best, but when you have so many more nodes, it would be hard not to top other clusters.

Re:They won the "Who has the most moneys" award. (3, Interesting)

Rockoon (1252108) | more than 5 years ago | (#27979363)

I was doing some back-of-the-envelope, and they are sorting 17.7GB/second, which at a minimum would require 177 HD's if each drive can write 100MB/sec.

If its not written to disk, then there is no achievement here (you don't perform 1 minute+ sorts and then throw the result away in real-world scenarios)

Re:They won the "Who has the most moneys" award. (0)

Anonymous Coward | more than 5 years ago | (#27979575)

There were over 15,000 disks in the cluster.

Re:They won the "Who has the most moneys" award. (0)

Anonymous Coward | more than 5 years ago | (#27979893)

If its not written to disk, then there is no achievement here (you don't perform 1 minute+ sorts and then throw the result away in real-world scenarios)

That's not true at all. I deal with a number of processes that have intermediate sort steps that do not require that the data touch disk.

Re:They won the "Who has the most moneys" award. (0)

gardyloo (512791) | more than 5 years ago | (#27979951)

So you have a TB of RAM?

Re:They won the "Who has the most moneys" award. (1, Interesting)

Anonymous Coward | more than 5 years ago | (#27980161)

You don't _always_ need that much main memory -- there's a concept of something called a data-flow architecture [wikipedia.org] .

The old Tandem (I think HP calls it Neoview now) does this w/ their SQL engine. Of course, you would likely still need the last step to use temporary/overflow files on disk but the intermediate steps could potentially be done w/ data touching disk -- depends on the generated query plan or how you are "reducing" the problem.

Re:They won the "Who has the most moneys" award. (0)

Anonymous Coward | more than 5 years ago | (#27980105)

I would assume they wouldn't want to introduce storage bottlenecks as a variable. You can buy the storage needed to support what your solution can do.

If the different teams were using different storage it would kind of destroy the results of comparing Apache Hadoop to something else. Infiniband could easily carry that amount of throughput to a single node. As far as where or how you want to store it permanently that really isn't relevant to the results.

Re:They won the "Who has the most moneys" award. (0)

Anonymous Coward | more than 5 years ago | (#27980159)

sed -e "s/a single node/the nodes/g"

Re:They won the "Who has the most moneys" award. (0)

Anonymous Coward | more than 5 years ago | (#27980657)

Infiniband could easily carry that amount of throughput to a single node.

*cough* bullshit *cough*

Re:They won the "Who has the most moneys" award. (0)

Anonymous Coward | more than 5 years ago | (#27979525)

It still shows that it scales!

Re:They won the "Who has the most moneys" award. (1)

marcosdumay (620877) | more than 5 years ago | (#27979689)

Yep, when you have a problem that big, you want scalability before anything else. How well could the other candidades use a machine that big?

Re:They won the "Who has the most moneys" award. (1)

drizek (1481461) | more than 5 years ago | (#27980023)

SO Open Source isn't all that great at sorting, but since they had the biggest toys, it obviously shows that it works as a business model.

Java (5, Insightful)

cratermoon (765155) | more than 5 years ago | (#27979101)

OK, so where are the "Java is slow" comments? o.O

Re:Java (-1, Troll)

Anonymous Coward | more than 5 years ago | (#27979161)

Java is slow. And it eats b^W memory too.

Re:Java (-1, Redundant)

buchner.johannes (1139593) | more than 5 years ago | (#27979163)

Java is slow

Re:Java (1)

French31 (1311051) | more than 5 years ago | (#27979199)

Here. [thedailywtf.com]

Re:Java (-1, Flamebait)

Anonymous Coward | more than 5 years ago | (#27979333)

.NET is even more insecure.

Re:Java (1)

dodobh (65811) | more than 5 years ago | (#27979335)

Waiting for similar hardware to become available for other languages.

I think Tim Bray is on the right track with his widefinder idea.

See Widefinder 1 [tbray.org] and Widefinder 2 [sun.com] for details.

Re:Java (4, Insightful)

hey! (33014) | more than 5 years ago | (#27979411)

Well, not to endorse the "Java is slow" meme or anything, but starting from a red light I can beat most cars across the intersection on my bike.

Likewise if I had to drive across country in the shortest time possible, I'd choose a Ford F250 if the challenge stipulated I had to bring 3000 pounds of bricks with me.

Speed is a very task specific notion.

Re:Java (1)

dodobh (65811) | more than 5 years ago | (#27980511)

Just use a Boeing 747. You drive so fast that your wheels don't touch the ground.

(That's like throwing RAM at an IO limited problem).

Re:Java (1)

asserted (818761) | more than 5 years ago | (#27979425)

well, actually... i sort of do. look, google has already reported on sorting 1 PB in 6 hours [blogspot.com] on a 4000 node cluster. their implementation is in C++. yahoo's result is 16.25 hours on a 3800 node cluster and hadoop is written in java. even taking into account the 200 node difference, yahoo's implementation is ~2.6 times slower than google's. it may not all be java's fault, but still.

Re:Java (1)

rackserverdeals (1503561) | more than 5 years ago | (#27979745)

Something doesn't seem right.

Yahoo's cluster had 3,800 nodes with 4 disks per node giving it roughly 15,200 drives plus or minus the dead nodes/drives in the cluster.

The Google cluster had 4,000 nodes with 48,000 hard drives. 12 drives per node doesn't sound like the typical Google servers [youtube.com] I've seen. That one looks like 2-4 drives. This other video [youtube.com] seems to show the storage node which looks like it has 5-10 drives.

The reason I bring up drives is that sorting 1PB likely involves hd access the more drives, the higher I/O throughput.

Whatever the case, the nodes seem to be vastly different and making a comparison based on the number of nodes doesn't seem appropriate.

Re:Java (1)

AlexBirch (1137019) | more than 5 years ago | (#27979841)

Let's compare orange and apples.
Since a node with 2 GB of memory is equivalent of 4 GB of memory, because a node is a node. What about myrinet is the same as ethernet. What about gigabits?
Or if we really wanted to compare the numbers we would ensure they sort the same data on the hardware.

Use C++ and save 10x the hardware (0)

Anonymous Coward | more than 5 years ago | (#27979477)

I'm always confused when teams use Java that don't REQUIRE complex cross platform support. IME, all the old complaints about java are still true.
It is slow, when compared to C/C++ or other mature compiled languages.
It uses more RAM than C++.
Development isn't any easier or faster than C++.

It is fairly easy to write cross platform C++ code that uses less compute resources and easily runs on 10 platforms. Besides having developers who are nearly clueless about the platform isn't good. I've seen some really bad java developer teams and some really bad C++ teams. Overall, the java developers knew less about the platform and hardware than the C++ teams. Java let them be lazy.

Don't get me wrong, there are many uses for java and as CPUs have gotten faster and hold more RAM, we aren't trying to suck every bit of performance. That's a good fit for java programs.

Re:Use C++ and save 10x the hardware (1, Insightful)

Anonymous Coward | more than 5 years ago | (#27980141)

Development isn't any easier or faster than C++.

Ridiculous. Java's library provides 100 times what C++'s library provides, which makes it a solid ground for application development (which is what it excels at).

Re:Use C++ and save 10x the hardware (0, Flamebait)

ClosedSource (238333) | more than 5 years ago | (#27980279)

But how much of those libraries exist to achieve Java's religious beliefs on abstraction?

Re:Use C++ and save 10x the hardware (2, Informative)

Yosho (135835) | more than 5 years ago | (#27980721)

But how much of those libraries exist to achieve Java's religious beliefs on abstraction?

Wow, how did this get modded insightful? For one, calling the design of a programming language a "religious belief", then asking a vague question about it without providing even a basis of an answer is just inflammatory.

But the answer that anybody who knows what they're talking about will tell you is, none of them. Java's abstraction mechanisms are built into the language. None of the standard libraries are necessary to support it. They take advantage of it, of course, and you'd be crazy to not take advantage of one of the language's features. Try taking a look at a tree representation [sun.com] of all of the classes in the standard library. The vast majority of classes are not more than one or two levels down from the top-level Object. The things that are deeper are typically things that are complex in any language -- CORBA, GUI toolkits, etc. It certainly looks much cleaner than many graphs I've seen of C++ libraries that abused multiple inheritence.

Re:Use C++ and save 10x the hardware (1, Funny)

Anonymous Coward | more than 5 years ago | (#27980145)

Use C++ and save 10x the hardware

You tell em brutha! I'm so tired of carrying 10 cell phones to play java games.

It was only a matter of time (0)

Anonymous Coward | more than 5 years ago | (#27979113)

Dang nabbit. I was depending on the World Sorting Records to be my reference for how people sort in other countries than my own. Avoid open source, next time it will break your nose.

C++ port of Java Hadoop? (1)

frik85 (951295) | more than 5 years ago | (#27979235)

Java doesn't fit for my environment, does someone know of an open source C++ port of Java Hadoop?

If there is no such port, is anyone interested in starting a port? An example of a Java port is "cLucene", a C++ port of Java Lucene search engine. It usually outperforms its Java sibling in an order of magnitude.

Re:C++ port of Java Hadoop? (1)

vbraga (228124) | more than 5 years ago | (#27979555)

It would be interesting to do it.

I'm willing to volunteer if you're starting a port. Just mail me (my email is on my user page).

Re:C++ port of Java Hadoop? (2, Informative)

Yosho (135835) | more than 5 years ago | (#27980659)

It usually outperforms its Java sibling in an order of magnitude.

Do you have any actual benchmarks for that? According to the benchmarks page at the official cLucene wiki [sourceforge.net] , cLucene is roughly twice as fast as the Java Lucene at indexing, and it's only about 10% faster at the actual searching. That's not even close to an order of magnitude.

The benefits of parallelizing everything! (2, Informative)

Celeste R (1002377) | more than 5 years ago | (#27979297)

According to a post on the Yahoo developer forums:

2005 winner used only 80 cores and achieved it in 435 seconds. So with 800 cores what 2007 winner achieved is 297 seconds ?

Its not only number of cores its how the logic to use parallel nodes properly to do a particular task is important.

Hadoop won with 1820 cores (910 nodes w/ 2 cores each) at 209 seconds.

I'm all for better sorting algorithms, but eventually the cost of parallelizing something overtakes the profit made. That being said, Hadoop's internal filesystem made to be redundant, which is an important feature whenever you're dealing with large amounts of data.

Hadoop uses Google's MapReduce, by the way, whereas the competition didn't. It's nice to see MapReduce being used in a more public eye.

While better sorting algorithms -do- matter, I have to say that maintenance and running costs also matter.

I'd also like to see how a compatible C version of this software compares with the Java version. However, as I see it, the Java overhead seems fairly limited; sorting code is wonderfully repetitive, and I'd expect that it's already been optimized a fair amount.

By the way, the number of nodes and the hardware in the nodes for this Hadoop cluster is -optimized- for this contest.

Re:The benefits of parallelizing everything! (1)

rackserverdeals (1503561) | more than 5 years ago | (#27979997)

By the way, the number of nodes and the hardware in the nodes for this Hadoop cluster is -optimized- for this contest.

The number of nodes was reduced to run the 100TB benchmark but I don't see anything that backs up your comment that the hardware was optimized for this contest. The cluster hardware doesn't look like anything special. Maybe it's optimized for Hadoop which is different than being optimized for the contest.

Not quite as impressive as it sounds (4, Informative)

Sangui5 (12317) | more than 5 years ago | (#27979441)

Google's sorting results from last yeat (link [blogspot.com] ) are much faster; they did a petabyte in 362 minutes, or 2.8 TB/sec. They minute sort didn't exist last year, but Google did 1TB in 68 seconds last year, so I think it may be safe to assume that they could do 1 TB in under a minute this year. Google just hasn't submitted any of their runs to the competition.

From the sort benchmark page [sortbenchmark.org] , the list the winning run as Yahoo's 100TB run, leaving out the 1PB run; that implies the 1PB run didn't conform to the rules, or was late, or something.

People have commented that this is a "who has the biggest cluster" competition; the sort benchmark also includes the 'penny' sort, which is how much can you sort for 1 penny of computer time (assuming your machine lasts 3 years), and 'Joule' sort, how much energy does it take you to sort a set amount of data. Not surprisingly, the big clusters appear to be neither cost efficient nor energy efficient.

Re:Not quite as impressive as it sounds (4, Interesting)

owenomalley (103963) | more than 5 years ago | (#27979959)

In sorting a terabyte, Hadoop beat Google's time (62 versus 68 seconds). For the petabyte sort, Google was faster (6 hours versus 16 hours). The hardware is of course different. (from Yahoo's blog [yahoo.net] and Google's blog [blogspot.com] )

Terabyte:
    Machines: Yahoo 1,407 Google 1,000
    Disks: Yahoo 5,628 Google 12,000
Petabyte:
    Machines: Yahoo 3658 Google 4000
    Disks: 14,632 Google: 48,000

Yahoo published their network specifications, but Google did not. Clearly the network speed is very relevant.

The two take away points are: Hadoop is getting faster and it is closing in on Google's performance and scalability.

Re:Not quite as impressive as it sounds (1)

fluffykitty1234 (1005053) | more than 5 years ago | (#27980375)

One other difference:

Google:
"we asked the Google File System to write three copies of each file to three different disks."

Yahoo:
"On the larger runs, failure is expected and thus replication of 2 is required. HDFS protects against data loss during rack failure by writing the second replica on a different rack and thus writing the second replica is relatively slow."

Google is using 4x as many disks, but writing 1.5 as much data.

I'm actually more impressed that Google is cramming 12 disks onto a single machine, how do they get them to fit?

Beowulf (1)

bhsbulldozer (1519367) | more than 5 years ago | (#27979479)

Imagine a beowulf cluster of those!

lets just hope.. (1)

pablo_max (626328) | more than 5 years ago | (#27979497)

I really hope that this works across multiple drives, because my p0rn collection is so spread out it would take for ever to sort manually!

Google Sort (2, Interesting)

jlebrech (810586) | more than 5 years ago | (#27979529)

Im looking forward to sorting my search results by Date, Title, Description, Author, etc..

Re:Google Sort (1)

daemonburrito (1026186) | more than 5 years ago | (#27979799)

This may get easier if HTML5 catches on. I've been playing with it, and the new <time> and <article> tags are extremely useful.

I used to be sympathetic to the "limited view of html" argument, but after writing a couple of tools that need to search the dom, I'm convinced that the semantic tags work a lot better than abusing css classes. The consistency is going to help search engines, too.

World Record? (0)

Anonymous Coward | more than 5 years ago | (#27979579)

The Gray sort metric is defined as TB/minute on a large data set (>=100TB). Apache Hadoop got 100TB in 173min = 0.578TB / min.

Half a year ago, Google's MapReduce sorted 1PB in 362 minutes. Rate = 2.762TB / min

http://developers.slashdot.org/article.pl?sid=08/11/23/1637219&from=rss

My algorithm can sort anything in 1 second (1)

nganju (821034) | more than 5 years ago | (#27979595)

My sorting algorithm operates in constant time. I should really enter it into one of these competitions. It's called Intelligent Design Sort: http://www.dangermouse.net/esoteric/intelligentdesignsort.html [dangermouse.net]

Re:My algorithm can sort anything in 1 second (0)

Anonymous Coward | more than 5 years ago | (#27980409)

Good luck on your lawsuit with DJ Danger Mouse.

(Kinda stupid to be whoring your vanity site out on the same day as a front page story about the person who could easily sue you and take it from you.)

Re:My algorithm can sort anything in 1 second (1)

dotgain (630123) | more than 5 years ago | (#27980853)

Gee, I actually thought it was funny. Now, maybe I'm just an idiot that's easily amused, but you sir have problems that could make just about anybody feel better about themselves.

This overturns all are fundamental assumptions! (1)

ClosedSource (238333) | more than 5 years ago | (#27980219)

Like the widely-held belief that sorting speed is related to the software license used.

Boy did I screw up that title (1)

ClosedSource (238333) | more than 5 years ago | (#27980247)

Make that "all of our" instead of "all are". A mind is a terrible thing to waste.

Re:Boy did I screw up that title (1)

dotgain (630123) | more than 5 years ago | (#27980879)

No no no, it's "ALL YOUR", look, I'll show you:

All your fundamentals are overturn. Move belief! What you pay!

Ivan (-1, Flamebait)

Anonymous Coward | more than 5 years ago | (#27980469)

How typical of the FOSS pilgrims! They love adorning themselves with wooden medals. It's not software quality but mostly quantity that beats in this contest. Besides, it's so an insignificant contest, that the big players simply didn't take part.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>