Beta

Slashdot: News for Nerds

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Microsoft Research Introduces Record-Beating MinuteSort Tech

timothy posted more than 2 years ago | from the smart-people-with-cool-ideas dept.

Databases 118

mikejuk writes "A team from Microsoft Research has taken the lead in the MinuteSort data sorting test using a specially-devised technology: Flat DataCenter Storage. The figures are impressive — 1401 gigabytes in 60 seconds, using 1033 disks across 250 machines. Not only is this three times as much as the previous record, but also, it uses only one sixth of the hardware resources, according to a blog post about the test from Microsoft. One thing that's interesting about the success is the technology used. While solutions such as Hadoop and MapReduce are traditionally used for working with large data sets, Microsoft Research created its own technology called the 'Flat Datacenter Storage,' or FDS for short. This isn't just academic research, of course. The team from Microsoft Research has already been working with the Bing team to help Bing accelerate its search results, and there are plans to use it in other Microsoft technologies."

cancel ×

118 comments

This is what I like about Microsoft (1, Troll)

Asksa (2646127) | more than 2 years ago | (#40089927)

Their support for research and innovation is top-notch. They are pretty much the only one of the large companies that fund this kind of research and they fund it with billions. Their work does lots of good for the world. Good job guys.

Re:This is what I like about Microsoft (3, Insightful)

allcoolnameswheretak (1102727) | more than 2 years ago | (#40090037)

"They are pretty much the only one of the large companies that fund this kind of research"
Bullshit alert.

"Their work does lots of good for the world."
For the world? Or for Microsoft?

Citations needed.

Re:This is what I like about Microsoft (5, Informative)

MikeyC01 (231948) | more than 2 years ago | (#40090397)

From the Wiki (http://en.wikipedia.org/wiki/Microsoft_Research#Laboratories [wikipedia.org] ), all of the following have come from MS Research

C#
Comic Chat (IRC Client)
F#
Sideshow (Became Desktop Gadgets)
Surface (TouchLight)
SenseCam
ClearType
Group Shot
Allegiance (Game)
Songsmith

I'd say C#, F#, and ClearType are pretty big contributions

Re:This is what I like about Microsoft (1, Interesting)

John Sugs (2646157) | more than 2 years ago | (#40090563)

Quora also has a discussion about some of these -> http://www.quora.com/Microsoft-Research/What-products-have-come-out-of-Microsoft-Research [quora.com]

And these are exact, high-profile products that have come out of Microsoft Research. You have to remember that they work on many smaller things that will be then integrated into other Microsoft products, or do work 'just for science' (which is pretty amazing from Microsoft).

Re:This is what I like about Microsoft (2)

allcoolnameswheretak (1102727) | more than 2 years ago | (#40090623)

I don't doubt that Microsoft Research has made important contributions (even though from the list you posted only C# is something I can put my finger on). Obviously, it's the "They are pretty much the only one" part that is complete nonsense.

Re:This is what I like about Microsoft (3)

Lord Lode (1290856) | more than 2 years ago | (#40090791)

For one moment there, I read "Comic Sans" instead of "Comic Chat".

Re:This is what I like about Microsoft (0)

Anonymous Coward | more than 2 years ago | (#40093137)

Comic Sans shipped as part of Comic Chat, so it is indirectly the fault of Microsoft Research.

BURN THE WITCH!

Re:This is what I like about Microsoft (3, Informative)

Yvan256 (722131) | more than 2 years ago | (#40090829)

ClearType invented nothing apart from the name itself.

Sub-pixel rendering was used two decades ago by Apple [grc.com] .

"Back in 1976, my design of the Apple II's high resolution graphics system utilized a characteristic of the NTSC color video signal (called the 'color subcarrier') that creates a left to right horizontal distribution of available colors. By coincidence, this is exactly analogous to the R-G-B distribution of colored sub-pixels used by modern LCD display panels. So more than twenty years ago, Apple II graphics programmers were using this 'sub-pixel' technology to effectively increase the horizontal resolution of their Apple II displays." - Steve Wozniak

Re:This is what I like about Microsoft (5, Informative)

Anonymous Coward | more than 2 years ago | (#40092761)

You immediately lose credibility by citing Steve Gibson.

The type of subpixel rendering done on old Apple IIs essentially treats the color display as a monochrome display of triple the resolution. This is clever and useful, but causes color fringing.

ClearType takes the concept substantially further by applying perceptual modelling to determine how the subpixels can be used. It's similar to MP3 audio, in that the process adds artifacts, but some artifacts will be invisible (or inaudible in MP3's case) to a human. The trick is minimizing the visible artifacts.

For example, if you have a one pixel wide line, it is always safe to shift it one third of a pixel to the left. RGB becomes BRG, which still appears the same.

However, if you have a one third pixel width line, you cannot just use one third of the subpixels. A "white" vertical line would be all red, all green, or all blue, depending on which subpixel it fell on. ClearType would render it using all three subpixels but in the correct color.

There's quite a bit more to it - sometimes you can use a single subpixel depending on what neighbors it, and/or you can adjust adjascent subpixels to mask fringing artifacts.

So yes, sub-pixel rendering isn't a wholly new concept, but saying ClearType isn't novel is willfully ignorant.

Re:This is what I like about Microsoft (0)

Anonymous Coward | more than 2 years ago | (#40092995)

You may be too young to have lived this, but the old CRTs were doing the color fringing by themselves. The only way to display colors was to display a columns of on/off/on/off/etc pixels. Even the Color Computer 2 used this method because that's how CRTs worked. Games even had a "Push the reset button until the square is red" messages when they started because the offset in pixels was random for some reason.

The only things ClearType did was to apply this to the RGB subpixels of LCD panels, but the technique is still the same. And it also brought us the insanity that is hammering fonts into pixels [earthli.com] .

Re:This is what I like about Microsoft (1)

sexconker (1179573) | more than 2 years ago | (#40093895)

For example, if you have a one pixel wide line, it is always safe to shift it one third of a pixel to the left. RGB becomes BRG, which still appears the same.

I have a pentile display in portrait mode you insensitive clod!

Re:This is what I like about Microsoft (-1, Troll)

znrt (2424692) | more than 2 years ago | (#40091113)

I'd say C#, F#, and ClearType are pretty big contributions

thanks for sparing me the time to check out most of the items in that list, most of which i had never heard of. i quickly looked up cleartype and it turns out that those ms-r&d guys ... just invented font anti-aliasing! 20ish years late, but nobody is perfect. but anyway if you consider that c# is a contribution to the world (and a big one for that matter) then the rest should be pretty irrelevant junk.

c#!!! ye gods & little fishes. the only thing c# has contributed to the world is largely replacing vd, thus keeping existing dev-base around, thus maintaining windows' sale figures. contribution in regards of academic research would be next to zero. in fact there's is no single concept in whole .net that was radically new, its just known stuff wrapped together with commercial purpose. it has indeed extensive application, but that doesn't mean it is crucial for that applications to exist. it just means royalties and licenses go to ms instead of others. this contributed to microsoft, not the world. your citation is invalid, try again. :)

Re:This is what I like about Microsoft (2)

MikeyC01 (231948) | more than 2 years ago | (#40091251)

it just means royalties and licenses go to ms instead of others. this contributed to microsoft, not the world. your citation is invalid, try again. :)

.NET, C#, VB.NET, and F# are all free... Download the .NET framework, fire up a text editor, and use the command-line compiler.
Visual Studio Express versions are free... Wrap a GUI around your development.
Parts of ASP.NET are even open source now... And they're accepting contributions from the public.

What exactly hasn't been contributed?

Re:This is what I like about Microsoft (-1)

Anonymous Coward | more than 2 years ago | (#40091679)

Alright, I'll go download all of this for my OS X and Unbuntu boxes!

Wait... they all require Microsoft Windows! And I suppose you can only compile for Windows, too.

So much for being free.

Re:This is what I like about Microsoft (1)

Anonymous Coward | more than 2 years ago | (#40092097)

C# seems a lot like Java, F# looks like OCaml and didn't Steve Jobs take calligraphy classes to produce the first "TrueType" like font?

Re:This is what I like about Microsoft (4, Funny)

robthebloke (1308483) | more than 2 years ago | (#40090489)

"Their work does lots of good for the world."
For the world? Or for Microsoft?

Dude, seriously! You do realise this algorithm has been developed to help Microsoft sort through all of the outstanding 'serious security flaw found in IE6' tickets? Why else do you think they'd need 1033 hard drives, and 250 machines?

Re:This is what I like about Microsoft (4, Informative)

Missing.Matter (1845576) | more than 2 years ago | (#40090731)

Citations needed.

Here you go [microsoft.com] . About 14,000 peer reviewed publications for the computer science community, about 10,000 of which were published completely in house by Microsoft Research, and about 4,000 of which were done in collaboration with Universities.

Re:This is what I like about Microsoft (1)

Fwipp (1473271) | more than 2 years ago | (#40091073)

The issue isn't whether they fund research - clearly they do. The GP was taking issue with "the only one" part.

Re:This is what I like about Microsoft (1)

Missing.Matter (1845576) | more than 2 years ago | (#40091195)

I guess I should have specified, I was providing citations to the second part of his post:

For the world? Or for Microsoft?

Publishing research is beneficial for the world. Not just Microsoft.

Re:This is what I like about Microsoft (1)

Anonymous Coward | more than 2 years ago | (#40090045)

Am I the only one that thinks these shill stories and comments are getting old?

Re:This is what I like about Microsoft (0)

Anonymous Coward | more than 2 years ago | (#40090801)

They indeed do support research. The parent post was a statement of fact. Do you intend to say that MS does not invest in research? If so, can you support it? You should visit http://research.microsoft.com and explore the fields in CS they work on and the different research labs they have in different countries. Few companies invest that much in research. Compare that to Apple, for example. I believe only IBM research is in the same league. In academia too, MS research is held in high esteem.

Re:This is what I like about Microsoft (2)

Bigby (659157) | more than 2 years ago | (#40091183)

The dispute has nothing to do with Microsoft Research. It is the claim that they are the only big company that researches. You even refute this yourself with your mention of IBM. Almost everything Google does is in the name of R&D (hence the high risk/reward business model). Then you extend that outside of the tech industry and look at pharmaceutical research. Look at Monsanto R&D. Look at Boeing R&D.

The GGP was complete troll material.

Re:This is what I like about Microsoft (0)

Anonymous Coward | more than 2 years ago | (#40090053)

Wouldn't GoOooogle also fit into that category?

Re:This is what I like about Microsoft (-1, Troll)

Asksa (2646127) | more than 2 years ago | (#40090107)

Wouldn't GoOooogle also fit into that category?

Google doesn't really innovate or do any research. The closest you get is the 20% time they give to engineers (note that not for other personnel). In fact, the only real products Google has made in-house are their search engine and gmail. Everything else (YouTube, Google Earth, Maps, Android) have been buy-outs of startups or copied, like Google+.

Re:This is what I like about Microsoft (0)

Anonymous Coward | more than 2 years ago | (#40090197)

gmail wasn't made in-house either.

Re:This is what I like about Microsoft (0)

Anonymous Coward | more than 2 years ago | (#40090839)

I think GMail was their own. I don't remember them acquiring any company/startup and offering GMail circa 2004.

Re:This is what I like about Microsoft (5, Informative)

sideslash (1865434) | more than 2 years ago | (#40090301)

Google doesn't really innovate or do any research. The closest you get is the 20% time they give to engineers (note that not for other personnel). In fact, the only real products Google has made in-house are their search engine and gmail. Everything else (YouTube, Google Earth, Maps, Android) have been buy-outs of startups or copied, like Google+.

What about their self-driving cars? What about their glasses and stuff? They have a lot of secret research projects that they are allegedly spending billions on. Are you trolling, or am I misunderstanding you?

Re:This is what I like about Microsoft (1)

Asksa (2646127) | more than 2 years ago | (#40090401)

Google's self-driving cars project comes from Stanford and was mostly US government funded(!).

Shrun's team at Stanford created the robotic vehicle Stanley which won the 2005 DARPA Grand Challenge and its US$2 million prize from the United States Department of Defense.

Re:This is what I like about Microsoft (2)

Missing.Matter (1845576) | more than 2 years ago | (#40090557)

What about their self-driving cars?

They bought the talent: aka Sebastian Thrun, who worked on many successful self driving cars before being hired by Google.

Re:This is what I like about Microsoft (1)

Rob Y. (110975) | more than 2 years ago | (#40090813)

Did you expect Sergey Brin to develop the self-driving car?

Re:This is what I like about Microsoft (1)

Missing.Matter (1845576) | more than 2 years ago | (#40091167)

No. I'm not saying it's a bad thing either. I'm just saying when Google's self driving car started, there were already numerous examples of self driving cars working around the world, and they basically bought the talent behind the best one to ensure their system was the best in the world. Which I think is a good thing.

Re:This is what I like about Microsoft (3, Insightful)

Anonymous Coward | more than 2 years ago | (#40090877)

Don't all research houses always 'buy' talent by recruiting qualified candidates? Isn't a university the place where people develop skills and then 'sell' themselves to employers?

Re:This is what I like about Microsoft (0)

Anonymous Coward | more than 2 years ago | (#40093543)

Of course. Microsoft hired font / graphics people to develop ClearType. They hired programming guys to develop their programming languages. It's not as if Google's research fails to be research because they hire experts and buy existing products in their infancy.

Re:This is what I like about Microsoft (3, Funny)

Sarten-X (1102295) | more than 2 years ago | (#40090313)

Exactly right. Functional self-driving cars aren't really innovations like a fancy coffee table [wikipedia.org] is!

People don't really need silly things like augmented reality glasses or street-level pictures of their mapped destinations - they need internally-inconsistent UIs that change at every major OS version! Thank God we have Microsoft to innovate for us!

Re:This is what I like about Microsoft (2)

zlives (2009072) | more than 2 years ago | (#40090447)

personally i wouldn;t knock any research, funded for "humanity" or profit. If it advances technology eventually it becomes accessible... plus its kinda cool

Re:This is what I like about Microsoft (1)

Sarten-X (1102295) | more than 2 years ago | (#40090641)

As someone who once got approval to build a computer-controlled catapult "for research", I agree. Research is good. Shills aren't.

Re:This is what I like about Microsoft (1)

Johann Lau (1040920) | more than 2 years ago | (#40093751)

What the fuck is it with everybody saying "shill!" every second word out of their worthless mouths on Slashdot these days?

Is that some new thing, like "noob" or "fag", or do you actually believe *anyone* who *ever* says something remotely positive about Microsoft, Google, Apple, or Facebook (etc.) is actually paid by them?

Why not simply pay attention to what they're actually saying? If you find fault with it, why not refute it?

If you can't refute it, saying "shill" just makes that even more clear, and if you can, saying "shill" kinda makes it seem like a random dumb fuck got something correct by pure chance.

It's been mediocre, it's been predictable, it's been utterly lame, and that dumb fucking meme -- it really isn't more, a fad for dimwits -- kinda could use some death about now.

Re:This is what I like about Microsoft (2)

Aguazul2 (2591049) | more than 2 years ago | (#40090217)

Typically Microsoft develops something impressive-sounding that is good for a news story, but that never gets out of the research labs -- i.e. vapourware as far as the rest of the world is concerned. Typically we hear about Google research projects when there is a chance to try them out. Intel innovations make it into the next chip generation. As a result, Microsoft 'innovation' stories fail to arouse any interest for me at all.

Re:This is what I like about Microsoft (0)

John Sugs (2646157) | more than 2 years ago | (#40090467)

Actually, many Microsoft Research projects do make it into final products, but they're usually one part of them. Their job isn't to develop complete products but new or improved technology. Then teams at Microsoft's business areas take those when fit and integrate into their own projects, and those are the ones you see.

This is different to Google as many Google's "research" projects will be instantly turned into actual products, and then most of them shortly afterwards abandoned.

Re:This is what I like about Microsoft (1)

Aguazul2 (2591049) | more than 2 years ago | (#40093689)

Yes, like I say, we get to try out a good part of Google's research. The only part of Microsoft's research that gets out of the lab is news of prototype vapourware. I wonder what the motivation is? To make MS seem cutting edge?

Re:This is what I like about Microsoft (4, Informative)

Galestar (1473827) | more than 2 years ago | (#40090253)

Not sure if troll. Yes, they fund "this kind of research", but to say they are "pretty much the only one of the large companies that [do so]" is absurd. Please see Hadoop's origins (Google). Oh and also IBM who eats this shit for breakfast.

Re:This is what I like about Microsoft (1)

Anonymous Coward | more than 2 years ago | (#40090771)

The one thing about Microsoft that I respect is their seriousness about R&D - MS has the highest R&D budget ($9 billion) of all the companies. And they have Turing award winners working with them (C.A.R Hoare and Charles Thacker come to mind). I once had the honour of listening to C.A.R Hoare at a conference where he said that the most difficult job for MS R&D people is to make the rest of the organization use what they create.

Re:This is what I like about Microsoft (0)

Anonymous Coward | more than 2 years ago | (#40090903)

They might spend a lot on R&D, but that rarely if ever makes it's way into their actual products.

So, if a company spends money on R&D, but doesn't make use of it, then what is the point?

Re:This is what I like about Microsoft (1)

Missing.Matter (1845576) | more than 2 years ago | (#40091333)

Can you actually back that up? Microsoft does a lot of fundamental CS research: algorithms, graphics, hardware, machine learning, number theory, statistics, etc. Microsoft also has thousands of products with thousands of components each. How exactly do you know the inner workings of all these products, and how can you confirm the research in thousands of peer-reviewed journals isn't making it into their products?

Re:This is what I like about Microsoft (1)

mikestew (1483105) | more than 2 years ago | (#40091069)

I don't know about troll, but certainly a brand new UID that posted to one or two other articles while waiting for this to come out of the firehose. My guess is an MSR employee that found out "hey, our stuff's going to be on ./!"

But that's neither here nor there, the post is inaccurate garbage.

Re:This is what I like about Microsoft (1)

CODiNE (27417) | more than 2 years ago | (#40091275)

Shooter McGavin: You're in big trouble though, pal. I eat pieces of shit like you for breakfast!
Happy Gilmore: [laughing] You eat pieces of shit for breakfast?
Shooter McGavin: [long pause] No!

- Happy Gilmore 1996

Dude, it's a sort (0)

Anonymous Coward | more than 2 years ago | (#40090305)

"using 1033 disks across 250 machines"

They sorted a terrabyte and a half in 60 seconds using 1000 disks and 250 machines.

You realize that sort is one of those things I can simply pre-section the data into smaller chunks (e.g. a,b,c,da-df,dg-dz,...etc) and then sort the smaller chunks?

With 192MB PC's it's trivial to pre-filter during the load, then sort in memory.

Re:Dude, it's a sort (4, Interesting)

CannonballHead (842625) | more than 2 years ago | (#40090357)

Not only is this three times as much as the previous record, but also, it uses only one sixth of the hardware resources, according to a blog post about the test from Microsoft.

The important part is not that this is a new approach, but that they beat the previous record using less hardware.

Re:Dude, it's a sort (0)

Anonymous Coward | more than 2 years ago | (#40090501)

my 200+GB mailbox can use this

Fewer CPUs, but it seems like the network is beefy (0)

Anonymous Coward | more than 2 years ago | (#40091219)

The important part is not that this is a new approach, but that they beat the previous record using less hardware.

Non-expert here, but they require a "full bisection bandwidth network" in which every computer gets full bandwidth to every other computer.

Re:This is what I like about Microsoft (3, Informative)

rjr3 (658693) | more than 2 years ago | (#40090341)

http://www.research.ibm.com/ [ibm.com]

They used to have one of the most amazing IT geek magazines.

Re:This is what I like about Microsoft (1)

alvinrod (889928) | more than 2 years ago | (#40090369)

No, they're the only one that talk about it all the damned time. Remember all of the Courier "leaks" from a year or so ago, and all of the talk about the Surface?

Other companies also spend a lot on R&D, but they just don't publicize it. Do you think Apple pulled the iPhone out of a hat or something? Hell, here's a recent blog post [arcfn.com] where the author tore down Apple's power adapter for the phone and found some interesting design work. Google probably does a lot of stuff to improve their search algorithms that they don't talk about much either and probably have stuff they've been working on for years that won't be announced or released until it's ready.

Re:This is what I like about Microsoft (5, Insightful)

rockmuelle (575982) | more than 2 years ago | (#40091093)

The big difference is that Microsoft Research is one of the last large corporate research labs focused on pure research. That is, research done for the sake of the research, not to drive product development. Research done at MSR doesn't have to be product driven (it has to be in the general space of software and computers, but that's about the only requirement). MSR is well funded by Microsoft and an integral part of the company's culture.

Sure, IBM, HP, and Intel all have research labs, but their charters have been re-written over the last ten years to focus more on product-centric research. Most research projects at these companies must start with a business plan that shows how the work will be commercialized within 5 years before being approved. This is not the pure research these labs were once known for.

Google, Facebook, Yahoo, and many other internet companies have some interesting projects (self driving cars, for instance), but these tend to be one-off projects and aren't part of a larger, long lived research organization.

Another interesting aspect of MSR is that they encourage all MS developers to take a stint in the organization, not just specially recruited Ph.D.s. It's not uncommon for someone to go from working on a product for a few years, take some time in MSR, then go back to product work.

I've worked directly with many of the research groups mentioned in this post over the last 20 years. Based on my experiences, MSR is truly the last real corporate research group (in the spirit of 20th century PARC/Watson/et al). The others are just part of the product funnels or whims of the founders.

-Chris

Re:This is what I like about Microsoft (2, Interesting)

Rakishi (759894) | more than 2 years ago | (#40090621)

First of all tons of companies fund research. Lots of papers come out of them of all kinds and plenty more that is never published.

Second of all Microsoft is actually known for being a black hole of research. Researchers go in and almost nothing comes out. They hire people just so their competitors can't hire them. They may do a few demos but nothing commercial comes from them.

Re:This is what I like about Microsoft (1)

shutdown -p now (807394) | more than 2 years ago | (#40091477)

Second of all Microsoft is actually known for being a black hole of research. Researchers go in and almost nothing comes out.

Except for all the published academic research papers with stuff like what's described in TFA?

Re:This is what I like about Microsoft (1)

Rakishi (759894) | more than 2 years ago | (#40091975)

All of which have patents attached to them ensuring that they never become too useful to anyone.

Re:This is what I like about Microsoft (1)

JustSomeProgrammer (1881750) | more than 2 years ago | (#40092225)

What corporation funds research, publishes papers, and DOESN'T patent the results? That's kindof the point... businesses don't do things out of the goodness of their hearts.

Re:This is what I like about Microsoft (1)

kiwimate (458274) | more than 2 years ago | (#40093825)

Microsoft is actually known for being a black hole of research. Researchers go in and almost nothing comes out. They hire people just so their competitors can't hire them.

Citation?

As for nothing coming out, you're apparently not including published papers [microsoft.com] (lots published by respectable bodies like IEEE, ACM, Oxford Publishing, etc.), or downloads [microsoft.com] such as Excel plug-ins to simplify working with genomic sequences, Differentially Private Network-Trace-Analysis Tools [microsoft.com] , an e-mail loss detection add-in [microsoft.com] , etc., etc.

Sure, not as sexy as self-driving cars. But serious, hard research usually isn't that sexy or appealing to the general public. I thought this was a geek web site...?

Re:This is what I like about Microsoft (0)

Anonymous Coward | more than 2 years ago | (#40090645)

Most research never leaves Microsoft. Only the developers of the project leave. (with their research they cannot use anymore)

They are far from the one of the only companies who fund this sort of research.
This isn't even anything out of the ordinary, either.
This sort of effort is constant, the search for new algorithms and approaches to working with large datasets.
Whether it is from the amateurs, or all the way up to the big guys like Microsoft, Google and IBM.

Re:This is what I like about Microsoft (1)

Missing.Matter (1845576) | more than 2 years ago | (#40091369)

Most research never leaves Microsoft. Only the developers of the project leave. (with their research they cannot use anymore)

You know... except the 10k+ peer reviewed publications [microsoft.com] available to the CS community, nothing ever comes out of MS R&D.

First post (5, Funny)

Anonymous Coward | more than 2 years ago | (#40089975)

Sorted by Microsoft

doesn't resolve confusion (0)

j-stroy (640921) | more than 2 years ago | (#40090079)

between their ass and a hole in the ground..

/smart people working for dumb people working for smart people

Re:doesn't resolve confusion (0)

Anonymous Coward | more than 2 years ago | (#40090143)

Thanks for your contribution.

So what? (-1)

Anonymous Coward | more than 2 years ago | (#40090091)

WHo cares? It's a commonly known and obvious algorithm! I bet M$ will try to patent this! Fuck M$ Fuck Micro$oft. $ $ $ $ $ Micro$oft. F/L/O/S/S FTW and RMS!

faster than ever... (2)

schlachter (862210) | more than 2 years ago | (#40090275)

...yet MinuteSort still takes a minute!

Re:faster than ever... (1)

Creepy (93888) | more than 2 years ago | (#40090323)

...or is it minute sort, as in tiny. Minute Maid or Minute Maid? Am I going mad, yes I've gone mad. The article is slashdotted already, and my mad mind will never know.

Re:faster than ever... (0)

Anonymous Coward | more than 2 years ago | (#40090995)

hey now, don't get hung up on minutiae..

Re:faster than ever... (1)

Anonymous Coward | more than 2 years ago | (#40090505)

Phhbt, you think that's bad? Race cars are way faster now than they were in the '60s, and yet the 24 Hours of Le Mons still takes a whole day!

Re:faster than ever... (0)

Anonymous Coward | more than 2 years ago | (#40090759)

That's because in the 24 Hours of Lemons they still use the same (now old POS) cars from the 60s....

Re:faster than ever... (0)

Anonymous Coward | more than 2 years ago | (#40091143)

I believe the correct spelling is "Le Mans".

Re:faster than ever... (1)

louaish88 (731196) | more than 2 years ago | (#40092427)

The contest is how much can be sorted in a minute, so the 1400gb is the important number.

Good job Microsoft! (-1, Flamebait)

Hemi Rodner (570284) | more than 2 years ago | (#40090363)

You are such a good boy. Here. Let me pat you on the head. *pat*
Now, if only you could play well with Linus.. it'd have made me a much happier father.

Other technologies... (5, Funny)

fahrbot-bot (874524) | more than 2 years ago | (#40090371)

The team from Microsoft Research has already been working with the Bing team to help Bing accelerate its search results, and there are plans to use it in other Microsoft technologies.

So Bing is going to scrape their search results from Google *and* other search engines? :-)

Faster! (0)

Anonymous Coward | more than 2 years ago | (#40090471)

The team from Microsoft Research has already been working with the Bing team to help Bing accelerate its search results,

Ah, so now you can get irrelevant results even faster!

Did they do anything? (0)

Anonymous Coward | more than 2 years ago | (#40090519)

Did they actually do anything or is this just from them using better hardware. The article says the record holder is yahoo from 2009 3 years better technology and unmentioned (from what I read) amount of funds I would assume anyone could beat it.

remote filesystem (0)

Anonymous Coward | more than 2 years ago | (#40090607)

And the filesystem was remote...

I would almost bet its the updated SMB protocol on Server 2012.

Did they do anything? (2)

jeff8j (2646163) | more than 2 years ago | (#40090631)

Did they actually do anything or just build a machine using todays hardware and lots of funding. A team from yahoo got the record in 2009 hardware has changed alot in the 3 years and when money is not a object couldnt anyone do about the same?

Re:Did they do anything? (0)

Anonymous Coward | more than 2 years ago | (#40090879)

Did they actually do anything or just build a machine using todays hardware and lots of funding.

As even the summary says, they developed a different way of referencing the massive amount of data involved that allowed higher throughput with less hardware than the other options.
From skimming the linked article, it looks like they are using a method where every computer listens to all data on the LAN, then locally decides how much it cares about in its own segment of the algorithm. Until someone links the patent that's the best analysis I can come up with.

Re:Did they do anything? (1)

jeff8j (2646163) | more than 2 years ago | (#40091067)

"they developed a different way of referencing the massive amount" Different doesnt mean much I could do somthing different doesnt mean it is any better or any worse. The say less hardware and sure maybe quantity wise but not by much microsoft - 1033 disks yahoo - 1406 difference 373 - that alone I would say would just be from advancements made in hard drives yahoo nodes 2x quad core xeons 8GB of assuming ddr2 ram which was current at the time 1gb ethernet port on each node 40 nodes per rack microsoft 2 - 12 cores a cluster 24GB - 96GB assuming ddr3 ram 10gb ethernet ports 78% were 10,000 rpm disks crazy interconnects so the hardware quantity may be lower but again being that their hardware is so much more advanced I dont see that they actually did anything.

Re:Did they do anything? (1)

jeff8j (2646163) | more than 2 years ago | (#40091141)

Sorry ignore the last posting it lost formatting and I cant edit it

"they developed a different way of referencing the massive amount"
Different doesnt mean much I could do somthing different doesnt mean it is any better or any worse.

They say less hardware and sure maybe quantity wise but not by much
microsoft - 1033 disks
yahoo - 1406
difference 373 - that alone I would say would just be from advancements made in hard drives

yahoo nodes
2x quad core xeons 8GB of assuming ddr2 ram which was current at the time
1gb ethernet port on each node
40 nodes per rack

microsoft
2 - 12 cores a cluster
24GB - 96GB assuming ddr3 ram
10gb ethernet ports
78% were 10,000 rpm disks
crazy interconnects

so the hardware quantity may be lower but again being that their hardware is so much more advanced I dont see that they actually did anything.

Re:Did they do anything? (0)

Anonymous Coward | more than 2 years ago | (#40091471)

My thoughts exactly.

Actually with the difference in hardware, I feel that the 3x increase feels pretty poor.

Re:Did they do anything? (1)

jeff8j (2646163) | more than 2 years ago | (#40091617)

Thats what im saying I actually think what yahoo did was amazing given the hardware they used this is just a disgusting purchase to the top. Im not a fan of microsoft or yahoo I just dont like it when people are all microsoft is the greatest when all they did was open their wallet.

Re:Did they do anything? (1)

jeff8j (2646163) | more than 2 years ago | (#40091659)

You know whats also convenient out of the different tests they beat the only one that was held by yahoo. I try not to think this way but it makes me wonder.

Downside (1, Redundant)

ThatsNotPudding (1045640) | more than 2 years ago | (#40090817)

It only works using IE6.

Oh Look (3, Insightful)

degeneratemonkey (1405019) | more than 2 years ago | (#40090841)

More irrational Microsoft hatred from the peanut gallery. Interesting accomplishment from Microsoft Research (a group which has produced all kinds of useful advances in computing and software development, and which has very little to do with shipped products like Outlook, IE6, etc.); Average /. luser interpretation? LOL SHILL ARTICLE FROM TEH MICRO$OFT FAGGORTZ YOU SUCK LOL.

Good to see that a nerd site is inundated with droves of empty-headed group-think religious fanatics!

When you're done masturbating to your imaginary universe, maybe you'd like to sit down with the likes of Simon Peyton-Jones and discuss some of the finer points of the terrible work he and his peers have been doing.

Baa-hahahaha. Right.

Re:Oh Look (1)

Anonymous Coward | more than 2 years ago | (#40091005)

The initial post heaps on un-warranted praise on Microsoft and that post was made using an account that is only getting it's first post today and will not have any further posts. So yes it is shill post and people bitch about shill posts as they should.

Second, it is well known that yes Microsoft spends tons of money on research, it is also well known that almost none of that research makes it's way into their products.

Yes, the individuals who did this deserve praise, but no one will benefit from this research.

Re:Oh Look (1)

Johann Lau (1040920) | more than 2 years ago | (#40093975)

The initial post heaps on un-warranted praise on Microsoft and that post was made using an account that is only getting it's first post today and will not have any further posts. So yes it is shill post and people bitch about shill posts as they should.

Why will it not have further posts? Are you some kind of prophet? It certainly has previous posts [slashdot.org] , among them stuff like

http://slashdot.org/submission/2061021/mozilla---ms-is-blocking-browser-choice---again [slashdot.org]

http://slashdot.org/submission/2055677/why-do-we-tolerate-ie-just-say-no [slashdot.org]

Of course, being such a clever shill, those posts were just to distract from the SUPER HARDCORE SHILL POST that would end up on the front page, right?

No wait, actually the people who scream "shill" are just dumb fucks who picked that word up from other dumb fucks. And if you weren't one of them, you'd realize that. Now fuck off and die.

Re:Oh Look (1)

jeff8j (2646163) | more than 2 years ago | (#40091203)

Do me a favor and look at the previous record holder in close detail and tell me that microsoft actually did anything other than buy the record...

microsoft - 1033 disks
yahoo - 1406
difference 373 - that alone I would say would just be from advancements made in hard drives

yahoo nodes
2x quad core xeons
8GB of assuming ddr2 ram which was current at the time
1gb ethernet port on each node
40 nodes per rack

microsoft
2 - 12 cores a cluster
24GB - 96GB assuming ddr3 ram
10gb ethernet ports
78% were 10,000 rpm disks
crazy interconnects

I actually think yahoos setup was much more amazing on what they acheived and what they acheived it with. Im not a fan of microsoft or yahoo so I feel the same either way im just disgusted at microsoft buying the top then everyone is like wow they are doing something when in realty they are doing very little.

Re:Oh Look (1)

jeff8j (2646163) | more than 2 years ago | (#40091245)

Yes this is my first day under this account but that is irrelevant, I have been coming here for years I just decided to stop posting as anonymous.
Microsoft spends lots of money on many things like you say does this actually help anyone other than microsoft? Of course not so why would anyone praise a marketing scheme?

Re:Oh Look (0)

shutdown -p now (807394) | more than 2 years ago | (#40091525)

Microsoft spends lots of money on many things like you say does this actually help anyone other than microsoft?

Go to Google Scholar and search for papers published by MSR folk.

Re:Oh Look (1)

jeff8j (2646163) | more than 2 years ago | (#40091591)

Isnt that irrelavent because "does this actually help anyone other than microsoft" this being in question not other publications that I care not get into.

Re:Oh Look (2)

shutdown -p now (807394) | more than 2 years ago | (#40091953)

Pretty much all stuff from MSR ends up published, so presumably whatever is new & special here would be published as well, for others to use and build upon.

Re:Oh Look (1)

jeff8j (2646163) | more than 2 years ago | (#40092073)

Well thats good but I guess well just have to wait and see if the algorithm is actually any better which is what im doubting.

Re:Oh Look (2)

vux984 (928602) | more than 2 years ago | (#40092165)

Do me a favor and look at the previous record holder in close detail and tell me that microsoft actually did anything other than buy the record...

Microsoft actually did something other than buy the record.

im just disgusted at microsoft buying the top then everyone is like wow they are doing something when in realty they are doing very little.

Sorting at that scale is fundamentally an i/o bound problem; and distributed sorting is bound by communications between the nodes. Scaling the problem up to more computers makes the problem HARDER, precisely because of intercommunication requirements.

You said it yourself - Microsoft had "Crazy interconnects"
You can't just buy those. They had to invent them.

Here's a much better link than the full article.

http://www.i-programmer.info/news/197-data-mining/4257-new-minutesort-record-set-by-microsoft-research.html [i-programmer.info]

Re:Oh Look (1)

jeff8j (2646163) | more than 2 years ago | (#40092943)

"Microsoft actually did something other than buy the record." What did they do? Sorting on many disks does become harder but hey they had less disks than the last record holder so they actually had it easier If they invented interconnects then they broke the rules of the competition because its supposed to be 100% off the shelf hardware so its all about the algorithm Thanks for the link that the post points to and the first thing I read since yes I read the article and the blog post and the pdfs

Re:Oh Look (2)

vux984 (928602) | more than 2 years ago | (#40093175)

Sorting on many disks does become harder but hey they had less disks than the last record holder so they actually had it easie

They sorted 3 times as much data. Hard drives didn't get 3x faster since 2009. And how many hard drives were involved is nearly irrelevant.

If they invented interconnects then they broke the rules of the competition because its supposed to be 100% off the shelf hardware so its all about the algorithm

They used off the shelf hardware to build the network, but they connected and used it an innovative way. No one else has ever done a sort like this.

Why exactly are you being such a tool about it?

Why is it not higher? (0)

Anonymous Coward | more than 2 years ago | (#40092333)

I understand that communication adds time and complexity, but doesn't this number seem quite low? I wrote a simple External Sort program in Java that can sort 1GB in 2 mins restricted to 150MB of memory on a mid-range laptop, so 1401GB (and especially the previous record of 500GB) doesn't seem that impressive! I understand that the complexity is more than linear, but I really would've thought the world record would be higher than that...

Website, PDF and excerpts (1)

mounthood (993037) | more than 2 years ago | (#40092521)

Website: http://sortbenchmark.org/ [sortbenchmark.org]
PDF: MinuteSort with Flat Datacenter Storage [sortbenchmark.org]

The sorts were accomplished using a heterogeneous
cluster consisting of 256 computers and 1,033 disks, di-
vided broadly into two classes: storage nodes and com-
pute nodes. Notably, no compute node in our system
uses local storage for data
; we believe FDS is the first
system with competitive sort performance that uses re-
mote storage. Because files are all remote, our 1,470 GB
runs actually transmitted 4.4 TB over the network in un-
der a minute. No strong assumptions are made around
key or record lengths; keys and records of other lengths
can be handled with only a performance-neutral config-
uration change.

Summary
FDS is a general-purpose scalable parallel blob store
that exploits a full-bandwidth interconnect to expose the
entire cluster’s disk bandwidth to remote clients. The
sort performance results in this paper demonstrate the
power of the architecture: in both Daytona and Indy
sorts, the system reads the data remotely to the sort ma-
chines, sorts the data across the network, and writes it
remotely back to storage.
Performant remote file access imparts a flexibility ab-
sent in contemporary distributed storage systems. Be-
yond sort, FDS supports a broad variety of scalable large-
data applications. It does so without demanding that
cluster nodes balance compute and disk performance;
more importantly, it does so without demanding that ap-
plications observe locality constraints.

tract locator table (metadata) and P2P? (1)

mounthood (993037) | more than 2 years ago | (#40092559)

Could someone knowledgeable comment on their "tract locator table" (or TLT) metadata system and it's possible relation to P2P protocols? If Bittorrent didn't focus on peer-speed as measured by reads and writes, couldn't it gain an advantage using this? TLT is expected to have consistent membership, but if it was updated once a minute (say), wouldn't that be enough to get the advantages without it taking to long to join a group?

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Create a Slashdot Account

Loading...