Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Microsoft Exploit Predictions Right 40% of Time

timothy posted more than 5 years ago | from the statistics-94pct-nonsense dept.

Microsoft 182

CWmike writes "Microsoft today called its first month of predicting whether hackers will create exploit code for its bugs a success — even though the company got its forecast right just 40% of the time for October. 'I think we did really well,' said Mike Reavey, group manager at the Microsoft Security Research Center (MSRC), when asked for a postmortem evaluation of the first cycle of the team's Exploitability Index. 'Four of the [nine] issues that we said where consistent exploit code was likely did have exploit code appear over the first two weeks. And another key was that in no case did we rate something too low.' Microsoft's Exploitability Index was introduced last month."

cancel ×

182 comments

Sorry! There are no comments related to the filter you selected.

Attention U.S.citizens (-1, Offtopic)

Anonymous Coward | more than 5 years ago | (#25759009)

A MESSAGE FROM THE QUEEN

To the citizens of the United States of America from Her Sovereign Majesty Queen Elizabeth II

In light of your failure in recent years to nominate competent candidates for President of the USA and thus to govern yourselves, we hereby give notice of the revocation of your independence, effective immediately. (You should look up 'revocation' in the Oxford English Dictionary.)

Her Sovereign Majesty Queen Elizabeth II will resume monarchical duties over all states, commonwealths, and territories (except Kansas ,which she does not fancy).

Your new Prime Minister, Gordon Brown, will appoint a Governor for America without the need for further elections.

Congress and the Senate will be disbanded.

A questionnaire may be circulated next year to determine whether any of you noticed.

To aid in the transition to a British Crown dependency, the following rules are introduced with immediate effect:

1.Look up aluminium, and check the pronunciation guide. You will be amazed at just how wrongly you have been pronouncing it.
2. The letter 'U' will be reinstated in words such as 'colour', 'favour', 'labour' and 'neighbour.' Likewise, you will learn to spell 'doughnut' without skipping half the letters, and the suffix '-ize' will be replaced by the suffix '-ise'. Generally, you will be expected to raise your vocabulary to acceptable levels. (look up 'vocabulary').
3. Using the same twenty-seven words interspersed with filler noises such as 'like' and 'you know' is an unacceptable and inefficient form of communication. There is no such thing as US English. We will let Microsoft know on your behalf. The Microsoft spell-checker will be adjusted to take into account the reinstated letter 'u' and the elimination of -ize.
4. July 4th will no longer be celebrated as a holiday.
5. You will learn to resolve personal issues without using guns, lawyers, or therapists. The fact that you need so many lawyers and therapists shows that you're not quite ready to be independent. Guns should only be used for shooting grouse. If you can't sort things out without suing someone or speaking to a therapist then you're not ready to shoot grouse.
6. Therefore, you will no longer be allowed to own or carry anything more dangerous than a vegetable peeler. Although a permit will be required if you wish to carry a vegetable peeler in public.
7. All intersections will be replaced with roundabouts, and you will start driving on the left side with immediate effect. At the same time, you will go metric with immediate effect and without the benefit of conversion tables. Both roundabouts and metrication will help you understand the British sense of humour.
8. The former USA will adopt UK prices on petrol (which you have been calling gasoline) of roughly $10/US gallon. Get used to it.
9. You will learn to make real chips. Those things you call French fries are not real chips, and those things you insist on calling potato chips are properly called crisps. Real chips are thick cut, fried in animal fat,and dressed not with catsup but with vinegar.
10. The cold tasteless stuff you insist on calling beer is not actually beer at all. Henceforth, only proper British Bitter will be referred to as beer, and European brews of known and accepted provenance will be referred to as Lager. American brands will be referred to as Near-Frozen Gnat's Urine, so that all can be sold without risk of further confusion.
11. Hollywood will be required occasionally to cast English actors as good guys. Hollywood will also be required to cast English actors to play English characters. Watching Andie MacDowell attempt English dialogue in Four Weddings and a Funeral was an experience akin to having one's ears removed with a cheese grater.
12. You will cease playing American football. There is only one kind of proper football; you call it soccer. Those of you brave enough will, in time, be allowed to play rugby (which has some similarities to American football, but does not involve stopping for a rest every twenty seconds or wearing full Kevlar body Armour like a bunch of Nancies). Don't try rugby - the Kiwis will thrash you, like they regularly thrash us.
13. Further, you will stop playing baseball. It is not reasonable to host an event called the World Series for a game which is not played outside of America. Since only 2.1% of you are aware there is a world beyond your borders, your error is understandable. You will learn cricket, and we will let you face the Aussies first to take the sting out of their deliveries.
14. You must tell us who killed JFK. It's been driving us mad.
15. An Inland Revenue agent (i.e. tax collector) from Her Majesty's Government will be with you shortly to ensure the acquisition of all monies due (backdated to 1776).
16. Daily 'Tea Time' begins promptly at 4 pm with proper cups, with saucers, and never mugs, with high quality biscuits (cookies) and cakes; plus strawberries (with cream) when in season.

God Save the Queen !

Re:Attention U.S.citizens (0, Offtopic)

Barny (103770) | more than 5 years ago | (#25759049)

Actually that was John Cleese, even posting anon you should give credit where its due.

Most of its correct though, even if it is as old as your mom :)

Re:Attention U.S.citizens (3, Informative)

91degrees (207121) | more than 5 years ago | (#25759111)

Actually that was John Cleese, even posting anon you should give credit where its due.

Actually it originated with One Alan Baxter of Rochester and expanded by other people on Usenet. So if you do give credit where it's due give it where it's actually due.

Re:Attention U.S.citizens (0, Offtopic)

Barny (103770) | more than 5 years ago | (#25759397)

/bow to your knowledge

Afaik that was where it was from, thankyou :)

Re:Attention U.S.citizens (2, Informative)

Barny (103770) | more than 5 years ago | (#25759449)

Ahh, here we go.

http://www.snopes.com/politics/satire/revocation.asp [snopes.com]

More exciting than reading about how badly microsoft can classify security bugs eh? :)

ps. NO FIREFOX, I WILL NOT CAPITALISE THE "M" IN mICROSOFT!

Re:Attention U.S.citizens (0)

Anonymous Coward | more than 5 years ago | (#25759477)

Thank God for a sensible post! Better than that guy eating shit.

Well.... (1)

morgan_greywolf (835522) | more than 5 years ago | (#25759501)

Well, I for, one, welcome our new stiff-upper-lipped, bland food eating, emotionless British overlords!

Dear Queen (2, Funny)

tjstork (137384) | more than 5 years ago | (#25759525)

We would be delighted to become subjects of the crown again, but we doubt that her majesty could afford all the cameras that her subjects are so accustomed to.

PS. The Irish make better beer than you do, and soccer still sucks.

Re:Dear Queen (1)

wisty (1335733) | more than 5 years ago | (#25759689)

Well, it sounds like Slashdot has it's quota of citizens who are, as Peter Cook once wrote, "neutral, i.e. from a foreign country, and probably bearing a deep seated resentment towards a nation that once ruled three quarters of the world, and ruled it well".

Re:Attention U.S.citizens (3, Funny)

Exitar (809068) | more than 5 years ago | (#25759621)

With the exception of points 7. and 9. it all seems quite reasonable.
Maybe one day you'll learn to drive on the right side.
And vinegar is acceptable on salad only, not potatoes.

By the way, I live in Continental Europe and my ancestors, at the time you were wearing animal furs and piling rocks in bizarre patterns, were building aqueducts.

So, in the end, Her Majesty, please
1. learn to drive
2. learn to cook
3. understand that fox hunting isn't a sport
4. stop using that absurd currency that is the pound sterling

Re:Attention U.S.citizens (0)

Anonymous Coward | more than 5 years ago | (#25759889)

Oh, we can drive on the right. We just choose to drive on the left so we can defend ourselves with swords. Comes in useful oh-so-often.

As a Brit, I'd be happy to get rid of Association Football, but replacing it with American Football? Ice Hockey would be my choice. Or Rugby Union. Or even Cricket.

I have to admit that whenever I've had food cooked by an American, it has been the blandest rubbish I have come across. Modern British cooking for me, any day.

What's absurd about pounds? Unless you're referring to pounds, shillings and pence.

Re:Attention U.S.citizens (0)

Anonymous Coward | more than 5 years ago | (#25760113)

That might have been funny when it was posted after the 2000 election, or even after the 2004 election (really, what happened there? Dubbya stole the election in 2000 and you promptly actually elect him the next time round?!).

I for one think that the election this time round was carried out according to the rules; the biggest budget^H^H^H^H^H^H^H^H^H^H^H best candidate won.

Congratulations? (3, Insightful)

Smidge204 (605297) | more than 5 years ago | (#25759027)

That's great, guys, but don't you think being proud that you were right about your code being exploited is... backwards? That's like being proud you correctly predicted you would get stabbed while walking through a ghetto wearing gang colors.

Then again, this is Microsoft. They probably throw an office party every time something compiles without errors.
=Smidge=

Re:Congratulations? (2, Interesting)

David Gerard (12369) | more than 5 years ago | (#25759147)

Indeed. I swear, I called it: it's easier to predict the holes when you release them yourself [today.com] .

After what was expected to be an unusually quiet Patch Tuesday, Microsoft has released eight patches for applications with an insufficient number of security holes. "Our market is the enterprise," said Microsoft security marketer Jonathan Ness. "Information technology professionals know that Windows is the greatest IT job creation scheme in history. Without Patch Tuesday, there's no reason for the experienced IT worker to spend his time hiding out in the server room watching progress bars and getting over his hangover. Also, you can't tell people a virus ate their mail, you actually have to get it back for them."

Re:Congratulations? (2, Interesting)

Roland Piquepaille (780675) | more than 5 years ago | (#25759175)

That's great, guys, but don't you think being proud that you were right about your code being exploited is... backwards?

Well, they're not proud of making exploitable code (if they were, there would have been a giant endless party at Microsoft for the last 20 years), they're proud of predicting when/how fast their code will be exploited.

That's like being proud you correctly predicted you would get stabbed while walking through a ghetto wearing gang colors.

No, it's like correctly predicting that you'll get stabbed 17 minutes after entering the ghetto, by 6 gang members dressed in red.

Re:Congratulations? (2, Insightful)

TheCycoONE (913189) | more than 5 years ago | (#25759423)

No, it's like correctly predicting that you'll get stabbed 17 minutes after entering the ghetto, by 6 gang members dressed in red.

Not at all. It's much more like guessing that you will be stabbed 6.8 minutes after entering a ghetto by 8-9 gang members dressed in red, then actually being stabbed after 17 minutes by 6 gang members wearing pink.

Re:Congratulations? (1)

Zxarr (1109195) | more than 5 years ago | (#25759479)

Wait, aren't we supposed to use car analogy's on /.??

Re:Congratulations? (4, Funny)

NoisySplatter (847631) | more than 5 years ago | (#25759553)

It's like running your own car into a pole, providing the mechanic with your estimate of the damages and claiming you were right when he only overcharges you by 60%.

Re:Congratulations? (1)

davetv (897037) | more than 5 years ago | (#25759931)

rofl

Re:Congratulations? (2, Funny)

hairyfeet (841228) | more than 5 years ago | (#25759585)

Okay.....It is like predicting you will get hit by a VW Bug crossing the street,and instead a Mac Truck nails you before you even get off the curb and drags you twenty feet. With a four out of ten pretty much the only thing they got right was they were going to get hit and it would hurt.

Re:Congratulations? (1)

gEvil (beta) (945888) | more than 5 years ago | (#25759253)

I can think of a few ways they can get that number up. Of course, none of them would be good for the consumer. But when has Microsoft put the consumer above having numbers that it can tout?

Re:Congratulations? (1)

Sockatume (732728) | more than 5 years ago | (#25759259)

If you're sailing in a yacht made of cake with sails of tissue paper, with pegs for both legs and hooks for both hands, it's useful to know where the leaks in your boat are.

Re:Congratulations? (1, Funny)

Anonymous Coward | more than 5 years ago | (#25760085)

Actually, in that case I think it's more useful to know where the lifeboat is!

Re:Congratulations? (5, Insightful)

iammani (1392285) | more than 5 years ago | (#25759271)

Slashdot crowd *loves* MSFT bashing doesnt it.

Ok lets see... Some company (say Canonical or MSFT) builds a huge software and releases it. And a third party finds a bug and reports it to them. Now would be good to predict the severity of the bug, so that the more exploitable ones can be fixed first? Thats exactly what they are doing, and they are able to get the severity 40% of the time right, with no false negatives (that not a single severe one has been classified as a low priority one).

So, now, do you think this is bad or wrong or something?

Re:Congratulations? (4, Insightful)

MrMr (219533) | more than 5 years ago | (#25759409)

They build enough security holes in their applications to do meaningful statistics on the monthly number of exploits in the wild.
So, now, do you think that that is not a reason for criticism on their internal software testing?

Re:Congratulations? (1)

iammani (1392285) | more than 5 years ago | (#25759453)

Hmmm I dont have statistics about number of security holes in MSFT apps vs say adobe acrobat/flash or any close sourced software.

But given that they are closed source, I would tend to think they are doing ok.

And yes I am playing a devils advocate here, though I do hate their bloatware

Re:Congratulations? (0)

Anonymous Coward | more than 5 years ago | (#25759593)

They sell enough copies of their software to do meaningful statistics on the monthly number of exploits in the wild.

Fixed that for you.

Re:Congratulations? (2, Insightful)

mobby_6kl (668092) | more than 5 years ago | (#25760181)

No, the criticism of either their coding practices or QA has nothing to do with a new and fairly efficient way to prioritize bug fixes. They already have the software with all the holes built in. Now they should deal with what they have in the best way possible, don't you agree?

Re:Congratulations? (1)

argStyopa (232550) | more than 5 years ago | (#25759599)

Yes, this IS bad or wrong or something.

Wouldn't it make MORE sense to perhaps spend the human/technical resources FIXING the most exploitable bugs rather than standing around with a beer in hand saying 'yep, that's going to explode for sure'.

*BOOM*

'See? I told you so.'

Re:Congratulations? (2, Informative)

iammani (1392285) | more than 5 years ago | (#25759671)

Wouldn't it make MORE sense to perhaps spend the human/technical resources FIXING the most exploitable bugs rather than standing around with a beer in hand saying 'yep, that's going to explode for sure'.

Yes it indeed would, and thats exactly what they have done and the story is about the review of the practice that happened at the end of the month (read during a review of what became an exploit and what got fixed at the right time)

Re:Congratulations? (3, Insightful)

LordKronos (470910) | more than 5 years ago | (#25759755)

Sure, if you have unlimited resources and can devote an infinite number of people to fixing everything, that would be great. However, if you have finite resources available and have to devote them to fixing up certain areas, how do you know where to devote your attention? If you can come up with a methodology for predicting such a thing, put it to the test, and get decent accuracy in your predictions, then wouldn't that be useful for confirming for you how you should devote your limited resources?

There is nothing unique in what they are doing. I mean, look at the auto industry, for example. They don't just randomly assign engineers to try and make random things safer. They do studies, try to figure out what are the most dangerous aspects of a vehicle, and then assign engineers to work on those specific things.

Fortunately for the auto industry, it's a little easier to do your predictions pre-release, since the "attack vectors" are more limited and well known (there are typically only so many ways you can get into an accident, so it's easier to model a majority of those cases). This allows them to be proactive in fixing flaws. Unfortunately, the attacks vectors in software are a bit more numerous, and you often have to take a more reactive approach. What Microsoft is doing here is trying to model things to see how reasonable it would be to devote resources in certain ways to be proactive.

So again, in what way is this bad?

Re:Congratulations? (1)

Jaktar (975138) | more than 5 years ago | (#25760287)

If I had mod points you'd have them, but next time don't go so in-depth with the auto industry. You may have gone over some peoples heads by not explicitly calling them 'car makers'.

Re:Congratulations? (1)

TheP4st (1164315) | more than 5 years ago | (#25759777)

To me it seem as most of the time there is a greater love for pointlessly bashing twitter and his sockpuppets than MS in the MS article threads.
Effectively making the threads hopeless to read, for an example of what I mean have a look at http://tech.slashdot.org/article.pl?sid=08/11/13/210255 [slashdot.org] with your prefs set to show all comments.

*Curiously awaiting the mod results*

Re:Congratulations? (0)

Anonymous Coward | more than 5 years ago | (#25759901)

Agreed...the 40% figure applies to nothing of importance. It just says that only 40% of the bugs they thought hackers would be smart enough to exploit actually got exploited. If anything, Microsoft is overestimating the brilliance of today's current crop of hackers. The real percentage should take into account all the bugs the deemed as non-critical as well. If they did that, they'd come out at, what...like Six Sigma?

Re:Congratulations? (3, Funny)

sjames (1099) | more than 5 years ago | (#25759921)

Based on their success rate, they should flip a coin instead, then they'll be at 50%. That's what everyone's laughing at.

Re:Congratulations? (0)

Anonymous Coward | more than 5 years ago | (#25760315)

Yawn.

Slashdot crowd *loves* MSFT bashing doesnt it.

When deserved, yes. And it is deserved. Your point?

Yes, I'll bash others when they deserve it too. You'd think a corporate behemoth like Microsoft could take it without having its fans go all "boo hoo hoo, they are meansies!" every single time. I guess not.

What kind of douche bag would vote for Stewart... (0)

Anonymous Coward | more than 5 years ago | (#25759295)

Smalley as their Senator? Is there anything more pathetic than a bitter old self-hating Jew?

That's not too bad (5, Insightful)

91degrees (207121) | more than 5 years ago | (#25759031)

A little heavy on the false positives but no false negatives so it allowed more efficient targeting of the risk areas. Also good enough to provide useful feedback.

It is TERRIBLE (0)

SmallFurryCreature (593017) | more than 5 years ago | (#25759771)

What REALLY happened is this: Every security hole that MS discovered on its own, was exploited BUT we are supposed to be happy because in 40% of the cases MS correctly predicted that it would be exploited.

Let me translate. I left fish out on the kitchen counter, EVERY single time it was eaten by my cat, but don't worry, I predicted he would correctly 4 times. Ain't I a genius.

This prediction ain't just bad by MS, it is idiotic. EVERY SECURITY HOLE WAS EXPLOITED. As you would expect it to be. Then claiming that you are smart by claiming you saw this coming 40% of the time is meaningless.

Coin tossing. If flip a coin and predict 40% of the time that it will land face side up, nobody would assume you have any special powers, it would just be down to randomness. Harmless and useless. But this is not what MS predicted. MS predicted that 40% of the time a coin flipped into the air, would come back down. That ain't a good guess, it shows that MS has a fundemental lack of understanding software security. Then again, that is hardly going to come as a suprise to anyone who has been following them for the last 2 decades.

Re:It is TERRIBLE (3, Informative)

91degrees (207121) | more than 5 years ago | (#25759855)

What REALLY happened is this: Every security hole that MS discovered on its own, was exploited BUT we are supposed to be happy because in 40% of the cases MS correctly predicted that it would be exploited.

No. What happened was this - MS spotted 18 potential security holes. 9 of them were regarded as more serious. A company that focussed on protecting against those 9 would not have been affected at all and would have had less disruption than a company that protected against all 18.

They are offering this as a means to tell their bug fixing department and other companies which areas to prioritize.

Re:It is TERRIBLE (3, Informative)

Nick Ives (317) | more than 5 years ago | (#25760139)

What REALLY happened is this: Every security hole that MS discovered on its own, was exploited BUT we are supposed to be happy because in 40% of the cases MS correctly predicted that it would be exploited.

I know we don't RTFA but please at least RTFS.

'Four of the [nine] issues that we said where consistent exploit code was likely did have exploit code appear over the first two weeks. And another key was that in no case did we rate something too low.'

So no, at least according to the summary not every security hole was exploited. If you're going to claim otherwise at least provide some links to an article; hopefully one supporting your claims although that's not always necessary for the +5 informative.

In fact I just actually bothered to RTFA, just to make sure, and it said that no exploit code appeared for the low ranked vulnerabilities.

Still not getting it. (1)

Barny (103770) | more than 5 years ago | (#25759033)

And another key was that in no case did we rate something too low

Well, that's like saying, after you block all your email from getting through, "We rated all the spam accordingly, and let none of them through".

How about, we just guess, a rough fucking guess, that any "remote code execution" or "run with elevated privileges" exploit or hell ANY GOD DAMN FUCKING BUG YOU FIND, needs fixing, right Microsoft?

Re:Still not getting it. (2, Insightful)

c_forq (924234) | more than 5 years ago | (#25759127)

Wow, have some anger issues there? This isn't about not fixing bugs, this is about prioritizing bug fixes. Anything this large is going to have massive amounts of bugs (I can't count the times I've updated packages in Ubuntu, and the OS-X bug fixes come by the hundreds per .x release). Microsoft, just like Apple and Canonical, has limited resources to fix said bugs (and actually Apple and Canonical get some free work done for them, due to use of open source packages).

Re:Still not getting it. (3, Insightful)

Roland Piquepaille (780675) | more than 5 years ago | (#25759235)

or hell ANY GOD DAMN FUCKING BUG YOU FIND, needs fixing, right Microsoft?

Any goddamn bug doesn't need fixing asap the same way. Software always has bugs, even really good software, so it's a matter of prioritizing which bugs are show-stoppers, which are less problematic and which are minor.

The problem with Microsoft is their habit of releasing bananaware: they ship green software that matures at the customers, at the expense of the customer of course who essentially pays to become a beta-tester for Microsoft. In other terms, when other reputable software shops iron out most bugs in-house before releasing their products, Microsoft just removes show-stoppers and let its customers report all the other bugs.

Re:Still not getting it. (2, Interesting)

Khuffie (818093) | more than 5 years ago | (#25759597)

In other terms, when other reputable software shops iron out most bugs in-house before releasing their products, Microsoft just removes show-stoppers and let its customers report all the other bugs.

You mean, like Apple's Leopard release? Or Apple's iPhone 3G release? Or Apple's mobileme release?

I fail to see how Microsoft has a reputation of releasing 'bananaware' whereas Apple doesn't. I don't recall hearing about major, crippling bugs when Office 2007 came out (one of their biggest apps), and regardless of what you hear on Slashdot, Vista was actually a solid enough release and most of the issues were due to bad drivers that manufacturers didn't bother updating a year beforehand when they had betas and release candidates. (Not saying that neither had bugs, they did, but they were in no way 'beta' software.)

Re:Still not getting it. (1)

lytithwyn (1357791) | more than 5 years ago | (#25759729)

The problem with Microsoft is their habit of releasing bananaware: they ship green software that matures at the customers, at the expense of the customer of course who essentially pays to become a beta-tester for Microsoft.

I've been telling people this for a long time. It really is true. Also notice that in the Microsoft KB, the vast majority of the troubleshooting articles including some wording that attempts to place blame on the user. For instance, we all remember the Windows 98 shutdown bug. If you read the KB article about it, it supposedly only happened if you had "too many" network drives mapped. The fact that this bug occurred even if you weren't ON a network was irrelevant.

This is why Microsoft software sucks (2, Insightful)

QuantumG (50515) | more than 5 years ago | (#25759035)

Any engineer who says that "40% is pretty good predicting" is incapable of writing good software, or managing a project, or, even, applying the scientific method.

Hint: 40% is worse than guessing.

Re:This is why Microsoft software sucks (1)

gbjbaanb (229885) | more than 5 years ago | (#25759063)

Dear MS. I have a foolproof way of enhancing and improving upon your algorithms to determine the exploitability index.

if it comes up heads, its exploitable. Tails its gonna be ok.

I estimate you will increase your predictive capabilities by a whole 10% using this method.

Re:This is why Microsoft software sucks (4, Insightful)

Mateo_LeFou (859634) | more than 5 years ago | (#25759075)

>if it comes up heads, its exploitable. Tails its gonna be ok.

In this case, wouldn't there be as many false negatives as false positives?

Re:This is why Microsoft software sucks (0)

Anonymous Coward | more than 5 years ago | (#25759497)

Depends if Martus sold him a gambling device or not.

Re:This is why Microsoft software sucks (5, Insightful)

Anonymous Coward | more than 5 years ago | (#25759077)

No, it means that they were able to cut the field of their immediate focus nearly in half while not missing any issues. For such a complex system without any precise mathematical model, that's pretty good.

In this case, flipping a coin is statistically likely to let an unaddressed issue through, and that's a big no-no for applications like this.

Re:This is why Microsoft software sucks (4, Informative)

rugatero (1292060) | more than 5 years ago | (#25759131)

Hint: 40% is worse than guessing.

No - from TFA:

The index, launched last month, rates each vulnerability using a three-step system.

Random guesses would be expected to yield 33% success.

Re:This is why Microsoft software sucks (4, Insightful)

Anonymous Coward | more than 5 years ago | (#25759373)

If the steps are sequential, it's less than 33%. The correct figure is 12.5% (50 percent of 50 percent of 50 percent).

Re:This is why Microsoft software sucks (1)

rugatero (1292060) | more than 5 years ago | (#25759747)

If the steps are sequential...

They're not - they are three discrete levels of severity.

The term 'three-step' used in the article is a little misleading.

Re:This is why Microsoft software sucks (1)

iammani (1392285) | more than 5 years ago | (#25759133)

Hint: 40% is worse than guessing.

Hmm, lets see...

If they were guessing the answer of an yes or no question, I would agree with you, getting an yes or no wrong 60% of the time is pretty bad.

But I doubt MSFT would find such a measurement for exploits useful. I would think that, they probably, would guess the probability of an exploit code being created. Like, there is a 90% probability that an exploit code would run amok in the internet.

And getting this probability right 40% of the time, is not bad at all, sounds pretty significant to me.

Re:This is why Microsoft software sucks (2, Informative)

mdmkolbe (944892) | more than 5 years ago | (#25759135)

40% is worse than guessing only if you have only two choices (e.g. heads or tails). If you have more choices it is a bit better than guessing.

MS was predicting not just whether exploits would appear but the kinds of exploits that will appear. Depending on how specific (e.g. there will be a buffer overrun in module XYZ) or general (e.g. there will be an exploit in Windows *somewhere*) they were about the kinds of exploits, 40% could be either pretty good (i.e. they were insightful) or pretty bad (i.e. they chose the obvious things). In either case they would still be better off than pure random chance.

It depends on which exploits they call correctly (1)

Gazzonyx (982402) | more than 5 years ago | (#25759375)

Granted, they're doing better than guessing... but in reality, I only care that they get it right on the risks that count. They could be 1 for 10, if the harm that the single exploit would cause was more than the sum of the other 9, and be doing decent.

For instance, if they patched the priv. escalation to SYSTEM that has a broad surface area (think, say, remote IIS exploit) over 9 exploits that require physical access and can only get guest access. If someone else has physical access to your box, it's no longer yours, anyways. Risk assessment has to account both for the opportunity and consequences of a given security hole.

Re:This is why Microsoft software sucks (4, Interesting)

abigsmurf (919188) | more than 5 years ago | (#25759145)

No it isn't. Unless of course you assume that for every bug hackers flip a coin and go "heads, I'll write an exploit for this".

40% accuracy in predicting with no false negatives? There are plenty of distaster agencies around the world who would be incredibly pleased with that kind of accuracy

Re:This is why Microsoft software sucks (1)

Raynor (925006) | more than 5 years ago | (#25759191)

Actually 40% is quite good considering, as others have mentioned, that 33% would be the random chance.

it is also worth noting that they have 40% prediction of KNOWN threats.

I would bet there are about as many undiscovered exploits re: these updates, which could drive up or down the percentage.

If I can predict the stock market by +7% over random guessing, that is pretty damn good predicting.

Re:This is why Microsoft software sucks (0)

Anonymous Coward | more than 5 years ago | (#25759471)

The horrid acting and overall cheesiness in Babylon 5 becomes slightly less noticeable the more you watch.

This is true. However, that's at least partly because the first season of B5 was very uneven, not helped in the slightest by Michael O'Hare (Commander Sinclair). If you watch the series in order, it slowly improves, with seasons 3-4 being probably some of the best TV ever aired.

The only recurring bad actor I can think of is Jeff Conaway (Zack Allan). All the rest of the main cast was quite good. Maybe not the best of the best, but far better than a lot of sci-fi schlock that came after B5.

Re:This is why Microsoft software sucks (0)

Anonymous Coward | more than 5 years ago | (#25759495)

40% is a bad rate if you assume 50% of the code will be exploitable and the other 50% will not.

But if the proportions are something like 90% x 10% and your predictions are right with a 40% rate, then your predictions are doing a fairly decent job - nothing awesome, but still decent.

Re:This is why Microsoft software sucks (1)

poot_rootbeer (188613) | more than 5 years ago | (#25759761)

Hint: 40% is worse than guessing.

I'm assuming you meant "worse than flipping a coin". But this was not a heads/tails judgment; it was "for this given defect, is it Highly Likely, Somewhat Likely, or Not Very Likely that it will be exploited"?

Re:This is why Microsoft software sucks (0)

Anonymous Coward | more than 5 years ago | (#25759955)

Any engineer who says that "40% is pretty good predicting" is incapable of writing good software, or managing a project, or, even, applying the scientific method.

Hint: 40% is worse than guessing.

Even from the summary we know that there were no false negatives. Sometimes when designing a classifier you're willing to trade off overall classification success in exchange for a reduced false positive or negative rate.

For example, imagine I'm a judge sending people to be executed. I would be willing to misclassify 10 guilty people as innocent, if it meant I could avoid misclassifying 1 innocent person as guilty. In other words I would trade off average classification success for reduced false positives.

I don't know if not knowing that means you're incapable of writing good software, managing a project, or applying the scientific method. It makes your slashdot comments ring false, though.

Here's why only 40% of the time (1)

KWTm (808824) | more than 5 years ago | (#25760067)

"This month, we're going to predict whether evil hackers will exploit bugs in our code. What do you predict?"

Steve Ballmer: "No."
James Allchin: "Yes."
Mike Reavey: "Yes."
Jim Gries: "No, I fixed all the bugs."
Sarah Ford: "I dunno. I'd say no; I'm confident in Microsoft."
Val Mazur: "No."
Rui Chen: "Well, the possibility is there, but they'll never prove that they did, so it's the same as no."
Kathleen Dollard: "Of course I will! er --I mean, THEY will. Yes."
Michel Fournier: "How am I supposed to know? How many people said Yes so far? Oh, okay, then I'll say yes."
Bill Gates: "No. Of course I count as part of Microsoft! Write my vote down. No."

Mike Reavey: "Okay, so, what was the right answer? Oh, umm... we were 40% correct. That's not too bad --there's been improvement."

Shows the confidence they have in themselves (1)

MosesJones (55544) | more than 5 years ago | (#25759041)

Interestingly what they are saying here is that they think that

a) Hackers are smarter than they actually are
b) Microsoft code is easier to exploit than it actually is

So the perception is that Microsoft is better than their prediction, but the implication of that is that Microsoft think they are rubbish.

Maybe all these years of "Microsoft sucks" posts on Slashdot have actually come from the MS security team.

Re:Shows the confidence they have in themselves (1)

Raynor (925006) | more than 5 years ago | (#25759211)

No. What they say is:

You should fix this bug first, since we believe it is the most likely to be exploited.

You can save these for later, since we don't believe it will be immediately exploited.

There is, however, something to be said for hackers referring to this list to find "unlikely" bugs to exploit.

Re:Shows the confidence they have in themselves (1)

JasterBobaMereel (1102861) | more than 5 years ago | (#25759241)

So Microsoft thought their code was exploitable and said so, and it was, and instead of doing something about it they just congratulated themselves on predicting it!

Now here's an odd idea rather than predicting if something is exploitable and then publishing it, why not just not write code that is easily exploitable....!

and note the 40% is only the exploits they know about ....so even that is suspect....

Re:Shows the confidence they have in themselves (1)

maugle (1369813) | more than 5 years ago | (#25760017)

b) Microsoft code is easier to exploit than it actually is

Wait... how does that work?

In progress.. (1)

mat (25086) | more than 5 years ago | (#25759089)

Only 40%, which is already "a success", but they can improve this score, and this would become a triumph !

Re:In progress.. (1)

iammani (1392285) | more than 5 years ago | (#25759169)

Only 40%, which is already "a success", but they can improve this score,

Ahh you are a manager arent you?

I wish my manager was as optimistic.

Re:In progress.. (1)

mat (25086) | more than 5 years ago | (#25760211)

No, I'm George Bush, and now I can say to Microsoft guys: Mission accomplished !

Re:In progress.. (1)

duguk (589689) | more than 5 years ago | (#25759197)

This was a triumph? [wikipedia.org] *

No... this was a Triumph! [wikimedia.org]

* sorry, as much as I love portal... it's getting old!

Exploitability Threat Level Announcement. (3, Funny)

140Mandak262Jamuna (970587) | more than 5 years ago | (#25759115)

Nov 14, Redmond, Washington. Today Head of Vistaland Security of Microsoft, Mr Ima F Anboi announced that Microsoft has raised the Exploitability Threat Level from Light Purple to Sunset Yellow. He urged the users to continue their normal activities and not take precipitous actions.

Microsoft Exploitability Threat Level Indicator is a series of color codes starting from Dazzling Arctic White to Heart of Dick Cheney. Though exact number of these colors is considered a secret, from the past announcements we deduce there are at least 22 million of them.

For PRNewswire, copy edited by Anurag Chakraborty in Bangalore and supervised by Robert Zimmermann in Pittsburgh.

4/9 = 40%? (1, Insightful)

Anonymous Coward | more than 5 years ago | (#25759149)

Research also shows Slashdot editors verify submission figures 112% of the time.

Re:4/9 = 40%? (1)

LordKronos (470910) | more than 5 years ago | (#25759817)

Research also shows Slashdot readers read the articles less than 100% of the time:

FTA:

All told, Microsoft correctly predicted eight out of October's 20 vulnerabilities' exploitability, an accuracy rate of 40%.

and in the previous paragraph:

Of the nine October vulnerabilities marked "Consistent exploit code likely," four did, in fact, end up with exploit code available, said Reavey, for an accuracy rate of 44%.

Wow, and I didn't even have to read the article to respond to you. Simply clicking on the link and spending 2 seconds telling the browser to search for "40%" and then reading one single sentence was enough. But I know, that's a lot to ask.

Stupid Metricians (0)

Anonymous Coward | more than 5 years ago | (#25759179)

"But our metrics said we were save!"
"Windows 7 now ranked no 1 in unexploitability with 1723 unexploitability points ahead of Debian"

Am I the only one who thinks this talk starts we people don't understand the matters that they are dealing with?

Do exploit or dont exploit a bug (0)

Anonymous Coward | more than 5 years ago | (#25759221)

sounds like a yes or no question... won't flipping a coin give you a 50% success rate?

Curious (1)

tuxgeek (872962) | more than 5 years ago | (#25759249)

So ... Are they admitting that bugs in their software that are being targeted by crackers are there by design? Or just incompetence?
If they know their software is filled with bugs, why not just fix them and be done with it before it's released.

Re:Curious (1)

iammani (1392285) | more than 5 years ago | (#25759377)

If they know their software is filled with bugs, why not just fix them and be done with it before it's released.

Ahh that wouldnt be interesting, would it be? Microsoft of course wants to release them with lot of bugs. Thats how they get all the free media coverage with MS fixing the bugs while the Open Source community simply does not fix bugs

/sarcasm

but but but... (1)

3seas (184403) | more than 5 years ago | (#25759255)

there is so many to chose from...

Awesome (0)

Anonymous Coward | more than 5 years ago | (#25759263)

That only puts it slightly below random guessing. Great work guys!

Being right 40% of the time... (1)

tangent3 (449222) | more than 5 years ago | (#25759281)

...is the same as being wrong 60% of the time.

Doesn't look so impressive when you look at it this way.

Re:Being right 40% of the time... (1)

Chrisq (894406) | more than 5 years ago | (#25759301)

I was going to say the same thing. Still, it didn't do George Bush any harm.

Re:Being right 40% of the time... (1)

Icarium (1109647) | more than 5 years ago | (#25759385)

Without knowing the baseline they're working on, this could range from extremely impressive to completely useless.

Ok. So 4 out of the 9 bugs they expected to see exploits codes for actually had exploits meterialise. How many bugs had exploits coded that were not in thier 9 candidates? What is the total number of bugs taken into consideration?

If you were playing "battleship" on a 3x3 board with 4 "ships", taking 9 guesses to hit all 4 would be pretty dismal. Change that into a 30x30 board and suddenly 9 guesses to hit all 4 looks pretty damn impressive.

Re:Being right 40% of the time... (2, Insightful)

dubl-u (51156) | more than 5 years ago | (#25759391)

Doesn't look so impressive when you look at it this way.

Depends on the payoff.

It's not good if you're betting even money on coin tosses. But if you're a venture capitalist, it's great. The general rule for tech VCs is that 7 bets out of 10 will fail, 2 will do ok, and 1 will be a big success. If that 1 success is buying 10% of Google in the very early days, your 70% failure rate is still pretty awesome, because you're still up billions of dollars.

I can predict the same thing if I were Microsoft (0)

Anonymous Coward | more than 5 years ago | (#25759323)

While i do not work for Microsoft (but i do work for another multinational corporation), it seems to me many of the exploits that Microsoft 'finds' in their products are emailed to them. For those who have not heard of the standard story before:
1. Regular Joe the geek finds an exploit on a product
1a. If Joe is malicious, he will exploit this right away. However, this seems to happen very rarely as most people have at least a bit of a conscious.
3. If Joe is not malicious, he will keep the exploit to himself, do some research, and email the details to the company.
4. Joe waits for a reply and a fix. Since the exploit seems so serious, he is at least expecting a fix within a month or so.
5. Joe waits one or two months, without getting much more than a standard response.
6. Joe grows tired of waiting. He will start surfing to see how other people deal with these issues
7. Joe publishes hints of the vulnerability. If he leaves a good trace back to himself, the company or the law might get involved.
7a. Some script kiddie or someone else might pick up on the vulnerability and start exploiting it.
7b. The company might actually have a fix ready by now (6 months to two years later).
7c. Nothing happens.
8. Joe either is thrown in jail, sued to extinction, or forgets about the whole thing.
Any way you stick it, Joe is trying to do the company a favour, but only can get crap in return.
From the company perspective, it is certainly possible to predict exploits if the company just gathers all the information available to them.

Posting anonymously because I know too many "Joe"'s.

40% Is good when ... (0)

Anonymous Coward | more than 5 years ago | (#25759357)

...predicting exploitation of vulnerabilities.

This studies which vulnerabilities should be prioritised for patching. It is not studying which code will have a vulnerability as some here seem to think.

I use Linux anyway.

How they work it out (0)

Anonymous Coward | more than 5 years ago | (#25759381)

Is this code part of Microsoft's code base?

Yes - it is exploitable
No - it is probably not exploitable*

* This accounts for the 60%

Mickeysoft = Dumbass (0)

Anonymous Coward | more than 5 years ago | (#25759437)

Microsoft is ignorant.

If hackers can exploit source code THEY HAVEN'T SEEN, wait until more real source is revealed.

Microsoft is an ignorant company sitting on top of a shell of a business.

The Sex Panther prediction method (0)

Anonymous Coward | more than 5 years ago | (#25759445)

60% of the time it's wrong, every time.

Woah, it works forty percent of the time? (0)

Anonymous Coward | more than 5 years ago | (#25759485)

That means that forty percent of the time, it works every time. [wikiquote.org]

Sorry guys, I know it's off topic, but I couldn't help it. :)

This is what it's come to? (1)

joedoc (441972) | more than 5 years ago | (#25759511)

Microsoft is now bragging about the fact that they predicted 40% of their bugs would be turned into exploits?
I realize that Windows is a complex hunk of crap...errr...operating system, but wouldn't they be better served trying to find and correct these issues rather then just releasing them into the wild and keeping their fingers crossed?
Their attitude is sort of like pointing the gun at your foot and firing five times, and bragging that you only hit two of your toes.
This is why, every day when I arrive at work, I log into this XP box and ask myself why my organization continues to put up with this garbage.

Flip a coin (0)

Joebert (946227) | more than 5 years ago | (#25759527)

I've decided to start my own Exploitability Index & my main selling point will be that I will be right 50% of the time compared to Microsofts mere 40%.

Why should I care? (0)

Anonymous Coward | more than 5 years ago | (#25759595)

So microsoft predicted these with some accuracy, and used this to give priorities to it's developers.

Jolly good, but apart from perhaps knowing that microsoft is trying to do a good work, why should I care about the process? This is a microsoft internal thing.

I can't bear anymore stupid executives bullying to the world about their stupid internal work. Do your homework and shut up. I really don't care how they do their work, just do it.

P.S. in microsoft developers are definetly not doing good work...Marketing instead has always been doing a great work there.

Toss (1)

ezwip (974076) | more than 5 years ago | (#25759743)

Kudos to Microsoft for choosing to toss the coin rather than to spin it. My research team argued for months with the board that spinning a penny instead of tossing it results in heads only about 30% of the time.

What do ya mean 40 percent? (2, Insightful)

WheelDweller (108946) | more than 5 years ago | (#25759767)

Has there ever been a Microsoft bug that hackers have left alone?

We've been through this 'a million times' since DOS; there are literally more than a million active viruses out there, with another 100,000 per month. 40 percent chance of an exploit being used seems kinda low, doesn't it?

Are they serious? (1)

abcjared (1126307) | more than 5 years ago | (#25759867)

Predicting something will change the outcome in this case.
Not only will hackers know the most exploitable bug but they will also know the least likely to be updated.
Yet another bloat to innocent users.
FFS can someone make them a feature usefulness vs crap predictor..

Worse than random (1)

QuietLagoon (813062) | more than 5 years ago | (#25759947)

So let me get this correct. Microsoft's determination whether or not there would be an exploit was correct less frequently than if they had just randomly chosen yes or no, and Microsoft calls that good performance?

With such low standards of good performance, it is no wonder that the software coming out of Redmond lately has been so horribly poor.

News at 11 (1)

Henry V .009 (518000) | more than 5 years ago | (#25760019)

Hackers are missing 60% of opportunities to exploit Microsoft code.

More fail from MS (2, Insightful)

foldingstock (945985) | more than 5 years ago | (#25760179)

They can predict exploits in their own software. Well paint me yellow and call me a phone directory!

How can a PR team for one of the largest corporations in the US seriously release a statement like this? What kind of company fails so badly that they can only predict 40% of exploits in their own [proprietary] software?

If a major car (or car part) manufacturer "accurately" predicted that 40% of their automobiles would explode and burn their owners alive due to a fuel system defect....would people still buy their cars? Oh right...firestone.
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>