Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Internet Explorer 9 Caught Cheating In SunSpider

CmdrTaco posted more than 3 years ago | from the well-that's-not-nice dept.

Microsoft 360

dkd903 writes "A Mozilla engineer has uncovered something embarrassing for Microsoft – Internet Explorer is cheating in the SunSpider Benchmark. The SunSpider, although developed by Apple, has nowadays become a very popular choice of benchmark for the JavaScript engines of browsers."

cancel ×

360 comments

Sorry! There are no comments related to the filter you selected.

Embarassing? (2, Funny)

intellitech (1912116) | more than 3 years ago | (#34253450)

I would think Microsoft would be used to embarassing by now..

Re:Embarassing? (1, Insightful)

Pojut (1027544) | more than 3 years ago | (#34253468)

They're kinda like the rich fat cat who constantly puts his foot in his mouth. He knows he should shut up, but then again why should he care...he's rich, bitch!

Re:Embarassing? (0)

commodore64_love (1445365) | more than 3 years ago | (#34253678)

Not for long . . . I've been hearing that MS is losing tons of money and heading towards IBM territory.

Re:Embarassing? (1)

eleuthero (812560) | more than 3 years ago | (#34253746)

um... I'll admit they've not been doing as well as they used to, but their stock [yahoo.com] would seem to indicate that they are not losing money. Individual divisions might be (they did get in the news yesterday for playing games with sales figures in one division--but not in a way that the money didn't come from somewhere in the company), but as a whole, their P/E is better than Apple's (even if they do have a lower market cap).

Re:Embarassing? (2, Informative)

nedwidek (98930) | more than 3 years ago | (#34254008)

And their 10-Q definitely indicates that they're not losing money. http://biz.yahoo.com/e/101028/msft10-q.html [yahoo.com]

The stock price is a meaningless indicator unless you are using it indirectly with their P/E ratio.

Re:Embarassing? (2, Informative)

poetmatt (793785) | more than 3 years ago | (#34254094)

Re:Embarassing? (1, Interesting)

eleuthero (812560) | more than 3 years ago | (#34254184)

87 million shares is only a quarter of what he owns in the company (I know, he has a ridiculous amount of money in MSFT stock). Is it a lot? probably so, but he owns the third highest number of MSFT shares (second highest individual owner) and is unlikely to want to rock the boat. As a case in point, his announcement to sell was not something that did much to the price of the shares over the long term. People are unconcerned. Further, selling from time to time is expected - if the company is doing badly, he might sell, but if it is doing well, he will sell some shares for precisely the reason he has claimed--diversity. Gates owns only ~15 billion dollars' worth of MSFT shares and yet has a net worth over 40 billion.

Re:Embarassing? (1)

poetmatt (793785) | more than 3 years ago | (#34254212)

please. it's almost trivial to sell shares in a company that puts anything above $1b in revenue a year, even if they're on the verge of bankruptcy.

Re:Embarassing? (0)

Anonymous Coward | more than 3 years ago | (#34254146)

yeah, because we all know that stock price is a perfect reflection of business performance and has nothing to do with public opinion :rolleyes:

Re:Embarassing? (1)

Attila Dimedici (1036002) | more than 3 years ago | (#34253910)

Not for long . . . I've been hearing that MS is losing tons of money and heading towards IBM territory.

Why would heading towards "IBM territory" be a bad thing? IBM is a profitable, successful company that has been around for longer than most companies of its size.

Re:Embarassing? (1)

Nutria (679911) | more than 3 years ago | (#34253932)

Libertarian==fiscally-conservative/socially-liberal

How can one be fiscally conservative (spend less money) and simultaneously be socially liberal (spend more money on those sad, pitiful poor people)?

Re:Embarassing? (1, Interesting)

poetmatt (793785) | more than 3 years ago | (#34254012)

anyone who says libertarian doesn't understand the conflict in the term. it's basically republicans who don't want to be called republicans or someone who hates both parties but still leans republican. if someone declares themselves independent libertarian, then they're more acknowledging that they don't necessarily align with "libertarian" views. I have a friend like this, and it's basically republican but he doesnt' want to admit it.

Re:Embarassing? (3, Informative)

commodore64_love (1445365) | more than 3 years ago | (#34254040)

Socially liberally means allowing people to do whatever they want. In their bedrooms, in their homes, in their personal lives, so long as their actions don't physically harm another.

It does Not mean politically-or-fiscally liberal (using gov't to steal money from workers and redistribute it).

Re:Embarassing? (1)

nschubach (922175) | more than 3 years ago | (#34254050)

We're not... true libertarians (small "L") believe people should be responsible for themselves and their own actions. Fiscal conservatism is a means to the end... where people can pay their own bills and take care of their own kids.

Re:Embarassing? (1)

BubbaDave (1352535) | more than 3 years ago | (#34254084)

Libertarian==fiscally-conservative/socially-liberal

How can one be fiscally conservative (spend less money) and simultaneously be socially liberal (spend more money on those sad, pitiful poor people)?

Socially liberal does not mean "spend money".

Dave

Re:Embarassing? (1)

k_187 (61692) | more than 3 years ago | (#34254148)

I usually self-identify as libertarian and describe my politics as "I believe in tax cuts and gay marriage". Although I will admit that as time passes I'm moving toward the American left on the tax cut side.

Re:Embarassing? (0)

Anonymous Coward | more than 3 years ago | (#34253730)

They're called Bidenisms [slate.com]

Re:Embarassing? (1, Insightful)

Anonymous Coward | more than 3 years ago | (#34253516)

I would think Microsoft would be used to embarassing by now..

Embarassment requires a sense of shame.

Re:Embarassing? (0, Troll)

Sponge Bath (413667) | more than 3 years ago | (#34253680)

Why would Microsoft be embarrassed by business as usual?

Re:Embarrassing? (1)

SimonTheSoundMan (1012395) | more than 3 years ago | (#34253690)

Embarrassing the article got slashdotted. Try the web site on port 8090.

http://digitizor.com.nyud.net:8090/2010/11/17/internet-explorer-9-caught-cheating-in-sunspider-benchmark/ [nyud.net]

Also embarrassing that you spelt "embarrassing" incorrectly. ;)

Re:Embarrassing? (1)

SimonTheSoundMan (1012395) | more than 3 years ago | (#34253700)

Embarrassing, it's CCed at nyud.net.

Re:Embarassing? (4, Informative)

Mushdot (943219) | more than 3 years ago | (#34253714)

Another misleading tabloid headline from Taco et al.

Short story: Someone notices a perhaps too-fast result for a particular benchmark test with IE 9 and modifies the benchmark code which then throws IE 9 performance the other way. One *possible* conclusion is that MS have done some sort of hardcoding/optimisation for this test, which has been thrown out by the modifications.

Re:Embarassing? (3, Insightful)

BLKMGK (34057) | more than 3 years ago | (#34253846)

Thanks for someone pointing this out. I mean really, if they were going to throw this test why would they throw it quite this much? And is this the ONLY portion of this test that seems to act this way? If so then why in the world would they throw only this portion and why this much? The original result was uber fast, the result on the modified test pretty slow - if they were going to try and hide something why make it uber fast and not just slightly better?

Something is weird, possibly hinky, but to outright declare cheating based just on this? Really? O_o

Re:Embarassing? (1)

Richard_at_work (517087) | more than 3 years ago | (#34253884)

I distinctly remember this being heavily discussed in the last few IE9 benchmark stories as well, so its not new and its not necessarily cheating.

Re:Embarassing? (5, Insightful)

Anonymous Coward | more than 3 years ago | (#34254178)

Optimisations done purely for use only on a benchmark to achieve far better results than normal is the exact definition of cheating. Benchmarks are meant to test the browser with some form of real performance measure and not how good the programmers are at making the browser pass that one test. If the thing is getting thrown off by some very simple instructions to the tune of 20 times longer then it is seriously broken. Optimization or not.

It is like when ATI/Nvidia made their drivers do some funky shit on the benchmarks to make their products seem way better; This was also called cheating at the time.

Microsoft cheating? (0)

Anonymous Coward | more than 3 years ago | (#34253452)

Inconceivable!

I'm sure there's no hyperbole in this article (2, Insightful)

Anonymous Coward | more than 3 years ago | (#34253454)

No, none what-so-ever.

Welcome to your daily two minutes hate, Slashdot.

Re:I'm sure there's no hyperbole in this article (1)

stewbacca (1033764) | more than 3 years ago | (#34253552)

*although it was developed by Apple*

Doesn't this only make sense if there is reason to suspect Apple doesn't develop things well?

Re:I'm sure there's no hyperbole in this article (1)

king neckbeard (1801738) | more than 3 years ago | (#34253654)

No, it's just stating that it has basically become a standard benchmark despite Apple not being a standards organization.

Re:I'm sure there's no hyperbole in this article (1)

beelsebob (529313) | more than 3 years ago | (#34253772)

But then, that's how most things become standards

A, who is not a standards organisation develops it.
B, who is also not a standards organisation uses it.
If sufficient numbers of Bs use it, it becomes a de-facto standard
Sometimes C, who is a standards organisation says it's a standard.
Then it becomes a de-jure standard.

Re:I'm sure there's no hyperbole in this article (1, Insightful)

Anonymous Coward | more than 3 years ago | (#34253590)

Why don't you try reading it before you make that claim? The article is a few simple benchmark results and mild speculation as to what caused them. The summary may be inflammatory, the article goes out of its way not to be.

Old news... (-1, Offtopic)

Anonymous Coward | more than 3 years ago | (#34253458)

This is very old news - I remember reading this via Planet Mozilla weeks ago...

Re:Old news... (3, Insightful)

Gadget_Guy (627405) | more than 3 years ago | (#34253952)

It is actually a couple of months old [mozilla.com] . The thing that makes me doubt the claims of cheating is that nobody has been able to find other examples of performance variations in this benchmark in all the time since this came to light. If they were going to cheat, why limit it to the cordic test? Nobody would base their browser choice on this obscure test.

I don't have the beta installed yet, but what I would like to see is the actual calculation changed and then run the tests again. Don't just put in weird code like "true;" but make the javascript plausible. It could be that the addition of these unusual statements are enough to confuse the optimiser so that it resorts back to a completely unoptimised version.

Benchmarks (4, Insightful)

Mongoose Disciple (722373) | more than 3 years ago | (#34253490)

This is the nature of benchmarks... whenever people start caring about them enough, software/hardware designers optimize for the benchmark.

Next we're going to be shocked that 8th grade history students try to memorize the material they think will be on their test rather than seeking a deep and insightful mastery of the subject and its modern societal implications.

Re:Benchmarks (2, Insightful)

Lunix Nutcase (1092239) | more than 3 years ago | (#34253508)

This is the nature of benchmarks... whenever people start caring about them enough, software/hardware designers optimize for the benchmark.

Except that the article writer tries to claim that that couldn't possibly be the case and thus claims that Microsoft is "cheating" instead. Basically this is an invented controversy.

Re:Benchmarks (1, Troll)

Rockoon (1252108) | more than 3 years ago | (#34253716)

Fear not, Slashdot, for I have read the fucking article!

The unnamed "Mozilla Engineer" didn't even bother adding his 'true' and "return' to non-sunspider code, but has the balls to declare that the only possible reason for the negative performance when added to the sunspider code is that Microsofts JavaScript engine is cheating.

Now, maybe Microsoft is cheating here and maybe they arent, but in this article the conclusion obviously preceded the result, that this unnamed mozilla hack didnt even do any of the extremely simple things that might nullify his hypothesis.

Re:Benchmarks (5, Informative)

Nutria (679911) | more than 3 years ago | (#34254096)

Fear not, for I have RTFA and the original article that the digitizor article is based on.

Fortunately for the ethics of Mozilla, the named Mozilla engineer (Rob Sayre) never claimed that IE9 cheated. Instead, he diplomatically refers to it as a "oddity" and "fragile analysis" and filed a bug w/ MSFT.

http://blog.mozilla.com/rob-sayre/2010/09/09/js-benchmarks-closing-in/M [mozilla.com]
http://blog.mozilla.com/rob-sayre/2010/11/16/reporting-a-bug-on-a-fragile-analysis/ [mozilla.com]

So, blame Digitizor and ycombinator for putting words in Rob Sayre's mouth.

Re:Benchmarks (1, Insightful)

hedwards (940851) | more than 3 years ago | (#34254122)

Most likely they are cheating. The other possibilities are far less plausible. Even if you discount that possibility, either they're not competent at optimization or they're not competent at writing a robust engine.

In none of the cases is MS doing something legitimately. Optimizing to one test is invariably a bad idea, no matter how well designed, and quite honest at this point they should be able to code an engine that's a lot more resilient than that.

Re:Benchmarks (0)

Anonymous Coward | more than 3 years ago | (#34253524)

Optimize != cheat

Re:Benchmarks (3, Informative)

Lunix Nutcase (1092239) | more than 3 years ago | (#34253568)

And there is no actual evidence that they are actually cheating. The article writer puts forth 2 other highly likely situations but then dismisses them for no good reason for the "cheating".

Re:Benchmarks (0)

Anonymous Coward | more than 3 years ago | (#34254214)

And there is no actual evidence that they are actually cheating. The article writer puts forth 2 other highly likely situations but then dismisses them for no good reason for the "cheating".

Yeah, we just have to take everything that monopoly says as true. Like moving Office to open standards, preferring HTML5 for SL and so on...
Somehow I have always trusted more on companies which compete side by side for my attention and do not have OS-monopoly to protect.

Really? You're Going with That? (5, Insightful)

eldavojohn (898314) | more than 3 years ago | (#34253544)

Next we're going to be shocked that 8th grade history students try to memorize the material they think will be on their test rather than seeking a deep and insightful mastery of the subject and its modern societal implications.

Some things to consider: 1) I'm not doing business with the 8th grader. Nor am I relying on his understanding and memorization of history to run Javascript that I write for clients. 2) You are giving Microsoft a pass by building an analogy between their javascript engine and an 8th grade history student.

Just something to consider when you say we shouldn't be shocked by this.

Re:Really? You're Going with That? (1)

Sockatume (732728) | more than 3 years ago | (#34253864)

2) You are giving Microsoft a pass by building an analogy between their javascript engine and an 8th grade history student.

Indeed. The student would make a better Javascript engine.

MS decision makers ARE 8th grade (0)

Anonymous Coward | more than 3 years ago | (#34253868)

MS decision makers ARE 8th grade-like.

Win at any cost, even if you have to give all your software away for people to choose it!

Re:Benchmarks (1)

QuietLagoon (813062) | more than 3 years ago | (#34253548)

This is the nature of benchmarks... whenever people start caring about them enough, software/hardware designers optimize for the benchmark.

It shows that Microsoft is more concerned about getting a good score on the benchmark than they are about providing a good customer experience.

Re:Benchmarks (1)

Jugalator (259273) | more than 3 years ago | (#34253578)

It shows that Microsoft is more concerned about getting a good score on the benchmark than they are about providing a good customer experience.

Could the same be said about the numerous bugs issued for Firefox about optimizing TraceMonkey's SunSpider performance?

Re:Benchmarks (1)

hedwards (940851) | more than 3 years ago | (#34254174)

No it couldn't. Firefox has for a long time lagged on pretty much all the tests, including that stupid ACID test. They lagged specifically because they were more focused on real improvements over faking it or optimizing for conditions that one is unlikely to encounter.

Or, it could be that they're just incredibly incompetent at cheating. I suppose that's possible. But given the degree to which the real speed has improved with the 4.0b7, I think we can largely rule out that level of incompetence.

Re:Benchmarks (1)

aliquis (678370) | more than 3 years ago | (#34253726)

Many customers make the benchmark result their experience of the soft- or hardware.

Re:Benchmarks (1, Insightful)

Mongoose Disciple (722373) | more than 3 years ago | (#34253780)

It shows that Microsoft is more concerned about getting a good score on the benchmark than they are about providing a good customer experience.

For that to be true, you'll need to demonstrate that they put more effort into scoring well on the benchmark than they did in improving performance in general. I don't think you can.

Improving performance in general is worth doing and I'm sure it's being done, but it's hard. Improving performance on a benchmark dramatically is often not that hard, and it's worth doing if it gets your product noticed.

I'm sure all browser makers are doing the exact same thing on both counts -- anonymous Mozilla guy is just bitter because he did a shittier job of the less-important task of benchmark performance.

Re:Benchmarks (0)

Anonymous Coward | more than 3 years ago | (#34253566)

Why not create a benchmark which boots the operating system and launches a web browser. Then I wouldn't care if they cheat!

Re:Benchmarks (0)

Anonymous Coward | more than 3 years ago | (#34253748)

They did, its called ChromeOS ;)

Re:Benchmarks (1)

gl4ss (559668) | more than 3 years ago | (#34253650)

read the article. their js performance is quite suspect if their results are "too good to be true" when the benchmark is unmodified and then too bad to be true when it's very slightly modified. some more 3rd party testing should be done.. and actually it would be pretty easy to do.

Re:Benchmarks (1)

hey (83763) | more than 3 years ago | (#34253704)

Shows a problem with benchmarks in general. Too easy to game.

Re:Benchmarks (1)

99BottlesOfBeerInMyF (813746) | more than 3 years ago | (#34253744)

Shows a problem with benchmarks in general. Too easy to game.

Benchmarks are great, for improving the performance of your code. Benchmarks are terrible, as soon as they start to get press and companies try to deceive users by gaming them. That's why it is important that we call out when they are caught so they get more bad press and maybe think twice about gaming the benchmark in the first place.

Re:Benchmarks (1)

hedwards (940851) | more than 3 years ago | (#34254204)

That's not a problem with benchmarks per se, that's a problem with the idiots that insist that benchmark performance is the same thing as good performance in general.

It really depends how the benchmark is set up, certain things are known to be costly in terms of IO, RAM and processing time. And a benchmark which measures things like that and gives some meaningful indication where the time is being spent is definitely valuable.

Re:Benchmarks (1)

aliquis (678370) | more than 3 years ago | (#34253664)

They also have one more thing in common.

They're both utterly useless ;)

Nostalgic purpose / dreams / cool numbers, yeah, but what matter is today, and actual perceived performance.

Re:Benchmarks (1)

Mongoose Disciple (722373) | more than 3 years ago | (#34253728)

Nostalgic purpose / dreams / cool numbers, yeah, but what matter is today, and actual perceived performance.

Absolutely.

Except in the sense that you can get a lot of good press / free advertising by stomping a mudhole in the other guy's performance in a benchmark. There's a clear incentive to improve your actual performance, because when real people get ahold of your benchmarked piece of software/hardware/whatever they're going to notice that actual performance -- but there's also some incentive to improve you benchmark performance for cheap advertising. The former is more valuable than the latter, and I'm sure it gets more time/money/effort proportionately.

Re:Benchmarks (5, Informative)

TheRaven64 (641858) | more than 3 years ago | (#34253862)

There is a difference between optimising for a benchmark and cheating at a benchmark. Optimising for a benchmark means looking at the patterns that are in a benchmark and ensuring that these generate good code. This is generally beneficial, because a well-constructed benchmark is representative of the kind of code that people will run, so optimising for the benchmark means that common cases in real code will be optimised too. I do this, and I assume that most other compiler writers do the same. Cheating at a benchmark means spotting code in a benchmark and returning a special case.

For example, if someone is running a recursive Fibonacci implementation as a benchmark, a valid optimisation would be noting that the function has no side effects and automatically memoising it. This would turn it into a linear time, rather than polynomial time, function, at the cost of increased memory usage. A cheating optimisation would be to recognise that it's the Fibonacci sequence benchmark and replaces it with one that's precalculated the return values. The cheat would be a lot faster, but it would be a special case for that specific benchmark and would have no impact on any other code - it's cheating because you're not really using the compiler at all, you're hand-cmpiling that specific case, which is an approach that doesn't scale.

The Mozilla engineer is claiming that this is an example of cheating because trivial changes to the code (adding an explicit return; at the end, and adding a line saying true;) both make the benchmark much slower. I'm inclined to agree. The true; line is a bit difficult - an optimiser should be stripping that out, but it's possible that it's generating an on-stack reference to the true singleton, which might mess up some data alignment. The explicit return is more obvious - that ought to be generating exactly the same AST as the version with an implicit return.

That said, fair benchmarks are incredibly hard to write for modern computers. I've got a couple of benchmarks that show my Smalltalk compiler is significantly faster than GCC-compiled C. If you look at the instruction streams generated by the two, this shouldn't be the case, but due to some interaction with the cache the more complex code runs faster than the simpler code. Modify either the Smalltalk or C versions very slightly and this advantage vanishes and the results return to something more plausible. There are lots of optimisations that you can do with JavaScript that have a massive performance impact, but need some quite complex heuristics to decide where to apply them. A fairly simple change to a program can quite easily make it fall through the optimiser's pattern matching engine and run in the slow path.

Re:Benchmarks (0)

Anonymous Coward | more than 3 years ago | (#34254166)

Seriously, how have you not been modded up? I'm a bit better for having read this comment.

I wish... (1)

joe2tiger (1883232) | more than 3 years ago | (#34253502)

I could still use Internet Explorer 8. I have 9 right now, but certain websites that my job requires me to go to locks me out of some content. Autotask.net and some other sites that my school has access to is nonfunctional. I know most pages comply, but the few that don't - why are we forced to use Microsoft software?

Level playing field (0, Troll)

QuietLagoon (813062) | more than 3 years ago | (#34253520)

Microsoft knows that they cannot win on a level playing field. Microsoft knows they are unable to compete without having their finger on the scale. So they cheat.

Do not attribute to malice ... (5, Insightful)

Tar-Alcarin (1325441) | more than 3 years ago | (#34253522)

what can be attributed to stupidity.

1) Microsoft cheated by optimizing Internet Explorer 9 solely to ace the SunSpider Bechmark. To me, this seems like the best explanation.
2)Microsoft engineers working on Internet Explorer 9 could have been using the SunSpider Benchmark and unintentionally over-optimized the JavaScript engine for the SunSpider Benchmark. This seems very unlikely to me.

I see no reason why explanation number one is more likely than explanation number two.

Re:Do not attribute to malice ... (1)

Spad (470073) | more than 3 years ago | (#34253564)

Only because you could argue that Microsoft has a vested interest in doing #1 - I guess it depends on how malicious you think Microsoft is :)

Re:Do not attribute to malice ... (0, Troll)

Lunix Nutcase (1092239) | more than 3 years ago | (#34253572)

Because explanation number 2 doesn't feed into the hatred of Microsoft and create all sorts of controversy in order to drive more ad revenue?

Re:Do not attribute to malice ... (2, Insightful)

The MAZZTer (911996) | more than 3 years ago | (#34253610)

Let's check out some other benchmarks/parts of Sunspider IE9 does good on and tweak them similarly to see if the performance suddenly suffers.

Re:Do not attribute to malice ... (3, Insightful)

MobyDisk (75490) | more than 3 years ago | (#34253710)

Re:Do not attribute to malice ... (1, Troll)

Tar-Alcarin (1325441) | more than 3 years ago | (#34253812)

Are you calling me stupid? Because I'm sure you're not implying that I misquoted that on purpose.

Re:Do not attribute to malice ... (0, Offtopic)

Tar-Alcarin (1325441) | more than 3 years ago | (#34254144)

Troll? Seriously?
Come on! It was a direct reference to the nature of the quote being discussed.
I would have thought the ";)" was implied.

Re:Do not attribute to malice ... (1)

Inda (580031) | more than 3 years ago | (#34253834)

Accuse someone of something when phishing for information. Watch the reactions, watch people back-peddling, listen for lies, listen for an overly reactive explaination, watch for the ultra-defensive, nose scratching, bullshitters, beads of sweat...

Does no on else use this trick in life? I doubt I've invented it; I'm sure it's taught somewhere and there's probably a fancy name for it.

Accuse Microsoft of cheating and see what information flows back.

Re:Do not attribute to malice ... (2, Insightful)

LordKronos (470910) | more than 3 years ago | (#34253900)

I see no reason why explanation number one is more likely than explanation number two.

I do. Given the nature of the changes that were used to uncover this, to me (as a programmer) it seems very unlikely that such over-optimization could happen in such a way that it would degrade so severely with those changes. Here is what was changed (look at the 2 diff files linked near the bottom of the article):

1) A "true;" statement was added into the code. It was not an assignment or a function call, or anything complex. Just a simple true statement. Depending on the level of optimization by the interpreter/compiler, this should turn into a noop at best, or simply loading a constant into a register or memory location (and then doing nothing more with it) at worst..
2) A "return" added at the end of a function, instead of using the implicit return. This also should have minimal impact. In the best case, the code generated should be identical to the return code already generated by the implicit return. In the worst case, the compiler might be sloppy and generate 2 return instructions (one for the explicit return, and another unreachable instruction for the implicit return).

So from my experience, it seems EXTREMELY unlikely that this could have happened by accident just through optimization. I'm not going to go so far as to say it CAN'T happen, because I know from experience that things can get really complex and things not so obvious can happen (like cache issues as mentioned in the article). However, in all likelihood, I suspect this is going to be just like when video card drivers detect the filename of the executable and optimize specifically for that (I think it was quake3.exe that we saw that happen with before).

Re:Do not attribute to malice ... (1)

sbates (1832606) | more than 3 years ago | (#34254004)

The second example--a general optimization of the engine that over-optimizes a corner case--is very difficult to pull off in these engines. When you consider that the over-optimization only occurs when the source is aligned a certain way things become very suspicious, because the engine isn't running against the source but an abstract representation of the code.

In today's engines there are also routines to strip out unnecessary and unreachable code, which is relevant because the code snippets added by the tester were extraneous and (slashdotted, so going from memory) unreachable. This means the bytecode would either have been generated and stripped or not generated at all.

Then consider that these engines are not actually running the bytecode but machine-optimizing it. So now you have a case where:

1. Extraneous and unreachable code is added
2. Extraneous and unreachable code is removed by the compiler to bytecode
3. The bytecode is further optimized to machine code
4. The code is executed

Step 4 is where the optimization is lost. This is why it's extremely unlikely that someone checked in code to attempt to optimize the engine which resulted in an over-optimization of the corner case.

Microsoft is irrelevant (0)

Anonymous Coward | more than 3 years ago | (#34253538)

Oh please. Don't overreact. Microsoft is quickly becoming irrelevant. Google and Facebook are the new borg that we should be worried about.

Re:Microsoft is irrelevant (2, Funny)

Rosco P. Coltrane (209368) | more than 3 years ago | (#34253638)

AC is right on the money there. Open-source software has come such a long way that Microsoft products and business are entirely avoidable these days, and therefore are no longer a threat. Google is the true danger of the age because they're fast on the way to make off-line applications obsolete altogether and render the open-source vs. closed source debate moot, as we'll have to swallow their online applications shenanigans without being able to do a thing about it.

Real-world usability (1)

Rosco P. Coltrane (209368) | more than 3 years ago | (#34253554)

Benchmarks are very nice and all, but in the end, users using different browsers for real should decide which *feels* faster or better (which isn't the same as being faster or better). If real-world users can't feel the difference, then benchmarks are just there for masturbation value, and quite frankly, on reasonably modern hardware, I've never felt any true difference in rendering speed between the various "big" browsers out there.

I reckon the only thing that truly matters is the speed at which a browser starts up without prefetching (another semi-dishonest technique used by Microsoft for years to make users believe their products start so much faster than the competition incidentally).

Three explanations FTFA (5, Insightful)

davev2.0 (1873518) | more than 3 years ago | (#34253584)

There are three possible explanation for this weird result from Internet Explorer:

Microsoft cheated by optimizing Internet Explorer 9 solely to ace the SunSpider Bechmark. To me, this seems like the best explanation.
Microsoft engineers working on Internet Explorer 9 could have been using the SunSpider Benchmark and unintentionally over-optimized the JavaScript engine for the SunSpider Benchmark. This seems very unlikely to me.
A third option (suggested in Hacker News) might be that this is an actual bug and adding these trivial codes disaligns cache tables and such throwing off the performance entirely. If this is the reason, it raises a serious question about the robustness of the engine.

Everything in italics is unsupported opinion by the author, yet is treated as fact in the summary and title by CmdrTaco and Slashdot. Perhaps if Slashdot would stick to actual news sites (you know NEWS for nerds and all that), this would be a balanced report with a good amount of information. Instead, it is just another Slashdot supported hit piece against MicroSoft.

Re:Three explanations FTFA (1)

gl4ss (559668) | more than 3 years ago | (#34253672)

it's just speculation on the possible reasons why it happened. should he have waited for the first replies to his post to post the replies to those obvious replies? of course not.

now if you want, you could run the benches yourself - and this is what the blogger wants you to do.

Re:Three explanations FTFA (4, Interesting)

davev2.0 (1873518) | more than 3 years ago | (#34254088)

So, instead the blogger should declare that MS cheated at the benchmarks with nothing more than his results for which he admits that there are at least three plausible explanations?

And, then Taco should treat the author's biased opinion as fact? Remember, the title of this post is "Internet Explorer 9 Caught Cheating in SunSpider."

I don't think so.

And, where is the response from MS? Did anyone ask MS, or did someone find this and go "MS is CHEATING!!11!!one!" without actually investigating or even asking MS? Because, it really looks like the latter, which would make this just more MS bashing blogspam.

Re:Three explanations FTFA (0)

Anonymous Coward | more than 3 years ago | (#34253978)

You must be new here.

No proof? (4, Informative)

1000101 (584896) | more than 3 years ago | (#34253596)

FTFA:

There are three possible explanation for this weird result from Internet Explorer:

1. Microsoft cheated by optimizing Internet Explorer 9 solely to ace the SunSpider Bechmark. To me, this seems like the best explanation.
2. Microsoft engineers working on Internet Explorer 9 could have been using the SunSpider Benchmark and unintentionally over-optimized the JavaScript engine for the SunSpider Benchmark. This seems very unlikely to me.
3. A third option (suggested in Hacker News) might be that this is an actual bug and adding these trivial codes disaligns cache tables and such throwing off the performance entirely. If this is the reason, it raises a serious question about the robustness of the engine.


I'm not saying if what they have done is right or wrong, but this is a sensationalist headline that offers two other "less evil" alternatives to the outcome.

Re:No proof? (1)

99BottlesOfBeerInMyF (813746) | more than 3 years ago | (#34253712)

Headlines are supposed to be succinct summaries and that is enforced by the character limit here. Maybe a better headline would be "Internet Explorer 9 Probably Cheating On Sunspider, But Maybe Just Horribly Written In Ways That Make SunSpider Apply Poorly". Of course that is too long for the title.

The important take away is that a particular SunSpider test is not a valid test for IE 9's performance in that category and that IE 9 will do much, much worse in many real world scenarios. The likelihood is that this is because Microsoft cares more about SunSpider than actual performance (marketing versus functionality) but it is vaguely possible it is the result of really crappy coding that just happens to have the same result.

Re:No proof? (0)

Anonymous Coward | more than 3 years ago | (#34254202)

There are three possible explanation for this weird result from Internet Explorer:

1. Microsoft cheated by optimizing Internet Explorer 9 solely to ace the SunSpider Bechmark. To me, this seems like the best explanation.
2. Microsoft engineers working on Internet Explorer 9 could have been using the SunSpider Benchmark and unintentionally over-optimized the JavaScript engine for the SunSpider Benchmark. This seems very unlikely to me.
3. A third option (suggested in Hacker News) might be that this is an actual bug and adding these trivial codes disaligns cache tables and such throwing off the performance entirely. If this is the reason, it raises a serious question about the robustness of the engine.

I'm not saying if what they have done is right or wrong, but this is a sensationalist headline that offers two other "less evil" alternatives to the outcome.

Frankly, the original article isn't much better. 2 and 3 sit there as weasel words to cover the author's back while he insists that 1 is "the best explanation".

Cheating allegation too strong (3, Insightful)

js3 (319268) | more than 3 years ago | (#34253606)

Meh I think claiming they are cheating with no evidence seems a little too out there. I've never seen MS brag about how fast their browser is on this particular benchmark, and frankly seems more like a bug than a cheat.

Re:Cheating allegation too strong (4, Informative)

king neckbeard (1801738) | more than 3 years ago | (#34253696)

They have shown their Sunspider results quite a few times on http://blogs.msdn.com/b/ie/ [msdn.com]

mod parent up (1)

markdowling (448297) | more than 3 years ago | (#34253930)

While MS-IE have disclosed a lot of information lately on their blogs, if they're going to discuss Sunspider results (as they did on 28 October with the IE9PP6 tests [msdn.com] ) then use of sleight of hand to sex them up is fair game for criticism.

Re:Cheating allegation too strong (1)

coofercat (719737) | more than 3 years ago | (#34253808)

I don't disagree with you, but I'd love your "bug" idea to become mainstream...

"Critical bug in Internet Explorer boosts performance by 10%"

"Microsoft keen to develop more bugs in the hope they boost performance in other products"

"Mozilla unavailable for comment on how come they persist in reducing bugs when everyone else seems to want more of them"

Re:Cheating allegation too strong (0)

Anonymous Coward | more than 3 years ago | (#34253876)

If bugs produce this kind of benchmark winning performance, then apparently what Microsoft needs is more bugs. I'm shutting off Windows Update.

Broken Link (0)

Anonymous Coward | more than 3 years ago | (#34253658)

Here's the google cache:

http://webcache.googleusercontent.com/search?q=cache:http://digitizor.com/2010/11/17/internet-explorer-9-caught-cheating-in-sunspider-benchmark/

Im shocked! (0, Troll)

miffo.swe (547642) | more than 3 years ago | (#34253662)

Microsoft, cheating? Umpossible!

They ALWAYS cheat because its not in their organisational capacity to deliver something good. They deliver mediocre and then warp the surrounding world into their own desired image.

3 possible explanations, so why accuse? (3, Interesting)

digitaldc (879047) | more than 3 years ago | (#34253670)

The article clearly states:
There are three possible explanation for this weird result from Internet Explorer:
1. Microsoft cheated by optimizing Internet Explorer 9 solely to ace the SunSpider Bechmark. To me, this seems like the best explanation.
2. Microsoft engineers working on Internet Explorer 9 could have been using the SunSpider Benchmark and unintentionally over-optimized the JavaScript engine for the SunSpider Benchmark. This seems very unlikely to me.
3. A third option (suggested in Hacker News) might be that this is an actual bug and adding these trivial codes disaligns cache tables and such throwing off the performance entirely. If this is the reason, it raises a serious question about the robustness of the engine.

So, what proof do we have that Microsoft actually cheated?

Re:3 possible explanations, so why accuse? (1)

jav1231 (539129) | more than 3 years ago | (#34253898)

They're Microsoft. :)

So I click on the link... (1)

Kc_spot (1677970) | more than 3 years ago | (#34253676)

And it leads me to a 500 internal server. Dohohoho, Microsoft.

Re:So I click on the link... (0)

Anonymous Coward | more than 3 years ago | (#34253888)

They've over-optimized their server to survive a Slashdotting by serving up brief error pages instead of full content. Ingenious!

Internet Explorer? (1)

TheBilgeRat (1629569) | more than 3 years ago | (#34253684)

Since when has IE ever been faster? And why bother cheating when you have 51% of the market share?

Browser share from Dec 2009 to Oct 2010 [netmarketshare.com]

Re:Internet Explorer? (1, Insightful)

Anonymous Coward | more than 3 years ago | (#34253794)

> And why bother cheating when you have 51% of the market share?

Because that 51% used to be 91%?

Re:Internet Explorer? (1)

commodore64_love (1445365) | more than 3 years ago | (#34254172)

Mozilla (netscape) used to have 91% share too but I don't see them cheating.

I choose browsers based on features not speed. I like Firefox's addons to enable me to download youtube vids, SeaMonkey's builtin newsgroups/chat/email features, and Opera's Turbo for slow connections (dialup, cellular). As for speed they all seem about the same although FF3.6 does have a memory leak that can be annoying.

Re:Internet Explorer? (1)

black6host (469985) | more than 3 years ago | (#34253998)

The data you reference shows 59%. However, 51% or 59% it doesn't really matter. What is important is the trend shown. IE is consistently losing market share over time. Were they climbing up the ladder that would be one thing. They're not. This doesn't say anything about whether or not they "cheated", only that your claim that their market share is so high why would they bother to cheat doesn't make much sense, to me.

People miss the point (0)

Anonymous Coward | more than 3 years ago | (#34253796)

Perhaps Microsoft cheated intentionally. Perhaps it's accidental they over-optimised for the benchmark. Either way, their browser is again proven to be inferior than advertised, to the great disappointment and irritation of their developers and users alike. Seriously, I keep giving them the benefit of the doubt and I'm let down every time...

TDD (0)

Anonymous Coward | more than 3 years ago | (#34253840)

Why would it seem weird to develop TDD where SunSpider is the test. If that is the current best benchmark test, then normal browsing should be simple.

free mp3 license (0)

Anonymous Coward | more than 3 years ago | (#34253894)

http://web.archive.org/web/19981202072738/http://www.mpeg.org/MPEG/mp3-licensing-faq.html [archive.org]

The following letter from Mr. Henri Linde (Thomson) explains the patent and licensing situation for MP3 (aka Layer-3) as of April 1998.

Business Answers MPEG Layer-3 - Version 04.98-3 ...thank you for your interest in our MPEG Layer-3 audio compression technology.

In this document you will find answers to the most frequently asked questions about MPEG Layer-3, the Fraunhofer Gesellschaft, THOMSON multimedia and the licensing of MPEG Layer-3 technology.

The Fraunhofer Gesellschaft.

The Fraunhofer Gesellschaft is the leading organization of applied research in Germany. It operates 47 research centers in Germany with about 9,000 employees, about half of them scientists and engineers. The Fraunhofer Gesellschaft expands to a worldwide Organization, especially in USA and Asia. Home of the Fraunhofer Gesellschaft is Munich.

One of the goals of the Fraunhofer policy is a rapid transfer of innovations into products. The total research expenditure is about US $ 700 million.

The Fraunhofer Institut Integrierte Schaltungen (Fraunhofer IIS), based in Erlangen, is one of the 47 research centers. In the field of high quality low bitrate audio coding, Fraunhofer IIS is the leading international research lab. Fraunhofer IIS has been the main developer of the most advanced audio coding schemes, like MPEG Layer-3 and MPEG-2 AAC (Advanced Audio Coding). Fraunhofer IIS plays a major role in the ongoing work for the MPEG-4 Audio standardization process and contributes to many other standards bodies as well, like ITU-R TG10/4, ITU-R WP10C, AES, MPEG-IPR, and others.

WWW: http://www.iis.fhg.de/audio/ [iis.fhg.de]

THOMSON multimedia.

The world's fourth-largest consumer electronics group, THOMSON multimedia develops, manufactures and markets: televisions, VCRs, camcorders, audio and communication appliances, satellite decoders, DVD players, as well as color TV tubes and professional television equipment (studio equipment, TV cameras, mixers, etc.).

With its pan-European brands - THOMSON, TELEFUNKEN and SABA - and its American brands - RCA, GE and ProScan - today the group is the leader in the United States, No. 2 in Europe and is strongly increasing its market presence in Asia, Latin America and Eastern Europe.

WWW: http://www.thomson-multimedia.com/ [thomson-multimedia.com] http://www.rca-electronics.com/ [rca-electronics.com] or
http://www.nipper.com/ [nipper.com]

OPTICOM

OPTICOM is a "spin-off" firm from Fraunhofer IIS. OPTICOM was founded in 1995 with the intention to assist you in finding the technical solution best suited for you through consulting and specialized tools.

WWW: http://www.opticom.de/ [opticom.de]

I) GENERAL

Q. I am interested in source coding technology. What can you do for me?

A. Licensing source code, developing real-time DSP code as well as DSP-based hardware, and developing codec libraries for desktop computers is part of the Fraunhofer IIS activities. The focus is on (mobile) high-quality low bitrate audiovisual communications (e.g. digital broadcasting systems or internet services).

Q. Regarding MPEG Layer-3, do any patent rights exist?

A. As for practically any important technology (and particularly for publicly established standards), you should know that patent rights for MPEG Layer-3 exist. Although others may also hold patents related to the MPEG Layer-3 technology, THOMSON multimedia and Fraunhofer IIS have pooled their interests and grant licenses under their combined patent portfolio.

Q. Who grants licenses under this combined portfolio to use MPEG Layer-3?

A. We have split the licensing activities in three parts:

i) THOMSON multimedia handles high volume consumer applications, such as desktop software, broadcast systems, pay-audio services, codec ICs and the like.

ii) Fraunhofer IIS handles professional applications, such as ISDN codecs based on DSP hardware, developed as part of a R&D project with Fraunhofer IIS.

iii) OPTICOM is an authorised distributor, handling desktop software in low-volume consumer applications, promotional use of MPEG Layer-3 on WEB-sites, distribution of MPEG Layer-3 data and decoders on CD-ROMs, and turnkey systems for audio on demand.

Q. So, whom should I contact?

A. You may address your request to any of us, but the descriptions in the previous answer and the rest of this FAQ should give you some guidance. In order to speed up the process, we would appreciate to receive a description of your business model and your expected use of the technology. Here are the names and addresses of the persons to contact:

THOMSON multimedia
Mr. Henri Linde
Manager, Licensing & Intellectual Property

lindeh@thmulti.com
46 Quai Alphonse Le Gallo
F-92648 BOULOGNE Cedex
France
voice: +33 141 86 52 87
fax: +33 141 86 0921 or +33 141 86 5638

Fraunhofer IIS
Mr. Niels Rump
Manager, Business Relations

layer3@iis.fhg.de
Am Weichselgarten 3
D-91058 ERLANGEN
Germany
voice: +49 9131 776 611
fax: +49 9131 776 399

OPTICOM GmbH
Mr. Michael Keyhl

info@opticom.de
Am Weichselgarten 7
D-91058 Erlangen
Germany
voice: +49 9131 691 160
fax: +49 9131 691 325

Q. I just want to buy a MPEG Layer-3 software and/or hardware product. What is available?

A. Many companies already licensed our MPEG Layer-3 technology and offer those products. Please visit the Fraunhofer website (http://www.iis.fhg.de/audio/) for more information.

II) PATENT LICENSING TERMS AND CONDITIONS

Q. And how much royalties or other fees do you charge?

A. The rules of ISO/MPEG determine that licenses have to be granted on a non-exclusive basis and on fair and reasonable terms. For identical applications, we add that the terms are non-discriminatory. The details depend on the application and the specific business model.

Q. What kind of licensing models exist?

A. To date, we have developed and granted license agreements for the following applications:

THOMSON multimedia
- Software decoders (see II.a)
- Hardware decoders for consumer-type applications (see IIb)
- Software encoders (see II.c)
- Pay-audio systems (see II.d)
- Broadcast systems (see II.e)

Fraunhofer IIS
- Professional applications (see III)

OPTICOM
- Distribution of MPEG Layer-3 data and decoders on CD-ROMs (see IV)

CREDITS

In all cases, we expect that MPEG Layer-3 products reference the licensors, like "MPEG Layer-3 audio compression technology licensed by Fraunhofer IIS and THOMSON multimedia".

Output? (1)

ebcdic (39948) | more than 3 years ago | (#34254154)

Does this part of the benchmark produce a result or output, and if so is it correct?

And if it doesn't produce any output or a result that's checked, there is plenty of scope for innocent explanations. It could be a bug that doesn't arise when the extra statements are added. Or it could be that part of the code is being optimised away (because the result isn't used) and the analysis isn't clever enough to handle it when the extra statements are present.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?