Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

ESR and the MindCraft Fiasco

CmdrTaco posted more than 15 years ago | from the stuff-to-read dept.

Microsoft 204

The one and only Eric S. Raymond has submitted his response to the Mind Craft report that we've talked about a bit here lately. This is a good wrap-up type piece which nicely summarizes the flaws with the testing (which range "yeah maybe" to "you gotta be kidding!"). Anyone who thought the tests had any validity should read this.The followingw as written by Slashdot reader, Jargon File Maintainer, Fetchmail Author, Open Source Evangelist, Eric S. Raymond

The Mindcraft fiasco

Microsoft's latest FUD (Fear, Uncertainty and Doubt) tactic may be backfiring.

A 21 April ITWeb story reported results by a benchmarking shop called Mindcraft that supposedly showed NT to be faster than Linux at SMB and Web service. The story also claimed that technical support for tuning the Linux system had been impossible to find.

Previous independent benchmarks (such as "Microsoft Windows NT Server 4.0 versus UNIX") have found Linux and other Unixes to be dramatically faster and more efficient than NT, and independent observers (beginning with a celebrated InfoWorld article in 1998) have lauded the Linux community's responsiveness to support problems. Linux fans smelled a rat somewhere (uttering responses typfied by "Mindcraft Reality Check"), and amidst the ensuing storm of protest some interesting facts came to light.

  1. The benchmark had been paid for by Microsoft. The Mindcraft press release failed to mention this fact.
  2. Mindcraft did in fact get a useful answer to its request for help tuning the Linux system. But they did not answer the request for more information, neither did they follow the tuning suggestions given Also, they forged the reply email address to conceal themselves -- the connection was made after the fact by a Usenetter who noticed that the unusual machine configuration described in the request exactly matched that of the test system in the Mindcraft results.
  3. Red Hat, the Linux distributor Mindcraft says it asked for help, reports that it got one phone call from them on the installation-help line, which isn't supposed to answer post-installation questions about things like advanced server tuning. Evidently Mindcraft's efforts to get help tuning the system were feeble -- at best incompetent, at worst cynical gestures.
  4. An entertainingly-written article by the head of the development team for Samba (one of the key pieces of Linux software involved in the benchmark) described how Mindcraft could have done a better job of tuning. The article revealed that one of Mindcraft's Samba tweaks had the effect of slowing their Linux down quite drastically.
  5. Another Usenet article independently pointed out that Mindcraft had deliberately chosen a logging format that imposed a lot of overhead on Apache (the web sever used for the Linux tests).

So far, so sordid -- a fairly standard tale of Microsoft paying to get exactly the FUD it wants from a nominally independent third party. But the story took a strange turn today (22 Mar) when Microsoft spokesperson Ian Hatton effectively admitted [8] that the test had been rigged! "A very highly-tuned NT server" Mr. Hatton said "was pitted against a very poorly tuned Linux server".

He then attempted to spin the whole episode around by complaining that Microsoft and its PR company had received "malicious and obscene" email from Linux fans and slamming this supposed "unprofessionalism". One wonders if Hatton believes it would be "unprofessional" to address strong language to a burglar caught in the act of nipping the family silver.

In any case, Microsoft's underhanded tactics seem (as with its clumsy "astroturf" campaign against the DOJ lawsuit) likely to come back to haunt it. The trade press had largely greeted the Mindcraft results with yawns and skepticism even before Hatton's admission. And it's hard to see how Microsoft will be able to credibly quote anti-Linux benchmarks in the future after this fiasco.

Sorry! There are no comments related to the filter you selected.

Say thanx and get to work! (0)

Anonymous Coward | more than 15 years ago | (#1920104)

I am worried by the general attitude of the community towards the Mindcraft test. IMHO it would be much more productive if we said:
"Thanx Mindcraft for working in the Free|Open spirit, test our software for us, and report problems back". The study shows at least that it is not difficult to get sub-optimal performance from a Linux box. Since nobody got such a monster at home, the programs were never tested on it before, not to mention tuned or even developed for such a machine. I sincerely hope that Kernel, Apache and Samba hackers will take this opportunity to improve their software for high-end servers:
- SMP
- big-memory support
- better|alternative defaults
- optimization guides

HOW-TO put an end to the fiasco! (0)

Anonymous Coward | more than 15 years ago | (#1920105)

Demonstrate performance on SPECweb96 on a 1, 2 and 4-way Xeon. All vendors can tune their solution to the max. Presto: even playing field!

Compare against existing NT4, Solaris, and NT2000 benchmarks on similar 4-way Xeons.

http://www.specbench.org/osg/web96/

--G

i think they did (0)

Anonymous Coward | more than 15 years ago | (#1920106)

there was something posted on /. a couple days ago where someone did a test of a poorly tuned linux server against a poorly tuned NT server (the controversial tests here were a poorly tuned linux vs. well-tuned NT).
linux won.

Mainframes (0)

Anonymous Coward | more than 15 years ago | (#1920107)

I couldn't agree more. We have been using a mainframe for almost 20 years, and just recently the decision came down to replace it with SAP running on some heavy IBM servers. The only problem is we got about 5 seconds into production of one section of the organization and the whole thing screeched to a halt.

Now we'll likely go back to SAP running on a newer Mainframe.

Sure we use Novell, NT and Linux for desktop authentication servers, passthrough stuff, web etc. But for anything vitally important, it's running on a Mainframe, or really, incredibly large servers that can not be classed as PC's.

PC's didn't win, they're just more visible.

Yes... But...There's always Linus (0)

Anonymous Coward | more than 15 years ago | (#1920108)

Linus uses a quad xeon at home (don't remember the exact RAM) and he's most interested in SMP...so I would think developer support is there.....

Dave

Irrelevant (0)

Anonymous Coward | more than 15 years ago | (#1920109)

Even if Linux is currently slower than NT on that configuration (since, as others have pointed out, few Linux developers have access to such hardware), any speed comparison is irrelevant.

The important things to remember are:

the NT system will never get faster, modulo new releases from Microsoft. (And those releases tend to be infrequent, bug-ridden, and bloated.)

the Linux system will never get slower.

the Linux system will get faster as developers get access to such high end systems.

Finally, 4-way Xeon servers aren't exactly a consumer item. Any department that will buy one probably has an annual budget well over $1M. If I were that manager and I wanted a screaming server, I would buy at least *five*.

Two would be deployed (for redundancy).
One would be used for final testing of any changes (and would be a hot spare).
One would be loaned to the Apache team
One would be loaned to the Samba team

The ones loaned to the Apache and Samba teams would be carried on the books as spares (and the terms of the loan would spell this out)... and internally the cost of the systems would be offset by the advertising benefits of donating hardware to those popular projects.

How about Fight FUD with facts? (0)

Anonymous Coward | more than 15 years ago | (#1920110)

Demonstrate performance on SPECweb96 on a 1, 2 and 4-way Xeon. All vendors can tune their solution to the max. Presto: even playing field!

Compare against existing NT4, Solaris, and NT2000 benchmarks on similar 4-way Xeons.

http://www.specbench.org/osg/web96/

--G

This moron has a point, of sorts. (0)

Anonymous Coward | more than 15 years ago | (#1920111)


Halloween Letter LXII or maybe I Quit IV

This Zippo character is dumber and pettier than Raymond is (no easy task, IMHO :), but he does have half a point. There's a dim scent of bandwagon-jumping wafting from Raymond's thing here, you know?

On the OTHER hand, as has been mentioned above by several posters, the actual purpose of Raymond's post here is to put all the valid claims of the debunkers together in one spot, associated with a name that trade-press editors have heard of. The point is publicity, along with what passes for credibility in the trade press. Raymond can provide that. He can get these statements out of the advocacy newsgroups and into column inches in IT publications. Doing so is important. IMHO he should be more cool, calm, and collected when he talks specifically about Microsoft, but on the whole this post is just what we all like best to see him do: Advocate free software armed with facts and logic. This post has no silly infighting with competing OSS Rock Stars and no bullshit. I like it. From where I sit, Raymond's earned back some of the respect that he lost (from me, anyway; YMMV) over the last few months.

Zippo just doesn't get it, that's all.

Mindcraft Labs' 'Independence' a Complete Fraud (0)

Anonymous Coward | more than 15 years ago | (#1920112)

If anything, they are nothing more than a thin shell for Microsoft PR. It would be a great story for some good investigative IT reporter to check out the background and funding of this supposedly 'independent' lab. I think it's clear from their reports who they're actually working on behalf of. Witness their home page:

"Windows NT Server Outperforms Linux"

"What's Wrong with the Quicktime Plug-In and How to Fix It"

"White paper compares Window NT Server 4.0 and NetWare 5 as file servers"

"White paper compares Windows NT Server and Solaris Web Servers"

In all cases, Windows comes out on top, and in all cases, Microsoft funded these supposedly 'independent' reports. Gee, wonder what's going on here? I also wonder how much they paid to have their findings published on Mindcraft's home page. Some clever reporter from ZDNet (how about Dvorak?) could make this into a fantastic expose.

What about the numbers? (0)

Anonymous Coward | more than 15 years ago | (#1920113)

Blah, blah blah, blah blah, blah. Who cares about all these thousands of comments?

The only thing that should be of interest is the performance numbers on the same four-way Xeon/RAID system with both platforms tuned to the gills.

Could a spider be used to count existing servers? (0)

Anonymous Coward | more than 15 years ago | (#1920114)

I dont know if there is a way for the client to to determine which server it is talking to, but if it is then a spider could be written to accuratly count Apache vs IIS servers. Anyone ?

Yes but be careful. (1)

Anonymous Coward | more than 15 years ago | (#1920155)

Netcraft is site surveying site that has always been objective and fair. MINDCRAFT are the idiots in question.

HOW-TO put an end to the fiasco! (1)

Anonymous Coward | more than 15 years ago | (#1920156)

Demonstrate performance on SPECweb96 on a 4-way Xeon. Compare against existing NT4 and Solaris benchmarks on similar 4-way Xeons.

--G

Does it matter? (1)

Anonymous Coward | more than 15 years ago | (#1920157)

Does it really matter whether Linux outperforms NT on a high-end SMP server? We know that Linux outperforms NT at the low end, and other Unixes outperform NT at the high end. Linux has NT looking like a poor choice for both the small-scale departmental server that has always been its bread-and-butter, and Unix has always made NT look like a poor choice for the high-end enterprise servers, and that doesn't seem to be changing.

So what is left for NT?

Careful, you can get sued.... (1)

Anonymous Coward | more than 15 years ago | (#1920158)

You have to be real careful performing benchmarks against commercial software. If you read thru some of the Micro$oft EULA's there's specific prohibitions on publishing benchmarks. In short, without an agreement from Micro$oft, it's a good way to get sued!

The commercial product I work on would love to publish direct comparison benchmarks against the competing MS product. Our legal department won't let us.

Interesting Hatton comment (1)

whoop (194) | more than 15 years ago | (#1920160)

With the recent report on the untuned NT vs Linux with Oracle server, many of the marks were 20 times faster on Linux than NT. So, I got to wondering, wouldn't it be funny if someone was able to achieve similar/better scores in a Linux system with a less powerful machine (how many of us have quad xeons?). If it were shown that a dual-p2 or something running Linux was just as good as their xeon beefed up NT box, that would make quite the statement, both on the study and on NT.

Yes... But... (1)

whoop (194) | more than 15 years ago | (#1920161)

As the guy above said, a quad-cpu machine isn't terribly high end, but more than the average desktop certainly.

But with regards to the study, wouldn't it have been more truthful to tune the Linux box appropriately? Then if the results showed that NT was better, a point could be made that Linux needs more work in these areas. I really believe people would accept that as honest criticism (again, IF the results showed that NT was faster, I have my doubts that would happen :)).

Mindcraft obviously didn't make much of an effort in tuning the box (ie hire a decent Linux admin), posted a message to a couple newsgroups (lacking sufficient details, and not responding when someone requested more details), one call to RedHat (which was directed to the wrong group as ESR stated). And yet, they build an entire study around these points, making claims that Linux support is bad and all. For these reasons they were blasted, not for saying NT is better than Linux (as some people I've talked to think is the motive for Linux people's outrage).

Don't forget their motto (4)

whoop (194) | more than 15 years ago | (#1920162)

In the Performance Testing section on their web page [mindcraft.com] , second paragraph they say flat out:

"...we work with you to define test goals. Then we put together the necessary tools and do the testing. We report the results back to you in a form that satisfies the test goals."

Since they say Microsoft sponsored the test, we can replace "you" with "Microsoft." So they worked with MS to define the test goals (NT is 2 or more times better than Linux). Then they put together the tools to do that, hacking the registry and all to beef NT up, slowing Linux apache/samba servers. And finally, report the results back in a form that satisfies the test goals, lo and behold NT is 2-3 time faster than Linux. Such a surprise, right?

Apache Benchmarking (1)

SuperQ (431) | more than 15 years ago | (#1920163)

Very interesting, What was the total cost of your system?

Things I could see improve your system:

drop the Apache RPM's and compile it.. Specificaly with the PGCC compiler, I have heard of 30+% speed improvements with it. you could even go for the PGCC based distro stampede for your base install.

look at mylex RAID boards.. they are supposed to work a little better than the AMI cards that Dell uses.

It'd be fun to send that box to mindcraft, and have them test it as same preformance or 90% preformance of the NT box.. but cost less than $5k
not $18,000

Hard to find... (1)

Isaac-Lew (623) | more than 15 years ago | (#1920165)

an impartial third party that can optimize both Linux *and* NT fairly.

Maybe everybody is at Comdex? (1)

Eric Green (627) | more than 15 years ago | (#1920166)

Please note that this report was (deliberately?) released a week before Comdex, when most Linux vendors were running around like chickens with their heads cut off trying to get everything straight for their Comdex displays. And most of the top people for Linux vendors have been at Comdex all of this week, while the home offices are understaffed and slowly going crazy because there's not enough people to do everything that's needed to keep the business running...

Given all of that, it's a little early for sponsored benchmarking by members of the Linux community. This stuff takes time, and if it's a choice between doing Comdex right and repeating a discredited benchmark, doing Comdex right comes up at the top of the list every time.

Isn't that what I said? (2)

Eric Green (627) | more than 15 years ago | (#1920167)

Isn't that what I said in the "Mindcraft Reality Check"?

There are valid limits to Linux scalability, problems that need fixing, and honest benchmarking can help us find those limits. Unfortunately, Mindcraft's benchmarking was so flawed by misconduct and poor judgment that it is not useful for that purpose.

BTW, I do agree with the Microsoft spokesman who said that he was certain that NT would have come out on top even with an honest test. I suspect the SAMBA results would have been quite competitive, within 3% (more or less) of the NT numbers, but the Apache server has never been known for its static file serving speed (though mod_mmap_static may change that!). On the other hand, there is a big difference between the 5%-10% advantage that I bet Mindcraft would have found, and the ridiculous numbers that they actually reported. They actually shot themselves in the foot here, because if they'd reported the real numbers, the Slashdot Crowd would have howled, but Jeremy Allison and other technical heavyweights would have stayed on the sidelines working on fixing the problems found, and the media would have ignored the Slashdot Crowd.

Just count it as another example of Microsoft Arrogance (tm) outweighing their good sense. It's amazing how such bright people can do such stupid things.

-- Eric

Put an end to the fiasco! (1)

gavinhall (33) | more than 15 years ago | (#1920169)

Posted by kkkotta:

Everyone is complaining how Microsoft rigged this test, but no one is surprised or doing anything about it. Why doesn't someone just sponsor an HONEST test, and let the best system win.

How about a range of machines? (1)

gavinhall (33) | more than 15 years ago | (#1920170)

Posted by smich:

Wasn't the test supposed to be about how nt can scale up but Linux can't? We all know the test was rigged, and I'd like to see it run again, but over a range of machines.

What does nt vs Linux look like on a 486 66 w. 8 meg of ram? Hmmm?

Proud of the Linux community and I learned a lot! (1)

gavinhall (33) | more than 15 years ago | (#1920171)

Posted by kewlmann:

I work for a fairly large company thats coming out with a kick ass machine pretty soon. Its enterprise class for sure. Lots of processors. I would love to see linux tuned for it.

Yes... But... (2)

AshNazg (830) | more than 15 years ago | (#1920174)

Yes the study was flawed. But I remember a comment by Matt Welsh in which he said that Linux is not properly represented in the high-end machines.

That is a natural consequence of the open source
development. Many of the features of Linux are there because some users needed them (scratch an itch, as ESR says...)

So, bearing in mind that there aren't many Linux
users with quad xeons with 4 Gb Ram, it's only natural that issues relating with that kind of machine have a lower priority to the Linux user community

Mainframes (1)

jabbo (860) | more than 15 years ago | (#1920175)

The PC won over the mainframe? Perhaps in the general-utility-computing arena, but for truly obscene loads and outrageous availability requirements, mainframes still rule. Ask E-Schwab, for example, or REI.

Mainframes will never die. The legacy system of tomorrow will be mainframe transaction-processing systems fronted by SP/2 analysis clusters, with something like Linux or NT gating the whole mess to the web. I can almost guarantee it.

Of course, Linux is being ported to the S/390...

About Netcraft, semi-off-topic question (1)

Ami Ganguli (921) | more than 15 years ago | (#1920176)

I'm curious about this too. I wonder if people who pay Netcraft money get that kind of breakdown.

One thing to remember is that Netcraft counts domains, not IP addresses. ISPs that host sites for clients probably dominate the survey and most of them run Apache. Companies that run their own web site off a T1 or ISDN line probably favour NT, but they would be in the minority.

I like to take each months numbers and pretend that nobody is switching (clearly not true, but might be a good approximation). This month, for example, Apache gained 423063 sites and MS gained 132943. So new Apache sites outnumbered new MS sites by MORE THAN 3 TO 1! That's impressive.

Huh? (1)

sterwill (972) | more than 15 years ago | (#1920177)

I've seen very little comment that Linux might actually really be slower on a 4-way. I would be disappointed, but not amazed if Linux were slightly slower on a 4-way given the maturity of NT SMP compared to Linux.
Maturity of NT's SMP? Wait, I thought we were talking performance. Obviously you've never used SMP NT. Add a processor it gains 20%. Add two more and it loses 10%. Scalable like mud.

Not an easy problem. (1)

j.e.hahn (1014) | more than 15 years ago | (#1920178)

I would still like to see some more conclusive stuff on Linux's highend SMP abilities (4 or more), bother on i386 and on Alpha and UltraSPARC. Alan Cox claimed there were some speedups to SMP late in 2.1.x (i think) that should have significantly improved hi end perf. Perhaps VAR would sponsr a test?

If Linux doesn't beat all on up to 16 or so processors, we should fix it.

It's not that simple. Currently the biggest problem with the Linux SMP implementation is the IO subsystem. (SCSI and IDE, etc.) The problem is that this subsystem isn't SMP safe. So whenever the kernel enters this portion, it grabs what is known as the kernel lock. Thus disk activity can only happen on 1 processor at a time (bad). In Linux 2.3 they're going to strip this away, so that Linux 2.4 (or 3.0, or whatever) will probably scale far better than Linux 2.2 (of course they'll be making other nice optimizations along the way). But this isn't something you can just fix on a whim.

Not to side with Mindcraft/MS, but... (1)

sjames (1099) | more than 15 years ago | (#1920179)

One could interpret it that way, but considering that they willfully avoided being helped to tune Linux (ask for samba help anywhere but on the samba lists/newsgroups, call the wrong helpline, don't ask for the right one, when they accidentally do find a helpful person, ignore him, etc...) it seems that the goal must have been to find that NT outperforms Linux.

It reminds me of an old Mad Magazine comic, A father orders his steriotypical hippy son living at home to go get a job. The son, dressed in his most casual attire (torn cutoff shorts, open shirt, many beads etc), goes to a clothing store and says "You ain't got no jobs do you?". Later to his father, "Well, I TRIED!".

The only difference is, Mindcraft actually did get an offer, so they had to ignore it.

And...? (1)

Matthew Kirkwood (1344) | more than 15 years ago | (#1920180)

Hmm... I think that most of us are well-versed with the arguments against those benchmark results.

I feel the need to ask why this piece was worthy of "airtime", and can reach only one conclusion: That a refutation can somehow become more valid because it comes from the pen of Eric Raymond.

I'm now happy that Katz periodically contributes his dozen screenfuls of drivel, and I don't have a problem with Eric either, but would I have got a whole item on /., had I summarised the obvious flaws of that ridiculous benchmark? I doubt it.

Matthew
- what happened to our great meritocracy?

Microsoft's credibility (1)

sql*kitten (1359) | more than 15 years ago | (#1920181)

the PC community *won*?

what have you been smoking?

with the rise of n-tier architecture and thin client, mainframe-style computing is getting stronger than ever. centralised data and logic, just the display on the desktop.

So, is linux faster than NT on a 4-way w/ 2GB mem? (3)

peel me a grape (1399) | more than 15 years ago | (#1920183)

I've seen very little comment that Linux might actually really be slower on a 4-way. I would be disappointed, but not amazed if Linux were slightly slower on a 4-way given the maturity of NT SMP compared to Linux.

I would like to know if Linux does scale as well or better than NT with 4 and 8 processors -- both systems properly tuned and using the same webserver. When that question is answered, I'd like to know what to expect in the future. Is Linux going to leave NT in the dust, or will this be the key niche ground for NT servers that Microsoft will defend to the end, and Linux will never conclusively defeat?

Mainframes (1)

mangino (1588) | more than 15 years ago | (#1920184)

As an extremely unhappy user of an sp/2 system, I hope the sp/2 isn't the system of the future. I can do some things faster on my pentium 90 with Linux than the SP/2 system. We have 32 nodes with 4G each and 2 Terabyte of disk spread out over them. With this setup, you'd think things would be fast. The only thing that happens fast is the corruption of my data by their filesystem. In short, the SP/2 system isn't made for databases or web serving. It's meant as a compute box (and it doesn't even do that well!)

But how is Linux SMP (1)

Matts (1628) | more than 15 years ago | (#1920187)

While our box is not SMP, I very much doubt that SMP on 4 processors (which even 2.0.36 was known to cope quite well with, especially for simple tasks like web serving) would cripple a box to the extent provided by the mindcraft survey.

I can see an SMP system to only improve performance up to a point, eventually hitting a limit (e.g. 8 processors won't be twice as good as 4 because of time spent scheduling), but to see the effects that mindcraft saw you would have to do something pretty crafty...

Matt.

No - you're wrong. (slightly) (2)

Matts (1628) | more than 15 years ago | (#1920189)

You're only right on the kernel front - that hasn't really been used enough on high end machines AFAIK.

But both Samba and Apache have been heavily tested on very high end servers. The Samba crew have even been heavily involved in making Samba fast on high end servers.

HostNameLookups (3)

Matts (1628) | more than 15 years ago | (#1920190)

Please note that the dejanews reference that ESR links to is quite wrong. The presence of %h does _not_ cause host name lookups under Apache - only the directive "HostNameLookups on" causes that to occur. I don't believe this to be the case.

I strongly believe however that their httpd was running under inetd, and that would cause the effect they saw.

Apache Benchmarking (5)

Matts (1628) | more than 15 years ago | (#1920191)

At a large company I'm working with we're trying to prove to the phb's that Linux is a good thing. The mindcraft study set us back a ways. So what did we do? We did our own tests.

Server:
- Hand built by our best hardware guy
- PIII 500 (single CPU)
- Adaptec 2940U2W SCSI Adapter
- 10,000 rpm LRW drive. 1 drive only.
- 100Mb/s network card
- 256Mb PC100 RAM.
- Linux 2.2.6, upgraded from stock Linux-Mandrake box
- Apache 1.3.6, configured for best performance.

No changes to the /proc fs to speed things up. Stock kernel options selected from "make xconfig". Apache was the apache+mod_perl srpm found on redhat/contrib, compiled with no configuration changes. We didn't test NT on this box - we were trying to compare against Mindcraft's results.

Want to know the results so far?

Well, we can get about 2200 requests per second out of that box. The Quad Xeon NT box that mindcraft tested got 3700 requests per second at its maximum rate. We are at very early stages so far, and I think I can squeeze more out of the box by dumping Apache and using thttpd or something else that uses a threaded model. But since this is to be a pure mod_perl box I don't think that's important.

Things to remember:

The mindcraft server had 1Gb of RAM.
The mindcraft server had RAID (RAID/0 I believe).
The mindcraft server had 4 10/100 network cards.

We're so far pretty pleased with our little Linux box... It was a fair bit cheaper than Mindcraft's server....

Eric made a factual mistake (3)

cjr (2590) | more than 15 years ago | (#1920194)

Here is what the ITWeb editorial says:

"Linux supporters have reacted violently to the Microsoft SA release (Independent research shows NT 4.0 outperforms Linux) published on ITWeb yesterday, saying "the study was paid for by Microsoft" and that "a very highly-tuned NT server was pitted against a very poorly tuned Linux server".

That is, the claim attributed by Eric to Ian Hatton was really made by reacting Linux supporters.

What Hatton did admit, was:

"Microsoft did sponsor the benchmark testing and the NT server was better tuned than the Linux one."

This isn't much, but it is sufficient. Hatton admits that "the NT server was better tuned than the Linux was" and even without adjectives that invalidates the report.

Suspicious (1)

wardk (3037) | more than 15 years ago | (#1920197)

I am not suprised by how "blatant" this whole episode is. Relatviely speaking, this is nothing to the FUD heaped on OS/2 by MS after the split with IBM.

Those who witnessed "OS Wars" of the early, mid-90's are well aware of the ability of MS to bludgeon superior technology into submission through marketing.

The main (and important) difference I can see is that today, MS has less credibity, and their target, being Linux in this case has no corporate "owner" like IBM. IBM sat idle while MS (and the IBM PC Company, "MS' biggest customer") and the trade press trashed OS/2 into oblivion.

I do not see the Linux community standing idle and taking it, ESR's post is a fine example of this. Note that benchmarks like mindcrafts were done with NT vs OS/2 over and over with no real response from IBM. The OS/2 users who protested were categorized as "zealots" and written off. On Compuserve, false user accounts (see "Barkto") were alleged to have been created to depict "real users" who then went on and on about serious OS/2 errors that "trashed my hard disk" and "my backups", ad nauseum. (Such reports were then published in PC Week, Infoworld, Computerworld to drive home the FUD).

MS has it's hands full trying to FUD Linux into obscurity. But be assured, they are experienced at this type of "warfare" and will attack furiously. With such deep pockets, I expect they feel a war of attrition is winnable.

This remains to be seen. The Linux community is not an impotent IBM. And today, we have a maturing internet to get some real facts distributed that the traditional "legacy" trade rags tend to not report.

That's beautiful (1)

aheitner (3273) | more than 15 years ago | (#1920198)

You guys bring tears to my eyes, matching a fully tuned $25k NT box with an untunes $3000 linux box.

If you want some more performance, I suggest moving to a dual system. My experience in SMP linux has only been with dual systems, and it's been very good. In addition, there are some very nice, cost effective dual motherboards out there. Tyan makes one with onboard aha2940 SCSI, Intel 10/100mbit ethernet, and sound too :). Hey, if it's good for /., it must be good enough for everyone else.

I wouldn't be surprised if with some tweaks to your box configuration you could make it as fast as the NT box without any hardware mods, perhaps by following some of the advice Eric links to...

----

I would still like to see some more conclusive stuff on Linux's highend SMP abilities (4 or more), bother on i386 and on Alpha and UltraSPARC. Alan Cox claimed there were some speedups to SMP late in 2.1.x (i think) that should have significantly improved hi end perf. Perhaps VAR would sponsor a test?

If Linux doesn't beat all on up to 16 or so processors, we should fix it...

I'm curious (1)

aheitner (3273) | more than 15 years ago | (#1920199)

how good PGCC is on p6-core processors. In my experience, the key on p6 is not pairing, like it was on the Pentium, but avoiding partial stalls, which empty the pipeline and really fsck things up.

In our own tests, we found that VisualC++ 5.0's (otherwise an excellent compiler) ftol() stalled like crazy on pII's, eating 10% processor power in Fire and Darkness. How good is PGCC at avoiding similar problems.

Are there tools under linux (analogous to Intel's VTune) for analyzing this?

Scaling is what counts (2)

thomasd (3336) | more than 15 years ago | (#1920200)

There have already been plenty of demonstrations that Linux works well on small server. What's really needed now is an impartial test run on a nice big SMP box with oodles of memory and a decent RAID array -- the system Mindcraft were (ab)using would do fine -- to demonstrate that, especially with 2.2 kernels, Linux scales quite well.

Remember that a certain number of sites really need big-iron servers (hey, slashdot isn't exactly gentle on its hardware, although in that case I suspect database performance may be more of an issue), and even when they don't it's the results from high-end server tests which impress the management the most.

Having seen Linux/SMP in action and made some subjective judgements I'm quite confident that, properly configured, it ought to scale fairly well onto hardware of the class Mindcraft were `testing'. But it would still be nice to have some number...

Yes... But... (1)

mdxi (3387) | more than 15 years ago | (#1920201)

I think you're a little bit out of touch with the vast majority of the Linux community.

One of Linux's prime virtues is what it can do with older hardware. I have a friend who wouldn't believe that the Linux machine she was using at my house was a P90 because Netscape/WordPerfect/etc. "felt" as fast as W95 on her PC (a PII/333).

I have three Linux boxes, the (dual) P90 mentioned above, a K6/166 that acts as my web/mail/ftp/telnet/IRC/MOO/everything server and my 386/25 laptop that I use to do my homework.

I would wager that the vast majority of Linux boxes are not high-end monsters but old machines that are "worthless" in the eyes of many people. That is the true power of Linux in my eyes, an attribute that I think is all-too-often ignored here lately.

IMHO, YMMV.

--

I wonder. (1)

Shane (3950) | more than 15 years ago | (#1920204)

I didn't see anywhere a reference to duplexing or
ethernet card configuration. In redhat's default
configuration I have noticed that it selects 10Mbit most of the time. And HALF duplex all the
time..

Only way out of this is to pass the options= command when loading the ethernet module.

Re: Fight FUD with FUD (1)

srobert (4099) | more than 15 years ago | (#1920205)

No, you don't fight fire with fire. You fight it with water. You fight FUD with the truth. You made a valid point though when you said that unbiased testing cannot be done or paid for by the Linux community. Nor can it be done by Microsoft. So what is needed is for the Linux community to throw down the gauntlet. Challenge Microsoft to a duel. Let both sides agree upon an independent referree to determine how each system performs on a series of benchmarks. Half of the testing criteria could be determined by MS. The other half by the Linux community. The types of machines involved would be agreed upon and each side would be responsible for optimizing their own setup.

GNUPro (1)

akharon (4824) | more than 15 years ago | (#1920206)

then it shouldn't be tough to distribute freely, just one person buys it, and then gives it to the world.

Put an end to the fiasco! (1)

stevew (4845) | more than 15 years ago | (#1920207)

First - sponsoring a rebuttal in the
form of another benchmark will have to
be left to the Distro guys making money...

Cause it costs money to put together such
a test system, or purchase NT for that matter!

On the other hand - putting out a decent
rebuttal in the form of accurate criticism
such as ESR has done(I REALLY like his
article) is perhaps the best way to point
that Emporer Bill isn't wearing any clothes.
The only remaining trick is to get that
rebuttal circulated amongst the press
widely. ESR has the credibility to get
quoted in such places. Looks like a
good combination, and the right path
to me.

Steve

benchmarks. (3)

law (5166) | more than 15 years ago | (#1920210)

Good summary.
Seems to me that what we really need is a bench marking rebuttal; is there another
benchmark going on? I saw that in Jeremy Allison's article he was working with PC Week,
does anyone else know any other active bench marking going on?

I think that the only way to prove against FUD is education, bench marking can go a long
way.

I have about 7 Linux servers with no down time, great performance on lesser hardware then
my commercial servers in my company, that should be proof enough; but my pointy haired
boss still asks "Why not NT?". I do not need any more fuel for that fire.

We need Benchmarks on larger servers, with more memory, RAID, and a high-end server
guide.

Proud of the Linux community and I learned a lot! (2)

John Kacur (5703) | more than 15 years ago | (#1920213)

Good article Eric, and I'm proud of the Linux community for the way we've reacted to the Mindcraft "benchmarks". I think the Linux community fought back, but in a mature balanced way. I think it is important that we continue to do so, and try not to appear too much like reactionary fanatics which somtimes happens too.

Also, I must say I really learned a lot by following the debates. Next time I need to install Apache and Samba you can bet I'll be referencing the responses to Mindcraft to see the proper way to optimize this stuff.

Kudos and thanks to the Linux Community!

Spread the word (2)

Rob Kaper (5960) | more than 15 years ago | (#1920214)

Without being too fanatic, I think that we all should inform any magazine publishing the Netcraft results (and thus concluding Linux is sh** compared to NT) of the facts and unreliability of this survey.

Admit, "M$ guilty of consumer fraud" is a better headline than "NT beats Linux on all fronts".

No! Get someone else! (1)

NeoTron (6020) | more than 15 years ago | (#1920215)

It would, IMO, not be a good idea to get Mindcraft to do another test.

It would be a far better idea to get a truly _impartial_ party to re-do these tests, with proper help from the Linux community. Then we'll see the results!

I simply don't _trust_ MindCru^Haft.

cjr used unclear language (1)

Jeff Kandt (6763) | more than 15 years ago | (#1920218)

The language makes it very unclear whether the quote came from the Linux supporters or from the Microsoft SA release.

Of course, it's unlikely that the quote came from the Microsoft SA press release (It is referring to the original brag release: "Independent research shows NT 4.0 outperforms Linux"). I think that cjr was quoting Linux users and therefore Eric was wrong to attribute it to Microsoft, much less to Ian Hatton directy.

Spread the word (2)

szyzyg (7313) | more than 15 years ago | (#1920220)

Ummm That should be Mindcraft right?

This is an alramingly common mistake - poor netcraft.

Microsoft's credibility (3)

Signal 11 (7608) | more than 15 years ago | (#1920221)

Well, I think this incident has damaged microsoft's credibility, but that's beside the point. Microsoft isn't talking to us, the technical community. They aren't trying to convince us that NT is better. For those of us in server closets, in the operations center, and in system administration - we already know the truth. We don't need benchmarks and statistics to tell us NT is unreliable.

The plain fact is, Microsoft did this to appeal to middle/upper-management, not us. They need to keep feeding them reasons to keep their NT investment without looking stupid. Remember the mainframe days? Shortly after the PC came out, a torrent of similar "debate" emerged from the mainframe community. First they laughed, then they fought, then the PC community won. Suprise. History repeats itself.



--

I've wondered about this too... (1)

somebody else (8966) | more than 15 years ago | (#1920222)

I remember when I first implemented the RC5DES cow on my home Linux box (AMDK5PR90) and on a mostly idle Win95 workstation at work (Intel P166), and my home Linux box was running circles around the blocks of the Win95 workstation (in fact, a few of my friends were embarrassed by their WinXX Intel PII's for not being all that much better than my Linux AMDK5PR90).

So, it makes me wonder about this test as well... Is it possible for someone to tweak a common-joe-affordable Linux box to outperform a supercharged, out-of-affordability-range NT box? Can someone duplicate the server load from the Mindcraft test on a highly tuned Linux PC and show that Linux can beat NT even when Linux is on a smaller machine?

The Mindcrap Affair: second-order effects (3)

kzinti (9651) | more than 15 years ago | (#1920223)

What strikes me about the entire "Mindcrap Affair" is the resulting coverage. I can recall seeing only one press article covering the original story (the "benchmarking"), but I have seen many press articles covering the resulting controversy. Of course, my impression may be biased because I take pointers to news stories from Slashdot and Linux Today. On the other hand, I have done some looking outside of the "linux community", at sites such as CNet News, and they definitely seem more interested in covering the fiasco than in the original benchmark. Maybe these sites too can smell a rat.

--JT

Spread the word, err please dont... (1)

Dion (10186) | more than 15 years ago | (#1920225)

Please call the imposter MindCraft, not netcraft...

I'm begining to think that this is all a conspiracy to undermine Netcraft:)

Sorry, guy... (1)

DH1 (10825) | more than 15 years ago | (#1920228)

Sorry, but a test (particularly one that is supposed to be objective and scientific/engineering in nature) has but ONE GOAL: to determine the facts as objectively as possible.

That isn't 'your' goal or my goal, or Saddam Hussein's goal. It's THE goal, PERIOD. Anybody who makes statements like that admits they run a bovine excrement factory instead of a testing facility.

Microsoft's credibility (1)

jms (11418) | more than 15 years ago | (#1920229)



Remember the mainframe days? Shortly after the PC came out, a torrent of similar "debate" emerged from the mainframe community. First they laughed, then they fought, then the PC community won. Suprise. History repeats itself.


Of course, IBM didn't help themselves by trying to alienate their customers in the early 90s by trying to withdraw all their mainframe source code. That was about the time we started looking into unix systems. BTW we're going to dump our entire machine room full of IBM mainframe equipment at the end of the year ... and the source issue had a lot to do with it.

Not to side with Mindcraft/MS, but... (1)

StimpyBoy (11864) | more than 15 years ago | (#1920230)

Hasn't anyone considered for a second that perhaps their definition of test goals is "determine which OS is a faster web server in a RAID/SMP system" or some other criteria?

Not to defend the report (it's not Scottish, so it's craaaaaaaap :) but I read that sentence in a completely different way.

Yes... But... (1)

IntlHarvester (11985) | more than 15 years ago | (#1920231)


A Quad Xeon is the highest end box that NT runs on (barring Alpha).

Note that WinNT is driving the "high-end" x86 hardware market. Vendors like Dell and Compaq make boxes with only 4 CPUs because that's all vanilla NT will support. When Win2000 comes out, it will support 8? processors, which means the hardware companies will immedeatly follow with 8 CPU iron. (Implictly making this hardware available to some Linux folks.)

Of course a better benchmark would be the $50,000 NT/Dell box versus the $50,000 Sun/HP/DEC box, etc.
--

Does it matter? (1)

IntlHarvester (11985) | more than 15 years ago | (#1920232)

A few years ago, nobody in their right mind would have proposed any x86 (Novell, OS/2, NT, Linux)solution for anything other than workgroup filesharing or a ccMail postoffice. The fact that you can now seriously consider Linux or NT as a contender at the low end of the midrange market is primarily due to the advances in Intel hardware.

In a few years, Solaris, Tru64, HP/UX, Linux, and NT will all be running on essentially the same Intel IA64 hardware. At this point, the appeal of NT's one-size-fits-all design is going to start breaking down. But on the other hand, hardware equality is going to get Microsoft's salesmen in the door for midrange solutions that were previously above their heads. And Microsoft is more price competitive than commercial Unix, so NT deployment is probably going to increase in this market, not decrease. (Same argument for Linux.)

--

About Netcraft, semi-off-topic question (1)

IntlHarvester (11985) | more than 15 years ago | (#1920233)


If I understand correctly, those numbers are public webservers only. MS IIS's market strength has been internal Intranet solutions (where there's probably an existing NT file+print setup). IIIS's intranet market is probably going to being going up, not down, as things like the Office 2000 server get deployed.


--

Content is beside the point. (2)

IntlHarvester (11985) | more than 15 years ago | (#1920235)


The 'Rush Limbaugh' principle is a very valid point, especially in this context. Don't forget the target market for this study is Microsoft partners and WinNT-based shops.

Aside from all the meaninless numbers (who cares if your web server can saturate a 100BT line with static pages!), the study drives home an important point to NT Administrators - If you've invested in a high end IIS system, and you've got it tuned, there's probably no good reason to switch that box over to Linux. If the Linux box was tuned correctly, I doubt the difference would be that great performance-wise.

Of course, the study didn't address stability, which is the number one problem with IIS.
--

About Netcraft, semi-off-topic question (1)

Venomous Louse (12488) | more than 15 years ago | (#1920237)


Okay, so Netcraft says that Apache's market share is 1.3% greater than the previous month, and IIS's market share is 0.41% smaller.

But what does that mean?

Netcraft also says that the total number of web servers just exceeded five million. Is all of this Apache vs. IIS activity happening on existing web servers, on the new ones? Is Apache growing slowly-but-steadily across the board, or is it growing like a weed on new web servers, while market share on the existing ones remains frozen? That's good news, too, but it's different news. Among other things, it suggests that people aren't so unhappy with IIS that they're willing to put up with the annoyance of moving to a different server.

I dunno, I'm just wondering.


"Once a solution is found, a compatibility problem becomes indescribably boring because it has only... practical importance"

Content is beside the point. (5)

Venomous Louse (12488) | more than 15 years ago | (#1920238)


The truth or falsehood of the Mindcraft study is irrelevant to its intended audience. The point is to give NT "believers" something to quote in arguments, that's all. It's the Rush Limbaugh Principle. In a disagreement, it's helpful to have official-sounding statistics to back up your point. It doesn't matter where they came from, and it doesn't matter whether they're even remotely accurate. What counts is that somebody "important" (read "well-known") said it in public, which "validates" it. This "validation" isn't about truth. What it means is that the proper forms have been followed, and so it's acceptable to introduce the "evidence" in an argument. What's being offered is not evidence in the conventional sense, but the appearance of evidence, or the outward form of evidence. In poker, what does the four of diamonds mean? It means the four of diamonds. It's pure, disembodied symbol.

Disagreement and debate in our culture (especially on the net) isn't a whole lot less stylized (nor a whole lot less predictable) than Noh drama. You have to play by the rules and observe the forms. The content of the Mindcraft study is arbitrary. The study is a signifier, or token. A yacc parser says, "hey, this token is a function, hey, that one's an operator." The actual content of the token is not significant; what matters is what kind of token it is.

Everybody should learn at least a bonehead popularized minimum of semiotics (which is all I know, obviously :)

While we're at it, let's be honest with ourselves: How many of us are going to check Eric Raymond's facts for ourselves -- even to the minimal extent of clicking on the links he provides? And how many of us who don't check the facts are going to run around repeating them? Quite a few, probably. Dammit, I think Raymond's right on the money with this, and I'm confident that he's done his homework -- but I don't have the time to go about proving it. As far as many of us are concerned, Eric has given us a counter-signifier. Some "good spin" to match against the "bad spin". (That makes it sound dishonest, but IMHO if the "good spin" is factual and accurate, then "good" is a perfectly reasonable thing to call it.)

Think about it.



(Experienced sysadmins are a bit of a special case here. They can judge for themselves. The Limbaugh Principle applies mainly to people who are arguing in an area outside of their field of expertise -- I don't recall who it was who said that "every man is gullible outside his specialty", but it's true even of the best of us.)


"Once a solution is found, a compatibility problem becomes indescribably boring because it has only... practical importance"

a very fair test (1)

Xtacy (12950) | more than 15 years ago | (#1920239)

Why don't we just send a couple of linux guru's down to netcraft and tune the linux side of that machine? I'm sure they still have the NT side still rigged up. Then run the tests again and watch linux beat NT on every test!

GNUPro (1)

raistlinne (13725) | more than 15 years ago | (#1920240)

You might want to check out Cygnus's GNUPro utilities, which have lots of PII/PIII optimizations, I think. THey're fully GPL'd, too (not free, though, they cost $).

Raymond growing more pathetic by the hour (0)

Zico (14255) | more than 15 years ago | (#1920241)

p>Good job, Eric! Way to take the same things that have been said over and over and over and try to sound meaningful! Wow, just how long did it take you to come up with that? I can just see you sitting there, rubbing your low and protruding forehead, trying to think just how you can win back all that respect you've lost because of your recent pettiness. Let's see, should I attack all those "kiddies" at Slashdot again, maybe that'll do it. Oh wait, I know! I'll attack Microsoft! Absolutely brilliant!

Nice try, Eric, but pretty transparent. I've gotta give this one a thumbs down and await Halloween Letter LXII or maybe I Quit IV, aka "But Mommy, I Use Linux Too So Why Does Everyone Hate Me?"

Cheers,
ZicoKnows@hotmail.com

But how is Linux SMP (1)

sterno (16320) | more than 15 years ago | (#1920245)

Your comments validate the fact that Linux can make far better use of a lower end box, but it MIGHT indicate that Linux SMP is not too impressive. The Mindcraft box appears to be roughly four times the machine that you tested with. Yet the number of connections is only about 1.7 times as much. Granted you tuned your box and they didn't tune their box and may very well have crippled it, but still it may indicate some lack of SMP efficiency.

If somebody will buy me a system like Mindcraft used, I'll be more than happy to benchmark it myself! It might take me a while though, so be patient with me. If I have the box back to you in say 5 years, is that sufficient? :)

---

No, cjr made a referential mistake. (3)

dillon_rinker (17944) | more than 15 years ago | (#1920250)

You have taken a quote your first quote COMPLETELY out of the context of the article.

"Linux supporters have reacted violently to the Microsoft SA release (Independent research shows NT 4.0 outperforms Linux) published on ITWeb yesterday, saying 'the study was paid for by Microsoft' and that 'a very highly-tuned NT server was pitted against a very poorly tuned Linux server'. In response, Ian Hatton, Windows platform manager at Microsoft SA, says these comments are valid."

What a sham... (1)

Praxxus (19048) | more than 15 years ago | (#1920252)

No no! They just used a variation of "Inductive Reasoning" as outlined in Science Made Stupid by Tom Weller (copyright 1985). The outline is as follows, with Mindshaft's specific steps explained:

Formulate Hypothesis
In this case "NT is faster than Linux"

Apply for Grant
This is a bit out of sync, because I think the "grant" came in before the hypothesis was ever formulated

Perform Experiments/Gather Data to Test Hypothesis
Benchmark! Benchmark! Benchmark!

Alter Data to Fit Hypothesis
Here they varied from the outline again. Instead of altering the data, they altered the test conditions.

Publish
Press release, etc.

So while they varied from the outline in some areas, it is clearly a valid use of the inductive method. Of course, I had no idea anyone took Science Made Stupid so seriously....

--

So, is linux faster than NT on a 4-way w/ 2GB mem? (1)

Larry_Dillon (20347) | more than 15 years ago | (#1920259)

Well, not actually having one it's hard to say but... I personally would spend the same dollar ammount in a different way, like four boxes with 512MB RAM on a 100meg switch. Think of the possibilities!

Yes... But... (1)

rhuff (22750) | more than 15 years ago | (#1920260)

Ummm, how does one consider a four processor Intel box a "high-end" machine. To my way of thinking, that's very much a mid-end machine. A high end machine would be one with > 16 processors and a ton of memory.

That's an area where Linux truly doesn't play at the moment, but with SGI's donation of an O2K, we may see some big changes there soon. That said, I doubt if we'll ever have great scalability to lots of processors, simply because the user/developer community for those type of machines is comparatively small. Then again, maybe I'm wrong.

Microsoft's credibility (1)

BeemerBoy (24030) | more than 15 years ago | (#1920262)

Are you saying Microsoft has credibility? That's almost as unbelievable as the Mindcraft "study!"

Microsoft's "interest" in a real test... (1)

physics-boy (24181) | more than 15 years ago | (#1920263)

I agree that you would expect Mindcraft to want to clear their name by performing the test again, but it strikes me that your parenthetical comment, "But since microsoft isn't interested in a real test that isn't going to happen" may have missed the mark. If I were Microsoft, I would be very interested in a second, more fair test. I would be very interested to see that it didn't happen.

Food for thought.

No - you're wrong. (slightly) (1)

chris.bitmead (24598) | more than 15 years ago | (#1920266)

Don't forget that Linus Torvalds computer is something like a Quad-Xeon. I've got confidence that Linux on 4 processors would be pretty competitive. Better or worse than NT I don't know, but it wouldn't have to hide.

No more benchmark... a contest (3)

Le douanier (24646) | more than 15 years ago | (#1920267)

I think a contest would be better than a benchmark.

In a benchmark their are great odds that the benchmark will be sponsored by one of the party (M$ in this case).

If you do a contest, like the best ratio performance/price : you benchmark the performance of all the competing teams and then divide by the price the team involved in the hardware (not the software because due to Linux openness many people would say Linux price biased the contest).

If someone do so you can have a M$ team which will try to tune NT to is best, a Linux/Samba/Apache team which will try to tune Linux to his best, a Novell team, a Sun team...

You could choose your hardware so small team can try to compete. Even companies unrelated to NT/Linux/Novell/Another OS could compete so that can do a lot of publicity to these companies if they are well placed in the results.

It would be a good thing so every people supporting an operating system and so knowing how to tune it would be able to compete and their would be a greater range of results than in a single benchmark.

Of course we now need to find somebody to finance the contest :)

April, May, whatever (1)

chamont (25273) | more than 15 years ago | (#1920268)

Yesterday was 22 Apr, not 22 May. Maybe someone should change the block, in case someone picks this story up.

Monty

er (1)

chamont (25273) | more than 15 years ago | (#1920269)

Yesterday was 22 Apr, not 22 Mar. Maybe someone should change the block, in case someone picks this story up.

Monty

HOWTO for Performance tuning for high-end servers (0)

theHippo (28682) | more than 15 years ago | (#1920272)

I apologise if anyone's suggested this before, but perhaps what's needed is a HOWTO for performance tuning of high-end servers, with a special mention that benchmarkers are advised to adhere to the information in the document.

High end (1)

RasmusKaj (29279) | more than 15 years ago | (#1920273)

True, Linux (or FreeBSD [freebsd.org] , which I happen to preffer) is not really in the high end. Neiter is Microsoft. They should try to compare themself to a good system from SGI, Digital (compaq) or Sun.

The good thing here is that those "real" high-end servers run Un*x. There is alomost complete source-code comatibility (meaning you can just type make in most cases) from Linux / *BSD to those systems.

Please help me out here (1)

DonkPunch (30957) | more than 15 years ago | (#1920275)

I don't have a whole lot of experience with Suns and I have zero experience paying for them, but....

When I read the report, the first thing question that popped into my head was, "If I had the money to by a quad-Xeon Intel, why would I? Wouldn't it be smarter to put the money into a Sun?"

It seems to me that quad-Xeons are getting into the upper range of Intel-based machines. Why not go for a mid-range (low-end?) Sparc-based system. I'm sure Solaris would spank NT in uptime (and probably speed), and I would have the ability to scale up even further if necessary.

SMB would be a non-issue to me. I would just use an NFS client on the NT workstations (I've done this before).

I'm thinking more in terms of a real-world "Here is your budget. Get the best you can afford" situation. I'm a missing something here?

Microsoft's credibility (1)

remande (31154) | more than 15 years ago | (#1920276)

Microsoft has credibility by the plainest definition: they can get people to believe what they say. Specifically, they have credibility with many of the management types that determine what they're spending their software budget on this year.

As long as they're credible with the deep-pocket types, they don't have to be credible with the techie/geek types. Of course, if the techie/geeks start gaining credibility with the deep pockets at the expense of Microsoft...

So, is linux faster than NT on a 4-way w/ 2GB mem? (2)

remande (31154) | more than 15 years ago | (#1920277)

I am probably in the minority here, but I don't believe that it really matters how well an OS scales to a piece of hardware. It really matters how well an OS scales to a job.

The interesting question to me isn't "How much power can you get out of hardware X with OS Y", but rather "How much hardware do you need to throw OS Y on to do job Z".

From what I've been reading, NT does better SMP than Linux does. Frankly, Linux doesn't need SMP nearly as badly as NT. If uniprocessor Linux can do the same job as SMP NT, who cares how good or bad SMP Linux is?

DOJ lawsuit (1)

deacent (32502) | more than 15 years ago | (#1920282)

Not that their credibility is looking all that good as it is, but this can only hurt them in the anti-trust trial. Mindcraft was an "independent" lab that contradicted what Apple was saying about Microsoft trying to undermine Quicktime for Windows. Mindcraft released a fix that would make most of the problems go away, but I wonder if that fix wasn't given to them by Microsoft to patch Microsoft's code instead of Quicktime.

Stooping to their level (2)

Remus Shepherd (32833) | more than 15 years ago | (#1920283)

You don't want to fight FUD with FUD in that way. It's not just a matter of morality, either. Microsoft has clout because of their success as a big corporation with an established monopoly. They can afford to lose a little credibility by spinning a few lies. The Linux community has only one source of credibility -- that their stuff *works* -- and that's the very thing M$ is attacking. If you bend the truth and are caught, your credibility will suffer a lot more than Microsoft's. You'll be helping their FUD campaign, not hindering it.

Keep the high ground, folks. It's really in your best interests.

What a sham... (1)

aaronjb (33087) | more than 15 years ago | (#1920284)

"...we work with you to define test goals. Then we put together the necessary tools and do the testing. We report the results back to you in a form that satisfies the test goals."

"Define test goals"?

"..satisfies the test goals."?

Isn't the goal of testing to reach unbiased, truthful results? Mindcraft simply takes the cash and criteria, and gives you the results you want. I couldn't sleep at night if I worked for a company devoid of any morals.

---
aaron barnes
part-time dork

Microsoft's credibility (1)

Madhatter (33678) | more than 15 years ago | (#1920285)

Yeah, that's the way I've viewed this affair myself. The constant backlashing at Microsoft really isn't helping much. The IT guys I work with are literally in bed with Microsoft. They take what ever Microsoft says as gospel and their software as manna. Many people in IT groups just don't look at servers and software like we do. They want to keep buying the newest servers and the newest Microsoft OSs. It dosen't matter what you show them about Linux vs. NT benchmarks are whatnot. They take this one study, funded by MS and carried out by a reputable testing group, and say "look, I told you so".

In the end Microsoft is going to win in the majority of markets simply because you can't get the "big brother" feeling from open-source that most people seem to get from Microsoft. Linux seems on the right path for that. As long as we as developers and operators continue to support Linux and support the groups that are working to produce standards and support, the future looks pretty good.

Interesting Hatton comment (2)

cje (33931) | more than 15 years ago | (#1920287)

Hatton also admits that the Linux system would have performed better if it had been better optimised. "Having said that, I must say that I still trust the Windows NT server would have outperformed the Linux one."

Trust? Obviously you don't have too much confidence in NT, Ian.

Fight FUD with FUD (2)

T.E.D. (34228) | more than 15 years ago | (#1920288)

I see lots of calls for doing another benchmark that's "fair" to prove the Linux system superior. The problem with that is that it could never be "fair" if carried out by Linux partisans. Even if it were, likely there would be one missed tweak which would throw the whole thing in doubt.

Instead, why not fight FUD with FUD? Mindcraft claims the study's still valid even though the systems weren't tweaked equally. If that's the truly the case we're home free! Do a study designed to show how *badly* an NT server can be tweaked, and publish the results. As long as you promote the results as "just as valid as the Mindcraft benchmarks", you are being perfectly honest. :-)

So next time MS throws out the invalid Mindcraft survey (NT 2.5 times better), don't attack the survey. Just throw out the new Linux survey (Linux 153 times better) done using the "Mindcraft method".

Get some of the Big Boys to put on a REAL test (1)

Cpt_Kirks (37296) | more than 15 years ago | (#1920293)

A couple of the big companies that are pushing Linux (Oracle for instance) should have put on a new benchmark session with SEVERAL test systems. They should have a variety of hardware, from low to very high-end.

Get some NT and Linux gurus to tune the hell out of their respective OS (lets make this damn fair). Run a series of tests that reflect theoretical and real world conditions.

Get the tech media involved as much as possible and be very honest about it.

Write it up in several formats from guru to PHB level (lots of pretty charts).

Let the results speak for themselves. I would love to read them.

Fight FUD with FUD (1)

-1 (40015) | more than 15 years ago | (#1920295)

truly beutiful, completely evil

I love it . . . make it so

Suspicious (1)

GaspodeTheWonderDog (40464) | more than 15 years ago | (#1920296)

Doesn't it seem just a little odd that Microsoft is so blatant about this. Reminds me of a certain "hacker" type that likes to leave an obvious cookie trail to let the detectives find him and then says "yeah so, now what are you going to do?"...

Microsoft may be trying to snowball the general public, but I don't think they even care if we know what they are up to or not. Them's politics for ya.

How about a range of machines? (1)

Abigail (70184) | more than 15 years ago | (#1920305)

smich wrote: What does nt vs Linux look like on a 486 66 w. 8 meg of ram? Hmmm?

About the same as Linux vs DOS look like on a 286.

Such a test is pretty useless. It won't convince anyone to go Linux in their company, no more than it would when you port Linux to a toaster. Companies buy using either "we have so much to spend, what's the best we can buy for it?" or "we need this and this, what is the best deal we can get?". A 486 will seldom be an option.

--- Abigail

Oops! ESR Misqouted Hatton (1)

Timbo (75953) | more than 15 years ago | (#1920306)

Ian Hatton didn't say "A very highly tuned NT server was pitted against a very poorly tuned Linux server." That quote was an excerpt from an email received by Microsoft.

Why can't they run it again? (2)

demon-D (83540) | more than 15 years ago | (#1920307)

I remember that shortly after the report was published MindCraft semi-officially said something to the effect that : "if we were to run the test again....we would not make those optimizations" (e.g. the ones that slowed samba down).
My question is: Why cant they do it again? Just do the tests again....
I realize it will be expensive. But someone paid for it originally and came up with flawed results. Most companies would be looking at how to do it right the second time instead of saying
"Well ya know if we were to do it again we would not screw it up (But since microsoft isn't interested in a real test that isn't going to happen)"
Most companies do their best to cusion bad publicity. But microsoft seems to be proving time and again that ANY publicity is good.
Even if its bad.
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?