Software Code Quality Of Apache Analyzed 442
fruey writes "Following Reasoning's February analysis of the Linux TCP/IP stack (putting it ahead of many commercial implementations for it's low error density), they recently pitted Apache 2.1 source code against commercial web server offerings, although they don't say which. Apparently, Apache is close, but no cigar..."
So if they found them... (Score:5, Funny)
Re:So if they found them... (Score:4, Insightful)
I'm just glad I'm not the poor go-coder who has to go through the code to find and fix these few "errors."
Here are the links to the defect reports (Score:5, Informative)
Metric Report [reasoning.com]
They make you fill out a form that asks for your email and then do an opt out checkbox at the bottom of the form (you have to check it to NOT get spam from them). The site's a bit slashdotted right now though.
Re:So if they found them... (Score:5, Informative)
For instance, the first bug is
Each bug report is followed by the snippet of source code containing the defect.
The metric report simply reports the statistics. For instance, the most bug ridden file is otherchild.c. The most common bug class is "dereferencing a NULL pointer".
If the Apache developers simply want to fix the bugs, they can use the Defect Report. If they want conduct a brutal purge of their contributors, they can use the Metric report.
*Yes, Reasoning wants an email address. They will mail you a URL (a rather simple one at that) to access the reports.
Re:So if they found them... (Score:5, Insightful)
I mean, yeah, it would be nice if code would explicitly check for a NULL before dereferencing, but if there's no earthly way for the pointer to actually BE a NULL pointer at that time (barring memory corruption -- in which case all bets are off and your code is doomed anyway) then I wouldn't call those errors.
This whole exercise seems very suspect to me.
Re:So if they found them... (Score:5, Interesting)
int some_func(char *somebuf)
{
if (somebuf == NULL) return ERROR;
somebuf[0] = 'a';
return OK;
}
Will generate a warning with splint saying "pointer may be null" despite the fact it cannot be.
Those tools are generally too sensitive and give too many false positives to be useful in the long run.
Tom
Re:So if they found them... (Score:5, Informative)
Well, Yes and No. The problem is that there may be no logical way that the pointer may be NULL today. But tomorrow, a new coder will add something that modifies the preconditions and suddenly that pointer can indeed be NULL. Even where you are sure that a condition is impossible, it is usually a good idea to check for NULL in order to avoid future errors.
And for those who haven't seen this trick before, a nice habit to get into is to write your checks like so:
This lets the compiler catch errors where you meant '==' rather than just '='. As in
Re:So if they found them... (Score:3, Insightful)
That's what assert() exists for. And 'preconditions' you are referring to are actually 'invariants', so if "suddenly that pointer can indeed be NULL" it means that someone broke a fundmental design assumption and should not be tweaking the code anyway.
And for those who haven't seen this trick before, a nice habit to get into is to write your checks like so:..
I found this trick p
Microsoft C++ catches this. Doesn't gcc? (Score:3, Informative)
MY compiler (Microsoft C++) does catch this
and issues a warning. Doesn't gcc?Re:Microsoft C++ catches this. Doesn't gcc? (Score:4, Informative)
MY compiler (Microsoft C++) does catch this
if (myPointer = NULL) {
and issues a warning. Doesn't gcc?
Yes, it does. So does every other C compiler I've ever used (quite a few). I suspect the original poster may be the sort who ignores warnings....
Re:So if they found them... (Score:3, Insightful)
So shut up, you little twerp.
Re:So if they found them... (Score:5, Insightful)
Note that current_provider is set to conf->providers on line 257. The loop starts and neither current_provider or conf->providers change. Then on line 287 there's a conditional break if conf->providers is NULL.
If current_provider is going to be NULL at line 291, then conf->providers must be as well, so the conditional break will happen and the NULL dereference will be skipped.
Or am I missing something else?
Re:So if they found them... (Score:3, Interesting)
It says pretty clearly that they purposely chose a less mature sample of open source software than they did last time. The point is, does open source software start out bug free or do the bugs get worked out with age?
Re:So if they found them... (Score:5, Insightful)
The metrics report does mention the version number (dev-1/31/03), though the fact that this is development code is not explicitly noted No mentions is made who commissioned this study. Perhaps the company is simply fishing for clients.
apache 2.1? (Score:5, Interesting)
So the error level in pre-release Apache ... (Score:5, Insightful)
Re:So the error level in pre-release Apache ... (Score:5, Insightful)
That, too, but I'm damn certain that they must have tried it on recent stable 2.0.46ish release aswell. The question is, why weren't those results made public?
I'm guessing it's because the results were something that would've placed their "defect detection sw" into bad light. I.e. nothing as fancy as the forementioned "use of uninitialized variable" and "dereference of a NULL pointer" (which strikes really odd to me in the first place).
Naturally the other explanation is endorsement. It would be so much not-the-first-time that I don't even bother... but I wouldn't bet that this is the case here, because the defect counts were only compared to production release code averages (which strikes me as the other extremely dubious part of this whole "experiment").
Re:So the error level in pre-release Apache ... (Score:4, Insightful)
It's not fair! (Score:5, Funny)
Re:It's not fair! (Score:5, Funny)
inklude
dephine
retern
brake... etc.
Code defects appear to be a small part of the equa (Score:5, Insightful)
But here's the kicker: the vast majority runs Apache on either BSD or Linux. All of this code, from the kernel to the library that tells Apache how to use PHP, is open source. Every hacker on the planet has full access to the code - which means that they can review it and find vulnerabilities in it. Not many people have access to Windows or IIS code. So why does IIS and Windows come out as far less secure, and is exploited so much more?
I think the answer lies in the severity of the code defects, and the architecture and design of the operating system that powers the web server. And yes, I know that Apache can run on Windows.
what is a "software error"? (Score:5, Insightful)
First, are all of IIS's issues "software errors" per se? I'm wondering if all security problems would have been caught, or if that was really the goal of the analysis. Perhaps it was, but I'm not sure. One could contest that IIS has a lot of things unprotected, but that this doesn't constitute a software error.
And as you say, severity would be another issue. It's always been typical open-source style to get the mission-critical parts hardened against nuclear attack, but leaving the other bits a tad soft. I wouldn't be surprised to learn that was the case with apache.
One thing I want to know - did MS (or whoever) give these guys source or were they analyzing the binaries?
Re:what is a "software error"? (Score:5, Insightful)
IMNSHO, that ought to be standard for any mission-critical software. Bugs and the places that bugs live in are not created equal. The beauty of Apache (at least 1.13) is that the overall system can be very robust and reliable with rather buggy modules. I suspect the problem with IIS is that everything assumes everything else is perfect, which overall doesn't quite work so well.
automatically detected defects exclude security (Score:5, Insightful)
to be expected from Open Source (Score:4, Interesting)
So seems pretty clear to me that in Open source, the ratio of showstopper bugs to miscolored widget bugs will be much lower than for commercial software.
Re:Code defects appear to be a small part of the e (Score:5, Insightful)
The majority of the secruity holes are from the people setting up the web servers. The holes are usually abused by "wanna-be" hackers, or script-kiddies. The problem is that people are not educated enough to run some of these programs. Being able to understand Apache, and how to make it operate correctly is not everyone's top priority. As long as it works, people don't care how it works (as goes for many other things in this world).
Re:Code defects appear to be a small part of the e (Score:4, Interesting)
For the star wars geeks out there, if you were a Jedi, you don't go around telling everyone you're a Jedi, nor do you flash your light saber in public places. They do realize when to show their light saber, and when they can tell people they are a Jedi. Nor do they not tell anyone who they are, or never show their lightsaber.
You might want to check out Secrets and Lies [amazon.com] which will give you a better understanding of security philosphy.
Re:Code defects appear to be a small part of the e (Score:3, Insightful)
Nobody's saying that the information should be published - what they're saying is that you can't rely on that information being a secret.
Is Fort Knox secure? Probably. If so, then why don't they publish the blueprints, guard rotation schedule and security policies?
That's pretty much the point you're missing - even if that information was published, it wou
It's all in how you calculate a defect (Score:4, Insightful)
Apache is just a webserver, and that's all. PHP, JSP, etc, are all separate applications treated separately. The integration does make things more efficient, yes, but also more prone to problems.
Re:Code defects appear to be a small part of the e (Score:3, Interesting)
Is IIS just inherinetly insucure because it is used on a Windows platform? Is it because hackers generally target IIS and not Apache (most people will rush to this conclusion)?
Microsoft will try to make people belive whatever is in their interests .. Even if it means contradicting themselves ..
Last Friday Microsoft called all their Premier customers in France with "information" related to the upcoming "hackerfest" last Sunday.
According to Microsoft mostly Unix and Linux servers would be the target
Re:Code defects appear to be a small part of the e (Score:3, Interesting)
Maybe that's because the majority of web servers are running on Unix/Linux?
True, but according to statistics [attrition.org] 56% of defaced webservers run Microsoft IIS, and (only) 34% Apache..
This is not brand new data, but it is the latest I can find ... And If Microsoft had some stats showing different results, you can be sure they would publish them..
The competition was about defacing 6000 webservers in 6 hours, so one would tend to conclude from the above that Microsoft IIS would be the primary targets..
Re:Code defects appear to be a small part of the e (Score:3, Insightful)
Do you know how long it takes to read someone else's code on something like an Apache-level webserver and understand it to the point where you can make useful changes and fixes? The big lie of the "all bugs are shallow" argument is that such a thing is simple, when in fact it is not.
Fixing a non-obvious bug in a 100k or so line C or C++ project is hard enough when you wrote the
Re:Code defects appear to be a small part of the e (Score:4, Interesting)
One of the best ways to get to know a large code base like Apache or something else is to find a repeatable bug and track it down. To fix a bug you do not need to understand the whole program, just the relevent parts. I've submitted bug fixes to several projects, so I must strenuously disagree, especially because, ahem, I have never submitted a bug fix to a proprietary project because its impossible.
Re:Code defects appear to be a small part of the e (Score:4, Informative)
For example, I managed to code, test, and patch a "fix" for PostgreSQL this weekend in under 2 hours, having never seen the code before.
The "fix" wasn't a bug, per se, i't just that the output of pg_dump wasn't optimal in my usage for dumping the schema for CVS revision control. I added two flags, -m -M, which molded the output to my liking.
If you haven't seen your code in two months, you and an outsider have about the same chance at finding and detecting bugs/misfeatures.
Re:Code defects appear to be a small part of the e (Score:5, Interesting)
And not just the architecture of the web server, but the architecture of the entire platform. But specifically looking at the architecture of Apache versus the architecture of IIS, you'll immediately see that the goals of the two pieces of software are not the same. Look at things like IIS's metabase - the structural details of the server's configuration are kept in an in-memory data structure, which is easily modified while the server is running. Apache, in contrast, reads its configuration at startup, and uses it to determine which modules of code are loaded, and how they are used to process requests - fixing the behavior of the web server at startup.
IIS follows typical MS enterprise software design - it has to interface with COM, and the NT security model, and active directory, and the registry, and a million other systems, all in the name of integration, and enterprise management. Apache doesn't have PHBs telling it that it needs another way for the metabase to be edited, or a new instrumentation API, or whatever else a particular large customer asked for - and can get on with just providing its facilities cleanly.
That's why IIS has so many more security holes, even if it does (as may or may not be the case) have the same raw coding error rate as Apache.
Re:Code defects appear to be a small part of the e (Score:4, Informative)
Wait a second (Score:4, Insightful)
Re:Wait a second (Score:3, Interesting)
Has Apache 2.1 been released as a stable, non-developmental release?
According to the official site [apache.org]. ....
The latest 2.* relase is "2.0.46 " and version 2.1 is nowhere to be seen
So the question is : Which version did they audit ??
Re:Wait a second (Score:3, Funny)
Defect? (Score:5, Interesting)
so what are the calling a defect?
Re:Defect? (Score:5, Informative)
NULL Pointer Dereference (Expression dereferences a NULL pointer) 29 instances
Uninitialized Variable (Variable is not initialized prior to use) 2 instances
They also list the files and code snippets where the errors were found.
In addition, the comparison is made against an industry average of commercial code they have tested this way, NOT against other webservers.
How do they get to look at closed source? (Score:4, Interesting)
Or am I just too far out of that line of work to know how these things work?
2.1 ? (Score:4, Insightful)
What do reasoning do? (Score:5, Insightful)
This is probably a publicity stunt for them although a good one. I think it would be a good idea for them to sell software suites of their product if they don't already.
FACT: 3 is a larger number than 2 (Score:5, Insightful)
As far as I can see, this article says 'We have two arbitary numbers, and one is bigger than the other. From this we deduce that Apache is not as good as commercial software.'
Re:FACT: 3 is a larger number than 2 (Score:3, Insightful)
Re:FACT: 3 is a larger number than 2 (Score:3, Insightful)
Suppose I had 100K lines of code with 100 defects. After reviewing my code I discovered that I could refactor it to 80K lines and suppose further that doing so had no effect on the defect count. Defects per line of code would look worse after
FACT: Reading is Good (Score:5, Informative)
Actually the article suggests apache is better (Score:5, Insightful)
Apache 2.1...? (Score:5, Insightful)
Either way, to have only 31 errors in close to 60,000 lines of code is impressive!
Re:Apache 2.1...? (Score:3, Insightful)
Re:Apache 2.1...? (Score:3, Insightful)
Agreed. It would be interesting to know whether this low LOC is accomplished through good architecture that emphasizes simplicity and maintainability or "clever" hacks that compress a 10-line loop down into a three-line abomination of pointer arithmetic. I genuinely hope it is not the latter.
Regardless, 59K lines is small enough a program that--given a good architecture--can be studied and debugged relatively e
"Defect Density"? (Score:5, Insightful)
Since LOC is a poor metric, a "defect density" measurement based on that will be just as poor.
Yes, I know there's not much else to go on, but something along the lines of putting the program through its paces, stress testing, load testing, etc. would be a much better measurement than a metric based on LOC.
Open Source versus Closed (Score:3, Informative)
their own code? (Score:5, Funny)
Recursion (Score:3, Funny)
Re:Recursion (Score:5, Funny)
A magazine reviewed the product. In their review they included a formal mathematical proof that such a program could never work. The vendor responded to the proof by saying that they would fix that problem in the next release!
Re:Recursion (Score:3, Interesting)
Imagine you made the program go into an infinite loop whenever the program it was analysing did not have an infinite loop.
Them run the program on itself......
What kind of BS test is this? (Score:3, Interesting)
Why don't they compare it to apache 2.0.46 if they want a newer, but release product? I expect they did, but they didn't get the results they wanted.
This is a development version, it's an odd numbered release for crying out loud.
I wouldn't be suprised to see this is bankrolled by M$. Let's compare IIS in development to Apache 2.1, and then see what IIS bug density rate is.
Bah!!
Re:Confuse with Linux? (Score:3, Informative)
Apache 2.1 does not yet exist (Score:5, Informative)
I can only assume that they're looking through the current DEVELOPMENT codebase -- finding a higher ``defect density'' in such a development codebase compared with commercial offerings is not exactly unexpected.
They're also some automated code inspection product; the press release doesn't go into details as to the severity of the defects found or the testing methodology.
It'll be necessary to read through the full report [reasoning.com] before drawing any sound conclusions.
Re:Apache 2.1 does not yet exist (Score:5, Informative)
The direct URLs for the reports are:
Defect Report [reasoning.com]
Metric Report [reasoning.com]
Having read the reports.. (Score:5, Insightful)
Their automated checker also searched for out-of-bounds array accesses, memory leaks, and bad deallocations. It found none.
They also state that they ran the same checks against other codebases, and found that they did marginally better, on average.
In short, this report says that OLD development code for an unreleased opensource project is nearly as good as current commercial offerings. That's at best, when you consider the huge gamut of possible defects that this checker won't pick up. That margin probably disappears in the +/- of the sampling if you were to do a proper statistical analysis.
The report is fairly useless. It certainly should not be taken as a reason to not trust Apache; to do so would be foolhardy particularly given Apache's track record.
Oh, and Reasoning's webserver is being pounded into the ground. You can get my local copy of the reports from here [ic.ac.uk].
more to it than # flaws-per-unit-"whatever" (Score:5, Insightful)
What bothers me about these articles is that there is more to software quality than the # of flaws-per-unit-"whatever".
Like design.
It seems to me most of the problems with Apache's main competitor in terms of software quality are the result of design and engineering choices made by MS's IIS development team.
In other words, it does exactly what they designed it to do, but what they designed it to do was a very bad idea.
Interesting, with or without modules? (Score:4, Interesting)
I know that Apache has vulnerabilities but it should come better than IIS. You can't realisticly give a verdict on IIS without looking at the libraries called.
As for the rest, I can imagine some commercial products coming in better, but not many.
No cigar, my ass. (Score:5, Insightful)
The problems with this are:
Re:No cigar, my ass. (Score:4, Informative)
5100 != 58,944
58,944 is the number from the article.
BSD codestyle... (Score:3, Funny)
The defect density of the Apache code inspected was 0.53 per thousand lines of source code...
We can bring this number down to 0.2 by avoiding the BSD style guidlines. No kiddings, have you seen the density of MFC code?
BSD code:
char*
foo(int bar, double baz)
{
return bar + random();
}
MS code:
char* Foo(int nBar, double dBaz) { return bar + random() + m_ExtraWindowsBugModifier(); }
Wrong Math (Score:5, Insightful)
The longer and more content you have per line the higher the likelyhood of error/ line.
As example with one errror in 100 lines you get 1% error. Imagine you could do the whole thing in one line. Now you have 100% error.
Does it matter? (Score:5, Interesting)
So?
There are errors and there are errors. There are error that don't matter a jot, and there are errors that are show-stoppers.
I've worked on banking software containing code that was written in assembly for PD11s and developed over decades. The most horrible spaggetti code you could ever imagine. Why did the banks keep using it? Because for any particular input it always gave the correct output.
Years of bug fixing had made the code horrible and probably full of errors if you were looking at it from a purely theoretical/software engineering viewpoint. But from an input/output point of view, it was faultless.
That's so weird ... (Score:3, Interesting)
Since when are unfounded results from a company that doesn't explain what the "32 defects" were, newsworthy. Don't act like these guys are worth my time, this is bullshit.
Dubious (Score:5, Insightful)
If the company has developed proprietary tools to enable them to identify defects in medium-sized software projects, which of the following business models do you think is more effective:
1. Design proprietary tools to identify defects in medium-sized software projects.
2. Fix defects
3. Profit
or
1. Design proprietary tools to identify defects in medium-sized software projects.
2. Sit around mumbling about defects, Open Source software, closed source software and why farting in the bath smells worse
3. ???
4. Profit
Secondly, where on earth did they get hold of a closed source enterprise level (which Apache undoubtedly is) web server software codebase?
"Hi, is that BEA? Do you mind if we take a copy of your entire code base so that we can peer review it against Apache's? What's that? Yes, Apache might come out on top, and we will make the results public..."
How do they define a defect anyway? A memory leak? A missing overflow check? A tab instead of 4 spaces?
It just sounds like bullshit to me...
Different standards? (Score:5, Insightful)
If Apache is so poor in quality... (Score:5, Funny)
Of course it is Apache 1.3.23...
Bad Statistics... (Score:5, Insightful)
My general rule is that if someone is quoting statictics to you, they are lying. At least on average.
Re:Bad Statistics... (Score:5, Funny)
39% of Slashdot readers already know that.
In other news... I have begun testing (Score:5, Funny)
I compared this to my 'other' server, for now unnammed.
My 'other' server brought me coffee, 2 pieces toast, 2 eggs OVER EASY, 4 strips of bacon, *and* Smucker's Grape Jelly with nary a mistep, or hesitation. This other server smiled, asked how my wife was, and brought me a new fork when I dropped my first one.
Congratulations, Gloria! You win the 'great server' award!
This article isn't worth the 2 dollar tip.
Here's an idea (Score:5, Funny)
Well, seriously, I wouldn't put much in their obvious estimation.
Don't assume IIS (Score:5, Insightful)
It could also be Zeus, SunOne or one of the other lesser known web servers out there.
Apache 2 is not Apache 1 (Score:3, Insightful)
The test may be more interesting if applied to Apache 1. As someone who has had to migrate a mod_perl site from Apache 1 to Apache 2, I can tell you that Apache 2 is a very new beast, and it doesn't shock me at all that there are dozens of bugs that still need to be shaken out. Fewer users are running Apache 2 in a production environment as well, since it's considered a development branch. See less eyeballs rule.
Defect Details (Score:5, Informative)
Some things I found interesting:
One of the explanations (given by Reasoning) for a NULL pointer dereference is "can occur in low memory conditions," which I think means the original allocator did not check for malloc failure.
So you can get a sense of what a defect looks like, here is #21. The orignal uses bold and fonts improve readability, but I don't know how to reproduce that in slashcode:
DEFECT CLASS: Null Pointer Dereference
DEFECT ID 21
LOCATION: httpd-2.1/srclib/apr/misc/unix/otherchild.c : 137
DESCRIPTION The local pointer variable cur, declared on line 126, and assigned on line 128, may
be NULL where it is dereferenced on line 137.
PRECONDITIONS The conditional expression (cur) on line 129 evaluates to false.
sorry, but thats pure BS... (Score:3, Informative)
One of the explanations (given by Reasoning) for a NULL pointer dereference is "can occur in low memory conditions," which I think means the original allocator did not check for malloc failure.
appache got its own malloc() that kills the child (and closes connection) if it fails to allocate enough bytes.
Defects and maturity of code base (Score:5, Insightful)
I think the next step for these folks would be to take a project that has a long history, say perhaps Apache 1.x and show defect rates over the life of the project.
Null dereferences and uninitialized variables (Score:3, Informative)
Something is wrong here... (Score:3, Insightful)
There is no magic "defect detector" for software. If there was such a thing, they would be making a helluva lot more money than they get for doing little defect tests.
It is very difficult to prove a program to be correct, and there's a lot of REALLY smart people who have tried.
Maybe these people have stuff than can look for buffer overflows and stuff, but actually being able to tell if Apache is returning the correct results requires far more than generic tests.
And I'll all but guarantee they didn't get together an entire development team to understand the code base and how it works as apache is a very large and complex code base.
Maybe they take what the find for their generic tests and extrapolate that if they find more generic problems there are probably more specialized errors as well, but they make it very clear in the report that the difference between
Anyways, I'm not saying the entire thing is worthless, just not to read too much into it -- either this one that puts Apache slightly behind some unnamed commercial implementation or the one that put the Linux TCP/IP stack ahead of some other commercial implementation (though I'd say it would probably be easier to test a TCP/IP for correct behaviour than a web server).
This is a dupe (Score:3, Informative)
Lies, damned lies, and statistics (Score:5, Insightful)
1) Apache 2.1 has more bugs than some unknown commercial competitor. If the version is correct, a development (not-ready-for-release) build was pitted against a released commercial build. Not fair playing ground.
2) Reasoning does not detail the severity or kind of the bugs. Certainly, a web server not being able to handle a type of format (pdf, csv, ogg vorbis) is less severe than a security hole. Pitted against IIS, I would trust Apache even if it had more bugs, because historically it has had fewer security patches. Check out Apache's 2.0 known patches [apacheweek.com] vs IIS 5.0 [microsoft.com]
RTFAdvertising (Score:4, Insightful)
Heck, forget confidence - YOU CAN JUST CHECK.
The fact that Reasoning didn't have to go and get permission from Apache to run this test - coupled with the fact that we don't even know what Apache is being compared to - is the *real* point behind this "article".
ps: IANAL but don't they have to include a copy of the Apache License given that they publish fragments of the source code in their defect report?
Defect is too strong a word... (Score:5, Insightful)
I suspect the following code will be flagged as a defect:
as long as doOrDie() does its job and never returns a NULL then where's the defect? The guys who wrote this tester seem to want you to check any pointer dereferencing against NULL before use - I might be doing this in my doOrDie() function, I dont want to have to do it twice.BINGO (Score:3, Informative)
Apache has it's own malloc that kills the connection (and the child) if it fails.
That code can never be reached. Their test is invalid.
Re:Defect is too strong a word... (Score:3, Interesting)
!!conf->providers => conf->providers => conf->providers != NULL
Their program has detected "defects" where there are none. Perhaps the greater coding style variation on open source projects exposes more defects in their automated program!
Useless information presented confusingly (Score:4, Interesting)
*plonk*
Null pointers and uninitialized variables (Score:3, Insightful)
As a rather "stupid" example, I had to initialize a Map to an empty HashMap just last week to get Sun's Java compiler accept my code, although the only two references to the Map where within two if-blocks, within the same function, both of which depended on the same boolean value, which wasn't changed in the whole function.
There's a difference between defect and a bug. Tools that help in finding problems are great, but after all, they can only point possibly unsafe points. Ofcourse it's good to write code that doesn't trigger any such possibilities in the first place.
Thank you, Captain Obvious (Score:3, Funny)
Wow, next we'll learn how you shouldn't buy any Ford, GM, or Chrysler product in the first year of production.
Coding errors & program logic errors (Score:3, Insightful)
The most worrying errors in programs are generally not coding errors as they are either terminal (ie. crash) or they are benign (the error may cause memory corruption in a place where it does no harm). Of course, there are exceptions such as buffer overflows, but I'd class those, in general, into the logic error category.
Logic or algorythmic errors are far more dangerous as they can be well hidden and are more likely to make the code do things unintended. The code itself may be perfect but if the algorithm is faulty then there's a major problem.
Development release (Score:4, Insightful)
Re:Development release (Score:4, Insightful)
Why did they use the development branch of Apache
Let me restate this: why are they comparing pre-alpha software with production releases?
Most simple answer: because they wanted to find flaws. The second most popular web software is ISS. This looks like a Microsoft tactic: anonymously hire this company to "evaluate" code so that the results look unbiased. Everyone will likely realize that the competitor is Microsoft's ISS, so it doesn't need to be stated bluntly. MS wins; another (small) battle for mindshare is won.
Apache 1.3? (Score:5, Interesting)
For a valid comparison versus commercial software, the testers should have used Apache 2.0.46, the most current STABLE series release.
Second, I'd be interested to see a comparison of 2.0.46 versus 1.3.27. I have a pet theory that multithreaded C code has more bugs than single-threaded C code, and I'd like to see whether there is evidence to support it.
Worst kind of science (Score:5, Interesting)
Am I the only one who looks at reasoning's results with suspicion (even when I agree with them). Any analysis using methods that are not open and repeatable is not science. This just feels like marketing to me. (it is sad because the study of code quality is such a worthwhile pursuit)
prove it. (Score:4, Interesting)
when they tell us what they used, then I will believe it.
this smells microsoft.
bring it on! we want to know what it was compared against, sure as hell was NOT IIS...