Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Software Code Quality Of Apache Analyzed

Hemos posted more than 11 years ago | from the analyze-THIS dept.

Programming 442

fruey writes "Following Reasoning's February analysis of the Linux TCP/IP stack (putting it ahead of many commercial implementations for it's low error density), they recently pitted Apache 2.1 source code against commercial web server offerings, although they don't say which. Apparently, Apache is close, but no cigar..."

Sorry! There are no comments related to the filter you selected.

God hates fags and muslims! (-1, Offtopic)

Anonymous Coward | more than 11 years ago | (#6382645)

beware!

Love is warm and chunky! (-1)

I VOMIT ON TODDLERS! (642865) | more than 11 years ago | (#6382647)

Show some love all over a toddler today!

Thank God Hemos is back (-1, Offtopic)

Anonymous Coward | more than 11 years ago | (#6382648)

So many stories by michael recently I thought he pulled another censorware and locked the other editors out of Slashdot...

GNAA (-1, Troll)

Anonymous Coward | more than 11 years ago | (#6382650)

GAY NIGGERS LOVE APACHE

YOU join the GNAA!

This message brought to you by the GNAA (GAY
NIGGER ASSOCIATION OF AMERICA)

Make sure to stop by the official GNAA irc channel #GNAA
on EFNET.

If you have trouble locating #GNAA official irc channel, make sure
you are connecting to the proper IRC server.
for EFNET, you can connect to irc.secsup.org or irc.isprime.com

ANAQI~

So if they found them... (5, Funny)

Marx_Mrvelous (532372) | more than 11 years ago | (#6382652)

Why don't they fix them? It seems almost paradoxical, if you find .53 errors per thousands lines of code and fix them, then you'll have 0 errors. But since we can only fix errors we can detect, we only detect errors we can fix. Ok, it's too early on a Monday morning...

Re:So if they found them... (3, Insightful)

dkh2 (29130) | more than 11 years ago | (#6382824)

Sure, they found them but, did they catalog them in any way. .53/KLOC errors translates to approx. 1 error every 1886 LOC on average. On top of that, on further investigation, which of these are actual errors and which only look like errors?

I'm just glad I'm not the poor go-coder who has to go through the code to find and fix these few "errors."

The reports do document and explain the defects. (-1, Redundant)

Chuck Chunder (21021) | more than 11 years ago | (#6382886)

So yes, they have been identified and can be looked at quite easily.

Presumably the reasoning (heh) behind releasing these reports is so they can show their service is useful and drum up business.

apache 2.1? (5, Interesting)

fishynet (684916) | more than 11 years ago | (#6382657)

2.1 is'nt even out yet! the latest is 2.0.46!

So the error level in pre-release Apache ... (4, Insightful)

burgburgburg (574866) | more than 11 years ago | (#6382784)

is equivalent to the error level in post-release commercial web serving software. Sounds like an endorsement to me.

It's not fair! (5, Funny)

jpmahala (181937) | more than 11 years ago | (#6382659)

Just because Open-Source coders can't spell when they insert comments doesn't mean that they can't write good code!

Why is that modded down? (-1)

Anonymous Coward | more than 11 years ago | (#6382679)

IT'S TRUE!

Code defects appear to be a small part of the equa (4, Insightful)

mao che minh (611166) | more than 11 years ago | (#6382660)

I suppose now we have to question the severity of the defects (and also factor in the implementation and use of the code). If Apache and, say, IIS are roughly equivalent in terms of code defects, you have to ask yourself "well, why does IIS have so many more general problems and security flaws then Apache, when they both carry the same general amount of coding defects?". Is IIS just inherinetly insucure because it is used on a Windows platform? Is it because hackers generally target IIS and not Apache (most people will rush to this conclusion)?

But here's the kicker: the vast majority runs Apache on either BSD or Linux. All of this code, from the kernel to the library that tells Apache how to use PHP, is open source. Every hacker on the planet has full access to the code - which means that they can review it and find vulnerabilities in it. Not many people have access to Windows or IIS code. So why does IIS and Windows come out as far less secure, and is exploited so much more?

I think the answer lies in the severity of the code defects, and the architecture and design of the operating system that powers the web server. And yes, I know that Apache can run on Windows.

Re:Code defects appear to be a small part of the e (1, Funny)

demaria (122790) | more than 11 years ago | (#6382695)

Hypothesis: Taking down IIS, Windows or Microsoft is more fun/cool.

Re:Code defects appear to be a small part of the e (1, Insightful)

Anonymous Coward | more than 11 years ago | (#6382731)

For the same reason that windows boxes get hacked more often. The more a platform is used the more attacks on it.

Re:Code defects appear to be a small part of the e (4, Informative)

phre4k (627961) | more than 11 years ago | (#6382753)

Prette lame when we are talking server software where apache has the lead. (apache 63% vs IIS 25% netcraft.com)

/Esben

what is a "software error"? (5, Insightful)

siskbc (598067) | more than 11 years ago | (#6382745)

If Apache and, say, IIS are roughly equivalent in terms of code defects, you have to ask yourself "well, why does IIS have so many more general problems and security flaws then Apache, when they both carry the same general amount of coding defects?". Is IIS just inherinetly insucure because it is used on a Windows platform? Is it because hackers generally target IIS and not Apache (most people will rush to this conclusion)?

First, are all of IIS's issues "software errors" per se? I'm wondering if all security problems would have been caught, or if that was really the goal of the analysis. Perhaps it was, but I'm not sure. One could contest that IIS has a lot of things unprotected, but that this doesn't constitute a software error.

And as you say, severity would be another issue. It's always been typical open-source style to get the mission-critical parts hardened against nuclear attack, but leaving the other bits a tad soft. I wouldn't be surprised to learn that was the case with apache.

One thing I want to know - did MS (or whoever) give these guys source or were they analyzing the binaries?

automatically detected defects exclude security (5, Insightful)

brlewis (214632) | more than 11 years ago | (#6382762)

Another post seems to indicate this was done via software to automatically detect defects. Many (most?) security defects cannot be detected automatically, as they involve using the software in an unintended way.

to be expected from Open Source (3, Interesting)

Illserve (56215) | more than 11 years ago | (#6382788)

By its very nature, Open source will tend to fix important bugs and leave unimportant ones unfixed, while standard QA processes associated with commercial software will tend to fix little UI issues during the release schedule before dealing with vulnerabilities.

So seems pretty clear to me that in Open source, the ratio of showstopper bugs to miscolored widget bugs will be much lower than for commercial software.

Re:Code defects appear to be a small part of the e (5, Insightful)

jdh-22 (636684) | more than 11 years ago | (#6382887)

Every hacker on the planet has full access to the code - which means that they can review it and find vulnerabilities in it. Not many people have access to Windows or IIS code.
To quote Bruce Schneier: "If I had a letter, sealed it in a locked vault and hid the vault somewhere in New York. Then told you to read the letter, thats not secruity, thats obsecurity. If I made a letter, sealed it in a vault, gave you the blueprints of the vault, the combinations of 1000 other vaults, access to the best lock smiths in the world, then told you to read the letter, and you still can't, thats security." Open source does have an upper hand on holes and bugs, but the code isn't where we should be looking.

The majority of the secruity holes are from the people setting up the web servers. The holes are usually abused by "wanna-be" hackers, or script-kiddies. The problem is that people are not educated enough to run some of these programs. Being able to understand Apache, and how to make it operate correctly is not everyone's top priority. As long as it works, people don't care how it works (as goes for many other things in this world).

It's all in how you calculate a defect (3, Insightful)

sterno (16320) | more than 11 years ago | (#6382893)

The thing that always kills IIS, is the integration it has with Windows. This isn't a defect in IIS, or Windows, per se, but rather a defect that arises because of how they integrate with eachother. A script executes on IIS in a way that's not inately a bug, but then when it interacts with Windows, Exchange, etc, suddenly it becomes one.

Apache is just a webserver, and that's all. PHP, JSP, etc, are all separate applications treated separately. The integration does make things more efficient, yes, but also more prone to problems.

Re:Code defects appear to be a small part of the e (2, Interesting)

AftanGustur (7715) | more than 11 years ago | (#6382898)


Is IIS just inherinetly insucure because it is used on a Windows platform? Is it because hackers generally target IIS and not Apache (most people will rush to this conclusion)?

Microsoft will try to make people belive whatever is in their interests .. Even if it means contradicting themselves ..

Last Friday Microsoft called all their Premier customers in France with "information" related to the upcoming "hackerfest" last Sunday.

According to Microsoft mostly Unix and Linux servers would be the target of the hackers but it did not exclude IIS Web servers to come under attack.

The FUD coming from MS is absolutely unbeleavable..

Wait a second (3, Insightful)

Knife_Edge (582068) | more than 11 years ago | (#6382661)

Has Apache 2.1 been released as a stable, non-developmental release? If not I would say testing it for defects is a bit premature.

Re:Wait a second (2, Interesting)

AftanGustur (7715) | more than 11 years ago | (#6382843)


Has Apache 2.1 been released as a stable, non-developmental release?

According to the official site [apache.org] .
The latest 2.* relase is "2.0.46 " and version 2.1 is nowhere to be seen ....

So the question is : Which version did they audit ??

Defect? (5, Interesting)

Jason_says (677478) | more than 11 years ago | (#6382663)

Reasoning found 31 software defects in 58,944 lines of source code of the Apache http server V2.1 code.

so what are the calling a defect?

Re:Defect? (2, Funny)

Anonymous Coward | more than 11 years ago | (#6382701)

so what are the calling a defect?

I guess would be quite a good example.

Re:Defect? (0)

Anonymous Coward | more than 11 years ago | (#6382736)

lol

Re:Defect? (5, Informative)

richie2000 (159732) | more than 11 years ago | (#6382816)

From the report:
NULL Pointer Dereference (Expression dereferences a NULL pointer) 29 instances
Uninitialized Variable (Variable is not initialized prior to use) 2 instances

They also list the files and code snippets where the errors were found.

In addition, the comparison is made against an industry average of commercial code they have tested this way, NOT against other webservers.

Re:Defect? (1)

Malc (1751) | more than 11 years ago | (#6382875)

"Uninitialized Variable (Variable is not initialized prior to use) 2 instances"

How does this happen? I know the MS Visual C++ compiler has been issuing warnings for this for years.

How do they get to look at closed source? (3, Interesting)

3.5 stripes (578410) | more than 11 years ago | (#6382664)

And don't most NDAs for when they do let you look forbid any competetive analysis?

Or am I just too far out of that line of work to know how these things work?

it's a dupe!! (-1, Redundant)

Anonymous Coward | more than 11 years ago | (#6382666)

this story was posted b4. see here [slashdot.org] for the original!!

2.1 ? (4, Insightful)

Aliencow (653119) | more than 11 years ago | (#6382667)

Wouldn't that be unstable? I thought the latest was 2.0.46 or something.. If I'm not mistaken, it would be a bit like saying "Freebsd 4.8 has less bugs than Linux 2.5!"

Re:2.1 ? (1)

penguinblotter (599271) | more than 11 years ago | (#6382760)


These guys can't even round up !

Re:2.1 ? (1)

pthomsen (68685) | more than 11 years ago | (#6382829)

But the point is that they were testing "less mature" OSS against something commercial.

To me, the upshot is that even stuff that's still in development is about as "bug-free" as commercially available wares. A win for OSS in my book!

Re:2.1 ? (1)

Aliencow (653119) | more than 11 years ago | (#6382868)

By less mature I thought they only meant less, as in less than the TCP/IP stack itself, not "beta software" or anything..

What do reasoning do? (4, Insightful)

SystematicPsycho (456042) | more than 11 years ago | (#6382670)

So basically they offer a service like lclint [virginia.edu] only many times more advanced ? What is to say they haven't missed anything?

This is probably a publicity stunt for them although a good one. I think it would be a good idea for them to sell software suites of their product if they don't already.

Re:What do reasoning do? (1)

ichimunki (194887) | more than 11 years ago | (#6382845)

This was exactly my thought. How much of this is just an attempt to sell their proprietary software?

And the point is? (0)

Anonymous Coward | more than 11 years ago | (#6382672)

Would someone please tell me what the point of releasing an article comparing one known product against an un-named one?

Re:And the point is? (-1, Redundant)

hpavc (129350) | more than 11 years ago | (#6382874)

agreed, i dont see iplanet, iss, etc being examined line by here.

FACT: 3 is a larger number than 2 (4, Insightful)

TheRaven64 (641858) | more than 11 years ago | (#6382673)

Hmm, so they looked at 58,944 lines of code, and found 31 defects? Did they find every defect? Can they prove this? What about those found in commercial code? If it were possible to find all of the defects in a piece of code this big in a small amount of time, then there would be no defects, since they would all be identified and fixed before release.

As far as I can see, this article says 'We have two arbitary numbers, and one is bigger than the other. From this we deduce that Apache is not as good as commercial software.'

Re:FACT: 3 is a larger number than 2 (2, Insightful)

frankthechicken (607647) | more than 11 years ago | (#6382738)

Completely and utterly agree, I mean hell, I could write fifty thousand lines of code, each line completely and utterly with no meaning, run it through the checker and produce 0 defects, except for one overall defective piece of software. Does this article have any point whatsoever to it at all, I mean, even if the results had any meaning, what on earth is the point of comparing a known to an unknown ?

Re:FACT: 3 is a larger number than 2 (1, Funny)

rootofevil (188401) | more than 11 years ago | (#6382766)

Turing says no.

The article doesn't say that at all. (1)

Chuck Chunder (21021) | more than 11 years ago | (#6382772)

It says the results are "objective and comparable across software applications, development methodologies, and coding styles".

FACT: Reading is Good (5, Informative)

Cancel (596312) | more than 11 years ago | (#6382790)

That's not what they're saying at all. In fact, Reasoning concluded that there was no statistically significant difference in 'defect density' between Apache and the unnamed commercial product.
"In our February study that compared the defect density of the Linux TCP/IP stack to the average defect density of commercially developed TCP/IP stacks, we concluded that Open Source had a significantly lower defect density compared to commercial equivalents," said Bill Payne, President & CEO of Reasoning. "We received numerous inquiries about that study and took seriously requests for us to examine defect density rates in a less mature Open Source application and compare it with the commercial equivalent. Taking advantage of our database of automated software code inspection projects, we were able to do exactly that,
and found the difference in defect density between the two was not significant." (emphasis mine)

Actually the article suggests apache is better (4, Insightful)

sterno (16320) | more than 11 years ago | (#6382791)

This doesn't indicate that the commercial equivalents are better. You've got the DEVELOPMENT branch of Apache, which is derrived from the 2.0.x code which is a complete rework from the original 1.X branch of code. So it's a rather new code base and it's showing similar defect rates to a code base that has been around for a while. I'd say this prooves that open source is better.

Re:FACT: 3 is a larger number than 2 (1)

StrawberryFrog (67065) | more than 11 years ago | (#6382797)

Hmm, so they looked at 58,944 lines of code, and found 31 defects? Did they find every defect? Can they prove this?

Proving program correctness and bugfreeness is real hard. If they did find every defect and they can prove it, then I supect that it would be a significant breakthrough in Computer Science, not to mention a comercial goldmine.

As you can imagine, I am a bit sceptical.

That was not the conclusion: RTFA (2, Interesting)

arrogance (590092) | more than 11 years ago | (#6382867)

As others have stated, the article states that "the difference in defect density between the two was not significant." Meaning that defect density, especially with such a small differential, has little bearing on the overall quality of the software. We know nothing of the severity, impact, etc of the defects: they could all be cosmetic for all we know. This is probably nothing more than a marketing strategy by Reasoning: publish a study without any details on a hotly debated topic and see how many people check out their site. It'd be nice if they had a downloadable version of their software to test drive.

FxCop [gotdotnet.com] is an example of a "defect" or code analysis tool. While I have NO idea of Reasoning's methodology, I know that with FxCop (which is specifically for .NET code analysis), you have to set it up to filter out the majority of its rules or you'll get 3000 instances of "You didn't name this variable the way MS says you're supposed to." FxCop is extensible though. The point is, not a single poster on this page (unless they work for the companies involved) knows what Reasoning's methodology or rule set was when they did this so we can glean virtually zero value from this analysis. I look forward to 600 anti-Microsoft posts because of it though....

Re:FACT: 3 is a larger number than 2 (1)

imAck (102644) | more than 11 years ago | (#6382879)

If you read the actual report, it does cite what type of defects they looked for, and what they actually found.

29 NULL pointer dereferences

2 Unitialized variables


The unitialized variable is just a -Wall issue, the NULL pointer thing may or may not be serious depending on the context...

In term of looks Apache is quite good (0)

r6144 (544027) | more than 11 years ago | (#6382674)

I once traced sendmail's source code. Absolutely messy.

Re:In term of looks Apache is quite good (1)

Tony Hoyle (11698) | more than 11 years ago | (#6382860)

I've been through sendmail myself and it jumps around a bit but isn't too bad (once you've got your head around it it's relatively understandable). Some of the commercial code I've worked with looks like an explosion in a code factory...

Apache 2.1...? (4, Insightful)

bc90021 (43730) | more than 11 years ago | (#6382675)

According to Apache.org [apache.org] , Apache's latest stable version is 2.0.46. Is that a typo on their part, or are they testing a development version? Also, since 1.3.27 is widely used, it would have been interesting to see how that stacked up as well, having been developed longer.

Either way, to have only 31 errors in close to 60,000 lines of code is impressive!

Re:Apache 2.1...? (2, Funny)

Bu Na Dan (575203) | more than 11 years ago | (#6382712)

the error density in the announcement of reasoning.com is pretty high ... testing a non released software against an unknown commercial software ... sounds like an ancient tale. where are the people who accept this kind of crap?

Re:Apache 2.1...? (3, Insightful)

jbp4444 (193803) | more than 11 years ago | (#6382713)

I was quite impressed by the fact that Apache can cram all the functionality into ~59k lines. So besides defect rate, I would like to know how many lines of code the commercial package had ... 0.51 defects per 1000 lines sounds good, unless there are 1,000,000 lines more code in the commercial package.

"Defect Density"? (4, Insightful)

sparkhead (589134) | more than 11 years ago | (#6382676)

A key reliability measurement indicator is defect density, defined as the number of defects found per thousand lines of source code.

Since LOC is a poor metric, a "defect density" measurement based on that will be just as poor.

Yes, I know there's not much else to go on, but something along the lines of putting the program through its paces, stress testing, load testing, etc. would be a much better measurement than a metric based on LOC.

Re:"Defect Density"? (1)

p3d0 (42270) | more than 11 years ago | (#6382782)

LOC is a good measure of the effort involved in a project, and the complexity of the code base, but not anything else. Particularly, it certainly doesn't indicate how much functionality the code has.

So I consider LOC a metric to be minimized. For a given project, the less code the better, within reason.

Open Source versus Closed (3, Informative)

ElectronOfAtom (685701) | more than 11 years ago | (#6382680)

The difference is that now that someone has found 31 errors in the open source Apache software, they will be fixed fairly quickly whereas closed source software will have to have the company do a cost-benefit analysis, put together a team to do the fixes, probably charge to put out patches or minor upgrades (assuming the product is Microsoft's IIS ;b)...

their own code? (5, Funny)

Jearil (154455) | more than 11 years ago | (#6382681)

Why does it seem a bit odd to be testing software quality with other software? I wonder if they ran their own software through its own program, but then that gets kinda weird when a program starts noticing errors about itself... maybe it'd get depressed and start ranting at the creator on how they should have taken better care of it... ok, I need more sleep

Recursion (2, Funny)

sterno (16320) | more than 11 years ago | (#6382800)

They didn't do that because if they did that, then they'd find bugs in their bug finder, so they'd have to run the bug finder on the bug finder to find bugs there, but then they'd have to run the bug finder on the bug finder on the...

What kind of BS test is this? (2, Interesting)

dtolton (162216) | more than 11 years ago | (#6382685)

They are comparing a development version to an un-named commercial web server?

Why don't they compare it to apache 2.0.46 if they want a newer, but release product? I expect they did, but they didn't get the results they wanted.

This is a development version, it's an odd numbered release for crying out loud.

I wouldn't be suprised to see this is bankrolled by M$. Let's compare IIS in development to Apache 2.1, and then see what IIS bug density rate is.

Bah!!

Confuse with Linux? (1)

yerricde (125198) | more than 11 years ago | (#6382793)

This is a development version, it's an odd numbered release for crying out loud.

You refer to the version numbering rules used by the developers of the Linux kernel. Does Apache follow the numbering scheme of Linux?

Absolute crap (1, Interesting)

degradas (453730) | more than 11 years ago | (#6382694)

I can't think of any reason why should anybody trust this analysis until they publish the methods used. Anybody can say "Hey, I tested something using my proprietary method, and $foo has more bugs than $bar!". Unfortunately, such tests really don't say anything substantial about the quality of software. IMHO.

Apache 2.1 does not yet exist (4, Informative)

David McBride (183571) | more than 11 years ago | (#6382697)

Umm, Apache 2.1 hasn't been released yet. Current latest stable is 2.0.46 [apache.org] .

I can only assume that they're looking through the current DEVELOPMENT codebase -- finding a higher ``defect density'' in such a development codebase compared with commercial offerings is not exactly unexpected.

They're also some automated code inspection product; the press release doesn't go into details as to the severity of the defects found or the testing methodology.

It'll be necessary to read through the full report [reasoning.com] before drawing any sound conclusions.

Re:Apache 2.1 does not yet exist (4, Informative)

David McBride (183571) | more than 11 years ago | (#6382751)

The above link wants your email address. Bah.

The direct URLs for the reports are:
Defect Report [reasoning.com]
Metric Report [reasoning.com]

Having read the reports.. (4, Insightful)

David McBride (183571) | more than 11 years ago | (#6382862)

Well, the reports simply state that, in the 360 files they checked (most of them header files) they found 29 cases of a potential NULL pointer dereference and 2 potentially uninitialized variables. This is from the Apache 2.1 codebase as of 31st Jan this year, about 58k lines of code.

Their automated checker also searched for out-of-bounds array accesses, memory leaks, and bad deallocations. It found none.

They also state that they ran the same checks against other codebases, and found that they did marginally better, on average.

In short, this report says that OLD development code for an unreleased opensource project is nearly as good as current commercial offerings. That's at best, when you consider the huge gamut of possible defects that this checker won't pick up. That margin probably disappears in the +/- of the sampling if you were to do a proper statistical analysis.

The report is fairly useless. It certainly should not be taken as a reason to not trust Apache; to do so would be foolhardy particularly given Apache's track record.

Oh, and Reasoning's webserver is being pounded into the ground. You can get my local copy of the reports from here [ic.ac.uk] .

Links to the Reports (no free reg required) (2, Informative)

Anonymous Coward | more than 11 years ago | (#6382702)

AC, thank you for contacting Reasoning!

Here are the links to the Apache Open Source Inspection Report you requested:

Apache Defect Report: http://www.reasoning.com/pdf/Apache_Defect_Report. pdf [reasoning.com]
Apache Metric Report: http://www.reasoning.com/pdf/Apache_Metric_Report. pdf [reasoning.com]

Reasoning provides the world's leading automated software inspection service. We boost the productivity of development teams by finding software defects faster and at a far lower cost than traditional approaches. Please let me know if you would like additional information. Thank you again for contacting Reasoning!

Sincerely,
Reasoning

more to it than # flaws-per-unit-"whatever" (5, Insightful)

Asprin (545477) | more than 11 years ago | (#6382703)


What bothers me about these articles is that there is more to software quality than the # of flaws-per-unit-"whatever".

Like design.

It seems to me most of the problems with Apache's main competitor in terms of software quality are the result of design and engineering choices made by MS's IIS development team.

In other words, it does exactly what they designed it to do, but what they designed it to do was a very bad idea.

Interesting, with or without modules? (3, Interesting)

hughk (248126) | more than 11 years ago | (#6382707)

If anyone has an Apache 2.1 dist around, they say they checked 58,000 lines - does this seem reasonable? Is this with any of the modules such as PHP or Perl or is this raw????

I know that Apache has vulnerabilities but it should come better than IIS. You can't realisticly give a verdict on IIS without looking at the libraries called.

As for the rest, I can imagine some commercial products coming in better, but not many.

No cigar, my ass. (5, Insightful)

KFury (19522) | more than 11 years ago | (#6382710)

The article claims Apache's error density, based on a meager 5100 lines of code, is 0.53, while that of 'comparable commercial applications' is 0.51.

The problems with this are:
  • 5100 lines of code does not give you a confidence range of less than 0.02, especially when the error rate can be expected to be heterogeneous across the code base, as would be the case in an open-source product where different code pieces are created by entirely different groups.
  • 'Comparable' my ass. If they can't provide details of what software they're comparing to (I somehow doubt they got a look at IIS source code) then the stats are worthless, because anyone who's ever programmed knows that quality control isn't a constant factor across commercial products any more than it is among open-source products.
  • What's the error rate of their 'defect analysis'? If they're so good at finding defects, why aren't they out there writing perfect software? If their defect detection rate is less than 98% accurate, then the difference between a rate of 0.51 and 0.53 is meaningless anyhow.
  • There's a big difference between caught coding exceptions and fundamental security problems. The first can cause code to run a little slower, the second can destroy your company. This testing methodology doesn't even look at the second.

Correction: 58,000 lines of code (1)

KFury (19522) | more than 11 years ago | (#6382742)

Still, mitigate that with the pre-release status of Apache 2.1 and it cancels out.

Re:No cigar, my ass. (3, Informative)

HowlinMad (220943) | more than 11 years ago | (#6382768)

FYI

5100 != 58,944

58,944 is the number from the article.

BSD codestyle... (3, Funny)

BigBadDude (683684) | more than 11 years ago | (#6382719)


The defect density of the Apache code inspected was 0.53 per thousand lines of source code...


We can bring this number down to 0.2 by avoiding the BSD style guidlines. No kiddings, have you seen the density of MFC code?

BSD code:

char*
foo(int bar, double baz)
{

/* do something */
return bar + random();

}



MS code:

char* Foo(int nBar, double dBaz) { return bar + random() + m_ExtraWindowsBugModifier(); }

Re:BSD codestyle... (1, Interesting)

Anonymous Coward | more than 11 years ago | (#6382755)

have you seen the kernel code?

about half of it is "comments", that are really arguments/fights between Linus, Cox and Russell.
(no kidding, if you ever read the kernel mailinglist you should already know this).

Did they include comments in the test?

Wrong Math (4, Insightful)

bstadil (7110) | more than 11 years ago | (#6382803)

You got the math reversed

The longer and more content you have per line the higher the likelyhood of error/ line.

As example with one errror in 100 lines you get 1% error. Imagine you could do the whole thing in one line. Now you have 100% error.

Re:Wrong Math (2, Informative)

BigBadDude (683684) | more than 11 years ago | (#6382849)

yeah, that was actually my point. nice someone got it :)

The source of most free software [KDE is an exception] tend to be smaller, more readable and more effective. Ever wondred why winword.exe is 10.598.984 bytes?

This is an ad for their software (2, Insightful)

Sikmaz (686372) | more than 11 years ago | (#6382725)

This looks like it was just an ad/demo of their code testing software.

I am trying to get the main analysis downloaded now, but they must have been prepared for a slashdot posting ;)

code defects? (1, Insightful)

Anonymous Coward | more than 11 years ago | (#6382728)

I see the point in automatically checking the
source code for common programming errors,
but how can such a system ever find semantic
errors, such as complicated protocol handling
issues?
It seems to me that those just happen to be
strong points of open source software.

Does it matter? (5, Interesting)

pubjames (468013) | more than 11 years ago | (#6382730)


So?

There are errors and there are errors. There are error that don't matter a jot, and there are errors that are show-stoppers.

I've worked on banking software containing code that was written in assembly for PD11s and developed over decades. The most horrible spaggetti code you could ever imagine. Why did the banks keep using it? Because for any particular input it always gave the correct output.

Years of bug fixing had made the code horrible and probably full of errors if you were looking at it from a purely theoretical/software engineering viewpoint. But from an input/output point of view, it was faultless.

THIS IS FUD! (0)

Anonymous Coward | more than 11 years ago | (#6382732)

This comparision is tottaly fishy!

Apostrophe abuse (1, Funny)

worst_name_ever (633374) | more than 11 years ago | (#6382734)

putting it ahead of many commercial implementations for it's low error density

...but behind the Slashdot editors in terms of number of abuses of the word "its" per story.

That's so weird ... (3, Interesting)

SuperDuG (134989) | more than 11 years ago | (#6382752)

I found just the opposite.

Important Tech City, CA, July 7th 2003
For Immediate Release
Sbj: Apache beats other webservers

Recently we had our staff (some guys kid) look over the source code of 3 major webserver packages available, in that code nearly 8 million lines of error were found, but surprisingly the damned things still worked?!

We placed a performance test (click a link and see if porn comes faster) with apached and 3 other commercial offerings. Apache seemed to knock them all of the water, boy will those other three companies be mad now.

While we cannot tell you what the other three offerings were (that might make this whole thing more believeable) we can tell you that we think they're popular.

Here's the results

Apache ------------------- 104
Com 1 --------32
Com 2 -----------45
Com 3 ---------------53

As you can see by the clear test results, apache wins in all tests.

Since when are unfounded results from a company that doesn't explain what the "32 defects" were, newsworthy. Don't act like these guys are worth my time, this is bullshit.

Dubious (4, Insightful)

cca93014 (466820) | more than 11 years ago | (#6382754)

Is it just me that finds this entire concept of "code defects per 000 lines" sounding like a little bullshit?

If the company has developed proprietary tools to enable them to identify defects in medium-sized software projects, which of the following business models do you think is more effective:

1. Design proprietary tools to identify defects in medium-sized software projects.
2. Fix defects
3. Profit

or

1. Design proprietary tools to identify defects in medium-sized software projects.
2. Sit around mumbling about defects, Open Source software, closed source software and why farting in the bath smells worse
3. ???
4. Profit

Secondly, where on earth did they get hold of a closed source enterprise level (which Apache undoubtedly is) web server software codebase?

"Hi, is that BEA? Do you mind if we take a copy of your entire code base so that we can peer review it against Apache's? What's that? Yes, Apache might come out on top, and we will make the results public..."

How do they define a defect anyway? A memory leak? A missing overflow check? A tab instead of 4 spaces?

It just sounds like bullshit to me...

Different standards? (5, Insightful)

NotClever (635709) | more than 11 years ago | (#6382757)

When the same group said that the IP stack in Linux was cleaner than a comparable one, everyone was screaming from the rooftops that it validated the open source model. When they say that an open source project and a closed source project are roughly comparable, all of a sudden everyone criticizes the methodology of the report!

Defect Density Indeed! (0)

Anonymous Coward | more than 11 years ago | (#6382779)

You compare 50k lines of semi-optimized beta code against 1000k lines of a "commercial" product. This is why statistics are the best liars. What does it take to get the "commercial" product to lower the contamination in parts per thousand?

Bloat.

Add a few more comments as unessential error checking, hell, add DRM to check to see if you are hosting the lastest Emimem MP3s. That should do it.

If anything is defective or dense, it's the people who came up with the statistics for the sake of PR.

If Apache is so poor in quality... (4, Funny)

tsetem (59788) | more than 11 years ago | (#6382783)

...then why is it their webserver [netcraft.com] ? :)

Of course it is Apache 1.3.23...

Bad Statistics... (5, Insightful)

FunkZombie (322039) | more than 11 years ago | (#6382787)

Also keep in mind that defect density is just an average. If you have 31 defects in 60k lines of code, that is potentially 31 security risks, or out-of-operation risks. If the other software tested had double the lines of code (120k), the density would imply that they had slightly less than double the defects, so say 58 or 60. That implies _58_ potential security or uptime risks. In this case, imho, defect density is not a good indicator of the reliablity of the software.

My general rule is that if someone is quoting statictics to you, they are lying. At least on average. :)

Re:Bad Statistics... (4, Funny)

Lxy (80823) | more than 11 years ago | (#6382856)

My general rule is that if someone is quoting statictics to you, they are lying. At least on average. :)

39% of Slashdot readers already know that.

Magic software (1)

pubjames (468013) | more than 11 years ago | (#6382804)


So if they can write software to automatically spot coding errors, then it must be possible for them to automatically fix them, no?

In other news... I have begun testing (4, Funny)

teamhasnoi (554944) | more than 11 years ago | (#6382809)

Apache 4.2 Alpha, a release that is yet to be even a twinkle in it's Daddies' eyes. I have found a whole bunch of errors, bad comments, a few scribbles on napkins, some old Populous save games, and a letter to 'Mom' asking for money.

I compared this to my 'other' server, for now unnammed.

My 'other' server brought me coffee, 2 pieces toast, 2 eggs OVER EASY, 4 strips of bacon, *and* Smucker's Grape Jelly with nary a mistep, or hesitation. This other server smiled, asked how my wife was, and brought me a new fork when I dropped my first one.

Congratulations, Gloria! You win the 'great server' award!

This article isn't worth the 2 dollar tip.

Dupe (1)

Sardonis (596687) | more than 11 years ago | (#6382812)

here [slashdot.org]

PWP!!!! (-1, Offtopic)

JoshMarotti (685722) | more than 11 years ago | (#6382817)

I miss page widening...

scope (0, Redundant)

thoolihan (611712) | more than 11 years ago | (#6382818)

I wonder what scope of errors they are looking at? For instance, are they counting assignment errors (overflow), IIS->Com higher level type errors, or both.
-t

Here's an idea (4, Funny)

Daath (225404) | more than 11 years ago | (#6382821)

Why doesn't Reasoning fill the niche, and code a completely error free web server? They know other peoples mistakes, so they should know how to code an error free one.
Well, seriously, I wouldn't put much in their obvious estimation.

Don't assume IIS (5, Insightful)

m00nun1t (588082) | more than 11 years ago | (#6382828)

Ok, IIS is the obvious choice as being the second most popular web server after Apache. But I hardly think Microsoft will be letting these guys all over the IIS source code.

It could also be Zeus, SunOne or one of the other lesser known web servers out there.

Apache 2 is not Apache 1 (2, Insightful)

defile (1059) | more than 11 years ago | (#6382833)

The test may be more interesting if applied to Apache 1. As someone who has had to migrate a mod_perl site from Apache 1 to Apache 2, I can tell you that Apache 2 is a very new beast, and it doesn't shock me at all that there are dozens of bugs that still need to be shaken out. Fewer users are running Apache 2 in a production environment as well, since it's considered a development branch. See less eyeballs rule.

Defect Details (5, Informative)

Eustace Tilley (23991) | more than 11 years ago | (#6382842)

Interested persons can download the full defect report free of charge. [reasoning.com]

Some things I found interesting:
  1. Apache 2.1 (dev) is a mere 76,208 LOC.
  2. No memory leaks detected
  3. 29 NULL pointer dereferences
  4. 2 Uninitialized variables
  5. No bounds errors, no bad deallocs
  6. otherchild.c had a rate of 7 NULL pointer dereferences per 1000 KSLC


  7. One of the explanations (given by Reasoning) for a NULL pointer dereference is "can occur in low memory conditions," which I think means the original allocator did not check for malloc failure.

    So you can get a sense of what a defect looks like, here is #21. The orignal uses bold and fonts improve readability, but I don't know how to reproduce that in slashcode:
    DEFECT CLASS: Null Pointer Dereference

    DEFECT ID 21

    LOCATION: httpd-2.1/srclib/apr/misc/unix/otherchild.c : 137

    DESCRIPTION The local pointer variable cur, declared on line 126, and assigned on line 128, may
    be NULL where it is dereferenced on line 137.
    PRECONDITIONS The conditional expression (cur) on line 129 evaluates to false.
    CODE FRAGMENT
    124 APR_DECLARE(void) apr_proc_other_child_unregister(void *data)
    125 {
    126 apr_other_child_rec_t *cur;
    127
    128 cur = other_children;
    129 while (cur) {
    130 if (cur->data == data) {
    131 break;
    132 }
    133 cur = cur->next;
    134 }
    135
    136 /* segfault if this function called with invalid parm */
    137 apr_pool_cleanup_kill(cur->p, cur->data, other_child_cleanup);
    138 other_child_cleanup(data);
    139 }

sorry, but thats pure BS... (3, Informative)

BigBadDude (683684) | more than 11 years ago | (#6382892)


One of the explanations (given by Reasoning) for a NULL pointer dereference is "can occur in low memory conditions," which I think means the original allocator did not check for malloc failure.


appache got its own malloc() that kills the child (and closes connection) if it fails to allocate enough bytes.

Defects and maturity of code base (4, Insightful)

the eric conspiracy (20178) | more than 11 years ago | (#6382847)

This study makes a lot of sense to me - that the defect rate is tied to the maturity of the code base. I have long felt that Microsoft's business model where they redo the operating system in order to churn their user base and induce cash flow will always result in more defects and security problems than a model where software change is driven on a solely technical basis.

I think the next step for these folks would be to take a project that has a long history, say perhaps Apache 1.x and show defect rates over the life of the project.

i tried to submit this story (-1)

Anonymous Coward | more than 11 years ago | (#6382853)

look.. it got rejected a week ago: my story [slashdot.org]

Null dereferences and uninitialized variables (2, Informative)

ByTor-2112 (313205) | more than 11 years ago | (#6382857)

29 possible "null dereferences" and 2 possible "uninitialized variables". Some of them are simple "fail to check return value of malloc() for null", and others are not bugs in the code but bugs in the logic of the scanner. This is, of course, a precursory review of their document. All in all, these are absolutely minor bugs if they are real at all.

WHAA? (0)

Anonymous Coward | more than 11 years ago | (#6382878)

"for it's low error density"

for it is low error density?

TARD.
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?