Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

450 Million Lines of Code Can't Be Wrong: How Open Source Stacks Up

timothy posted about a year ago | from the all-a-blur-at-this-point dept.

Bug 209

An anonymous reader writes "A new report details the analysis of more than 450 million lines of software through the Coverity Scan service, which began as the largest public-private sector research project focused on open source software integrity, and was initiated between Coverity and the U.S. Department of Homeland Security in 2006. Code quality for open source software continues to mirror that of proprietary software — and both continue to surpass the industry standard for software quality. Defect density (defects per 1,000 lines of software code) is a commonly used measurement for software quality. The analysis found an average defect density of .69 for open source software projects, and an average defect density of .68 for proprietary code."

cancel ×

209 comments

Sorry! There are no comments related to the filter you selected.

Correction (4, Insightful)

Sla$hPot (1189603) | about a year ago | (#43653161)

"450 Million Lines of Code Can't Be Wrong"
should have been
"450 Million Lines of Code Can't ALL Be Wrong"

Re:Correction (1)

bobbied (2522392) | about a year ago | (#43653303)

You mean: "exit(-1);" ?

Re:Correction (4, Funny)

hduff (570443) | about a year ago | (#43653945)

It mean over 300,000 lines of code are wrong, most of it in the app I keep trying to use.

Re:Correction (0)

Anonymous Coward | about a year ago | (#43655099)

It mean over 300,000 lines of code are wrong, most of it in the app I keep trying to use.

Not necessarily wrong, it could be that it is insecure or exploitable.

Re:Correction (0)

Anonymous Coward | about a year ago | (#43655165)

The solution is simple. Add more meaninglessly verbose lines!

Write

if (blah.something(nain,jahaha,verboten)) while (betty == evil) kill(kayne);

like this


if
    (
        blah.something(
            nain,
            jahaha,
            verboten
        )
    )
{
    while
        (betty == evil)
    {
        kill(
            kayne
        );
    }
}

And when they complain, reply with the usual thought-terminating "Think of the retards! " clichee, and get the bonus of implying all your coworkers are morons who can't even comprehend the former version, while making them say sorry to *you*. (At least if they follow standard "social" norms.)

P.S.: Inb4 "Your English is bad.": I know. It's my fourth one our of five, and I still have my bad moments when it's three in the morning. But I felt like reading Luxemburgish is a bit too much for the average Slashdotter. ^^

65 million lines of HOST file can't be wrong (3, Funny)

Anonymous Coward | about a year ago | (#43653177)

Just ask apk!

Re:65 million lines of HOST file can't be wrong (0, Troll)

Anonymous Coward | about a year ago | (#43653407)

Just ask apk!

You mean Jeremiah Cornelius?

Re:65 million lines of HOST file can't be wrong (0)

Anonymous Coward | about a year ago | (#43654149)

I see one of his sock puppets voted me down.

Defects fixed for proprietary may differ. (2, Informative)

CodeReign (2426810) | about a year ago | (#43653185)

Propietary defects are ones that may cause financial harm. FOSS defects are ones that cause annoyance.

I know that our code has more defects than we'd consider fixing purely because the CBA isn't there.

Re:Defects fixed for proprietary may differ. (-1)

Anonymous Coward | about a year ago | (#43653299)

FOSS defects can cause financial harm too. Stop being such a fanboy.

Re:Defects fixed for proprietary may differ. (-1)

Anonymous Coward | about a year ago | (#43653521)

How do you know if he's a fanboy? Maybe he meant that errors in FOSS don't cause financial harm because nobody uses it for serious work. Which would be just as wrong, but would make him the exact opposite of a fanboy.

Re:Defects fixed for proprietary may differ. (1)

Vanderhoth (1582661) | about a year ago | (#43654195)

A fangirl?

Re:Defects fixed for proprietary may differ. (1)

Dishevel (1105119) | about a year ago | (#43654353)

A fangirl?

No. The exact opposite would be a HateBitch.

And eating toast CAN cause cancer. (0)

Anonymous Coward | about a year ago | (#43654025)

What, however, are the propensities for it compared to other sources?

Re:Defects fixed for proprietary may differ. (4, Insightful)

Cenan (1892902) | about a year ago | (#43653333)

Propietary defects are ones that may cause financial harm. FOSS defects are ones that cause annoyance.

I know that our code has more defects than we'd consider fixing purely because the CBA isn't there.

I'm guessing you mean defects in propietary software only gets fixed if they have an impact on the bottom line? Otherwise that whole reply makes no sense.

Anyways, that is not much different from the OSS model. Whoever cares about the sub-system that has a bug, fixes it, and if nobody cares (or has the skills to fix it) it can go ignored for years. The selector for OSS is different, but the end result is the same: nobody gives a fuck about the end user unless it directly affects their day/paycheck/e-peen.

Re:Defects fixed for proprietary may differ. (2)

Bigby (659157) | about a year ago | (#43654527)

No, I think the GP is getting at the point that code analyzed in the analysis likely includes critical proprietary software. Software that needs to work and so they invest the time in making sure it does.

Meanwhile, the open source side probably included code that is not critical, based on reverse engineering, or experimental in nature. Not that both the proprietary and open source code bases didn't contain both, but I think the context of the code is quite different.

The results would be much more meaningful if they compared the code quality of GIMP vs Photoshop, Firefox vs IE, or Linux/X/KDE/Gnome vs Windows. Comparing anything in the open source world to code using in NASA missions or medical devices is bad, especially when you include those with useless stuff like xclock or some open source tetris.

Re:Defects fixed for proprietary may differ. (1)

Anonymous Coward | about a year ago | (#43654031)

My company uses FOSS, and I can assure you that there have been bugs in the FOSS software that have caused financial harm.

Conversely, there are bugs in proprietary software that we use that are annoying and have to be worked around, but cause us no great financial difficulty, because they can be worked around.

FOSS has no exclusive claim to "only has defects that will annoy you," just as proprietary software has no exclusive claim to "only containing defects that will affect your bottom line."

I'm honestly not sure if you're trying to knock FOSS, or jock FOSS, but either way - you're completely wrong.

Fight'n Words (1)

Anonymous Coward | about a year ago | (#43653189)

Code quality for open source software continues to mirror that of proprietary software ....

That thar is fight'n words, pardner!

Open Source is SUPERIOR!

Re:Fight'n Words (2)

bzipitidoo (647217) | about a year ago | (#43654403)

Sure makes the emacs vs vi wars look petty. This is a religious dispute between the believers in greedy capitalism, who think such forces lead to the best balance of highest possible quality at reasonable expense, in all endeavors, and everyone else.

The greedy capitalists think that if you aren't sweating and stressing over your job and the money it provides to feed your hungry children, not to mention your house and car payments, fearing that the loss of your job will ruin your career so that you will never be able to find another, then you aren't motivated enough to really help a capitalist endeavor succeed. They pressure you to put yourself on the hook with none too subtle hints, couched in plausibly deniable talk of "team spirit" and "dedication" and "commitment". The coworker who buys a new car gets all kinds of praise for, in essence, being such a good wage slave/capitalist consumer. If that doesn't do the trick, managers reveal a few personal details about their choices in housing and transportation, to set an example, and encourage a little bit of "keeping up with the Joneses" envy. If that still doesn't work, they remove the people who aren't falling for it, as those sort set a bad example. Makes for a good object lesson for the survivors of the layoff. Extreme balances on credit cards, massive house payments, and other such horrible burdens become, in this whacked out world, bragging rights. The savvy worker therefore has to appear to toe the party line, while carefully holding back in ways that do not show, because sometimes one ends up stuck under a boss who doesn't know when to quit pushing, and the only healthy alternative the worker has is to leave.

Naturally, they're afraid of open source, readily equating it with Socialism and even (gasp!) Communism, which they have feared for decades. They don't understand the motivations behind open source, and therefore don't trust it. They've been told that open source fits just fine with capitalism (and it does!), but they can't believe that. I think that attitude more than anything, this dogmatic belief in the holiness of wealth and ownership, has propped the likes of Microsoft up beyond all reasonable objective assessment of the true value of their offerings.

firsty posteh? (-1)

Anonymous Coward | about a year ago | (#43653203)

frosty piss!

Re:firsty posteh? (0)

Anonymous Coward | about a year ago | (#43653697)

Dude,

Go in for treatment. Please.

it contradicts the definition (5, Interesting)

nimbius (983462) | about a year ago | (#43653219)

the very definition of 'proprietary software' indicates you dont have access to the code to calculate defect density, and even if you did you cannot independently verify the code you have is production code. how did the researchers quantify it?

Re:it contradicts the definition (0)

Anonymous Coward | about a year ago | (#43653227)

Pay no attention to the man behind the curtain.

Re:it contradicts the definition (5, Informative)

GrugVoth (822168) | about a year ago | (#43653309)

We use coverity where I work on proprietary code and part of their service is to report, anonymously obviously, the defect count, type and lines of code etc back to coverity if you want to. Via this they can get an idea of the defects found using their tool over a very large code base.

Re:it contradicts the definition (5, Interesting)

Chris Mattern (191822) | about a year ago | (#43653887)

We use coverity where I work on proprietary code and part of their service is to report, anonymously obviously, the defect count, type and lines of code etc back to coverity IF YOU WANT TO.

Am I detecting a selection bias here? Coverity can run their tests against all of open source. Coverity can run their tests only against that proprietary code that decides to use it and report the results--and it strikes me that only the better, and more open, proprietary shops would be doing this. Is Mircrosoft reporting their code? I doubt it. Is Oracle?

Re:it contradicts the definition (1)

Bigby (659157) | about a year ago | (#43654667)

I want to know how a tool can automatically detect defects. Sure, it can get syntax and a few semantic stuff. But most defects are not syntax errors. How does coverity catch when a required field isn't required for some reason? How does it catch UI glitches? How does it test performance? Memory loads?

Re:it contradicts the definition (2)

Registered Coward v2 (447531) | about a year ago | (#43654855)

We use coverity where I work on proprietary code and part of their service is to report, anonymously obviously, the defect count, type and lines of code etc back to coverity IF YOU WANT TO.

Am I detecting a selection bias here? Coverity can run their tests against all of open source. Coverity can run their tests only against that proprietary code that decides to use it and report the results--and it strikes me that only the better, and more open, proprietary shops would be doing this. Is Mircrosoft reporting their code? I doubt it. Is Oracle?

I doubt they ran it against all open source software; just some subset that ideally mirrored the proprietary code in complexity and application. If so it would be a reasonable comparison. Since TFA says they used some 300 OSS programs of various sizes I'd say it was a reasonable approximation of real world defect rates. Since the TFA doesn't name any proprietary products included in the survey it is harder to decide if they are valid results but I am willing to give them the benefit of doubt.

Re:it contradicts the definition (1)

chill (34294) | about a year ago | (#43655011)

You might try just RTFA.

...and an average defect density of .68 for proprietary code developed by Coverity enterprise customers.

Re:it contradicts the definition (2)

steelfood (895457) | about a year ago | (#43655091)

Is Mircrosoft reporting their code?

That would be unfairly skewing the numbers upwards against proprietary software, what with both Windows RT and 8 being completely defective and all.

Re:it contradicts the definition (2, Insightful)

Anonymous Coward | about a year ago | (#43653441)

Wrong. There are quite a few organizations who have access to Windows source code, yet Windows is still proprietary software. Proprietary just means that you cannot freely share, not that you have no chance to get the source code.

Re:it contradicts the definition (1)

robthebloke (1308483) | about a year ago | (#43653673)

Is AOL [iwastesomuchtime.com] one of them?

Re:it contradicts the definition (1)

Chris Mattern (191822) | about a year ago | (#43653977)

Wrong. There are quite a few organizations who have access to Windows source code, yet Windows is still proprietary software.

For the purposes of evaluating the Coverity Scan results, it's irrelevant whether other organizations have access to Windows source code. The question is: Does Coverity have that access, and did they use it in compiling their results? I will admit I don't know, but I sincerely doubt it. According to the article, the proprietary results are only from those who are Coverity clients.

Re:it contradicts the definition (0)

Anonymous Coward | about a year ago | (#43654525)

Considering that Microsoft is listed as a Coverity customer here [coverity.com] (select the "Software and Internet" tab), it's actually quite possible that they do have access to Windows metrics. I know it's gonna seem impossible to believe, but Microsoft isn't still shipping Windows ME. Their software has dramatically improved in recent years. It's not perfect, but it's quite a bit more stable than it used to be.

And for all the people whining about selection bias, there isn't any more bias for FOSS than there is for Proprietary: FOSS projects have to register with coverity to be included in their scan service. Any project reporting to Coverity (FOSS or Proprietary) must value quality enough to USE Coverity and report metrics. Coverity is not just spidering SourceForge and going "LOL FOSS SUCKS."

Here's the list of open source projects [coverity.com] that Coverity covers currently - looking through the list, there's quite a few heavy hitters in there (including the Linux kernel, which their report notes as a "benchmark" for FOSS quality), and not a lot of one-off SourceForge abominations.

Compelling numbers (0)

Anonymous Coward | about a year ago | (#43653239)

Compelling numbers, friend.

I'd like to have my own personnel verify thi... ah, right.

Auto scan (0)

Anonymous Coward | about a year ago | (#43653247)

It must be just me, but if we could simply find all these defects with a scan, why weren't they fixed before release?

Re:Auto scan (0)

Anonymous Coward | about a year ago | (#43653613)

Because they were all marked as WONTFIX in the bug database. :-)

69 is good (0)

youn (1516637) | about a year ago | (#43653257)

definitely more fun than 68 :P

Re:69 is good (1)

BasilBrush (643681) | about a year ago | (#43653437)

The 1970s called. They want their joke back.

Re:69 is good (0)

Anonymous Coward | about a year ago | (#43653627)

And the definition of 68 is...
You do me and I'll owe you one.

and all the children are above average (5, Funny)

Anonymous Coward | about a year ago | (#43653259)

"Code quality for open source software continues to mirror that of proprietary software — and both continue to surpass the industry standard for software quality."

What is this third kind of software that is neither open source nor proprietary which is bringing down the average industry standard for software quality? Because if there is only open source and proprietary then they can't both be better than average. Or perhaps the programmers are from Lake Wobegon?

Re:and all the children are above average (2)

clodney (778910) | about a year ago | (#43653321)

"Code quality for open source software continues to mirror that of proprietary software — and both continue to surpass the industry standard for software quality."

What is this third kind of software that is neither open source nor proprietary which is bringing down the average industry standard for software quality? Because if there is only open source and proprietary then they can't both be better than average. Or perhaps the programmers are from Lake Wobegon?

I had the same reaction, right down to the Lake Wobegon reference. Perhaps they are differentiating between software offered for sale versus tools internal to a business? To some extent that would also explain the difference in quality - cost to fix is much higher if you have shipped thousands of copies, versus telling the one consumer of a report in finance to ignore the one number that is wrong.

Re:and all the children are above average (2)

ZahrGnosis (66741) | about a year ago | (#43653501)

Wow, yeah, I posted an almost identical sentence myself. Eerie. (Although I didn't have a Wobegon reference... sorry). But yeah, it seems like an odd sentiment. Internal use software is still either "proprietary" or "open source"... isn't it? But good point. If someone calculated the bugs in my excel macros as if they could be used for general purpose computing I'd be in sad shape. (ObNote: I use excel macros as rarely as possible, and normally only at gunpoint).

Re:and all the children are above average (1)

Registered Coward v2 (447531) | about a year ago | (#43654729)

"Code quality for open source software continues to mirror that of proprietary software — and both continue to surpass the industry standard for software quality."

What is this third kind of software that is neither open source nor proprietary which is bringing down the average industry standard for software quality? Because if there is only open source and proprietary then they can't both be better than average. Or perhaps the programmers are from Lake Wobegon?

I had the same reaction, right down to the Lake Wobegon reference. Perhaps they are differentiating between software offered for sale versus tools internal to a business? To some extent that would also explain the difference in quality - cost to fix is much higher if you have shipped thousands of copies, versus telling the one consumer of a report in finance to ignore the one number that is wrong.

An industry standard has nothing to do with actual practice. It is not an average. All it says is it is an acceptable error rate is x.

Re:and all the children are above average (3, Insightful)

ZorroXXX (610877) | about a year ago | (#43653629)

The selection of sample projects is biased. For proprietary software, the data is taken from projects that at least cares as much for code quality that they run some tools (e.g. at least Coverity) to analyse it. I would suspect that the industry standard is below that because there exists some companies that mostly only consider "get the product out the door". For open source the selection is probably also somewhat scewed, in that they have analysed relatively large, mature and highly successfull projects. I would assume those have higher quality than the average sourceforge/github project.

Re:and all the children are above average (3, Interesting)

Gr33nJ3ll0 (1367543) | about a year ago | (#43653909)

This is a good point. To build on it, the results reported from the propertiary code has had coverity at least run against it, and usually the problems that it reports fixed. This does not appear to have been done in the case of the Open Source software, which was just scanned, but never given a chance to fix. In that circumstance I would have expected a much much higher result for the Open Source software, because Coverity often reports on very pedantic issues, which are often not important to overall software quality. Further these issues would not show up in anything other than Coverity, making the initial scan the first time these issues were brought to life.

Re:and all the children are above average (1)

Cenan (1892902) | about a year ago | (#43654127)

The selection is biased yes. But not for the reason you imagine. It's biased because only developers who care about the quality of their code run tools to determine that quality. All the shitty OSS and propietary code outthere didn't participate in the study. The dataset was built with usage statistics from the service and you have to register your project with Coverity in order to participate.

Re:and all the children are above average (2)

hardluck86 (2653957) | about a year ago | (#43653811)

Industry Standard != Industry Average

Re:and all the children are above average (1)

Bigby (659157) | about a year ago | (#43654763)

Correct. I would hope the industry average for heart surgery is above the industry standard. Likewise, the industry standard for politicians is far higher than what you get with the average.

Re:and all the children are above average (0)

Anonymous Coward | about a year ago | (#43654805)

So who is setting this "standard" that is so low that virtually all software is exceeding it?

Re: third kind of software (1)

rnturn (11092) | about a year ago | (#43653919)

``What is this third kind of software that is neither open source nor proprietary which is bringing down the average industry standard for software quality?''

Internally-written software that is not being released for ``external'' consumption, perhaps? There's likely far more of that in use than what is being sold for profit or being given away.

Re:and all the children are above average (0)

Anonymous Coward | about a year ago | (#43654269)

"industry standard for software quality" says nothing about averages, dummy. Industry standard is a minimum threshold, apparently commonly set by QA and Engineering organizations, that code must have "fewer than X defects per KLOC." And apparently both FOSS and Proprietary software exceed that industry rule of thumb.

I'm not sure why this is throwing you for such a loop - were you dropped on your head as a baby? Did you experience a near-drowning that cut off blood supply to your brain and thus damaged your cognitive function? We'll never know. But, perhaps this thought exercise will let light dawn on Marblehead:

The speed limit on a road is marked as 65 mph. Two cars drive down the road, both doing 75 mph. They both get pulled over and ticketed for exceeding the speed limit. Why? (Hint: Averages and Standards aren't the same. Consider the difference as you formulate your answer.)

Extra credit brain busters for you:
1) Imagine a 3rd car joins the first two on the road, doing 50 mph. Now the average speed of all cars on the road is 66.6 mph - above the speed limit. Do all 3 cars get a ticket? Why or why not?

2) Imagine a 3rd car joins the first two on the road, doing 40 mph. Now the average speed of all cars on the road is 63.3 mph - below the speed limit. Does any car get a ticket? Why or why not?

Re:and all the children are above average (0)

Anonymous Coward | about a year ago | (#43654531)

Many companies have software that is developed for a specific purpose and does not fall under these broad categories.
Firmware is software, but is part of hardware.
Some companies develop software that is not for sale but it serves some specialized purpose internal to the company. It is not really proprietary but more likely IT or domain-specific.
I would expect that consultants who retain copyright on their code but provide the code to their customers fall into another bucket.

Re:and all the children are above average (1)

Zero__Kelvin (151819) | about a year ago | (#43654619)

I just read the Wikipedia article on Lake Wobegon, and it seems that you are referring to "The Lake Wobegon Effect". Unfortunatley this term is ill-coined. It is indeed possible for all the children to be above average in fictional Lake Wobegon, since they are a very small subset of all the children in the world.

"What is this third kind of software that is neither open source nor proprietary which is bringing down the average industry standard for software quality?"

As far as exceeding industry standards you are confusing a standard with a mean. For example, the industry standard size for a byte is 8 bits. If your bytes have 9 bits in the software you design then you have exceeded industry standards ;-)

I'm pretty sure they can (0)

Anonymous Coward | about a year ago | (#43653265)

I don't think that it will be that hard to find at least 450 million lines of code that are wrong.
Heck, I can probably find that in my unfinished/abandoned projects folder.

Some things can't be measured objectively (2, Insightful)

Hentes (2461350) | about a year ago | (#43653271)

Errors per lines of code may give you a hard number, but that number has nothing to do with the quality of code. It only takes one well-placed error to ruin a piece of software.

Re:Some things can't be measured objectively (2)

MadKeithV (102058) | about a year ago | (#43653621)

It's still a better measure than not trying to measure it at all

Re:Some things can't be measured objectively (1)

Hentes (2461350) | about a year ago | (#43653861)

No it isn't. In this case, you have to ask the subjective but professional opinion of developers.

Re:Some things can't be measured objectively (0)

Anonymous Coward | about a year ago | (#43654655)

you have to ask the subjective but professional opinion of developers.

Ah, that explains the most famed professional riposte: "But it meets the Spec!"

Because "grossly unfit for actual purpose" is manifestly not a "defect", as every Luser shude kno.

Re:Some things can't be measured objectively (3, Insightful)

gorzek (647352) | about a year ago | (#43654467)

You are wrong, and here's why.

With no measurements at all, you cannot make informed judgments about the quality of your software. You can only guess. This means you would be unable to convince anyone (sane and intelligent) that your product has n bugs. "Because I say so" is not a metric.

With a poor measurement--such as one that ranks all defects equally--you have information, but now it's bad information. If you share the information but not the method(s) used to gather it, you can convince people you're right, because you have data about it. Never mind if you are stacking up Product A with 1 show-stopping bug against Product B with 50 cosmetic bugs or unhandled corner cases. By this bugcount-only metric, Product A looks better, and that's just stupid.

You need good measurements, and sometimes that includes measurements which cannot be quantitatively calculated without human intervention. A human programmer (or QA or other support person) who is familiar with a product will know just how severe a given bug is in terms of its impact. It is why, after all, bug tracking systems generally allow you to prioritize work by severity, fixing the worst bugs first.

Poor information is worse than no information because it can lead you to make the wrong decisions with confidence. With no information, at least you know you are shooting in the dark.

Re:Some things can't be measured objectively (0)

Anonymous Coward | about a year ago | (#43654941)

A code analyzer isn't going to detect cosmetic bugs and it keeps track of the types of bugs it finds. The analyzer can compare similar categories of bugs and weight different ones differently.

Re:Some things can't be measured objectively (0)

Anonymous Coward | about a year ago | (#43654475)

Not necessarily, and the question whether it is or not is empirical.

Re:Some things can't be measured objectively (1)

Bigby (659157) | about a year ago | (#43654807)

I could be. But it is a bad assumption to think it is. It will lead to a false sense of quality.

Re:Some things can't be measured objectively (0)

Anonymous Coward | about a year ago | (#43653681)

Unless you are saying proprietary or open source software has a particular penchant for making such "well-placed" errors, you can in fact get a very good idea of quality of code in general from these numbers.

Defect density (0)

Anonymous Coward | about a year ago | (#43653319)

If the defect density is 0.69 per 1000 lines of code, then of 450 million lines of code, more than 300000 are wrong. Therefore, so is the title.

The study is not really conclusive (2)

brainscauseminds (1865962) | about a year ago | (#43653353)

Actually, this study does not say anything directly about code quality, because Density = Total Defects Found / Code Size. The problem is with the "Total Defects Found" part. How they are found and how they are reported may differ vastly from one project/company to other. The report sais that the quality of code increases with larger codebases in propertiary projects. In fact, the best you can say is that the metric decreases with larger codebases in propertiary projects. Maybe many of the defects have not been found yet in propertiary projects. Maybe they have less manpower to seek the errors, maybe they just don't care as long anything does not crash. But smaller defects may be in the code. Open source code is more open to "finding the defects", thus possibly obtaining worse "quality" they are talking about in the article. I think this has to be kept in mind when reading the report.

OSS defects (2)

Cyko_01 (1092499) | about a year ago | (#43653389)

Everyone knows OSS doesn't have defects, it just develops random features

Re:OSS defects (1)

TWiTfan (2887093) | about a year ago | (#43655123)

The problem with open source software isn't the code quality, it's poor UI and poor documentation. Way too many open source projects bring on great programmers, but few, if any, designers or technical writers. The result is software with great functionality, but buried beneath horrid UI's and poor (or non-existent) documentation. I wish I had a nickel for every OSS project website I've been to where the only documentation in sight was a long list of bug-fixes, or whose UI was so confusing as to make it unclear what the software is even FOR.

Not every software project has to be as well designed as Apple, but Jesus, if you expect a typical consumer to memorize a ton of obscure command-line commands to use software that doesn't even properly document those commands, then you really need to help from a proper designer and technical writer (much as programmers are loathe to admit that designers and writers serve an actual useful purpose).

The more the code sucks, the more it costs (0)

Anonymous Coward | about a year ago | (#43653411)

Stuff written in less popular old obsolete languages that is very sloppy yet unique and confusing and difficult to wrap your head around and extremely buggy is a pain and thus it costs companies money to pay someone to do work on it. Not only does it take an intuitive person who can tolerate annoyingly crappy code, but they'd be stupid to take the job in the first place unless the payoff was worth it. Since open source is free and clean and popular, its friggin awesome.

What else is in the "industry"? (4, Insightful)

ZahrGnosis (66741) | about a year ago | (#43653431)

and both [proprietary and open-source software] continue to surpass the industry standard for software quality

... What else is there? And why is this unknown third type of code dragging down the "industry"?

Re:What else is in the "industry"? (1)

fredrated (639554) | about a year ago | (#43653625)

Exactly my question, what else is there besides proprietary and open source? How can they both surpass industry standards?

Re:What else is in the "industry"? (1)

swillden (191260) | about a year ago | (#43654291)

Exactly my question, what else is there besides proprietary and open source? How can they both surpass industry standards?

I think that's based on the unstated and unsupported -- but not entirely unreasonable -- assumption that proprietary and open source projects that don't care enough about quality to run Coverity on their code have lower quality levels than those that do.

However, I don't believe Google uses Coverity, and we have a pretty serious focus on code quality. At least, in my 20+-year career I haven't seen any other organization with quality standards as high as Google's, so I'd put Google forth as a counterexample to the assumption, and I'm sure there are many more. I have no idea if it the assumption is valid overall.

Re:What else is in the "industry"? (0)

Anonymous Coward | about a year ago | (#43654347)

Perhaps you've overlooked the little factoid that industry 'averages' and industry 'standards' are not the same?

Re:What else is in the "industry"? (1)

stillnotelf (1476907) | about a year ago | (#43654465)

There is a huge third group: the military and aerospace industries. Unfortunately, their standards are even higher, like one bug per 420000 lines of code, [fastcompany.com] so they're obviously not the group we need to make this math work.

Maybe the "industry standard" is whatever buggy math it is that makes that statement make sense to the original author?

Re:What else is in the "industry"? (1)

ZahrGnosis (66741) | about a year ago | (#43654995)

Military still seems "proprietary" to me. If they meant "commercial", I could see a difference. I also considered "embedded" or "firmware" style code that, while software, is more closely tied to a physical hardware implementation. All of those still seem either "proprietary" or "open source", though, and you're right (@stillnotelf) that these would raise rather than lower industry averages.

It could include things like javascript that is just out-in-the-wild. If you were to strip programmatic pieces from websites that were one-offs... things that were neither marketed nor sold, and not really managed as software, just put out there, code quality would probably drop. I'm thinking of websites with funny animations or just hand-coded scripts to do navigation or whatnot. These wouldn't be "open source" in the sense that there's no statement of open copyright, and they wouldn't be "proprietary" in the sense that noone is marketing or working to save or publish or reuse the code, and while copyright may exist noone is really worried about projecting it (due to the one-off nature). Still, it's a weird statistic.

I wonder if they meant a "standard" as in a target or an accepted limit that is somewhat arbitrary rather than an "average" (which the article actually uses) of real-world code. This would make sense since the average they cite is exactly 1.

Unforeseen consequences (2)

tedgyz (515156) | about a year ago | (#43653445)

Quality metrics can have unexpected side effects [dilbert.com] .

Re:Unforeseen consequences (1, Funny)

Alain Williams (2972) | about a year ago | (#43653567)

/* This
* comment
* is
* part
* of
* the
* corporate
* edict
* to
* reduce
* the
* defect
* rate
* reported
* by
* Coverity
*/
  printf("hello world\n");

Re:Unforeseen consequences (1)

rnturn (11092) | about a year ago | (#43654045)

Good grief... I certainly hope that Coverity's analyzer strips out comments before it starts evaluating code. Even the dimmest pointy-haired manager would see right through that scam.

Re:Unforeseen consequences (1)

Chris Mattern (191822) | about a year ago | (#43654109)

Most code metrics (except for those that specifically evaluate comments) strip out comments before compiling. However, you can always do this:

print
  (
      "hello world\n"
  )
;

Could probably split up the string too, but I'm too lazy to look up the exact syntax for that.

Re:Unforeseen consequences (1)

angel'o'sphere (80593) | about a year ago | (#43654733)

Most code metric tools don't use new line seither but only count ";", "," and "}".
(Because depending on definition ... see: Watts S. Humphrey, Personal Software Process ... every parameter you pass to a function is considered ONE LINE OF CODE).
So this: f(1, 3*4, "sup?") and this
f(1,
3*4
,
"sup?")
are the same kines of code.

The quality fairy (2)

swm (171547) | about a year ago | (#43653507)

FTA:

As projects surpass one million lines of code, there’s a direct correlation between size and quality for proprietary projects, and an inverse correlation for open source projects.

The article gives numbers: above 1M LOC, defect density increases for open source projects, and decreases for proprietary projects.
Increasing defect density with size is plausible: beyond a certain size, the code base becomes intractable.
Decreasing defect density with size is harder to understand: why should the quality fairy only visit specially big proprietary projects?

Perhaps the way those proprietary projects get into the MLOC range in the first place is with huge tracts of boilerplate, duplicated code, or machine-generated code.
That would inflate up the denominator in the defects/KLOC ratio.
But then that calls the whole defects/KLOC metric into question.

Re:The quality fairy (1)

K. S. Kyosuke (729550) | about a year ago | (#43653571)

Decreasing defect density with size is harder to understand: why should the quality fairy only visit specially big proprietary projects?

That might have something to do with market penetration and resources dedicated to maintenance...? Those huge proprietary projects probably happen to be the stuff that almost everyone gets to use.

Re:The quality fairy (0)

Anonymous Coward | about a year ago | (#43653983)

Decreasing defect density with size is harder to understand: why should the quality fairy only visit specially big proprietary projects?

That might have something to do with market penetration and resources dedicated to maintenance...? Those huge proprietary projects probably happen to be the stuff that almost everyone gets to use.

Actually, what most likely happens is that for large corporate projects there's a quality initiative that is throwing actual resources at the code quality task - for example, static analysis tools, teams of QA, etc., etc., that are typically lacking from a large open source project.

Re:The quality fairy (1)

gutnor (872759) | about a year ago | (#43654117)

Maybe also 1 MLOC means popular in both OSS and Proprietary world. In proprietary, popular is slowly becoming legacy, the stuff you cannot change. On the other hand, in the OSS world, popular means load more contribution from people, the time they chose to keep quality on the core feature and the community go wild with the rest of the codebase.

Re:The quality fairy (0)

Anonymous Coward | about a year ago | (#43654189)

It's probably safe to assume that there's a direct correlation between the size of a project and its age; i.e. bigger projects are older and hence more mature.

In the proprietary arena, there is also a likely correlation between the age of a project and its popularity; i.e. unpopular ones don't get funded so they fail. And of course, there's a correlation between the popularity of a project and the amount of resources (e.g. man-hours) that can be spent on it because popularity translates directly to sales. And of course, the more resources spent on it, the better the quality.

In other words, big proprietary projects go along with big corporations that can afford to spend the time and money to fix the bugs.

Colours in graphs (1)

Alain Williams (2972) | about a year ago | (#43653523)

Why on earth do they choose 2 colours that are hard to tell apart in that graph ? They were black & dark blue. It took me several seconds to work out which was which. Many other reports/... seem to do similar.

Re:Colours in graphs (1)

drinkypoo (153816) | about a year ago | (#43653643)

Why on earth do they choose 2 colours that are hard to tell apart in that graph ?

here you go [ebay.com]

Code quality (2)

140Mandak262Jamuna (970587) | about a year ago | (#43653581)

First of all code quality is difficult to measure, and the number of (known) defects per 1000 lines of code is a very poor metric. I could do more (good and bad) in one line of code than a novice who write voluminous code. Leaving that aside, what drives/motivates creating good quality code?

In open source, a defect gets fixed when someone feels the urge to fix it. Most of the time it is because it is their own dog food. Many open source projects are actually used by their own developers and they fix the issues that irritate them most. And rest of the bugs are based on impact on other users and passion about the software project

In a closed source project, it is often the bugs that affect the loudest paying customer gets fixed. If it is not going to advance sales, it wont get fixed.

Given this dynamic it is not at all surprising both methods have similar levels of that elusive "quality". I think software development should eventually follow the model of academic research. There is scientific research done by the universities that have no immediate application or exploitation potentials. The tenured academic professors teach courses and do research on such topics. Then as the commercialization potential gets understood, it starts going towards sponsored projects and eventually it goes into commercial R&D and product development.

Similarly we could envision people who teach programming languages to college maintaining open source projects. The students develop features and fix bugs for course credit. As the project matures, it might go commercial or might stay open source or it could become a fork. The professors who maintain such OSS projects should get similar bragging rights and prestige like professors who publish academic research on language families or bird migration or the nature of urban planning in ancient Rome.

Re:Code quality (1)

bussdriver (620565) | about a year ago | (#43653993)

Given the massive bias the US government has towards expensive private software contractors, I am surprised the results were so close.

MBAs, Politicians and incompetent journalists LOVE poor metrics. Americans love simplistic binary metrics (sorry no citation just experience, it's the culture.)

Remember klocs? That went on a while. Sounds like this metric dates back to those days-- they don't measure programmers by 1000s of lines coded anymore but they didn't learn their lesson and kept the defect rate measures...

Academics? Sciences? get with the real world, nobody bothers unless it is found to improve the bottom line Usually, it has to be forced to even be seriously considered... Proof by example using science and academic results is SUCH a change they call those who employ it "innovators" and they usually get all the undeserved credit as well.

Re:Code quality (1)

Dcnjoe60 (682885) | about a year ago | (#43654641)

Given the massive bias the US government has towards expensive private software contractors, I am surprised the results were so close.

Well, it could be that there really isn't a correlation between quality and what you pay for programming, at least beyond some point, so a good, but lower paid, open source programmer writes just as good code as a good, but higher paid proprietary programmer.

Or, it could be the higher paid programmers really do turn out better code, but the nature of open source, with multiple people reviewing it mitigates the difference. I hate to use a sports analogy, but I will anyway. I am a lousy golfer, but I can putt fantastically. If I am playing a normal round of golf, I will most likely lose. However, in a four man scramble, where the best ball is played, people want me on their team, because once the ball makes it to the green, I can usually get it in the cup with a single putt. Likewise, the programmers working on an open source project might not have the expertise as the high priced specialist, but they may have those who contribute the right parts to make the whole project successful.

I would venture that in reality, it is a combination of both of the above. Good programmers turn out good code, regardless of whether it is proprietary or open source. Plus, the open source model makes up for weaknesses in the skills of individual programmers.

Re:Code quality (1)

chrism238 (657741) | about a year ago | (#43654445)

I think software development should eventually follow the model of academic research. There is scientific research done by the universities that have no immediate application or exploitation potentials. The tenured academic professors teach courses and do research on such topics. Then as the commercialization potential gets understood, it starts going towards sponsored projects and eventually it goes into commercial R&D and product development.

It sounds like you have a very 1980's appreciation of university research.

Re:Code quality (1)

LeadSongDog (1120683) | about a year ago | (#43654745)

number of (known) defects per 1000 lines of code is a very poor metric

It's not a poor metric, but it is a metric of something which isn't very useful. If I already knew the unfixed defects in the product, I'd just fix them.

More useful metrics relate to simplicity and testability. Is every module understandable on its own by a cleanroom reviewer who first saw it ten minutes ago? How free is the code from hand-tuning? How few parameters are passed? Are there state variables that go uninitialized? How small are the largest individual modules? How completely does the test code exercise all branches? How thoroughly has the test code itself been tested?

"both continue to surpass the industry standard" (0)

Anonymous Coward | about a year ago | (#43653605)

Where everybody is above average. Or is there a 3rd category of software other than proprietary and open?

Re:"both continue to surpass the industry standard (0)

Anonymous Coward | about a year ago | (#43653817)

Proprietary, open, and 'subject to endless litigation regarding its status' (SCO and friends), maybe?

Question? (1)

Dcnjoe60 (682885) | about a year ago | (#43654501)

Code quality for open source software continues to mirror that of proprietary software — and both continue to surpass the industry standard for software quality. Defect density (defects per 1,000 lines of software code) is a commonly used measurement for software quality.

Since there are two types of software open source and proprietary and both of them surpass the industry standard for software quality, what exactly is the industry standard based on?

The article states that the industry standard is 1 defect per 1,000 lines of code. But at the rates given, open source is 1 defect in 1,449 lines of code and proprietary software is 1 defect in 1,470 lines of code. Maybe it's time to change the industry standard?

HIgher defect density indicates BETTER code (1)

raymorris (2726007) | about a year ago | (#43654893)

Counterintuitively, defect density is actually an INVERSE indication of quality - better quality code will have MORE defects per line.
The reason I say that is because better code has fewer lines per problem. Consider strcpy(), a function to copy array of characters (a C string). You can't use strcpy() in your cd - you're supposed to create strcpy(), copying each element of the array.
Take a moment to consider how you'd write that before looking below.

Roughly how many lines of code did you use to copy an array? Here's what a typical corporate programmer might do:
while (source[i] != '\0')
{
    dest[i] = source[i];
    i++;
  }

So one error in that code would be 1 defect per five lines or so.

Here's all the code you need, what a better programmer would write:
while (*dest++ = *src++);

If the typical programmer and the expert both had exactly one error, the expert would have five times as many bugs PER LINE than the typical programmer! So you're better off with code that has a higher density of errors - better code will have fewer lines per error.

This is the same reason LOC is an inverse indicator of productivity. Yesterday I fixed a junior programmers code tat looked like this:

if ($category = 'rings') {
        $page = 'rings.html'
}
if ($category = 'necklaces') {
        $page = necklaces.html'
}
if ($category = 'bracelets') {
        $page = 'bracelets.html'
}
if ($category = 'loose_stones') {
        $page = ''loose_stones.html'
}
if ($category = 'charms') {
        $page = 'charms.html'
}

Of course I changed that code to, well, zero lines, I just used the $category variable where he had used the $page variable. Code which accomplishes a task in zero to one line is better software, written by a better programmer, than code that uses eightteen lines to accomplish the same thing.

Open is better (1)

Murdoch5 (1563847) | about a year ago | (#43655027)

The more eyes that can view a piece of code the more bugs can be spotted and the better the algorithm development can be. Open source code is also a great way to teach young developers because the best way to learn to programming to is to read code and program, something which can't be easily done by locked down software.

At least half of those lines (0)

Anonymous Coward | about a year ago | (#43655135)

At least half of those lines come from PHP alone! Magnificent!

the inconvenient truth (0)

Anonymous Coward | about a year ago | (#43655163)

most "open source" is written by people working for corporations, so this comparison is idiotic in its design.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?