Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Can ISO 29119 Software Testing "Standard" Really Be a Standard?

timothy posted about two weeks ago | from the so-many-questions dept.

Programming 152

New submitter yorgo writes The International Organization for Standardization (ISO) will soon publish part 4 of a 5 part series of software testing standards. According to the website, "ISO/IEC/IEEE 29119 Software Testing is an internationally agreed set of standards for software testing that can be used within any software development life cycle or organisation." However, many in the testing community are against it. Some wonder how the ISO/IEC/IEEE achieved consensus without their input. James Bach speculates that exclusion helped build consensus. Others, such as Iain McCowatt, argue that something as variable as software testing cannot be standardized, at all. And others believe that the motive behind the standards is not increased quality, but economic benefit, instead. Michael Bolton explains "rent-seeking" as he builds on James Christie's CAST 2014 presentation, "Standards – promoting quality or restricting competition?"

A comprehensive list of many other arguments, viewpoints, and information has been collected by Huib Schoots. Opponents of ISO 29119 have even started a petition aimed at suspending publication of the standard. Even so, this might be an losing battle. Gil Zilberfeld thinks that companies will take the path of least resistance and accept ISO 29119.

So, where do you stand? What constitutes a consensus? Can a standard be honored without consensus? Can an inherently sapient activity, such as testing, be standardized, at all? What is the real purpose of a standard? Will companies acquiesce and adopt the standard without question?

cancel ×

152 comments

Sorry! There are no comments related to the filter you selected.

Who cares (1)

hinchles (976598) | about two weeks ago | (#47817161)

Considering ISO certification costs money and most companies refuse to spend money on stuff that isn't ISO 9001 then very few will probably get it. Most dedicated software houses I've encountered seem to prefer getting the TickIT certs instead of ISO certs for actual code certification but nobody bothers with the testing side of things.

Re:Who cares (2)

Jane Q. Public (1010737) | about two weeks ago | (#47818839)

I also suspect that this "standard" would be unworkable with some established de facto industry standards that aren't likely to change.

Standards (3, Insightful)

Crizzam (749336) | about two weeks ago | (#47817193)

Are rules for some and suggestions for the rest of us. The IEEE can put a standard on cleaning the toilet. If your company wants to follow it to the letter, or just use it as another reference, that's your call. I think the organization of conceptually difficult concepts is a good thing, overall. What we do with that is a whole other thing.

Re:Standards (1)

UnderCoverPenguin (1001627) | about two weeks ago | (#47817309)

At least where I work, most likely is that it will be more paper work to get done - never mind that we already have too much paper work to do. Like us in development, the testing people will make whatever they can conform to this new standard, then file waivers for the rest.

Re:Standards (4, Insightful)

JeffAtl (1737988) | about two weeks ago | (#47818103)

It's not quite that simple - customers can require the certification. If a customer is sizable enough, the suppliers will have comply to stay in business.

For real world examples, look at the power that the state of Texas has wielded in text book requirements and the state of California with warning labels.

Re:Standards (1)

michaelbolton (554690) | about two weeks ago | (#47820219)

Organization of conceptually difficult things is fine. A lot of us do this. A group of uninformed and (in my view) insufficiently critical self-appointed process enthusiasts mandating a particular organization for conceptually difficult things is more problematic.

Can see it now: (5, Insightful)

some old guy (674482) | about two weeks ago | (#47817195)

MBA CEO: I want our new product to be QA'd according to ISO 29119 before shipping.

Project Manager: Good idea, but that will add some time and overhead cost to my budget.

MBA CEO: Never mind, just ship it.

Re:Can see it now: (5, Informative)

UnderCoverPenguin (1001627) | about two weeks ago | (#47817269)

MBA CEO: Never mind, just ship it.

More likely response: "Figure out how to get it done within the existing budget and schedule."

Re:Can see it now: (3, Funny)

tomhath (637240) | about two weeks ago | (#47817457)

Even more likely response: "Have the secretary fill out the paperwork and change our website to say we're ISO 29119 compliant".

Re:Can see it now: (2)

mariox19 (632969) | about two weeks ago | (#47818363)

And if anything goes wrong, we'll crucify the QA people that we never gave enough testing time to in the first place.

Re:Can see it now: (1)

lgw (121541) | about two weeks ago | (#47818545)

You still have QA people? What luxury!

Re:Can see it now: (2)

awebb (963424) | about two weeks ago | (#47820119)

Amen to that! No one gives a shit about QA or the 100 issues they find during testing. It's the 1 issue that slips out that causes a riot.

Which Michael Bolton? (3, Funny)

Type44Q (1233630) | about two weeks ago | (#47817257)

Michael Bolton explains "rent-seeking"

The no-talent ass clown known for his God-awful "music" or the one who hates him??

Re:Which Michael Bolton? (1)

CaptainDork (3678879) | about two weeks ago | (#47817449)

You jumped my stuff.

Re:Which Michael Bolton? (2)

michaelbolton (554690) | about two weeks ago | (#47817801)

Ah. Another primitive attempt at what some of your fellow Earthlings call "humour". Keep practicing! ---Michael Bolton

Re:Which Michael Bolton? (1)

gstoddart (321705) | about two weeks ago | (#47818015)

But but ... how are we supposed to live without you, now that we've been loving you so long?

Dude, seriously, it must be suck to evoke such a reviled musician every time someone hears your name.

Re:Which Michael Bolton? (1)

michaelbolton (554690) | about two weeks ago | (#47818421)

Only when people bring it up. So, if you want to help...

Re:Which Michael Bolton? (1)

gstoddart (321705) | about two weeks ago | (#47818889)

LOL ... well, tell me how you feel [azlyrics.com] , time is on my side, [azlyrics.com] it's just a feeling [azlyrics.com] , but I almost believed you [azlyrics.com] .

This is the internet, everybody's crazy [azlyrics.com] , don't like it, walk away [azlyrics.com] .

From now on [azlyrics.com] you'll know that forever isn't long enough [azlyrics.com] until you hear another Michael Bolton joke.

Just remember to love somebody [azlyrics.com] . The one thing [azlyrics.com] to remember, this is the time [azlyrics.com] you're asking yourself Why me? [azlyrics.com]

So, before you turn a whiter shade of pale [azlyrics.com] , I wanna hear you say it [azlyrics.com] ... If I could [azlyrics.com] go the distance [azlyrics.com] , That's life! [azlyrics.com]

All the best" [azlyrics.com] .

There's two lessons to be learned here (one for you, and one for me) ... first, never ask for mercy on the interwebs, not likely to happen. Second, your namesake had way too many damned albums for that to not be a painful process for both of us. ;-)

Cheers (and, yes, feel free to say it ... it was rather childish and annoying. ;-)

Re:Which Michael Bolton? (1)

michaelbolton (554690) | about two weeks ago | (#47820257)

Thank you, but you've already helped more than enough. :)

Re:Which Michael Bolton? (2)

ESD (62) | about two weeks ago | (#47818807)

As this particular thread is off-topic anyway: Thanks for your blog, Mr. Bolton! It's always an interesting read and I try to point new testers to it at least once to get some exposure to a usually unknown view on testing.

Doesn't change a thing for QA (0)

Anonymous Coward | about two weeks ago | (#47817299)

The job still sucks and everyone hates you.

Wrong focus? (2)

QuietLagoon (813062) | about two weeks ago | (#47817307)

... According to the website, "ISO/IEC/IEEE 29119 Software Testing is an internationally agreed set of standards for software testing that can be used within any software development life cycle or organisation."...

If I were to jump upon a standard for testing software, the fact that it is "internationally agreed" is way down on the requirements, yet it seems to be mentioned as the main feature here.

Re:Wrong focus? (1)

rhazz (2853871) | about two weeks ago | (#47817527)

Yep, and it only takes two countries agreeing for it to be "internationally agreed"!

Re:Wrong focus? (2)

TechyImmigrant (175943) | about two weeks ago | (#47817835)

This shit matters if you are selling things internationally. It makes it difficulty for country A to refuse imports of county B's software because it wasn't tested to their local testing standard. If it was tested to an ISO standard it'll do and the WTO will be on their ass if they try to claim otherwise.

That doesn't mean the specs are any good. They aren't. But there is a very real interaction with international trade regulations.

Re:Wrong focus? (3, Insightful)

sinij (911942) | about two weeks ago | (#47817853)

You are incorrect. ISO standards are recognized by multiple countries. You can Google the full list.
 
The main advantage of any kind ISO is that it enables your software to get on government procurement lists. This doesn't impact small shops, they don't generally have resources or organizational maturity to sell to governments. There are multiple international, proprietary, and country-specific standards (e.g. ISO, FIPS, Common Criteria, PCI-DSS). International means you only have to go through certification once, not once per country that you were doing before the standard was adopted.

Re:Wrong focus? (1)

michaelbolton (554690) | about two weeks ago | (#47818451)

That is both an advantage AND a disadvantage, since it affords the potential for rent-seeking (http://www.developsense.com/blog/2014/08/rising-against-the-rent-seekers/ ). Rent-seeking may lead to wasteful documentation, compliance vs. competence, and other forms of goal displacement. See also http://www.developsense.com/pr... [developsense.com] .

Re:Wrong focus? (1)

sinij (911942) | about two weeks ago | (#47818477)

Compliance establishes baseline level of competence. As a system integrator, I have seen numerous examples that fall under gross negligence category. As a result, I strongly believe that even ineffective standards testing with associated increased overhead is better than a status quo.

Re:Wrong focus? (2)

michaelbolton (554690) | about two weeks ago | (#47819253)

Lots of competent testers point out that compliance with ISO 29119 does not in the least establish a baseline level of competence. Your argument "even ineffective standards testing with associated increased overhead his better than a status quo" is basically equivalent to saying "friendly fire is better than not shooting" or "ineffective medicine with severe side effects is better than no medicine at all".

Shades of 2167 (3, Insightful)

Snotnose (212196) | about two weeks ago | (#47817325)

In the late 80s and early 90s I was involved in 2 projects run under MIL SPEC 2167, which was supposed to ensure product quality. Both were epic disasters. IMHO, 2167 pretty much guaranteed mediocre at best software, taking 3x longer to do, at a cost at least 6x of non-2167

This sounds like the 21st century version of 2167.

Re:Shades of 2167 (1)

gstoddart (321705) | about two weeks ago | (#47818153)

Both were epic disasters. IMHO, 2167 pretty much guaranteed mediocre at best software, taking 3x longer to do, at a cost at least 6x of non-2167

But, I've always assumed that the function of government specs was to achieve precisely that.

I mean, a quality standard defined by a government committee can't actually be expected to have been designed to product actual quality outputs, can it?

I've just assumed they were there to give the bureaucrats something to tick off on their checklist, even if it has nothing to do with the real world.

Re:Shades of 2167 (3, Interesting)

luis_a_espinal (1810296) | about two weeks ago | (#47819505)

In the late 80s and early 90s I was involved in 2 projects run under MIL SPEC 2167, which was supposed to ensure product quality. Both were epic disasters. IMHO, 2167 pretty much guaranteed mediocre at best software, taking 3x longer to do, at a cost at least 6x of non-2167 This sounds like the 21st century version of 2167.

MIL SPEC 2167, iirc, deals with documentation and deliverables. The actual software development process was "guidelined" by some other MIL SPEC. With that said, those were supposed to act as guidelines for a waterfall methodology (which surprisingly, it can actually be used in some contexts, or subverted into a spiral/iterative process.)

I worked at a defense contractor not long ago, and alas, the software projects were horribly run. But I always saw that it was the execution, not the specs per say that was the culprit for each and every single snafu/fubar hybrid I encountered. That and management, and life-long-career jockeys from the punch-card era dictating how to write software, and department infighting.

It's just like CMM/CMMI - A CMMI level X or Y only indicates that, to some degree, an organization a) has a formal process, and b) follows such process.

It doesn't indicate that the process is good - it doesn't even guarantee that *it is not shit*.

What it does, is that it helps an organization guarantee that its constituent parts know what activities to do under what circumstances and tasks in a business lifecycle. And that helps an organization improve things (assuming said organization has the necessary technical wherewithal and culture.)

In private business and with defense contractors, it is the companies who fail to execute (let's think of how many companies ditch source control to become *agile*!) particular practices. Defense contractors have a lot of leeway in negotiating and tailoring the actual execution of a process. Typically, they do not do it because they suck (and for a lot of other political and financial motivation$.)

Re:Shades of 2167 (1)

michaelbolton (554690) | about two weeks ago | (#47820297)

"What it does, is that it helps an organization guarantee that its constituent parts know what activities to do under what circumstances and tasks in a business lifecycle." That claim is unsupportable, since it ignores the role of tacit knowledge; it ignores all the things that the process manual doesn't say; and it ignores all the ways in which the process manual overstructures the work. As James C. Scott points out in /Seeing Like a State/, this is exactly the reason why work-to-rule campaigns can be as effective (that is, more disruptive) than a strike.

Automated test in is a minimum (3, Interesting)

LostMyBeaver (1226054) | about two weeks ago | (#47817331)

First of all. I HATE WRITING UNIT TESTS!!! Know what I hate more? When I get bit in the ass because something that did work before stopped.

Unit testing is step one in any decent software development and I will never enter into or manage another project without unit tests being a critical component of the project. I'll just hire a QA guy to unit test all my code... I don't want to do it haha.

Second, there is absolutely nothing which can't be automatically tested too. When you write code, GUI, Web, Command Line, message based, etc... An automated script to test the code is critical. There are tools for it.

Everything should be tested automatically... That even includes memory leaks when exiting. I would never hire someone even for a C#, Java, Python or even PHP position who doesn't write code which cleans up properly after itself (even if that means correct use of the garbage collector).

I have worked on several multi-million line commercial applications, some with 500 million+ users. I have never seen a piece of code which could not be properly tested using the right tools. That can even include small embedded systems where we would have to actually implement a QEMU module or three.

So... Quit your bitching and write a test suite.

Re:Automated test in is a minimum (5, Interesting)

omglolbah (731566) | about two weeks ago | (#47817511)

You would love the control system software we use at work... (world leading platform for the industry).

No revision control. You have 'current' revision. That is it.

Integrated code editor that has no syntax highlighting.

Patches to the system will break components that are not considered 'core'. Which forces updates of ALL components in the system. This has lead to bugs persisting at sites for years with no patch because nobody wants to fix bugs when it costs tens of millions of dollars to do so.

No automatic testing. Of anything. When we update a component everything has to be tested manually. Someone will sit for 2 weeks and check every state of GUI symbols for the whole HMI. Oh joy...

If you change ANYTHING in code, you can no longer connect to controllers to view live data. You need to do a download to the control with the code changes before live data can be observed. This means that as soon as you make changes, you lose the ability to view the old code running. There is no way to have both a 'online capable' version of the code and a changed codebase in the same control system. We operate two separate virtual environments and switch VLANs or just move a cat6 when testing...

This is for oil rig control systems. There is no automated testing of any kind, even for critical emergency shutdown systems. Every test is done manually.
The ESD systems are usually a complex matrix of casues and effects with hundreds of inputs, hundreds of outputs... This is all tested manually as the software does not support any reasonable simulation of the controller input/output systems.

Enjoy that little gem.

Re:Automated test in is a minimum (1)

UnderCoverPenguin (1001627) | about two weeks ago | (#47817619)

I'll just hire a QA guy to unit test all my code...

Actually, that's how the testing should be done. Give the requirements to both teams - testers and developers. Developers design/write the product code. Testers design/write the tests. Then let the testing begin. Problems entered into issue tracking. Both teams fix their respective problems. Retest. Repeat as needed.

Unfortunately, many companies fail to adequately fund testing so devs end up writing tests, which, in turn, catch fewer problems

Re:Automated test in is a minimum (0)

Anonymous Coward | about two weeks ago | (#47817883)

No. Unit testing is not a tester's responsibility. It is the responsibility of the coder to validate the code they wrote does the basic function it supposed to do. Unit testing is for that. Functional, performance, load, etc. testing is for testers to do. Can they overlap? Yes. In scrum/agile, they often do. Should devs NOT test? Hell to the no they should definitely test.

I wouldn't hire a dev that didn't test his/her own code.

Re:Automated test in is a minimum (1, Insightful)

gnupun (752725) | about two weeks ago | (#47818219)

No. Unit testing is not a tester's responsibility. It is the responsibility of the coder to validate the code they wrote does the basic function it supposed to do.

Unit testing is white box testing and it's okay to get a tester to develop the tests as long as there's feedback from the developer about how the white box code is supposed to behave (internal behavior).

Should devs NOT test? Hell to the no they should definitely test.

Does the average dev have the skill set/ mind set of a tester? No. Even Microsoft often pairs a tester with a developer. Unit tests are likely to find as many or more bugs than black box testing and the typical developer will not be happy/motivated to find bugs in his own code.

So are you willing to deliver a buggy product just because you don't want to spend extra on test developers?

Re:Automated test in is a minimum (2)

uncqual (836337) | about two weeks ago | (#47819011)

Does the average hobbyist dev have the skill set/ mind set of a tester? No.

FTFY.

If a developer doesn't have the mindset of a tester, perhaps they should find something else to do (maybe the local Walmart needs a greeter?) or be assigned support "on-call" more regularly so they can experience being waken at 3AM due to someone failing to design, implement, and test well. If someone doesn't have the mindset of a tester and their title is something like Senior Software Developer or higher, there is a serious title inflation problem in their workplace.

Design and implementation decisions, the classic "developer" roles, impact testing. Someone who doesn't grok and appreciate testing has no business playing a significant role in design. In my experience, one of the reasons that unit testing is sometimes quite hard (sometimes updating the unit tests takes much more effort than the corresponding modest change to the base code took) is that systems are rarely designed to be easily unit testable - commonly there's a "shadow" unit testing world of hacks layered on hacks relying on side effects of this subsystem or that subsystem that is cobbled together and labeled something like "the unit test environment".

When a developer commits code, they should be sure it works and take it personally (i.e., as a personal failing) and use it as a learning experience if the code fails either due to a design/coding error within the scope of the code (generally extending throughout the entire subsystem even if just one line in one module was changed) or misuse of some external service/API. Failures that arise from inter-subsystem interactions due to ambiguous specifications or not yet understood interactions are what testers should be finding (and the architects should usually be held accountable for those failures).

As well, you want developers to have a mindset of testing because they are then the most likely to develop and adapt tools that actually make testing a repeatable and low cost process.

Re:Automated test in is a minimum (0)

Anonymous Coward | about two weeks ago | (#47819461)

So you want a coder to design, code and test? That's 3 jobs, i hope you pay them 3 times.

Re:Automated test in is a minimum (1)

uncqual (836337) | about two weeks ago | (#47819899)

Qualified senior developers I know do some amount of all three and all have done a substantial amount of all three during various points in their career.

If you're considering just a "coder" (working, presumably, from some detailed design spec), that's probably a different matter (I've never worked anywhere that a "coder" job role existed), but a "dev" (a.k.a. "developer") is a much broader role than "coder" (and pays much better - so, yes, I probably pay them three times what I would pay a "coder" if I could find an efficient use for the latter). Anyone on my staff who only has potential to be a "coder", is sent on their way as it's faster to write the damned code than to write a detailed design spec that a one dimensional "coder" can code from -- and the result is generally much better (unless, of course, the detailed design spec is so detailed that the coder has to make no decisions -- in which case, just write the spec in Java or C++ and compile it).

Re:Automated test in is a minimum (0)

Anonymous Coward | about two weeks ago | (#47819161)

Did you just fucking say a QA tester knows more about testing than a programmer? HAHAHHAHAHAHA!!!!

Re:Automated test in is a minimum (1)

Livius (318358) | about two weeks ago | (#47820155)

Unit testing legitimately comes under both job descriptions, though
1) the coder really understands the meaning of 'unit' in a particular context, and
2) the tester is the subject matter expert on testing.

The two need to have a dialogue. If they aren't talking, there are bigger problems.

Re:Automated test in is a minimum (2)

Whatsisname (891214) | about two weeks ago | (#47818509)

That's not going to work, because you'll never be able to economically write a requirements document so complete that the behavior is so well defined that you can get meaningful test coverage from it.

To get that kind of completeness you'd have to code the entire software in MSWord, which is a terrible programming language, and without ever testing it along the way.

Testing needs to be a continuous process as part of software development, not something that happens parallel or afterwards.

Re:Automated test in is a minimum (1)

angel'o'sphere (80593) | about two weeks ago | (#47817641)

In a static typed language unit tests are pretty pointless. You only have them because your developers are using cut/paste to much and accidentally hit return when a line was selected (and delete it by that).
You are far better off if you write user acceptance tests on use case or story level.

I have worked on several multi-million line commercial applications, some with 500 million+ users. I have never seen a piece of code which could not be properly tested using the right tools. That can even include small embedded systems where we would have to actually implement a QEMU module or three I would not count web site visitors as "users", but well ... technically they are.

I can write you in 40 lines a function which you wont be able to automatically test. It only needs some nested loops and an 'if' cascades with loops inside.
Sure: you can refactor that into a lot of small functions, containing only a single loop or a single 'if'.

You lost quite some credibility with using terms or sentences like That even includes memory leaks when exiting. and "... even if that means correct use of the garbage collector ..."
Unfortunately a exiting program can not leak memory, and similar unfortunate, you don't use a garbage collector, it runs in the background. You can parametrize it perhaps ... but thats it. (and please don't tell me you are doing System.gc() in Java programs at "random" intervals)

Re:Automated test in is a minimum (3, Interesting)

swillden (191260) | about two weeks ago | (#47818355)

In a static typed language unit tests are pretty pointless.

Because static typing catches all bugs? That must be some statically-typed language that I've never seen. Unit tests are perhaps marginally less necessary than in dynamically-typed languages, but they're still necessary. Test-Driven Development is a life saver regardless of your toolset.

I can write you in 40 lines a function which you wont be able to automatically test. It only needs some nested loops and an 'if' cascades with loops inside.

There's nothing untestable about such a function. Basic code coverage tools will even identify any branches within the code that aren't taken, so you know to look for ways to write test cases that cover those branches. What's harder is ensuring coverage of all of the issues that could be provoked by different input data, even after you've covered all of the paths. With experience you learn to do that quite effectively, though.

Sure: you should refactor that into a lot of small functions, containing only a single loop or a single 'if'.

FTFY. Change it to "must" if I'm your code reviewer.

You lost quite some credibility with using terms or sentences like That even includes memory leaks when exiting. and "... even if that means correct use of the garbage collector ..." Unfortunately a exiting program can not leak memory

Sure it can. If there are any heap-allocated blocks remaining (not freed) at exit, the program has a memory leak. Again, there are good tools to help you find these leaks, like valgrind memcheck.

you don't use a garbage collector, it runs in the background. You can parametrize it perhaps ... but thats it. (and please don't tell me you are doing System.gc() in Java programs at "random" intervals)

And yet you can still have leaks in garbage-collected environments, and there are ways to test for them. It's a bit more complex than in non-GC'd environments, but it can -- and must! -- be tested if you want to have reliable software.

Re:Automated test in is a minimum (1)

angel'o'sphere (80593) | about two weeks ago | (#47818551)

Rofl.
Acacademic half knowledge with complete wrongs: If there are any heap-allocated blocks remaining (not freed) at exit, the program has a memory leak
No it has not. Where should the memory go to after the program has exited?

Technically you can test a 40 lines mess of loops and if cascades, practicaly you can't ... or how likely is it that you can prove me in a reasonable time that a certain branch in an if - else inside of a cascade of nested loops and if's is executed with meaningfull data in your test? Especially if I have written the function and you want to write the test?

The rest I leave as it stands if you like to argue about how likely it is that a single compilation unit has a bug that is not dicovered by a functional user acceptance test ... unit tests without user acceptance tests or integration tests are pointless. The next thing is you tell me to test getters and setters ...

But you will figure that soon enough when you have 100% code coverage and still have bugs and wonder why

Btw, I never was in a team that had a memory leak in a GCed language. And I certainly don't waste resources on testing for that unless I have a strong indication that we have a leak.

As I said above: you are full with academic half knowledge and lack (despite your claims to the contrary) real live experience in large systems

Re:Automated test in is a minimum (2)

Your.Master (1088569) | about two weeks ago | (#47818991)

He's saying you can detect runtime memory leaks at exit time.

The memory won't be leaked anymore post-exit, and the leak doesn't matter at exit time, but that's when it is possible to deterministically detect memory leaks over the course of the software running.

Re:Automated test in is a minimum (1)

angel'o'sphere (80593) | about two weeks ago | (#47819397)

Nevertheless he is wrong, even if your interpretation of his words is right.
In C/C++ like languages no one is going to free/delete any memory at the point of exit(), you simply call exit() and are done with it.
but that's when it is possible to deterministically detect memory leaks over the course of the software running.
No it is not :) how should that work?

Same in garbage collected languages. It is close to impossible to free all memory in a garbage collected environment ... you programmed the whole application agnostic of memory allocation and then you want to travel all objects and set all references to null (and somehow trigger the GC) to have everything 'cleared'? (Sure, I can write a reflection based graph traversal that sets every reference it encounters to null. But what would be the point?)
How should any of that be related to memory leaks that happen during normal operation of the program?

Re:Automated test in is a minimum (4, Interesting)

swillden (191260) | about two weeks ago | (#47819751)

Rofl. Acacademic half knowledge with complete wrongs

Dude. I've been a professional software developer for 25 years, and am a respected engineer at one of the biggest names in software. You use my software daily.

If there are any heap-allocated blocks remaining (not freed) at exit, the program has a memory leak No it has not. Where should the memory go to after the program has exited?

Well, obviously it'll all be returned to the system after exit. The point is to check AT EXIT. If there are any blocks still allocated, you've screwed up.

Technically you can test a 40 lines mess of loops and if cascades, practicaly you can't ... or how likely is it that you can prove me in a reasonable time that a certain branch in an if - else inside of a cascade of nested loops and if's is executed with meaningfull data in your test? Especially if I have written the function and you want to write the test?

I've done it many times. Just check to see which branches aren't being executed and work out what needs to be done to execute them.

Though it's much, much better to refactor the function first.

The rest I leave as it stands if you like to argue about how likely it is that a single compilation unit has a bug that is not dicovered by a functional user acceptance test ... unit tests without user acceptance tests or integration tests are pointless.

I've found thousands of bugs with unit tests that were not discovered by functional tests, integration tests, or user acceptance tests. In fact, unit tests are the most likely ones to find thing like subtle boundary condition errors which are hard to reproduce and are the source of nasty, rare bugs that are seen in production only once in a while.

The next thing is you tell me to test getters and setters ...

Typically those get exercised by integration tests... and it is a good idea to have coverage of them at some point in your automated suite, because the whole point of getters and setters is that they may someday become non-trivial. Writing tests for them at that point isn't as good as already having tests in place, because you want to verify that your new implementation doesn't have subtle differences in behavior.

But you will figure that soon enough when you have 100% code coverage and still have bugs and wonder why

No one is claiming that unit tests are sufficient, only that they're necessary.

Btw, I never was in a team that had a memory leak in a GCed language.

Then you haven't worked on many non-trivial systems.

Re:Automated test in is a minimum (0)

angel'o'sphere (80593) | about two weeks ago | (#47820229)

Sorry, Well, obviously it'll all be returned to the system after exit. The point is to check AT EXIT. If there are any blocks still allocated, you've screwed up.
Sorry, this is simply wrong.
No one is calling free/delete before exit() on all still living objects to show that he has no memory leak. You simply do the 'do you realy want to exit', 'do you want to safe ....' routine and be done with it.
Your 25 years helps you nothing if you don't understand such simple stuff (BTW: I'm lomger in business than you).
What would be the point of freeing memory in an exit routine, where the memory management has absolutely nothing to do with the memory management during runtime of the program? Just an excercise 'I know where all my pointers are'?
You really want to do that and then you conclude: during normal operation (before exit) I have no memory leak? Rofl ...

The rest of your posts shows that you did not learn much in your 25 years career.

Then you haven't worked on many non-trivial systems. Unfortunately I only worked on non trivial systems.
And you might not believe it but I never had a memory leak in my own code, regardless of C/C++ or Java. Perhaps during developent ... but never when released or shipped. And as I said before in Java I never was in a team where we had a memory leak. A memory leak in a GC language means not that you messed up memory management but that your algorithm is wrong!.

Hence my point about unit testing.

I take it you don't do user acceptance tests on story or use case level? So why do you want to argue with me about unit tests? Or don't you know what a unit test actually is and use the term wrongly? A unit test tests a single compilation unit. Hence its name. E.g. a single class in Java/C#/C++

I've found thousands of bugs with unit tests that were not discovered by functional tests, integration tests, or user acceptance tests. In fact, unit tests are the most likely ones to find thing like subtle boundary condition errors which are hard to reproduce and are the source of nasty, rare bugs that are seen in production only once in a while.
Every sentence here is nonsense. To find thousands of bugs you must work in an environment where people produce bugs instead of code. 25 years * 45 work weeks * 5 days = 5625. So you need to find a bug at least once a week (over your 25 years programming career).

If you have a unit test that 'finds a bug' but the acceptant tests did not, your acceptence test was: wrong, or did at least not cover the relevant line :) Before you write unit tests for existing code you should write user acceptance tests for it, then check the code coverage. After all users don't care if a method in a class that is never called is not tested, but they care if the button they click still does the same thing.

Unit tests don't find the things you state above. They are only usefull to confirm a class works according to the specc (and that specc can be wrong, too) They fail or succeede in tests, they don't find bugs. The only point where unit tests are really helpfull is if you either write libraries or you do heavy refactoring. Or to teach new kids how to code, having the test first and let it call the methods you want to implement helps in thinking. And then still user acceptance tests are much better.

Starting with unit tests in a language like Java, or C# (strongly typed languages) is in most cases a waste of time and give you a false sense about your code quallity (which matches with your claim to have found thousands of bugs ... where did they come from at the first place?)

WTF the spelling correction is broken again on this iPad, please ignore spelling errors, I can not see them if they are not red underlined.

Re:Automated test in is a minimum (1)

DarkOx (621550) | about two weeks ago | (#47818681)

Sure it can. If there are any heap-allocated blocks remaining (not freed) at exit, the program has a memory leak. Again, there are good tools to help you find these leaks, like valgrind memcheck.

Really can you point to any contemporary operating system that would NOT free all the memory allocated to a process when it exists? I guess you might mean if your process asks other "servers" to do things like say just exists without closing database connections etc, the other process might not free resources associated with yours but that is not the same thing as a memory leak.

Re:Automated test in is a minimum (1)

swillden (191260) | about two weeks ago | (#47819835)

Really can you point to any contemporary operating system that would NOT free all the memory allocated to a process when it exists?

Obviously not. If there were such an OS it would be broken.

The point is that if you're managing your memory well, by the time you exit every allocated block will have been freed by your code. Yes, this is a little more work than appears to be absolutely necessary, but when that function which allocated a block which you expect to be released at exit, and which you know is only going to be called once, ends up being called in a loop a few years later, the maintainer who makes that change will hate you. And note that said maintainer may well be you.

Re:Automated test in is a minimum (1)

DeBaas (470886) | about two weeks ago | (#47817683)

First off, I would love it if people took unit testing more seriously and automated it! In my view that helps greatly for getting robust software. However, not for all tests automating is the answer. Especially when you get to acceptance testing (where we validate, rather than verify) or when you do integration testing for systems that communicate with other systems it is not a silver bullet. Aside from feasibility, as automating is time consuming, there are more drawbacks. Automating all tests means assuming that you can anticipate everything that can be wrong and even anticipate every test that should be done just based on the specs. Exactly the context driven testers (such as James Bach and Michael Bolton) believe you can't, we humans are not 'wired' that way. In fact they even are of the opinion (as am I) that the best testing is done by people that design tests to a great deal as you go while testing as long as these are skilled people that understand how the software should work.

Rigorously testing software via automated tests, please do. But in my view there should always still be people that test and see for themselves. The automated tests can and will miss things that are plain obvious to human testers.
   

Re:Automated test in is a minimum (1)

Snotnose (212196) | about two weeks ago | (#47818703)

The automated tests can and will miss things that are plain obvious to human testers.

True dat. The solution is that for every bugfix submitted there is also an automated test to verify it stays fixed.

Automated test suites are not static. They should grow as the project matures and users/developers gain experience with it.

Re:Automated test in is a minimum (1)

Anonymous Coward | about two weeks ago | (#47817719)

I agree almost 100%, let's say 95%. Automated testing is fantastic. It's not everything though. You can't automatically test for "does it annoy and/or confuse the operator". For that, you need a human to test it. Also I think there are going to be aspects of the test that could be automated and you just didn't think of them. Acceptable performance falls under this category somewhat. An automated test won't tell you that something takes too long to render unless you know what "too long" is for a human. A human has to sit in front of the integrated app and test it at some point.

Like I said though, 95% agreement.

The only automated tests I don't like are the "test suites" that are nothing more than glorified asserts. They're a wast of code. Somebody tried to get me to use one of those one time. I was already doing all my developer unit tests with text files and diff. You print the outputs of functions under test to output.txt. You diff it against expected.txt. If diff outputs anything, it's failure. This is so much more powerful than an assert, and it doesn't require polluting the code under test with anything, or linking a special lib:

Re:Automated test in is a minimum (2)

uncqual (836337) | about two weeks ago | (#47819183)

Asserts are fairly useless as a testing tool. Their excessive use can be extremely annoying as they clutter up the code and that clutter can increase the chances of introducing bugs.

However, responsible use of asserts can be quite useful in debugging - esp. when a customer is encountering a problem that is difficult to recreate (or, perhaps, nearly impossible from a practical standpoint because they won't give you access to their confidential data/environment). In those cases, running a "well asserted" debug version of the system on the customer site can sometimes help narrow the problem space substantially very quickly - not always of course, but the cost of running the test is often very small so it has a good ROI. Of course, if you're going to do this, you need to keep the asserts valid so systems test often needs to validate both the version WITH and the version WITHOUT the asserts enabled - else it's likely that the "well asserted" version has rotted and will result in a flurry of assert failures that actually don't represent a problem.

Of course if you're writing code that controls the United States nuclear weapon launch system, ignore the advice above.

Re:Automated test in is a minimum (1)

michaelbolton (554690) | about two weeks ago | (#47817839)

You are confusing testing (evaluating the product by learning about it through experimentation) with what we call checking (evaluating functions in the product by algorithmic observations and decision rules). http://www.satisfice.com/blog/... [satisfice.com] . Checking can find lots of functional problems in the product, and it's an important thing to do. But that's not the same as testing the product. Knuth made this distinction using different words years ago: "Beware of using this product. I have merely proven it correct; I haven't tested it."

but how many tests? (1)

pr100 (653298) | about two weeks ago | (#47817845)

Whilst unit testing is a good thing in general it's actually hard to know how many tests to write. Whilst it is in theory possible to write tests for most things, it's completely infeasible to a priori write tests every for possible interaction with any reasonably sized bit of software. In practice everyone ships code with gaps in the test suite.

There are contexts in which we should be formally proving the correctness of code. This tends to be a niche thing tho' because it's hard and time-consuming.

Re:Automated test in is a minimum (2)

Just Some Guy (3352) | about two weeks ago | (#47818585)

I love testing. Honestly, if forced at gunpoint to give up testing or version control, I'd be hard pressed to pick. Testing means I can update an innocent-looking line of code without worrying whether it's going to break some arcane business logic that a customer depends on, and that I can gut and reimplement an API while having a reasonable expectation that consumers will keep working afterward. I'm a huge testing advocate.

But.

I have no desire whatsoever to conform to an ISO document about their idea of the right way to do things. Standards are great for ensuring interoperability, but how often do you care about that in a test suite (beyond the minimal "can I make its output look like JUnit so Jenkins can yell at someone when it breaks")?

I'll sit down next to you and help you crank out a comprehensive test suite, because while that's a tedious pain in the ass, it's far less of a pain in the ass than not having one. But I couldn't possibly care less whether the result of our labors is "ISO 29119 Certified (tm)!", probably by a $300 outside consultant, so that we can put an "ISO 29119 Certified (tm)!" logo on our website for PHB's to smile approvingly and longingly at and then forget about.

This sounds... familiar.... (3, Funny)

QilessQi (2044624) | about two weeks ago | (#47817337)

By implementing these standards, you will be adopting the only internationally-recognised and agreed standards for software testing, which will provide your organisation with a high-quality approach to testing that can be communicated throughout the world.

....Be the first on your block to collect all five! GOTTA CATCH 'EM ALL!

SSL, anyone? (1)

fuzzyfuzzyfungus (1223518) | about two weeks ago | (#47817347)

While the idea of having a well vetted testing system in place that would allow customers to choose software that had been so vetted seems like a good idea, I have to wonder if it's doomed to the fate of SSL, at least outside of a few niche applications that mostly demand high levels of verification anyway.

With SSL, we all wanted the security; but everyone wanted it to be cheap, and wanted to avoid a monopoly over certificate authorityship. So, what did we get? A mass of CAs, many painfully shoddy, who will issue you a fancy looking cert for peanuts. Then we got "EV" certs, which are supposed to actually do what the original certs were supposed to do, only more expensive.

Standards Committees on Common Sense (0)

Anonymous Coward | about two weeks ago | (#47817351)

The point of a standard is to pick a convention and build on it. Screw threads are standardized. Resistor color codes and values are standardized.

That's a good standard.

The part that makes me boil is where standards committees attempt to codify what a manager does by writing management standards. Most of them are overwrought variants of Plan, Do, Check, Act. DO WE REALLY NEED ANOTHER STANDARD LIKE THIS?

This kind of foolishness has gone way past the point of humor, past the point of ridicule, past the just plain stupid, and gone to just awfully sad.

Make It STOP!

Where is the standard????????? (0)

Anonymous Coward | about two weeks ago | (#47817365)

THe link in the article points to: http://www.softwaretestingstandard.org/
THe links there only have tables of contents:
http://www.softwaretestingstandard.org/part1.php

Where is the actual content?

Re:Where is the standard????????? (0)

Anonymous Coward | about two weeks ago | (#47817717)

http://www.iso.org/iso/home/store/catalogue_tc/catalogue_tc_browse.htm?commid=45086

Re:Where is the standard????????? (1)

Mostly a lurker (634878) | about two weeks ago | (#47817817)

I guess the website's testing process failed to catch the edge case where someone wanted to navigate past the home page.

Re:Where is the standard????????? (1)

TechyImmigrant (175943) | about two weeks ago | (#47817855)

You have to pay.

just like ISO 9000, that worked well! (2)

thebeastofbaystreet (3805781) | about two weeks ago | (#47817387)

I mean, seriously, what kind of idiot thinks that have a standard for this will make any difference at all? Quality costs money and time end of story.

Re:just like ISO 9000, that worked well! (0)

Anonymous Coward | about two weeks ago | (#47817451)

... but spending money and time does not guarantee quality.

Re:just like ISO 9000, that worked well! (1)

itsdapead (734413) | about two weeks ago | (#47817771)

... but spending money and time does not guarantee quality.

Nor does a certificate that proves that you spent time and money on paperwork.

Re:just like ISO 9000, that worked well! (0)

Anonymous Coward | about two weeks ago | (#47817779)

... but spending money and time does not guarantee quality.

That's not why you do it.

You expend the effort to mitigate risk. That's it. Spending a dime more is often considered a waste, so this has little to do with quality.

Re:just like ISO 9000, that worked well! (1)

angel'o'sphere (80593) | about two weeks ago | (#47817671)

In RL quality saves money.
Unfortunately many small (and also big shops) have not realized this yet.

Re:just like ISO 9000, that worked well! (1)

sjames (1099) | about two weeks ago | (#47817825)

ISO9000 is a really sad joke. If you have a shitty process that produces poor quality, you can pass ISO9000 just fine. From that point on, it will make sure you never accidentally produce a mediocre or (god forbid) a good product.

Most likely against it... (0)

Anonymous Coward | about two weeks ago | (#47817435)

Most of the people likely against it probably had or already have no intention of testing their shit software in the first place!

See: pretty much every website that runs PHP, most Microsoft products for some large examples
Google too. Goddamn Google.
A company so huge they can't even make non-corruptable file formats. That was solved decades ago.
I gave up after Picasa corrupted its database for a 4th time.
Google Chrome, regularly screws its own installations up. I've had it happen to me. I've had it happen to family. My friends have had it happen to them.
I've been programming from 9, and my friends mostly work in software and networking as well, and I can't say I have seen a worse program than Chrome, besides maybe Skype. (AND I STILL USE THEM)
Skype v5 had an almost instant BSOD if there was anything even remotely intense running while you answer a call. The videoframe wouldn't be loaded and it would try to draw to it... It was still there up until even Microsoft bought them. (oddly enough, the MSN Developers were actually the good devs in Microsoft, and the MSDN people for the most part)

What. The. Hell. Are. You. Doing. Developers. ?. That was totally accidental. OR WAS IT?!

There will almost certainly be a few legit reasons for people being against it, however.
One of them being "if you aren't certified, it must mean you are crap".

Re:Most likely against it... (0)

Anonymous Coward | about two weeks ago | (#47817927)

This shows you know nothing about software development and software testing. You can test the shit out of the software but you cannot test every single possible scenario because of the number of variables that exist.

Re:Most likely against it... (1)

michaelbolton (554690) | about two weeks ago | (#47819307)

"Most of the people likely against it probably had or already have no intention of testing their shit software in the first place!" You are spectacularly wrong about that. Apparently you probably had or already have no intention of testing your theory; not even by the most cursory reading of any of the materials referred to at the beginning of this thread.

It will "catch" on (1)

Anonymous Coward | about two weeks ago | (#47817467)

The moment when someone with little real experience is tasked with designing a "testing methodology", they will look around the internet an choose an "industry standard". They have seriously tried to implement something like that at my current gig (due to gov regulations), and the results are hilarious. I wrote an application last week, this week I'm doing the design of the application, and I will hopefully have requirements by the end of the week. Yep, in that exact order...
Ohh, and they also forgot to have a real testing phase (as opposed to the validation phase), so now the Q&A people only do testing (in a clandestine way of course, since it's not part of the process) when they're not too busy documenting the process.
So the end result is a mix of a dysfunctional procedure and just plain sneaking things in under the radar to try to get anything done.

Re:It will "catch" on (1)

uncqual (836337) | about two weeks ago | (#47819433)

Wait... I thought Healthcare.gov [healthcare.gov] was coded long ago. Are you working on version 2?

Yuo ]F4il It? (-1)

Anonymous Coward | about two weeks ago | (#47817545)

NetBSD p0sts on during play, this Is the group that (7000+1400+700)*4 1n a head spinning First avoid going shall we? OK!

I see a bunch of whiners (2, Informative)

sirwired (27582) | about two weeks ago | (#47817551)

It seems as if their chief complaint is that they were not asked to provide input, and the personal communications with members of the committee didn't go anywhere. That's not how the standards process works (I'm speaking from the IEEE perspective, anyway; don't know how ISO works)... your organization (at least from the IEEE end, this is open to pretty much anybody that can muster up the nominal dues) signs up to be on the standards committee, you pay a nominal fee to be included in the working group, and Pow! Your organization is now a full voting member for the standard.

If you don't sign up for the working group, then it should be no surprise that your input is considered entirely optional and/or ignored entirely.

In the first article, the author describes a management course where a group was supposed to form a consensus. He complains that he disagreed with everyone else, wouldn't change his mind (because of his self-proclaimed "high-standards"), and was therefore excluded from the final output from the group, which then was reported to be a consensus. He disagreed that there was a consensus at all, since he didn't agree with it. That's not how "consensus" works; it does not mean that everybody will be satisfied with the outcome, or even want to be associated with it. He goes on to complain that the ISO process requires "consensus", but since he, and like-minded individuals, disagree with the standard, it should not be cleared as a standard.

Again, not how consensus works. In a consensus process, the majority approve of whatever the final output is, and the objections of the dissenters are noted and made available as part of the standards record. You can look on the website of pretty much any standards organization and access drafts, comments, meeting minutes, presentations, the whole works. This full record can help potential adopters of the standard decide if they want to utilize it or not.

Re:I see a bunch of whiners (1)

Anonymous Coward | about two weeks ago | (#47818129)

If you don't form industry consensus then don't expect a consensus of a minority of people from that industry (working on a standard) for it to be meaningful to the industry as a whole, regardless of who is to blame. This will be another dead standard no-one uses at worst or at best, another checklist item in the list of certifications some stakeholder somewhere wants but no-one actually cares about... and I speak of this as someone who has worked with a working group who developed a standard no-one cared about for the same reasons.

Re:I see a bunch of whiners (0)

Anonymous Coward | about two weeks ago | (#47818447)

Yeah the man is a whiner and his complaints are pathetic. When someone asked to present his criticism regarding the standard, his only answer is that there is no standard. And why is that? Because as he didn't agree with the process, he is pretending that nothing was ever defined.

And thus he avoids providing a single anwer or present a single tangible complaint regarding the standard.

To me, this whining is just grandstanding. It looks like this clown is trying to use the standardization committee to advertise his business and his supposedly rebel philosophy regarding unit testing. He is also depicting himself above any other tester, particularly the ones in the standard, because he supposedly has higher quality standards than his critics, and he claims that his astonishingly high quality standards that he supposedly expects from himself and others were the reason why he was kicked out of the process.

It's a whiner's version of "bitches can't handle my quality standards".

The whole idea is a pathetic attempt at grandstanding.

The bit were it gets really pathetic is the part where he starts to complain about rent-seeking and someone using the standards to start consultancy gigs and related business deals, as this only became a problem because he failed to shoehorn his opinions and professional expertise accepted into the standards.

Them grapes, they are sour.

Re:I see a bunch of whiners (1)

michaelbolton (554690) | about two weeks ago | (#47818527)

Would you agree that consensus gained by attrition and rent-seeking should be used to determine the way things are done in health-, safety-, or finance-related enterprises upon which you, your loved ones, and your nest egg depend? Are you saying that IEEE/ISO working groups should ignore what's going on in the world because the process is designed to favour those who are driven by profit, but who do not have skin in the game? http://www.developsense.com/bl... [developsense.com] ---Michael B.

Then participate! (1)

sirwired (27582) | about two weeks ago | (#47818923)

Firstly, I'm not going to answer stupid leading questions. What is this, some kind of sound-bite-driven political debate?

If you don't like the way the standard is going, you form an organization of like-minded individuals and join the working group. Spread amongst a group of people, the costs are not that extreme, nor the commitment that dire.

And I don't the ability of education providers and consultants being able to advertise "We teach/use the XYZ Standard" as being some sort of nefarious plot. If you are looking for advice on something, being able to have a decent idea what you are going to learn about without extensive interrogation and negotiation can be quite useful for both parties.

The signing of an online petition is guaranteed to be utterly and completely ineffective... the costs of actual participation are so low, it's not entirely unjustified to ignore an online petition as anything other than an isolated group. There's a process to get heard on the issue, and internet petitions, and combative communications with those involved are not it.

I read that petition: "Because the people that signed this petition who couldn't be arsed to participate in the process disagree with the standard, consensus is not possible." How was THAT every going to go anywhere? "Consensus" is not reached (or not reached) by waiting until any schmoe with an axe is satisfied; consensus is reached when the committee votes on a standard and approves one. And yes, sometimes consensus cannot be reached, and the standard simply dies... that's a perfectly valid outcome too.

Curse my typos! (1)

sirwired (27582) | about two weeks ago | (#47818967)

*And I don't see how the ability*...
*ever*
*not reached (or failed to be reached)*

Re:I see a bunch of whiners (0)

Anonymous Coward | about two weeks ago | (#47819363)

Would you agree that consensus gained by attrition and rent-seeking should be used to determine the way things are done in health-, safety-, or finance-related enterprises upon which you, your loved ones, and your nest egg depend?

After complaining about click-baiting laden titles, in your own writing at the link you provided, that's a pretty ironic statement.

Another thing is you are not exactly clear how you know that the either the majority, or maybe even all involved in this standards working group and process, is apparently rent-seeking and intends to gain a dominate position through size, attrition, or whatever, to abuse the process or the standards. I said intends, because you fail to produce a single piece of evidence, or even reference to anything, that the standards themselves or the working group's process was abused. It sounds like pure speculation, conjecture, and fiction. Otherwise known as fear mongering.

Are you saying that IEEE/ISO working groups should ignore what's going on in the world because the process is designed to favour those who are driven by profit, [...]

Are you really trying to claim that your business and yourself have no financial motivation in your actions? Or do you do all of your consulting gratis, merely on principle alone?

You are free to ignore this standard, just as you are free to ignore SAE, ISO 9000, ITIL, PRINCE2, IEEE 802.11abcgn, ITU V.32bis, or any of the other thousands of standards that already exist, and are not legally binding. At worst, you are free to stop consulting about software testing if all your potential clients require IEEE/ISO happy-shiny TQM [wikipedia.org] , and go build wooden sailing ships if you wish. I expect the market may be rather soft in its demand for wooden sailing vessels, but that's you choice.

ISO/IEC 29500 (1)

fustakrakich (1673220) | about two weeks ago | (#47817561)

ISO? Who are they?

Re:ISO/IEC 29500 (1)

pjt33 (739471) | about two weeks ago | (#47819179)

International Social Outcasts.

Can it be a *useful* standard (2)

g01d4 (888748) | about two weeks ago | (#47817791)

The Bolton-Christie argument, to me, boils down to: you can have too much of a good thing, e.g. documentation. This can impose unnecessary costs and defeat the purpose if, following the above example, onerous documentation doesn't get read. Too much of a standard means unnecessary cost goes out to the standards industry (rent seeking).

Re:Can it be a *useful* standard (2)

michaelbolton (554690) | about two weeks ago | (#47818603)

That's certainly an important point. There are others. To me, the issue not just the cost of preparing the documentation, but the degree to which compliance with the standard displaces the goal of actually testing the product or service. A moment that a tester spends on useless documentation is a moment in which she's not focused on identifying risks and finding problems that would cause loss, harm, or annoyance. http://www.developsense.com/bl... [developsense.com]

Ian Betteridge says no. (1)

NemoinSpace (1118137) | about two weeks ago | (#47817793)

Standards are written in ivory towers. MCA and BetaMax were some of the most impressive standards i never used. I'm not saying standards aren't useful, but they are not a substitute for "just works".
Companies make cheap crap because that is what most people want. Other people buy expensive crap. What they NEVER do is pay a lot of money for cheap crap. Standards help define these segments.

Consensus? You have to be kidding... (1)

NReitzel (77941) | about two weeks ago | (#47817961)

The free dictionary (by Farlex) defines consensus: 1. An opinion or position reached by a group as a whole.

That's very democratic. Unfortunately, reality is not democratic.

Software testing is designed to unveil real vulnerabilities and errors in a complex system. Having a bunch of people hold up their hands and say, "Is this a problem?" is flatly ludicrous. In point of fact, it's the error that isn't noticed by the majority that constitutes the deepest problem. Remember the Columbia shuttle? A group of people got together and came to the concensus that the ice impact at launch was not a problem.

Testing, by it's very nature, is not subject to regimentation. It's a lot like "Job Descriptions" -- in real terms, establishing a job description is publishing a whole list of things that don't need to be addressed. Why does anyone think software testing will be different?

"Your piece of software has problems." "No, it doesn't. We fulfilled the standard for testing."

Giveth me a break.

Only $775 to read the first 3 parts (1)

daverk (38859) | about two weeks ago | (#47818117)

First 3 parts of 29119 cost $775, the total for all the parts when they are finished will probably be around $1200. This seems to be more about money than anything else.

microsoft's snide refusal (0)

Anonymous Coward | about two weeks ago | (#47818347)

Posting AC for obvious reasons...

When IEEE came calling to our org, they were repeatedly referred to "Captain Canada," a no-talent ass-clown PR guy in MSRC with an engineer title. He simply pointed to the SDL and told IEEE that "this is a solved problem, don't waste our time." At one point he was so rude to the IEEE coordinator he made her cry, and while that doesn't excuse IEEE's weak consensus-gathering, it does put some light on Microsoft's utterly tone-deaf and solid-bone-stupid response.

This mess is as much the community's fault (*our* fault) as IEEE's.

-AC

Of course you can have a standard (1)

geekoid (135745) | about two weeks ago | (#47818495)

You may have to extend it in some cases, but that's normal.

On standards (2)

mr_mischief (456295) | about two weeks ago | (#47818669)

The best known standard quip about standards itself has multiple versions and attributions. How meta:

"The nice thing about standards is that you have so many to choose from." - Andrew S. Tanenbaum, Computer Networks, 2nd ed., p. 254 [wikiquote.org]

"The nicest thing about standards is that there are so many of them to choose from." -- Ken Olsen [brainyquote.com]

“The wonderful thing about standards is that there are so many of them to choose from.” -- Grace Murray Hopper [goodreads.com]

See also:

Obligatory (but who set that standard?): xkcd : Standards [xkcd.com]
Why are there so many plugs and sockets? [www.iec.ch]

‘Mediocrity finds safety in standardization.’ -- Frederick Crane
‘It is not enough that X be standard, it should also be good.’ -- Rob Pike (Window Systems Should Be Transparent)
The two above can be found on the cat -v page on standards" [cat-v.org]
"Standards are like toothbrushes. Everybody wants one but nobody wants to use anybody else’s." -- Connie Morella [niso.org]

No (0)

Anonymous Coward | about two weeks ago | (#47818873)

I am disinclined to acquiesce your request.

That can't come up with a standard without a consensus, and they never asked me nor the general public. It most likely is a way to control the software that is released into the wild, and I expect some OS's like Windows to perform the so called "standard test" and if specific hidden code (set of commands) is not included in it, then it will fail the test.

I won't go along with it under any circumstances, and I encourage the rest of you to ignore it as well.

On "concensus" (1)

DutchUncle (826473) | about two weeks ago | (#47818951)

"It sometimes happens in a people amongst which various opinions prevail that the balance of the several parties is lost, and one of them obtains an irresistible preponderance, overpowers all obstacles, harasses its opponents, and appropriates all the resources of society to its own purposes. The vanquished citizens despair of success and they conceal their dissatisfaction in silence and in general apathy. The nation seems to be governed by a single principle, and the prevailing party assumes the credit of having restored peace and unanimity to the country. But this apparent unanimity is merely a cloak to alarming dissensions and perpetual opposition." - Alexis de Tocqueville, published 1831

Standards are not "All or Nothing" (0)

Anonymous Coward | about two weeks ago | (#47819591)

I've done lots of work under various DOD, DOE, FAA, ISO, IEEE, and many other publishers of standards.

Not once have I ever achieved "full compliance" with any but the most trivial standards. What we typically did was cherry-pick standards to highlight what was most useful in our specific situation, eliminate what was inapplicable (and try to find suitable replacements), and assess the rest for usefulness and effectiveness ("worth the effort").

This is called working to a "tailored" standard, and is very a common industry practice. For standards such as the FAA's DO-267, tailoring is required, because parts of the standards actually conflict (in a "choose path A or path B" sense).

I've found many standards to include some truly useful and insightful gems, though that's under 1% of the bulk. Still well worth finding and using. I'm especially a fan of the ISO-900x series, which has positively transformed every business I've been with that applied them properly (rather than wasting time and effort paying lip-service to them).

There are also "guidelines" that are just short of being standards that every software practitioner should be aware of, such as those from Carnegie Mellon's Software Engineering Institute for secure/mature software development. The MISRA guidelines are "the book" for embedded C programming.

Standards are not religious doctrine, to be slavishly adhered to upon fear of condemnation. They are the collected wisdom of groups of experienced practitioners and managers (perhaps groups that don't include you, but often still insightful).

Take the best, leave the rest. Move along.

Re:Standards are not "All or Nothing" (0)

Anonymous Coward | about two weeks ago | (#47819733)

I hate replying to my own posts, but 2 more thoughts:

1. Learning to quickly read and comprehend standards is a skill of its own, especially as too many standards appear to be written in obfusucatese. Most have a "Table of Terms" or "Glossary", which is the first part that must be understood. Most also rely on other standards, including standards for creating standards: Learning these makes for a great foundation.

2. Nearly all standards compliance checks rely on "artifacts" that are generated as part of adhering to the process. Some of these artifacts are the prodedures used within an organization to adhere to the standard, but most are products of doing the work (reports). To the greatest extent possible, the reports should be generated automatically, such as by an automated build system. Once automated, compliance becomes close to free.

CAPA vs. Barrier to Entry (1)

retroworks (652802) | about two weeks ago | (#47819973)

Most people don't understand the compliance. There's good and bad, but there's no going back once your industry (candle makers, software writers, barbers, whoever) adapts a standard it invariably becomes a tool of an authority.

Good: What I like about it is that our certifications increase accountability by encouraging recording mistakes. The "routine" of flagging mistakes and finding root causes and formalizing "corrective and preventative action" has been good and improved our company.

Bad: These standards are adapted by many companies in order to reduce competition, take away via consensus unique individual methods for doing things. They become almost like a "union", punishing individual innovation via auditors that view the world inside a "box". Uniqueness and innovation are an increased cost and risk to the third party auditor, and the auditor is ready to adapt the majority interpretation - which is usually to increase barrier of entry into the field of competiton.

As Morris Kleiner, the AFL-CIO chair in labor policy at the Humphrey School of Public Affairs at the University of Minnesota, put it "Occupational licensing has either no impact or even a negative impact on the quality of services provided to customers by members of the regulated occupation."

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>