Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

How Do You Prove Software Testing Saves Money?

samzenpus posted more than 3 years ago | from the taking-it-for-a-test-drive dept.

The Almighty Buck 312

cdman52 writes "I work at a small software development company. We have one app that is used by a few hundred clients and was initially developed by a few undergrads about 10 years ago. The app is collection of about 25 developers preferences and ideas. Testing wasn't an initial concern since it was created as an internal application, I guess. Anyway, the app is now large and used frequently. Part of my duties are to fix bugs users find, I'm on a team with a few other people and at least once every 2-3 months I see some bug I fixed come back, and I can only assume it's because we don't have a formal test suite. The owner doesn't want to invest time or money in getting one set up, but I'm sure that in the long run it would save time and money. Can anyone offer suggestions for how to convince the owner that setting up a test suite is in his own best interest?"

cancel ×

312 comments

Sorry! There are no comments related to the filter you selected.

Design from the beginning is important too. (3, Insightful)

ls671 (1122017) | more than 3 years ago | (#34741948)

I would say the most important thing to save money is a good design and programming guidelines for the application.

Test suites are obviously a good idea but it may end up as a tedious task if the application as poor or no real starting design and if everybody just did things their ways.

I have seen many applications that had grown larger than expected that typically cost 10 times more than it should to maintain, bug fix and implement change requests.

It is sometimes possible to fit existing code in a new clean design and that saves a lot of money if the application is going to be used for a long time.

Let's hope your application has proper design ! ;-)

Re:Design from the beginning is important too. (1)

Chatsubo (807023) | more than 3 years ago | (#34742046)

I agree with the above, when you set out to write code knowing you're going to have to test it, it will influence how you do the design and insulate functionality from the rest of the system, in order to be easily testable.

The problem, however, is you've just made this guys problem even worse. He can't even get allocation for testing. Now he'll need allocation for refactoring AND testing.

My opinion is to break it down into small incremental changes that may take years to filter through the whole system. He can get into the habit RIGHT NOW of writing a small test case for every problem he finds, this will solve his immediate problem, well... immediately. It will focus testing only where it's most badly needed.

Then design wise, the thing to do would be for every problem found, see what can be re-factored and improved right there and then, and do it immediately (with test cases to prevent regression).

The trick then is to keep metrics of how many issues boomerang now, how much time is spent fixing them, and then also keep track of how many "test-cased-and-refactored" issues boomerang. Yes, this will require rigorous timekeeping.

Re:Design from the beginning is important too. (1)

ls671 (1122017) | more than 3 years ago | (#34742096)

> My opinion is to break it down into small incremental changes....

I agree, I just posted a reply that is along the same line:

http://slashdot.org/comments.pl?sid=1932550&cid=34742086 [slashdot.org]

Re:Design from the beginning is important too. (2)

Kjella (173770) | more than 3 years ago | (#34742164)

Then design wise, the thing to do would be for every problem found, see what can be re-factored and improved right there and then, and do it immediately (with test cases to prevent regression).

Well, in my experience it goes like this: You're looking to refactor the code because the code already is quirky. Some 99% of the time, that's just because it's sloppy coding, poor structure and logic or it is simply undefined or irrelevant, the corner cases are never reached, the function is always called with data of the correct format and so on. Then in 1% of the cases, the application depends on this quirky behavior and all in all works correctly - or at least works - until you clean the code. If you have to assume that everything that is there should be there, you end up with code almost as bad as what you started with. This is the problem trying to reverse engineer the intended behavior from existing code.

That is why you have to find some kind of border where you have pretty solid control on all the ways this code is called, it can be a module or a group of classes. Then you preserve the exterior while you build a new and clean interior, where you can safely remove the quirks because you're sure nothing depends on them. The kind of "micro-refactoring" that you seem to suggest he could do along with bug-squashing just doesn't have much effect in my experience. You need to have a certain scale to it before you get any real clean-up effect in the middle.

Re:Design from the beginning is important too. (4, Insightful)

mcvos (645701) | more than 3 years ago | (#34742216)

Well, in my experience it goes like this: You're looking to refactor the code because the code already is quirky. Some 99% of the time, that's just because it's sloppy coding, poor structure and logic or it is simply undefined or irrelevant, the corner cases are never reached, the function is always called with data of the correct format and so on. Then in 1% of the cases, the application depends on this quirky behavior and all in all works correctly - or at least works - until you clean the code. If you have to assume that everything that is there should be there, you end up with code almost as bad as what you started with. This is the problem trying to reverse engineer the intended behavior from existing code.

1%? I think even if the code is buggy, it's quite likely that the code now depends on those bugs in various ways.

In any case, the right way to refactor is to make sure you've got proper tests for the code you're going to refactor. That way, after refactoring, you can check whether the functionality is still unchanged.

So yeah, the submitter has a problem. What he really needs from a code-quality point of few is pretty big and expensive. If there's no money for it, but there is money for continuing as before, then I guess that's the only option here.

First things first (5, Insightful)

Anrego (830717) | more than 3 years ago | (#34741954)

Can anyone offer suggestions for how to convince the owner that setting up a test suite is in his own best interest?

Are you sure that it will be?

but I'm sure that in the long run it would save time and money

How much time and how much money?

What you need here is hard numbers to back up your opinion. Specifically, and I hate to say the word, but you need metrics.

- How much do these bugs cost (these are a mixruee of hard numbers (dev time, etc) and "fuzzy" numbers (customer relations and so forth)?
- How much is setting up this test suite going to cost? How much is it going to cost to use it?
- How effective do you think it will be (look at existing bugs.. would they have been caught)?

That last one is especially important. Testing is good, but it doesn't solve all problems and there are other options. Really take a good look at whether previous bugs would have been found in your proposed system.

Once you have this information, the math then becomes pretty simple.

If your program is modular-ish .. it might be possible to do a trial. This is probably a good idea anyway. There are different ways to do testing, some of which are incompatible with certain development mindsets. Best to find that out early on a small piece vice the entire code base.

Eventually you need a solution (3, Interesting)

nanospook (521118) | more than 3 years ago | (#34742000)

The metrics is a good idea. This will allow the owner to see what it costs him to not have good QA. However, eventually, the conversation will float to how expensive will it be to have QA? Like any security situation, you will want to find the happy medium where the trade off in time/money balances the risk. Given you only have a few hundred users you might wish to experiment by adding QA incrementally. Also, you might find why those bugs are coming back? Poor documentation? Or are developers not maintaining up to date code bases? Process is just as important as QA when it comes to fixing bugs.

Re:Eventually you need a solution (1)

Billly Gates (198444) | more than 3 years ago | (#34742094)

Well his salary is a cost. If he spends 35% of his time fixing previous bugs then y x .35 is the cost that could be saved right there or better put to use developing software instead of fixing it.

Re:Eventually you need a solution (1)

Anrego (830717) | more than 3 years ago | (#34742146)

Implementing a test suite isn't free.. nor is working testing into your process.

If he implements the wrong test methodology (I feel dirty using that work) he may end up spending 40% of his coding time satisfying test requirements, which ultimately only catch 1/4 of the bugs.

I would generally say some testing is always a good idea.. but too much testing can actually be a bad thing. Especially if it is very process and time intensive.

In other words, more analysis than "less bugs good" is required for something like this. Salary and dev time is definitely a factor, but there is a lot of other stuff to consider.

Re:First things first (1)

mcvos (645701) | more than 3 years ago | (#34742224)

but I'm sure that in the long run it would save time and money

How much time and how much money?

And how long is that run exactly? I'm all for proper test frameworks, but if it takes 5 years before it starts paying off, and nobody knows where the app will be in 5 years, then it's really not such a great investment at all. So, like parent said: do the math. Figure out how much the test framework will cost, how much the recurring bugs cost, and then you've got some numbers to convince your boss. Or yourself.

Re:First things first (1)

tibit (1762298) | more than 3 years ago | (#34742504)

Frankly said it's not much of a business if they can't at least wish for where they'll be in 5 years.

Re:First things first (4, Insightful)

Splab (574204) | more than 3 years ago | (#34742304)

Also, putting testing infrastructure into an existing system is a huge undertaking. And it requires a very strict handling for all future commits - if you just slip a little on the test suite you are basically invalidating the whole thing.

Many a times I've tried to implement test suites, both as request from employers and also for own sanity sake - and every time it has failed because the workload was simply too overwhelming to support both testing and development.

In the end, the basic economics of it has dictated that it is cheaper to hot fix than to make it right the first time around.

Re:First things first (2)

arth1 (260657) | more than 3 years ago | (#34742518)

Also, testing introduces bugs. Both because there will be bugs in the test code itself, and because of bias - even when not meaning to, new code will be skewed towards passing the test case, and the tests skewed towards validating the code instead of finding bugs. I've seen examples of bugs that wouldn't have existed if it wasn't for the testing.

Then there's time wasted on testing things that simply don't go wrong, or where the impact would be so huge that it doesn't require a test case - if it prevents the software from running in the first case. Or where the impact of the bug is so small that it doesn't really matter, and rewriting the code is too expensive compared to the theoretical risk of triggering the bug (like a system not allowing someone to be have a name with all digits, or a rounding error only showing up at speeds over 1000 mph).

Yes, testing is overall good for quality, but can increase costs quite substantially.

If the same bugs keep reappearing, perhaps the bug fixer could be better at commenting when fixing bugs -- not only in comment blocks above routines (because they can be missed), but inline comments about what was fixed, and more importantly why?

I don't know... (0)

Anonymous Coward | more than 3 years ago | (#34741970)

But when you find out, let Steve Jobs know.

Charge him (1)

MrQuacker (1938262) | more than 3 years ago | (#34741978)

Charge him by the bug. You'll see how quick he'll make sure hes not paying to fix the same one twice.

Re:Charge him (1)

Billly Gates (198444) | more than 3 years ago | (#34742068)

Then he could save money by firing him instead.

Why will it save money? (3, Insightful)

SuperKendall (25149) | more than 3 years ago | (#34741982)

You need to think very hard about exactly how any why this will save the business money. What do YOU think will be gained, for your software, from a technical standpoint?

Many people just think testing will make things better. But I have seen testing make things better, and I have also seen testing make software far harder to maintain and later to deliver.

What kinds of problems are you seeing that you think testing will solve? Do you see a lot of maintenance causing other errors? Do you want to do this in preparation for overhauling some part of the system?

You see bugs come back every 2-3 months, but is preventing that really worse than the overhead of maintaining and adhering to a test suite that all of the developers would have to do? That alone may not be enough of a reason to impose the overhead that testing can cause.

Since you are fixing bugs you have access to the codebase. Start a test framework yourself, of bugs you cover, and run it against the code day by day. See how often it catches issues, vs. how often you have to alter the tests because the code has changed in ways that are valid. Then you will have more real world metrics about problems solved vs. costs...

Re:Why will it save money? (1)

ls671 (1122017) | more than 3 years ago | (#34742086)

> Start a test framework yourself, of bugs you cover,
> and run it against the code day by day.

I second that. It is amazing sometimes how you have to take the decision and just bill it on normal bug fix time or whatever task is assigned to you. Often, managers won't care about it as long as the average time to fix a bug or the time spent on your regular tasks remain the same.

It is harder to do it this way because you then have do it gradually. We even have redesigned application cores without letting the end customer know ! If we had told him, he would have been disturbed and worrying about it. For him, all we did was change requests and bugfixes. In this case, in was more complex this way because the old design had to be compatible with the new design for quite a while.

In the end, we managed to implement changes more easily, we could afford to charge the customer less while making better margins. ;-)

start small (5, Insightful)

sugarmotor (621907) | more than 3 years ago | (#34741986)

Just start small, with "failing tests" for each new bug. Bug fixed, test passes. Keep expanding the test coverage.

Re:start small (0)

ratbag (65209) | more than 3 years ago | (#34742080)

Just start small, with "failing tests" for each new bug. Bug fixed, test passes. Keep expanding the test coverage.

This needs modding up since it is the "right" answer to the question - I've got no points at the moment.

Re:start small (0)

Anonymous Coward | more than 3 years ago | (#34742188)

Adding unit tests to code that was not designed for testing often requires refactoring -- a considerable effort and a source of bugs itself.

Re:start small (0)

Anonymous Coward | more than 3 years ago | (#34742516)

Whereas integration tests don't have that problem. Don't limit yourself to one type of test.

Re:start small (5, Insightful)

VirexEye (572399) | more than 3 years ago | (#34742098)

+1 to moding parent up.

You won't get far convincing a product owner that you should spend months writing tests for the entire system.

Convincing someone that you should write unit tests for all new functionality to help guarantee the bug fix/new feature will continue to always work into the future is a much easier sell.

Re:start small (0)

Anonymous Coward | more than 3 years ago | (#34742158)

If the owner does not get convinced even on a incremental start small plan, do it anyway under his nose. It's for your own good.

Re:start small (1)

Usagi_yo (648836) | more than 3 years ago | (#34742268)

Best Answer by far for small non mission critical apps. If you can't justify the cost of testing vs the cost of a bug, then you'll never get funding for a distinct project for testing. When you do fix a bug, you invariably code review and write a test app against it. When it comes to internal apps, testing is typically overlooked and bugs are fixed ad hoc or work around become tribal knowledge. But even the tiniest mission critical application needs to have a test suite developed side by side with the app in question. Think of the team that let slip the bug that crashed the mars probe. Not only will that slow your career down, it will scar you knowing that your bug cost billions of dollars and years of work hours.

Re:start small (0)

Anonymous Coward | more than 3 years ago | (#34742396)

+1 ... if the unit under test is reasonably testable, such that you can achive some degree of isolation from other interfaces/components/systems

Otherwise, exploratory testing might be more reasonable at this stage in the game. Legacy, mutated mish mash might have more side affects "by design" than a framework based scripted testing would reasonably allow or handle.

Re:start small (0)

Anonymous Coward | more than 3 years ago | (#34742552)

Just start small, with "failing tests" for each new bug. Bug fixed, test passes. Keep expanding the test coverage.

Exactly! One has to duplicate bug before fixing it. Do that with an automated test. Use the test to verify that you fixed the bug. In that way you have slightly increased test coverage and charged it to fixing the bug. After a while, you'll start getting less bugs.

WHY do you have to prove software testing saves mo (1)

dbIII (701233) | more than 3 years ago | (#34741990)

WHY do you have to prove software testing saves money? Even a cheap and nasty electrical appliance from China is tested despite there being a lot of motivation to cut corners and not do so. Why should software be any different?

Re:WHY do you have to prove software testing saves (4, Insightful)

Rosco P. Coltrane (209368) | more than 3 years ago | (#34742066)

WHY do you have to prove software testing saves money? Even a cheap and nasty electrical appliance from China is tested despite there being a lot of motivation to cut corners and not do so. Why should software be any different?

Rest assured that Chinese companies test their products precisely *because* it saves money. If it didn't, they wouldn't; they don't have a commitment to good quality for its own sake.

An untested product leads to high rates of return (lost money), customer dissatisfaction and brand name damage (HUGE loss of money in the medium to long term). Nobody puts up with bad products anymore. Software is one of the last kind of products where it's still somewhat accepted but people are quickly becoming intolerant to bugs nowadays.

Still, what is true of physical products (that extensive testing on top of proper design and good manufacturing practices) may not be true of software. I.e. the question is: is laborious and careful design and implementation with minimal testing more or less expensive than quicker, laxer design and coding and a strong test/correction feedback process? I'm not sure the answer is clear-cut. As a (former) programmer, I can see argument for both approaches.

Re:WHY do you have to prove software testing saves (3, Insightful)

Kjella (173770) | more than 3 years ago | (#34742236)

Still, what is true of physical products (that extensive testing on top of proper design and good manufacturing practices) may not be true of software. I.e. the question is: is laborious and careful design and implementation with minimal testing more or less expensive than quicker, laxer design and coding and a strong test/correction feedback process? I'm not sure the answer is clear-cut. As a (former) programmer, I can see argument for both approaches.

Didn't you just answer your own question: extensive testing on top of proper design and good manufacturing practices

For the first I'll just quote Knuth: "I have only proved it correct, not tried it." And if you send piles of shit to testing/QA and expect them to be the impenetrable barrier between you and the customer, you're equally deluded.

I've found that for anything that should last a while and be stable, there should be a test case. It's so easy to subtly break something. However, I've found writing a proper test suite that deals with databases, network communication+++ and not just the application itself is pretty hard.

Re:WHY do you have to prove software testing saves (1)

outsider007 (115534) | more than 3 years ago | (#34742430)

So you think American companies have a commitment to good quality for its own sake. Just out of curiosity, what do you drive?

Re:WHY do you have to prove software testing saves (2)

tibit (1762298) | more than 3 years ago | (#34742546)

Nobody puts up with bad products anymore.

-- Ha, ha, ha. The U.S. market obviously demands plenty bad, nearly worthless crap, if one were to judge from what's on the market.

There are simple things like cast-iron hand-cranked meat grinders where the market is saturated with extremely poorly made chinese crap, with absolutely no alternatives. I know, I've tried to buy a new one -- ended up getting a used Porkert instead.

Another simple thing: portable DVD players. I've already had to fix two of them because they had liquids spilled on them. It's not a rare occurence -- they are often used by kids who either don't care or just forget themselves and get unlucky. While the manufacturing quality is perfectly acceptable, the design quality just sucks big time. The clearances between many parts seem randomly chosen, so that either things that rub on each other seize, or they rattle. An no, this isn't a molding issue, it's by design. Molding on the units I looked at was by the book. You can tell that many small cross-section plastic parts, prone to breakage, were designed by people who have no clue or no experience in mechanical design. There are plenty of changes that one could make that would not increase the material costs, but would make those devices much more rugged.

It seems that many U.S. companies that brand products imported from China have no engineering on the other side of the pond, just marketing and purchasing people.

Re:WHY do you have to prove software testing saves (1)

Opportunist (166417) | more than 3 years ago | (#34742256)

Because it is. Do you know the term "bananaware"? It's used internally by a huge German company. It means "letting the goods mature at the customer's (just as you do with bananas, picking them while green and letting them mature during transport and at the customer's shop)".

At the customer's shop, and at his expense, of course.

The core reason is that people got so used to crappy software that they don't even expect it to work "out of the box" anymore. It's also trivially easy today to replace broken software. No need to send it to the shop for replacement, we'll send you a patch via internet. You do have internet, right? You don't? How do you exist today?

Testing or no testing is only a matter of money. A product only has to be good enough to keep enough customers from throwing it back at the maker, or rather, as long as the cost from RMA process and customer complaints are lower than the cost for testing, no testing will occur.

Re:WHY do you have to prove software testing saves (1)

jellomizer (103300) | more than 3 years ago | (#34742526)

Because some people and companies like to have a quality product.

Maybe it won't save money (3, Insightful)

initialE (758110) | more than 3 years ago | (#34741992)

Perhaps the app is just more important to you than it is to the company? First take your ego out of the equation, i guess, then make your case before the boss. Just sayin'. Also if you have a time sheet of how much time you've spent on this app so far it would be helpful in drumming up some numbers, plus a few scenarios where the failure of the app causes lost time for the company as a whole.

Cost-Benefit Time! (5, Interesting)

wild_berry (448019) | more than 3 years ago | (#34741994)

We've had to justify creating auto test suites where I work.

Over the last decade our product has grown from one code-base into three strands, each with separate customer foci, and we've had a healthy amount of staff turnover so that there are still brilliant, creative and skilled people working on it but some of the original knowledge has left us.

We found* numbers to justify that automated testing of existing features, applied later will protect against regressive changes. Even where there are complicated features which were not modular in design, or which lack good interfaces, the tests have saved us massive amounts of time testing by hand. The real win is hidden under something we didn't realise until later: creating the tests have forced us to really document what the features are and how they work**, sometimes from a unit-test level, sometimes at the interface level and sometimes in a top-to-bottom vertical slice. Once you have a record of what your software does, in a computer which is skilled at remembering exactly and repeating exactly what some former staff member told it a couple of years ago, you have a decent reason to be confident that your bug fixes won't cause more harm than good.

*: ballpark figures / educated guesses / made up.
**: We favour working code over comprehensive documentation, until our agile team is reassigned to other projects or leaves the company.

Tough luck (1)

hardtofindanick (1105361) | more than 3 years ago | (#34741996)

In small companies most of the time it is not really worth the money to invest time and money in test suites and engineers. It does not make sense to operate in such a manner and is very frustrating but the goal of the owner is just to make money. You can start charging him by the bug, but then the way you are describing this is a trivial software so he will find someone else who will work the old way.

All it takes is one CVS idiot... (0)

Anonymous Coward | more than 3 years ago | (#34741998)

With 25 developers, all it takes is one who doesn't merge correctly to cause old bugs to resurrect themselves.

There's your problem... (1)

SanityInAnarchy (655584) | more than 3 years ago | (#34742018)

...using CVS.

Re:There's your problem... (1)

Anonymous Coward | more than 3 years ago | (#34742050)

Yeah I know. Real developers copy .zip snapshots over using Explorer!

Re:There's your problem... (2)

SanityInAnarchy (655584) | more than 3 years ago | (#34742120)

No, real developers use real version control software. Even SVN would be an upgrade, let alone Git, Mercurial, bzr, etc.

Re:There's your problem... (0)

Anonymous Coward | more than 3 years ago | (#34742192)

GP here. I was using Darcs long before Git and Bazaar even existed. But thank you for stating what we all knew in 2002 -- about SVN's superiority to CVS.

Re:There's your problem... (2)

Vegard (11855) | more than 3 years ago | (#34742474)

I have seen the same happen with SVN. Trust me, people who don't understand version control systems will *always* find a way to fuck up! :)

Re:All it takes is one CVS idiot... (1)

tibit (1762298) | more than 3 years ago | (#34742558)

With 25 developers, you should have customary code reviews before things get merged into production.

Well... (0)

Anonymous Coward | more than 3 years ago | (#34742004)

How can you be so sure that it'll save money? If you need help to prove it, then it's likely that you don't know enough to make a significant change.

it won't save him money, thats your problem (0)

Anonymous Coward | more than 3 years ago | (#34742006)

fixing bugs won't save him money unless your willing to recommend firing a few developers because it'll reduce workload. thank your lucky stars he's smart/kind enough to know this and say no rather then let you go to all this trouble then fire you because your not needed anymore.

Re:it won't save him money, thats your problem (1)

SanityInAnarchy (655584) | more than 3 years ago | (#34742114)

I don't know about you, but I've never worked anywhere where the problem was developers with too little to do. If they aren't spending all their time fixing bugs, maybe they can devote a bit more time to actual enhancements?

Re:it won't save him money, thats your problem (0)

Anonymous Coward | more than 3 years ago | (#34742160)

but how is THAT going to save money? the only way eliminating bugs saves money is to fire developers. i'm betting these guys are salaried like the rest of us so their time is already paid for. if on the other hand this guy has some other grand plan that involves turning that time currently spent fixing bugs into delivering a new feature that sells more product, i'm all ears.

he first needs a solid delieverable he can hold out like a carrot. slashdotters really do need to have their hands held on business thinking don't that.....

You are almost to the point of needing QA (1)

linzeal (197905) | more than 3 years ago | (#34742010)

QA can be contracted out for 20 dollars an hour in house, I would suggest you look into that. It is a full time job for anything but the smallest of code bases.

Re:You are almost to the point of needing QA (1)

Jane Q. Public (1010737) | more than 3 years ago | (#34742194)

QA is not the same as testing, in this context. The kind of tests he is referring to tell you whether the code is broken before it ever gets to the QA stage.

Baby steps (1)

sjames (1099) | more than 3 years ago | (#34742014)

Start off easy, you must have test cases for the bugs you fix (so you'll know they're fixed and some sort of script or procedure you used for that testing. Just hack together a quick script on top to run them in succession as part of your regular duties fixing bugs.

Now, put them somewhere where other developers can access them. Mention it around the coffee pot.

That's not exactly comprehensive testing, but it's a start and should only take a few minutes. You could argue that you had to do it anyway as part of your regular duties, so it cost nothing. With any luck it'll catch on and prove itself useful.

the 200 Million answer (0)

Anonymous Coward | more than 3 years ago | (#34742016)

It is not just that testing is needed, when bugs are found, they need to be fixed (security bugs with priority).
I have personally seen bugs that were not deemed 'need to fix' that went out with the following fallout (this is NOT a joke or embellishment):
1. possible loss of life - as in death of a real person
2. loss of revenue that was supposedly in the millions
3. death of company
4. loss of accounting revenue in the multiple thousands

I have been in the industry long enough to see these things. Per NDA, I cannot give details. But this is real and depending on the company and its politics, could happen in front of your eyes.

- SPT

A test suite doesn't have to be expensive. (1)

julesh (229690) | more than 3 years ago | (#34742022)

If all you do to make the test suite is add one test each time a new bug is discovered, you'll probably find it doesn't take much longer than fixing the bug and retesting manually anyway, and over time you'll build up a reasonable suite. Yes, the benefits won't be instant, but you can do it without much outlay of time (or money) and therefore it's easier to justify to the product owner.

In case you haven't, you should read Robert C. Martin's book Working Effectively With Legacy Code, which is full of advice on just this kind of situation.

It saves money if the data is worth something (1)

finwind (975756) | more than 3 years ago | (#34742024)

If the data is worth anything, you can always argument that software that has not been tested thoroughly can mess up the data and then the worth/value of the whole system collapses. Yes, there are backups, but will the backup schmea work if the software gradually corrupts the data? And will the customers suffer damage during the downtime that you try to debug/correct the bug (that will eventually occur) ?

Just do it! (0)

Anonymous Coward | more than 3 years ago | (#34742036)

Why are you asking for permission? Most programmers today know that there is no point in asking for permission when it is much easier to ask for forgiveness after the fact ... if they ever find out and care.

Just start in one corner, create test cases for exactly those bugs that you don't want to see again. Then add incrementally whenever you have a couple of minutes to spare.

Just do it!

Kids these days (1)

oldhack (1037484) | more than 3 years ago | (#34742070)

I don't have a solution to your problem, but I can tell you the shit only gets worse.

You're welcome. :)

Wrong Question tag appropriate (1)

wrook (134116) | more than 3 years ago | (#34742074)

The better question to ask yourself is, "Why am I still working here?"

If I were working on that team, I would walk up to the owner and tell him straight out, "You hired me for my ability and experience in software development. It's my job to give you the most customer satisfying functionality for the money. You've seen my credentials. You've seen me work. Stop second guessing me and let me get on with my job." But my employers have historically hired me because of my experience in these kinds of areas.

Asking, "How do I prove to management that X practice is a good idea?" is precisely the wrong thing to be doing. Do you *know* the practice is good? Do you have experience implementing the practice? Do you know when it is appropriate and when it isn't? Do you know how to implement the practice in the particular circumstance you are in? If not, who on your team does? If nobody, who will pay for your learning experience in the probable case that it takes time to get things right?

I'm not telling you that you shouldn't be doing these things. But if the guy paying you is unsure of your ability to pull it off and you have no prior experience, I highly recommend relocating to a job where you can get the experience.

Re:Wrong Question tag appropriate (2)

Rosco P. Coltrane (209368) | more than 3 years ago | (#34742106)

Yeah, walk up to your boss and tell him you know more than he does. Great career move...

Have you heard of diplomacy? You may know more than he does, but he doesn't want to hear it. Instead, you should convince him that what *you* want to do is *his* idea.

Re:Wrong Question tag appropriate (0)

Anonymous Coward | more than 3 years ago | (#34742246)

Instead, you should convince him that what *you* want to do is *his* idea.

Achievement earned: verbal cocksucker.

Re:Wrong Question tag appropriate (1)

mcvos (645701) | more than 3 years ago | (#34742292)

If you need to use that kind of deception to do your job, maybe you're better off looking for a better job.

Re:Wrong Question tag appropriate (0)

Anonymous Coward | more than 3 years ago | (#34742508)

The question is what the distribution of reasonable managers vs. ego-maniac managers is. Posting anonymously expresses my opinion in the matter (and also my opinion about my current managers, who have just fired someone because they didn't like what he had to say).

If the probability of getting into the same kind of environment is 80% or above, what's the point in leaving your job?

I have the same opinion about unpaid overtime (everybody expects you to do it), no testing (most companies don't invest in quality because the managers are ex-sales that are used to sell crap and not take responsibility), over-committing, ridiculous dead-lines, etc. etc. This is the face of the industry, there's not point in looking for another job in the same industry, because it's exactly the same or worse in other companies.

The only way to deal with it is to put a mental barrier between yourself and your company - the job allows you to have a life, but it does not have to reflect your own values and opinions.

Re:Wrong Question tag appropriate (1)

Anrego (830717) | more than 3 years ago | (#34742132)

That's a little.. extreme. That kind of trust isn't assumed with credentials (at least not in my book) but builds over time.

Not only that but the boss may (and likely does) have a broader knowledge of the company and it's objectives, and any time you're talking about spending more money I would expect to have to at least show the boss a few power point slides or something. At the very least, having to convince management that an idea is sound is a good exercise, and may cause you to look at things you would have conveniently ignored.

It's all about cost (1)

morbingoodkid (562128) | more than 3 years ago | (#34742082)

The problem is you need a software (bug control system) already setup to prove that it is important to have a testing framework.

But the calculation I would use goes something like this.

Needed
1. Number of bugs fixed.
2. Number of bugs returned.
3. Time added to original bug to fix additional requirements
4. Time to test properly.

Cost factor for bugs not tested properly (time to test)/(time added to original bugfix)

This would be your cost of not having a testing framework. Work this out over the period you would like to aggregate the testing framework cost.

Say there was 30 bugs that returned
If took you about 60 hours to fix those 30 bugs.
It would have taken you 20 hours to test the bug fixes properly.
You will get
(20/60)=0.3 (30 %) reduction in cost.

To calculate if it is worth the companies money to implement the testing framework.
0.3*(cost to fix returned bugs over 2 year period) if this value is less than the cost of the framework it is not necessary to setup the testing framework.

Honestly even on the simplest system it always works out cheaper to have a proper testing framework. But I understand your frustration to bring this concept under the eyes of the managers are quite difficult.

The easiest way is to try and calculate a factor of cost savings and bring it to their attention.

I can't believe the incompetence (1)

Billly Gates (198444) | more than 3 years ago | (#34742092)

No QA?? Sadly, I see many slashdotters in similar situations. How unprofessional and arrogant have the bean counters become? I have been out of I.T. full time for many years but times were different earlier last decade. Any manager who refused to check the quality of the work used to be fired. Are you really managing if you wont even do basic QA?

Even fast food restaurant managers do QA and have district managers nail them on satisfaction and statistics of food ingredient use and statistical sampling.

Time to update your resume. From what you described it sounds like a cheap work environment if they have college interns writing critical software. Quality is free and any good manager knows this which saves money. Six sigma started out in software design before being a big hit on the factory floor.

Re:I can't believe the incompetence (1)

Opportunist (166417) | more than 3 years ago | (#34742336)

College interns writing mission critical software isn't as far fetched as it seems. Especially if it wasn't meant to be but just "developed", as software often does. Allow me to give you a piece of anecdote "evidence".

The year is 1993 and little Opportunist is working as an intern during the college holidays. Instead of stuffing envelopes as usual, the company found out that this guy can do "stuff with computers" and we have those boxes that record where our traveling salesmen go, maybe he can get the data out of them and into a more meaningful format than the software we got. Opportunist reverse engineers that format, writes a cute little piece of code (IIRC it was even originally in Access Basic) and finds out after two days that he's done. Back to stuffing envelopes? Not if I can fight it!

To make a long story short, I kept suggesting features and adding them to it to stay out of the assembly line hell for as long as I could and eventually we ended up with a tool that could do the travel plan for a salesman. Yes, they didn't have that. Their people had to do it by hand and it took them always almost the whole Friday to get a travel plan down for next week. The system was able to learn and adapt and it was pretty good (for my knowledge then, today I would toss such a piece of junk before letting the guy writing it finish it).

Now human nature invariably struck. My boss boasted how he can now send his salesmen around a day longer since they're not tied down with planning, the bigwigs came flying in and a month later I received a call that I have to make a few changes that $headoffice wants, because now the whole company should be able to use it.

This software (or rather the successor I wrote in C++ Builder a little later) is still being used by a rather big company to send around its traveling salespeople. And we're talking about a make-or-break piece of software, if these people do not travel and visit the right people at the right time, the company will lose serious amounts of money. All that hanging on an application a college intern wrote almost two decades ago. Today I would write a LOT of it differently, knowing a lot more about algorithms and efficiency (Fortunately they didn't care that the program took almost 30 minutes to complete on the ancient 486 machines they used, they were used to IBM and overnight calculations), and I am very, very sure it has some very, very nasty bugs that rear their ugly heads at the worst possible times because back then I had not the slightest idea about sensible testing.

But, and that's the point, it still beats the alternative: Not having anything at all, doing it by hand and hence relying on human ability. Even given the bugs, I am sure it makes fewer errors than human does. And when I compare my hour rates then and now, I can see why they wouldn't hire the back-then equivalent of me (i.e. someone who "really learned" how to do it). The intern does it well enough and he costs about a tenth.

Whether maintenance is manageable or tenfold (because, well, easy maintenance was not really a focus for me when I wrote the code, I never had to maintain code before so I didn't know just how nasty it could be and how important it is), is a different matter.

But it was cheap! And that's what counts! Ask your manager if you don't believe me!

It's ready now! (1)

hcs_$reboot (1536101) | more than 3 years ago | (#34742100)

how to convince the owner?

Replace the owner's default homepage with this Slashdot article.

Hmm ... (1)

lennier1 (264730) | more than 3 years ago | (#34742102)

Send the suits one of their cherished Excel sheets?

    ((downtime duration) * (customer's lost income per hour) * (customers))
+ ((bugfix duration) + (rollout duration)) * (hourly developer salary)
+ (overhead costs to appease angry customers)

Paying technical debt. (3, Insightful)

SanityInAnarchy (655584) | more than 3 years ago | (#34742112)

As others are saying, if you can get hard numbers, get them. Nothing like actual evidence to back up your claim.

However, there's a more important reason for doing this which is harder to find solid evidence for -- the concept of "paying down your technical debt." Often, we hack software together relatively quickly, without fully testing, etc -- every time we do that, we're effectively borrowing against the future stability and maintainability of the product, or realistically, against our own debugging and refactoring time in the future.

Like all debt, it has interest -- the longer you let it fester, the worse of a mess it's going to be. So like all debt, it makes sense to pay it down when you can -- a little refactoring now can save a lot of headaches later.

So, it may be difficult to get hard numbers -- though again, those are better. But if you can't do that, sell it as an investment. Invest a little time right now at least putting the automated tests in place, and maybe start with just some regression testing. Twenty minutes writing a regression test will almost certainly save you an hour the first time that bug appears -- and those are pessimistic numbers (though also made up on the spot -- find your own actual facts!)

Test your own code? (1)

Old Wolf (56093) | more than 3 years ago | (#34742116)

I guess I'm missing something, but don't you test your bug fix when you write it?

You said that your bug fixes "come back", it sounds like you are cutting some code and sending it off without testing that you actually fixed the problem?

Answer: Write a test and see! (1)

VortexCortex (1117377) | more than 3 years ago | (#34742128)

#!/usr/bin/perl -w
#####################
# File: money-test.pl
# Desc: Tests if Money is Saved.
#####################

use strict;
use FileHandle;

# This Should Save Money!
sub saveMoney {
    my $tout = new FileHandle( "> test-output.txt" ) or return 0;
    $tout->print( "Money\n" );
    $tout->close();
    $tout->open( "< test-output.txt" ) or return 0;
    die "Lost Money!" if ( <$tout> !~ /^Money/ );
    return 1;
}

saveMoney() or die "Money not saved: $!\n";
print "Money Saved.\n";

Once every two-three months? (0)

Anonymous Coward | more than 3 years ago | (#34742130)

How do you prove (automated) software testing saves money? Many factors are involved. A test suite and unit tests can cost a lot of time to set up. You may very well find your system wasn't designed to allow automated testing and would require refactoring to allow for automated testing. How much time would you spend on setting up a test suite? How much time would it save? (Keep in mind right now you may not be doing manual tests- there is added value to introducing such tests as well). One factor is how long does it takes to re-fix a bug if one comes back every two-three months- some bugs take a bit more, some a bit less, but I find handling two (unknown) bugs a day is typical. Another, more important factor to take in account is how much damage can potentially be done by bugs. Depending on your system this may range anywhere from lost entertainment value in a video game [kotaku.com] to a plane crashes in air control systems. That is to say, in some systems, automated testing simply matters more than in others.

Just do it... (1)

rastoboy29 (807168) | more than 3 years ago | (#34742144)

Just do it and show him the tests running.  It always impresses people to see a bunch of automated tests happening--it makes it more intuitive what the value is.

Anybody can understand how that can save valuable human tester time (particularly if that human is expensive and talented).

Re:Just do it... (1)

benjamindees (441808) | more than 3 years ago | (#34742238)

Anybody can understand how that can save valuable human tester time (particularly if that human is expensive and talented).

Well, wherever you live that might be true. But in the US, home of gangster-capitalist-accounting-fraud-as-a-defense-against-the-social-democratic-kleptocracy, showing your boss something you've automated would just make him think "Why did you waste time doing this when I could have outsourced it to a shell-company in which I have a 49% stake and hired some Indians to write it and taken that new tax write-off for creating jobs in a gay woman minority owned software outsourcing firm. Now I'm going to have to track it as a capital expense, ugh."

Re:Just do it... (1)

gabebear (251933) | more than 3 years ago | (#34742440)

You really sound like you need to quit your job. It's not like that everywhere.

Re:Just do it... (1)

Lennie (16154) | more than 3 years ago | (#34742464)

It isn't very impressive if it hasn't found any bugs. Then it's just plain boring.

You see... (1)

benjamindees (441808) | more than 3 years ago | (#34742162)

There's this thing, called 'specialization'. You work for a small business somewhere in the Western world, so you probably have never heard of it.

Basically, the way it works is this. One person does one job. Another person does another. Everyone sort of agrees ahead of time what the jobs will be, and how they can interact in the most efficient and least obtrusive way. That way, you don't end up with five people on a 'team' spending all their time either in meetings or overwriting each other's work.

For instance, one person might fix bugs. Another person might test the software. Each person can 'specialize' in one relatively small area of 'special' knowledge and skills and not have to worry about everything his co-workers are doing. The person fixing bugs can know which bugs have been fixed, and spend time thinking about how the software should evolve in the future in order to fix or prevent other bugs. The person testing the software can spend his time creating automated tests, finding new bugs to be fixed, and double-checking the work of the people fixing the bugs.

Watch out, though! Because there's this other thing, of which you are probably well aware, called the 'pointy-haired boss'. Pointy-haired bosses don't like 'specialization' because they are fairly dim-witted and actually enjoy going to meetings and re-doing the same work over and over again. So, if you do decide to 'specialize' in your work, you should probably have a few websites lined up to browse and look like you're staying busy during all that extra free time you will soon have. Good luck!

IT (Project) Management 101 (0)

Anonymous Coward | more than 3 years ago | (#34742182)

Calculate the statistics regarding the costs that fixing the bugs already takes in your outfit (such as hours taken, damage recovery costs caused by impacts of bugs etc.) and then calculate the capital costs for setting up the processes, procedures, assets etc. for properly testing software and estimate how many bugs it could have prevented by examining old data and working out what technique, process or procedure you're planning to implement would have caught said bug(s).

Deming... (1)

Aiwendil (556873) | more than 3 years ago | (#34742228)

Get him to read anything written of/about Deming about what is wrong with current management I'd say.

The book "Dr. Deming: The American Who Taught the Japanese about Quality" by Rafael Aguayo is probably the easiest to grasp for most people (and is a book I would recommend to pick up and read in general anyways).

Speak their language (1)

BESTouff (531293) | more than 3 years ago | (#34742260)

However you prove the need for a test suite, express it on a shiny powerpoint. That way, even if the people to whom you'll show it are techs, they'll be able to pass it to their PHB and have him understand just what's needed: allocated time slots for tests.

Don't do that ! (1)

eulernet (1132389) | more than 3 years ago | (#34742262)

The owner doesn't want to invest time or money in getting one set up, but I'm sure that in the long run it would save time and money.

No, it will NEVER save money nor time. It will just improve quality.
Testing requires investment, and there is no ROI (Return On Investment) on it, except that the quality of the software improves, since you are able to avoid regression bugs.

In my company, we spent a lot of time (several man years) writing tests to strengthen our software, and I can assure you that this was very expensive, and I still believe that it was a waste of time and money (disclaimer: I worked onto writing the tests), since this could be done in a cheaper way.
In our case, we have a 7 years old project, and it was very difficult to write tests, since the code was not designed to be tested, and thus not easily testable.

Just putting the tests in place will probably require a large amount of time and effort.

If there is no development on your software, but only bugfixes, then it's better to use the cheaper approach of writing tests that fail, as somebody suggested above.
If the cost to write a single test is too expensive (in effort/time/money), you'll have to live with the current situation.

If there is still some development on the software, it's good practice to start adding tests on the NEW parts of the code.
Forget about the legacy code, it will require too much effort.

Metrics... (1)

theVarangian (1948970) | more than 3 years ago | (#34742280)

Can anyone offer suggestions for how to convince the owner that setting up a test suite is in his own best interest?

I actually had to write a paper on this subject for an SQA class I took. The result among other things was (as if we didn't know that already) that PHBs respond to numbers and little else (except maybe Blackberrys) and a scary percentage of them think software testing isn't worth doing beyond weeding out the worst of the bugs because it offers a low return of investment. To convince your PHB you have to gather metrics. How much does an error cost if that error makes it undetected into a marketed product? How much would do save if we caught it early by testing? How much of a reduction in the frequency of, say... buffer overflow or SQL injection bugs, etc... have you achieved by implementing software testing in some other project? If you don't have any SQA enabled projects to fall back on you can try to cite case studies or CS papers. Case studies are probably better since they are less theoretical and not loaded with a huge amount of statistics. Keep in mind that the level of SQA that is justifiable for projects varies according to their nature. If this is an internal product never used by external parties you can get away with much less SQA than if it is used by external customers. Metrics gathering is also a key component in pretty much everything else SQA related, root cause analysis, test coverage analysis, determining code coverage, deciding on where to focus tests... the list goes on and on.

TIMEZONE SLASHDOT (0)

Anonymous Coward | more than 3 years ago | (#34742284)

I happen to read Slashdot at night when the sun rise in the US.
Now They are making a good move, somewhat like GOOGLE's targeted ads. Stories as per geographies.
Thanks slashdot.

Develop a mathematical model of costs (3, Informative)

Paul Johnson (33553) | more than 3 years ago | (#34742288)

Basically you have to develop a mathematical model of the costs of the current situation, and compare it with a mathematical model of the costs of using tests. As part of this you will have to produce a plan for introducing tests, with the costs for each step itemised. Use the best numbers available, but don't worry if some of those numbers are "best guess". Just don't try to hide the fact. Put both models in a spreadsheet and come up with a number for how long it will take to recoup the initial investment (break even). Don't forget to discount future cash flows. In MBA-speak this is known as a "business case".

One bug 2-3 months (1)

ArIck (203) | more than 3 years ago | (#34742310)

One bug every 2-3 months. How long does it take to resolve that bug. An estimate of 4-6 bugs a year is not that severe of an issue and like the owner is better off without the extensive bug testing.

Re:One bug 2-3 months (1)

Shooter28 (1564631) | more than 3 years ago | (#34742528)

One bug every 2-3 months. How long does it take to resolve that bug. An estimate of 4-6 bugs a year is not that severe of an issue and like the owner is better off without the extensive bug testing.

How many months are in your year?

A good approach? (1)

gedeco (696368) | more than 3 years ago | (#34742312)

I think there might be something wrong with the problem analysis. You should consider why a bug does reappear. Wrong version in Production? Or as you stated not enough tested.
1) First of all, You have to make a inventory of all existing bugs in the application.
2) you need to describe the possible Workarounds. Here you can already start calculating the estimated cost of each incident provoked by a bug.
3) prioritize the bugs. The more urgent, the more they will cost the bussiness by not functionning and resolving time
4) Focus on some Quick wins. bugs who doesn't consume much time and money to solve.
5) Keep in mind that having a good Workaround is sometimes better
6) Make a bussiness plan. Explain the benefits versus the money lost (consider lost worktime as a los off money)would be cheaper to redesign the app.

Managers are willing to pay for improvement, but they have to be convinced. The only language they understand: MONEY.
Briefly you talk about the money you loose doing nothing, versus the Return on Investment (ROI) doing something.

bug regressions (1)

rhendershot (46429) | more than 3 years ago | (#34742320)

at least once every 2-3 months I see some bug I fixed come back, and I can only assume it's because we don't have a formal test suite.

Many have already mentioned that you need metrics if you want to prove that a test suite would be cost effective. You don't really say if that's to be automated and that's harder. You also don't say if the intent is module unit testing or integration testing or some other slice, nor do you say what technology the system is built around. 10 years probably would be VB6 and there was a unit testing framework a few years ago available but the tool-set is decidedly sparse AFAIK.

Anyway, the statement about bugs returning caught my eye. Perhaps they really are a regressive bug but take a close look at that idea. To illustrate suppose the complaint is that "a failure was not logged to the logging file". This could be *caused* by, among other things, a misconfiguration of the logging system, a missing logging statement within the code that handles the failure, a locked logging file which could not be written, use of a module that disabled logging, or which did not re-enable logging, and so on.

Since these are all discreet sources they really are different bugs. It's unlikely (disadvantageous actually) that one test would find re-emergence of all of these even though they result in the same basic problem.

The point I'm hoping to make is to be certain you really are identifying the same bug as haunting you not just similar bugs and be sure you can faithfully relate that to your stakeholders.

And if you can make that case then it should be a no-brainer that they'd *not* wish for their teams to spend time and their resources re-doing the same thing. At that point the question becomes "What's the best way to insure that we don't?"

Sometimes it comes down to better documentation of the modules being used or training of the people involved. It's very hard to quantify the cost of training folks to create useful tests that don't actually increase costs (a lot). Also hard is predicting the point of break-even where all the costs in training, tools, and coding (and testing) the tests have started to return enough that the net starts to be positive.

We often seek perfection but removal of all regressive bugs is probably not the best first step. Another clue I got was that these bugs are found late in the process. In any case, perhaps better defensive coding in the modules would make them more apparent or obvious, and found sooner or prevented from propagating in the development process or the system itself.

Can anyone offer suggestions for how to convince the owner that setting up a test suite is in his own best interest?

The question really is "Are they?".

Few things I'm aware of (1)

ThePhilips (752041) | more than 3 years ago | (#34742332)

How Do You Prove Software Testing Saves Money?

Generally, in short term testing doesn't save money. Though in long term, there are some benefits.

0. Software testing doesn't save money to the software developer itself - it generally saves money to the customer. Having software being tested, allows customers to cut the time and the effort in the critical phases of the larger project deployments. Some companies I know sell testing separately. You can buy their product and test the particular configuration yourself - or the ISV can do it for you. (Feature bloat is the main reason why the bogosity exists.)

1. Customer relationships. There is nothing worse than a minor design defect requiring coming up before (and causes delays of) deployment.

2. Long term software development generally builds on previous enhancements. If the enhancements were not tested properly beforehand, later development might come to the unpleasant surprise, when it would come to the light that the foundation for the new work wasn't as stable as people did tend to believe (or the foundation wasn't really satisfying the requirements it was presumed to). And it is a well known fact that software developers are poor at spotting problems in their own code. Testers having a different angle on the software complement well the development process.

Also, I would generally recommend to retrain software developers to be testers. Though not obvious to many, any effective testing requires knowledge of the source code. I had seen lots of wasted effort due to the fact that testers were not aware of the implementation peculiarities.

For smaller companies I would advise a simpler scheme: rotate primary focus of developers between coding and testing (and support). Idea is that developer should have enough time to learn particularities of each of the development phases and bring the new experience back into the other development phases.

and I can only assume it's because we don't have a formal test suite.

For internal applications, make sure that you have requirement specification. Testing is impossible if the expectations are not spelled out formally.

If you do not have a requirement specification, then try to create one before advocating for testing. It might turn out that the software has too many interested parties and finalizing requirements would prove impossible. Then testing might cause more harm than good, by breaking the unwritten unspoken balance. By becoming a tester you should be ready to make an enemy out of better half of the company.

Answers: (1)

Alex Belits (437) | more than 3 years ago | (#34742340)

once every 2-3 months I see some bug I fixed come back, and I can only assume it's because we don't have a formal test suite

No, it's because you didn't fix it. "Bug" is usually described as a set of symptoms. A very simple test can determine if the changes you made altered the behavior of a program that was recognized as a bug, however it's your responsibility to have understanding of a product that allows you to recognize and fix problems. Tests merely help to determine if some known aspect of product's behavior is unaltered (if tests could find bugs, products would never be released with them).

The owner doesn't want to invest time or money in getting one set up, but I'm sure that in the long run it would save time and money.

You are on a fixed salary, you idiot! No one spends money or saves money depending on what you do -- it all depends on what product does (and how poorly) because it affects sales. Unless you are asking them to buy some software (what would be a wrong approach), there is nothing he is supposed to do -- it's your job to implement everything you need to do your job properly.

Simplicity? (1)

Fri13 (963421) | more than 3 years ago | (#34742348)

What if you would just simply drop all money metrics and say that the pleased client who gets working product is happy customer and happy customer brings more customers in long run.

If the boss can not understand how valued for customer is that they are happy from the product they buy, he (or she) ain't good boss.
The company will go down when there comes competition whom gives better products. If you can not gain a monopoly or dominant market position or any other way to tie customers to your services and products. But that's it. No more new customers as what the company would get with happy customers.

Been there... (1)

dogsbreath (730413) | more than 3 years ago | (#34742360)

"at least once every 2-3 months I see some bug I fixed come back, and I can only assume it's because we don't have a formal test suite"

I assume that what you really want is to improve the development process and reduce your frustration time. A shiny test suite seems to be the answer.

Hmmmm... before you go buying a test suite you have to understand that the business should drive what you do. Business cases are all about research, analysis and documentation:

1. Metrics: What is the value of the app? What is the cost of maintaining it? ie: How many person hours are spent on development and maintenance? How many bugs are fixed per period and what is the average cost? Can you create some metrics for code quality and code brittleness based on version release stats and fault stats? You should have a handle on all of these items. If you are not in a position to know then you might start by persuading your boss to do some research on these topics. If you have an experienced crew then you should be able to estimate at a high level what the important metrics are, even if there are no formal stats being tracked. What do these metrics reveal as hot spots?

2. Processes: What are your current development, maintenance, and release processes? How are bugs tracked? If there is no formal process then what is the informal process? How do things happen? Do other people have thoughts about the processes and issues? Document how things are done and what the known or perceived issues are. If bugs are being reintroduced because of bad process then document this. If you don't have good processes then software tools will just be anchors that help drag you to the bottom. The worst thing you can do is introduce a tool that only you know, which then becomes part of a hidden, informal process that only you know.

3. Analysis: What does the research into metrics and processes show? How do you compare against standards either from industry or from inside the company? If you can't find a comparison then how do things look versus where you would like to be? What is the low hanging fruit: the obvious things that would improve quality? Lack of formal bug tracking, change control, and other metric gathering may be the obvious place to start.

4. Make a plan to improve things. This means that there has to be a sequence of changes and for each change there must be details of what the change will improve and how to measure the improvement. Be clear on what the goals are. No airy-fairy stuff. Hard, documentable, measurable changes. At this point you may identify some software that would help, such as a test suite. Remember though, software tools are just that: tools. No sense using a hammer to push in screws. Simply creating a change control process with a regression test may get rid of a lot of issues with spending a dime on tools or hardware. Simply formalizing and adhering to strict development and release processes that utilize (free) version control and (free) bug tracking may cure many problems. Similarly, standardizing methods can go a long way to improving quality and reducing problems.

5. Buy in: at every step of the way, from initiating the research into the situation to creating an improvement plan, you need support from your boss and from your peers. If nobody wants to change then fuhgetaboutit.

The above may seem like a lot but it isn't really. The initial metrics gathering, analysis, and plan development should not require a huge effort, especially if you are a relatively small shop. Some of the numbers may be sourced as WAGs from subject matter experts but I prefer to use the term "heuristics" or educated WAGs ;->

It only costs money if you need to hire help (1)

petes_PoV (912422) | more than 3 years ago | (#34742424)

Although a lot of consultants like to try and pin dollar costs on "quality", coding standards, development tools and other intangibles, the inescapable truth is that unless these processes and methods actually stop additional money from leaving the company they have no real-world, measurable value. Sure they might make your life easier (though most just increase the amount of time it takes to get something done), they might reduce the number of bugs, or the time taken to fix one, However if that time is already accounted for in your working day then the only way these things could save money is if they get you fired and can save your salary.

Since you appear to have the time to spend on these support tasks, there would only be a saving (and one that would take a considerable time to realise and even longer to pay back) you could only really make a case if you can say that int he future your workload will increase to the point where another person will have to be hired to assist ot take over these tasks. That's the only solid, dollar cost you will have to offer. Even then it's going to be an uphill struggle as the zero-cost alternative is just to make you work harder for the same money.

Have you tried sucking his cock? (0)

Noogie Brown (889153) | more than 3 years ago | (#34742442)

I find that's the best way to get what you want in life.

Minimal Test Suite (1)

ATestR (1060586) | more than 3 years ago | (#34742460)

How do you prove software testing saves money? Not going to happen in this case. The place testing saves money is during development. Once the software is deployed, its too late.

This is a pretty much the same place I'm in with my current position. 10-15 year old application that just "grew" into its current state, generally without formal testing. The decision to introduce formal testing into the process was probably made to meet standards of external users in my case. However, as the Software QA analyst who ended up setting testing up, I quickly realized that it would be impossible to set up complete testing in any reasonable time frame. Instead, what needs to be done is to establish a basic framework for testing, and introduce some very high end tests that will run the application's key points. Then, every time a new version is generated, run the tests. Shouldn't take very long. If a bug fix is introduced (other than cosmetic), add a check for that specific item. Do not bother to check everything. As the old axiom goes, if it ain't broke, don't fix it.

But does it? Not always. (1)

blind biker (1066130) | more than 3 years ago | (#34742476)

Civilization V was released with a ton of bugs, many of which were (are!) showstopping crashers, as long as you got to a few hundred turns. Add a trillion smaller, non-crasher bugs, and you have one of the buggiest releases of all time. And yet, between brick-and-mortar and Valve/Steam, they sold about a million units. That's about $50M in the bank.

Testing? What testing?

Overall cost may not be "in his own best interest" (4, Insightful)

dtmos (447842) | more than 3 years ago | (#34742478)

If you truly are working for a small software development company, cost may not be the issue. Frequently (I may even say universally) the issue facing the owners of small companies is not cost, or even profitability, but cash flow -- making sure that there is enough money coming in to make the next big expense coming up (frequently payroll). In small companies there typically aren't the cash reserves available to spend on unanticipated expenses or program delays. The boss may even agree with you that the overall cost would be reduced if software testing were introduced, but may not "want to invest the time and money" because the company does not have the cash available to make the investment.

For example, the software you're working on needs to ship by the end of January to receive payment from the customer by the end of February, so that payroll can be made in March. If bug-fixing stops in January for development of the bug-fixing program, the customer doesn't get his software until the end of February, so he doesn't pay until the end of March and so payroll is missed that month. Having fewer bugs in the long term has to be balanced by the need to stay in business until the "long term" is reached.

It's like anything else in finance: You have better options and get a better return on your investment if you invest $100,000 than if you invest $1,000, but if you don't have $100,000, it doesn't much matter -- you do what you can with the resources you have available. Similarly, you get a better return if you're able to invest your money for a year than if you must have it back in a week. (The guy who said "time is money" actually knew something.) This trade between cash flow and long-term efficiency (leading, one hopes, to profitability), i.e., knowing when and how much to invest, is the real art of business management.

The solution to your problem? IMHO, incremental investment. I agree with those above who suggest implementing tests for new bugs found. This should enable you to begin to work more efficiently over time, without substantially delaying current work, while collecting metrics (I know, I know...) that can help quantify the cost of the bug re-work. Once a substantial body of tested bugs has been collected, one can institute a formal testing program (preferably by including it in the next project, so that it is a planned expense) as a cost reduction, since by then it should speed up development work over the ad hoc method then in place.

Rewrite the fucker (0)

Anonymous Coward | more than 3 years ago | (#34742492)

You don't mention what this application is written in, but anything that was written ten years ago needs rewritten from the ground up. There are probably libraries and other much-improved functions that you could take advantage of now that didn't exist ten years ago.

What is test for you? (1)

Parker Lewis (999165) | more than 3 years ago | (#34742522)

See in this post comments, how people think that test is only unit tests. Testing must not be write by developers. Period. Reasons: developers are the bottleneck (about time) of the development process, they think that software as their chield, they are so much time over the software that they cannot see obvious defects, developers look for a defect how a negative point for his software (and we known that ANY software in the world, even the most simple one, has a lot of bugs), they have a lot of software know-how that average joe user do not have, so if you ask to a developer test your own software, you're wasting money. Sure, they must write and run unit testing, but this is the minor portion of testing. Testing starts when someone starts write requirements docs. Someone from testing team must read this document and check for ambiguities. At this moment, find some ambiguity, it's the most cheap testing. If you find a bug in the internal testing, it's more expensive then find it on doc. More expensive than that, is find a bug on user acceptance test. But, the really most expensive bug, is the one found in a running/deployed software: you have to take time to really understand the bug, put the analyst to re-design, programmer to fix, and testers to run it again. It's hard to find a excellent tester, cause he must really understand the application, it's good if he have a base background of develop (when automation is needed), he must understand how to write negative tests, he must have excellent communication skills (write and talk), etc And, the best documentation to start learn about testing, is the International Software Testing Qualifications Board syllabus: http://istqb.org/download/attachments/2326555/Foundation+Level+Syllabus+(2010).pdf [istqb.org] The good thing about ISTQB: it's not related to any commercial company. And, testing is not about run an audit in the testing team. Testing team has the same objective that all the company: deliver a free bug software (free bug is impossible, but at least at an acceptable level).

Re:What is test for you? (1)

Parker Lewis (999165) | more than 3 years ago | (#34742530)

Sorry, I forgot to send the comment as pure text (it lost the paragraphs).

Are You trying to kill Your own job !? (1)

hugetoon (766694) | more than 3 years ago | (#34742536)

Let it be, if it's the same bug, you'll fix it faster and go home earlier.
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>