Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Test, Test and Test Again

Hemos posted more than 7 years ago | from the did-you-test-that? dept.

41

snikkersnak writes "Richard Collins has written a piece about developers and testers; the article is arguing that in closed development these two roles have to be chained together one-on-one in order to reproduce the 'release early and often' effect of open source development."

cancel ×

41 comments

Sorry! There are no comments related to the filter you selected.

ABsolutly (5, Funny)

Ingolfke (515826) | more than 7 years ago | (#17506858)

I am in 100^ agrement hear. Far to often developer dont test and check they're code before they release it This resultsi n a poor edn-user experience. Tey wrok two fast always trying to git on to to the next thing.

Test test tst!!$

Re:ABsolutly (1)

pestie (141370) | more than 7 years ago | (#17516986)

The average quality of postings on the intarwebs being what it is, I was well into the second sentence before I even realized that post was a joke.

one-to-one (3, Funny)

brenddie (897982) | more than 7 years ago | (#17506886)

one tester for each developer?? bah, I have around 300 users^Wtesters.....

Re:one-to-one (0)

Anonymous Coward | more than 7 years ago | (#17509704)

>I have around 300 users^Wtesters.....

And there's what's wrong with OSS in a nutshell.

Re:one-to-one (2, Funny)

soloport (312487) | more than 7 years ago | (#17509908)

I have around 300 users^Wtesters.....

And there's what's wrong with OSS in a nutshell.


No, the GP was referring to portion of code the GP developed for Vista.

:-p

Wrong. (1)

Threni (635302) | more than 7 years ago | (#17506902)

Besides, there are books on testing. What's the point of this little spurt of typing?

Re:Wrong. (2, Insightful)

DrSkwid (118965) | more than 7 years ago | (#17507104)

welcome to the brave new world of 'anything will do' user driven content

One tester per developer? (2, Interesting)

ErGalvao (843384) | more than 7 years ago | (#17506948)

The article is very good, but I disagree with the part of "one tester per developer". I don't believe there is a formula to use, since each system is/can be a different case. I don't have a good theoretical background about the subject, but for me it always seemed a good idea to divide the system in specific parts like 'login routine', 'add foo routine', 'database connection' (...), and then test them.

Too amateur?

Re:One tester per developer? (3, Informative)

aadvancedGIR (959466) | more than 7 years ago | (#17506994)

That's the kind of testing you can actually do fast and close to the development team, but the real problems come from the impact your code could have on parts you barely know they exist in the full system.
Parts testing is just step 0 of the testing process.

Re:One tester per developer? (2, Interesting)

eric76 (679787) | more than 7 years ago | (#17507080)

I always test everything in detail as I write it. A good tester would be left with checking for something I missed, but that wouldn't take a full time tester.

At one time, I had a tester assigned to me half time. That was when I was given someone else's code to fix. That was the worst code I've seen outside of a classroom. The writer, for some bizarre reason, had been promoted to project manager in spite of the fact that he had only been a software developer for a couple of years or so. Prior to that, he was an insurance salesman!

I asked time and time again to let me rewrite the application instead of the useless task of fixing it. Instead, they assigned a tester to find bugs for me to fix and hinted that if I didn't get the code polished, they would fire me. He had no problem finding bugs but I had already seen the futility of trying to fix them.

I left a short time later for a new job. The task was handed over to a guy in the office next to mine who went hog wild fixing bugs for a week or two before realizing that it just didn't matter. Instead of asking permission as I had done to rewrite it, he just did it on his own. When someone would holler for a bug fix, he'd fix that bug for them and then get back to work on rewriting it.

About the time he finished the application, the project had been sold to the company for which we were developing the code. They had been sitting in on our meetings for a while and so one of the first things that company did was replace the project manager with someone who knew what he was doing. So when the other guy showed off the new code, instead of getting fired for insubordination, everyone thought it was great.

In short, the one time I had a tester assigned to me, I didn't want a tester. I wanted to start over. The rest of the time, I've had to beg to get anyone to test the code instead of just using it for its intended purpose.

Re:One tester per developer? (1)

functor0 (89014) | more than 7 years ago | (#17522514)

That was the worst code I've seen outside of a classroom. The writer, for some bizarre reason, had been promoted to project manager in spite of the fact that he had only been a software developer for a couple of years or so.
If he's that bad at coding, I'd promote him to project manager just to keep him from coding!

Automated unit tests (4, Interesting)

tcopeland (32225) | more than 7 years ago | (#17507170)

I think the trick here is for the developers to write lots of automated unit tests. This catches the vast majority of obvious bugs, and lets testers become more like "power users" or something. That is to say, rather than the testers writing bugs like "I did xyz and it crashed" they can write bugs like "it'd be great if this screen had a 'quick user lookup' field". The testers then begin to think of ways to improve and streamline the application, the end users get a better experience, and the developers have more fun because they're actually adding features rather than constantly fixing easy bugs.

Andy Glover [thediscoblog.com] has a good blog on testing and QA in general. He uses FitNesse and TestNG and various other Java testing tools so he's pretty up on all the latest junx.

Re:Automated unit tests (1)

Ritchie70 (860516) | more than 7 years ago | (#17509102)

You seem to have confused QA and BA.

It's OK, our QA group tends to as well.

Re:Automated unit tests (2, Interesting)

Anonymous Coward | more than 7 years ago | (#17511176)

Oh, good idea! Have the developers do testing, which frees up the testers to do design work.

Uh...

If you think that the work of testers can be replaced by unit tests, you either have really stupid testers or really stupid developers. Or both.

Unit tests don't fix bugs. A unit test tests a unit (hence the name), so if 'xyz' is a complex setup, it's likely your unit test won't have caught it. If a developer thought it could happen, he would have fixed it to begin with. Testers are there to find bugs that developers *didn't* think of, and unit tests written by developers can only catch bugs developers *did* think of.

And counting on testers to do design work is just stupid. It takes a special kind of person to design a good user experience, and (the vast majority of) testers aren't that person. You'll end up with a frankenstein app, where each tester's pet feature is assembled into one app. If that's better than what you have now, then you're in pretty sorry shape.

Re:Automated unit tests (2, Insightful)

ChunderDownunder (709234) | more than 7 years ago | (#17519024)

Well... Testers can't be replaced by unit testing, nevertheless unit testing does have some merit.
When a tester spots a bug, a unit test might be written to document the behaviour, that the developer didn't think of, thereby reducing regressions.

In additional to 'bugs', testing is useful in checking whether an application performs a task as dictated by a specification, i.e. missed functionality. Unit testing won't catch bugs in code that hasn't been written in the first place!

As for the 'design work' matter, the development ought to be a collaborative effort between the Business Analyst, UI Designer/Usability specialist, Tester and Coder. The tester is often the first person to see the application. If a user interface isn't usable in the eyes of a tester it probably won't be to the end user either. But rather than delegate enhancements to the coder, the tester ought to refer these thoughts back to the usability person. As in the example, if the UI specialist agrees that yes, "it'd be great if this screen had a 'quick user lookup' field" then that could be written as a requirement in the specification in consultation with the BA. - e.g. 'the system shall provide quick user lookup'.

Agreed, it's not the role of testers, and hence developers, to go adding features just for coolness sake. That's how projects never get finished. But in testing that the software meets requirements in a specification they're also, to some degree, testing that the specification is complete and workable. And if the spec is set in stone then at least suggested enhancements can be documented for a future revision.

Re:Automated unit tests (0)

Anonymous Coward | more than 7 years ago | (#17512094)

Emperically, I know that unit tests only capture certain classes of bugs which in practice, turn out to be minor (e.g. rounding). They typically miss basic issues like menu item or control disappearance, error dialogs which appear thousands of times, inappropriately or not at all, basic CRUD and failed i/o in general. Interface issues are inevitably missed, showing the the programmer's bias that the interface is an afterthought (Tell it to Apple).

Some of these classes of bug *could* be caught by unit tests (e.g. CRUD and I/O), but it's no easy task to convince a programmer to stop and write them.

I stopped reading... (1)

ErroneousBee (611028) | more than 7 years ago | (#17507208)

When he started out stating that all physics grads are stupid. If I wanted stupid name calling, I'd go eavesdrop on next doors kids.

Re:I stopped reading... (1, Insightful)

Anonymous Coward | more than 7 years ago | (#17507572)

...stating that all physics grads are stupid.

First of all, I don't believe that is what he said! Quite the opposite, he seems to imply that physics grads are smarter (in at least one area) than compsci majors! And I think that is right on! Physics grads are trained in mathematics, physics and, more generally, how to solve a host of complicated problems. Compsci grads are taught programming languages and algorithm construction and not a helluva lot of problem solving.

I personally prefer programmers who are engineers with a master's in compsci or even self-taught. They problem-solvers first and algorithm-artists (to coin a phrase) second.

Re:I stopped reading... (2, Interesting)

ErroneousBee (611028) | more than 7 years ago | (#17508388)

Just re-read his opening comment. I was imagining a class of physics students licking butter off of bread, and it may have led me astray.

As for the CS/Physics thing, I find physics grads (me, BTW) have a big weakness when it comes to 'correctness', and being able to simplify algorithms. If they can fix those weaknesses, they become better than ordinary CS grads. The same for CS grads not being able to break out of problems, if they can broaden their narrow focus on 'code elegance', they to can do great things.

See the original Netscape browser for an example of the awfulness of Physicist code, and regexp for an example of CS theory beating usability.

Of course, neither class of programmer have anything on an MBA implementing the company payroll as an Excel macro.

Re:I stopped reading... (0)

Anonymous Coward | more than 7 years ago | (#17511692)

"I cannot believe this isn't butter!"

Uh, from my experience... (3, Insightful)

Blakey Rat (99501) | more than 7 years ago | (#17507272)

From my experience, open source software may be released early and often, but it sure isn't tested thoroughly. Sometimes it seems to me that it's not tested at all, when extremely blatant bugs appear in releases. (One of the recent ones I found is a Import dialog in Inkscape that doesn't alphabetize correctly. This isn't 1985, open source developers, alphabetization of a list of files is a solved problem!)

Of course, I also don't think "release early and often" is a good philosophy. If you release early, by definition you're releasing something that isn't yet ready for the public, and when the public uses it they're going to be disappointed and a lot less likely to try your next version.

Re:Uh, from my experience... (1)

PingSpike (947548) | more than 7 years ago | (#17507570)

I think he may be refering to internal releases. Instead of hoarding a load of fixes/features for large releases, increase the amount of incremental releases so the testing team can get their hands on the code as it is written.

Re:Uh, from my experience... (1)

Blakey Rat (99501) | more than 7 years ago | (#17511132)

That's a possibility, and is probably good advice unless your build system is especially cumbersome.

Re:Uh, from my experience... (0)

Anonymous Coward | more than 7 years ago | (#17509776)

Could you post a report at the Inkscape bug tracker here: http://sourceforge.net/tracker/?atid=604306&group_ id=93438&func=browse [sourceforge.net] , reply with more details, or pop into #inkscape on Freenode and bring this up, if you see this problem in current builds? Thanks.

Re:Uh, from my experience... (1)

Blakey Rat (99501) | more than 7 years ago | (#17511006)

Already have done. I don't have the URL handy, but it's bookmarked on my home computer... look under OS X bugs. I believe an Inkscape developer actually pushed the bug to the GNOME project, but I haven't checked that yet.

Re:Uh, from my experience... (1)

kirun (658684) | more than 7 years ago | (#17513748)

http://sourceforge.net/tracker/index.php?func=deta il&aid=1609779&group_id=93438&atid=604306 [sourceforge.net]

Ahh, the old case-sensitive sort. I've never quite understood the point of doing it that way really, even given case-sensitive filenames. It is the "normal" X behaviour, though KDE does offer the option of insensitive sorting.

Re:Uh, from my experience... (1)

Blakey Rat (99501) | more than 7 years ago | (#17515696)

It's a plain old bug as far as I'm concerned. Bryce's comment suggests that just to cope with this, you need to careful plan how you're going to name your files... I say that's ridiculous. My computer should cope with what *I* do, not the other way around.

Or look at it from a usability standpoint... what is easier to remember? Whether a filename starts with S, or whether a filename starts with a capital or lowercase? Not only is S easier to remember, but it also narrowed the selection down tremendously.

The fact that it's the year 2007 and GNOME (apparently) doesn't sort filenames correctly is just... I don't even have a word for it. But the Linux worlds needs to HAMMER DOWN THE BASICS before they spend one more day talking about 'defeating Microsoft.'

Re:Uh, from my experience... (-1, Troll)

Anonymous Coward | more than 7 years ago | (#17517044)

what is easier to remember? Whether a filename starts with S, or whether a filename starts with a capital or lowercase?

The latter. You wierdo. I'm guessing you're an effeminate "verbal" thinker. Men are spatial thinkers, and the size of an object is an obvious basic attribute.

You're just not Manly enough for Linux, fool.

Just use small letters, jerkwad. I organise my files by lower/upper case based on importance - the more of the file name is upper cased, the earlier it appears in the list.

Re:Uh, from my experience... (1)

try_anything (880404) | more than 7 years ago | (#17535948)

what is easier to remember? Whether a filename starts with S, or whether a filename starts with a capital or lowercase?

It's the right way to sort for the kinds of people who were Unix users twenty years ago: programmers, scientists, and engineers. You're right that it's one of those things that will have to be "fixed" before Linux is "ready for the desktop." But it wasn't wrong in the first place -- it was (and still is, in most cases) right for the people who use it. The only reason to change it is to take over the world.

Sigh. I feel the same way thinking about that as I do when a local dive turns into a Starbucks or an Urban Outfitters. It makes we want to go help the Californians ruin Bozeman for a while.

Re:Uh, from my experience... (1)

Blakey Rat (99501) | more than 7 years ago | (#17540300)

Few thoughts:

1) I'd wager it was done that way CPU-saving or code-saving measure because it's easier to write a case-sensitive sort function than a case-insensitive one.

2) I'd wager that the reason "the type of people who use Unix" prefer it that way is only because they are used to it from using Unix.

3) I'd wager that if you polled just "the type of people who use Unix" that they would prefer an actual alphabetical sort. Since that's what, for instance, the index of every book they read, the card catalog at their brain institutes's library, etc all use.

Re:Uh, from my experience... (1)

try_anything (880404) | more than 7 years ago | (#17543914)

2) I'd wager that the reason "the type of people who use Unix" prefer it that way is only because they are used to it from using Unix.

Preferring it that way is a simple matter of being used to adhering to case-sensitive naming conventions. Programming in most languages will teach you that, as will sharing large amounts of data and files with colleagues. (When using mathematical notation, also, it is often helpful to use capitalization consistently.) On my system, for my work, if I know the type and approximate contents of the file I'm looking for, then I usually know its capitalization.

On the other hand, when I receive personal files from friends or name files for my friends, such as photographs, capitalization follows no system at all, so case-insensitive alphabetization makes more sense. Also, it isn't a big deal to look at a case-insensitively alphabetized list and mentally filter out the items with the wrong capitalization, so techies will be less burdened under the new way than non-techies were under the old way.

Closed source does not work like open source (1)

asb (1909) | more than 7 years ago | (#17507568)

I have to call bullshit on that argument.

First of all the word "early" in the good old "release early, release often" mantra really means releasing untested software.

Secondly, pardon my own misleading subject line, the division is not about libre vs. proprietary (open source vs. proprietary). It is between gratis vs. non-gratis software. When people use software they expect to get what they pay for so when the users do not pay in cash they pay in testing time. This time is what non-gratis developers have to pay when they do expensive system testing.

Re:Closed source does not work like open source (2, Informative)

TheRaven64 (641858) | more than 7 years ago | (#17508970)

Many Free Software projects (the BSDs in particular) split the tree into three sections:
  1. The active development 'this may compile, and if it does, then it may work, but don't expect it to be stable' branch.
  2. The 'this stuff seems to work, but may still break' branch.
  3. The 'we think this stuff is stable now' branch.
On FreeBSD, for example, these three branches are called -CURRENT, -STABLE, and -RELEASE. Typically, only active developers run the -CURRENT branch, but a lot of people run -STABLE. People running either can file bugs, which make -RELEASE more stable, and people running important services run -RELEASE (and possibly keep a -STABLE box around for testing).

Contrast this with closed development. In this case, the first branch will be internal-only, and snapshots of the second my be released as betas. This reduces the number of people testing both of these branches, which then has an impact on the stability of the third.

Ballmerism (1)

TempeTerra (83076) | more than 7 years ago | (#17507720)

When asked to comment on whether this methodology would be adopted at Microsoft, Steve Ballmer responded "developers developers developers testers testers testers!"

*Everything* should be tested! (1)

Richard Steiner (1585) | more than 7 years ago | (#17508796)

Heck, I even run internal tools that I write for internal company developer use past other programmers as a sanity check (after using them myself extensively) to make sure that things are actually working before I make the tools available to the general developer public.

Multiple testers are better than one, since different users often have different approaches, and I think testers are optimally obtained from a pool of actual end users (or in our case, quite often, from a pool of business analysts who either used to be end users or who are still end users). In any case, the folks testing should probably not be familiar with the code internals. It's too easy to unconsciously behave in certain ways (and avoid certain types of behaviors) if you know precisely what is expected by the program logic.

Moo (2, Interesting)

Chacham (981) | more than 7 years ago | (#17510284)

Don't forget The Career Programmer [slashdot.org] which discusses testers at length.

Thorough self-testing (2, Informative)

c0d3h4x0r (604141) | more than 7 years ago | (#17511276)

I've been a professional software developer for over ten years. Based on my experience, I can say with certainty that there are three main sources of bugs.

In order, they are:

1. Lack of sensible high-level design and architecture. This causes the bugs that require massive amounts of churn or ugly hacks on top of hacks to fix.

2. Source code that was written without regard for human maintainability/understandability/readability. This causes the bugs that take the longest time to investigate and which are the trickiest to fix without breaking something else without realizing it.

3. Lazy developers who neglect to thoroughly self-test their own work before calling it finished. This causes the most bugs in gener, but fortunately most of them are easy and quick to fix as they are mere oversights.

Re:Thorough self-testing (1)

ClosedSource (238333) | more than 7 years ago | (#17511958)

Your list appears to be of items that make finding or correcting bugs more difficult, not a list of sources of bugs. Source code that was written without regard to human M/U/R doesn't necessarily contain more bugs than code that does take these concerns into consideration.

Re:Thorough self-testing (0)

Anonymous Coward | more than 7 years ago | (#17516066)

In other words:

1. Bad design
2. Bad programmer
3. Lazy programmer

I wonder whether during your ten years of experience you had to modify a large piece of code in which each layer is new to you (hardware, drivers, OS, abstraction layers and application), and you had to do it in six weeks. All this while the company doesn't even have testers and you have to train someone at the same time.

So I think there's a much more common source of bugs:

4. Lack of time and resources to write code properly.

It may sound like an excuse, but sometimes you really can't afford a week to find that the audio driver has a problem when the timer is being used by two tasks at the same time, and that debug printings hide this problem, and that the debugger doesn't even work because there's an overflow in a hardware buffer when you put a breakpoint.

Oh yes, and tomorrow the customer is expecting to see "something running" before paying for the next phase.

I usually have another name for #4:

4. Bad management.

So poorly written! (1)

Dahamma (304068) | more than 7 years ago | (#17512326)

The first few paragraphs were so poorly written and full of unfounded opinions (even the EDITOR had to step in three times to comment on it!) that I couldn't even get to the potentially interesting parts... since I knew I wouldn't be able to trust anything else after that point.

Should testing drive development or ... (1)

BagOCrap (980854) | more than 7 years ago | (#17534176)

Without development, there won't be much testing. Likewise, without testing, there won't be much development. Farewell, reputation.
Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>