Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

How Healthcare.gov Changed the Software Testing Conversation

Soulskill posted about 10 months ago | from the polls-say-software-testing-is-patriotic dept.

Software 118

An anonymous reader notes an article about how the tribulations of Healthcare.gov brought the idea of software testing into the public consciousness in a more detailed way than ever before. Quoting: "Suddenly, Americans are sitting at their kitchen tables – in suburbs, in cities, on farms – and talking about quality issues with a website. The average American was given nightly tutorials on load testing and performance bottlenecks when the site first launched, then crumbled moments later. We talked about whether the requirements were well-defined and the project schedule reasonably laid out. We talked about who owns the decision to launch and whether they were keeping appropriate track of milestones and iterations. ... When the media went from talking about the issues in the website to the process used to build the website was when things really got interesting. This is when software testers stepped out of the cube farm behind the coffee station and into the public limelight. Who were these people – and were they incompetent or mistreated? Did the project leaders not allocate enough time for testing? Did they allocate time for testing but not time to react to the testing outcome? Did the testers run inadequate tests? Were there not enough testers? Did they not speak up about the issues? If they did, were they not forceful enough?"

Sorry! There are no comments related to the filter you selected.

First Post (-1)

Anonymous Coward | about 10 months ago | (#45778163)

Death panels be damned, AC lives!

WTF Not forceful enough? (3, Insightful)

Anonymous Coward | about 10 months ago | (#45778185)

Ya those damn testers, they just can't communicate the issues to management. Like that NASA engineer and the O-rings. Stop blaming the testers.

Re:WTF Not forceful enough? (4, Insightful)

ObsessiveMathsFreak (773371) | about 10 months ago | (#45778579)

Well you either blame the help, or else someone important is going to have to take the blame.

Someone important *should* take the blame (3, Insightful)

Anonymous Brave Guy (457657) | about 10 months ago | (#45778655)

One of the most insightful truths ever told to me:

It is always management's fault.

This goes right to the root of the tree, because by definition if the people further out couldn't get the job done or didn't have the right resources to do it, it was management's responsibility to fix those problems. The buck stops with the most senior managers on a project, whose only two choices are to explain what is needed to succeed and then do so if given those things, or to fail.

Re:Someone important *should* take the blame (0)

Anonymous Coward | about 10 months ago | (#45779685)

Bullshit. The President doesn't write code.

Re:Someone important *should* take the blame (3, Informative)

Anonymous Brave Guy (457657) | about 10 months ago | (#45780015)

The President doesn't write code.

No, but he presumably appoints the senior staff whose teams ultimately will write code, and capture requirements, and run tests, and all the rest. For something on this level he probably had a direct say in things like budget and timescales and the scope and high-level organisation of the project as well. If he asked for the impossible, he bears responsibility for that. If he didn't know it was impossible, he should have hired better advisors before committing to the project. And if it wasn't impossible but failed anyway, he should have appointed a senior management team that was up to the job.

(I'm not in the US and have no allegiance to any US political party, so please don't try to turn this into some sort of red vs. blue flamewar. I'm just translating the same general principles that apply to senior management for any infrastructure project into the specific context of healthcare.gov.)

Re:Someone important *should* take the blame (0)

Anonymous Coward | about 10 months ago | (#45780379)

No, he doesn't.

The government has a bidding procedure, where contractors bid for projects just like healthcare.gov. Then there's supposedly an impartial process for picking the best, meaning lowest bid. Once a large project gets started, the government paper-pushers are usually answerable to congress, and when congress, specifically the house in this case, wants to make sure the project is a miserable failures: the government project managers, who are way lower on the pole than a presidential appointee, are going to make certain the project goes up in smoke.

Re: Someone important *should* take the blame (3, Informative)

kenh (9056) | about 10 months ago | (#45780485)

What a staggeringly ignorant post.

The house controls the purse strings and has oversight powers over the operation of the government.

Healthcare.gov has allocated some $634M and to date had expended just over half of it by the time it went live, lord knows what they spent since Oct. 1... The house withheld not one nickel from the project - it's fully funded.

As for House interference, you are simply inventing a conspiracy out of whole cloth... You really think the House 'secretly' met with those overseeing the development of the website and told them to mess it up? It was obvious to anyone looking at the situation that this was a train wreck waiting to happen - remember Republicans saying the best way to defeat Obamacare was to let it play out and fail on it's own? If there was a conspiracy as you imagine, how is it not one person has stepped forward to expose it?

The contractor selected had a history of failed projects, why is it so hard to believe they failed again?

Re:Someone important *should* take the blame (2)

phantomfive (622387) | about 10 months ago | (#45780789)

Bullshit. The President doesn't write code.

His staff was saying, in the week before the website went live, that everything would be great. That's on him, for not knowing (or lying?) about the state of the website.

Re: Someone important *should* take the blame (0)

Anonymous Coward | about 10 months ago | (#45781267)

So obama is surrounded by syccophants. That also explains why western finance is in the crapper.
Plus, Obama showe his tyrant side when it came to mistreat the soldier Bradley Manning. A Corrupt Imperium, that is the U.S.

Re:Someone important *should* take the blame (2)

phantomfive (622387) | about 10 months ago | (#45780787)

It is always management's fault.

That is exactly the job of management: to get the job done. When the lead programmer quits? Management is the one who needs to find another one.

Ender's game gives a perfect example: Colonel Graff is management. He sorts through all possible candidates, develops a program with one goal in mind: beating the formics.
Ender is the guy who actually does the work, he is the engineer. He does the battle plan, and implements it. But the Colonel is watching, making sure things are going according to plan, and will put different people on the project if necessary. It is his project, and he will manage it (even if he doesn't actually do any of the actual work).

Re:Someone important *should* take the blame (1)

Anonymous Coward | about 10 months ago | (#45780955)

That is why good CEOs like Steve Jobs takes responsibility to 'test' the products and involve himself in the design process. Because in the end it is his responsibility.
Trust but verify.

Re: Someone important *should* take the blame (0)

Anonymous Coward | about 10 months ago | (#45781275)

Good colonels and generals know how to do the Work THEMSELVES. See Admiral Rickover or General Groves.
Rickover was universally hated because of his expertise and his ending of politico-officers careers who stumbled into nuclear duty.

Re: Someone important *should* take the blame (0)

Anonymous Coward | about 10 months ago | (#45781285)

If America were governed by Rickovers, the Chinese engineers who run China would not look as good as they do.
The West has been hijacked by the Social Science Frauds. Now, deal with it and prepare mentally for civil war. Because that is the logical results of Government Incompetence.
At NSA, BND: YOU will be held accountable, too, if you dont get a handle on these Gov-Crappers.

Re:WTF Not forceful enough? (1)

pigiron (104729) | about 10 months ago | (#45778583)

You are forgetting the first rule of bureaucracy:

It is NEVER upper management's fault.

inspecting an unfinished POS (3, Interesting)

harvey the nerd (582806) | about 10 months ago | (#45778629)

Imagine being the QA inspector on a 1985 Jugo car. No matter what you say, the entire thing is a POS. The only question is whether you need your paycheck that badly. Politics and unrestrained corruption simply don't mix well with code.

Re: inspecting an unfinished POS (0)

Anonymous Coward | about 10 months ago | (#45781305)

You can use your 100 billion brain cells and your throat to
A) rip another asshole into your lazy and dumb colleagues and
B) educate your colleagues about quality procedures and
C) think of colleagues as being a Team Of Warriors, trying to beat the competition at all cost

Worked out EXCELLENTLY for the japanese. Works for Yougos, Russkies and anybody else IF SOMEONE RIPS ASSHOLES.

development not complete (3, Insightful)

BradMajors (995624) | about 10 months ago | (#45778193)

It is not really a question of testing. Parts of the software were missing or incomplete. You can't test what isn't there.

Re:development not complete (0)

Anonymous Coward | about 10 months ago | (#45778391)

It is. But the tester is not to blame. As tester test whatever he is told to. Since software gets to be huge, the tester test specific parts.
The quality team determines the test. If they are not fully understanding the full thing, stuff goes wrong and certain connections never get tested.

Re:development not complete (1)

bsolar (1176767) | about 10 months ago | (#45778649)

Of course you can test incomplete software. The relevant tests should fail until the implementation is complete and correct.

Re:development not complete (0)

Anonymous Coward | about 10 months ago | (#45779763)

Are mixing it up with unit testing?

Re:development not complete (2)

ranton (36917) | about 10 months ago | (#45780215)

Are mixing it up with unit testing?

TDD isn't the only time you have tests before the code is written. It is a common practice to create test cases for functionality that has not been completed or even started on. The status on the test case may be "Implementation Not Complete", which is just one type of test failure.

This way you can still have reports on development progress and show that large sections of functionality have not been tested yet (because they aren't finished).

Re: development not complete (0)

Anonymous Coward | about 10 months ago | (#45781317)

Testing should be a continous activity from Day 1 of coding: As soon as some little features works, there should be Testers documenting the test procedure, building test harnesses and of course feeding test results back to developers and leadership.
All testing should be done as realistic as possible: Realistic scale, realistic hardware, realistic operating procedure. In the Âtrain as you fight spirit.
Thereby at day 280 of your project, the system will have the wortst bugs eliminated.
But you know what ? Leaders from Tokio to Washington to Talinn are corrupt, cynical, nasty, ignorant bastards. They will never be able to acutally exchange ideas with non-bastards.
Obama is exactly this kind of bastard, and that is why this thing exploded in his face.

Re: development not complete (0)

Anonymous Coward | about 10 months ago | (#45781339)

Of course developers must immediately act upon credible bug reports and fix them before implementing new functionality.
Managers/leaders/directors must not be above playing with the system themselves, reporting bugs and making sure they are fixed.
Instead, they sit in meetings and play politicks.

Re:development not complete (5, Interesting)

hguorbray (967940) | about 10 months ago | (#45778683)

Having been brought on late to QA a few death marches/trainwrecks in my time I have found that many projects don't get QA involved until way too late in the game.

This was very common in the .com boom days when everything was developer-centric and testing was seen as an unneeded cost that could be covered by the coders themselves -I don't need to tell anyone here why that is a bad idea.

Ideally QA gets to help validate that the functional requirements are adequately addressed in the design. In many cases, lacking a spec of any kind I would have to create one of my own based on what the product was able to do or close to being able to do at that time in order to make a test plan.

When you are brought on board a sinking ship there is no point in blaming the crew for the state of the ship -all you can do is damage control to validate whatever is working and then lower the bar as to what constitutes 'working' or 'functional' -particularly if some major components or functionality are missing.

You're going to be seen by management as the people who are going to point out what idiots and incompetents the developers were and be seen as the enemy by the developers who were probably led down the rabbit hole by changing or nebulous requirements and unrealistic schedules...

So it is important to try to walk the middle line -making observations about the current situation without casting blame or making guesses about how the project got to that state (although it may be obvious when you look at the principals and the agenda). Gap analysis of both testing and in the product functionality and features is another thing that needs to be done more often in order to present a realistic picture of the current state of the product or project.

As a consultant it is nice to be able to come into these things knowing that you didn't help cause the trainwreck -you are just there doing triage and trying to save the patient....and sometimes management will listen to you about project and requirements that they ignored when brought up by their own people. Even if it is 20/20 hindsight perhaps they will heed the techies the next time they embark upon this path -Nah!

-I'm just sayin'

Re:development not complete (1)

Dan667 (564390) | about 10 months ago | (#45779033)

Most of the time I find that many Developers want to be handed a perfect spec and let to go into a basement and code it. The real world does not work like this and I find the best work comes when the guy who will be writing the software actually collects the requirements and does a good job of that including a plan of how to test the code. I think part of the problem is pretending requirements are some uncertain quantity, when they are usually very straight forward to collect if you are willing to do the work.

Re:development not complete (4, Insightful)

ranton (36917) | about 10 months ago | (#45780303)

The real world does not work like this and I find the best work comes when the guy who will be writing the software actually collects the requirements and does a good job of that including a plan of how to test the code.

The worst code I see is written when the programmer is given some narrow requirements on some ticket and they code directly to those requirements with little to no knowledge of the overall system they are working on. In some magical world where all of your requirements are perfect this could work very well. But part of being a solid developer is knowing how to spot the "smell" of bad requirements. While sometimes you can do this without any knowledge of the overall system, you are much more likely to have the right insights if you have some relevant domain knowledge.

I have a current project that I am maintaining where I wrote most of the code in a vacuum without understanding the customer's real needs. It is a horrible mess and I would have done things completely differently if I was involved in meetings with the clients early on. I'm not saying my code would have been perfect, but there were some massive disconnects between the assumptions I drew from the written requirements and the explanations I got from the clients once I was given more access to them. Now there is no time or money for massive rewrites or refactorings, so it continues to be a thorn in my side.

Re:development not complete (1)

ShoulderOfOrion (646118) | about 10 months ago | (#45780473)

So it is important to try to walk the middle line -making observations about the current situation without casting blame or making guesses about how the project got to that state (although it may be obvious when you look at the principals and the agenda).

In my experience, it is *always* obvious when you look at the principals and the agenda how the project got screwed up. You're right though--diplomacy often requires walking the middle path.

Battleifeld 4 (2, Funny)

Anonymous Coward | about 10 months ago | (#45778225)

These same questions plague Battlefield 4

Re:Battleifeld 4 (3, Funny)

martin-boundary (547041) | about 10 months ago | (#45779007)

These same questions plague Battlefield 4

And not to mention, Battlestar Galactica too. A lot of the needless drama on that show could have been avoided if they had just licensed their testing framework from the Cylons.

Re:Battleifeld 4 (0)

Anonymous Coward | about 10 months ago | (#45779251)

As well as xfinity and their Motorola STB/DVRs - absolute crap software that just doesn't work in odd ways at odd times.

It also works in the strangest and stupidest ways by design. For example, it only records the last ½ hour of the live HD show you're watching even though there are hundreds of GB of free space on the disk, so if you pause the show for a total of more than a ½ hour due to phone calls or other distractions, you lose content -- sure, if you think to do so, you can hit the REC button before pausing, but come on, that's absurd to expect me to bother/remember to do.

Of course, there is no practical way for me, as a consumer, to explain a problem and how to reproduce it to someone who cares at xfinity - one just gets a canned response and the problems are never fixed even years later after several software updates.

Were known management tools used? (2)

whizbang77045 (1342005) | about 10 months ago | (#45778265)

It does not sound to me as though known management tools were used. Did they sit down with the government personnel in charge, and present their approach, and what the site would look like (menus, flow, etc) when finished? Were there testable milestones, and a final presentation of working software? It sure doesn't sound like it.

Re:Were known management tools used? (1)

Anonymous Coward | about 10 months ago | (#45778449)

They didn't even use a RDBMS even near mainstream, much less one that can be used with standardized process.

Can MarkLogic pass the ACID test? Their website states that they can, but it would be nice to see real world results, especially well before this was thrown as a pivotal site the American public has to deal with.

Maybe the ACA website would have had a far better fate had they used DB/2, Oracle, or even MS SQL. No, it doesn't have the "pop" of NoSQL, but RDBMS servers are made to just do the job quietly and reliably, with very strict ACID compliance... and this is something that should be thought crucial for medical records.

The right way would have been to do it old school... zSeries machines, DB/2, Parallel Sysplex. Expensive iron, for sure, but something that is tried and true. Then build the website on a standardized, well-secured stack. Yes, more initial investment in mainframes, but there would have been a lot fewer headaches all the way around.

Re: Were known management tools used? (0)

Anonymous Coward | about 10 months ago | (#45779019)

so deeply ingrained cynism

Re:Were known management tools used? (2)

hey! (33014) | about 10 months ago | (#45779031)

No, it doesn't have the "pop" of NoSQL,

More to the point, it doesn't have the scalability across distributed systems. Show me one application approaching this scale, just *one*, that relies RDBMS clusters and two-phase commit exclusively to support this kind of transaction volume. Don't get me wrong; I'm an old-school RDBMS guy myself; I know a lot about relational database systems, including their limitations. I'd look to the way outfits like Amazon, ,Google or LinkedIn design their infrastructure rather than Oracle on big iron. This is well outside the sweet spot for that approach.

RDBMS servers are made to just do the job quietly and reliably, with very strict ACID compliance...

This is a very simple-minded approach to architecture, one that's admittedly very serviceable in a wide array of applications. But useful as ACID is as a set of assumptions you can rely upon, it's not the only way to create a reliable, serviceable system. In fact there are situations where it's provable that ACID falls short. Google "CAP Theorem" and "eventual consistency". It's fascinating stuff.

Re:Were known management tools used? (0)

Anonymous Coward | about 10 months ago | (#45779683)

If they used RDBMS or not, it all depends how they design and architect their infrastructure and stress the system with millions of concurrence transaction before open it for the public. If we deploy and fine tune RDBMS, whether Oracle or other databases, keeping the overall expected load and performance targets, this infrastructure would have performed far better than what we have been seeing.

Re:Were known management tools used? (1)

hey! (33014) | about 10 months ago | (#45779743)

Well, we're talking far too abstractly here to be very meaningful. I'm not saying an RDMS couldn't be *part* of the picture. I'm saying that a system architecture that punted all the persistence and data consistency problems to a distributed RDBMS is a non-starter for something on this scale. People don't build systems like this one that way. If they don't, even though that technology is a mature one, that's a good reason to be skeptical of the idea that that approach would be a panacea.

Re: Were known management tools used? (0)

Anonymous Coward | about 10 months ago | (#45781373)

What we know is that banks, insurance companies, telephone companies run massive systems with billions of customers in some instances on Oracle and DB/2.
But you are right that Excellent Engineers are the most important aspect of this project. Good people will make you this thing even with PostgreSQL on 1000 distribuited machines, each running an RBDMS instance.
I am talking of people aged 40 or older, with 15 years of IT experience and a Computer Science degree.
But of course, in the Corrupt West, they go for the Best Bribing Bidder and his army of 23 year old Ruby-on-unicorn team.

Re:Were known management tools used? (2, Insightful)

Anonymous Coward | about 10 months ago | (#45778473)

How do you make a presentation of working software when the working software is designed to pull data from half a dozen databases outside of your control? You might as well throw a football in the air and then claim you're a world class quarterback, you're just waiting for the rest of the world class team to show up and assist you.

Re:Were known management tools used? (4, Informative)

hey! (33014) | about 10 months ago | (#45778713)

It does not sound to me as though known management tools were used. Did they sit down with the government personnel in charge, and present their approach, and what the site would look like (menus, flow, etc) when finished? Were there testable milestones, and a final presentation of working software? It sure doesn't sound like it.

They might well have done all these things and still failed to catch the problems before the site's launch.

Performance, like security (ack! scary!) is a non-functional requirement -- that is to say it's not the kind of requirement where you can sit down with a checklist and say, 'yep, this it works,' or 'no, it doesn't.' You have to develop a more sophisticated test.

Load testing is a step in the right direction, but you also have to look at system architecture. Remember the days before people figured out that you had to load web ads asynchronously, after the page content was loaded? Sometimes the page load would be slow, not because the page's server was loaded, or because of the user's browser or internet connection were slow. Often it would be the ad server that was overwhelmed, which if you think about it is bound to be more common than the content server being overwhelmed. You could functional test and even load test the heck out of a page with synchronous ad loads, but until it went into production chances are you wouldn't catch the fatal performance flaw. That kind of problem is architectural; some of the data being delivered is coming from servers outside your control.

Ordinary tests are about ensuring reproducible results, but when the architecture leaves you vulnerable to things happening on servers and networks outside your control your problems *aren't reliably reproducible*. You have to design around *possibilities*.

Some of the problems with Healthcare.gov were of this nature, although with not so simple a solution as "use window.onload()." The site is supposed to orchestrate a complex, *distributed* process *synchronously*. You have to go out the Homeland Security's database to confirm citizenship, then to the IRS databases to confirm claims about income, then get quotes from the private insurers that meet the customer's needs. There is, in my opinion, no way to be 100% sure, or even 80% sure that a system like that will work under real work load, unless you present it with real work load.

Were I architecting such a site, I'd plan to do a lot of that work batch; that is I'd build the healthcare application offline on the user's browser, with internal consistency checks of course. Then I'd send the user's application through a batch verification system, emailing him when a result was ready. This is a clunky and old-fashioned approach, but it wouldn't force the user to chain himself to his browser . And it would have more predictable performance. Predictability is a vastly under-rated quality in a system. A system which is fast most of the time may not be as desirable as one which provides the answer consistently.

Re:Were known management tools used? (1)

mlts (1038732) | about 10 months ago | (#45778775)

I think you hit the nail on the head, although I'd probably code in a messaging similar similar to what banks have so a user logs onto the healthcare site to receive notices... this will protect against phishing.

The batch system is by far the best way to do this, as it disconnects one from having to depend on other databases in real time.

I'd also consider using content delivery networks for static content. That way, the data coming from the dynamic website is as small as possible. Google does a good job at this with YouTube.

Re:Were known management tools used? (1)

ls671 (1122017) | about 10 months ago | (#45779093)

Google does a good job at this with YouTube.

I am not saying you are wrong, I have seen it being suggested, but it has never flown on governmental projects.

Re:Were known management tools used? (2)

lgw (121541) | about 10 months ago | (#45778865)

Let's talk about security of this website, because it illustrates what's going on here well.

The guy responsible for signing off on the security of Healthcare.gov - signing off on the waiver, mind you, not some actual audit - refused to sign. His boss signed for him and he resigned or was fired.

Last week this repeated - the new guy responsible for signing off on security - presumably the most compliant guy they could possibly find - refused to sign off. Again his boss signed for him, and he's gone.

Does anyone doubt this was happening with all areas of testing? When even management isn't willing to pretend it works, and you launch anyway to meet political goals, you sabotage the entire culture of the Test organization, when you say so clearly "rubber stamp it or get out". If this site ever works, it will be by accident.

Re:Were known management tools used? (1)

Oligonicella (659917) | about 10 months ago | (#45780185)

And it's a criminal accident to the tune of over 600M now. There is no excuse.

Re:Were known management tools used? (1)

Dan667 (564390) | about 10 months ago | (#45779063)

There is lots of integration, but lets be honest. A site you have to use over and over in order to get your insurance ordered is going to suck and I personally would not use it.

Re:Were known management tools used? (1)

phantomfive (622387) | about 10 months ago | (#45780799)

The problem here wasn't one of testing. QA did good tests, and found that the site wouldn't handle load.

Management decided to go live with the site anyway. The were hoping that at least some people would be able to register.

worst job ever... (0)

Anonymous Coward | about 10 months ago | (#45778355)

I cannot imagine a worse job than to have worked on that project.. The ratio of "status update" meetings and management pud-pulls to useful work accomplished must have been damn close to infinity..

Re:worst job ever... (3, Funny)

ColdWetDog (752185) | about 10 months ago | (#45778435)

I cannot imagine a worse job than to have worked on that project.. The ratio of "status update" meetings and management pud-pulls to useful work accomplished must have been damn close to infinity..

You haven't worked on my project.

Re:worst job ever... (0)

Anonymous Coward | about 10 months ago | (#45779487)

I cannot imagine a worse job than to have worked on that project.. The ratio of "status update" meetings and management pud-pulls to useful work accomplished must have been damn close to infinity..

You haven't worked on my project.

Nope, my project has to be worse.

Give the developers a break! (0)

Anonymous Coward | about 10 months ago | (#45778371)

There are over 2.8 million words of Obamacare regulations! [netrightdaily.com]

I challenge anyone to create a website that conforms to such a huge number of rules -- some of them probably contradictory!

Conservatives constantly point out how excessive regulation makes doing business difficult. Well it makes things difficult on the government, too. Let's be fair.

Re:Give the developers a break! (0)

ColdWetDog (752185) | about 10 months ago | (#45778445)

There are over 2.8 million words of Obamacare regulations! [netrightdaily.com]

I challenge anyone to create a website that conforms to such a huge number of rules -- some of them probably contradictory!

Conservatives constantly point out how excessive regulation makes doing business difficult. Well it makes things difficult on the government, too. Let's be fair.

10 PRINT "This is screwed up, it won't work"
20 GOTO 10

Not relevant to the web site (3, Interesting)

Mr2001 (90979) | about 10 months ago | (#45778511)

Most of the Affordable Care Act has nothing to do with the web site. The site didn't have to implement those "2.8 million words of Obamacare regulations" as code: it only had to match patients up with insurance plans, which means interacting with dozens (hundreds?) of government and industry databases.

Some states, like California, managed to implement their sites without any of the problems of the federal exchange. The federal exchange mainly suffered from (1) being rushed, and (2) having to deal with a larger number of external systems than any single state exchange.

anonymous (0)

Anonymous Coward | about 10 months ago | (#45778875)

The state exchanges also were planned solely by and for an individual state. Originally, the federal website was supposed to be used by 20 or so states. It's now being used by 36. The original scope of the project may well have worked for half as many states. It seems like it was never properly scaled and instead, a solution designed to accommodate a certain number of users was shoehorned to accommodate double that initial number.

Re:Not relevant to the web site (1)

CrimsonAvenger (580665) | about 10 months ago | (#45779435)

it only had to match patients up with insurance plans

It also had to determine whether you were eligible for subsidies, and if so, how much. Which was a non-trivial exercise, though nowhere near the whole ACA-worth of regulations.

Re:Give the developers a break! (2)

jedidiah (1196) | about 10 months ago | (#45778615)

> I challenge anyone to create a website that conforms to such a huge number of rules -- some of them probably contradictory!

Sounds like any other regulatory burden. Are the things at Amazon FCC approved? Are they UL listed? Do they pass muster by the USDA?

All of that stuff is outside the scope of the website and it should be the case for Obamacare too.

Your kind of thinking is why it was such a disaster and why 3 guys managed to throw together a window shopping frontend with little effort.

Re:Give the developers a break! (2)

Oligonicella (659917) | about 10 months ago | (#45778723)

Conservatives constantly point out how excessive regulation makes doing business difficult. Well it makes things difficult on the government, too. Let's be fair.

Isn't that rather like saying "Ladies and gentlemen of the jury, have compassion on my client. By killing his parents he became an orphan."?

Re: Give the developers a break! (0)

Anonymous Coward | about 10 months ago | (#45778915)

Not once you realize the government is not a monolithic entity, but a diversity of interests.

To put it another way, Ra's al-Ghul killed Bruce Wayne's parents by ordering Joe Chill into that alley.

Nothing to see here. (1, Redundant)

clickclickdrone (964164) | about 10 months ago | (#45778433)

This is just yet another big government project gone awry. We get these in the UK all the time. I seriously doubt anyone is talking about the testing of this particular project though. Those involved in testing will just keep doing what they do, good ones doing it properly, bad ones doing it, well badly. The other 99% of the population will just bitch about the site as being generally crap but they won't be saying 'They really should have done more integration and load testing'

Re:Nothing to see here. (0, Flamebait)

Jawnn (445279) | about 10 months ago | (#45778687)

This is just yet another big government project gone awry.

Jeezuz, another by-gawd, all-gubbamint-is-bad, Rand fan boy...
Straw man much?

Your assertion might have merit, if the actual development of healthcare.gov was a "government project". The fact of the matter is that it's construction and deployment were the responsibility of firms operating in the private sector, where everything is, to hear you fan boys tell it, rosy and oh-so-efficient. And yes, there are a lot of folks talking about testing. Did you even read TFA?

Re:Nothing to see here. (1)

r0ball (1848426) | about 10 months ago | (#45778975)

I don't think he meant a 'big-government project'; just a 'big project for the government'. There have been some very embarrassing very big IT project failures for the UK government in recent years - the NHS records system [bbc.co.uk] for one, as well as a few DWP and CSA projects [theguardian.com] . These have resulted in £bn write-offs.

I don't know whether in each case the project goals were just too ambitious, or whether the projects were incompetently planned or managed (probably all of the above), but there seems to be systematic failure when it comes to large IT projects procured by government bodies in the UK and elsewhere.

I'd like to know if there are there any examples, worldwide, of large (say >$400m) government IT projects that are completed and have been widely judged a success. If so, what made them different?

Nothing to C here (1)

rmdingler (1955220) | about 10 months ago | (#45778813)

The average citizen may or may not be proficient in a given area of expertise, and perhaps a few gifted ones in several, but Ernest's ability to turn on a light in a room probably begins at 'flip the on switch' and ends at 'reset the breaker'. Coding, system administration, the quality control thereof are no more relevant to your average facebook poster than car mechanics to your mean 16 year old with his head under the hood.

The little awareness they glean of the craft from this incident will just as likely be used to their detriment as their advantage. quote quote...something about a little bit of information...

"Average American was given nightly tutorials on (4, Funny)

JoeyRox (2711699) | about 10 months ago | (#45778461)

load testing and performance bottlenecks"

That's great but how about we teach the average American how to spot Europe on a map first.

Re:"Average American was given nightly tutorials o (1)

Anonymous Coward | about 10 months ago | (#45778553)

Why? Is Europe's location somehow significant to average Americans?

Re:"Average American was given nightly tutorials o (0)

Anonymous Coward | about 10 months ago | (#45778673)

Americans theoretically are supposed to approve of their overseas combat operations, so it might be better to have them identify Iraq and Afghanistan, but the percentage of positive responses would be so depressingly low, finding Europe would be a reasonable consolation prize for the poll-takers.

Re:"Average American was given nightly tutorials o (3, Insightful)

blackpaw (240313) | about 10 months ago | (#45778681)

Why? Is Europe's location somehow significant to average Americans?

And there is the proof of the OP's implied statement

Re:"Average American was given nightly tutorials o (0)

Anonymous Coward | about 10 months ago | (#45778827)

I don't get it. Failing to understand why JoeyRox assumes Europe's location is signifcant to average Americans is very differnt from being ignorant of its location.
What point are you trying to make?

Re:"Average American was given nightly tutorials o (1)

LordNacho (1909280) | about 10 months ago | (#45778979)

Yes. The average American is of recent European descent, speaks a European language, and lives under a system derived from European thought history.

Re:"Average American was given nightly tutorials o (0)

Anonymous Coward | about 10 months ago | (#45779545)

Because otherwise they look like idiots to th rest of the world

Re:"Average American was given nightly tutorials o (1)

OzPeter (195038) | about 10 months ago | (#45778701)

That's great but how about we teach the average American how to spot Europe on a map first.

I think finding Europe is the least of their problems: The Chaser: War on Everything - Americans [youtube.com]

(Yes I know it was probably all in the editing .. still .. you've got to be elfin joking)

Re:"Average American was given nightly tutorials o (2, Insightful)

Oligonicella (659917) | about 10 months ago | (#45778737)

A Swiss friend of mine visited DC. I live in So Mo. We chatted on the phone and he suggested we get together for lunch the next day. So, I agreed and told him the city in Virgina we could meet in after driving eleven hours.

They're no better with our geo than we are with theirs.

Re:"Average American was given nightly tutorials o (1)

ShoulderOfOrion (646118) | about 10 months ago | (#45780531)

It happens occasionally here in the Southwest U.S. that European tourists forget they can drive for eleven hours and not see anyone, or anything. Including a petrol station.

Re:"Average American was given nightly tutorials o (1)

PolygamousRanchKid (1290638) | about 10 months ago | (#45779365)

That's great but how about we teach the average American how to spot Europe on a map first.

That's great but how about we teach the average American how to spot the USA on a map first.

Bad design, not just bad implementation (2)

Bengie (1121981) | about 10 months ago | (#45778539)

Performance and scaling should have been addressed in the design phase

It's called "Design for Test" (2)

yayoubetcha (893774) | about 10 months ago | (#45778547)

Software get's so much grief about bugs and testing. Do you know how many times I had to send my Intel processor back for a replacement because of bugs? Seems like we're getting notifications every six to twelve weeks for that damn processor exchange again. Makes me so mad!

Luckily processors, chip-sets, and mother-boards must be a lot easier than software.... right? Perhaps there's something else going on in the hardware world?

No please done (0)

Anonymous Coward | about 10 months ago | (#45778549)

Give the QA guys a video camera! This charade won't last!

What a shit article (1, Interesting)

Mashiki (184564) | about 10 months ago | (#45778563)

No the project leads gave plenty of time for testing, development, and even kept the WH up to date on what was happening. What happened though was the Obama administration pushed through something that wasn't ready, and wouldn't be ready for long past it's actual inception date. And this of course is because the administration sat on it's backside for an extended period of time, then waved their hands and said a couple of years should be more than enough.

The committee meetings are chock full of very useful information on this, lots of waffle, but surprising bits in the waffle itself. And most of it revolves around, but we..and..they said...followed by...we were going to do it anyway, but it's not our fault we pushed it out early.

The underlying problem... (0)

mi (197448) | about 10 months ago | (#45778567)

The problem underlying the entire fiasco — and the less-impacting others like it (Amtrak [usatoday.com] , anyone?) — is that whatever the government does, is done poorly .

"Not bad for a government job," — is part of vernacular, yet, a curious mix of well-meaning idealists and self-serving demagogues manage to convince the public to try again every once in a while...

Re:The underlying problem... (3, Interesting)

DeathToBill (601486) | about 10 months ago | (#45778715)

Yes, yes, because every bridge the government builds falls down three or four times a day in the first couple of weeks after it's opened.

Re:The underlying problem... (1)

mi (197448) | about 10 months ago | (#45780687)

Yes, yes, because every bridge the government builds falls down three or four times a day in the first couple of weeks after it's opened.

I did not say, it does not get done at all — I said, it is done poorly. The government-managed highways and bridges suck — next time you are stuck in traffic, you'll be forced to agree with me....

Re:The underlying problem... (2, Interesting)

nbauman (624611) | about 10 months ago | (#45778815)

The problem underlying the entire fiasco — and the less-impacting others like it (Amtrak [usatoday.com] , anyone?) — is that whatever the government does, is done poorly .

I realize that that's a right-wing meme, and it's rare for conservatives to change their minds based on the facts, but it's not true.

The military and Veterans Affairs medical centers give some of the best care in the world. I've read the studies that compare them to other centers around the world. They've got the data.

Ronald Reagan got his colon and prostate surgery at Walter Reed. Watch what they do, not what they say.

If you got a head injury in Iraq, you'd have the best chance in the world of surviving with as much of your brain left as possible in the military health system. Ditto with saving a leg or an arm.

The National Institutes of Health is the biggest medical research center in the world. They've done more important research, and won more Nobel prizes, than the entire U.S. pharmaceutical industry put together.

I leave it to Gordon Crovitz to explain how the U.S. government created the Internet.

NASA put the first man on the moon.

Does the invasion of Normandy count?

Re:The underlying problem... (1)

mi (197448) | about 10 months ago | (#45780735)

You — and others — seem to have misconstrued my argument to mean, the government simply can not do anything. That's not, what I said. They can do it — just poorly.

it's rare for conservatives to change their minds based on the facts

Is not it a little early in the conversation for ad hominems?

The military and Veterans Affairs medical centers give some of the best care in the world

Citation needed? [usatoday.com]

Ronald Reagan got his colon and prostate surgery at Walter Reed. Watch what they do, not what they say.

A person of Ronald Reagan's station will get the very best care available in any country and under any regime.

They've done more important research, and won more Nobel prizes, than the entire U.S. pharmaceutical industry put together.

Several considerations destroy this argument too:

  • Who knows, how much more the same people would have achieved — having spent the same amounts of money — if they worked for competing corporations?
  • Could the pharmaceutical industry be more interested in actual cures, than in abstract research?
  • Could the UN committees be a tad biased towards non-profit researchers?

Everybody, who works for government grants hates the process with passion. Various writers (including stars like Asimov) mocked the it viciously.

U.S. government created the Internet.

Are you honestly not aware of the numerous problems with this wonder — where everything is spoofable and nothing is encrypted? Where all sorts of data travels in clear text and security considerations are still — decades later — being bolted on?

NASA put the first man on the moon.

Yes, they did. Poorly... Billions of dollars to take 3 men there and back — with a handful of rocks... Where are the lunar settlements? Where are the retirement homes for the elderly to enjoy the improved quality of life in lower gravity? The shuttle-program — after consuming the untold more billions of dollars — is scuttled, we are using Russian vehicles to bring stuff up. And the Russian are government made too — just cheaper, because everything is cheaper in a poor country.

Heinlein argued before a congressional committee in favor of funding space-travel and related research. But in his books it was the entrepreneurs — motivated by both profit and passion — who did the exploration. If those billions were allowed to stay in the private hands — rather than be taxed away — could there have been a Luna Hilton up there by now?

Does the invasion of Normandy count?

Military organizations, by the very nature of the domain, are not easily subject to competitive markets — government's monopoly must exist there. And yet... The security organizations like Black Water (currently known as Academi) have shown, how much better a privately organized force can be — for far less money — than an official military.

Government sucks at everything — an efficient government is a dictatorship said John Kennedy. Some things can not be done outside of government. But whatever can, should...

Best to keep silent and be presumed smart than (0)

Anonymous Coward | about 10 months ago | (#45781047)

...to do what you just did.

It's a left-wing meme to claim that conservatives refuse to change their minds based on facts, but the FACT is that you have presented no "facts" that would cause a rational person to change his views. Obamacare is just the latest proof that both libertarians AND conservatives are right about big government..... even when run by people who LOVE big government (and who we are told are the smartest and best-qualified to run it), the federal government can spend obscene amounts of money trying to do something the private sector regularly does and totally fail

VA Medical care is AWFUL compared to what's available in the private sector. Sure, they MAY be better than the socialized healthcare in some other countries, BUT the VA system is a mess compared to the private sector. Presidents and former Presidents get healthcare via the military system ("commander in chief" ring a bell?) but that has nothing to do with quality; they'd get the best of what's available no matter what system they are in. The fact that the military medical folks deal best with INITIAL care of battlefield injuries is simply because they face those injuries more than civilian docs at your local hospital. Please refrain from ranting about the wonders of military medical care unless you have endured it. None of these healthcare-related points you tried to make show that the federal government does things better than anybody else.

Yes, NASA managed to put a man on the moon..... there were some bright people running the place in the 60's who contracted with companies like North American Aviation, Grumman, Chrysler, MIT, etc and these non-government corporations actually designed and built the systems. NASA ran the thing and managed/coordinated the effort, but did NOT do all the work. Look at the old pictures and films of those times and notice all the technicians with lab coats with various corporate logos; read the documents and see all the back-and-forth between the government managers and the corporate implementers - Obama SHOULD have done that before attempting "Obamacare", he might have learned a few valuable lessons. NASA and the contractors made many missteps on the way to the moon landing success including miscommunication and poor oversight which lead to the Apollo 1 fire and dead astronauts. NASA overcame this by spending lots of money and changing the way they and their many contractors worked and interacted. The huge levels of obscene spending required to succeed were not sustainable, so only a dozen men walked on the moon before the program was terminated. Ever noticed how many post-Apollo NASA projects have been started, spent lots of money, and then been cancelled without every flying??? Ever noticed how many military projects get started in the US and then go way over budget and under-deliver on what was promised?

The thing the moon landing had in common with the Normandy invasion is this: The federal government can indeed achieve an occasional great thing by throwing huge numbers of people and massive piles of cash at a problem until it's "solved"; when money is no object, even the most incompetent and clumbsy outfit can achieve a goal BUT very few projects get enough public support for those levels of spending and when they DO the support only lasts a few years. Conservatives "get" this which is why they support military spending generally while complaining about government waste and inefficiency (liberals always misunderstand/mis-portray this as hypocrisy but it's just realism). We want as little government (which is wasteful and inefficient) as possible, but recognize that the military has a job that cannot reasonably be done in any other way (no, mercenaries are generally not a good idea), so the waste and inefficiency of the pentagon is a necessary evil (though most of us would like to see FAR greater oversight of it). When there is a viable alternative to govt, it should be used.

Re:The underlying problem... (1)

Anonymous Coward | about 10 months ago | (#45779681)

You don't know much about Amtrak, do you? Amtrak is the result of rail companies no longer wanting to deal with the passenger business they had left after the rise of the automobile. So they got together and convinced the government to take over that side of things, while leaving them free to engage in the business they wanted, freight.

Which they do very nicely.

But passengers? They don't want the bother, they don't want the trouble, they'd rather leave it to highways (run by the government, in case you didn't know) or the airlines (and fuck you if you've never noticed how poorly that deregulated shit is doing), even the government doesn't want the job, except in select areas where it works very well.

And you know what's a common part of the vernacular? The false notion that the well-meaning idealist and self-serving demagogues who propound the free market aren't full of as much crap as can be, but they manage to dazzle the public with their own empty rhetoric.

Re:The underlying problem... (1)

mi (197448) | about 10 months ago | (#45780761)

Amtrak is the result of rail companies no longer wanting to deal with the passenger business they had left after the rise of the automobile. So they got together and convinced the government to take over that side of things

Cool story, bro. I could've asked for some references, but it does not change a thing. The point was — and remains — that Amtrak sucks. The link I gave earlier [usatoday.com] discusses — as an example — the cost of a can of soda... Despite selling it to passengers for $2, Amtrak loses money, because their own cost is $3.40 per can... Nobody can explain this — not only can you buy the cans in the supermarket for $.30-.50, even privately-run vending machines on each station sell the same cans for less than $2. Their proprietors would've been ecstatic to supply the passing trains with as much as they could take... But no, for some reason, Amtrak's costs are $3.40 — would not you love to be their supplier? Somebody is...

And you know what's a common part of the vernacular? The false notion that the well-meaning idealist and self-serving demagogues...

Darling, whatever you are trying to say here is decidedly not part of vernacular...

Been a problem for decades (2)

bfwebster (90513) | about 10 months ago | (#45778727)

SQA as a red-headed stepchild has been an issue for many, many years. It's just that most troubles/failed software systems don't have the widespread public exposure that Healthcare.gov has; even the most brain-dead corporation would not have launched such an incomplete and bug-ridden system to a vast end-user bases.

Some years ago, I led a review of a late (4 yrs vs 2 yrs estimated) and very over-budget ($500M vs. $180M estimated) corporate software project. The core problems had everything to do with SQA, starting with the fact that there was no SQA organization; all testing was done on an ad hoc basis by individual teams and organizations. Adding to that problem was the fact that there was no coherent architecture. After 4 years and $500M, there were no systems that were ready to go into production. Far too common in industry and especially in government. ..bruce..

Funny thing: testing is not that important... (3, Insightful)

gweihir (88907) | about 10 months ago | (#45778791)

If you find out in testing that your architecture or design does not cut it, you are screwed. The only thing you can usually do is scrap the project and start again. Testing does only work for simple things like simple busiess logic and the like, where you know the characteristics very well beforehand. For anything that is a new design, the only thing that helps is very capable and experienced architects and designers that have a good change to get it right by intuition. This will be people that can do architecture, design and implementation and can do all three well. Not many of those exist, but there is no replacement for them. Those that think they can do things on the cheap without not only having this type of expert but also listening to them closely will fail. This can be observerd time and again and can alost be called a "well established industrial practice", because quite a few "managers" do not actually know that it can be done better. Funny thing, in other fields, you have chief enineers, architects and the like and the critical work is not given to people that are likely to fail. Only IT messess it up regularly, because talent and exerence is not respected.

Re:Funny thing: testing is not that important... (1)

saleenS281 (859657) | about 10 months ago | (#45779781)

Since when? Every large software vendor I've ever worked with breaks their product up into modules and do standalone module testing and once those tests are passed add it to the main code line and do additional integration testing from there. Why on earth would you not test a software project of this size until it was completed in it's entirety? While there may be modules that you can't go live without, there absolutely shouldn't be a requirement to start from the ground up because one module failed testing... That's just bad software architecture. Healthcare.gov should be a prime example of a system that can be modularized. Hell, half the codebase is likely code to talk to external API's for the various government and private institutions they're sharing data with.

Why weren't Google or Amazon asked to do it? (1)

presidenteloco (659168) | about 10 months ago | (#45778989)

You've got to ask yourself, who's building and deploying reliable, performant, extremely scaled apps these days? Who has been doing that successfully for over a decade? Why don't we ask them to build our big app? Or if they're busy, ask them who they would recommend.

It's usually the schedule, in my experience (1)

presidenteloco (659168) | about 10 months ago | (#45779055)

Ok, so some projects, as has been pointed out, are doomed from the very first bad architectural decision (or lack of architectural decision.)

But regardless of that factor, the most common thing I've seen is management/corporate promising a particular release date, in a contract, say, and eventually getting around to telling development/engineering, who say, if they're brave, um, that's not possible. If they are less brave, they smile and get on with faking it, all the while knowing it's impossible in the time given. If they're highly skilled and properly whipped, they'll get something that looks superficially ok out the door on schedule, but don't ever, ever, try to use it.

sex with A Shit (-1)

Anonymous Coward | about 10 months ago | (#45779255)

Maybe now Testers will have budget. (1)

Ducho_CWB (900642) | about 10 months ago | (#45779421)

I'm a developer that has been working as a tester in the last 5 years.
Always coding to run performance / load / whatever not manual testing.

In my experience working at bank or big publishers, my main problem is always budget.
Enough money for everything except the right number of people testing, the right tools or infrastructure for testing, and so on.

I don't expect that was different with healthcare.gov.

Re:Maybe now Testers will have budget. (0)

Anonymous Coward | about 10 months ago | (#45779829)

I haven't worked on huge Google style stuff, but in my experience, systems have always performed exactly as designed.

Re:Maybe now Testers will have budget. (1)

Ducho_CWB (900642) | about 10 months ago | (#45780197)

Except for one project that I destroyed with performance testing, leading to a huge re-factoring, the others always get delayed to correct all serious bugs.

But it was, as always, management decision. Postpone or release with bugs?

Re:Maybe now Testers will have budget. (1)

theGhostPony (1631407) | about 10 months ago | (#45780769)

A-freakin-men to that.

Suffering from that very issue as we speak with a major software update due for rollout by the end of Feb. Argh.

At this point what difference does it make (1)

John Jorsett (171560) | about 10 months ago | (#45779427)

By all accounts, there was some testing done, however inadequate. Thing is, the system utterly failed that testing and THEY DEPLOYED IT ANYWAY. If you're going to ignore the testing, why even do it? Just throw open the doors and hope for the best. Which is what they did, apparently.

Look who asked for it.. (0)

Anonymous Coward | about 10 months ago | (#45779537)

Democrats wanted it, so it was doomed to fail since nothing they do work. Obama was told it was not ready, but he ignored it. You can't really blame the programers or testers, there simply was not enough time to fully develop the website. So it comes down to Obama to be blamed for the website problems.

But it could have been Obama's plan for it to fail at the start so he can delay a lot of Obamacare till after midterms. That plan was not very successful though, since people are starting to see how horrible Obamacare is, and that Obama knowingly lied about keeping their healthcare, doctor and STILL have to pay more then their old plan.

The only defense I hear from democrat voters are "The healthy should have to pay unhealthy's medical bills, because its unfair that you are healthy."

Re:Look who asked for it.. (1)

ppanon (16583) | about 10 months ago | (#45779921)

Then they are people who understand insurance as poorly as you appear to. The healthy pay the unhealthy's medical bills because a) the risk and cost is shared by all premium payers when they are healthy (including those who get sick later) b) the unhealthy don't need the added stress of getting huge bills to impact their attempts to get better and usually can't work while they're in traction or a coma.

You know, sort of like how in fire or flood insurance the people whose houses are standing help pay for replace the houses that got burnt down/flooded/washed away (but whose owners had paid insurance premiums). That's how insurance works!

suddenly? (0)

Anonymous Coward | about 10 months ago | (#45779709)

hah, I've been doing software testing for 25 years. :p

hipster newspeak (0)

Anonymous Coward | about 10 months ago | (#45779887)

"changed the conversation"
"owns the decision"

What kind of meaningless hipster newspeak is this?

If hipster horseshit is representative of the methods used to build Healthcare.gov then no one should be surprised that it failed.

"The Word" (0)

Anonymous Coward | about 10 months ago | (#45779913)

http://dictionary.reference.com/browse/snafu+principle

In the beginning was the plan, and then the specification; And the plan was without form, and the specification was void.
And darkness was on the faces of the implementors thereof; And they spake unto their leader, saying: "It is a crock of shit, and smells as of a sewer." ...

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?