Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

New Virginia IT Systems Lack Network Backup

timothy posted more than 4 years ago | from the private-did-not-make-perfect dept.

Communications 211

1sockchuck writes "Virginia's new state IT system is experiencing downtime in key services because of a mind-boggling oversight: the state apparently neglected to require network backup in a 10-year, $2.3 billion outsourcing deal with Northrop Grumman. The issue is causing serious downtime for state services. This fall the Virginia DMV has suffered 12 system outages spanning a total of more than 100 hours, and downtime hampered the state transportation department when a state of emergency was declared during the Nov. 11 Northeaster."

Sorry! There are no comments related to the filter you selected.

Blame Northrop? (3, Insightful)

betterunixthanunix (980855) | more than 4 years ago | (#30212696)

In my experience, it is rare for a customer, even with professional IT staff, to properly specify their needs when it comes to technology. Why did Northrop, which presumably has experience in government systems, not design backups?

Re:Blame Northrop? (5, Insightful)

eht (8912) | more than 4 years ago | (#30212732)

Likely they were told they should have a backup, quoted a price, and said nah, we will be fine.

Re:Blame Northrop? (2, Insightful)

Daniel_Staal (609844) | more than 4 years ago | (#30213068)

That'd be my guess. Second guess would be that they agreed to having a backup - as soon as some politician determined where the backup site would be. (Which, of course, hasn't happened yet.)

Re:Blame Northrop? (5, Informative)

DRBivens (148931) | more than 4 years ago | (#30213258)

... as soon as some politician determined where the backup site would be. (Which, of course, hasn't happened yet.)

Actually, it has happened. The CoVA backup site is located in Lebanon, VA (SW part of the state).

What THIS article is discussing is the lack of network backup, not data backup.

This is an important distinction, to say the least.

Re:Blame Northrop? (1)

vegiVamp (518171) | more than 4 years ago | (#30213724)

So they're lacking a *backup network*, not a network backup ?

Re:Blame Northrop? (1)

LoSt180 (1481103) | more than 4 years ago | (#30213808)

You're correct, this article is referring to network REDUNDANCY for connections to offices. The word "backup" in the title is a bit misleading as most will immediately think of data backups. My guess is the T1's into an office are from the same provider instead of two as a fall-back.

Re:Blame Northrop? (1)

TheLink (130905) | more than 4 years ago | (#30213852)

Yeah, and the problem might be overblown (and the reason why it is overblown could be politics or part of "haggling").

100 hours in five weeks might be not much depending on how many branches there are. Same goes for the 4,677 hours in 6 months figure.

The branches may just have ADSL links to the HQ.

As for redundant network links. In many cases it's worthless paying the extra. It costs a lot to do it right.

1) The branch may only have one telco/carrier choice for connectivity (at a reasonable price)
2) Even if they have two choices, both could go through the same line/chunk of ground that the backhoe digs up. Nowadays with all the outsourcing and inter company deals, you might buy redundant links from two different companies, and later find out the hard way that they are all in the same cable!

Many years ago my ISP had two redundant international links (they were small then). But they lost connectivity even, so people asked "what happened". And they said - a trawler broke one line, and apparently not long later, another trawler (or the same trawler :) ) broke the second line[1]... Doh.

And I wonder how many paid for redundant transpacific fibre optic links that all went over the Taiwan earthquake zone ;).

[1] That's what they said anyway... For all we know, maybe it was the NSA doing a bad job of tapping the lines with their subs :).

Re:Blame Northrop? (1)

atamido (1020905) | more than 4 years ago | (#30213952)

2) Even if they have two choices, both could go through the same line/chunk of ground that the backhoe digs up. Nowadays with all the outsourcing and inter company deals, you might buy redundant links from two different companies, and later find out the hard way that they are all in the same cable!

For the small city that I work for, we could get AT&T ADSL, Time Warner Cable, or Time Warner Fiber. They all terminate into the same decades old AT&T building. It is certainly possible that Time Warner has their own fiber running between that AT&T building and the rest of the world, but I'd bet money that they just lease space on whatever data trunk AT&T pulled out there.

One misplaced backhoe is going to take out data and phone for the entire city, no matter who their provider is.

Re:Blame Northrop? (4, Insightful)

skgrey (1412883) | more than 4 years ago | (#30212758)

And not just backups, it sounds like they had no BCP plan at all. This is a massive oversight, but a fairly common one. I've consulted for a number of years, and it's amazing at how many companies don't have a BCP plan at all, and sometimes it includes simple backups of data.

The companies where I've seen this basically do a risk assessment and say "well, we are willing to accept the risk of downtime because BCP is too costly". Unfortunately they don't weigh the chance of an outage or disaster appropriately, and then find themselves severely screwed when a tornado, storm system, or fire occurs, and then they are either out of business (in a small company) or take enough of a hit to make a headline on Slashdot and cripple the business.

Seriously, when are companies going to realize that this is a critical component of IT? I've felt like I've talked till I was blue in the face about this over the years.

Re:Blame Northrop? (3, Interesting)

Anonymous Coward | more than 4 years ago | (#30212864)

You are correct. There should be SLAs, as well. The problem that most people don’t seem to comprehend. NG’s contract is NOT with the Stateits with VITA (yea a state agency..). NG does have SLA’s with VITA however most state agencies didn’t even SEE those SLA’s until oh I don’t know the last 3 months? VITA on the other hand, whom state agencies are REQUIRED to use by state law, has NO SLA’s..no MOU (memorandum of Understanding) nothing with the other state agencies. We are FORCED to use VITAand they have no staff, and they know that if and when it fails, NG will be blamed.

NG has failed and will continue to fail not because “agencies want us to fail” but because they came in with no clear understanding and NO desire to understand state governments business or needs. We are NOT a private company, we are NOT driven by profit. We are stymied by laws and procedures to protect the tax payer that in the end cost more money.

State workers just want to get this thing working. WE KNOW how its suppose to workwe’ve done it. We’ve done it many years without budget money, without staff, without support.

This VITA/NG “parnership” is a complete fiasco for those of us that live it every day. This is just the tip of the iceburg.

Re:Blame Northrop? (0)

Anonymous Coward | more than 4 years ago | (#30213006)

Where I work, we have a BCP... but it's a total fiasco. The county has spent nearly a million dollars, and for that we have a site hosted by another county a fraction of our size. We have a single DS3 running up there over which backup and test traffic runs. The network was designed to be a mirror of our current network, down to the IP addresses, while still accessible mid-day to the agencies down here. It's a NAT nightmare.

Oh, and the inbound link from the Internet to be used in case of disaster in the actual County is a DS3, and plans call for up to 4000 users to be able to access e-mail, remote desktop, and on occasion thick client via SSL VPN over that. In recent weeks, they've decided that they also need to run VoIP for several hundred users over it, too.

I'd rather have no BCP than this embarrassment.

Re:Blame Northrop? (0, Offtopic)

skgrey (1412883) | more than 4 years ago | (#30213034)

I absolutely love the fact that my post got moderated "overrated". Apparently we have some management types on Slashdot these day that got some mod points. It's a perfect example of how some people feel about BCP.

And uh, aren't "overrated" mod points for things that have been modded up a whole bunch that you don't agree with?

Re:Blame Northrop? (1)

WinterSolstice (223271) | more than 4 years ago | (#30213394)

Agreed - I've worked a number of places where a system is flagged as "Mission Critical!!!omgzwtfbbq" and they have no trouble waking you up to make you fix it, but if you suggest an HA/failover?

Sorry, too expensive. We have weighed the risk, and decided it's an acceptable risk.

Queue the freak-outs when something actually *does* break :)

Re:Blame Northrop? (3, Insightful)

TheLink (130905) | more than 4 years ago | (#30213548)

> they have no trouble waking you up to make you fix it, but if you suggest an HA/failover?
> Sorry, too expensive. We have weighed the risk, and decided it's an acceptable risk.

Yes because they can count on waking you up to fix it.

So seems perhaps the bosses are doing the right thing for the organization. They hired you, you will wake up to fix it, and they don't need to spend on HA/failover.

Now if they hired someone who can't fix it fast, or sleeps really soundly, then they should spend on HA/failover, or hire you instead ;).

Re:Blame Northrop? (4, Insightful)

mcgrew (92797) | more than 4 years ago | (#30212784)

Why did Northrop, which presumably has experience in government systems, not design backups?

Because they didn't have to. It wasn't in the contract, so they're not going to spend the money doing it. They're not in business to keep the state government afloat, their only purpose is to make money.

If you don't properly specify your needs, that's your fault. Don't rely on corporate good will, because there is no such thing.

Re:Blame Northrop? (0)

Anonymous Coward | more than 4 years ago | (#30213120)

Don't attribute to malice that which can be adequately explained by stupidity.

Odds are that Northrop engineers brought this up on multiple occasions, but if the site was designed as an HA environment and utilizes one or more SANs, there's a reasonable chance that management believed that a backup system would be unnecessary, especially given the cost of a proper implementation. I've seen similar things happen where I work as management pays more attention to the budget than to reality.

Re:Blame Northrop? (3, Insightful)

Eivind (15695) | more than 4 years ago | (#30213150)

True enough. But as you say, Northrop is in the business of making money, so it would've made sense for them to do the following:

* Deliver a offer for the system requested.
* Get the deal signed
* Say: We notice you've not specified any backup, do you want that additionally ?

Gives them a chance to upsell, AND potentially makes the customer happier -- a win-win.

Re:Blame Northrop? (1)

confused one (671304) | more than 4 years ago | (#30213300)

It's a tight budget year. Perhaps VA said no, when the additional work was suggested.

Re:Blame Northrop? (1)

dissy (172727) | more than 4 years ago | (#30213432)

so it would've made sense for them to do the following:

* Deliver a offer for the system requested.
* Get the deal signed
* Say: We notice you've not specified any backup, do you want that additionally ?

Who's to say they didn't? ;)

My guess is the disconnect happened in between steps 1 and 2, more precisely when the bean counter saw the price in the offer and mentally could not attach that to any value.

Re:Blame Northrop? (1)

Chris Mattern (191822) | more than 4 years ago | (#30213718)

"Thank you for your suggestion that we spend more money. We'll certainly take it under consideration."

--Later:

"File this somewhere where I'll be sure to never see it again, won't you, Miss Haversham?"

Re:Blame Northrop? (0)

Anonymous Coward | more than 4 years ago | (#30213206)

Sadly, the only benefit to designing a truly appropriate system is that you get to snicker derisively when the system installed by the guys that underbid you goes belly up. Alas, that doesn't pay the bills.

Re:Blame Northrop? (0)

Anonymous Coward | more than 4 years ago | (#30213308)

"They're not in business to keep the state government afloat"

So much for corporate social responsibility, which corporations do claim to have.

Re:Blame Northrop? (0)

Anonymous Coward | more than 4 years ago | (#30213452)

"They're not in business to keep the state government afloat"

So much for corporate social responsibility, which corporations do claim to have.

Ah. But if the Govt. does not wish to have a service or pay for a service. It is NOT the corporations' responsibility to provide the said service for free. Customer (Govt) refused and customer pays consequences.

Re:Blame Northrop? (1, Insightful)

eth1 (94901) | more than 4 years ago | (#30213312)

I don't buy that it's necessarily the government's fault for not specifying backups.

The customer should only have to say "we need a system that does X, it needs to be up Y% of the time, with an MTTR of no more than Z." They don't know, and shouldn't have to specify technical details. It's up to the provider to design a system that does that.

As another poster mentioned, though, it's quite likely that NG came back and said here's a system that will do that, and it will cost X, and the customer got sticker shock and decided to drop a few 9s from the SLA. I'm in that business, and this happens all the time.

Re:Blame Northrop? (4, Insightful)

WinterSolstice (223271) | more than 4 years ago | (#30213418)

You must not deal with the government much :)

If you are bidding for a government contract, it's a public bid. They state their requirements very precisely, and every single dollar you spend over is counted against you.

Basically to do network backup, you'd have to eat it out of the goodness of your heart. There is a potential to upsell later, of course, but it has to go back through the public approvals process.

Re:Blame Northrop? (0)

Anonymous Coward | more than 4 years ago | (#30213870)

It doesn't have to go through the same bidding as the original contract. The experts and Northrup should have realized that redundant carrier circuits were not in place and gotten a mod to their contract to put them in place. If the Commonwealth decided not to do so then shame on them.

Re:Blame Northrop? (0)

Anonymous Coward | more than 4 years ago | (#30213716)

Heh... Management not long ago was demanding five nines uptime for all critical systems, including external-facing websites (government agency communicating with the public). We asked them if they meant in terms of planned or unplanned outages. They said unplanned. We told them it would be expensive. They said to prove it. We gave them a number. They all but told us we were lying. We showed them a case study. They said we were cherry-picking data and hired a consultant, who came back with a bigger number than we did. It's never been mentioned since, and suddenly, three nines is considered nice-to-have.

Re:Blame Northrop? (4, Insightful)

nine-times (778537) | more than 4 years ago | (#30213592)

They're not in business to keep the state government afloat, their only purpose is to make money.

I hate when this is offered as an excuse for shoddy work. "It's not their job to do good work. It's their job to make money." Yeah? So what. It strikes me a little like saying, "Hey, can't blame a con man for stealing your money. That's what con men do!"

I don't know this particular situation well enough to say who is at fault and to what degree, but it's part of their business to service their customers well. It's part of every company's business to provide service to their customers in an ethical manner.

Re:Blame Northrop? (2, Insightful)

Publikwerks (885730) | more than 4 years ago | (#30212808)

It's the state's fault for not putting that in the contract. I have worked for state contractors who handle IT services, and the state always had a downtime penalty written in to the contract, so it was too expensive to be down not to have a redundant system. This is probably a case of penny pushers not doing their homework, seeing that one system is cheaper than two.

Re:Blame Northrop? (1)

John Hasler (414242) | more than 4 years ago | (#30212832)

How do you know that Northrup didn't suggest them? Or that Northrop wasn't the lowest qualified bidder by being the only one not to include backups in their bid? When you buy a computer do you expect the vendor to throw in a free backup system even though you didn't ask for it "because you should have it"?

Sounds to me like the state got what it ordered.

Re:Blame Northrop?Christmas gifts,shoes,handbags (-1, Offtopic)

coolforsale123 (1685254) | more than 4 years ago | (#30212878)

http://www.coolforsale.com/ [coolforsale.com] Dear ladies and gentlemen Hello, In order to meet Christmas, Site launched Christmas spree, welcome new and old customers come to participate in the there are unexpected surprises, look forward to your arrival. Only this site have this treatmentOur goal is "Best quality, Best reputation , Best services". Your satisfaction is our main pursue. You can find the best products from us, meeting your different needs.Ladies and Gentlemen weicome to my coolforsale.com.Here,there are the most fashion products . Pass by but don't miss it.Select your favorite clothing! Welcome to come next time ! Thank you! ugg boot,POLO hoody,Jacket,Air jordan(1-24)shoes $33,Nike shox(R4,NZ,OZ,TL1,TL2,TL3) $35,Handbags(Coach lv fendi d&g) $35,Tshirts (Polo ,ed hardy,lacoste) $16,free shipping Thanks!!! Advance wish you a merry Christmas.

Re:Blame Northrop? (1)

jollyreaper (513215) | more than 4 years ago | (#30213080)

In my experience, it is rare for a customer, even with professional IT staff, to properly specify their needs when it comes to technology. Why did Northrop, which presumably has experience in government systems, not design backups?

Consultants never seem to get it right. Granted, the situations I'm familiar with are a fraction of a fraction of a percent of the size of this contract but at that scale the issue is that the contractors and consultants are juggling multiple clients so the needs of any one must be balanced against the needs of all of them. But what really burns me is when they can't even provide decent advice on what should be bread and butter. "We need a backup solution." See, there you go. Many solutions on the market but they should be able to settle on a package they can confidently advise clients will work. But they don't. This astounds me. The consultants are just shooting in the dark, no better informed than the rest of us. You can't blame them for good advice ignored by the customer but for bad advice or no advice at all? Absolutely!

Re:Blame Northrop? (2, Insightful)

Chris Mattern (191822) | more than 4 years ago | (#30213130)

Welcome to the world of government low-bid contracts. The specification didn't call for backups, so you don't get backups, because that would've made the bid higher.

Re:Blame Northrop? (1)

bill_mcgonigle (4333) | more than 4 years ago | (#30213892)

Welcome to the world of government low-bid contracts. The specification didn't call for backups, so you don't get backups, because that would've made the bid higher.

It would be interesting to know who developed the specification - an intern in State IT or a professional software 'architect'.

This is a particularly vexing problem for local government, without the resources to properly specify, but I'd hope that the Commonwealth of Virginia could do better.

Re:Blame Northrop? (1)

angus_rg (1063280) | more than 4 years ago | (#30213144)

Ask and it shall be given, for an extra 10 mil.

Re:Blame Northrop? (1)

magarity (164372) | more than 4 years ago | (#30213780)

From the article is seems that NG installed and ran the system for a while but only after the state people took over did it experience a lot of problems. I'm curious what is causing these outages before I'll blame either NG for not installing backups or the state for not budgeting them.

That's the way of the future... (4, Funny)

master_p (608214) | more than 4 years ago | (#30212704)

Have you ever seen backup systems in Star Trek, for example? you haven't. The future requires no backups.

Re:That's the way of the future... (2, Insightful)

mikael_j (106439) | more than 4 years ago | (#30212798)

Actually, there are plenty of backup systems in Star Trek. Of course, a few of them fail in every episode to avoid having every episode end with a "Yay for Starfleet engineers!" after five minutes.

In fact, for some systems they apparently have up to four backups which all manage to fail magically at the same time *cough*transporters*cough*.

/Mikael

Re:That's the way of the future... (1)

corsec67 (627446) | more than 4 years ago | (#30213038)

Talking about that, is there a single instance in Star Trek where the "Manual Override" actually worked?

And then, how is it a manual override if you just flip some other switches. The way they use "manual overrides" in Star Trek the bridge should be similar to that of the Tardis.

Re:That's the way of the future... (1)

mcgrew (92797) | more than 4 years ago | (#30212904)

Actually, yes. In one episode of NG they had to purge the system of malware and indeed used their backup. The only time I ever saw a backup where I work was when it was part of my job to change the tapes.

Re:That's the way of the future... (1)

rwv (1636355) | more than 4 years ago | (#30213630)

when it was part of my job to change the tapes.

I had that job once. I hated that part of my job because it was repetitive and boring and the backups failed more than they should have.

Is that meant to be irony? (1)

realxmp (518717) | more than 4 years ago | (#30212906)

Have you ever seen backup systems in Star Trek, for example? you haven't. The future requires no backups.

Either that was a terrible attempt at irony or you really really weren't watching closely enough. The Galaxy class had at least two redundant paths for every key primary system.

Re:Is that meant to be irony? (1)

slaad (589282) | more than 4 years ago | (#30213084)

They had to install double backups for systems in DS9 after the Federation took over as well.

Re:Is that meant to be irony? (1)

theaveng (1243528) | more than 4 years ago | (#30213608)

This is true of virtually all U.S. Naval systems - redundancy. Unfortunately a lot of the primary/secondary wiring runs in parallel, so all it takes is a single bomb to knockout both at the same time. Rather dumb really, but even the U.S. government likes to skimp on costs.

I wonder if the UFP government also made similar stupid cost-cutting measures in the Enterprise and other ships of the line? That would explain why both systems seem to suffer failure at least once a year.

Re:That's the way of the future... (1)

sjames (1099) | more than 4 years ago | (#30213334)

The ideal backup system is invisible. You don't even know when you're using it because it took over seamlessly. An alert comes up in the noc and the primary is repaired behind the scenes.

That doesn't work in the world we live in. Such systems cost a good bit more money and even if funded initially, will go away soon enough due to budget cuts because management says "we don't need backups because we've never had an outage" while the techs know the backups seamlessly took over 3 times in the last year.

Re:That's the way of the future... (1)

TheLink (130905) | more than 4 years ago | (#30213622)

> The ideal backup system is invisible.

That already exists according to the Many-Worlds-Interpretation Quantum Physicists.

> That doesn't work in the world we live in.

That's the big problem though - when "stuff happens" the backup is in other worlds.

Re:That's the way of the future... (1)

dkleinsc (563838) | more than 4 years ago | (#30214068)

Sure it does. What the heck is all the talk about auxiliary and emergency power?

Of course, the real limitation is that any redundant control systems make the operator's console twice as likely to explode.

Easy (5, Insightful)

Spad (470073) | more than 4 years ago | (#30212716)

During the first six months of the year, state Department of Transportation workers faced 101 significant IT outages totaling 4,677 hours: an average of more than 46 hours per outage. One took 360 hours to fix.

That's 27 weeks of downtime in the space of 26 weeks, which raises a much more important question than why there's no network redundancy and that question is: What kind of fucking morons have they got running their systems?

Re:Easy (1)

Clover_Kicker (20761) | more than 4 years ago | (#30212790)

What kind of downtime are we talking? There's a big difference between a 5 hour total outage or 5 hour loss of connectivity to a small town DMV office.

PHB who say it's cheaper to let some out side peop (1)

Joe The Dragon (967727) | more than 4 years ago | (#30212872)

PHB's who say it's cheaper to let some out side people run there IT and likely say no to the higher cost plan that has backup with it.

Re:PHB who say it's cheaper to let some out side p (0)

Anonymous Coward | more than 4 years ago | (#30213764)

You mean outside people as in low-cost Indians on H1-B visa status? Yep, thought so.

Re:Easy (1)

kiehlster (844523) | more than 4 years ago | (#30212890)

'Running' is not the optimal term here. I'd say they have monkeys defecating on their systems.

What makes you think there are any people? (1)

Colin Smith (2679) | more than 4 years ago | (#30213670)

You think you get actual people when you outsource? People cost money, which reduces profitability.

As long as the systems are inside SLA, what's the problem?

 

outsourcing (5, Insightful)

Clover_Kicker (20761) | more than 4 years ago | (#30212730)

But I thought the magic pixie dust of free enterprise would make outsourcing something to the private sector cheaper, more efficient, and better in every possible way?

Re:outsourcing (1)

Bourdain (683477) | more than 4 years ago | (#30212766)

that "magic pixie dust" of free enterprise is only as effective as the competency of said free enterprise and we all know that well-run companies are the exception and not the rule

Re:outsourcing (1)

nanospook (521118) | more than 4 years ago | (#30212886)

You mean the people of Virgina had a slew of technology companies competing for this business slot and had a choice? No, it came down to the state govt, a budget, polititians, and one or two "allowed" choices. If Northrop could make money by providing a backup, I think they would have done it. I think the state agency dropped the ball and said no backup needed. Free competition comes down to a number of candidates, all competing to outdo the other so they can make money. The best way to do that is to do the best job for the best price. That's why something "smells" here..

Re:outsourcing (1)

jimicus (737525) | more than 4 years ago | (#30213088)

The best way to do that is to do the best job for the best price. That's why something "smells" here..

If you'll excuse my saying so, that's an incredibly naive way of seeing things.

IME there are lots of business plans which history has shown can all work very well indeed.

Off the top of my head, there's:

  • Provide a good product at a reasonable price, and try your damndest to have just one slight edge over the competition - maybe cheaper by a very narrow margin, maybe a slightly better product. (This is basically the one you describe, and is the most obvious).
  • Order 1,000 leaflets announcing you are "specialists in Education" and send them to every school you can think of. Don't worry too much about the quality of your product/service or its price.
  • As above, but the leaflets announce that you are "specialists in supplying the public sector" and send them to every government body you can think of.

Re:outsourcing (3, Informative)

idiotnot (302133) | more than 4 years ago | (#30212920)

Hey, it worked. Mark Warner won two-thirds of the vote in his senate run last year based on his stellar performance as governor. This was one of his big initiatives.

(He also *fixed* the revenue sources, so that there'd never be a problem like happened with Jim Gilmore. Yet, now, Virginia is in worse shape than when he got there.)

Re:outsourcing (2, Insightful)

darjen (879890) | more than 4 years ago | (#30213220)

The government is clearly involved here. So it's got nothing to do with free enterprise.

"the government exists (4, Informative)

circletimessquare (444983) | more than 4 years ago | (#30213868)

and has involved itself in the market in some way in the past

therefore, any prudent rational criticism of the free market and how it obviously fails can be explained away with creative rationalization that its the government's fault, somehow"

my favorite is how free market fundamentalists wish to blame the market crash of 2008 on government policies. rather than gee, i dunno, the clinton and bush administration deregulation policies? you know, deregulation: having the government less invovled int he market?

"what? my free market bubble and pop? nah, impossible! government's fault! pffft"

please study your banking panics of the 1800s: without regulation, free markets have innate imperfections which always result in catalcysmic failures. all you need is simple human psychology, no government need apply, to cause a market to crash. you either regulate it, leveling the playing field artificially, and therefore making it truly "free", or you leave it alone, letting it bubble and pop like mad, and allow monopolists to take advantage of natural imperfections in the market to leverage unfair behavior

free market fundamentalism is dead. your ideology is dead. fact: you need government involvement in the market for the market to run efficiently. fact: you need government policing and regulation of the marketplace to keep it "free" and egalitarian and equal for all players

if you don't understand these simple truths by now, or refuse to believe that despite the obvious proof, you're an idiot

Re:outsourcing (1)

Korin43 (881732) | more than 4 years ago | (#30213310)

Yup, that's what the free market is all about: The government paying someone to do something.

They have bigger problems than just this one... (3, Informative)

Cornwallis (1188489) | more than 4 years ago | (#30212752)

Remember how Virginia's health records were compromised earlier this year?

http://it.slashdot.org/article.pl?sid=09/05/05/1232240 [slashdot.org]

Sounds like systemic ineptitude which is why I'm really looking forward to more government involvement in health care!

Re:They have bigger problems than just this one... (3, Funny)

east coast (590680) | more than 4 years ago | (#30213076)

That wasn't a compromise of health records. That was transparency!

Re:They have bigger problems than just this one... (3, Insightful)

mcgrew (92797) | more than 4 years ago | (#30213122)

Bureaucracy is bureaucracy. [itbusiness.ca] Government involvement doesn't mean ineptitude, and the free market doesn't gurantee competence. Whether private or public, ineptitude as well as competence abounds.

Re:They have bigger problems than just this one... (1)

5KVGhost (208137) | more than 4 years ago | (#30213252)

That's often true. But private ineptitude tends to be a self-correcting problem. Businesses that are consistently unresponsive to the market and that do really stupid things will fail. And they should fail. Better businesses spend a great deal of effort to identify their mistakes so that they won't repeat them.

Public bureaucracies, OTOH, are essentially pure monopolies. They entrench themselves and always outlive whatever original purpose they had, they're given the power of law to enforce their decisions, and they have no reason to improve (or even care) because they have no competition. The aggregate incentives are all wrong.

And government owned corporations (like GM is turning into) and government granted monopolies (like cable companies) combine the worst of both worlds. Their real customer is not us, it's the government. We're just an expense. And better business have a much harder time competing against the advantages the favored companies are given.

Re:They have bigger problems than just this one... (1, Insightful)

Anonymous Coward | more than 4 years ago | (#30213322)

But private ineptitude tends to be a self-correcting problem.

And this helps me after my health records (or credit card numbers or whatever) have been leaked... how?

One inept company goes bankrupt, another takes its place, and the damage is already done irregardless.

Re:They have bigger problems than just this one... (1)

Uberbah (647458) | more than 4 years ago | (#30214064)

But private ineptitude tends to be a self-correcting problem.

Yes, as evidenced by the fact that all the bankers and investors that drove our economy into the ground are now delivering pizza and mopping floors. Oh wait, they're getting bigger bonuses than ever.

What you describe works nice in theory, but in practice executives will take enormous risks at the chance of enormous short term profit. Why make a million dollars a year through responsible investment if you can make 30 million dollars in two years? So what if the business goes belly up because of it - you made 30 million dollars!

Re:They have bigger problems than just this one... (1)

rwv (1636355) | more than 4 years ago | (#30213686)

Whether private or public, ineptitude as well as competence abounds.

It seems like in Virginia... ineptitude abounds.

I live in Massachusetts... competence is fairly prevalent here. Going to the RMV (our version of the DMV) still sucks, but unless you're getting your license for the first time you can circumvent that hellish trip and process your registration forms through a website. :)

Re:They have bigger problems than just this one... (1)

Jawn98685 (687784) | more than 4 years ago | (#30213320)

Remember how Virginia's health records were compromised earlier this year?

http://it.slashdot.org/article.pl?sid=09/05/05/1232240 [slashdot.org]

Sounds like systemic ineptitude which is why I'm really looking forward to more government involvement in health care!

Yeah, right. Because the private sector has done such a good job of protecting our privacy, banking info, etc.
Please..., go troll somewhere else.

Re:They have bigger problems than just this one... (0)

Anonymous Coward | more than 4 years ago | (#30213366)

That's why we should outsource our healthcare and our IT to the private sector!

Re:They have bigger problems than just this one... (1)

ffflala (793437) | more than 4 years ago | (#30213386)

Sounds like systemic ineptitude which is why I'm really looking forward to having health care!

FTFY

Re:They have bigger problems than just this one... (0)

Anonymous Coward | more than 4 years ago | (#30213450)

I'm looking forward to paying the insurance premiums of grossly obese programmers who sit at the keyboard all day eating twinkies.

NG, I call you out! (4, Interesting)

Anonymous Coward | more than 4 years ago | (#30212756)

Most forget that the network provided by the NG crooks is NOT part of the Comprehensive Infrastructure Agreement (CIA). It is a seperate agreement that is a fixed cost agreement under which NG was supposed to replace “like for like”. They were supposed to install an MPLS network. MPLS (Multi Protocol Label Switching) allows for the prioritization of traffic to allow Voice traffic to travel over the same circuit as the data. It also supposed to be intelligent enough to encrypt data to essentially allow a VPN to be created from point-to-point.

None of the VPN has been done as promised, very few sites have used the VOIP option unless dictated to by VITA as part of new construction and most sites complain about network performance. Some agencies had totally redundant networks but were forced to pay more for less. 65% of VITA staff make over 90,000 a year. Again we pay more for less.

While I am not a NG fan, interestingly enough, most state managers at Agencies will tell you that working directly with NG allows things to get done, VITA just gets in the way. VITA wants to always be the interface, Waste Fraud and Abuse to pay high salaries for mostly unqualified folks. Throw out VITA and let the agencies be treated like customers by NG.

The IT Community Frowns Upon Your Shenanigans...

Sounds like you're a VA state IT employee... (0)

Anonymous Coward | more than 4 years ago | (#30213834)

Hope you used an anonymizing proxy to post that. Nobody dares announce that the Emperor is indeed stark raving naked, lest the proclaimer get his head chopped off.

Re:NG, I call you out! (0)

Anonymous Coward | more than 4 years ago | (#30214014)

This may be a little biased, because I do work for a contractor to a government agency.

I agree with your point. When people come directly to us, we can usually provide them a pretty straightforward answer, and within a couple of days things are usually done. However, our policies dictate that we are not supposed to respond to such things, instead sending them to either management or to a project manager. The head of project management has (I am told) said that her staff does not need any technical knowledge -- that is why we have engineers. Either one will provide an answer that, while correct at some level, actually says nothing, but is interpreted by those who go to them as something useful. Anything that requires a certain number of hours is then deemed to be a formal project, requiring assignment of a project manager and customer liaison (which is factored into the initial time, thereby frequently bumping up the hours to the magic number). This is the simplest part of the revenue policies running around here and put in place by management, all of which are government employees.

When government is able to come up with mechanisms designed to increase billing hours that make consultants and contractors shy away from the evil, that says something.

Sometimes upper management rips reqs from RFQs (1)

chrispix (624431) | more than 4 years ago | (#30212824)

Working for a local government, it does not matter what the IT department recommends or requires in a project. Sometimes upper management just don't get it, and cut things out of projects to save a buck. One example I can think of was hand holds. At a few hundred dollars a pop, we wanted hand holds every so many yards (can't recall number several years later), and also require them at turns > than 45 degrees. That way if we needed to branch off, we could do it at a turn (usually at a street intersection). When the project got approved somewhere up the chain, locate wires, and the hand holds at turns got eliminated. When we added a new building on line, we ended up having to pay additional cost for a single hand hold, and putting fiber across a street... A couple months later, the fiber was cut b/c there was no locate wire to locate the cable that had not been reported to dig test, because it would cost too much according to engineering. Stinks that that kind of oversight for a few dollars costs the tax payers in the end. Things like backups, locate wires etc are called insurance, maybe us IT Folk should start calling it that... Insurance.

Northrop Grumman outsources part of it's own IT as (2, Insightful)

Joe The Dragon (967727) | more than 4 years ago | (#30212838)

Northrop Grumman outsources part of it's own IT as well and it does not own some of it's systems they rent them or at least they did 1-2 years ago.

Wait a second... (0)

Anonymous Coward | more than 4 years ago | (#30212926)

i thought government programs never fail. /s

Va/VITA/NG == Navy/NMCI/EDS (0)

Anonymous Coward | more than 4 years ago | (#30212996)

This is the same fiasco that the Navy went through with EDS and NMCI [pbs.org] . Does no one at the state IT level read trade mags? This crap was all over the IT news magazines when the EDS contract was blowing up.

sorry, dont mean to troll (0)

Anonymous Coward | more than 4 years ago | (#30213048)

But how stupid can these people be to not include backups? i dont see this as a matter of improperly specifying their needs. I see this as a matter of stupidity. thats all.

again, sorry for the troll post, but this is ridiculous.

Network connections, not system backups... (5, Informative)

tomhath (637240) | more than 4 years ago | (#30213110)

Is seems nobody RTFA (no surprise). The problem they're having is network outages at branch offices. I assume they're using DSL or such, with no way to connect if/when it goes down. Any one office probably has >99% up time, but when you have hundreds of offices and the remnants of a hurricane come through you can expect several of them to go offline, which is what's happening.

Re:Network connections, not system backups... (0)

Anonymous Coward | more than 4 years ago | (#30213762)

That doesn't sound nearly as bad as TFS. Thanks for RTFA for us!

Re:Network connections, not system backups... (1)

bill_mcgonigle (4333) | more than 4 years ago | (#30213948)

s/Backup/Redundancy/ and half the comments go away - vague headlines must be Slashdot's new evil plan to get page views.

Network redundancy not backups (4, Insightful)

zerofoo (262795) | more than 4 years ago | (#30213124)

The article does not mention "backups" as in tape drives and off-site storage.

The article does mention lack of redundancy at the network carrier level.

My guess is that Northrop Grumman designed a network around single circuits connecting offices to data centers, and did not design the network to tolerate WAN link failures.

A stupid oversight for sure, but nothing that can't be easily remedied by ordering redundant WAN circuits from your telco of choice. Redundant routing gear would also be smart.

For all that are blaming government for this - they outsourced the design and implementation to a private company. That company screwed the pooch in design and implementation. Shame on both parties for not recognizing the risk of WAN failure.

-ted

Backups are worthless, restores are priceless... (0, Offtopic)

adamsteinhoff (185899) | more than 4 years ago | (#30213146)

I've been in the business of running networks for many years. I generally find that, as the company gets smaller, the amount of redundancy and unknown problems goes up. I expect every small business that I walk into to have no redundancy and backup systems that work 10% of the time. However, the fact that someone as large as NG and the government overlooked this critical part of system design is amazing to me.

I always explain to my small business clients that 'backups are worthless, restores are priceless.' It generally takes a few seconds to set in before the questions or statements start coming. "But, I have a backup scheduled every night. I take the tapes offsite every Friday."

It's always an eye-opener when I show them the console on their server with lots of red errors indicating that the data on the tapes are incomplete at best. Then, I start talking about the fact that if their server were to die, not only would it take a day or so to get parts for that old thing, but that the likelihood of being able to get it running at 100% again is slim or very costly at best.

Even small businesses will invest in a backup/business continuity device after hearing that and reading our blog that contains 'my company saved the day for a client yesterday'.

Shameless plug: Backups are worthless, restores are priceless [dedicatedit.com] blog.

Epic Fail (2, Interesting)

halfEvilTech (1171369) | more than 4 years ago | (#30213236)

If any story deserves this tag it is this. from the article:

"Virginia declared a state of emergency Nov. 11 in the face of record nor'easter rains and winds.

But without backup circuits -- which VDOT had before the Northrop Grumman outsourcing -- to take up the load, the transportation agency's Hampton Roads' IT network went out of service 23 times during the event.

"We called at 5:35 in the morning," said Gary Allen, VDOT's chief of technology, research and innovation.

"It took VITA four hours to open the help ticket" and begin to solve the problem."

4 hrs on a critical system seriously just to get started solving it?

Re:Epic Fail (1)

confused one (671304) | more than 4 years ago | (#30213390)

Didn't you hear? There was a storm man... It was cold, windy and raining oceans of water. We were told to stay home. The tech was probably in his bed in the dark (no electricity) trying to ignore the ringing cell phone.

Funny math or multiple systems? (3, Insightful)

Cprossu (736997) | more than 4 years ago | (#30213244)

"During the first six months of the year, state Department of Transportation workers faced 101 significant IT outages totaling 4,677 hours: an average of more than 46 hours per outage. One took 360 hours to fix."

wait, 4,677 hours? how could that be? There were 181 days in the first 6 months of this year, that's only 4,344 hours.. there was more downtime on the system than days in it's operational life! (did someone /0 here?)

Outsourced, no thanks... I think I'd rather dig up a Univac I to do work on, at least it would be more reliable

Re:Funny math or multiple systems? (2, Informative)

halfEvilTech (1171369) | more than 4 years ago | (#30213292)

My guess is that would be multiple systems. They noted in TFA that they provided IT services to 1000 local governments and 85 state agencies in VA.

THIS explains all the closed rest stops in VA... (2, Interesting)

volxdragon (1297215) | more than 4 years ago | (#30213286)

I *knew* there had to be some other reason why they closed half the interstate rest stops in VA, this is obviously where the money was (mis)spent...

Re:THIS explains all the closed rest stops in VA.. (1)

halfEvilTech (1171369) | more than 4 years ago | (#30213358)

naw the system was down which told the rest stops to lock their doors. The should be open again now, assuming it hasn't crashed again... well fudge

15 day outage??? (3, Funny)

tinkertim (918832) | more than 4 years ago | (#30213348)

From TFA:

During the first six months of the year, state Department of Transportation workers faced 101 significant IT outages totaling 4,677 hours: an average of more than 46 hours per outage. One took 360 hours to fix.

Suddenly, I don't feel so bad for that 2 1/2 hour glitch last week :)

Sue Northrop... (0)

Anonymous Coward | more than 4 years ago | (#30213470)

... for every loss in productivity they are responsible for. This "mistake" is sheer incompetence.

Shocker! Gov't contractor lied! (0)

Anonymous Coward | more than 4 years ago | (#30213510)

From TFA "The 10-year, $2.3 billion project aims to modernize 85 state government agencies' computer networks, PCs, phones, servers and e-mail systems, while holding down costs."

So basically they gave 2.3 billion to a giant gov't contractor because they said they could do it better and cheaper. News flash ... they lied. The cheaper comes from eliminating anything you didn't specify as absolutely essential, like network backups. So far it looks like the only thing they do better is cash your checks. The state will now scramble around trying to add in some redundant network connections, starting with their high priority targets. Anyone want to bet that when they finally get everything stabilized the new outsourced system that was supposed to be "holding down costs" will end up costing significantly more than anyone imagined. That is how gov't contractors work. They put in a low ball bid, and make sure that anything that is not explicitly defined in the contract is billed at rates that make up the difference. Your fault VA.

More bias from the Richmond Time Dispatch (1)

Casharelle (746564) | more than 4 years ago | (#30213702)

The Richmond Times Dispatch really seems to hate the Northrop Grumman / VITA project. Actually, from my experience a lot of folks down in Richmond hate it and tend to bad mouth it endlessly.

There is a lot of bad sentiment here in VA as this consolidation/outsourcing of IT cost quite a few State jobs which obviously upset a lot of people. Of course, they forget the fact that Northrop Grumman is one of the State's biggest employers and that they were hiring people to work at the new location. I still hear of lot of anger on radio stations and in the news when I'm in that region.

Better Terminology (1)

Atomm (945911) | more than 4 years ago | (#30213820)

The OP would make a heck of a lot more sense if they used better terminology. Rather than Network Backup, call it Network Redundancy or High Availability. When people hear backup, they think storing data to tape.

Staffed by 2nd rate people? (1)

cryfreedomlove (929828) | more than 4 years ago | (#30213836)

Northrop Grumman's core business is making airplanes (at least it was). So what we have here is a non-core business effort on behalf of a state government contract. I'll bet it was staffed by the B team at Northrup Grumman because real IT hot shots just are not motivated to get out of bed in the morning to chase state government contracts. On top of this staffing issues, I'm sure the government had lots of non-standard 'requirements' from insecure bureaucrats that need to justify their jobs. This is a lethal combination of doom for this project. The solution is standard requirements shared by lots of similar customers that is chased after by multiple competing vendors but I suspect the politics of patronage in Virginia would never let that happen.

Re:Staffed by 2nd rate people? (1)

Casharelle (746564) | more than 4 years ago | (#30213982)

Northrop Grumman's core business is making airplanes (at least it was). So what we have here is a non-core business effort on behalf of a state government contract. I'll bet it was staffed by the B team at Northrup Grumman because real IT hot shots just are not motivated to get out of bed in the morning to chase state government contracts. On top of this staffing issues, I'm sure the government had lots of non-standard 'requirements' from insecure bureaucrats that need to justify their jobs. This is a lethal combination of doom for this project. The solution is standard requirements shared by lots of similar customers that is chased after by multiple competing vendors but I suspect the politics of patronage in Virginia would never let that happen.

Northrop Grumman has an entire arm of its business devoted to Information Systems that makes about $10 billion in revenue a year. They aren't exactly a small time operation when it comes to IT.

And I thought.... (0)

Anonymous Coward | more than 4 years ago | (#30213848)

.... getting a drivers license in Texas was bad!

Job opening in Virginia (0)

iamhigh (1252742) | more than 4 years ago | (#30214056)

But the site is down, so you can't apply.

(I'm trying to start a new meme, here)
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?