Beta

Slashdot: News for Nerds

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Northrop Grumman Says 'I'm Sorry' For Virginia IT Outage

Soulskill posted more than 3 years ago | from the wipe-the-tears-away-with-benjamins dept.

Networking 168

Lucas123 writes "After a storage area network in a data center run by Northrop Grumman went down last week, crippling 26 state agencies' websites — some for more than a week — Northrop Grumman has now apologized to Virginia, saying it will learn from its mistakes in order to recover systems faster in the future. Northrop's $2.6 billion service contract with Virginia's government has come under harsh criticism in the past for service outages, along with project delays and cost overruns."

cancel ×

168 comments

Apt Futurama quote (5, Insightful)

dkleinsc (563838) | more than 3 years ago | (#33465052)

Hermes: What do we do when we break somebody's window?
Dwight: Pay for it?
Hermes: Heavens, no! We apologize! With nice, cheap words.

Re:Apt Futurama quote (1)

operagost (62405) | more than 3 years ago | (#33466044)

Should VA sue Northrop Grumman, then? Or simply hold them to their promise of improving their infrastructure and response (both of these cost money)?

Re:Apt Futurama quote (1)

hondo77 (324058) | more than 3 years ago | (#33466086)

Should VA sue Northrop Grumman, then?

VA shouldn't have to sue NG. NG should do the right thing. It won't, of course.

Re:Apt Futurama quote (1)

Paspanique (1704404) | more than 3 years ago | (#33466130)

The only "Do the right thing" share holders know is making the profit and lower the cost. If you can avoid settling stuff like that, do it. Think BP will pay a dime if it sees it can avoid it?

Re:Apt Futurama quote (2, Insightful)

Amouth (879122) | more than 3 years ago | (#33466336)

Think BP will pay a dime if it sees it can avoid it?

Yes - and they already have. Under law when it happened they are only liable for 75m they have already paid out a hell of a lot more than than that.

Why? because it would cost them more not too. If BP ran with the law at the time and said too bad soo sorry we are only liable for 75m, the media and the politicians would have gone ramped with it in their own name (elect/pay me i'll fight the evil doers BP for you).. instead BP did the correct thing and said out front before ANYONE mentioned making them pay - that they would pay for it. That they would foot the bill, and so far they have.

Just because they can get out of paying for something doesn't make it wise to do it.. as for NG.. i doubt the VA contract is a large portion of the biz and isn't a core piece - so yea.. they are going to use cheap words.

Re:Apt Futurama quote (1)

Red Flayer (890720) | more than 3 years ago | (#33466402)

instead BP did the correct thing and said out front before ANYONE mentioned making them pay - that they would pay for it. That they would foot the bill, and so far they have.

Except now the line is, "Approve permits for our off-shore drilling, or we won't pay."

Re:Apt Futurama quote (1)

Amouth (879122) | more than 3 years ago | (#33466648)

and where have they been correctly quoted saying this?

Poor metropolitan area (2, Interesting)

Pojut (1027544) | more than 3 years ago | (#33465094)

The Maryland/VA/DC metro area is really starting to go down hill, from an infrastructure standpoint. Things are just falling apart around here...oh, and what's that? Instead of investing in fixing aging infrastructure, they instead are spending billions to build the ICC? [iccproject.com] Oh, and what's that? It's STILL going to be a toll road?

I've lived in Montgomery County my whole life, but I'm quickly getting tired of this place -_-;;

Re:Poor metropolitan area (1)

Shimmer (3036) | more than 3 years ago | (#33465256)

Another Montgomery County resident here, and I second this view. We've had an insane number of power outages, water main breaks, crippling snowstorms with unplowed roads, etc. in the last few years.

I love where I live, though. Who needs electricity?

Re:Poor metropolitan area (2, Insightful)

Wiarumas (919682) | more than 3 years ago | (#33465710)

http://www.thesentinel.com/mont/Pepco-Investigation2010-08-12T10-54-42

The governor of MD wrote a letter to Pepco regarding those power outages. As for the snowstorms - its because the DC area historically does not receive snowfall to justify a ROI on snowplows. Instead, they borrow them from the north. Its not like in PA, where if it blizzards overnight, the streets are clear by 6AM so the kids can go to school.

Re:Poor metropolitan area (1)

Shimmer (3036) | more than 3 years ago | (#33465836)

Yes, and the power outages are caused by overhead (rather than underground) power lines that are severed when one of our many large, beautiful trees fall on them. However, I think that if you compare this area to others with similar climate/foliage, we still fall short. The large state roads around here were not plowed for DAYS following the Snowpocalypse this past winter. (And then when they were finally plowed, the plowing was done poorly and left many roads with huge ruts and sudden narrowings.) I've lived in this area for most of my life (since the 1960s), and it's worse now than it ever was before.

Re:Poor metropolitan area (2, Insightful)

Anonymous Coward | more than 3 years ago | (#33466114)

http://www.thesentinel.com/mont/Pepco-Investigation2010-08-12T10-54-42

The governor of MD wrote a letter to Pepco regarding those power outages. As for the snowstorms - its because the DC area historically does not receive snowfall to justify a ROI on snowplows. Instead, they borrow them from the north. Its not like in PA, where if it blizzards overnight, the streets are clear by 6AM so the kids can go to school.

And the plowing was also poorly prioritized and executed. I live in Alexandria. My residential street was clear to pavement with repeated passes of salt and sand, but Duke Street and Route 1 were an unplowed rutted mess. All because I have a member of the General Assembly living in my neighborhood (that's at least what I've been told by those in the know).

Re:Poor metropolitan area (-1, Flamebait)

Anonymous Coward | more than 3 years ago | (#33465900)

We've had an insane number of power outages, water main breaks, crippling snowstorms with unplowed roads, etc

Only Republicans politicians in federal government can both cut taxes and issue debt to pay for ordinary day-to-day services without a political cost, as their voters are stupid and easily fooled.

Re:Poor metropolitan area (1, Informative)

Anonymous Coward | more than 3 years ago | (#33465264)

What does the infrastructure of highways have to do with with the State Governments reliance on a single storage area network?

Re:Poor metropolitan area (1)

Pojut (1027544) | more than 3 years ago | (#33465336)

I used the ICC as an example because it's the biggest one...but there are HUGE problems with all aspects of infrastructure here, from electric to water and sewer, to traffic signals to just about anything else you can name.

These systems being down caused widespread problems, including people being unable to use the DMV for driver's licence services. It's just one more ingredient in the shit sandwich, that's all.

Re:Poor metropolitan area (1)

Pojut (1027544) | more than 3 years ago | (#33465470)

In addition to my other post, it's also relevant because the ICC is a multi-billion dollar highway project that is wholly unecessary. Those billions could be used in much better ways...like, for example, fixing/upgrading the IT infrastructure that is clearly breaking down rather than simply maintaining it.

Re:Poor metropolitan area (0)

Anonymous Coward | more than 3 years ago | (#33465984)

They are both examples of infrastructure paid for by the government of Virginia which have gone downhill lately.

See GP post for more info, and watch out for those infinite loops. ;)

Re:Poor metropolitan area (1)

El Torico (732160) | more than 3 years ago | (#33466172)

The ICC is in Maryland.

Re:Poor metropolitan area (0)

Anonymous Coward | more than 3 years ago | (#33466262)

Yes, but worse yet is the handing out of services like candy. Look at the budget for Montgomery County Schools. All this money spent but quality is sliding.

$2.6 billion service contract? (1)

Futurepower(R) (558542) | more than 3 years ago | (#33465328)

"Northrop's $2.6 billion service contract with Virginia's government..."

What could they possibly be doing for Virginia that should cost $2.6 billion?

Re:$2.6 billion service contract? (4, Informative)

Pojut (1027544) | more than 3 years ago | (#33465376)

Here you go, direct from our local news radio station. [wtop.com]

"Northrop Grumman holds a $2.4 billion, 10-year contract with the Virginia Information Technologies Agency to build, operate and maintain the state's 7-year-old, problem-plagued consolidated computer services bureaucracy. It is the largest single-vendor contract in Virginia history. The partnership has been repeatedly criticized in JLARC studies for poor and tardy delivery of services, cost overruns and system failures."

These systems are directly integrated into the DMV, as well as the Department of Social Services and Department of Taxation, amongst others.

Re:$2.6 billion service contract? (0)

Anonymous Coward | more than 3 years ago | (#33465556)

Read "The Shock Doctrine: The Rise of Disaster Capitalism" by Naomi Klein for an insight into how these contracts get created as part of the effort to "down size" inefficient government agencies. Amazon link at -- http://www.amazon.com/Shock-Doctrine-Rise-Disaster-Capitalism/dp/0312427999/ref=sr_1_1?ie=UTF8&s=books&qid=1283529678&sr=8-1 ...

Re:$2.6 billion service contract? (1)

Pojut (1027544) | more than 3 years ago | (#33465688)

Interesting...thanks for the tip!

Re:$2.6 billion service contract? (1)

itlurksbeneath (952654) | more than 3 years ago | (#33465828)

Still, what could they possibly be doing for that amount of money. If you work it out, that's almost 30 thousand dollars every hour for 10 years (I didn't even take out weekends or holidays or anything). That's ~890,000 per month. I could run one bangin' IT organization on 240 million dollars a year. Hell, the company I work for now (which is in the top 10 on the Fortune 500 list), our yearly IT budget is smaller than that.

Re:$2.6 billion service contract? (1)

Lunix Nutcase (1092239) | more than 3 years ago | (#33465868)

Still, what could they possibly be doing for that amount of money.

Lining their pockets. What else?

Re:$2.6 billion service contract? (1)

Pojut (1027544) | more than 3 years ago | (#33465942)

Didn't say VA was getting their money's worth, just wanted to tell people what they were doing :-)

Re:$2.6 billion service contract? (1, Insightful)

Anonymous Coward | more than 3 years ago | (#33466392)

Clearly you don't understand how the government works. It's a circle.
Congress gives companies huge contracts -> Companies make mountains of profit -> Companies donate to congressman's compaign -> Congress gives companies huge contracts

Re:$2.6 billion service contract? (1)

nacturation (646836) | more than 3 years ago | (#33466438)

Still, what could they possibly be doing for that amount of money. If you work it out, that's almost 30 thousand dollars every hour for 10 years (I didn't even take out weekends or holidays or anything). That's ~890,000 per month.

That's about $20 million/month ($240 million per year / 12 months/year).

Re:$2.6 billion service contract? (0)

Anonymous Coward | more than 3 years ago | (#33465996)

> "...poor and tardy delivery of services, cost overruns and system failures."
>
> These systems are directly integrated into the DMV, as well as the Department of Social Services and Department of Taxation, amongst others.

Weren't poor/tardy delivery of services, cost overruns, and system failures always integrated into those departments?

Re:$2.6 billion service contract? (5, Informative)

Anonymous Coward | more than 3 years ago | (#33466028)

This kind of thing seems to be a growing trend in government IT. I'm posting anonymously because while I don't think I'm going to say anything that violates an NDA, it's better to be vague and sure.

The county that I live in recently made a move like this: they fired basically all of their IT staff and replaced them with the lowest bidding consulting company.

The upside of that from a certain fiscal standpoint is that they've eliminated a bunch of positions with pensions and good (and therefore expensive) benefits. These people have been replaced, in some departments, by the exact same people now subcontracting through the consulting company. This isn't really cheaper -- they won't have additional pension obligations to those people, but they're drawing much higher salaries than before, and obviously the consulting company gets a sizeable cut too.

In other departments, all the long-term employees have been replaced by new consultants. This is a problem in that, probably the people who had those jobs should have documented their networks and tasks much better, but the fact is, they didn't. The whole memory of those departments has been flushed. Inexperienced people are now trying to figure out how to maintain processes that literally no one who works there knows anything about. It's a disaster and it's going to get a lot worse before it gets better.

Meanwhile, the county executive is running for governor, touting the above as a great accomplishment. Hey, he shrunk the size of government by eliminating many permanent positions! By the time people realize that, not always, but sometimes, a lifer IT person is worth their pay because of the institutional memory they have of a thousand important things that were never documented, the election will be over.

Re:$2.6 billion service contract? (4, Insightful)

AB3A (192265) | more than 3 years ago | (#33466298)

Mod parent way up. I've seen it in other places too.

Often when you ask people in the accounting department, they'll say they don't really know what's going on because IT made the system. When you talk to IT, they'll say they don't know because the system was specified by Accounting. The truth is that some smart guys in each department got together and forged some sort of system together. Then the smart guys went on to bigger and better things, while the peons were left with some special voodoo system.

Now you want to move said voodoo system over to a consulting company. The assumption is that the Accountants know every detail of what the old system did. Well, they don't. But nobody is willing to step forward and say that. So the new consultants come along and gosh, nobody knows what the systems do.

Then people ponder why it "doesn't work." Sigh.

This is how shit happens.

Re:Poor metropolitan area (1)

arkane1234 (457605) | more than 3 years ago | (#33466692)

I lived in Carroll County, moved to Alexandria (VA) and realized things are never going to get better so I left to Arizona.
Never felt better about life.

An apology? (0)

Anonymous Coward | more than 3 years ago | (#33465102)

Oh no, their contract specifies that they get paid one million dollars for each time they say they are sorry!

State governments not as forgiving as the DoD? (1)

FatSean (18753) | more than 3 years ago | (#33465122)

Pretty weak apology if you ask me. I guess these military contractors are used to the "boys will be boys" pat on the hand they get from the Department of Defense when they screw up there.

My Project (4, Informative)

WED Fan (911325) | more than 3 years ago | (#33465124)

I have a project in a separate NG hosted dataspace, in Virginia. They are killing us with incompetence and their sub contractors are worse then they are. We are still trying to get things certified and they won't provide us information about their hosting. We think they have us on virtual servers that belong to another project, and the reason they don't want to tell us anything is that it would reveal they are in breach, since we are paying for dedicated servers.

Re:My Project (0)

Anonymous Coward | more than 3 years ago | (#33465178)

Sounds like GoDaddy.

Re:My Project (1)

Manip (656104) | more than 3 years ago | (#33465542)

GoDaddy sell both. Are you sure you know what you're paying for?

Re:My Project (0)

Anonymous Coward | more than 3 years ago | (#33465320)

You're giving NG way too much credit there. I doubt they're capable, institutionally, of that sort of duplicity; they have been in almost every way, at the corporate and division level, absolutely incompetent.

Re:My Project (3, Informative)

snspdaarf (1314399) | more than 3 years ago | (#33465332)

I do some work with NG sub contractors. One is a delight to work with. Working with the other is like staring at the sun through binoculars. People don't answer email, sometimes because they are lazy, sometimes because they quit. When an answer does come in, it is worse than none at all. It's like the worst troll postings on slashdot, full of errors, run-on sentences that confuse more than clarify, and off topic. Absolute nightmare.

Re:My Project (3, Insightful)

Monchanger (637670) | more than 3 years ago | (#33465930)

As much as Northrup is being bashed here, I don't think this problem is specific to NG, but common to many large contractors and their subs.

In the few times I've worked with subcontractors doing IT for the government I've been unimpressed. Even being one step away from the prime contract seems to allow for many problems, both technical and managerial. Requirements and deadlines aren't met, and they pull the BP-Halliburton-Transocean trick of avoiding responsibility by blaming each other (as well as everyone's favorite scapegoat: the government). Trying to get a subcontractor to build things they way they were supposed to will often require waiting for the next spiral, which means going way over budget.

I understand the difficulty with pushing too hard, punishing contractors who screw up and scaring them away from government work, but it seems we've gone too far in accepting very expensive third-rate work. As much as the public likes to say government can't do anything right, how much worse off would we be than $2.4B in the hole with nothing to show for it but a mediocre datacenter run by amateurs? I don't think I'm asking for much, I haven't touched on the very messy political poisoning of contracting.

Re:My Project (4, Informative)

Cylix (55374) | more than 3 years ago | (#33465452)

Unless they have gone to some very lengthy steps to hide this you can probably discover the information on your own.

http://www.dmo.ca/blog/detecting-virtualization-on-linux/ [www.dmo.ca]

This page details steps for many different types of virtualization environments. Though I think it would be just as fast to sort through the output of dmidecode and look for an identification in the mess.

I'm afraid this is rather linux centric, but even so similar data sets can be collected on windows.

Re:My Project (1)

Anonymous Coward | more than 3 years ago | (#33466380)

... similar data sets can be collected on windows.

Device Manager > Display adapters. Look for "VMware SVGA II".

*running away from flying tomatoes*

Re:My Project (0)

Anonymous Coward | more than 3 years ago | (#33466550)

I can't agree more. NG seems willfully incompetent because they make more money with their cost plus contracts that way. The more time they take, and the more inefficient resources they throw at the problem, the more they get paid. I've worked for the top three defense contractors and NG takes the cake for this kind of behavior. I heard more employees there use the phrase "I'm not going to jail for doing this", or "one more time and I'm going to call the Defense Hotline [dodig.mil] ". While the people of NG are some of the hardest working, honest and dedicated people I've ever met, the (mis)management at NG takes the cake for sheer incompetence, as well as overworking and underpaying their employees.

To be fair... (2, Insightful)

betterunixthanunix (980855) | more than 3 years ago | (#33465144)

To be fair, there is no evidence that Northrop is doing worse than anyone else would have done. We are talking about an enormously complex IT infrastructure here (or so I assume, since it is a government network), and this is not exactly a uniquely bad failure. A week may seem extreme, but I have seen smaller scale systems go down for that long.

I am not an apologist for Northrop, I am just saying that this is not exactly one-of-a-kind incompetence.

Re:To be fair... (3, Insightful)

bsDaemon (87307) | more than 3 years ago | (#33465170)

Meanwhile, the last week has marked the first time where there was really a valid excuse for apparently unmoving lines at Virginia DMV branches... glad I don't have to get my license renewed until 2017. They should be back up by then.

Re:To be fair... (1)

Mekkah (1651935) | more than 3 years ago | (#33465210)

Thank you for inserting logic. I used to work for NG, and I am certainly not defending them, but lets not just brand them as bad because of the size of their.. contract. We all know morons who work for every company..

Re:To be fair... (2, Insightful)

rotide (1015173) | more than 3 years ago | (#33465222)

Frankly, I've been witness to whole Enterprise datacenters going offline and then being brought up system by system in less than a week (candy red button). A week to fix a SAN issue? really? Why not classify it as a Sev0 (public exposure) (probably a somewhat unique code to the company I work for) and get the vendors in THAT DAY to fix it?

Re:To be fair... (1)

nacturation (646836) | more than 3 years ago | (#33466528)

Frankly, I've been witness to whole Enterprise datacenters going offline and then being brought up system by system in less than a week (candy red button). A week to fix a SAN issue? really? Why not classify it as a Sev0 (public exposure) (probably a somewhat unique code to the company I work for) and get the vendors in THAT DAY to fix it?

Depends on how much storage was in the SAN and the nature of the failure. If it was as easy as replacing a memory card then your point it completely valid. If, however, there was some data corruption then there might have been terabytes of data that needed to be verified and recovered. The article states that 26 state agencies were affected, "some for more than a week" which implies a variable amount of time. Without having more detail, it could have been that some were back up and running the same day but others had to wait for data recovery to occur.

Re:To be fair... (1)

Cylix (55374) | more than 3 years ago | (#33465506)

I think the problem is there is more then one complain against the company.

Still, 2.6 billion I could build way past four nines of availability.

I currently maintain better availability on a budget that is way smaller and the equipment would not be considered inexpensive. In fact, a failure on that scale would blow any availability numbers I have out of the water for years. Short of data corruption that works its way into the system a single SAN failure should not halt operations for a week.

Now, instead of using said dollars to build out proper infrastructure and relying on what I like to call "hope." Well, then these problems will eventually crop up because equipment does fail.

Re:To be fair... (2, Insightful)

Spazmania (174582) | more than 3 years ago | (#33465608)

Which is why you don't try to implement this broad an IT contract. Was a damnfool idea at the start.

Re:To be fair... (0)

Anonymous Coward | more than 3 years ago | (#33465852)

To be fair, there is no evidence that Northrop is doing worse than anyone else would have done.

Ahh, a variant of the conservative's favorite excuse of "But Bill Clinton did it too!" or the liberal's "But Dubya did it too!!". Sorry, but incompetence of this scale is bad no matter how bad everyone else may be at the same time.

Re:To be fair... (2, Informative)

Lunix Nutcase (1092239) | more than 3 years ago | (#33466104)

Really? This is hardly a great track record:

During the first six months of 2009, Virginia's Department of Transportation (VDOT) experienced 101 significant IT outages totaling 4,677 hours: an average of more than 46 hours per outage. One outage, the Times-Dispatch said, took 360 hours to correct. The state's Department of Motor Vehicles (DMV) has experienced over the course of 5 weeks this autumn some 12 outages that put individual DMV offices out of business for a total of more than 100 hours the paper says.

From here [ieee.org] . I'm sorry, but there is no excuse for that.

IT Bubble Syndrome (3, Insightful)

darien.train (1752510) | more than 3 years ago | (#33465154)

The "old business man discovering the internet" IT bubble culture is still alive and well in the defense industry. They have such a bad track record with networking technology it borders on scary. Transformation [pbs.org] comes to mind quickly and they keep repeating the same mistakes.

Re:IT Bubble Syndrome (1, Informative)

Anonymous Coward | more than 3 years ago | (#33465272)

The problem with DI contractors is that the government rewards good sales teams more than good engineering teams. It means that a disproportionate amount of overhead must be factored in to a winning bid, and since winning bids are often on the lower cost side, it means the engineering teams are always underfunded for the project.

I was involved with NG on a project once.... (4, Interesting)

quangdog (1002624) | more than 3 years ago | (#33465226)

Fresh out of school with my CS degree I went to work on a project for my employer that involved partnering and working directly with folks from NG. The original deadline for shipping the solution was something like 6 months after I started. The complete and utter incompetence of the NG side of things wound up stretching this out more than 18 MONTHS longer, and the final delivery lacked a lot of the original stated requirements.

Being the newbie to the whole corporate culture, I was shocked that people were not bothered at all by blown deadlines, missed estimates, and huge cost overruns. Shortly after the project finally delivered, I bailed to work for a much smaller company (fewer than 10 employees) where I discovered that I really love the smaller, more dynamic environment that only small companies can provide.

Working for huge corporations just sucks.

Re:I was involved with NG on a project once.... (4, Interesting)

Nadaka (224565) | more than 3 years ago | (#33465428)

I work for a smaller DoD related company (17k employees, so not that small really) and I hear nothing but horror stories about NG, and have experienced one of them myself. NG originally developed our project but was not allowed to bid on the re-compete where we picked it up. It was bad, but we are fixing it. I really can't say much more than that at the moment.

Re:I was involved with NG on a project once.... (0)

Anonymous Coward | more than 3 years ago | (#33465496)

Going to have to keep this ac for obvious reasons, but having experience working there I can say that deadlines are set with the expectation to fail. No it is not as blunt as saying we don't care about a deadline, but I would have to agree with you in referencing a smaller companies: they have the advantage that the "upper management" (or in this case team leader) knows what the hell is going on. Big companies are needed however in the long run, hell a 10 person, 5,000, even 20,000 staff is too small to run their aircraft carrier sector. My opinion is as follows:
"Deadlines are like a box of chocolates, you never know what the hell your going to get." - or something like that haha

cheers

Re:I was involved with NG on a project once.... (1)

IndustrialComplex (975015) | more than 3 years ago | (#33465500)

Working for huge corporations just sucks.

I used to work for Lockheed Martin. Now I work for a company with 50 employees.

Talk about a head rush of culture shock.

Re:I was involved with NG on a project once.... (1)

sconeu (64226) | more than 3 years ago | (#33465666)

I know. I used to work for Litton, and left 1 month before the NG purchase became final.

Went to work for a small house doing Fibre Channel... Seriously major cognitive dissonance.

Re:I was involved with NG on a project once.... (1)

tareko (1002904) | more than 3 years ago | (#33465774)

In the short term, wasn't an 18 month project that ran way over budget and over time more profitable than one that would have been on time and on spec?

Re:I was involved with NG on a project once.... (3, Insightful)

GrumblyStuff (870046) | more than 3 years ago | (#33465912)

Probably but you die on the inside every day you have to work up bullshit excuse why this was bound to happen and why it's a good thing.

Re:I was involved with NG on a project once.... (0)

Anonymous Coward | more than 3 years ago | (#33466390)

OH HAI! I was wondering where my Super-Ego went!

NGC Culture (4, Interesting)

CherniyVolk (513591) | more than 3 years ago | (#33465292)

I work with a fair number of ngc.com; I'm a contractor myself. At one point, I had a few interviews for Northrop and I'm so glad they over looked my talents because they seem to abuse the talent they have.

Now, this may not be for every department or division, but almost every NGC employee I know is basically well familiar with furlough. Whether good or bad, NGC is left with the ability to place entire departments on furlough to reduce overhead costs in the event a contract dries up. Now perhaps it's their size, perhaps they simply don't care about their workers, but this sort of thing seems to happen often. I'd guess that no NGC employee with a tenure more than 2 years hasn't been out of work for up to a month or so. But this is how things are run there.

See, government contracting works like this. You create a company, hire some folk to work on a contract. Whatever their salary is, you charge the government +50% or more, so essentially the government is not only flat out paying your salary but also the company for your services. If the contract ends, so does your job as the company may not want to charge overhead. In contrast to other business sectors, employment typically isn't grounded so harshly on the existence of a contract, which is where cost of business and business management can keep workers afloat even during down times (think department store).

I only point out Northrop because while all government contracting is essentially this contract+play model, Northrop has a reputation of placing people on furlough much more often than other companies such as CACI, Raytheon, General Atomics etc. Some Northrop employees seem to live the lives similar to actors and actresses in Hollywood, and I'm not talking about Tom Hanks acting, but maybe those actors that get little spots from time to time on your sitcoms. They literally live in apartments, and wait for the phone to ring day after day. Northrop employees seem to wake up in the morning, wondering if they'll still have a job at the end of the day.

What does this observation have to do with the op? Well, it seems that moral and motivation might be a bit low on a large scale at Northrop, so such blunders are no surprise to me.

Re:NGC Culture (3, Informative)

XorNand (517466) | more than 3 years ago | (#33466052)

See, government contracting works like this. You create a company, hire some folk to work on a contract. Whatever their salary is, you charge the government +50% or more, so essentially the government is not only flat out paying your salary but also the company for your services. If the contract ends, so does your job as the company may not want to charge overhead. In contrast to other business sectors, employment typically isn't grounded so harshly on the existence of a contract, which is where cost of business and business management can keep workers afloat even during down times (think department store).

I do a lot of business with both the federal gov't and private sector businesses on IT projects. You've over-simplified things to the point of painting an inaccurate picture. Federal contracting is extremely complex and there are myriad types of contracts that can be awarded, each with different terms. It sounds like what you're describing is a labor-hour contract. The contractor bills the gov't for the "fully burdened cost" of putting a warm butt in a seat. This includes the worker's salary, overhead, G&A (general & administrative), and profit. All together, it's typically a lot more than a 50% markup of the staff's straight salary.

Unlike most private sector contracts, when doing a fully burdened labor hour contract with the feds, the contractor will spell out exactly what their profit margin is. Generally this is only 6-10%, which is considerably lower than the private sector. Despite what everyone thinks, doing business with the gov't isn't all that lucrative. It's an extremely competitive market in which the bottom-line cost is almost always the most important factor. Contracting officers are even prohibited by law to give preferential treatment to companies that have previously done a great job.

I can't really comment on forced furloughs, because I'm not familiar with how Northrup operates. But just because they do "government contracts" doesn't necessarily mean they can afford to keep highly-skilled staff on the payroll until they find a new project for them. Federal contracts can really help with sales revenue because they can be large awards and the government *always* pays. However, the trade off is all the red tape (which increases G&A costs) and the low profit margins. Next time you hear about Company X getting a $10M contract, don't just roll your eyes. Get a hold of their proposal and the contract and see what their actual profit is on the contract. Both documents are public property and available upon request from the federal contracting officer that made the award. (Defense related contracts might need to be pried from gov't with a FOIA request though)

Re:NGC Culture (0)

Anonymous Coward | more than 3 years ago | (#33466616)

I don't know which gov't you work for but all a contractor has to do is win one contract and then every upgrade/improvement/repair after that makes them the sole source. Doesn't take a genius to realize you can charge more if you aren't competing to get the job. And are you trying to say that the contractor doesn't inflate the hours at all to make up for the fixed profit?

Re:NGC Culture (1)

wkcole (644783) | more than 3 years ago | (#33466674)

Now, this may not be for every department or division, but almost every NGC employee I know is basically well familiar with furlough. Whether good or bad, NGC is left with the ability to place entire departments on furlough to reduce overhead costs in the event a contract dries up. Now perhaps it's their size, perhaps they simply don't care about their workers, but this sort of thing seems to happen often. I'd guess that no NGC employee with a tenure more than 2 years hasn't been out of work for up to a month or so. But this is how things are run there.

See, government contracting works like this. You create a company, hire some folk to work on a contract. Whatever their salary is, you charge the government +50% or more, so essentially the government is not only flat out paying your salary but also the company for your services. If the contract ends, so does your job as the company may not want to charge overhead. In contrast to other business sectors, employment typically isn't grounded so harshly on the existence of a contract, which is where cost of business and business management can keep workers afloat even during down times (think department store).

FWIW, this makes it sound like NGC is a couple of steps better than the contract IT shops that I'm familiar with that service the private sector. 100% markup is fairly common, and keeping idle employees "on furlough" is a concept that doesn't really exist. When the contract ends, the job ends. A pure pimp agency will usually try to place a profitable contractor in another spot ASAP once a customer provides notice, but they won't guarantee anyones rates and I have never heard of a pimp agency doing anything to hold on to a contractor who is between contracts if they don't have anything suitable immediately available. For "managed project" type contractors it is as bad or worse. Agencies typically make sure they have the power to take their people with them if they lose a project, but often they only exercise that power only as a tool and/or weapon applied to customers and competitors to the detriment of their employees, rather than to actually move them to other work.

Animal House (1)

david@ecsd.com (45841) | more than 3 years ago | (#33465308)

Kinda like brother Bluto in Animal House when he smashes the guitar, then looks at the shocked people and goes, "Heh, sorry."

It was EMC storage failure (4, Informative)

Anonymous Coward | more than 3 years ago | (#33465318)

It was EMC storage that failed:

http://www.boston.com/business/technology/articles/2010/08/31/emc_system_serving_virginia_breaks_down/

"A major portion of the network shut down on Thursday after some of the EMC gear malfunctioned. As many as 400 server computers in various government departments relied on the storage network and were knocked offline.

Both Northrop Grumman and EMC declined to comment, directing all inquiries about the breakdown to the Virginia Information Technologies Agency, which oversees all of that state’s government computer systems. According to the agency’s website, EMC said that Thursday’s breakdown was unprecedented. “The manufacturer reports that the system and its underlying technology have an exemplary history of reliability, industry-leading data availability of more than 99.999% and no similar failure in one billion hours of run time,’’ the website said."

Re:It was EMC storage failure (1)

TheLink (130905) | more than 3 years ago | (#33466078)

> Both Northrop Grumman and EMC declined to comment,
> The manufacturer reports that the system and its underlying technology have an exemplary history of reliability

So who's lying ;).

Re:It was EMC storage failure (1)

Lunix Nutcase (1092239) | more than 3 years ago | (#33466128)

Both of them?

Re:It was EMC storage failure (1)

AB3A (192265) | more than 3 years ago | (#33466348)

Lesson: Don't put all your eggs in one basket. Nothing new about that...

Re:It was EMC storage failure (2, Insightful)

Lunix Nutcase (1092239) | more than 3 years ago | (#33466632)

Actually the lesson is to not let a bunch of incompetents draw up your IT contracts.

Apparently, when VITA negotiated its 10-year, $2.3 billion outsourcing contract with Northrop Grumman to modernize Virginia's 85 state government agencies' IT systems and networks, it forgot to require network that backup capability be provided in case of network failure, the Richmond Times-Disptach reported over the weekend.

I mean really? Requiring redundancy is such a basic requirement that you really have to wonder if the people in VITA even have a brain.

Re:It was EMC storage failure (1)

realmolo (574068) | more than 3 years ago | (#33466366)

So, for their $2.6 billion, Virginia doesn't get redundant systems? Everything is on ONE SAN?!

I would sue the living shit out of Northrop. That's insane.

Re:It was EMC storage failure (3, Informative)

Lunix Nutcase (1092239) | more than 3 years ago | (#33466554)

I would sue the living shit out of Northrop. That's insane.

Based on what would they sue? The contract they signed had no requirement of redundancy [ieee.org] . As much fault as NG has in this, it's not like they broke the contract or anything. This is as much the fault of the incompetents in Virginia's IT Agency as on NG's.

Apparently, when VITA negotiated its 10-year, $2.3 billion outsourcing contract with Northrop Grumman to modernize Virginia's 85 state government agencies' IT systems and networks, it forgot to require network that backup capability be provided in case of network failure, the Richmond Times-Disptach reported over the weekend.

Re:It was EMC storage failure (0)

Anonymous Coward | more than 3 years ago | (#33466638)

So, for their $2.6 billion, Virginia doesn't get redundant systems? Everything is on ONE SAN?!

I would sue the living shit out of Northrop. That's insane.

RTFA. There was a backup SAN.

Re:It was EMC storage failure (-1, Troll)

Anonymous Coward | more than 3 years ago | (#33466382)

You are injecting facts into a /. discussion. Are you new here;-)

Re:It was EMC storage failure (1)

Klinky (636952) | more than 3 years ago | (#33466498)

How are they counting one billion hours of run time? Are they counting uptime and multiplying it by each disk or something? One billion hours is 114,079 years... Five-nines for 1 billions hours is 1.14 years... At that rate they should be happy it only took Northrup a week to get things working again.

The trauma... (0, Offtopic)

ackthpt (218170) | more than 3 years ago | (#33465342)

It lead to the Great Pr0n Outage of '10.

Lechers had to actually go outdoors and look at live, clothed people for a change.

Re:The trauma... (1)

Lunix Nutcase (1092239) | more than 3 years ago | (#33465398)

look at live, clothed people for a change.

That's vile and disgusting.

Re:The trauma... (1)

Jedi Alec (258881) | more than 3 years ago | (#33465832)

And now you know why some European countries have chosen to consider internet access in the same category as gas, electricity and water. Society depends on it ;-)

at least they never said ... (1, Interesting)

Anonymous Coward | more than 3 years ago | (#33465402)

That "they'll fix it so it never happens again."

I used to work in IT and I hated it whenever some manager said that to a customer. Things fail, they can fail multiple times and different failures can have the same results so the statement that "it'll never happen again" would be a bold faced lie.

Re:at least they never said ... (0)

Anonymous Coward | more than 3 years ago | (#33466276)

"I used to work in IT and I hated it whenever some manager said that to a customer."

Yeah, specially when the incident study is over and it comes out that a techy already told what the problems were, what the possible outcomes would be and what the remediation should look like and how much it would cost. Then that very same manager from the beginning tells (again) to the technicians: "no we won't do that: is too expensive and missaligned with our bussiness goals" while getting his bonus intact.

I first read the headline as (2, Insightful)

edwebdev (1304531) | more than 3 years ago | (#33465414)

"Northrop Grumman Says 'I'm Sorry' For Virginia IT Outrage"

Incompetent IT folk (0)

Anonymous Coward | more than 3 years ago | (#33465636)

Yeah, them Indians, stealing all our jaaaaabs and doing crappy work on top of it!

Oh, wait! Don't tell me these were American IT staff, locally recruited in the good ol' US of A?? Meh, this has got to be someone else's fault. Americans in IT are simply flawless!

Re:Incompetent IT folk (0)

Anonymous Coward | more than 3 years ago | (#33466058)

Yes these are second rate locally recruited American IT staff. If they could get past the security requirements these positions would certainly be filled by Indians since a talented American would not touch it.

Re:Incompetent IT folk (0)

Anonymous Coward | more than 3 years ago | (#33466144)

these positions would certainly be filled by Indians

Second rate Indian programmers to be precise. The talented ones wouldn't bother.

Government contracting is the B leagues (1)

cryfreedomlove (929828) | more than 3 years ago | (#33465638)

This is happening because all of the best IT talent these days are doing startups or working at cool companies like Google and Facebook. This means that Northrup Grumann is staffing their teams with people that can't get those cooler IT jobs. That's the real cause of this disaster for Virginia.

Re:Government contracting is the B leagues (1)

DarkAce911 (245282) | more than 3 years ago | (#33465692)

It's Richmond, there are no cool start-ups like that anywhere in the city. Ok, maybe on the West End, but the data center is in Chester, south of Downtown.

Exactly well sort of (0)

Anonymous Coward | more than 3 years ago | (#33465878)

I recently turned down a offer from NG. I turned them down for a number of reasons but mainly past employee experience. Once I had learned that they unexpectedly cut employee salaries in the past to make numbers that was a very large part of the decision. There where other issues like competence and attitudes, that I expect in govt related positions. They are simply not going to attract top talent with their track record.

Not Quite (1)

geoffrobinson (109879) | more than 3 years ago | (#33466580)

I used to work at Lockheed Martin and I worked with a tremendous amount of talented people. The people tended to be a little more risk-averse, career-wise.

The likelihood of working 60 hour weeks is less, which is an attractions. Benefits used to be good. So it just attracts different people.

3-year-olds know better than that (2, Insightful)

swschrad (312009) | more than 3 years ago | (#33465782)

good thing Northrup Grumman doesn't do anything important, like, say, vital national security support.

oh, wait... .

Re:3-year-olds know better than that (1)

Monchanger (637670) | more than 3 years ago | (#33466072)

I wonder how much that's actually related. When I read the contractor was NG, I asked myself "what the hell are they in that business for?"

The answer, money, is of course, a major suspect in the failure.

This contract should not have gone to someone like NG, who probably just turned around and sold it to EMC who should have been competent. That is, if it really was EMC and not one of their subcontractors.

Re:3-year-olds know better than that (1)

Lunix Nutcase (1092239) | more than 3 years ago | (#33466244)

But according [northropgrumman.com] to Northrup Grumman they were "recently recognized by the National Association of Counties for Outstanding Achievement in the area of "Information Technology in State Government – Enterprise IT Management Initiatives".

More background. (1, Insightful)

Anonymous Coward | more than 3 years ago | (#33465940)

can't find my account info sorry.

More background for those interested.

http://www.govtech.com/gt/749378 this is the same state agency which last summer fired the head of the agency because he attempted to make Northrop Grumman comply with its obligations.

For some background on their previous problems try the IEEE.

http://spectrum.ieee.org/riskfactor/computing/it/virginia-information-technologies-agency-believes-in-the-perfect-network-fairy

And yes, this is the same agency which has been help up as an example of the benefits of taking diverse IT services and pulling them into a central cloud.

In other news ... (1, Insightful)

Anonymous Coward | more than 3 years ago | (#33466378)

NG reports increased profits and record bonuses for key executives

One Merger Away From Completness (0)

Anonymous Coward | more than 3 years ago | (#33466400)

Northrop-Grumman just needs to merge with Fairchild and their company initials will be complete - NFG

Go read Virginia IT's website... (2, Insightful)

HockeyPuck (141947) | more than 3 years ago | (#33466446)

Go see it yourself here [virginia.gov] .

On Wednesday, August 25, at approximately 3 p.m., the Commonwealth of Virginia experienced an information technology (IT) infrastructure outage that affected 27 of the Commonwealth's 89 agencies and caused 13 percent of the Commonwealth's file servers to fail. The failure was in the equipment used for data storage, commonly known as a storage area network (SAN). Specifically, the SAN that failed was an EMC DMX-3.

According to the manufacturer of the storage system, the events that led to the outage appear to be unprecedented. The manufacturer reports that the system and its underlying technology have an exemplary history of reliability, industry-leading data availability of more than 99.999 percent and no similar failure has occurred in more than one billion hours of run time. A root cause analysis of the failure is currently being conducted.

Anybody else read this like some middle age guy after "finishing a bit too quickly" and telling his , "I swear honey, this the first time this has ever happened to me..."

Apology (2, Insightful)

Curunir_wolf (588405) | more than 3 years ago | (#33466536)

So Virginia taxpayers will continue to get screwed, but Northrop Grumman has now extended a reach-around?

Irony (1, Insightful)

Anonymous Coward | more than 3 years ago | (#33466640)

Ironically, Microsoft has a case study touting the Virginia DMV project as a success story.

http://www.microsoft.com/casestudies/Microsoft-SQL-Server-2005-Enterprise-Edition/Virginia-Department-of-Motor-Vehicles/Virginia-DMV-Enhances-Decisions-Boosts-Safety-Through-Integration-with-Other-Agencies/4000004307

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Create a Slashdot Account

Loading...