×

Announcing: Slashdot Deals - Explore geek apps, games, gadgets and more. (what is this?)

Thank you!

We are sorry to see you leave - Beta is different and we value the time you took to try it out. Before you decide to go, please take a look at some value-adds for Beta and learn more about it. Thank you for reading Slashdot, and for making the site better!

Passport Database Outage Leaves Thousands Stranded

Unknown Lamer posted about 4 months ago | from the maintenance-considered-harmful dept.

Bug 162

linuxwrangler (582055) writes Job interviews missed, work and wedding plans disrupted, children unable to fly home with their adoptive parents. All this disruption is due to a outage involving the passport and visa processing database at the U.S. State Department. The problems have been ongoing since July 19 and the best estimate for repair is "soon." The system "crashed shortly after maintenance."

Sorry! There are no comments related to the filter you selected.

Change management fail (4, Insightful)

dave562 (969951) | about 4 months ago | (#47578095)

Rollback plan? What is that?

Re:Change management fail (1)

Fallen Kell (165468) | about 4 months ago | (#47578139)

Yeah I was going to say the same thing. You NEVER make a change that you don't have a way to backout to the previous operational state.

Re:Change management fail (5, Informative)

roc97007 (608802) | about 4 months ago | (#47578187)

It's the wave of the future. A typical contract with offshore IT is for "current minus one", which means that each new firmware, OS or driver release causes a flurry of "maintenance" by remote "admins" who follow written procedures to update the systems with no real understanding of what they're doing, in what order they should do it, or what to do if something goes wrong. A typical list of systems to update may randomly contain a haphazard collection of prod and development machines, and may include some but not all members of a cluster. Systems are patched in Asset Management order, with no thought to rolling through dev and QA first before doing prod.

The backout plan is to engage the vendor.

Our outsourced IT bricks a few servers a year. We try to take it in stride. We've argued hysterically that if they really have to do firmware updates, to at least do dev servers first for God's Sake. They seem to not understand this.

So yeah, I could definitely see this happening. We will be seeing more of same. You get what you pay for.

Re:Change management fail (-1)

Anonymous Coward | about 4 months ago | (#47578757)

> to at least do dev servers first for God's Sake.
Well there's your problem! God has no part in an IT management plan.

Re:Change management fail (4, Funny)

Areyoukiddingme (1289470) | about 4 months ago | (#47578949)

Well there's your problem! God has no part in an IT management plan.

Yeah, the other guy has it well in hand.

Re:Change management fail (3, Funny)

roc97007 (608802) | about 4 months ago | (#47578989)

I sometimes think that if I accidentally entered a church with an IT management plan in a back pocket, my pants would burst into flames.

Re:Change management fail (2, Insightful)

JustOK (667959) | about 4 months ago | (#47579243)

Why? Are you a liar?

Re:Change management fail (1)

roc97007 (608802) | about 4 months ago | (#47579297)

Sorry, I guess I deserved that. I meant, of course, that the plan would burst into flames.

Re:Change management fail (0)

Cryacin (657549) | about 4 months ago | (#47578193)

We are US Customs and TSA. We NEVER make mistakes!

Re:Change management fail (1)

Anonymous Coward | about 4 months ago | (#47578661)

No dummy. Passports are issued by the State Department. Totally not DHS.

Re:Change management fail (1)

davester666 (731373) | about 4 months ago | (#47579527)

Sure the state department may issue them, but DHS controls the database. It's one of the primary data sources for their terrorist watch list database.

Re:Change management fail (1)

roc97007 (608802) | about 4 months ago | (#47578201)

...in case my other article did not make it clear, we always ask if they have a backout plan, and they always say they do.

...and then, when the system does not reboot after an update, we find out that the backout plan is to call Dell and say "the system, it is not being working. What are we to be doing?"

...and we pay money for this.

Re:Change management fail (4, Insightful)

Anonymous Coward | about 4 months ago | (#47578309)

Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no". The answer is always "yes" or some other affirmative that makes you think they have it under control and can do the work. When the fact is, the work they just said "yes" to, they don't actually have a clue how to perform it, so they learn as they go, on your production servers. They don't know what development / test environments are.

Re:Change management fail (-1, Offtopic)

ruir (2709173) | about 4 months ago | (#47578373)

This is a very insightful comment, pity you posted it as anonymous. I do not mod up anonymous by policy, and often it is a waste of time to answer them. People hire techs by peanuts, get monkeys.

Re:Change management fail (0)

Anonymous Coward | about 4 months ago | (#47579201)

Pity you choose what to mod by the colour of their skin. Sorry, by their name.

Re:Change management fail (4, Informative)

ShanghaiBill (739463) | about 4 months ago | (#47578609)

Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".

It takes two to fail to communicate. You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.

Re:Change management fail (2, Insightful)

Anonymous Coward | about 4 months ago | (#47578631)

You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.

Do I care?

Re:Change management fail (2, Insightful)

Anonymous Coward | about 4 months ago | (#47579239)

You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.

Do I care?

If affects your ability to do your job, you should.

Re:Change management fail (1, Insightful)

pla (258480) | about 4 months ago | (#47578691)

It takes two to fail to communicate. You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.

Sorry, what part of paying you to do a job requires me to give a shit about whether or not your failed third-world culture doesn't like answering direct fucking questions?

"Rude" does not apply. Breach of contract, however, does. I just wish more companies would catch on to this before they decide to outsource, rather than paying extra for literally nothing more than a built-in scapegoat for any and all problems.

Re:Change management fail (5, Insightful)

ShanghaiBill (739463) | about 4 months ago | (#47578783)

Sorry, what part of paying you to do a job requires me to give a shit about whether or not your failed third-world culture doesn't like answering direct fucking questions?

The part about you paying them far less than you would pay someone culturally compatible. If you want to pay peanuts, you need to deal with the cultural consequences. I have dealt with Indians for years, and have learned how to ask questions so that I get the answer I am looking for. It is not that hard.

Re:Change management fail (1)

Anonymous Coward | about 4 months ago | (#47578821)

That must have been the most self- and Western-centric comment I've read in a while. It wouldn't hurt to realize that next to the cultural sensitivity towards direct questions, the concept of a "contract" is also wholly different in the East than in the West...

(Coming from a Westerner living and working in China for the past years)

Captcha: "shudders"...

Re:Change management fail (1)

Ol Olsoc (1175323) | about 4 months ago | (#47578987)

That must have been the most self- and Western-centric comment I've read in a while. It wouldn't hurt to realize that next to the cultural sensitivity towards direct questions,

I't also the silliest fucking thing I've ever heard. If I have the director and a couple higly placed people breathing down my neck at a phenomenal burn rate, I gotta say that if the dude from India cannot answer simple yes or no questions, The's the last of their computers that will ever enter the shipping dock.

They're giving support to Americans. This booboo feelings sensitivity BS is just one more reason the offshoring doesn't work. If I'm insulting the poor guy, I wont ever bother him or his company again.

Re:Change management fail (1)

JustOK (667959) | about 4 months ago | (#47579263)

Sounds like your feelings are hurt.

Re:Change management fail (1)

Ol Olsoc (1175323) | about 4 months ago | (#47579379)

Sounds like your feelings are hurt.

What an odd reaction. I did what I did and was paid very well for it because I got things done. If the help doesn't perform, you simply change the help.

Re:Change management fail (0)

Anonymous Coward | about 4 months ago | (#47578717)

Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".

It takes two to fail to communicate. You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.

Oh, FFS.. then those cultures are wrong.

I can perceive being accommodating of cultural tics if you're the contracting firm (i.e. if they are your client and are paying you for your services), but if they are your client and you are paying them money then they need to bridge the fucking culture gap. This goes both ways. I wouldn't expect to contract in another country and have the denizens of that place go out of their way to accommodate my personal cultural idiosyncrasies. "When in Rome...", you know.

Furthermore, this "always answer in the affirmative" bullshit is not only retarded, it's downright dangerous in certain circumstances. Cf. interviewing patients who are in the hospital for emergent care and you need to rapidly winnow down the possible causes. You ask specific yes/no questions for discriminator symptoms and you need to have a goddamn truthful answer. Oh well, I guess it won't be *me* that's dying thanks to my own cultural inability to communicate "yes" or "no" honestly. Would you rather be "boorish" and alive, or "polite" and dead?

Re:Change management fail (0)

Anonymous Coward | about 4 months ago | (#47579069)

It takes two to fail to communicate.

In an ideal world.

There are too many occasions where I have personally witnessed people being told literally what to do step-by-step, and still screwing it up. Mind you, I am not talking about outsourced workers.

In a development or IT related role, you should be able to follow step-by-step instructions with little difficulty. If the instructions are ambiguous, then it is fair to blame both parties, but when it comes down to things like "change line X to ABC" and that does not happen correctly, then it is very difficult to blame both sides. And, regardless of cultural clashes, this happens far too frequently in our industry.

Re:Change management fail (0)

Anonymous Coward | about 4 months ago | (#47579713)

Yeah because it's his job to make sure that a DBA in india knows how to run a select count(*) from foo ? (try this in an instant message tool for a good laugh, you'll get as reply : it doesn't work! ).
Did I mention that my client is paying these fucktards 250USD a day? and 150USD for closing a ticket?

Re:Change management fail (0)

Anonymous Coward | about 4 months ago | (#47579105)

And the takeaway from all this is that it pays to be a "Yes-man". They remain employed just so some top executive can have his ego stroked. Where I come from, we just call them folk whores!!!

Re:Change management fail (4, Insightful)

khchung (462899) | about 4 months ago | (#47579129)

Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".

On the other hand, I have encountered plenty of managers who literally do not know how to take "no" as an answer.

Takes two to make a pair.

Re:Change management fail (2)

bayankaran (446245) | about 4 months ago | (#47579657)

Outsourcing is bad, but as an Indian who renewed his passport recently, I have to say the process was smooth.
The document verification/IT/software/hardware part of the Indian passport application/renewal operation is now handled by TCS. You make an appointment online - for both fast track and slow track - and arrive at the local passport center with your documents. The TCS grunts allot a token, make sure documents are in order - if they are not you are sent back to get them corrected, take your fingerprints, photo and for final verification you meet an Indian government employee. He/she either says YES, or says YES with police verification.
In my case it was the latter as my passport had some damage - usual wear and tear, but visible.
A few days later a policewoman came home after fixing an appointment over phone, verified my documents/address and in two weeks I got a fresh passport.
I was impressed by the whole process. AFAIK Indian government and TCS is doing everything right as far as passport renewal goes...it was better than getting my experience in getting a drivers license/immigration papers in Toronto/Chicago/San Francisco.
May be US government should indeed outsource the operation to TCS...if they can do a good show in an anarchic chaotic mess like India, I am sure they can do the same in US.

Re:Change management fail (0)

ruir (2709173) | about 4 months ago | (#47578377)

One question roc9xxx, with that kind of service, would you think it would be time to think to change, pay a little more and hire more responsible people, or do you thing I am being a bit far-fetched?

Re:Change management fail (3, Informative)

roc97007 (608802) | about 4 months ago | (#47578493)

Oh man, don't get me started. It's not even clear that one would need to pay more -- we have not saved money so far by outsourcing, although the outsource company keeps telling us that savings are just around the corner. The first year, the excuse was that there is always startup issues, the second year, the excuse was that the outgoing employees did not document their jobs well enough, (probably true -- who would?) the third year the excuse was that the scope was bigger than we said it was. And so forth. Each year a new excuse and each year the total cost is more than what we were paying when we had our own IT department.

So yeah, insourcing, or at least selective insourcing, (let them keep doing what they do well, if anything) makes tremendous sense to me.

But I don't make the decisions.

And even where upper management has considered terminating our outsourcing contracts, it's only to give the contract to a different outsourcing company, which only means we're now calling a building across the street from the original building in Hyderabad. Who knows, we might even be dealing with some of the same people.

Re:Change management fail (1, Insightful)

ruir (2709173) | about 4 months ago | (#47578581)

Outgoing employees are a flaw THEIR lack of management and a flaw of their processes, and I would not even accept to hear such (lame) excuse. And if there are no savings, I would bring the IT back, at least you can make them accountable and control the process better. I have worked with Indian people in the past, or rather had competing businesses employing them, and was not really that impressed with their technical abilities. And then, it probably does not help most of those firms tend to work on the cheap. Low salaries policy dont attract the best and the brightest.

Re:Change management fail (2)

roc97007 (608802) | about 4 months ago | (#47578699)

Exactly. Consider the business model -- A company tells you that you can outsource your IT department, buy it as a service, and pay the outsourcing company's overhead and significant profit margin, *and* save money. And the only way this could possibly work is if the outsourcing company goes to the LCC (least cost country) and hires the cheapest labor possible. This is justified in that all that IT stuff, it's all just following procedures, and anyone can do that.

And of course, this is a blatant falsehood, but executives of the victim company either (a) don't know that, (b) *want* to believe the scam, (like any good scam) or (c) don't care, because they intend to take their bonus and get the hell out of dodge.

So, outsourcing companies go into contracts *knowing* that cutover is going to be a Big Fail, and they have excuses prepared for when it happens. And a strategy (a brilliant strategy, really, executed by brilliant if unethical people) to string the con along as long as possible.

And just incidentally, the victim's attempts to train the workers that they are stuck with also fails in the long run, due to the nature of the business model. The business model breaks down if the workers are paid more than starvation wages, and workers with a bit of experience can easily get a higher paid job elsewhere (perhaps as a second or third level admin for a different outsourcing company) and they quit. And then you have a new person who doesn't know what a kernel is, and you have to train them up. And all you're doing is giving out free training. How noble of you.

...so not only is the system *designed* to fail, the system is very specifically designed to fail continuously.

But at least it's cheap. Oh, wait...

Re:Change management fail (1)

ruir (2709173) | about 4 months ago | (#47578815)

I often can not get the mentality of wanting to have good and competent resources for peanuts. People work for a living, if customers pay, at least pay too to the people that helps running your business along. And you have summed quite nicely why I avoid working for shops whose core business is not IT, IT are viewed like some strange pumbling guys that just suck money (i.e a center cost). And then, they probably pay more to the proper plumbers...

Re:Change management fail (1)

Ol Olsoc (1175323) | about 4 months ago | (#47579001)

I often can not get the mentality of wanting to have good and competent resources for peanuts. People work for a living, if customers pay, at least pay too to the people that helps running your business along. And you have summed quite nicely why I avoid working for shops whose core business is not IT, IT are viewed like some strange pumbling guys that just suck money (i.e a center cost). And then, they probably pay more to the proper plumbers...

Don't worry, the cloud's basic purpose is to get rid of those IT guys.

Re:Change management fail (1)

ruir (2709173) | about 4 months ago | (#47579109)

Sure, the cloud runs itself magically, and you call Merlin when you have problems...That should be the most stupid comment I have read lately. The cloud is only outsourcing, you know? It is like you saying you dont need more mechanics because you go to the garage and leave there your car. ;-P

Re:Change management fail (1)

Ol Olsoc (1175323) | about 4 months ago | (#47579177)

Sure, the cloud runs itself magically, and you call Merlin when you have problems...That should be the most stupid comment I have read lately. The cloud is only outsourcing, you know? It is like you saying you dont need more mechanics because you go to the garage and leave there your car. ;-P

You don't think the suits are salivating over the possibilitiy of the people they can get rid of once they send as much work as possible to "the cloud"? There is a repeating cycle in the business world. They start out with in-house. Then they want to cut costs, so they outsource, then outsourceing doesn't work very well because of either costing more than it was supposed to, or it doesn't work like it was supposed to, so they start hiring in house again(and remember, you are just another customer to your cloud supplier, so you don't get any special treatment - same as outsourcing) Then they run with in house for a while, then they want to cut costs.....

I've seen the cycle many times.

Re:Change management fail (3, Interesting)

l0n3s0m3phr34k (2613107) | about 4 months ago | (#47578999)

This isn't always the case. A company can save money via outsourcing IT infrustructure if they go with the right vendor. VMWare, virtual servers, proper fail-overs, big multi-core blade racks were the VM is still more powerful than your original server and still costs less...but of course I work at HP in the Enterprise Services so the level I'm talking about probably isn't affordable for a "small company". We have VERY specific steps for everything, our "runbooks" detail everything from server configs, hardware, rack enclosures, port layouts, and responsible parties to contact for each part if it fails. When you have a rack of blades, it's far easier to snapshot, launch then test,,,we always have a "backup" in a hot image ready to go if anything fails. Often I'm working with 5-15 people spread across the globe all doing different functions (unix admin, wintel, recovery, netops, etc) but we rarely have any "HP owned" customer-impacting outages. Of course my major clients are airlines so it's all tightly regulated; your individual milage may vary LOL.

Re:Change management fail (1)

roc97007 (608802) | about 4 months ago | (#47579305)

Not to put too fine a point on it, but I've heard that sales pitch before. I agree, HP has some truly brilliant admins, as evidenced by the "demo". The day to day reality is very much different.

Re:Change management fail (1)

Stolpskott (2422670) | about 4 months ago | (#47579731)

...in case my other article did not make it clear, we always ask if they have a backout plan, and they always say they do.

I used to deal with a lot of Indian outsourced IT groups, and the only way to handle this is to either follow up the "Yes, we have a backout plan" response with "Tell me what your backout plan is" or just to skip straight to that without bothering to ask the "Do you have a plan?" question.
Things still got screwed up, but after the first occurrence we completely cut their access to the servers and re-enabled them on demand, so we forced their people to update a specific server first to show that they could do it on a system which is not mission-critical.
However, that approach really only works when the client does not turn into a whining tub of lard when the vendor starts putting pressure on.

Re:Change management fail (1)

gstoddart (321705) | about 4 months ago | (#47578725)

Yeah I was going to say the same thing. You NEVER make a change that you don't have a way to backout to the previous operational state.

And, really, if you have something which Absolutely Has To Be There ... you make damned sure you have an environment you apply the changes to first. So that you can apply the changes and at least try to make sure stuff don't break without messing up the real one.

This is basic change management.

(And, yes, I am saying this without any context for this outage -- but, really, if you maintain a production environment for critical software, this is what you do)

Re:Change management fail (1)

Anonymous Coward | about 4 months ago | (#47579115)

Yeah I was going to say the same thing. You NEVER make a change that you don't have a way to backout to the previous operational state.

Sounds great to MBA types, but easier said than done, the real world don't have a transaction log or a time machine you can just rollback. Especially when management decreed that ANY action not regularly scheduled automatically is considered a "change".

Tell me, how do you "backout" of these "changes" in a way you can be SURE you would be back to the "previous operational state":

  • - copying a file from directory A to directory B (how can you be 100% SURE your "backout" job would delete to correct file?)
  • - restoring a database from backup
  • - reorganizing a database
  • - rebuilding an index
  • - taking a backup
  • - collecting and updating the statistics of your database tables
  • - shutting down or rebooting a machine without any configuration change (what if it won't start? How do you "backout"?)
  • - shutting down or restarting a process on the machine without any configuration change (what if it won't start? How do you "backout"?)
  • - plugging a network cable to another socket (double penalty for answering "plugging it back to the original socket", it doesn't guarantee operational state if the cable plug lost a pin in process)
  • - replacing any hardware component ("putting the old one back in" in not an acceptable answer, see above)

Re:Change management fail (0)

Anonymous Coward | about 4 months ago | (#47578151)

Same as testing on a test server before pushing it out to a production server ?

Re:Change management fail (1)

TWX (665546) | about 4 months ago | (#47578211)

More like same as trying a test server, then after that appears to work, pushing out to only a portion of your production cluster, if the backend can handle having two different versions running concurrently, then seeing if those portion that are changed continue to work at least as well as the original ones for a functional duration, if not better. Then push out to the rest.

Even better, if the circumstances require significant DB changes, you back up the DB (you are backing up the DB, right?), start bringing up new nodes, add those nodes to the load balancer, then start bringing down old nodes. You also date-stamp the changelog so that if you have to rollback to a previous database for previous-generation servers, you can propagate the necessary changes that were committed to the new database post-roll to the old one, if you have to bring the old one back online.

Re: Change management fail (0)

Anonymous Coward | about 4 months ago | (#47578251)

Sorry but DevOps requires you upgrade all servers at the same time very fast, with no regard to individual server ordering.

Re: Change management fail (1)

ruir (2709173) | about 4 months ago | (#47578393)

DevOps also have maintenance, and production, pre-production and development systems. Automating things does mean throwing standard procedures of the industry out of the window, it only means working smarter. ...if you want to troll discussions and denigrate people who apparently work better than yourself, at least learn how to pretend that you know what you are talking about

Re: Change management fail (1)

plopez (54068) | about 4 months ago | (#47578647)

From my experience with developers, developers as Ops are a misshapen twisted wandering shambling nightmare.

Re: Change management fail (1)

l0n3s0m3phr34k (2613107) | about 4 months ago | (#47579011)

Sounds like you have an Ghisguth infection. I suggest calling Dresden the Wizard and maybe he can help out!

Re: Change management fail (1)

pla (258480) | about 4 months ago | (#47578755)

Sorry but DevOps requires you upgrade all servers at the same time very fast, with no regard to individual server ordering.

Did you mean NetOps? DevOps refers to a development paradigm. If your development paradigm risks actual user-impacting down-time, you need firing ASAP.
Assuming you meant NetOps, can they live with provisioning me at least four (dev, test, UAT, and training) clones of the entire production environment? No? well then, they can make their case to the CTO whether inconveniencing them or our end users will have more of an impact on the bottom line. If the CTO says "go", hell, I'll code right in the production environment - Oh, you wanted that mortgage payment to go through this week? Bummer!

Developers should never have the power to affect end users. If they do, it represents a failure not on their part, but on the entire IT corporate food chain, all the way to the top. Choosing customer-facing downtime over a few more terabytes and VMs amounts to corporate suicide.

Change management fail (0)

Anonymous Coward | about 4 months ago | (#47578205)

http://happyplace.someecards.com/confession/elevator-work-in-progress-funny-sign/

Re:Change management fail (1)

ruir (2709173) | about 4 months ago | (#47578357)

Forget the obvious rollback plans? At the very least working without computers, no? Or having alternative system for something so crucial...If computers are only there to complicate, than we are better off without them. The level of incompetence and lack of planning is strong in several levels.

Re:Change management fail (1)

dave562 (969951) | about 4 months ago | (#47578503)

You bring up a good point. Given the extent out of the impact of the change, they probably should have just declared a disaster and gone with Plan B. Yet, given that they blew a system change and did not have a rollback plan, I am fairly confident that any sort of DR strategy is equally broken and worthless.

Situations like this always put a smile on my face, because I know that my job is secure. If an organization as large as the United States government cannot get these basics right, but I can... I know that I will always be in a position to make improvements somewhere, and will never be faced with a shortage of things to do.

I see the same thing with the major Fortune 50 corporations that I work with. I am thoroughly convinced that from the smallest shops, up to the largest organizations, the majority of IT departments are barely functioning and are just one bad change away from serious down time with no hope of recovering in any sort of reasonable amount of time.

Re:Change management fail (1)

ruir (2709173) | about 4 months ago | (#47578561)

This kind of situation is unthinkable in 2014...haven't they heard about virtualisation technologies and clones before major changes (as at least a very minimum baseline?) How about pre-production system to TEST changes? Change management process? Backups? Alternative systems for disasters replicated from last week systems? If I had to guess, I would say they where hacked and could not count on backups, but then we are dealing with the government and DHS, and they are often known for not being that competent, everything is possible. I also agree we wont be anytime out of work. The only thing that worries me is that no matter how things are fucked up, and no matter how you are seasoned on the industry, often people want people to work for peanuts.

Re:Change management fail (3, Interesting)

dave562 (969951) | about 4 months ago | (#47578621)

As much as I am not a fan of government regulation, my professional experience has shown me that the only time people get IT anywhere close to right is when there is a risk of financial penalty involved in getting it wrong. Regulation seems to be the only solution to people working for peanuts. The people who work for peanuts make mistakes. If those mistakes cost the company more than the company saves by hiring those people, they will not hire those people.

Out of all of the industries that I have worked with, the financial services industries seem to be the most together. They are not perfect, but the penalties associated with losing customer data makes them more careful.

Re:Change management fail (0)

Anonymous Coward | about 4 months ago | (#47578597)

Fact 1: Business processes are complex. There are always lots of rules, more than is ever documented, and lots of exceptions and excuses.

Fact 2: IT makes business processes MORE complex and complicated. Complexity increases by relying on systems. Complications increase when you become dependent on so much infrastructure and special-privileged access and know-how just to access the systems.

Fact 3: Businesses wants to pay as little as possible. On paper, the business might save some nickles in the short run, but every cost-cut has hidden prices and risks, that at some point in the future will become critical.

Fact 4: Businesses are only as good as their PEOPLE. Screw your people, and your company is screwed. This goes for ALL of the customers, employees, management and owners.

Fact 5: Everybody operating a business except the direct owners really care about the bottom line of the business.

Fact 6: Owners usually have no competence over how the operations best can be handled, they only care about the bottom line.

I'm sure there is a catch-22 in there somewhere.. Shouldn't be hard to find.

Re:Change management fail (1)

dbIII (701233) | about 4 months ago | (#47579687)

There are sometimes very "political" problems that prevent rollback because for instance that means buying more time on the licences from an earlier vendor when the new vendor is heavily embedded with the management that are driving the change. Such a problem in my state resulted in such an enormous fuckup in Hospital payroll systems with no rollback that there was real political fallout - after the bill went beyond 500 million the government lost office to be replaced by a bunch of baby fascists led by a Pocket Putin.

Re:Change management fail (0)

Anonymous Coward | about 4 months ago | (#47578411)

>testing backups

Top kek.

Re:Change management fail (1)

Ol Olsoc (1175323) | about 4 months ago | (#47578939)

Rollback plan? What is that?

Umm - have you tried rebooting the computer sir. Try that first.

Maybe the IRS can help (1)

Bodhammer (559311) | about 4 months ago | (#47578111)

They are hard drive experts!

Damn (1)

rmdingler (1955220) | about 4 months ago | (#47578161)

Still, bet Sysadmin's the highest ranking head that'll roll.

Re:Damn (1)

ruir (2709173) | about 4 months ago | (#47578399)

Will they? Apparently they do not have sysadmins and are just a bunch of system operators, if that.

The solution (4, Funny)

Loopy (41728) | about 4 months ago | (#47578181)

Sic the healthcare.gov guys on it. I'm sure it'll be right as rain in no time.

Replication != Backup (5, Interesting)

Anonymous Coward | about 4 months ago | (#47578219)

From their Q&A:

Q: Why wasn’t there a back-up server?
Back-up capability and redundancy are built into the system. The upgrade affected our current processing capability, in part because it interfered with the smooth interoperability of redundant nodes.

We don't need backups, the data is replicated, we're cool.

Re:Replication != Backup (0)

Anonymous Coward | about 4 months ago | (#47578291)

How could they have known! LOL!

Re:Replication != Backup (4, Funny)

sumdumass (711423) | about 4 months ago | (#47578441)

http://xkcd.com/327/ [xkcd.com]

I wonder if little bobby tables applied for a passport while this was going on?

Re:Replication != Backup (1)

ruir (2709173) | about 4 months ago | (#47578403)

this one smells like BS even very far, far away. I change contractors on the spot.

Re:Replication != Backup (1)

Noah Haders (3621429) | about 4 months ago | (#47578429)

I would change governments on the spot. Kiss my ass, ins/dhs/TSA!

Over Improved (1)

bfmorgan (839462) | about 4 months ago | (#47578267)

We call this being over improved. So much for testing.

One Database to rule them all (1)

sehlat (180760) | about 4 months ago | (#47578293)

One Database to bind them.
One Database to keep them out.
And into the darkness send them.

Ask the NSA (4, Funny)

Daemonik (171801) | about 4 months ago | (#47578303)

I'm sure they have full copies of all the data already.

Re:Ask the NSA (0)

Anonymous Coward | about 4 months ago | (#47579417)

omfg, This is actually the first reasonable solution in this page that I can agree with an laugh at at the same time.

CAPTCHA:smutty

Everybody is illegal (0)

Anonymous Coward | about 4 months ago | (#47578311)

Glad I'm not 'murrican. With the inmates firmly in control of the asylum I'm inclined to listen to the tinfoilhatters who think it's a plot to control the populace.

I wonder (2)

ruir (2709173) | about 4 months ago | (#47578319)

That these breakdowns are lame excuses. If computers fails, have people forgot how to do the same process manually? It is better to halt all the flights than letting people through and risk "terrorists" flying? Are we that terrified?

Re:I wonder (0)

Anonymous Coward | about 4 months ago | (#47578497)

That these breakdowns are lame excuses. If computers fails, have people forgot how to do the same process manually? It is better to halt all the flights than letting people through and risk "terrorists" flying? Are we that terrified?

Yes.

Re:I wonder (0)

Anonymous Coward | about 4 months ago | (#47578847)

It's not about halting flights, it's about issuing visas. The system issued 220,000 visas in about a week while running at half capacity, so the manual option is simply not possible. It would take months just to hire and train the necessary staff. Plus, where would you put them?

Re:I wonder (2)

Kittenman (971447) | about 4 months ago | (#47579229)

That these breakdowns are lame excuses. If computers fails, have people forgot how to do the same process manually? It is better to halt all the flights than letting people through and risk "terrorists" flying? Are we that terrified?

You could just ask the questions that used to get asked back in the '50s. "Do you intend to bring down or otherwise defame the US government?"

Really. A UK humourist (Frank Muir?) wrote "Sole purpose of visit" on the form.

Re:I wonder (1)

dbIII (701233) | about 4 months ago | (#47579721)

If computers fails, have people forgot how to do the same process manually?

Yes.
As an example. I've been rushed to a steel mill rolling line with a pocket calculator because the operators were not taught how to divide the number on the dial of the test machine by the cross sectional area of the rod that they had measured the diameter of - they were just told to manually enter those two numbers into the computer. By knowing how to calculate the area of a circle I was saving downtime of hundreds of thousands of dollars per hour - which is pathetic on so many levels since the operators also knew how to find the area of a circle but nobody had told them that's what the computer was doing. They had only been told that the process was to put two numbers into a computer.

It's not about stupid operators/counter staff it's about relying on fragile links with no workaround when they break. There's also the fear of taking responsibility - it's seen as safer for the supervisor of all those staff to say "nothing should be done" than to work around the problem.

Not just the passports (2, Insightful)

Anonymous Coward | about 4 months ago | (#47578381)

The whole US customs and immigration system is massively dysfunctional. Last year I flew into Minneapolis from Asia. I'd been traveling for twenty hours straight and then I got to stand in line for a full hour waiting for an immigration agent to spend ten seconds looking at my passport photo to make sure it matched my face. Even the third world airports I've been through aren't that bad. There were even empty stations without agents. How much would it have cost to add a few more agents - $100? At the time they were doing this ridiculous upgrade to the airport that must have cost millions - they were setting up all these silly little tables with ipads in the waiting areas. But somehow they couldn't manage to have enough immigration agents. It made me wonder if people in the state of Minnesota are as silly as their ariport - they did elect Michelle Bachmann to congress - so there may be quite a few of them who were dropped on their heads as babies or something.

Re: Not just the passports (0)

Anonymous Coward | about 4 months ago | (#47578509)

Same happens all the time at JFK. They are simply understaffed for peak events

Re: Not just the passports (0)

Anonymous Coward | about 4 months ago | (#47578751)

Used to have visitors fly in to Minneapolis from Europe. At almost the exact same time, there was another large flight from Asia arriving. About 50/50 whether the passangers from the Scandinavian flight would make it to the customs/immegration before the asian passangers. Only time I ever felt like i should be bribing the Air Traffic Controller guys was when standing at the international arrivals door and see the first passenger to come out not wearing socks in their sandals and looking awefully pale. At that point you might as well drive home and take a nap...

Re:Not just the passports (0)

Anonymous Coward | about 4 months ago | (#47578591)

They also elected Stuart Smalley as their US Senator, so maybe you have a point.

Re:Not just the passports (1)

CRCulver (715279) | about 4 months ago | (#47578701)

At the time they were doing this ridiculous upgrade to the airport that must have cost millions - they were setting up all these silly little tables with ipads in the waiting areas. But somehow they couldn't manage to have enough immigration agents.

Airport infrastructure is typically managed by a consortium with private or state-level involvement. US customs officials are federal employees. The two parts of the airport have entirely different funding sources.

Re:Not just the passports (1)

Anonymous Coward | about 4 months ago | (#47578785)

The two parts of the airport have entirely different funding sources.

In a sane world, the airport director would phone up Michelle Bachmann, explain that it reflects poorly on the people of Minnesota for international travelers to be having a 3rd world immigration experience, that many international travelers have paid thousands of dollars for their tickets while extra immigration agents can be had for something like $10/hour, and then Michelle Bachmann would get together with her buddies in congress and they would have the situation fixed by the end of the day. But, thanks to all this nonsense about the glory of privatization and local government, Minnesota is stuck with a third world mess of an airport.

97% availability and getting worse (0)

Anonymous Coward | about 4 months ago | (#47578507)

Watch as government system availability falls in ways previously deemded "unlikely". Currently, 97% at most (http://en.wikipedia.org/wiki/High_availability#Percentage_calculation) and perhaps approaching the 66% US government systems are known for. All they need is more money for more warm bodies to make better mistakes, otherwise it wouldn't be proper welfare to have a government job.

Here's the problem... (2, Funny)

Anonymous Coward | about 4 months ago | (#47578553)

I think I found the problem, from the Department of State's own website:

"The Department of State is working with Oracle and Microsoft to implement system changes aimed at optimizing performance and addressing ongoing performance issues."

They're running Oracle on Windows.

Re:Here's the problem... (4, Funny)

AJWM (19027) | about 4 months ago | (#47579237)

Or worse, they're running SQLServer on Sun boxes...

A Welcome to The Crash (1)

Mister Liberty (769145) | about 4 months ago | (#47578685)

I have arrived at the point where any crashes experienced by whatever State Department of whatever so called and self proclaimed Democratic Country (traitor mark here) are welcomed by me with the utmost glee. The more disruption, the more chances for a turnaround.

Management FAIL (0)

Anonymous Coward | about 4 months ago | (#47578693)

somewhere there was a choice made that was a poor one.

no back out plan.
no test environment.
poor choice of vendor (software or hardware )
bad application/database config/design

speaking as a system admin, my bet would be a management person, that was promoted.
And was promoted for getting this system installed under budget !
Then left the company for more money.

my guess (0)

Anonymous Coward | about 4 months ago | (#47578743)

is this is something to do with Russia and Ukraine

Mission critical failure (1)

Anonymous Coward | about 4 months ago | (#47578777)

And how many families will be disconnected because of this?
How many jobs will be lost when people can't get back to work?

It's all nice conveniently glossing over the fact that people can't get home but they have lives to live, schedules to meet and contracts their obligated to perform under. You can't just say "Oh, sorry. You can't come back. Try later." The real world doesn't accept "Try later" as an excuse.

Two Reasons for this Fuck Up (0)

Anonymous Coward | about 4 months ago | (#47578835)

Two reasons for this fuck up come to mind immediately:

The Department of State is working with Oracle and Microsoft to implement system changes aimed at optimizing performance and addressing ongoing performance issues.

Of course, no one would expect the State Department to use anything else, but given what they're running there should be little surprise that an update caused unexpected, catastrophic, inexplicable and seemingly irreparable performance issues.

The System is Down! (0)

Anonymous Coward | about 4 months ago | (#47578861)

http://www.homestarrunner.com/systemisdown.html

Major Threat (0)

Anonymous Coward | about 4 months ago | (#47578955)

"The system 'crashed shortly after maintenance.'"

Payback for the DoJ's request of e-mails from a server in Dublin of a person of interest which bcc'ed Bill Gates.

M$ targeted the DoS servers and fead them malware and hack disguised as an Update.

The DoS's Visa and Passport databases have now been subversively transferred to Al Quida servers.

This will tie-up airline flights as every passenger on all US (out and in) flights must be co-verified on the DoS server databases through DHS.

Rough seas ahead.

Large Databases? (4, Insightful)

TechyImmigrant (175943) | about 4 months ago | (#47579055)

The article tries to wow us with the hugeness of the database, like this is a reason for the issues.

Yet the numbers quoted are not that big. Any modern PC isn't going to get too upset handling 75 million things. A real data center is going to sit there wondering what to do with the remaining 500TB of storage.

I don't doubt that there is some horrible flaw in the way the system was conceived that rendered it fragile, but whatever it is, it's nothing to do with the enormity of the problem, because it isn't very enormous.

Re:Large Databases? (0)

Anonymous Coward | about 4 months ago | (#47579609)

It's Oracle, enough said sadly.

Meanwhile the Mexican border (1)

mtthwbrnd (1608651) | about 4 months ago | (#47579181)

remains completely open and unguarded.

Computer Crash? (0)

Anonymous Coward | about 4 months ago | (#47579207)

Who let Lois Lerner in the State Department's office?

I understand that the database is large... (1)

Anonymous Coward | about 4 months ago | (#47579281)

I understand that the database is very large by any measure: 100 million records, 75 million pictures. But social security databases in most countries (or income tax databases) are at least that large (ok, likely much larger). Its a fail to have a large database really tank like this. If you need to shut the whole thing down for a day to avoid corrupt data, then shut it down. Fixing a corrupt database is much more difficult than correctly shutting a (slow) one down and then bringing it back up again.

1970s design mentality? (1)

narnian (9597) | about 4 months ago | (#47579681)

While it doesn't always go this way, often simple things like the User Experience of a business gives an indication of the ethos behind a whole lot of the processes and systems they are using. To wit, compare the US Arrivals card that all "aliens" need to use upon arrival into the US, with the one from Australia. A clear 1970s look-and-feel versus something from this millenium.

http://www.immihelp.com/visas/sample-i94-form.pdf

http://www.immi.gov.au/managing-australias-borders/border-security/travel/passenger-cards/_pdf/english-ipc-sample.pdf
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?