×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Post Mortem of GunnAllen IT Meltdown

Unknown Lamer posted about a year and a half ago | from the but-it-was-business-class-cable dept.

Security 192

CowboyRobot writes "The story begins when GunnAllen, a financial company, outsourced all of its IT to The Revere Group. Before long, it was discovered that 'A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem.' In addition to the obvious security concerns of sending information such as bank routing information and driver's license numbers, the act violated SEC rules because the routed information was not being logged. Regardless of whether the cause was negligence, incompetence, or sabotage, the matter was swept under the rug for a time until unpaid SQL Server licenses meant threatening calls from Microsoft as well. The rest of the story is one of greed, mismanagement, and neglect, and ends with the SEC's first-ever fine for failure to protect customer data."

cancel ×
This is a preview of your comment

No Comment Title Entered

Anonymous Coward 1 minute ago

No Comment Entered

192 comments

HAHA (-1)

Anonymous Coward | about a year and a half ago | (#41592799)

EOM

Re:HAHA (0)

Anonymous Coward | about a year and a half ago | (#41592831)

I agree. This is the funniest thing I've heard today. Through his home cable modem! HAHAHAHAHAHA!

Re:HAHA (2)

Frosty Piss (770223) | about a year and a half ago | (#41592851)

Well, you know, he had RoadRunner... In 2005, that was pretty wicked! If he had set up two or three accounts and load balanced them...

Re:HAHA (5, Insightful)

El Puerco Loco (31491) | about a year and a half ago | (#41592913)

'A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem.

That's got to be the funniest thing I've ever read on /. Seriously, it sounds like something from an Onion story.

Re:HAHA (5, Interesting)

cbhacking (979169) | about a year and a half ago | (#41593115)

It's not mentioned in the summary, but the first sign of the rerouting was, as you'd expect, their network slowing to a crawl. That earned the IT guy responsible for it a reprimand. A reprimand, for routing an entire company's trading data through his home modem for a week!

There's other gold in there too, like the time the guy pulled the cable on a production rack in order to create a catastrophe so he wouldn't have to travel to a business meeting, or his habit of remoting into IT infrastructure (Blackberry and Exchange servers were mentioned) on the weekends to fuck up their configuration, just so he could "magically" fix it on Monday morning.

He was, apparently, eventually fired.

Re:HAHA (5, Funny)

the_B0fh (208483) | about a year and a half ago | (#41593133)

I worked at a place where the Exchange admin - every so often - would have to heroically worked 72 hours or whatever to rescue the mail servers and we only have 2 days of downtime, etc etc, and the CIO would praise him for his hardwork.

I asked my boss if I should also reboot the firewalls every now and then - just to heroically bring them back up again, and get thanked for my hardwork. He gave me a nasty look...

Re:HAHA (3, Informative)

Anonymous Coward | about a year and a half ago | (#41593207)

If not given the resources to have Exchange load balanced, and if it happens to crash and requires a 200GB Store restored...72 hours sounds about right. The 2 days downtime should have been 4 hours (time to investigate and bring a backup VM online). Without a backup VM, it should have been down 1 day.

jeez, exchange is still used? (0)

cheekyboy (598084) | about a year and a half ago | (#41594283)

Yeah yeah we know it does work, mostly, and is probably written in VBscript or cobol.

But damn, you can afford a EX licence, but cannot afford a high end intel 512G SSD x 2.

Restore in 5mins.

Hardrives, puhhhh.... so 90s, like C64 tapes. Get with the future dude.

Re:jeez, exchange is still used? (4, Interesting)

PsychoSlashDot (207849) | about a year and a half ago | (#41594823)

Yeah yeah we know it does work, mostly, and is probably written in VBscript or cobol.

But damn, you can afford a EX licence, but cannot afford a high end intel 512G SSD x 2.

Restore in 5mins.

Hardrives, puhhhh.... so 90s, like C64 tapes. Get with the future dude.

Sure. So you restore in minutes but that's when you realize that the information store is - by definition - backed up dirty because it's in use. A moment later you discover that Exchange insists on you running some nice ISINTEG routines to mark the database as clean before it can be mounted. Those routines joyfully take a minor eternity, even on SSD if you have a huge database. Like... 450G. When you're done with ISINTEG, if you're really lucky you can have a bonus round of ESEUTIL followed by ISINTEG again if it turns out there was any minor database structural issues you didn't know about.

High I/O absolutely helps, but don't write this off as if massive database restores are trivial just because someone follows your advice. For businesses that are big enough to accrue huge amounts of data but not big enough to afford redundant servers, TIME is the cost they pay.

OOHHH GOD!! (0)

higuita (129722) | about a year and a half ago | (#41594935)

OOHHH GOD!!

WHY, but WHY people still use that exchange garbage!! With so many exchange replacements, so many webmails, so many SAAS alternatives... WHY!?

At least they pay the (heavy) price for it! (money, work hours, never ending troubles)

Re:HAHA (0)

Anonymous Coward | about a year and a half ago | (#41593585)

Fuck exchange for stuff the like this, the worst part is that it probably did it on its "own". Granted, it shouldn't take 72 hours, but a good 8 can occur if you have an entire raid array failing or something, coupled with a bad MS update and someone who tried to install Office 2007 on the CAS.

Don't be too hard on them (1)

dbIII (701233) | about a year and a half ago | (#41594377)

MS Exchange is difficult to care for from what I've seen and the competence or otherwise of the people that look after it doesn't seem to spare such dramas from what I've seen. The experienced seem to run several MS Exchange servers (even in small places of 100 users where a 300MHz machine with Sendmail would do the job) that way the blowups and disasters may happen on one server but the mail still gets through on another.
It's a shambolic pile of services and applications loosely stuck together with gum, and there was no reliable way to get usable backup without stopping it (ie. the entire fucking thing to put on a new server and not just a portion of the mailboxes), until volume shadow copy came around - the MS Exchange programmers never supplied what every other MTA provided on first release!
You probably do need to be a hero to keep a single instance of it running.

Re:HAHA (4, Insightful)

dbIII (701233) | about a year and a half ago | (#41594327)

However no jail time. Refusing to disclose a password in case it's used by such an incompetent carries jail time, but being deliberately criminally incompetent does not. It's a pretty nasty lesson we are teaching the next generation.

Re:HAHA (1)

1s44c (552956) | about a year and a half ago | (#41594619)

There's other gold in there too, like the time the guy pulled the cable on a production rack in order to create a catastrophe so he wouldn't have to travel to a business meeting, or his habit of remoting into IT infrastructure (Blackberry and Exchange servers were mentioned) on the weekends to fuck up their configuration, just so he could "magically" fix it on Monday morning.

He was, apparently, eventually fired.

Wha!??

What was this guy? The Harold Shipman of IT?

Re:HAHA (2)

ackthpt (218170) | about a year and a half ago | (#41593367)

'A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem.

That's got to be the funniest thing I've ever read on /. Seriously, it sounds like something from an Onion story.

He probably cooked lobsters in his dishwasher, too.

Re:HAHA (0)

Anonymous Coward | about a year and a half ago | (#41593853)

Apparently Scott Adams can retire, even the pointy haired boss couldn't top this.

Re:HAHA (1)

mlush (620447) | about a year and a half ago | (#41594713)

'A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem.

That's got to be the funniest thing I've ever read on /. Seriously, it sounds like something from an Onion story.

The thing I'm really struggling with is why on Earth would anyone do such a thing

Re:HAHA (1)

leonardluen (211265) | about a year and a half ago | (#41595581)

the summary says:

the act violated SEC rules because the routed information was not being logged.

are they sure he wasn't logging the data?

Trusted Advisor? (3, Informative)

Frosty Piss (770223) | about a year and a half ago | (#41592803)

Wow, according to the The Revere Group website:

WHEN TRANSFORMING THEIR BUSINESS, TOP PERFORMERS TURN TO A TRUSTED ADVISOR

Guess that's not The Revere! Group

Re:Trusted Advisor? (0)

Anonymous Coward | about a year and a half ago | (#41593149)

Been had a little horsy named Paul Revere
Just me and my horsy and a quart of beer
Riding across the land, kicking up sand
Sheriff's posse on my tail cause I'm in demand

Outsourced (5, Interesting)

Anonymous Coward | about a year and a half ago | (#41592841)

Yeah keep outsourcing the responsibility of something so crucial that IT people hold the keys to the kingdom.
This is nothing new in the world of IT. Save a dime to lose a million dollars.
I am in a comany right now where they hired IT consultants for well over 3 years and come to find out so called "Experts" where just patching the system but never really fixing the real issues. It's amazing to see what these contractors were selling to a company who had the money to buy great gear only to discover pure incompetence at implementing it. I am no expert by any means but I can smeel bullshit when I see a network in need of a lot of TLC.

Re:Outsourced (1)

bbelt16ag (744938) | about a year and a half ago | (#41592881)

indeed, no expert here either, but i know enough to be dangerous. this is a complete lack of understanding of even the basics of what is going on in their networks, or how to identify who is a good engineer or a bad one.

Re:Outsourced (0)

Anonymous Coward | about a year and a half ago | (#41592919)

IT doesn't make money. It COSTS money! Why should shareholders invest in IT when they receive no return on investment like you would with top talent or a bigger sales and marketing force to increase its value?

They are a finance company. Not an IT one so wouldn't an outsourcer provide a better job as it is its own strength?

Re:Outsourced (5, Insightful)

AK Marc (707885) | about a year and a half ago | (#41592991)

For the same reason they don't oursource their upper management. After all, CEOs cost money, why not outsource CEO to a management company and cut costs. After all, they are a finance company, not a management company, so all their management should be outsourced.

Re:Outsourced (0)

Anonymous Coward | about a year and a half ago | (#41593077)

Search Gogle for management consulting.

Re:Outsourced (4, Insightful)

AK Marc (707885) | about a year and a half ago | (#41593191)

Consultants are often used for outsourcing blame, rather than outsourcing capability. "Oh, our consultant recommended that."

Re:Outsourced (3, Interesting)

Anonymous Coward | about a year and a half ago | (#41593241)

With the revolving door nature of CEO and other top level jobs these days, you could argue that upper management is already outsourced away from the actual company. Just that they compete on paying the most instead of the least.

Re:Outsourced (0)

Anonymous Coward | about a year and a half ago | (#41593249)

No CEOs through their billiant ideas are what make products and things happen. Look at Nokia and Microsoft right now for what a bad CEO will do? An IT guy ... well he just plays with computers and costs money. Not making those sales or boasting share prices. Accountants do not know how to do IT stuff. IT companies do.

Re:Outsourced (4, Insightful)

JDG1980 (2438906) | about a year and a half ago | (#41593105)

They are a finance company. Not an IT one

If you run any business beyond the level of a mom-and-pop restaurant, you are in the IT business whether you want to be or not. The only question is whether you will leverage IT as a strategic asset or be outcompeted by those who do.

Re:Outsourced (4, Insightful)

Rhinobird (151521) | about a year and a half ago | (#41593223)

Eventually the people in charge are going to realize that any kind of financial institution is basically a database on the internet that holds and exchanges account information. And then they're going to turn ghostly white as they realize all these strangers are touching the equipment that, in a very real sense, IS the bank, er, financial whatever...or worse, those strangers OWN the equipment that IS the financial gobstopper.

And then, at least in finance, outsourcing IT will be seen as a form of insanity.

Re:Outsourced (0)

Anonymous Coward | about a year and a half ago | (#41594665)

If your system is down for a week because you outsourced your experienced IT and replaced them with cheap idiots, how much does that cost your business in terms of lost/delayed orders, pissed off customers, stock dives, and the finger-pointing blame game?

Re:Outsourced (0)

Anonymous Coward | about a year and a half ago | (#41594999)

I dread working with these companies, they have one outsourcing group for the desktops, one for the servers one for the network and another for the firewalls and load balancers. It is a pointing party every time we work on something.

Sigh... (4, Interesting)

Black Parrot (19622) | about a year and a half ago | (#41592847)

A financial company outsourcing its IT ought to be considered criminal negligence.

(Though an own employee could do the same thing, in this case.)

Re:Sigh... (5, Insightful)

DigiShaman (671371) | about a year and a half ago | (#41592973)

Agreed. I work in the MSP (Managed Service Provider) sector which is a fancy way of saying that we are outsourced IT. We focus on the SMB market where a company is too small to have a dedicated IT department, but just large enough that they place a trouble ticket in our queue once a week. Sometimes once a day. Anything ranging from tier 1 to 3 support.

However, once you as a company get involved with needing to be HIPPA, PCI, or SOX compliant, that should be synonyms with "dedicated in-house IT dept".

Re:Sigh... (2)

Charliemopps (1157495) | about a year and a half ago | (#41593101)

I'd have to disagree. We have our own in house IT department... but a small part of our business is providing outsourced IT. And our stuff ridiculously overbuilt and robust. I doubt anyone could do it in house better. But it's expensive as hell, and not very flexible. If you're not getting too creative with your needs, and you have the money, you can get something very robust. But if you want to go on the cheap and still get some crazy ass system no ones ever tried before to work, then I think you're shit out of luck no matter who you go with.

Re:Sigh... (3, Insightful)

LordLucless (582312) | about a year and a half ago | (#41593503)

I'd have to disagree. We have our own in house IT department... but a small part of our business is providing outsourced IT. And our stuff ridiculously overbuilt and robust.

It's not about robustness in these instances. It's about power and accountability. When you have hugely sensitive information (medical records, credit card details or financial records) you must be in control of your own systems. While downtime sucks, downtime is often better than data compromise in these cases.

Re:Sigh... (1)

KingMotley (944240) | about a year and a half ago | (#41593593)

That sounds grossly naive. What company over the size of 0 employees doesn't have one of the following: Medical records, Credit card details, or financial records? Every single company has those, even companies that have 1 part time person in it. I seriously don't think there is enough IT professionals in the entire world worth a damn that you could have 1 at every single company.

Outsourcing isn't the problem with data breaches. Outsourcing to companies that back up their promises with financial guarantees and fines is the problem. These companies that do outsourcing need to have a well trained staff that can actually do security well and they need to have a vested financial interest in doing it well. You would find less data breaches by having all the worlds "sensitive" information in a few dozen companies who actual responsibility to keep to safe than to have it in hundreds of thousands of companies who are totally incompetent.

Re:Sigh... (2)

drinkypoo (153816) | about a year and a half ago | (#41594681)

That sounds grossly naive. What company over the size of 0 employees doesn't have one of the following: Medical records, Credit card details, or financial records? Every single company has those, even companies that have 1 part time person in it. I seriously don't think there is enough IT professionals in the entire world worth a damn that you could have 1 at every single company.

You're being grossly obtuse. We're talking about a bank here. They are directly responsible for customer data, and they are explicitly on the hook in the case of data breaches. It is a gross failure of responsibility not to maintain IT in-house when your entire business is built on IT, which is the case in banking today. They can't do anything for you if the computers are down, except take a deposit and give you a handwritten slip in exchange. And if I walk into my bank and their computers are down in this day and age of clustering and high availability, I'm probably going to go to some other bank and open an account with that check.

Re:Sigh... (1)

bluefoxlucid (723572) | about a year and a half ago | (#41595275)

It is a gross failure of responsibility not to maintain IT in-house when your entire business is built on IT, which is the case in banking today.

Why? Contractors are still people, just their payroll department is elsewhere. They live in your building, sit at your desks, use your computers. I mean hell, I worked at the Social Security Administration and most people at the NSA are contractors. Some of my coworkers WERE NSA at one job for a while; we worked in the same office, I wasn't cleared and they didn't work on secret projects in the same office because that office wasn't a secure room or else, you know, I wouldn't be allowed in it.

Re:Sigh... (1)

KingMotley (944240) | about a year and a half ago | (#41595395)

The article is specifically about a bank, but lordlucless wasn't speaking about the bank specifically. He took one example then expanded it to encompass basically every company on the planet.

As for your post:
1) Banks are hardly "built on IT". Not much more so than any other company out there. You walk into a store and the "computers are down" (this included McDonalds), and either they do the same thing, take your money have hand you bank a handwritten slip, or they are just closed. Like most companies, the computers make doing things there more efficient, but they are hardly necessary to actually complete the work. Banks existed before them, and if they disappeared off the face of the planet, they would still be around (after everyone rioted and burned them to the ground).
2) There is very little reason why the day to day activities of a bank and it's tellers etc need to be run by an in house IT. The bank isn't in business of buying/selling computers. Sure you need a highly secure network for the money transactions, etc., but are you suggesting that billy bob down IT guy who grew up down the street from the bank is better than companies with experts in the field that build highly secure banking networks 1000 times before?
3) Banks do a hell of a lot more than just process your deposits and withdraws and keep a tally on them. Most do investments (Stocks, Bonds, IRAs, Treasury Certificates, etc), loans (Personal, Car, Mortgage), credit cards (Visa, Mastercard), etc etc.
4) If I'm banking at a small mom and pop bank (Actually I don't, but I did for many years, they just got sucked up into Chase -- but there are many reasons for wanting a smaller bank), then as someone who actually worked on clustering and high availability systems (building, and writing clustering drivers), I'd rather they spend it on other more important things like backups. I can deal with the infrequent computers being down problem. What I don't want it my account hacked, or financial records being completely lost (unless of course, I only have a mortgage with them, then please lose everything!)

Re:Sigh... (1)

hairyfish (1653411) | about a year and a half ago | (#41593589)

I'd have to disagree. We have our own in house IT department... but a small part of our business is providing outsourced IT. And our stuff ridiculously overbuilt and robust. I doubt anyone could do it in house better. But it's expensive as hell, and not very flexible.

I bet you can't even see the irony of your post. If it expensive and inflexible then it's quite easy to do it better don't you think? The problem you haven't addressed is that every business has different requirements and not all of them require super-robustness. I worked both sides of the fence, and MSP has it's place but it isn't the solution for everyone (as TFA quite nicely demonstrates).

Re:Sigh... (1)

cbhacking (979169) | about a year and a half ago | (#41593181)

Actually, given the specific expertise and experience required for such compliance (at least, for doing it right), I can see an argument for specialized IT services companies that handle the needs of companies up to a certain size (bigger than you were talking about, though not necessarily by much; still too small to make it worth hiring a team of such people). The problem is, you've got to assign responsibility along with that contract. LOTS of responsibility, as in no-feasible-way-in-hell-you-could-save-more-money-from-negligence-that-puts-our-compliance-at-risk-than-you'd-have-to-pay-for-breach-of-contract levels of responsibility. If the outsourced company has a serious stake in the matter, then it shouldn't be a problem... yeah, they could still screw up and be grossly incompetent or have a malicious insider, but the same is true of in-house people.

Not that I disagree that outsourcing such critical roles is a terrible idea in general... but sometimes, it really is the only economically practical option, and that shouldn't mean you can't do business at all. Besides, just because the current way the outsourcing is done is broken, that doesn't mean you have to throw the whole idea out; it may be possible to fix it instead.

Re:Sigh... (1)

AK Marc (707885) | about a year and a half ago | (#41593309)

I worked for a VAR for a while, and we sold a lot of wireless gear to a hospital and set up the system, with HIPAA compliance and all. For a relatively large hospital. There wasn't much of an IT department. I think IT people don't like working for doctors, I'm not sure anyone likes working for doctors.

Re:Sigh... (5, Insightful)

girlintraining (1395911) | about a year and a half ago | (#41593259)

A financial company outsourcing its IT ought to be considered criminal negligence.

Outsourcing IT isn't the problem. A failure to oversee the IT services provided was the problem; A complete lack of auditing and process control. I wish people would stop looking at outsourcing as somehow evil; It makes sense in a lot of cases. Most corporations have other companies contracted to replace and maintain printers. Most office printers have the ability to retain all documents printed from it, locally, to a harddrive inside it. That isn't a problem by itself -- unless you don't know that the functionality is enabled, and don't audit or remove the drives before the printers are rolled out the front door with all your confidential data... that you thought was secure because you had a contract to shred all your documents.

The story of GunnAllen's criminal negligence starts with the CTO and board of directors -- who fired people for coming forward with security problems, and had a very obvious closed-door policy. Nobody with the parent company wanted to hear about problems, and it's no surprise that the firm they contracted with heard that loud and clear -- and propagated the same attitude right on down the line. "See no evil, hear no evil" often leads to a lot of people doing evil.

GunnAllen's story is one being repeated by the thousand every morning of every workday across our industry. Managerial incompetence leads to otherwise trivial problems becoming fines, bankrupcy, and lawsuits. This story is not about the failures of IT -- IT was involved, but it was not that failed. It was the people at the top... and when the extent of the damage was finally discovered by the government, they tried to pin it all on former employees and the people under them. I'd like to know where those managers are now; Because I know they'll eventually find themselves in another position of power at another company. Whereas all the engineers and people who actually worked for a living, well... we all know what happened to them, whether the article says so or not.

You want to fix problems like this: Start with accountability.

Re:Sigh... (2)

mjwx (966435) | about a year and a half ago | (#41594075)

Outsourcing IT isn't the problem. A failure to oversee the IT services provided was the problem;

Which is difficult to impossible to do unless you're directly managing the technicians. In which case, why are you paying another company A$200 an hour when the same techs would jump at being directly offered A$35-60 an hour (consulting rates in Oz).

So we're back to outsourcing being the problem. There may be more to it than that, but if you need 100% control, you cant get that by going through third party.

Re:Sigh... (1)

bluefoxlucid (723572) | about a year and a half ago | (#41595309)

In the US, it's $35-$60/hr plus the cost of benefits plus compliance with EEO laws plus payroll taxes plus you actually have to run payroll and accounting for all that instead of dumping a brick of cash into a line-item on your accounting.

Re:Sigh... (1)

drinkypoo (153816) | about a year and a half ago | (#41594669)

You want to fix problems like this: Start with accountability.

Yes, and you start with accountability by keeping your IT in house, where you have some control over the IT workers. In fact, outsourcing is primarily a vehicle for disposing of accountability; as long as the company you're outsourcing to claims responsibility, you get to avoid it. And then you have situations like this. Anyone outsourcing their IT is a dumbfuck. The only businesses who should ever hire anyone external to do any computer work are those whose business is too small to justify a full-time IT employee, because computing infrastructure is now critical to business.

It would have been a step in the right direction to audit the outsourced services, but you need an IT employee you can trust for that. And if you're only going to have one, is he going to sit around twiddling his thumbs any time he's not performing an audit for you? Or are you going to have an IT department of your own to handle the critical services upon which your business depends? It seems like a no-brainer to me, but then, it also seems like there's a lot of people with no brain, especially in management.

Re:Sigh... (0)

Anonymous Coward | about a year and a half ago | (#41594761)

Outsourcing IS evil, because it is creating a giant race to the bottom where quality surrenders to profit. As long as there is a large income disparity between countries, companies will try to ship their "costs" over there to save a buck, never noticing or caring that it is costing them in so many other ways.

I would argue excessive, unregulated outsourcing is one of the largest reasons the world economy is in the toilet right now.

Coincidentally, I don't disagree with you that accountability is another huge problem. But it is pretty related.

Re:Sigh... (2)

slashmydots (2189826) | about a year and a half ago | (#41593323)

A financial company outsourcing its IT ought to be considered criminal negligence.

(Though an own employee could do the same thing, in this case.)

I worked at a hospital with around 1000 computers and IT was onsite but contracted from a 3rd party. So, that's odd but get this! They outsourced the support calls to Mexico! Yeah, you could walk right down to the damn IT office yourself on floor 1 and get your problem taken care of or you could call Mexico. You could even simply get an extension of someone in IT and call that...or call Mexico! MEXICO! AT A HOSPITAL! By the way, I was there on a 6 month PC replacement project from a different contractor that the other contractors hired. Oh and they all got fired 4 months later when the hospital didn't renew their contract.

BOOM! (0)

Anonymous Coward | about a year and a half ago | (#41592871)

What a clusterfuck!

Seriously, this seemed like a good idea? (0)

Anonymous Coward | about a year and a half ago | (#41592903)

GunnAllen, a financial company, outsourced all of its IT

I think I've found the first problem.

Astonishing Criminal Acts and Federal Negligence (-1)

Anonymous Coward | about a year and a half ago | (#41592937)

'President' Obama confers onto himself the rights of:

1) Kidnap Without Cause

2) Rendition Without Cause

3) Torture Without Cause

4) Murder Without Cause.

Yet, the Federal Communications Commission stands ... In the Shadows ... Hapless ... Without A Brain ... even with credible intelligence.

What a Circus of the Absurd This United States of America Executive Office and Departments ... The Curse Of The UnElected.

Re:Astonishing Criminal Acts and Federal Negligenc (-1)

Anonymous Coward | about a year and a half ago | (#41594191)

I think you meant to say:
President' Bush confers onto himself the rights of:

I guess your memory is shot due to all of the hours you spent praying to a non existent entity

Wait a minute... (5, Funny)

damn_registrars (1103043) | about a year and a half ago | (#41592985)

Are you trying to tell me that the SEC has rules? That they enforce? I don't believe this. This does not reflect the US that I live in; are you perhaps talking about some other country with more reasonable laws about this kind of thing - maybe you meant to say it happened in Armenia, not America?

Re:Wait a minute... (3, Interesting)

jamstar7 (694492) | about a year and a half ago | (#41593123)

Of course the rules get enforced, if you're small enough to where you can't outlawyer the Feds. Why you think none of the big brokerage houses faced prosecution? For every lawyer the DoJ fielded, the brokerages fielded *5* or more.. And it didn't help that a Republican-controlled Congress cut their funding to the point where the DoJ was damned near useless.

Re:Wait a minute... (3, Interesting)

khallow (566160) | about a year and a half ago | (#41593461)

And it didn't help that a Republican-controlled Congress cut their funding to the point where the DoJ was damned near useless.

Even with funding, the DoJ would be pretty useless. I'll just trot out the current Republican talking points about Fast and Furious since they'll illustrate a good reason why the Republicans wouldn't be inclined to fund the Department of Justice.

Here, you have a pretty much cut and dry case. ATF agents allowed roughly two thousand fairly high quality guns to pass to Mexican drug cartels with no attempt made to track those weapons. Since those weapons have turned up at many crime scenes, including the murder of a US border agent (which is what finally shut down Fast and Furious). Further, the ATF agents involved knew for a few months before that final murder that these weapons were turning up at crime scenes, including murders. So a prosecutor has a pretty good case that someone committed a bunch of acts of accessory to murder (with reckless disregard for human life) and other crimes, plus the murder of a federal law enforcement officer. So what is the Department of Justice doing with this case? Hiding the agents involved in Washington DC. When will they investigate this?

This is why the "more funding" argument doesn't work. If the Department of Justice isn't going to do its job, then it doesn't really matter how much they're paid so might as well make it a little rather than a lot. The SEC is particularly notorious for providing the illusion of security for novice investors, or in other words, helping keep the marks from getting scared off before they can be fleeced.

Re:Wait a minute... (1)

CodeBuster (516420) | about a year and a half ago | (#41593565)

I think that both of you are missing the essential mater. While it's true that the SEC reserves criminal prosecution for the most egregious cases, relying more upon fines and plea bargaining, it can also be argued, and indeed it has been, that this general strategy really is the most effective use of limited taxpayer resources; allowing the most correction to be achieved for the tax monies spent. Sure, you could increase the enforcement budget of the SEC and expand the number of prosecutors, investigators and associated support staff but what would that accomplish? The courts dockets are already jammed and even tripling the budget of the SEC would allow only a small fraction of additional cases to be investigated and prosecuted. Meanwhile, the US government is still drowning in debt with no viable long term policy to put the financial house in order. Could the SEC do better with what they're given? Probably. Is spending a majority of agency resources on a few high profile prosecutions each year, while letting many smaller fish pass untouched, in the best interests of the American people and the investing public? Probably not.

The Moral of the Story (1)

DingerX (847589) | about a year and a half ago | (#41594079)

So, this brokerage was set up as a flag of convenience fifteen years ago and, to all appearances, operates as a loose federation of unchecked agents. One broker is charged with defrauded his clients, assigning all profitable trades to his wife, and all losses to the client. Another gets busted in a massive Ponzi scheme involving retirees and refinancing. Only when they're on the ropes does the SEC come looking at their IT operation, outsourced, from what I can see in the article, via an obvious conflict of interest to a "see-no-evil" boss and a pathological engineer. And the SEC only finds the very tip of the problem.

And that's the only time the SEC fined anyone for IT breeches of customer confidence.

Sleep well, America.

Woe is me (0)

Anonymous Coward | about a year and a half ago | (#41593095)

Woe is me, shame and scandal in the family
Woe is me, shame and scandal in the family.

Sabotage (1)

girlinatrainingbra (2738457) | about a year and a half ago | (#41593127)

It seems a lot like "Backdraft" [wikipedia.org] , the movie in which the fireman is also the firebug arsonist.

_

The network engineer was sabotaging the system by logging in during the middle of the night and breaking servers such as the Blackberry server, etc., so that he could come in during the morning and be the hero by fixing everything as quickly as he wanted.

"The network would get screwy over the weekend ... then [he] would show up, and five minutes in on a Monday, he'd fix the problem," Saccavino said.

He got caught when they sent a different level of IT person to investigate the network slowdowns and who used a keylogger to catch the shenanigans.

_

The saboteur network engineer was also plain ol' lazy, he's also accused of

"purposely pulling a cable out of a production environment in order that you would not have to travel to Jacksonville to attend an HP event at the request of the CIO." As a bonus, Microsoft also threatened to revoke their licenses for their version of SQL because, get this, the CIO had not gotten around to paying the license fees. That part seems to be a management problem, and not the network engineer's fault. But obviously, if this is the first time for a stand-alone SEC fine, then there were very crazy things going on at this company.

Re:Sabotage (0)

Anonymous Coward | about a year and a half ago | (#41593175)

I can't stand it I know you planned it
I'm gonna set it straight, this Watergate
I can't stand rocking when I'm in here
Because your crystal ball ain't so crystal clear
So while you sit back and wonder why
I got this fucking thorn in my side
Oh my god, it's a mirage
I'm tellin' y'all it's sabotage

Re:Sabotage (1)

AK Marc (707885) | about a year and a half ago | (#41593327)

It seems a lot like "Backdraft", the movie in which the fireman is also the firebug arsonist.

way to ruin the ending, no spoiler alert.

But then, you could have used plenty of real-life examples, including firemen. http://en.wikipedia.org/wiki/John_Leonard_Orr [wikipedia.org]

Re:Sabotage spoiler (1)

girlinatrainingbra (2738457) | about a year and a half ago | (#41593537)

Sorry for the spoiler without the alert! ;>)

_

I meant to find a real example of another lazy network tech., sabotaging for the sake of self-aggrandization or for getting out of work, but I couldn't find an example easily, or think of the search-terms that would do it. ("Self-aggrandization" didn't lead to much..., though there are some good reads like http://www.metafilter.com/88359/Not-enough-women-have-what-it-takes-to-behave-like-arrogant-selfaggrandizing-jerks [metafilter.com]

http://www.shirky.com/weblog/2010/01/a-rant-about-women/ [shirky.com]

http://www.computerworld.com/s/article/9034438/Former_network_engineer_faces_jail_time_for_sabotaging_patient_data [computerworld.com] ) but that last one is more of a criminal sociapath.

. And there was the San Francisco City Network administrator who refused to hand over his password, even to his boss or the mayor until he was taken to court on a criminal charge.

If you know any other good tech example, I'd love to know about it.

Corruption in finance? Unpossible! (0)

Anonymous Coward | about a year and a half ago | (#41593163)

Or not... it seems from the small investment firm to the core of our financial systems are manned by corrupt, lazy, money-comes-through-grift-not-work types.

Negligence, Incompetence, or Sabotage? (5, Interesting)

techsurvivorman (2747221) | about a year and a half ago | (#41593215)

I say Sabotage. I'm presently a NOC engineer at an IT managed services provider. Before, I worked for a well-known financial market data provider. The most demanding client we have is a financial company. Everyone once in a while, they get unhappy with our service for whatever reason and decide to blast the blame-thrower. During the most recent hissy-fit episode, they threatened to not renew the service contract. Moreover, their CIO dropped in on the conference call and said not only are they not gonna renew the contract but he was gonna have us blacklisted with other financial companies that we were looking to grow business with. It's been my general impression that financial clients tend to be some of the most high maintenance, demanding, and nasty assholes. I've a hunch that a similar reason could be a factor In explaining this network engineer's actions.

Re:Negligence, Incompetence, or Sabotage? (1)

Billly Gates (198444) | about a year and a half ago | (#41593271)

Give the finance company credit? They are rich because they are dirt cheap and compensate their profit centers well. They find the the best bang for the buck and punish those who under deliver. I know it sucks for you as these guys demand metrics and have 1 guy support 1,000 users (I know I interviewed for Citigroup and turned them down after learning about that) but that is how they get rich.

It sucks on your end but on the other end you always get great service by demanding more for less.

Re:Negligence, Incompetence, or Sabotage? (4, Insightful)

GSloop (165220) | about a year and a half ago | (#41593539)

It sucks on your end but on the other end you always get great service by demanding more for less.

I have news for you. People have the most ingenious ways of paying back arseholes. Thus, you don't always get great service by demanding more for less.

As a matter of fact, you may [meaning almost certainly WILL] get pretty bad service when you treat people badly - by continually demanding more for less, past the point of reasonableness and fairness.

Re:Negligence, Incompetence, or Sabotage? (0)

Anonymous Coward | about a year and a half ago | (#41593319)

"It's been my general impression that financial clients tend to be some of the most high maintenance, demanding, and nasty assholes"

It's such a surprise that Wall Street are entitled pricks!

Re:Negligence, Incompetence, or Sabotage? (-1)

tconnors (91126) | about a year and a half ago | (#41593769)

Moreover, their CIO dropped in on the conference call and said not only are they not gonna renew the contract but he was gonna have us blacklisted with other financial companies that we were looking to grow business with.

I'd blacklist you for using the non-word "gonna".

Re:Negligence, Incompetence, or Sabotage? (-1)

Anonymous Coward | about a year and a half ago | (#41593847)

I'm gonna use the hell outta that word, and there ain't gonna be nuthin your gonna do about it, jerkface!

Aww, what's a madder? Are you gonna cry? For all intensive porpoises, you done got pwned, noobhead!

RE:Lazy, stupid, or troll? (0)

Anonymous Coward | about a year and a half ago | (#41595457)

Gonna, nothin... okay;
Madder, intensive porpoises, pwned... lame.

Just for fun... (4, Informative)

Anonymous Coward | about a year and a half ago | (#41593257)

Go to http://www.reveregroup.com/ [reveregroup.com] and search for anything in the top right search box. You'll get a licensing error. These guys are on the ball...

Re:Just for fun... (1)

Alex Belits (437) | about a year and a half ago | (#41593667)

Error Message: The license does not allow the use of this search interface.

lol

Use the Coveo search box inserted in the upper-right corner

wtf

of your Sharepoint sites.

BWAHAHAHAHAHAHAHAHA!!!

Re:Just for fun... (2)

Chris Mattern (191822) | about a year and a half ago | (#41594271)

While it's not as out-and-out broken as their search box or twitter link, I also like their main page selection. Because everybody wants web navigation that induces motion sickness! Complete with mystery meat selections, too.

BOFH Strikes again (0)

Anonymous Coward | about a year and a half ago | (#41593261)

FTFA:
  "He'd purposefully break things, then come in in the morning and be the hero,"
  "purposely pulling a cable out of a production environment in order that you would not have to travel to Jacksonville to attend an HP event at the request of the CIO."

Outsourced IT will bring down companies (1)

Anonymous Coward | about a year and a half ago | (#41593325)

I'm in a decent position at my company. My particular skillset is luckily in decent demand, so I'm not worried if I do get outsourced.

However, I like my company. It has good benefits and the working conditions are not bad. We are looking at co-location of our data center and outsourcing some of our support.

The biggest problem I see is that the outsourcing company really sucks. Their engineers are crappy, have little skills, and know little about regulatory or other compliance requirements.

We have already begun to outsource some web development efforts to another company. Our internal IT had to bid against the external company. Apparently internal IT's costs and delivery date were not 'aggressive' enough. Long story short, the external company won the bid but are now at least two months behind and 50% over budget. That 50% translates to over $1M US. Not only that, the external company has pretty much ignored any compliance requirements (PCI, internal baseline standards, change control processes, etc.). Why can they get away with it when internal IT cannot? Simply because this is a critical project and normal controls are being relaxed. Yes, it makes absolutely no sense that the more critical a project is, the less it has to adhere to standards, but welcome to my company.

As I said, I like my company, but some idiots got sold on a vendor promise and we will end up paying for it in lost revenue and jobs.

disappointing (0)

slashmydots (2189826) | about a year and a half ago | (#41593329)

I mean it's disappointing that a title like that wasn't a story about someone from IT going completely berserk apeshit. It's bound to happen, lol.

Milton in the Middle (2)

Mr. Lwanga (872401) | about a year and a half ago | (#41593363)

Why would senior network engineer need to send traffic home to verify his routing patterns? Yeah right, he scammed millions and they covered it up to avoid more fines. Now, he and his red stapler, are at some beach resort complaining about the Mai Tais.

Second paragraph has all you need to know (4, Informative)

Stiletto (12066) | about a year and a half ago | (#41593505)

Over a period of roughly seven business days, traffic had slowed to a crawl at the Tampa, Fla.-based firm, which had outsourced its IT department to The Revere Group. GunnAllen's acting CIO, a Revere Group partner, asked a member of the IT team to investigate.

Well, here we go! The CIO of the company outsourced the IT department to..... his own personal company. No conflict of interest there!

Re:Second paragraph has all you need to know (2)

Chris Mattern (191822) | about a year and a half ago | (#41594237)

Not his own personal company; he was a Revere Group employee. At one point in the narrative one IT minion discusses how he went to the CIO's Revere Group superiors. When they outsourced IT they outsourced the CIO position along with it.

Unions can be a big help in stopping BS like this (5, Insightful)

Joe_Dragon (2206452) | about a year and a half ago | (#41593563)

Unions can be a big help in stopping BS like this from happening.

When you have people purposefully break things just to look good for the bosses that's bad even worse is sweeping security and other issues under the rug.

Re:Unions can be a big help in stopping BS like th (0)

Z34107 (925136) | about a year and a half ago | (#41594027)

Are you kidding? If he was union labor, they wouldn't have been able to fire him.

Re:Unions can be a big help in stopping BS like th (0)

Anonymous Coward | about a year and a half ago | (#41594213)

Are you kidding? Unions are the first to resort to such underhanded behavior. Just look at what happened at American Airlines. Some maintenance worker loosened up a bunch of seats, and bingo within a week the Pilot's union has a new contract after over a year of negotiating. Some coincidence!

Re:Unions can be a big help in stopping BS like th (1)

DNS-and-BIND (461968) | about a year and a half ago | (#41594901)

What about when the mafia who controls the unions comes around looking to get paid? What about when politicians looking for paybacks for favors granted to the union demand you employ 50 people who will collect paychecks and yet never show up for work?

Re:Unions can be a big help in stopping BS like th (1)

furytrader (1512517) | about a year and a half ago | (#41594955)

Come to Chicago sometime and you can see how helpful the unions are when it comes to running a business ... right out of Illinois.

Hard time reading train wreck stories (3, Insightful)

HangingChad (677530) | about a year and a half ago | (#41594031)

It's hard reading IT train wreck stories, especially when the damage is self-inflicted. And yet I saw that same attitude, on both sides of the transaction, acted out over and over.

A long time ago a CIO I worked for said he wasn't worried as long as he had a throat he could choke if things went sideways. The only thing he cared about was having somewhere to cast blame.

Those were the days I naively cared about doing a good job.

Re:Hard time reading train wreck stories (4, Interesting)

dbIII (701233) | about a year and a half ago | (#41594469)

A long time ago a CIO I worked for said he wasn't worried as long as he had a throat he could choke if things went sideways

There seems to be a lot of that attitude with the cloud outsourcing. I put an example up here earlier of 25k email accounts inaccessible for a week due to a DNS typo and a long job queue to do the two second fix, but people seemed to think it was OK to have that so long as there was someone else to blame. In that case it was Microsoft doing the hosting so good luck in getting anywhere with blaming them, a customer with twenty-five thousand email accounts is ignorable small fry and legal action is pointless.

Re:Hard time reading train wreck stories (1)

Anonymous Coward | about a year and a half ago | (#41594743)

I remember an issue with PCs used in controlling automatic equipment where the plant manager ranted about how we should 'Get Microsoft in here because we use 400 copies of their software." The tone of the meeting went downhill after all the IT folks, along with most of the others started laughing so hard they couldn't talk for at least 5 minutes.

Re:Hard time reading train wreck stories (4, Insightful)

Turminder Xuss (2726733) | about a year and a half ago | (#41595059)

The five stages of IT projects: 1. Wild Enthusiasm 2. Cold Reality 3. The Hunt for the Guilty 4. Bayoneting the Wounded 5. Promoting the Absent

Something isn't right with the story (-1)

Anonymous Coward | about a year and a half ago | (#41594643)

Had a senior network engineer actually routed their trades through his home cable modem, the plot would have been discovered instantaneously, because trades would not be fulfilled on time. Broker-dealers have ultra-fast, ultra-low-latency network paths direct to each exchange, usually with their trade servers in the same datacenter as the exchange's. Day-traders would have pitched an unholy fit instantly, given a low-throughput cable modem connection is going to be choked like a chicken on the morning the SI swimsuit issue comes out as trade confirmations were drawn out from seconds to minutes.

That alone says there was "something else" going on, because it could not have happened the way the summary describes.

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Sign up for Slashdot Newsletters
Create a Slashdot Account

Loading...