Become a fan of Slashdot on Facebook

 


Forgot your password?
Close
typodupeerror
×
The Almighty Buck Government

Knight Capital Fined $12M For a Software Bug That Cost $460M 192

Posted by samzenpus from the slap-on-the-wrist dept.
Mark Gibbs writes "Knight Capital monumentally fouled up a software update. According to the SEC, 'Knight did not have supervisory procedures to guide its relevant personnel when significant issues developed.' In other words, not only was Knight's code management inadequate but their human management processes were just as bad. The fine for what could have been a biblical financial disaster? A measly $12 million."
This discussion has been archived. No new comments can be posted.

Knight Capital Fined $12M For a Software Bug That Cost $460M More

Knight Capital Fined $12M For a Software Bug That Cost $460M

Comments Filter:

  • The Fine was $12 M, but, (Score:5, Insightful)

    by Anonymous Coward on Thursday October 24, 2013 @08:10AM (#45221929)

    The cost to them was $472 M. I *think* that will discourage them.

    • Re: (Score:3)

      by Chrisq ( 894406 )

      The cost to them was $472 M. I *think* that will discourage them.

      Law enforcement love to rub salt in the wound. A friend of mine had an accident that wrote-off a car. He had no MOT and the insurance investigators found that the car was unroadworthy. This meant that his insurance defaulted to "road traffic act only", which is the legal minimum and only covers third party injury claims. He had lost a car, and had to pay £2,000 for repairs to another vehicle.

      On top of that the police then sent him a court summons, where he was fined £30 and had three points on

    • It cost the shareholders $472 million. The men and women mismanaging Knight Capital drew their paychecks and benefits as before and didn't lose a dime in remuneration.

      The only lesson learned here was that no lesson was learned.

  • Maybe I am stupid, please explain. (Score:4, Interesting)

    by Anonymous Coward on Thursday October 24, 2013 @08:14AM (#45221953)

    They were FINED 12M, and they LOST 460M discovering the bug. This cost them a total of 476M.

    I am not understanding the outrage. Why should the SEC care if Knight Capital wanted to lose a big pile of money.

    • Re:Maybe I am stupid, please explain. (Score:5, Interesting)

      by Anonymous Coward on Thursday October 24, 2013 @08:29AM (#45222049)

      Why should the SEC care if Knight Capital wanted to lose a big pile of money.

      Because SEC has the political mission to portray the stock market as a rational, efficient, professional and socially useful apparatus - not the cesspool of mother-fucking sharks that it is. Hence all the trading rules which are selectively enforced so as to maintain the illusion without scarring the sharks away to, say, London.

    • I think you're missing a number... 460 + 12 does not equal 476, unless M is some weird number I'm not familiar with.

    • Because they broke the rules while losing that money. Do you think the SEC should only fine institutions that make money when they break the rules?

    • Re: (Score:2)

      by AlecC ( 512609 )

      As it happens, they lost an amount of money they could just about afford. But they could have lost more money than they had - leaving other people with losses in what is supposed to be a safe market place. It is like fining people for speeding - they have not caused an accident, but they are driving in a way likely to cause an accident. As it was, it nearly wiped out Knight: if it had gone further, it would have wiped out other, innocent, parties and done serious damage to the market, which the SEC is taske

  • compensation (Score:5, Interesting)

    by MickyTheIdiot ( 1032226 ) on Thursday October 24, 2013 @08:14AM (#45221959) Homepage Journal

    Can someone tell me why these financial institutions are never forced to compensate the *individuals* that suffer from these events?

    For instance in the mortgage fraud scandal they were allowed to settle fraudulent foreclosures for pennies on the dollar. Why are these companies never required to make the people they hurt whole again? Individuals that paid thousands of dollars simply got a small payment while banks just had to deal with "the cost of doing business."

    I think I know the answer (lobbying/congresscritters in their pockets) but I think it's one of the most scandalous aspects of the financial mess of 2008.

    • How about suing? Did those who were hurt sue? I bet they didn't sue.

      I'm saying that you don't have to wait for other people to do the right thing. In fact, I wouldn't wait at all. Go grab the appropriate paperwork and file a Small Claims case. At the 2,000 received court summons or so they'll just start settling as a default action.

      • Re:compensation (Score:5, Informative)

        by Capt.Albatross ( 1301561 ) on Thursday October 24, 2013 @08:55AM (#45222219)

        How about suing? Did those who were hurt sue?

        The customers have probably signed an agreement to settle disputes exclusively through binding arbitration - I believe this is an almost universal practice in the financial industry, as a condition for doing business, and no, you can't take your business elsewhere. That arbitration is widely regarded (at least outside of the industry) as being biased towards the financial industry.

    • A lot of times, it's because punishing them too severely would just hurt even more innocent people. It's the same reason that criminals with kids are sometimes given a somewhat lighter sentence or lesser fines so as to avoid putting the kids through even more undue hardship as a result of punishing the parent.

    • Because they're careful to write contracts that legally protect them from such lawsuits.

    • Re: (Score:2)

      by unrtst ( 777550 )

      Can someone tell me why these financial institutions are never forced to compensate the *individuals* that suffer from these events?

      Seriously? Because that would complete the loop, and someone has to be putting in more than they're taking out.

      Granted, "financial institutions" is a pretty broad term, so I'm not entirely certain which situation you are talking about, but since you mentioned the mortgage stuff, I'm currently assuming you mean the big thing involving the banks and the gov't bail out to them. Doesn't really matter which situation though... someone has to lose. It's just like Vegas - lots of individuals put in money hoping fo

      • Re: (Score:2)

        by sjames ( 1099 )

        The bailout should have had a lot more strings attached. For example it should have allowed us to put the brakes on the foreclosure frenzy that is still going on.It should have forbid any kind of management bonuses.

    • Can someone tell me why these financial institutions are never forced to compensate the *individuals* that suffer from these events?

      Because in capitalism the power is in the money and financial institutions have more money than individuals. You could as well ask for an explanation on why kings were not forced to compensate peasants in feudalism.

      The society where all individuals are equal doesn't exist. For as much as we know, it would have other inequalities and problems, but we cannot be sure because it doesn't exist and never has.

    • Re:compensation (Score:5, Informative)

      by Capt.Albatross ( 1301561 ) on Thursday October 24, 2013 @09:16AM (#45222381)

      For instance in the mortgage fraud scandal they were allowed to settle fraudulent foreclosures for pennies on the dollar. Why are these companies never required to make the people they hurt whole again?

      I don't intend to defend Knight Capital, but there is a big difference between its incompetence and negligence in this case, and the deliberate and fraudulent actions that characterized the mortgage mess. No individual in the financial industry has been held accountable for these actions, even while some of the people they exploited have been prosecuted: http://www.nytimes.com/2011/03/26/business/26nocera.html?_r=1&pagewanted=all [nytimes.com] [In Prison for Taking a 'Liar Loan' - Joe Nocera - NY Times; may require registration, or try reaching it through a search engine.] This, I believe, is the "most scandalous aspects of the financial mess of 2008."

    • The schools teach that if you vote in an election, then your interests will be represented by your elected officials. Informed adults know that's such a steaming crock, but do we really expect government schools to teach that? Meanwhile, most voters don't bother to get informed (they did that in school, right?).

      Heck, my kids' school teaches that Columbus thought the Earth was flat and that Lincoln started the Civil War to end slavery - it's mostly all folklore with the varnish of history.

    • Can someone tell me why these financial institutions are never forced to compensate the *individuals* that suffer from these events?

      For instance in the mortgage fraud scandal they were allowed to settle fraudulent foreclosures for pennies on the dollar. Why are these companies never required to make the people they hurt whole again? Individuals that paid thousands of dollars simply got a small payment while banks just had to deal with "the cost of doing business."

      I think I know the answer (lobbying/congresscritters in their pockets) but I think it's one of the most scandalous aspects of the financial mess of 2008.

      Conceptually it's more like they have the (bankrupt) government as a whole in their pocket.

      Got enough money to buy off a (bankrupt) government and you can get away with an awful lot.

    • Customers are investors, and investment carries risk. Sometimes it's good investors see a good chunk of their money disappear, it makes them more careful about their money. This is not exactly the proper way of losing that money, of course.

      I'm actually surprised that this company hasn't gone belly-up yet. It lost US$ 460 mln - that's enough to kill most companies if it's their own capital, and if it's their customers capital, those customers would likely take their losses and take their money to another com

  • The fine wasn't all of the punishment (Score:5, Informative)

    by Jay Maynard ( 54798 ) on Thursday October 24, 2013 @08:15AM (#45221967) Homepage

    That $460 million came out of Knight Capital's pockets too...and is far more effective than any fine the SEC could levy. Why should the SEC pile on, aside from the populist outrage that goes along with people handling billions of dollars?

    • Furthermore, why would millionaires trust their money to a company that is getting pilloried in the press for fundamental failures of management, not to mention development practices?

      Cutting corners on developing the software that handles your money: penny wise and pound foolish.

    • Re:The fine wasn't all of the punishment (Score:4, Informative)

      by Software ( 179033 ) on Thursday October 24, 2013 @09:04AM (#45222289) Journal
      If Knight had put the $460 million in a pile and burned it, there would be no fine. The problem was that their algorithm was wildly buying and selling shares in the open market, and thus distorting that market. See the graph at http://www.businessweek.com/articles/2013-06-06/the-knightmare [businessweek.com] for an example of a stock that was affected. What if you were an investor in that stock who had set a stop-loss at $10? Knight's wild selling would have triggered the stop-loss, and you'd lose money because of Knight's actions. This gross market distortion is what the fine was meant to punish.

      • Re:The fine wasn't all of the punishment (Score:5, Insightful)

        by JesseMcDonald ( 536341 ) on Thursday October 24, 2013 @11:27AM (#45223997) Homepage

        What if you were an investor in that stock who had set a stop-loss at $10? Knight's wild selling would have triggered the stop-loss, and you'd lose money because of Knight's actions.

        No, you'd lose money because you sold at the wrong time based on an automated trading rule rather than your own informed judgement. That is the risk you take when you enter a stop-loss order: if the drop in price is temporary, you're going to lose money. It makes no difference why the price moved. Knight's inadvertent trades are not to blame here. No one who payed attention to the fundamentals lost anything. Only those who panicked took a loss, and deservedly so.

    • Of course, the people who suffered because of the destabilization of the markets caused by such cavalier trading algorithms can sue to collect compensation for damages. But the same people who valiant rise to the defense of free markets, and "let people do what they want with their money and if they lose it, its their problem" are the same ones who rail against the trial lawyers and push for "tort" reform to reduce penalties.

      Remember this folks, when you are pushing for complete free markets, trial lawyer

      • Re: (Score:2)

        by sjames ( 1099 )

        Of course an advantage to fiction is that the author gets to gloss over all of the 'Atlases' moaning in bed every night when they discover that that unappreciated back breaking manual labor is really hard and takes a toll on health.

    • That $460 million came out of Knight Capital's pockets too...and is far more effective than any fine the SEC could levy. Why should the SEC pile on, aside from the populist outrage that goes along with people handling billions of dollars?

      As a punishment, obviously.

      If you break a law, and the action of breaking the law costs you money, you would still be penalized for breaking the law.

      i.e. if you spend five million dollars on trying to have someone killed and the assassin takes off with your cash you could still be held accountable for conspiracy to commit murder (IANAL but whatever) along with whatever punishment that merits.

  • They lost their own money (Score:5, Insightful)

    by Anonymous Coward on Thursday October 24, 2013 @08:22AM (#45222007)

    As a proprietary trading firm, they were working entirely with their own money. They had no external investors or whatnot (like hedge funds do). So, they made a mistake and they paid for it dearly. It's not clear to me that they should have paid any fine.

    The article's whole argument seems to be made by comparing the size of the trading loss to the size of the fine, but no logical reasoning is given for why the one should have any relation to the other.

    TFA sucks.

  • Not joking with this... (Score:4, Insightful)

    by EmperorOfCanada ( 1332175 ) on Thursday October 24, 2013 @08:23AM (#45222015)
    I'm not joking when I say that procure number one when money is flying out of your servers is to Shut Them Down instantly. I would have pulled the cables out so fast the CPU might have been yanked out with the network cable. Or a good old shutdown -h now !!!!! (The exclamation marks speed up the shutdown)

    And I wouldn't have done this one server at a time it would have been all the servers at the same time. I suspect they would lose money by not having the servers up but not at the firehose rate that they were losing money as they were.

    The worst part is that the admins were probably following some procedure in their book and were refusing to just pull the plug in some vain attempt for 99.9 percent up time or other admin related metric instead of the clear "Don't Lose $48 Million a minute!!!!" metric. So probably another clear case of IT's priorities getting way out of sync with the company's actual priorities.

    • when stuff like that happens the Correct And Right response is to fire an AA-12 loaded with breaching rounds through the main power /data conduits. (this should be faster than finding and yanking cables)

    • Re:Not joking with this... (Score:5, Insightful)

      by locofungus ( 179280 ) on Thursday October 24, 2013 @08:49AM (#45222185)

      The problem with this is that they didn't know they were losing money.

      The trading had gone haywire and they didn't understand why it was doing what it was doing but at the time it was happening they couldn't say if they were making or losing money.

      They built up positions of billions of dollars and only once it was all unwound and settled were the losses finally known.

      I can feel for the programmers and sysadmins. Maybe this was right - maybe pulling the plug out would prevent the unwinding of positions that would then make money for the firm.

      I wouldn't be surprised if there weren't previous problems where programmers and admins had been criticized by management for "doing the wrong thing with hindsight" when they didn't understand what was going on. If you have that sort of management culture then the natural inclination becomes to do nothing and push the responsibility up the chain.

      Similar disasters have happened in the past - one that springs to mind was Piper Alpha where the other nearby rigs continued to pump gas to it even when they could see it was on fire - because if you stop pumping it takes days and costs a fortune to get things back up and running and it might just have been an easily controllable fire.

      There's a very fine line to be drawn between reacting to the unknown too soon and reacting too late. There's also a fine line between making a reasonable best guess with the facts available and just making a random guess.

      http://en.wikipedia.org/wiki/Piper_Alpha [wikipedia.org]

      The fire would have burnt out were it not being fed with oil from both Tartan and the Claymore platforms, the resulting back pressure forcing fresh fuel out of ruptured pipework on Piper, directly into the heart of the fire. The Claymore platform continued pumping until the second explosion because the manager had no permission from the Occidental control centre to shut down. Also, the connecting pipeline to Tartan continued to pump, as its manager had been directed by his superior. The reason for this procedure was the huge cost of such a shut down. It would have taken several days to restart production after a stop, with substantial financial consequences.

      • The problem with this is that they didn't know they were losing money.

        That is a load of utter bullocks.

        The problem is that they have any kind of process for testing things before putting them into production. It was a cowboy operation without following best practices and a result like this was inevitable. If they had followed best practices and done testing ahead of time, they would have known before losing hundreds of millions of dollars. But they didn't do that, they didn't do peer review or a number of o

        • So, there are eight production servers. How many configurations do you think they should have tested?

          There are 256 ways of deploying a combination of the old and new code to those servers. Times two because the platform sending orders to those servers was changing too.

          The old code was good, and the new code was good, the bad deployment was the problem.

          Maybe even the order of deployment to those 8 servers matters - even if they're nominally identical I doubt they're exactly identical.

          Ok. So there's around 20

          • They had 8 systems in the cluster and only rolled out the code to 7 of them. The upgrade only ran on part of the cluster before it was put into production. A simple peer review would have caught the failure on the 8th system and prevented in from going live. A simple test would have caught that code on system didn't match the others. The more critical the system is the higher the level of review it should have before going live.

            This is what maintenance windows and peer review are for. So, yes, these failure

        • Re: (Score:2)

          by Zalbik ( 308903 )

          I agree 100%, but it makes me wonder: How do you test a system like this?

          You can't just put in in a test environment & try it out, as you need the rest of the market to exist. You can't just simulate it's responses based on current market conditions as the trades your system makes likely result in changes to the market, which wouldn't happen with simulated trades.

          It still looks like there is plenty of blame here, but it would be interesting to see how these systems are tested at other institutions....

        • Re: (Score:2)

          by sjames ( 1099 )

          That could have been OK if they were appropriately instrumented to be able to see if they were making or losing money. Alas, they didn't do that ether.

    • I guess it depends on what the trading volume was. Maybe they were losing 48 million a minute, but shutting everything down would have caused them to hold on to stuff longer than they should have. They possibly could have been losing more than 48 million a minute just by doing absolutely nothing, by not being able to sell stuff that was dropping. Personally, I wonder why it took them 45 minutes though. It should have been a simple script to revert everything back to the previous software version. It's
      • Yes a key strategy when upgrading your servers is to have a retreat plan. Quite simply if you upgrade your servers regularly one of your upgrades is going to blow up. I know that I am nervous every time I push a new version and never push a new version and then run out for lunch.

  • TFA states that the $460 million was lost by Knight Capital themselves. If they'd been fined $12M for stealing $460M, I'd be as outraged as the article author, but from where I'm standing it looks like the SEC turned a $460M loss into a $472M loss.

    Sure, they're idiots, they've punished themselves amply!

    • So if I break the traffic laws and while doing so crash my car, you are saying I should not be fined/jailed/whatever for breaking those laws since I already punished myself amply with the damage to my car?

  • Bad code wasn't the problem (Score:5, Informative)

    by onyxruby ( 118189 ) <onyxruby&comcast,net> on Thursday October 24, 2013 @08:33AM (#45222077)

    This had absolutely jack to do with bad code, that wasn't the problem. The problem was a failure to adhere to best practices that would have prevented the bad code from ever seeing production to begin with. The lack of a process for the distribution of code to production made a failure for bad code inevitable.

    This was sheer incompetence of the highest magnitude and should have been readily caught in the lab. This is what happens when cowboys run the show and ITIL is considered a four letter word. Take your younger staff, the wannabe cowboys and make them read this report. Let them learn at others incompetence. As for getting your management to read this, that's an entirely different story.

    • Re: (Score:2)

      by jimicus ( 737525 )

      It's not as simple as "badly-tested code" - it's actually "badly-designed deployment procedure and insufficient oversight".

      TFA is light on details, but other articles have picked up the details and explained them a bit better: basically, Knight Capital were running their code on a cluster of 8 nodes.

      They used a flag to signal a module to run. A particular module had been out of use for some years, so the flag to signal that module was re-used for a new module.

      With me so far? OK, this is all very nice. Excep

      • I actually made your point on a comment [slashdot.org] I made about this story yesterday and am in complete agreement with you. The lack of proper process is what did this in and their failure to have proper change management and follow industry best practices is incompetence of the highest level. This was entirely preventable and I would have been fired in a heartbeat if I had done this when I was doing that work.

        • Re: (Score:2)

          by jimicus ( 737525 )

          Thing is, much of this isn't particularly sophisticated.

          Hell, even the most basic change control process forces you to think about how you're going to do the job, what the criteria are for successful completion and how you're going to back out if there's the remotest sign of anything going wrong. That's noddy stuff you learn in a 3 day intensive ITIL course with zero real-world experience; there is precisely zero excuse for a trading firm not doing all that and more besides.

    • Seriously, there *was* no bad code. What happened was that one of their systems didn't get upgraded and they re-used a variable that was previously used to make systems to keep buying until they were told to stop by a master system. When the server that didn't get upgraded got that variable switched, it just started buying and nobody told it to stop. They knew something was wrong for 45 minutes and kept on letting it buy stuff, didn't just switch it off because there was nobody authoritative that could make

    • This had absolutely jack to do with bad code, that wasn't the problem. The problem was a failure to adhere to best practices that would have prevented the bad code from ever seeing production to begin with. The lack of a process for the distribution of code to production made a failure for bad code inevitable.

      This was sheer incompetence of the highest magnitude and should have been readily caught in the lab. This is what happens when cowboys run the show and ITIL is considered a four letter word. Take your younger staff, the wannabe cowboys and make them read this report. Let them learn at others incompetence. As for getting your management to read this, that's an entirely different story.

      After working in the early days of computer trading, I can tell you that every minute that the new code is not in place they are "losing money, and IT is to blame." If there's a glitch like this one, then they "lost money, and IT is to blame." Do you see a pattern. The managers and traders are never to blame.

  • Failures in Large Software Systems Seem the Norm? (Score:3)

    by BoRegardless ( 721219 ) on Thursday October 24, 2013 @08:35AM (#45222085)

    I am a bit numbed by the number of failures of software systems at big companies (& governments) who should know better.

    If you are designing critical systems, there has to be an incredibly detailed master system describing fallbacks, trip points and fail safe conditions, let alone a gross shutdown (seen multiple times recently.) What do these failures in both checking and security and logic mean for trusting large institutions and government?

    The question: What overview system of principles of software design are going to be needed to properly organize a major software program from day one to prevent, at least, the obvious failure modes? There is something inherently wrong by design when hundreds to thousands of security breaches occur in the US on public websites and databases each year.

  • It's a bad thing...

    Just make sure they suffer all the pain caused by the $450 Million loss

    In other words: don't allow them to pass any of this loss on to their customers by drawing funds from their accounts.

    • Just make sure they suffer all the pain caused by the $450 Million loss

      They did. It was their own money they lost. The summary and TFA gloss over that fact in some circuitous attempt to grind a non-relevant axe. Fining them makes about as much sense as charging someone who fails at committing suicide with attempted murder.

  • Wall St systems in general are bad (Score:5, Insightful)

    by Andover Chick ( 1859494 ) on Thursday October 24, 2013 @08:56AM (#45222233)
    Most all Wall St firm's systems are bloody awful. There are many reasons for this. First, the true business is sales/brokerage so the engineering side, though it is a strategic asset, is often neglected. This includes putting clueless business side people in charge of IT system. Second, the boom and bust cycles of tech investment are a bad way of building tech systems. It's like not watering your garden all summer except for one day when you use a high-pressure fire hose on it. Third, as part of the boom/bust cost cutting they have no employee longevity in tech so no one understands how the mind-bogglingly complex and obscure layers of technology work. Fourth, and more recently for cost cutting, they've dispersed their dev teams around the globe so communication and teamwork are seriously compromised. Fifth, when there is a boom they try to build their systems so quickly that they take all sorts of dangerous engineering short cuts. All this adds up to engineering disaster.
  • Given the cronyism masquerading as capitalism in USA, you should be glad this behavior is considered bad enough to be punished. Be glad they did not get the contract to "improve" healthcare.gov
  • Tell you what, if 12 Mil is measly to you, then I'm sure you wouldn't even notice if half a mil went missing. And I sure could use half a million dollars . . .
  • The SEC (Slashdot Effeciency Committee) have released their findings and conclude that:

    Slashdot pushed their new code to all but one app server. That one app server reposted the same Knight story as yesterday. Slashdot has been fined 12 karma.

  • These are all elementary software mistakes:
    1) You never reuse a flag for a working code, because it makes it impossible to revert back to older deployment.
    2) You always double check deployments to make sure it actually succeeded.

  • I don't get what's the purpose of these "remedial sanctions," especially coming from the SEC.

    If the SEC is doing this to deter Knight's management from being un-diligent with Knight's owners' assets, then it ought be a fine or prison time for the people who were responsible, not the company (the owners, who were also the victims). That's like punishing someone for the crime of being raped, while talking about how irresponsible the rapist was.

    If the SEC is doing this to deter Knight's owners (who worked t

  • After all, they needed more money to continue funding KITT and that damn trailer that drives it around.
    Michael Knight doesn't come cheap, either.

Slashdot Top Deals

"If it ain't broke, don't fix it." - Bert Lantz

Close