Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

How To Lose $172,222 a Second For 45 Minutes

Soulskill posted about 10 months ago | from the step-one-accrue-at-least-$172,222 dept.

The Almighty Buck 327

An anonymous reader writes "Investment firm Knight Capital made headlines in 2012 for losing over $400 million on the New York Stock Exchange because of problems with their algorithmic trading software. Now, the owner of a Python programming blog noticed the release of a detailed SEC report into exactly what went wrong (PDF). It shows how a botched update rollout combined with useless or nonexistent process guidelines cost the company over $172,000 a second for over 45 minutes. From the report: 'When Knight used the Power Peg code previously, as child orders were executed, a cumulative quantity function counted the number of shares of the parent order that had been executed. This feature instructed the code to stop routing child orders after the parent order had been filled completely. In 2003, Knight ceased using the Power Peg functionality. In 2005, Knight moved the tracking of cumulative shares function in the Power Peg code to an earlier point in the SMARS code sequence. Knight did not retest the Power Peg code after moving the cumulative quantity function to determine whether Power Peg would still function correctly if called. ... During the deployment of the new code, however, one of Knight's technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Knight had no written procedures that required such a review.'"

cancel ×

327 comments

Sorry! There are no comments related to the filter you selected.

The efficiency of capitalism (5, Funny)

Tablizer (95088) | about 10 months ago | (#45207921)

See, the private sector can blow money faster than the public sector (OmabaCare site).

Re:The efficiency of capitalism (CORRECTION) (1)

Tablizer (95088) | about 10 months ago | (#45207941)

Correction: "Obama" not "Omaba". Sorry 'bout that. It was not intentional (although there's probably a mean joke in there somewhere.)

Combining nerdism with capitalism (2, Informative)

Taco Cowboy (5327) | about 10 months ago | (#45208113)

Capitalism is very effective in what it does

Nerdism, on the other hand, is very detail in what it does

Combining both and you will get an invincible beast

Unfortunately (or fortunately, depending on how you look at it), the investment firm "Knight Capital" does not treasure the nerds enough to put them into position that can have effective oversee powers over technology deployment

I hope the 400 million loss will wake them up

No more we nerds should work under them capitalists --- they need us MUCH MORE than we need them

This is what I like best about /. (5, Insightful)

rsilvergun (571051) | about 10 months ago | (#45208235)

In the same thread where I can find 1000 people going on about how efficient capitalism is I can find another (sometimes the same) 1000 people complaining all the dumb things their companies do. Well, which one is it? It doesn't work both ways people. Could it be that people are people, no matter what banner they're organized under?

Re:This is what I like best about /. (5, Insightful)

Anonymous Coward | about 10 months ago | (#45208297)

Capitalism produces large corporations that are very efficient at doing dumb things.

Re:This is what I like best about /. (2, Insightful)

MightyMartian (840721) | about 10 months ago | (#45208417)

I'm too busy thinking about how many children with cancer could be saved for 400 million US dollars.

Re:This is what I like best about /. (5, Insightful)

Anonymous Coward | about 10 months ago | (#45208481)

The money wasn't destroyed, it went to other traders. Maybe they will donate it to children with cancer?

Re:The efficiency of capitalism (0)

Anonymous Coward | about 10 months ago | (#45207947)

See, the private sector can blow money faster than the public sector (OmabaCare site).

But can they shake off the extra pounds this fast [youtube.com] ?

Re:The efficiency of capitalism (1, Insightful)

Anonymous Coward | about 10 months ago | (#45207971)

This corporation can't send people with guns to your house to get more money. Obama Can Do It!
This corporation can't print more money, watering down the money you yourself have, Obama Can Do It!
This corporation won't get the smug approval of 4 out of the 5 major news outlets, Obama Can Do It!
This corporation won't blame its losses on Bush. ...won't borrow 40 cents for ever dollar it spends, from China, leaving the mess for your kids to pay for. ...won't forward this email to 10 friends!

Re:The efficiency of capitalism (2, Insightful)

iserlohn (49556) | about 10 months ago | (#45208007)

The corporations (and other moneyed interests) don't have to do any of that because it pays the government du jour to do its bidding. The whole idea of representative democracy is to prevent and minimize that by introducing checks on power by the public.

Re:The efficiency of capitalism (0)

Anonymous Coward | about 10 months ago | (#45208047)

The corporations are just as bad, but under another name. Take my mortgage company. Private sector company. They force me to buy fire insurance for my house (which I don't want!) just because some corporate douche thinks he knows what's best. I have never in my life had a fire at my house, so why the fuck should I have to pay for some other dumbass who can't properly extinguish a cigarette? It is an afront to my freedom as an American.

Re:The efficiency of capitalism (2)

Fjandr (66656) | about 10 months ago | (#45208247)

You didn't have to get a mortgage.

Re:The efficiency of capitalism (0)

Anonymous Coward | about 10 months ago | (#45208389)

You didn't have to get a mortgage.

And if you don't like paying taxes, just quit your job. Problem solved. Welfare is the new work.

Re:The efficiency of capitalism (-1)

Anonymous Coward | about 10 months ago | (#45208579)

Thanks, but Obamacare REQUIRES insurance or a tax. If you have no income you don't get a subsidy either, because it is done as a tax credit.

How about that, he made just being an American citizen illegal unless you do his bidding. I thought the 13th amendment was supposed to prevent that.

Re:The efficiency of capitalism (1)

ArbitraryName (3391191) | about 10 months ago | (#45208265)

You're free to do business with any mortgage company, or none at all.

Re: The efficiency of capitalism (0)

Anonymous Coward | about 10 months ago | (#45208283)

Almost every private loan has conditions attached to protect the lender

Re:The efficiency of capitalism (0)

Anonymous Coward | about 10 months ago | (#45208293)

An affront to your freedom as an American? What about the freedom of your neighbours to not have THEIR houses burn down next to yours because you're a fucktard who doesn't understand the danger?

Re:The efficiency of capitalism (1)

blue trane (110704) | about 10 months ago | (#45208295)

Govt funding costs are essentially 0. China owns something like 15% of the US Debt. The Fed rebates interest on t-bills it buys. Conclusion: the national debt is a distraction, not a crisis except for those who want to gain political advantage by cynically drumming up fears about it when they know that once they get into power, they can run up the debt with impunity because "Reagan proved deficits don't matter."

$400 billion / year is "essentially zero"? (3, Informative)

raymorris (2726007) | about 10 months ago | (#45208665)

$400 billion per year in interest payments is "essentially zero"? That number is actually almost manageable, the BIG problem is that it's snowballing.

You'll recall Obama has been saying that if he wasn't allowed to borrow more, the government would default - wouldn't be able to pay the interest due. We're borrowing to the interest on our borrowing. That's when you know you're fucked, when you're maxing out one credit card to pay the minimum payment on another card. That's essentially what the US government is doing. We're in a trap the we're borrowing more and more in order to make the payments on existing debt, so the debt and the interest just keeps getting bigger and bigger until 100% of our tax money goes to pppay interest, leaving no money for essential government services.

Looking at what has happened in other countries, the "you're fucked" point, the point at which you can't escape the death spiral, is about 100% of GDP - when a country owes as much as it generates. Eight years ago, our debt to GDP ratio was about 35%. In 2014, it should hit 70%. That tells us we are about six to eight years from becoming Greece.
    The difference between Greece and though, is that Greece is small enough to be bailed out. Nobody has $5 trillion to bail the US out.

Re: The efficiency of capitalism (0)

Anonymous Coward | about 10 months ago | (#45208335)

Yes they can. Yes they can. And yes they can.

Re:The efficiency of capitalism (0)

Anonymous Coward | about 10 months ago | (#45208395)

This corporation can't send people with guns to your house to get more money

They got a taxpayer bailout. Yes they can!

Re:The efficiency of capitalism (1)

Phoeniyx (2751919) | about 10 months ago | (#45207989)

And the private sector has a way of correcting it. Look what happened to Knight. In the public sector, wasted government expenditure is rewarded with a higher debt ceiling.

Re:The efficiency of capitalism (0)

Anonymous Coward | about 10 months ago | (#45208131)

In the public sector, wasted government expenditure is rewarded with a higher debt ceiling.

Government expenditure on public programs grows with population.
Government income from taxes shrinks as corporations fund election campaigns.

If (expenses - income) is positive, a debt is incurred.
We either have to kick grandma and grandpa to the curb (end medicare, end social security, etc.)
Or we have to make the corporate freeloaders pay their taxes.

Re:The efficiency of capitalism (1)

blue trane (110704) | about 10 months ago | (#45208271)

Create more money. As long as we keep innovating, it doesn't matter how big a deficit we run. Reagan proved that. Old people can help innovation, participating in web forums, winning 3D printer contests (see http://news.slashdot.org/story/13/03/04/1428241/ [slashdot.org] "83-Year-Old Inventer Wins $40,000 3D Printing Competition").

Re:The efficiency of capitalism (1)

Anonymous Coward | about 10 months ago | (#45208085)

If Knight Capital had run in this condition 24/7 for a year, it would have pumped 5.431 trillion. Total government spending, all levels, in 2012 runs in at 6.189 trillion. So uber fast computers running wide open on an unchecked and egregriously stupid software error fall 12% shy of American excellence.

Footnote: It only took Knight Capital 45 minutes to realize they'd made an error.

Re:The efficiency of capitalism (0)

Anonymous Coward | about 10 months ago | (#45208307)

Interesting numbers, but to me that proves how fast imaginary money can be created or destroyed - almost as fast as the entire human race (or were your figures just for USA?) can generate it.

Re:The efficiency of capitalism (0)

Anonymous Coward | about 10 months ago | (#45208327)

herp derp all government spending is waste derp herp

Re:The efficiency of capitalism (-1, Flamebait)

Bite The Pillow (3087109) | about 10 months ago | (#45208135)

-1 off topic. Nothing to do with FartbongoCare. Plenty of things to rail on here, left or right wing, without bringing in unrelated nonsense.

See how I used a derogatory term? I'm just like you, we agree, we're friends. Now junk-punch yourself as penance.

Re:The efficiency of capitalism (0)

MightyMartian (840721) | about 10 months ago | (#45208445)

Fartbongocare? Is that Ted Cruz's up and coming proposed replacement to the ACA?

Re:The efficiency of capitalism (5, Insightful)

TsuruchiBrian (2731979) | about 10 months ago | (#45208169)

The wealth that the money represented was not "lost", but rather redistributed. Efficient redistribution of wealth is a strength of the private sector not a weakness. The private sector is good at redistributing money where it needs to go for economic growth. This company was not exercising an appropriate level of caution and it's money was redistributed elsewhere.

Re:The efficiency of capitalism (4, Insightful)

techno-vampire (666512) | about 10 months ago | (#45208273)

I think you're being a tad too literal here. As far as Knight was concerned the money was lost because they didn't have it any longer and they had nothing to show for it.

Re:The efficiency of capitalism (2)

TsuruchiBrian (2731979) | about 10 months ago | (#45208401)

I was responding to a comment that suggested this was a failure of the private sector. I responded that this is an example of what the private sector is supposed to be doing, and actually does quite well. When you talk about 1 company in the private sector losing money, this doesn't mean the private sector as a whole lost wealth (e.g. what happened here). When you talk about the public sector losing money, it almost always translates to lost wealth.

As a society we don't need to care what happens to some random company. This is a great strength of the private sector, and this property is what is referenced by the phrase "the market is self-regulating". The government however is not self regulating. It is regulated by voters. We *do* need to care if a government agency loses money, because it is usually not losing it to some competing government agency. It is usually just wasted.

Re:The efficiency of capitalism (2, Insightful)

Waffle Iron (339739) | about 10 months ago | (#45208693)

It is usually just wasted.

By your own previous logic, money can never be "wasted", just redistributed.

Every dollar that the government collects in taxes or creates out of thin air is redistributed back into the private sector.

Re:The efficiency of capitalism (1)

Anonymous Coward | about 10 months ago | (#45208469)

The private sector is good at redistributing money where it needs to go for economic growth.

I see this kind of claim a lot but I've never been able to figure out what it's based on.

There are certainly examples of economies that are/were micromanaged by corrupt dictatorships in the name of "communism" that seem to be particularly unsuccessful (e.g. North Korea). But then there are also economies without much government intervention at all that don't seem to be much better (e.g. Somalia). And at the other end of the spectrum, countries that provide a mix of individual incentives and strong governemt social programs seem to be particularly successful (e.g. the Scandinavian countries).

And from a theoretical perspective, I don't see any reason to believe that the total welfare of a collection of individual agents is maximized when each agent focuses on maximizing its own individual welfare. Among other things, it seems that such systems would be inherently unstable: a small number of agents end up controling all the wealth and all the other agents basically become slaves.

But is there actually a sound mathematical/scientific reason to conclude that your claim is valid in a very general sense? Or is it just that you can sometimes find specific cases where it's true and you'd like to believe that it's true as a matter of wishful thinking?

Re:The efficiency of capitalism (0)

Anonymous Coward | about 10 months ago | (#45208521)

Are you fucking kidding? In a normal circumstance I'd be sure that you are, but this is Slashdot, the lightbulb for autistic libertarian moths. Here I just can't be sure.

Re:The efficiency of capitalism (0)

mysidia (191772) | about 10 months ago | (#45208321)

See, the private sector can blow money faster than the public sector (OmabaCare site).

The federal government incurs over $3 million in new debt per minute on average.

The annual extra interest expense on the national debt (taxpayer money lost due to interest volume) is on the order of $500 billion.

$172,222 a Second For 45 Minutes is an order of magnitude less .

Re:The efficiency of capitalism (1)

roman_mir (125474) | about 10 months ago | (#45208331)

Private sector did exactly as it is supposed to: somebody lost money by not being efficient or vigilant, somebody else made money by being more efficient and or more vigilant.

As long as the government does not step in with some form of a bail out there is no problem there at all. This bank or whatever financial firm it is will be HELD ACCOUNTABLE by the shareholders, clients, but at least there are mechanisms for doing that in the private sector.

Obamacare is about stealing money from people and apparently there is no mechanism to stop this theft at all.

Re:The efficiency of capitalism (1)

phrackthat (2602661) | about 10 months ago | (#45208515)

Obamacare is about stealing money from people and apparently there is no mechanism to stop this theft at all.

Well, actually there is a mechanism. It's called the vote. However, you have a lot of low information voters out there and a media that's as loyal as Pravda to the Democrats. The people voted to stop Obamacare by voting in people to oppose it.

Democrats bitch and whine that a "democratically passed law" is being opposed. The media parrots the Dems and tells people what to think - "The evil Republicans are opposing a democratically passed law." But they don't tell the sheeple that the Democrats in the House were voted out over Obamacare and that it is the #1 job of the people who were to voted in to oppose it and shut it down because that is why they were voted in!

The 4th estate is an utter failure. They do not operate to keep those in power honest - they serve the interests of Democrats for the purpose of aggregating power.

Re:The efficiency of capitalism (2)

Laxori666 (748529) | about 10 months ago | (#45208661)

Eh that's the point of capitalism. People who fail at things go out of business, and others who don't fail as much replace them. Maybe this would have put Knight out of business. Great, then another company can fill that spot. Yet what happens if the government fails? They can just raise taxes or create money to cover the gap, thus making everybody else in the nation poorer.

Garden Variety Upgrade SNAFU (3, Insightful)

mythosaz (572040) | about 10 months ago | (#45207925)

I'm sure someone will chime in and claim to be the em-effing Change MASTER! but this seems like an ordinary error, the sort I've seen a hundred times before, where one server is a tiny bit wonky, and during the change, something doesn't happen as expected. Normally it's an "inexpensive" error, where some of your VPN users get randomly disconnected... ...and sometimes it's the sort of error where you lose half a billion dollars an hour by HFT trading... ...badly.

Re:Garden Variety Upgrade SNAFU (1)

Qzukk (229616) | about 10 months ago | (#45208433)

It's true that mistakes can be made and accidents can happen.

It's also true that if you bother to try, you might fix some of those mistakes and catch those accidents before they happen.

Knight Capital didn't even bother to try. At least they managed to find private investors to bail them out after the SEC refused.

Re:Garden Variety Upgrade SNAFU (5, Insightful)

Alan Shutko (5101) | about 10 months ago | (#45208441)

There were a number of errors made here.

  1. They failed to deploy to one of eight servers
  2. They failed to automate the deployment to the servers such that it would be impossible to deploy to all servers without knowing.
  3. They didn't have a step between code deployment and production activation where they could validate all 8 servers. For instance, in our company, we deploy the prod code to the prod servers but leave them in a "stage" environment, where the production load balancer doesn't hit those instances. Once we've validated, we then switch the load balancers to point to the correct instance.
  4. They failed to quickly back out a change when they realized it was having problems. In fact, they backed out the part on seven servers but not the flag that was being sent to the servers, which made things worse.
  5. They failed to have a risk-mitigation backstop in place which would have prevented these orders from being submitted once they hit a certain amount, and which was required by SEC Rule 15c3-5(c)(1)(i).

There were a lot of places that you could put in a control to prevent or limit the effect of these kinds of errors, and that's the lesson people need to learn. Yes, mistakes happen! But try to make it hard to make a mistake, easy to recover from a mistake, and really easy to NOTICE a mistake.

There should be a mandatory one second delay. (5, Insightful)

Anonymous Coward | about 10 months ago | (#45207931)

This level of trading does not do the market any good, and puts individual investors at a severe disadvantage against firms like this.

It can be stopped. And it should be stopped. And the only reason it is not being stopped is because too many rich and powerful people make too much money on it.

Re:There should be a mandatory one second delay. (2)

sexconker (1179573) | about 10 months ago | (#45208001)

This level of trading does not do the market any good, and puts individual investors at a severe disadvantage against firms like this.

It can be stopped. And it should be stopped. And the only reason it is not being stopped is because too many rich and powerful people make too much money on it.

I'd go as far as 10 seconds just to eliminate the possibility of assholes pulling shit with different rules for rounding, or horseshit like "our clock was slightly off", etc.

Re:There should be a mandatory one second delay. (5, Interesting)

Anonymous Coward | about 10 months ago | (#45208067)

Personally, I believe if you're going to buy stock in a company then you should be required to hold said stock for at least 24 hours, if not much longer. the stock market was created to allow people to invest their money in a company, thus allow that company to use that money to grow which should result in a return (or loss). It was not designed for gambling, which is what it has become.

Re: There should be a mandatory one second delay. (1)

Ant2 (252143) | about 10 months ago | (#45208221)

The company that issued stock has long since been paid for it at the initial public offering. Any subsequent trades in that stock really do not affect the company to any significant extent.

Re: There should be a mandatory one second delay. (1)

Anonymous Coward | about 10 months ago | (#45208311)

Trades affect the stock's value. The stock value affects the value of the company to a pretty significant extent.

Re: There should be a mandatory one second delay. (3)

gumbi west (610122) | about 10 months ago | (#45208477)

The market price of the stock affects the company's ability to raise money through additional issues. This affects their ability to take out loans/sell bonds.

Re:There should be a mandatory one second delay. (4, Insightful)

ShanghaiBill (739463) | about 10 months ago | (#45208077)

You are confused. "Algorithmic trading" and "High Frequency Trading" are two different things, used for two different purposes. This was caused by AT, and you are complaining about HFS. That makes no sense.

Re:There should be a mandatory one second delay. (0)

Anonymous Coward | about 10 months ago | (#45208137)

172,000 per second seems pretty high frequency to me..

Re:There should be a mandatory one second delay. (0)

Anonymous Coward | about 10 months ago | (#45208217)

172,000 per second seems pretty high frequency to me..

Here, just for you: A clue [wikipedia.org]

Re:There should be a mandatory one second delay. (1)

complete loony (663508) | about 10 months ago | (#45208193)

"SMARS is an automated, high speed, algorithmic router that sends orders into the market for execution"

All "High Frequency Trading" is by definition "Algorithmic trading", though the reverse doesn't necessarily hold.

Re:There should be a mandatory one second delay. (0)

Anonymous Coward | about 10 months ago | (#45208325)

Neither does the forward. HFT can simply be contingency orders and does not have to be driven by a computer 'responding' to the market.

They are the same thing. (0)

Anonymous Coward | about 10 months ago | (#45208365)

Alogrithmic trading and HFT are the same thing. HFT needs an (insert term here) in order to do its trades. That is the only reason to use a computer in the first place; otherwise you just need a handheld calculator or at most a Python sctipt for portfolio management.

Re:There should be a mandatory one second delay. (1)

mysidia (191772) | about 10 months ago | (#45208373)

This was caused by AT, and you are complaining about HFS.

If it was not HFS; then why were the orders not being presented for human approval?

Re:There should be a mandatory one second delay. (0)

Anonymous Coward | about 10 months ago | (#45208483)

From the Wikipedia article on Knight Capital:

The Knight Capital Group was an American global financial services firm engaging in market making, electronic execution, and institutional sales and trading.[1] With its high-frequency trading algorithms....

"high frequency trading algorithms"

Re:There should be a mandatory one second delay. (3, Interesting)

Anonymous Coward | about 10 months ago | (#45208101)

Just tax EVERY transaction. Done.

Re:There should be a mandatory one second delay. (1)

Pinhedd (1661735) | about 10 months ago | (#45208139)

This is not HFT and as such a small delay would not have changed anything. It went on for 45 minutes, not 45 microseconds.

Re:There should be a mandatory one second delay. (0)

Anonymous Coward | about 10 months ago | (#45208245)

how about a minimum fee?

Re:There should be a mandatory one second delay. (2)

Skapare (16644) | about 10 months ago | (#45208269)

The method I have proposed to fix this is "cycle trading". Traders submit their buy/sell requests for a trade cycle that happens every minute. Each cycle's trades go to completion if possible, based on buy/sell amounts and prices (bid/ask). Requests will be left over if they cannot be bought or sold in that cycle. They can be flagged to hang in there for the next cycle, or canceled, or changed.

But instead of a one minute cycle, let's do a one day cycle.

Re:There should be a mandatory one second delay. (2)

mysidia (191772) | about 10 months ago | (#45208367)

This level of trading does not do the market any good, and puts individual investors at a severe disadvantage against firms like this.

Well; it does do the market good... it helps with eliminating inefficiencies.

However; I am of the opinion that there should be a minimum "execution delay"; with all trades timestamped.

Trades should be cleared at 90 second increments; with no trade younger than 120 seconds eligible to be executed. After 90 seconds have elapsed; the trade should become non-cancellable, non-modifiable, until it as at least 180 seconds old total.

All the orders that arrived within the same 60 second increment, should be randomly shuffled, so that the exact order they arrived within that narrow time frame has no effect on priority.

Finally, trades cleared in order --- with only trades placed in different 60 second windows having priority over one another.

Re:There should be a mandatory one second delay. (5, Informative)

girlintraining (1395911) | about 10 months ago | (#45208421)

Eh, other posters have already pointed out that you're referencing high frequency trading, not algorithmic trading, so this is offtopic. Nonetheless... where exactly do you think this 1 second delay should be put in, and what would it accomplish? Make the wires "longer"; That would mean less contention for premier data centers in NYC. In one second, you can send a signal around the world five times over. But that doesn't help with the propagation of trade data from which the trades are based on; By adding all that extra lag only in terms of trade execution, but not market data, you're potentially putting billions of dollars at risk as trades are now following market data, instead of running concurrently with them. Think of it this way: You swipe your card to pay for gas. The price shown is $3.55. But when you start the pump, the price drops to $3.54. But you started the pump a second too late, so you're billed a penny more than the guy who waited a split second. Now, multiply this a few million times and suddenly you've got a market crisis. It's the same if you lag the market data but allow trades at full speed.

Let's say you put this one second latency in for both sides; trade execution and market data. How exactly do you syncronize the data when the price itself is determined by trades -- you potentially have more trades waiting to be executed than you have shares... the price is now in some kind of weird state whereby it cannot be accurate until the trades are complete, yet as the trades complete the price is trading. Now you've turned a tiny amount of speculation into a massive amount of speculation. You've made the problem a thousand times worse!

You see, no matter where you put in your "one second delay", you're reducing liquidity, increasing costs, and causing money to be lost out of the system. Your idiotic attempt to help the "little guy" has resulted in utter chaos at best, and only made it harder for him at worse!

High volume trade is just margin trading; Buying low and selling high. Now there's a lot of macroeconomic theory to go into what I say next, too much for a slashdot post, but fundamentally... the more trade there is, the more wealth there is. Lots of trades mean the market is healthy. It means money is moving... and the more money moves, the more it trades hands, the more value that money has. The only time money loses value is when it sits in an account doing nothing. It's like potential energy versus kinetic energy. You cannot harness the power of something that isn't moving.

Every time I hear about people bitch about high volume trading and "the little guy" I die a little inside; It shows a shocking lack of understanding of how markets actually operate, and how these sorts of trades benefit everyone by improving liquidity. The last economic crisis, in fact, the core of all economic crisis, is the lack of money moving. You can't invest because nothing is producing. You can't produce because nobody's investing. These kind of mexican standoffs are what lead to recessions and depressions. Liquidity is at the very heart of any boom, and its absence at the heart of every bust.

The reason why the "rich and powerful" have created a wealth gap is because money isn't trading hands. There's no trade going on -- the middle class isn't buying anything new, they're just paying off old debts. The upper class are the only ones with any liquidity, and they're holding onto money because there's nothing to invest in; If nobody's buying anything, what then is the point of investment? There's no return then. And the poor... they can't invest. They're living hand to mouth, paycheck to paycheck... economically, they're useless. They'll spend every dollar they're handed on the same things every day -- food, shelter, clothing, gas, rent... these things are essential to daily life, but they don't grow an economy. To get economic growth, you need people buying laptops, cars, services, luxury items.

And what started all of this? Ironically, it was a small segment of the population -- the baby boomers. They decided, as an aggregate group, to live beyond their means -- they went into debt to maintain their lifestyle of botox injections, hipsterism, and living in houses when apartments would do, and buying everything from cars to vacations on credit cards. Our consumer credit card debt is nearly equal to the government's total debt -- it's measured in the many trillions of dollars, and eclipses the GDP of our country. Or put another way... most of our production capacity as individuals is going to pay off debt, instead of buy new goods and services.

That's why everything went to hell; Blaming the 1% for this is stupid. They certainly didn't stop you from piling on the debt like a fat guy at an all you can eat bar... they shoot for long-term gain... 20, 40 year bonds, stuff like that. Because their money doesn't have to come back to them immediately... they can invest. And they did. They invested in your greed and need for immediate gratification.

And now they're collecting, credit card payment at a time.

Validation? (4, Funny)

egcagrac0 (1410377) | about 10 months ago | (#45207977)

Damn the validation. Full speed to prod!

Re:Validation? (1)

Frosty Piss (770223) | about 10 months ago | (#45208035)

What does this have to do with "validation"? TFA talks about *partial* deployment, one server didn't get the update.

Re:Validation? (1)

Bite The Pillow (3087109) | about 10 months ago | (#45208171)

No testing, no test deployment in a mirrored environment, no one validated that the deployment was correct.

If these things are the heart of your income, you take a hash of everything pre-deploy and post-deploy, and make sure it matches everywhere it has to.

Then you validate that the hash list matches everywhere. That's just common sense. I've worked plenty of places where income was not directly tied to what was deployed I developed a manually intensive way to do exactly the same thing - make sure you deploy what should be deployed, nothing less and nothing more.

Is the deployment valid? That's validation. And if you have the ability to lose piles of money by fucking it up, you fucking well validate.

Re:Validation? (1)

steelfood (895457) | about 10 months ago | (#45208251)

Yeah, sounds more like a botched migration than any failure to test. And that was allowed to go forward because of a lacking in their formal migration process.

There are a lot of growing pains when moving to entire systems of new technology. Testing the technology in pieces (unit testing) and as a whole (system testing) is already a given. For things that really matter though (like money), the processes themselves need to be tested and refined. I've seen a lot of processes go to production, i.e. used on production machines, only to go wrong when they could have been tested on the existing test setup and the kinks worked out then.

Re:Validation? (0)

Anonymous Coward | about 10 months ago | (#45208231)

Damn the Configuration Management. Fixed for Me..wait..

Failure of best practices (4, Insightful)

onyxruby (118189) | about 10 months ago | (#45207993)

No proper change management, no peer review, no proper lab testing. Dev should always reflect production to the greatest reasonable level. No proper maintenance windows. You should never be surprised by a change in production. This is a case study in incompetence and the failure to execute industry best practices. I'm guessing the guy or gal who raised the best practices flag was ignored as being inconvenient or too expensive.

If I'd done this kind of thing when I was working with the exchanges I would have been fired in a heartbeat. Whoever failed to utilize best practices, or whoever failed to allow the utilization of best practices had damn well better have been fired. This is incompetence of the highest level and a perfect example of why ITIL based best practices were born.

Re:Failure of best practices (1)

mythosaz (572040) | about 10 months ago | (#45208055)

It's very easy to do all of these things, and then have one server get missed from your production change list. ...and your validation check *seems* OK.

Re:Failure of best practices (1)

ShanghaiBill (739463) | about 10 months ago | (#45208111)

No proper change management, no peer review, no proper lab testing.

Also, there appears to have been no run-time checks or assertions. I have seen high reliability systems where 90% of the code is various run-time sanity checks, and only 10% implements the actual functionality. And that was for systems with a failure cost of way less than $400M.

Re:Failure of best practices (4, Insightful)

Billly Gates (198444) | about 10 months ago | (#45208155)

Dude, Wall Street traders demand software engineers get things done in minutes and hours!

There is no process as it all had to be done yesterday if you ask anyone who has worked in that environment.

They pay top dollars and change core algorithms on the fly as each millisecond costs money to a competitor who has a more efficient trading algorithm that steals from their own HTC network.

So when when skims and manipulates the prices in milliseconds they steal from those who have process engineered designs who are too slow to react.

If it messes up they get bailed out by the SEC anyway and they can just fire the programmer.

Re:Failure of best practices (2)

Howitzer86 (964585) | about 10 months ago | (#45208461)

I'm sure they were fired. That's how scapegoating works. You tell them about the problem, they ignore you, it fails as you said it would, they fire you.

Re:Failure of best practices (1)

ToasterMonkey (467067) | about 10 months ago | (#45208697)

No proper change management, no peer review, no proper lab testing. Dev should always reflect production to the greatest reasonable level. No proper maintenance windows. You should never be surprised by a change in production. This is a case study in incompetence and the failure to execute industry best practices. I'm guessing the guy or gal who raised the best practices flag was ignored as being inconvenient or too expensive.

If I'd done this kind of thing when I was working with the exchanges I would have been fired in a heartbeat. Whoever failed to utilize best practices, or whoever failed to allow the utilization of best practices had damn well better have been fired. This is incompetence of the highest level and a perfect example of why ITIL based best practices were born.

I didn't read TfA, but from TfS, none of what you said would solve this problem, or a better way to put it is they all could have actually taken place to a reasonable degree.

Is it generally expected or practical to test combinations of versions of the same software in a cluster? Only automated testing could catch a problem like that, and you'd need a simulated production workload.
A "reasonable" development environment would NEVER reach that far. That is a very above average QA environment.

Of course everybody would LOVE to have that, but I doubt that is widely considered a best practice.

At the other end, a monitoring system should have flagged the condition where all nodes are not running the same revision, and discovered new nodes automatically.

Another big "nice to have".

Sure they could have taken measures to prevent this kind of problem, I'm not disputing that, but to generalize the problem as no change management, peer review.... um, no sir.

Just imagine (0)

Anonymous Coward | about 10 months ago | (#45208023)

Computers that create no value for mankind now make more in a second than you to all year.

In other news.. (1)

sharknado (3217097) | about 10 months ago | (#45208049)

So...where did the money go?

Re:In other news.. (0)

Anonymous Coward | about 10 months ago | (#45208075)

They went to buying shares for a particular order against some company that wasn't mentioned. Someone saw their stock price go up.

Re:In other news.. (0)

Anonymous Coward | about 10 months ago | (#45208095)

Up with the magic smoke.

Re:In other news.. (2, Insightful)

ShanghaiBill (739463) | about 10 months ago | (#45208143)

So...where did the money go?

The money went to the people on the other side of the trades. Knight lost $400M. Many other people, accumulatively, gained $400M.

Re:In other news.. (1)

TsuruchiBrian (2731979) | about 10 months ago | (#45208201)

To other companies running better trading algorithms.

IT Debt (5, Insightful)

Anonymous Coward | about 10 months ago | (#45208099)

They had some code that processed orders in a special way. There was a flag on the order they could set that would trigger that code. We will call this Power Peg. They later moved away from that functionality but it still existed in the system. It sat there for years untested and unused. 9 years later they added new functionality and decided to reuse that same flag. The new code also disabled Power Peg.

When they pushed the new code into production, they missed a server. That missed server still had Power Peg looking for that flag. Orders started setting that flag and it was processed correctly on all but one server. But that last server was placing orders incorrectly. The logic that Power Peg used was not valid anymore. In a panic they rolled back the code on the servers. Not knowing that Power Peg was the issue, they now had all the servers running Power Peg again.

Re: IT Debt (0)

Anonymous Coward | about 10 months ago | (#45208409)

Unlike the sunmmary, this was actually concise and made sense. Thank you!

return *0; (0)

VortexCortex (1117377) | about 10 months ago | (#45208103)

Stubs that don't bring the system to a halt if called? This is why I assert, C is superior.

Re:return *0; (0)

Anonymous Coward | about 10 months ago | (#45208333)

And if you can't see incrementally beyond that,
then this is not a pointer.

They got bailed out (0)

Billly Gates (198444) | about 10 months ago | (#45208133)

What is neglected in this article is the story how the FEC compensated them for their loss and reversed the transactions.

So nice they get the same protections that you (slashdot readers) and I get when we invest. I wonder if any if it is tax based. Better get to work to pay for someone elses mistakes then.

Re: They got bailed out (0)

Anonymous Coward | about 10 months ago | (#45208427)

They didn't get bailed out. Getco and a couple other firms gave them cash to continue operating, and then eventually Getco bought them out a few months later. A relatively small number of trades were busted

Re:They got bailed out (3, Informative)

Qzukk (229616) | about 10 months ago | (#45208447)

Actually, they asked the SEC to bail them out and got the boot. They had to do a round of private investment and diluted shareholders' value quite a bit. It's generally cited as the "right way" to deal with companies that fuck up and lose billions. Shame we can't do the same with the banks.

RE:Application of TP2anus until the paper is clean (1)

rmdingler (1955220) | about 10 months ago | (#45208185)

Knight had no written procedures that required such a review. And as the deity of your choice only knows, you are not required to expect uncommon sense from your employees.

$172,000 a second for 20.148 years... (-1, Offtopic)

Bartles (1198017) | about 10 months ago | (#45208227)

...would bring you to the point of losing 17 trillion dollars.

Re:$172,000 a second for 20.148 years... (0)

Anonymous Coward | about 10 months ago | (#45208403)

I think your calculations are off.
172,000*60(seconds in a minute)*60(minutes in an hour)*24(hours in a day)*365(days in a year)*20 = $108,483,840,000,000.

I may have not really used math all that much in the past 10-years but I think I still remember this much.

No proper change management, no peer review (-1)

aimua rao vat (3406057) | about 10 months ago | (#45208253)

No proper change management, no peer review, no proper lab testing. Dev should always reflect production to the greatest reasonable level. No proper maintenance windows. You should never be surprised by a change in production. This is a case study in incompetence and the failure to execute industry best practices. I'm guessing the guy or gal who raised the best practices flag was ignored as being inconvenient or too expensive. http://aimua24h.com/ [aimua24h.com]

Re:No proper change management, no peer review (-1, Troll)

Anonymous Coward | about 10 months ago | (#45208291)

Fuck you, spammer.

...had no written procedures... (0)

Anonymous Coward | about 10 months ago | (#45208315)

"Knight had no written procedures that required such a review."
They will now!

New rule! (0)

Anonymous Coward | about 10 months ago | (#45208377)

All trades shall be entered ONLY by a human and signed for in blood!

If there are any illegal problems with your trade. We come get the rest of your blood.

The have your punk kid nephew do it mentality (3, Insightful)

RightwingNutjob (1302813) | about 10 months ago | (#45208387)

Didn't RTFA, but summary makes me go WTF in several places:
1. Python. I thought all the quants liked C, assembler, and even VHDL for their high frequency stuff. No matter
2. "2nd technician to review". If this were flight hardware or a bridge or skyscraper, there would be a second "technician" to review and at least one "engineer" to personally sign off that what was built/deployed is a) done right and b) is what you want
3. "no written procedures". There are a very small number of things in life about which it is absolutely imperative to keep a rod firmly up one's ass: a. moving machinery, b. formal mathematics, and c: hundreds of millions of dollars of your clients and shareholders' money.

Re:The have your punk kid nephew do it mentality (0)

Anonymous Coward | about 10 months ago | (#45208465)

it was someone with a python blog that noticed the SEC report...not that they were using python for the trading

Re:The have your punk kid nephew do it mentality (2)

ShaunC (203807) | about 10 months ago | (#45208497)

It was a Python blogger who happened to report about it, but Knight's code was not written in Python.

They just dropped the fractions of cents on every (0)

Anonymous Coward | about 10 months ago | (#45208405)

transaction....into an account that they created!

(plus one informative) (-1)

Anonymous Coward | about 10 months ago | (#45208577)

and I 4robably [goat.cx]

And nothing (2)

The_Star_Child (2660919) | about 10 months ago | (#45208689)

And nothing of value was lost.
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>