Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Wall Street and the Mismanagement of Software

Soulskill posted about 2 years ago | from the of-barn-doors-and-horses dept.

Businesses 267

CowboyRobot writes "Last week, a bug in high-frequency trading software from Knight Capital Group resulted in erroneous trades costing almost a half-billion dollars. So, what went wrong and how can they, or any other software developer, prevent something similar from happening again? In hindsight, it's clear that the developers did not verify the code under enough conditions. But the real issue is how these high-frequency trades work in the first place. Robert Dewar at Dr. Dobb's suggests the financial industry needs to take a page from the avionics rulebook, which has very strict guidelines about what code can be implemented due to the high cost of failure in that field. 'High-frequency automated trading is not avionics flight control, but the aviation industry has demonstrated that safe, reliable real-time software is possible, practical, and necessary. It requires appropriate development technology and processes as well as a culture that thinks in terms of safety (or reliability) first. That is the real lesson to be learned from last week's incident. It doesn't come for free, but it certainly costs less than $440M.'"

cancel ×


Sorry! There are no comments related to the filter you selected.

That's What We Did (3, Insightful)

crrkrieger (160555) | about 2 years ago | (#40944573)

Back in the late 90s when I was system admin for a trading company, they recruited me from a place that did 911 computer aided dispatch software. My shop, at least, recognized that some of the same reliability issues were at stake, so some people get it.

Re:That's What We Did (0, Troll)

Anonymous Coward | about 2 years ago | (#40944605)

They should arrest whoever made the software along with whoever approved if if they new it was faulty.

Also, any CEO should be arrested as well.

And Romney murdered a person's wife.

Re:That's What We Did (-1)

Anonymous Coward | about 2 years ago | (#40945099)

Obama is showing what he thinks of women. We will see if women agree in November.

Women will ignore that they can't get jobs and that the president is going out of his way to expand the executive branch and destroy any rights he does not agree with. They will ignore it as long as they can have non-protective sex without the risk of pregnancy for free.

Re:That's What We Did (2)

gl4ss (559668) | about 2 years ago | (#40944617)

loss stop.

trading is done against other people or other peoples computer algos, so no matter what I guess you had some sort of loss stop? or pre-allocated funds which were manually accepted to go into trading? puzzling thing about knight capital is that they seemingly had neither and the robot had access to whole company credit as if were..

Re:That's What We Did (4, Insightful)

Anne Thwacks (531696) | about 2 years ago | (#40944961)

You seem to assume that they want to stop the "accidents". You misunderstand the senario - the whole thing is a distributed Ponzi scheme. It works like a sort of "reverse pass the parcel". Each transaction takes a small percentage in brokerage fees as the underlying "commodity" gains or loses value. However, over time, the brokerage fees exceed the true worth of the commodity has in real life.

The spectacular "losses" are the path by which the money leaves investors, and is syphoned off to pay the disproportionate brokerage fees.

As a point of reference see gold - the value of gold traded between speculators each day is 1,000 times the value actually sold into industry/jewelery/etc - so a 0.25% brokerage fee on both buyer an seller is worth 5* the value of the gold that enters or leaves the market each day. Obviously, the additional margin actually paid by real life users for the benefits of a liquid market is a tiny fraction of the value of the gold - sometimes there has to be a "software problem" or "rogue trader" or, to use the colloquial term "scapegoat", to provide the money distributed on brokerage fees.

I am not suggesting any one person is responsible for this - the banks have colluded to look the other way while this system has developed in an ad-hoc manner. In all probability, many of the so called "masters of the universe" are too stupid to understand what is going on, as it requires an understanding of the laws of maths, which they generally don't have. The few "whizz-kids" who could understand are "highly motivated" to look the other way.

Wny not just tax trades? (5, Insightful)

Anonymous Coward | about 2 years ago | (#40944589)

First 100 trades in a day: free
Next 1000, taxed at 0.02%
Next 1000, taxed at 0.1%

And so on.

This would do wonders for the problem.

Re:Wny not just tax trades? (0)

Anonymous Coward | about 2 years ago | (#40944619)

You've failed to define the problem. The captcha is telling ... envious

Re:Wny not just tax trades? (0)

Anonymous Coward | about 2 years ago | (#40944643)

The "you're just envious, WAAA!" meme only plays to your base. It doesn't encourage anyone to change their minds AT ALL. It just makes you look more like the dick you are.

Re:Wny not just tax trades? (-1)

Anonymous Coward | about 2 years ago | (#40944869)

The "you're just envious, WAAA!" meme only plays to your base. It doesn't encourage anyone to change their minds AT ALL. It just makes you look more like the dick you are.

And "Tax the rich!" plays to parasitic idiots dependent on government handouts.

Re:Wny not just tax trades? (1)

Anonymous Coward | about 2 years ago | (#40945049)

The "you're just envious, WAAA!" meme only plays to your base. It doesn't encourage anyone to change their minds AT ALL. It just makes you look more like the dick you are.

And "Tax the rich!" plays to parasitic idiots dependent on government handouts.

Does that statement also cover the legion of rich bastards who depended on government taxdollar handouts to avoid bankruptcy not that long ago?

Re:Wny not just tax trades? (1)

BVis (267028) | about 2 years ago | (#40945081)

What's your point?

Re:Wny not just tax trades? (0)

Anonymous Coward | about 2 years ago | (#40944623)

This would just complicate the software and legal structure a bit further as the major trading houses suddenly created n/100 subsidiaries (for n = 80th percentile of daily trade volume) to do the trading.

Re:Wny not just tax trades? (-1)

Anonymous Coward | about 2 years ago | (#40944625)

First 100 trades in a day: free
Next 1000, taxed at 0.02%
Next 1000, taxed at 0.1%

And so on.

This would do wonders for the problem.

Because like all things government, once the government gets it going it won't know when to stop.

Re:Wny not just tax trades? (4, Insightful)

sycodon (149926) | about 2 years ago | (#40944745)

Outlaw HFTs They just pervert the market.

Re:Wny not just tax trades? (5, Funny)

ShieldW0lf (601553) | about 2 years ago | (#40944835)

If code like this works better than human judgement does, doesn't that mean we've made rich people obsolete?

Re:Wny not just tax trades? (0)

Anonymous Coward | about 2 years ago | (#40944921)

I think the whole point is that it doesn't necessarily work better.

Basically, we thought we handed the market over to GLaDOS to manage, and it turns out it was actually Wheatley.

Re:Wny not just tax trades? (1)

leonardluen (211265) | about 2 years ago | (#40945125)

so...GLaDOS operates the system so efficiently that she gets bored and decides to kill everyone, mostly just to see what would happen.

Wheatley on the other hand manages the system so poorly that it all explodes and kills everyone.

seems either way you get the same result.

Re:Wny not just tax trades? (2)

Shavano (2541114) | about 2 years ago | (#40945093)

They long ago decided that the middle class was obsolete.

Re:Wny not just tax trades? (0)

Anonymous Coward | about 2 years ago | (#40944859)

And what, exactly, is an "acceptable" frequency of trading?

Re:Wny not just tax trades? (1)

Anonymous Coward | about 2 years ago | (#40944905)

You make a simple rule that "all trades must be initiated by a human", or words to that effect. That is, after all, exactly how it used to work in the past, and it worked just fine.

Also, Trading Places would have been a far less interesting film if Eddie Murphy were replaced with a HFT algorithm.

Re:Wny not just tax trades? (2)

digitalaudiorock (1130835) | about 2 years ago | (#40944965)

...or as others have suggested in the past, put a miniscule tax on each trade. That alone would be enough to make it go away. Either way, you're totally correct, there's just no place in the market for that BS.

Re:Wny not just tax trades? (1)

mister2au (1707664) | about 2 years ago | (#40945047)

HFT by itself does not pervert the market ... it creates liquidity which typically reduces volatility as well

There are 2 main concerns though:
1. algorithm errors which can cause runaway price movements
2. front running

in the first case, it has no impact on long-term investors and typically little impact on short-term traders but can act as a catalyst for wider sentiment moves .. but then so can many things (major bankruptcies, terrorist attacks, move in debt markets, etc) when the human traders are looking for triggers

typically, the market will correct straight back to its original level and someone will have profited on the opposite side of the error

in the second case, this is a major problem which is not strictly HFT but is an add-on to HFT ... it allows the HF traders to effectively take fractions of cents on every deal done at the expense of institutional and private investors (ie investors not traders) in return for increase exchange fees being paid

so effectively the exchange are taking a small backdoor fee on every deal done while allowing HFT to keep some and assumed the real investors wont notice fractions of cents

So i'd suggest HFT is fine but front running isn't - every trader/investor should have equal information or a simple trading lag (eg 1 second) would suffice

Re:Wny not just tax trades? (0)

Anonymous Coward | about 2 years ago | (#40944767)

Taxing won't work, since you can't tax everyone consistently, and anyone can perform a "trade" with very little infrastructure. Folks have this idea that bulk of the trading happens on wall st... it doesn't. Bulk is happening internally within thousands of firms.

That madhouse NYSE Floor scenes you see on TV... that's perhaps only ~0.001% of the total shares traded.

Re:Wny not just tax trades? (1)

sourcerror (1718066) | about 2 years ago | (#40944937)

I would be enought to tax them at 0.0001% each, it wouls still seriously discourage HFTs, as they work with very small margins, but high volumes.

Re:Wny not just tax trades? (1)

BadgerRush (2648589) | about 2 years ago | (#40944943)

This is difficult (maybe impossible) to oversee/audit. The traders will just set thousands of subsidiary companies and each one will do only 100 trades/day.

Re:Wny not just tax trades? (1)

Shavano (2541114) | about 2 years ago | (#40945085)

That would make being a large broker unprofitable.

Not a bug, but a test harness (5, Informative)

Anonymous Coward | about 2 years ago | (#40944609)

Reports have it that rather than releasing a buggy system, the problem was caused by running the test harness in a production environment. More here:

Re:Not a bug, but a test harness (2)

fuzzyfuzzyfungus (1223518) | about 2 years ago | (#40944621)

Running a test system in a production environment sounds like a "buggy system"(albeit a system one level higher in the chain) to me...

Re:Not a bug, but a test harness (1)

Waffle Iron (339739) | about 2 years ago | (#40944851)

Reports have it that rather than releasing a buggy system, the problem was caused by running the test harness in a production environment. More here:

That sounds like the time in 1979 when they mistakenly loaded a nuclear attack simulation tape into NORAD's actual online defense system and nearly started WWIII.

So, sue the developer for the cost he caused. (2)

G3ckoG33k (647276) | about 2 years ago | (#40944631)

So, sue the developer for the cost he caused.

That should teach him a lesson. ;)

Re:So, sue the developer for the cost he caused. (0)

Anonymous Coward | about 2 years ago | (#40944685)

And it will make it real easy to recruit his replacement.

Re:So, sue the developer for the cost he caused. (1)

kraut (2788) | about 2 years ago | (#40944783)

So, sue the developer for the cost he caused.

That should teach him a lesson. ;)

Good luck recovering $400 million from an unemployed coder

Re:So, sue the developer for the cost he caused. (5, Insightful)

140Mandak262Jamuna (970587) | about 2 years ago | (#40944981)

They will sue the unemployed coder for 400 million dollars. Some CXO will certify in good faith he hopes to collect the money. S&P will accept the certificate and rate the credit worthiness of the company AAA. Goldman Sachs will use S&P rating to sell the company to some poor smuck, your 401K mutual fund or my pension fund or our municipalities long term fund at 400 million over the true worth. Everyone involved in the racket will award themselves huge bonus, consultation fee and commissions. That is how 400 million dollars of profits are created, transferred to private individuals and the corresponding 400 million dollar loss for the counter parties are socialized. Either small investors, or government institutionalized investors, or straight forward government bail out. We are always the counterparties who are on the losing end of every such gigantic whoppers.

America once had a great capitalism. Now we have the system where no matter what risk the rich insiders take, all the profits are theirs and all the losses are ours. A system where the ruling elites are protected from the consequences of their actions, where they can rig the game so that they win no matter what, is how societal collapse begins. Jared Diamond's book "Collapse" discusses specific case studies showing how it collapses. Greenland colonization from Iceland, Pueblo Indians, Easter Island were what he discusses in great detail.

Re:So, sue the developer for the cost he caused. (3, Interesting)

Dan Dankleton (1898312) | about 2 years ago | (#40945089)

Oh, for an "I wish it weren't true" moderation.

Re:So, sue the developer for the cost he caused. (1)

TheUnFounded (731123) | about 2 years ago | (#40945021)

I assume this was tounge-in-cheek, but realistically, we're actually headed that way. Licensing boards for software developers, etc. The problem is, as a software developer, it's much more costly to fix 100% of bugs than it is to fix 98% and not worry about that last 2%. If we held every developer to task for every bug in software, we'd have a whole lot less software written, and what would be written would cost 10x as much.

Re:So, sue the developer for the cost he caused. (1)

BVis (267028) | about 2 years ago | (#40945121)

So it's OK that out of 100 planes, 2 crash?

Yes, bug-free software is essentially impossible. But you can get 99.99% of the way there. Yes, it costs a ton, in the short term. In the long term, you save/make money. But, like every other business in America, they're focused on quarterly earnings, not how the company does over 5 years.

Completely wrong (5, Informative)

Anonymous Coward | about 2 years ago | (#40944633)

It wasn't HFT software, it was regular trading software. The developers created a build of the software that included a module that generated lots of silly trades as test data. The software was hooked up to the exchange as a live test to ensure it would talk to the exchange correctly. Unfortunately, they used the software build that included the test trade generator, and those test trades started executing for real.

It wasn't actually a bug; everything worked perfectly. It was more of a configuration management problem.

The only relevance it has to HFT is that if the NYSE limited the rate of trades then a lot less money would have been lost.

Re:Completely wrong (2, Funny)

Anonymous Coward | about 2 years ago | (#40945127)

Unfortunately, they used the software build that included the test trade generator, and those test trades started executing for real.

And no one immediately caught the "real" vs "silly" trades bug because NO ONE CAN TELL THE DIFFERENCE!

Shadow Trading (1)

smtierney (1251288) | about 2 years ago | (#40944639)

Every time I have been involved with re-vamping some site or application that involved the handling of money, we would always shadow the existing system for an undefined period of time to make sure that things were working properly. Can't the trading world implement something similar to real-world test the software without actually launching it live? I always found this invaluable, and a huge stress-reliever.

Re:Shadow Trading (1)

Shavano (2541114) | about 2 years ago | (#40945149)

That only works if you're a small time investor that doesn't affect prices significantly. Once you offer and accept trades in the real world the market starts to react to what you're doing in a chaotic fashion.

And Save What? More Fantasy? (3, Insightful)

knapper_tech (813569) | about 2 years ago | (#40944653)

It's already such a waste that so much talent is getting thrown at problems that seek to make money while producing absolutely nothing. HFT is cleverly sanding in the middle of a river in an eddy and dipping your hand in to tap power without getting pushed downstream. What does Wall Street actually produce? What is their product? Why should we care that they periodically lose their minds and shirts? If anything HFT should be taxed into oblivion so that excellent minds aren't recruited to deliver nothing of social value.

Re:And Save What? More Fantasy? (1)

Anonymous Coward | about 2 years ago | (#40944743)

Exactly. It's not as if anyone died. A company pushed software to production, it was buggy, and they lost $BIGNUM. Shit happens, move on. This is exactly the microsecond-latency Darwinian survival game that Knight and others helped to create; I don't see why I should shed a tear when it bites them in the ass.

Re:And Save What? More Fantasy? (1, Insightful)

dkleinsc (563838) | about 2 years ago | (#40944781)

In theory, what Wall St is supposed to produce is investment directed at useful activity - for instance, if making solar panels is useful, and making fake cold fusion devices is not, Wall St is supposed to ensure that the solar panel company gets investment capital to make more solar panels while the cold fusion company does not.

In practice, this doesn't happen as well as it should because many investors are stupid and believe the hype (e.g. Facebook IPO), and even more try to profit off of other people believing the hype by successfully selling securities for more than they're worth.

Re:And Save What? More Fantasy? (1)

Sique (173459) | about 2 years ago | (#40944867)

It's just the "law of the great numbers fallacy" in disguise. Yes, long term the coin will flip to each side with about the same rate. But for the next coin flip, it's 50%, whatever the current rates are. Each coin flip is completely independent from all the coin flips that happened in the past.

Yes, long term, Wall Street will funnel investments to the right companies, and if you calculate an average over all trades you will find that with 99.% certainity it works. But the next trade is more or less random chance, and with random chance, it will produce a negative outcome. That means each trade can be bad, and just because you had a long sequence of bad trades, it doesn't mean the next one will not be bad too.

Re:And Save What? More Fantasy? (4, Interesting)

fuzzyfuzzyfungus (1223518) | about 2 years ago | (#40944959)

The really troubling thing (to my mind) is not so much the 'what does Wall St. produce?' question(which, as you note, is ostensibly 'capital allocation'; but the 'how efficiently do they actually produce it?' question.

In a non-pathological market situation, you would hope to see Wall St.'s share of the economy as a whole be static or declining(as newer technology makes allocating capital easier and less expensive) and the demand for 'capital allocation' exist only so far as other business sectors find that more efficient capital allocation makes them more efficient and productive(in the same way that you would want to see any other support function of a business kept in line with the business overall. You wouldn't want your IT group consuming a greater percentage of your total economic output every year). Trouble is, that isn't what the numbers reflect.

Instead of acting as other suppliers do, and having their health and size depend on the success of the customers, the financial services sector has managed to capture a steadily increasing share of the value relative to other sectors. Absolute growth would be one thing, if the economy as a whole is growing; but relative growth, in terms of percentage of total output captured, suggests a substantial increase in the market(and regulatory) power of financial services without necessarily any increase in the value of their product to their customers. That is bad.

Re:And Save What? More Fantasy? (0)

Anonymous Coward | about 2 years ago | (#40944843)

In Theroy we could dump Wall Street and banks all along. However, they control also the flow of money between people and companies so they can organize their businesses and private live (people only). If you can provide credits and bank accounts for those two gorups, you could dump the money trinkering device called Wall Street without any trouble.

This will slow them down (2)

TVmisGuided (151197) | about 2 years ago | (#40944667)

Want to stem the flood of HFT software into the Wall Street environment? Pass a law that requires any such software to be written in Ada. Think that one through...

Re:This will slow them down (1)

Vintermann (400722) | about 2 years ago | (#40944787)

You did notice the article was written by the president of AdaCore - developers of GNAT, The GNU Ada complier?

For situations like this, where there's a fixed day the system has to be ready to (due to the opening of a new market) it may actually not be the dumbest thing you could do... but I don't think that is a common constraint for HFT trader programmers. In general, I think Jane Street Capital is more on the right track with their focus on functional programming. It seems you can get a lot of speed, correctness promise and good development time with Haskell/OCaml, if you can find people who are actually good at it (but that's a problem with Ada too).

Re:This will slow them down (1)

TVmisGuided (151197) | about 2 years ago | (#40944865)

Why do you think I mentioned it? Apparently, my sarcasm didn't translate...

Re:This will slow them down (0, Redundant)

dkleinsc (563838) | about 2 years ago | (#40944855)

There's a much better way to stop it, actually: Tax each trade $0.01 per share. This isn't a significant burden on typical investors - you make a few trades of $10000 a year, you pay only a fraction of that in tax. But it is a big deal for the HFTs who are trading millions of shares a day trying to profit off of 10 cent differences.

This is of course not my idea: Tobin Tax []

Not comparable (1)

Vintermann (400722) | about 2 years ago | (#40944671)

High frequency trading programmers can't just talk to the government to try to convince them to give a bigger budget for safety. HFT programmers work under time pressure unlike anything in Avionics, because it doesn't just have to be ready by a certain fixed date, millions in profits may be gained for every day you can shave off development time.

It's interesting, from a programming technical point of view, that functional languages like Haskell and OCaml are used in this domain. They don't offer the correctness guarantees of formal methods-style stuff like Misra C and SPARK Ada, but they have another approach to correctness, and probably much better development times.

From a social point of view, of course, I'm pretty worried about HFT.

Re:Not comparable (5, Insightful)

MickyTheIdiot (1032226) | about 2 years ago | (#40944693)

Is it me or are we continuously using "profits" as a excuse for bad *anything* and pushing that idea to an extreme?

Re:Not comparable (0)

Anonymous Coward | about 2 years ago | (#40945105)

Saying that out loud anywhere in the USA will get you labeled a communist.

Re:Not comparable (0)

Anonymous Coward | about 2 years ago | (#40944717)

"From a social point of view, of course, I'm pretty worried about HFT."

You should be. It does not add anything of value and takes money away from every stock holder, retirement plan.

Re:Not comparable (2)

hierofalcon (1233282) | about 2 years ago | (#40945117)

I'm not sure I'd agree with that part about taking money away from every stockholder, and all. Back in the old days before high frequency trading was in vogue, you would place a stock trade with your broker. Depending on the brokerage firm, it was amazing how many trades that were made claimed to be near the low of the day if you were selling or near the high of the day if you were buying. Of course proving anything was impossible, but the people actually doing the trading were still making a killing.

Now, when you place a stock order you will in most cases in normal markets get it filled at the price that was being quoted when you hit the sell or buy, even if the stock has moved several dollars per share over the course of the day. This is largely due to HFT. Yes, they are making fractions of a cent each share traded and are making lots of money. But they are also providing the opposite side of trades that you want to make at a price near what is being quoted, That lets you, as an individual investor, get the price you have considered to be fair for the security.

Yes, I know all about limit orders and other options to try to control the price you pay or receive and that is always an option, but HFT - when working correctly - has largely eliminated the need for them.

All software trading algorithms are dangerous and can easily cause massive problems when they play against each other or feed on each other. I don't necessarily like them and wouldn't mind them going away. But to say they provide nothing of value is not true -even for a common stock trader who only does a few trades every few months.

Re:Not comparable (1)

UnknowingFool (672806) | about 2 years ago | (#40944833)

The problem isn't the software but the culture. Frankly most code has bugs when released. The problem was there was a deadline and instead of delaying the release until the software was fully verified, they went with the release anyways to meet the deadline. The culture of Wall Street rewards risk taking and abhors rules even if the rules are best practices. I firmly believe one of the leading causes of the meltdown of 2008 was the repeal of Glass-Steagall in the 90s. Commercial banks and investment banks have diametrically opposite philosophies when it comes to money. Add in the pressure of shareholders for growth, it was a disaster waiting to happen. Even now after it turns out that the lack of regulation has caused these crises, Wall Street is still opposed to any regulation.

Warning to execs (4, Insightful)

jerpyro (926071) | about 2 years ago | (#40944679)

Maybe it should serve as a warning to executives to not release buggy software. I know a lot of shops that push things out the door before they're fully baked.

In terms of the stock market, I don't see a problem. The long-term market wasn't affected, no value was created or destroyed, and those who played the game improperly lost out big time. Short term trades on the exchange are gambling. Anyone who tells you otherwise just wants your money. Don't forget, there's always a buyer *and* a seller. Just because Knight lost $450m doesn't mean other people didn't gain $450m.

Re:Warning to execs (0)

Anonymous Coward | about 2 years ago | (#40944909)

That is what will probably really come of this, bots that look to exploit bugs in other trading bots to rack up quick $450m gains. Or at least try.

Quick, cheap, good- pick two (0)

Anonymous Coward | about 2 years ago | (#40945109)

Disclaimer- I haven't actually worked with algorithmic trading systems, but I do work in a bank and I read job descriptions, heard rumors, etc.

As far as I know, some algo systems have releases during the day. If a trader notices some opportunity, he might want system modified to exploit that opportunity, and changes to be released & deployed on same day before trading closes. If that is true, for developers working under these conditions it must be the most stressful position I could imagine. Although as far as I know these developers are paid very very well. But even with plentiful resources at your disposal, there is only so much testing you can do over several hours. So I do believe mistakes will slip by and cause losses.

I guess this is a matter of strategy and priorities. You might have (relatively) slow well tested releases that don't lose you 450 million, but you will miss on some opportunities to make money as well. Or you go with very flexible and less well tested the bleeding edge systems, potentially make more money, but risk losing big like this time. Knowing appetites for risk in finance industry, they are probably going with 2nd approach...

maybe some like FAA code reviews (1)

Joe_Dragon (2206452) | about 2 years ago | (#40944699)

maybe some like FAA code reviews there they just don't let any piece of code go in to a autopilot system.

Not a problem (2, Funny)

fastgriz (1052034) | about 2 years ago | (#40944709)

The SEC usually gives them a mulligan when a software boo-boo costs a big institution a lot of money. I never get a do-over when I make a bad trade though...

Early-Breaking News: AGILITY! (5, Funny)

Tackhead (54550) | about 2 years ago | (#40944711)

A "safety culture" has infused the entire industry, with hazard/safety analysis a key part of the overall process. Until the software has been certified as compliant with the standard, the plane does not fly.

Blair K., Certified Master of the Scrum, responded: "Well, that doesn't sound like a very agile process to me! "Certified" and "compliant with a standard" sound pretty waterfallish. Why not just have a 15-minute standup and decide to launch the plane? At last the aerospace industry could deliver aircraft on time and under budget."

Customer wants their plane painted hot pink? We can totes do that, bros! Shouldn't take more than 24 hours to get to Home Depot and get a few cans of spraypaint. Delivered! And if bits of paint peel off at altitude and get sucked into the engine, gluing themselves to the turbine blades until catastrophic failure of an engine, well, we can just patch the paint recipe in the next sprint! Paint that's "hot pink" is part of this sprint. The user story about engines that don't fail is part of the next sprint.

The real problem with aircraft design is that all our little user stories are in a big clunky database. If we printed out the database's contents (by hand!) on little 3x5 index cards, then we'd be using the best practices of both Scrum and Kanban. Our planes would be so damn agile they'd have turning radii measured in inches.

When a senior engineer piped up that an aircraft with a turning radius measured in inches would kill everyone on board due to G-forces measured in the thousands of Gs, and would likely tear itself apart because the centripetal force far exceeds the tensile modulus of steel, titanium, carbon fiber, or anything else available, he was terminated because "switching from traditional tube-construction to blended-wing-body design made of unobtanium" was part of the next epic.

Re:Early-Breaking News: AGILITY! (1)

Vintermann (400722) | about 2 years ago | (#40944825)

Very funny, but the avionics industry is based on real specs. As in, non-junk specs from people who actually know what they're talking about. In a perfect world, business software would have that too.

But since it doesn't, doing enterprise systems in formally verified Ada would be equally much of a disaster in practice.

Re:Early-Breaking News: AGILITY! (0)

Anonymous Coward | about 2 years ago | (#40944911)

Very funny, but the avionics industry is based on real specs. As in, non-junk specs from people who actually know what they're talking about. In a perfect world, business software would have that too.

Fair enough. Airplanes have been around long enough (and failed hard enough) that we know most of the failure modes and can write specs around them. A safety culture also involves digging back to find a root cause, and that's something that's not incompatible with agile development.

But the Knight Capital situation (and countless like it) could arguably have been prevented with something as simple as a checklist before a handoff/delivery/installation process. What is the version of the software we're about to install? Was it intended for test or production? Has anyone signed off on it being ready to hand off to a production machine? Has the person in charge of the production machine compared the version they received is indeed the version they were promised? Etc. Etc. Etc.

Re:Early-Breaking News: AGILITY! (0)

Anonymous Coward | about 2 years ago | (#40945011)

"But for every second that our new shiny algorithm spends NOT in production, our competitors might be getting millions of dollars that we should be getting! I don't care that the dev team has been on a thirty-hour shift already, give them another case of Red Bull and get the new code out there five minutes ago!"

I guaran-fucking-tee that's what happened, because that's the attitude endemic to the financial software industry. Aviation is willing to say to the customer "hey, there's been a holdup in integration and test, and it looks like we might ship a couple weeks late." And the customer nods and says "okay, I guess" because otherwise people might die.

Re:Early-Breaking News: AGILITY! (1)

Rob Riggs (6418) | about 2 years ago | (#40944983)

Very funny, but the avionics industry is based on real specs. As in, non-junk specs from people who actually know what they're talking about. In a perfect world, business software would have that too.

Exactly. First, imagine a typical American MBA. Now imagine the requirement specs from that MBA.

If you imagined written specs, you failed.

Re:Early-Breaking News: AGILITY! (1)

140Mandak262Jamuna (970587) | about 2 years ago | (#40945095)

You don't work for Fluent, or do you?

what they really mean (2)

slashmydots (2189826) | about 2 years ago | (#40944713)

What they really mean is they were too lazy to write "catch code." They spend all that time making it hyperintelligent then lazy out and don't write any checks and stops and tests into it. Just a tiny bit of heuristics would have detected that as a trade that maaaaybe might need a human to review for a second first. Even every version of Halo is the same way. Yes, is "shouldn't" happen but put in a check in the movement engine to see if the player is currently moving faster than running or falling should allow. Tada, no "superbounce" glitch. For MW3, if someone gets a score of 15 kills and 0 deaths with 100% accuracy, MAYBE they might be cheating and should be booted from the game. A little AI goes a long way, people. But nope, programmers are just too lazy. Once the product is "done," they're out of there!

Re:what they really mean (2, Insightful)

Anonymous Coward | about 2 years ago | (#40944845)

You probably aren't thinking enough about the 'problem area' here. Note that I personally think HFT should be taxed into obscurity as it produces nothing and has the potential to cause real damage. But setting that aside, the whole point of these algorithms is to be HIGH FREQUENCY. Ie: fast. If you can get a quote (order) into the market, one ms faster than your competitor, you can make the profitable trade and your competitor won't. Any checks and balances you put in - will result in slower algorithms. It is a constant balance to squeeze out even a few more nanoseconds of performance versus putting in checks. You can make it perfectly smart and safe - and you will never make any money because its slower than the rest. Put in too few checks, and you risk losing a lot of money when some exceptional circumstance occurs that wasn't covered.

Clearly Knight erred on the wrong side of the balance here, but within the context of HFT, its not just a matter of being 'too lazy' to write the checks.

Disclaimer: I worked for a while in a HFT firm.

Re:what they really mean (2)

Walter White (1573805) | about 2 years ago | (#40945075)

What they really mean is they were too lazy to write "catch code." ...

Hardly the case.

Trading systems make money by being the first to respond to incoming bids and offers. Add some "catch code" and the system will lose that race and make no money. Trading system developers routinely examine the code in system calls and libraries to see which calls will execute the fastest and if they can write custom code that will execute faster. It's not laziness but rather a design requirement.

Secure software? (-1)

Anonymous Coward | about 2 years ago | (#40944715)

Why should they Bother? If they screw up the Goverment will step in and bail them out.

no incentive (0)

Anonymous Coward | about 2 years ago | (#40944721)

why would they invest in robust software? It doesn't make the chance to win much higher, it would only make the chance to lose lower.

And, as we have seen over and over and over again, on wall street, profits from wins go to you, but damages for losses goes to everybody else.

There's no downside (1)

cellocgw (617879) | about 2 years ago | (#40944731)

When HFT "works," the trading company makes tons of money. When a "bug" hits -- and said bug causes a loss, rather than an unintended gain --, the trading company writes it off its taxes or gets TARP-III to cover. Why worry about bugs? Or, more accurately, it's like being chased by a bear. You only need to run faster than the other guy. Fewer bugs in your HFT code than Other_Big_Trading_Co and you're ahead of the game.

Totally different motivations (0)

Anonymous Coward | about 2 years ago | (#40944747)

The aviation industry is driven by two things: regulations and fear of being sued. It will naturally support safety and the way they do software is the result of that.

The financial industry, on the other hand, is driven mostly by greed. Never mind tweeking the software, the whole industry needs to go back to the kind of regulatory environment we had before Regan. Finance should be a lot more like plumbing and a lot less like the wild wild west.

If we try to regulate the financial industry's software, they will, guaranteed, find a way to circumvent those efforts.

High Frequency Gambling (2, Interesting)

dutchwhizzman (817898) | about 2 years ago | (#40944753)

It should be called high frequency gambling and taxed as such. It has nothing to do with the (perceived) value but only with a gamble on what the sentiment and competing algorithms will produce as the next stock or derivative price.

Any derivative trade and any stock trade that is done within 28 days of purchase should be taxed as gambling. It's nothing more or less than that so it's fair if these big online casino's get their profits taxed so the rest of us can profit too.

Re:High Frequency Gambling (0)

Anonymous Coward | about 2 years ago | (#40944887)

It should be called high frequency gambling and taxed as such.

Ok, but in many countries of the world gambling winnings aren't taxed.

Gambling winnings are taxed in the USA, but so are stock market gains.

And if you earn your living from gambling in the USA, then you can deduct your losses & business expenses (not unlike being a HFT trader).

Re:High Frequency Gambling (2)

ed1park (100777) | about 2 years ago | (#40944925)

Warren Buffett suggested a 100% short term capital gains tax to eliminate all market volatility to foster real growth and investment. So simple, it's genius. Short term capital gains is all investments less than 1 year old. We could start off with just securities.

Occupy Wall Street should make this their top goal.

Re:High Frequency Gambling (1)

Anonymous Coward | about 2 years ago | (#40944993)

used to be here that gains on stock owned less that three years were taxed as regular income

Re:High Frequency Gambling (1)

fuzzyfuzzyfungus (1223518) | about 2 years ago | (#40945053)

What I'd like to see, just for giggles, would be a rewrite of the rules for various games commonly gambled on(eg. poker) into the forms and terms of 'legitimate' financial transactions, followed by some test cases in jurisdictions where gambling is tightly regulated; but derivatives trading(that, um, just so happens to be based on the future outcomes of certain playing card distributions...) is not. You might even be able to treat your winnings as capital gains, rather than income, and write off your 'capital losses'(which I'm pretty sure you don't get to do on normal gambling losses)...

One loss of $500m is another's gain of $500m (5, Informative)

Anonymous Coward | about 2 years ago | (#40944771)

This isn't the equivalent of $500m of infrastructure burning down. The money was lost by some investors, but gained by other investors.

tax all HFT profits at 80% (0)

Anonymous Coward | about 2 years ago | (#40944777)

The state of the economy confirms that trickle-down is failing us: if so-called "wealth creators" give up or leave then it doesn't matter as they're doing us no good anyway and can just be replaced by people more interested in society than a few extra dollras.

Test People (3, Interesting)

stuffduff (681819) | about 2 years ago | (#40944823)

"Anyway, no drug, not even alcohol, causes the fundamental ills of society. If we're looking for the source of our troubles, we shouldn't test people for drugs, we should test them for stupidity, ignorance, greed and love of power." -- P. J. O'Rourke

costing almost a half-billion dollars (1)

bug1 (96678) | about 2 years ago | (#40944847)

Cost the company half a bil, the rest of the market says Thanks :)

It's all ethically wrong so karma is served. (1)

pointyhat (2649443) | about 2 years ago | (#40944861)

High frequency trading is ethically wrong and high risk so this sort of shit should just be accepted as collateral from the investors. Instead they sue the shit out of each other. Unfortunately, it's survival of the fittest and richest asshole with the best lawyers at the end of the day.

As for bugs, they happen. HFT/algothmic trading is based on constant optimisation hacks which are rarely tested properly. There are no test. A model is provided, someone codes it and they throw it out. This can take a matter of minutes to hours. Don't expect it to work every time.

If you piss around with funds via HFT, expect to get burned.

(I used to work in HFT - it's a scam).

Re:It's all ethically wrong so karma is served. (1)

Shavano (2541114) | about 2 years ago | (#40945055)

It doesn't matter how much testing you do if your system doesn't have an absolute failsafe condition that simply stops trading when things start looking dicey. You can never model what other HFT programs are going to do to the market because they keep changing and they are the only thing you're playing with on the HFT bandwidth. Real investors are too slow to respond to make a difference to your program.

Re:It's all ethically wrong so karma is served. (0)

Anonymous Coward | about 2 years ago | (#40945155)

High frequency trading is ethically wrong

Cite? I'm serious. Whose ethics? Based on what?

I think the article is premature. (1)

tazan (652775) | about 2 years ago | (#40944897)

I don't think the problem was a software bug at all, or that they deployed without enough testing. Another article mentioned they deployed their test system with the production software. I think this was probably a packaging issue. Or even a network issue, where they plugged their test system in to the live network accidentally. It's entirely possible the problem is too much software testing.

Good explanation (0)

Anonymous Coward | about 2 years ago | (#40944939)

The original article is well written guess work. This is much closer to the likely explanation:

The software was not buggy - the deployment was buggy as it contained a test component that was not meant to be on in production.

Knight is not a HFT or day trader, they are a bonefide liquidity and service provider. They make money by providing a service and getting people cheaper trades

Can't test this enough. Period. (1)

Delgul (515042) | about 2 years ago | (#40944945)

Under normal circumstances you can build a test-suite that will present the software with a large number of scenario's and see if it still gives the 'correct' answer after a code update. However, here the 'correct' answer is not really clear. You are trying to test the results in a highly chaotic system of which you do even understand the mechanics, cannot simulate correctly and cannot measure. On top of that, your software is going to actively influence the dynamics of the very system you are working in, making things even more difficult. Impossible to test anything but some basic stuff really...

You could create software monitoring the actions of the trading software to some extend however. This is probably the only way to go...

Just let them blow up (1)

GlobalEcho (26240) | about 2 years ago | (#40944953)

It makes no sense for everyone to be so concerned about the survival of companies like Knight -- especially people opposed to algorithmic trading in the first place. Just let firms like Knight blow up! Their loss is others' gain, after all.

There was half an hour of wacky behavior in certain stock prices during Knight's whole blowup process, but that affected essentially zero long-term investors. Long-term investors don't need protection from this sort of incident.

Now, a flash crash is a bigger deal since it is more of a market-wide disruption. I still believe that long-term investors have little to worry about in one since the essential characteristic of such an incident is that it is over quickly (certainly none of my personal investments were ultimately affected by the flash crash). But, to the extent it is worth regulating to prevent another flash crash, I think software verification would be an overcomplicated and ineffective means of achieving that goal.

I'll also state that long-term investors have little to worry about from high-frequency traders. The whole point of HF firms is that they make a few pennies per trade. That's far less than the brokerage fees paid by long-term investors, so why complain about the profits made by HF firms rather than by the grasping brokers?

This is not a S/W development problem. (1)

Walter White (1573805) | about 2 years ago | (#40944967)

Avionics and trading systems differ in a fundamental way.

Once the plane leaves the ground (or even reaches significant speed on the runway) any malfunction becomes catastrophic. And it becomes catastrophic to third party participants. This warrants the extreme measures taken to vet avionics S/W, H/W, pilots, manufacturing, maintenance and so on.

Financial systems have no such characteristic. A problem can result in losses that can run up to catastrophic levels only if allowed to run unchecked. And the losses accrue to the organization operating the S/W. Third parties are harmed only to the extent that they trusted the trading organization. In this case it was not the bug that resulted in $440M loss but rather the lack of oversight that would have shut it down before the loss grew so large. Had someone been watching, it would have been obvious that there was a problem within seconds of the open and the system could have been shut down and rolled back.

Anyone suggesting that the S/W should have more checks built into it to prevent this problem has a shallow understanding of the domain. In order to maximize profits the trading system must be the first to respond when an order or bid is received. Developers routinely examine the source code for system calls to determine which ones will execute the quickest in order to minimize latency in the system. The systems also reside in closets in the same building that houses the exchange to reduce network latency. They are not going to add additional code to provide sanity checks. And in any case, it is unlikely that sanity checks can be guaranteed to catch all problems so there will always exist the need to monitor system behavior.

Another facet is 'time to market' for the trading S/W. Having the "next great algorithm" ready in days vs. weeks or months may mean the difference between making money or not.

This approach entails risks that the business people are aware of and willing to take based on the potential reward. It boggles my mind that the business interests at Knight did not address these risks by putting into place effective monitoring that would have prevented this catastrophe.

It really has little to do with S/W development and all to do with addressing and managing business risk.

Avionics != Cutting edge (2)

torkus (1133985) | about 2 years ago | (#40944973)

Sorry but this whole idea doesn't 'fly'. Avionics are *NOT* the type of cutting edge technology used in stock market matching engines or HFT engines. Avionics are designed to be utter reliable to such a degree that they wind up using older tech. The deployment and approval cycle is also long enough that 'new' for a plane is probably years out of date.

HFTs on the other hand, are bleeding edge systems with essentially a cost-is-no-factor approach. You're talking about a world where microseconds are very litterally counted. 10G and higher network connections - not for data throughput but because it lowers latency but a small but appreciable amount. No, they obviously don't want the FUBAR situaion Knight had because of pushing tech but to assume the stock market is using tech with any resemblance to what's in the DreamLiner shows a lack of understanding of both worlds.

Let's use a car analogy! Sure you can make a race car utterly reliable and safe - it's called a Volvo.* It will undoubtely get you to the finish line for race after race after race with no maintenance while the cars meant for the race break down, crash, need maintenance and so on. Just like race car accidents, you don't usually hear about trading mistakes unless they're spectacular.

*substitute your own preferred car mfg

Here's a proposal (1)

Shavano (2541114) | about 2 years ago | (#40944987)

Assess actual damages to other investors on the exchange caused by high frequency trader actions to the high frequency traders that caused the meltdown. Since nobody can absorb a loss of that size without being destroyed, traders will exercise reasonable caution or other investors will wind up owning all of their assets.

Checkout the story, informative (0)

Anonymous Coward | about 2 years ago | (#40945001) The "bug" was mis-calibration.

Step 1 (2)

ilsaloving (1534307) | about 2 years ago | (#40945005)

Stop bailing these bastards out every time they screw up.

Remember it's a zero sum game (0)

Anonymous Coward | about 2 years ago | (#40945019)

hey, we should all rejoice - Knight lost a bunch of money which means someone else made that much money. Financial markets are a zero sum game so for each dollar lost by Knight, someone else (let's hope it was Joe the Plumber ;) ) made a dollar.

You cannot make competitions 'safe'. (3, Insightful)

Remus Shepherd (32833) | about 2 years ago | (#40945025)

The purpose of avionics is to get a plane from one point to another without incident.

The purpose of automated stock trading software is to make as much money as possible while screwing the other guy if you get the opportunity.

You'll never make automated stock software 'safe'. Its purpose is inherently risky and combative. You're not up against the laws of physics and the occasional thunderstorm; you're up against other people who have similar software and an urge to hurt you. This is Wall Street PvP (that's Prick-versus-Prick). It's unsafe by its nature.

You cannot make competitions entirely 'safe'. What you can do is pen them in so that they do not hurt bystanders. Just like putting crash walls around a NASCAR track, we need to put up regulations around Wall Street so their blood combat does not spill out and harm the larger economy. Re-implementing Glass-Steagall is the least that we can do to keep Wall Street's fiery crashes from hurting the common people. There are probably more reforms we could make to wall them off properly.

A waste to treat symptoms (0)

Anonymous Coward | about 2 years ago | (#40945039)

This is just a symptom of the problem. It is time to place a minimum holding period of 3 months or more on all stocks and derivatives to channel people's time and resources back into production that has social value.

Oh, Really? (1)

RivenAleem (1590553) | about 2 years ago | (#40945051)

So, what went wrong and how can they, or any other software developer, prevent something similar from happening again?

There one sure fire way of preventing this from happening again, and it is a quite obvious one too. Stop.

Operations Trumps Development (0)

Anonymous Coward | about 2 years ago | (#40945151)

Although the root cause was with software -- be it due to errant inclusion of test code, an actual bug, misconfiguration, etc. -- this is fundamentally an operations issue.

When you are losing ONE MILLION DOLLARS every MINUTE, somebody better wake up, kill the programs, unplug the network ports, call the exchange, and so on.

The most perfect of development processes, static verification, rigorous testing, inclusion of safety checks and "fail-soft" won't help you if your organization lacks the structure to properly deal with an operational crisis. They would have been better served understanding how Curiousity's Mission Control works, rather than studying their developers.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>