Beta

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

London Stock Exchange Tackles System Problem

samzenpus posted more than 3 years ago | from the fix-the-glitch dept.

Novell 237

DMandPenfold writes "The London Stock Exchange has taken steps to resolve a system problem that occurred at 4.30pm Tuesday, which saw a delay to the start of the closing auction and knocked out automatic trades during a 42 second period. The problem occurred a day after the high profile launch of its new matching engine on the main equities market, based on the SUSE Linux system from Novell."

cancel ×

237 comments

Sorry! There are no comments related to the filter you selected.

Doh! (-1, Offtopic)

dave562 (969951) | more than 3 years ago | (#35229476)

First post?!

Re:Doh! (-1)

Anonymous Coward | more than 3 years ago | (#35229508)

I was going to be first post by I got the story 42 seconds late :(

fail (-1)

Anonymous Coward | more than 3 years ago | (#35229888)

You were a bit more than 42 seconds late, you nub.

Tell your wife I said hi.

first post (-1)

Anonymous Coward | more than 3 years ago | (#35229490)

yay first post

Re:first post (-1)

Anonymous Coward | more than 3 years ago | (#35229542)

So... is this post off-topic or not?

SUSE Linux? (-1)

Anonymous Coward | more than 3 years ago | (#35229528)

Is SUSE still Linux? I thought they would have migrated to the NT kernel by now.

Re:SUSE Linux? (0)

Anonymous Coward | more than 3 years ago | (#35229622)

Of course guilty are SUSE with NT kernel and Ballmer.

First posters are lame (1)

VMSBIGOT (933292) | more than 3 years ago | (#35229560)

So, they replaced a C#/.NET infrastructure with C++/Linux and apparently didn’t test it? Now, I’m not the systems engineer involved with this, but I have a hard time believing that something like this didn’t come up in testing.

At least now they have the ultimate answer, now they just need to work on the ultimate question.

Re:First posters are lame (4, Insightful)

mwvdlee (775178) | more than 3 years ago | (#35229618)

No test can ever be a 100% accurate representation or real use.
It's probably somewhere in the 0.1% mismatch where this problem occurred.

Re:First posters are lame (-1)

Anonymous Coward | more than 3 years ago | (#35230322)

A point I made to Adrian from People Can Fly (team working on Bulletstorm w/ CliffyB) is that in a test scenario with 100,000 people, one of them might find a 1:100,000 bug. But no one can ever afford to test software with 100,000 people off the bat. Simple as that.

As a proponent of .NET programming, I won't say much about their decision to migrate away, but I will say the following: The inverse could very well have held true. If they were changing from a SUSE/GNU C core to C#/.NET, they could easily have faced similar or worse problems. Platform changes don't actually address the underlying issues of incompetent programmers to begin with. Platform changes just wash the bad taste of a now fired programmer out of your mouth when you replace them with someone else who works in a different environment.

Re:First posters are lame (1)

ThePhilips (752041) | more than 3 years ago | (#35232620)

Platform changes don't actually address the underlying issues of incompetent programmers to begin with. Platform changes just wash the bad taste of a now fired programmer out of your mouth when you replace them with someone else who works in a different environment.

I would argue that in that particular case platform matters.

Most modern platforms (C#/.Net, Java) encourage sloppy programming. (*) That works fine on desktops and even in corporate data centers, where minor problems can be brushed off, where excessive resource consumption can be mitigated by throwing in more hardware. But that isn't acceptable for customers like LSE, where even adding piece of dumb hardware might require exhausting test: to make sure that software works with more hardware and the extra hardware itself works without problems too.

(*) Do not get me wrong. As developer, in general, I'm all for the sloppy coding practices, anything what makes life of developer easier.

Re:First posters are lame (2, Insightful)

michelcolman (1208008) | more than 3 years ago | (#35229644)

If I were to introduce a new trading system like that, I would probably run it in parallel with the old system for a couple of months, feeding it the same data and checking the results. I can't imagine them just doing some offline tests and then throwing the switch and hope for the best? Surely not?

Re:First posters are lame (4, Insightful)

pieterh (196118) | more than 3 years ago | (#35229686)

This works for a component with one set of inputs and one set of outputs.

A trading system is essentially chaotic in the way it processes data because it gets so many inputs and their relative arrival times determine the system's behavior. You'd have to be replacing the old system with an identical new one, and then add heavy and slow synchronization to all the inputs going to both systems (so e.g. a trade A hitting the old system one microsecond before trade B also hits the new system the same way).

So yes, it comes down to running a whole lot of offline tests using real data and then bringing it online.

Re:First posters are lame (4, Informative)

iserlohn (49556) | more than 3 years ago | (#35230046)

The system has been online since the middle of last year. Nice for you to drop by and dish out out your golden nuggets of wisdom tough.

Re:First posters are lame (1)

michelcolman (1208008) | more than 3 years ago | (#35230340)

I did add a "surely not" at the end, didn't I?

Re:First posters are lame (0, Funny)

Anonymous Coward | more than 3 years ago | (#35230800)

Oh sorry, I didn't see your get out-out-of-being-a-douche-free card.

Re:First posters are lame (0)

michelcolman (1208008) | more than 3 years ago | (#35230946)

What's a "get out-out-of-being-a-douche-free card"?

Re:First posters are lame (0)

Anonymous Coward | more than 3 years ago | (#35232202)

your mom

Re:First posters are lame (1)

jrumney (197329) | more than 3 years ago | (#35232598)

They have been running it in parallel since at least October. Search for "LSE Turquoise bug" for all the past outages they've experienced. There's lots of talk about "human error", "suspicious circumstances" and "network problems", but noone seems to be pointing the finger at the software itself, which was thrown together in a few months by a previously unknown software development house in Sri Lanka which LSE bought outright for less than they pay Microsoft and Accenture every year for maintenance of the previous system.

Re:First posters are lame (4, Informative)

Anonymous Coward | more than 3 years ago | (#35230576)

The summary is even lamer. It didn't knock out any trading. What happened is that the closing auction announcement was delayed by 42 seconds. Normally any trades submitted prior to the announcement get ignored. However, in this case I'll quote from the article:

LSE traders are required to wait for the message before trading. Normally auction trade instructions sent before the message would not complete, however because of the surprise delay, on this occasion order book trades were allowed to complete later.

So, no outage, no downtime, no lost trades.

Re:First posters are lame (1)

hattig (47930) | more than 3 years ago | (#35230742)

So, no outage, no downtime, no lost trades.

So everything worked fine even when an unforeseen issue occurred and this should be seen as a glowing endorsement of the system that is in place now?

Re:First posters are lame (1)

LingNoi (1066278) | more than 3 years ago | (#35233064)

Perhaps you'd like to compare it to the old system which shutdown trading for a whole day if not longer? I don't remember the specifics.

A 42 second delay at the end of trading is nothing. I really don't understand why people hold this Linux platform to a much higher standard and overlook the poor performance of the previous system which was terrible.

Newsworthy? (1)

Anonymous Coward | more than 3 years ago | (#35229562)

Why is this news? None of the several hour long outage calls I've been involved with were ever on the news.

Re:Newsworthy? (1)

c0lo (1497653) | more than 3 years ago | (#35229578)

Why is this news? None of the several hour long outage calls I've been involved with were ever on the news.

I'm afraid I can't remember what you said about the name of company you work for, can you please repeat it?

Re:Newsworthy? (1)

Anonymous Coward | more than 3 years ago | (#35229584)

Why is this news? None of the several hour long outage calls I've been involved with were ever on the news.

I'm afraid I can't remember what you said about the name of company you work for, can you please repeat it?

One of the major exchanges in Chicago, as well as one one of the bigger global banks. Not a small firm.

Besides, if Reuters or Currenex had this same issue for two hours, we STILL wouldn't hear about this.

Re:Newsworthy? (2)

c0lo (1497653) | more than 3 years ago | (#35229718)

Why is this news? None of the several hour long outage calls I've been involved with were ever on the news.

One of the major exchanges in Chicago, as well as one one of the bigger global banks. Not a small firm.

Ah, I see... well, maybe it would be news-worthy, but... too pity there's too much noise coming from the open outcry trading in the pit... nobody hears about an outage and even during an outage the business goes as usual in that pit... everybody furiously shouting and gesticulating.
(just kidding)

Maybe it will help to compare the LSE trade-volume per day with the same for your workplace?

Re:Newsworthy? (1)

h4rm0ny (722443) | more than 3 years ago | (#35229820)

Why is this news? None of the several hour long outage calls I've been involved with were ever on the news.

I'm afraid I can't remember what you said about the name of company you work for, can you please repeat it?

One of the major exchanges in Chicago, as well as one one of the bigger global banks. Not a small firm.

Besides, if Reuters or Currenex had this same issue for two hours, we STILL wouldn't hear about this.

I can't answer why it is "news", but I can answer why it is on Slashdot. Over the past couple of years, Slashdot has increasingly posted stories that have flamebait as their primary feature and where it isn't intrinsic to the story, it is often editorialised to add a little prompting to get us all arguing. This has an ostensible Linux vs. MS angle hence qualifies to be featured prominently for the sake of the post-count boosting flamewars it could induce. I say "ostensibly" because in reality, any change over between two such large, complex and hyper-used systems will experience problems.

Re:Newsworthy? (5, Funny)

DeBaas (470886) | more than 3 years ago | (#35229730)

Why is this news? None of the several hour long outage calls I've been involved with were ever on the news.

Simple, when a Linux system has a 42 seconds outage that's news. When a windows systems has several hours outage, there is nothing new about.
Odd that you didn't get that..

Re:Newsworthy? (1)

Gruturo (141223) | more than 3 years ago | (#35230268)

Why is this news? None of the several hour long outage calls I've been involved with were ever on the news.

Cause it's about Linux, it's News for nerds, stuff that matters (to us, at least).

Re:Newsworthy? (1)

cbope (130292) | more than 3 years ago | (#35230432)

Actually, cue someone claiming that X millions of dollars or pounds were lost during that 42 seconds of downtime...

It's all about trading speed these days, with milliseconds being critical. Not that I agree, because usually this means someone is gaming the system through automated computer-based trading systems. I would like to see a mandatory delay added to all trades to slow things a bit and let the market become more manageable, by you know, humans. After all, it's we, as humans, that are ultimately doing the trades. The problem is automated trading is too widely used and distorts the markets. I would even go out on a limb and say this was a major contributor to the markets crashing.

Re:Newsworthy? (1)

Americano (920576) | more than 3 years ago | (#35231828)

God, why won't this stupid meme die? "Inserting a mandatory delay" does NOTHING to solve the problem you're asserting it will. All it does is mean that it'll take $delay seconds after market open for high-speed / automated trading to begin hitting the exchange, and it will continue ALL DAY LONG until market close.

This assertion that a delay would somehow make the market "manageable" for individual humans is like saying that making everybody wait in their driveway for 2 minutes before they begin their commute to work will somehow alleviate traffic jams.

Simple (1)

snookerhog (1835110) | more than 3 years ago | (#35231130)

42

Re:Newsworthy? (1)

jrumney (197329) | more than 3 years ago | (#35232670)

Because those several hour long outages happened during the testing phase. From Monday the system went live, so outages are causing real problems now.

Well, well, well... (1, Interesting)

LordNacho (1909280) | more than 3 years ago | (#35229566)

This just shows that it's hard to build these highly available, low latency, massive usergroup systems. Previously there was a lot of chatter about the platforms (.NET, MSSQL 2003, etc...)

The problem is more likely to be internal organisation than specific platforms.

Re:Well, well, well... (4, Insightful)

_Sprocket_ (42527) | more than 3 years ago | (#35229628)

This just shows that it's hard to build these highly available, low latency, massive usergroup systems. Previously there was a lot of chatter about the platforms (.NET, MSSQL 2003, etc...)

Yes. And let us not forget that a lot of that chatter came from Microsoft's PR department.

Re:Well, well, well... (0)

Anonymous Coward | more than 3 years ago | (#35229872)

Yes, that's why you can build also on open platform. MS says you can't.

Re:Well, well, well... (-1)

Anonymous Coward | more than 3 years ago | (#35230146)

LOL

so funny.

You linux geeks would jump up and down like orgasmic bunnies if the system was windows and crashed. Now its linux and you have to kid yourself the O/S isn't tio blame.
Linux is a bug ridden pile of shit, and ever serious IT pro knows this. Its sad to see kiddies pretending this sin't happening.
grow up hippies.

Re:Well, well, well... (3, Interesting)

the_womble (580291) | more than 3 years ago | (#35230428)

Given that this was a showcase client for MS, it does not make them looks good.

Given that the MS was involved in developing the TradeElect system (the Windows based one), so even is the fault lies in TradeElect rather than the MS platform, then its still at least partly MS's fault.

Microsoft and Accenture developed a system that turned out not to be as good as the one developed by a small Sri Lankan company no-one had ever heard of.

Teething problems (4, Insightful)

mangu (126918) | more than 3 years ago | (#35230620)

Previously there was a lot of chatter about the platforms (.NET, MSSQL 2003, etc...)

It's one thing to have a 42 seconds glitch in the first day a totally new system is powered up. That's perfectly normal, and had been predicted [computerworlduk.com] :

"Observers watching today's Linux-based launch will likely note that such a large change could bring about some teething problems, as with any technology overhaul."

It's a totally different thing to have it stop for a whole day after having been in operation for three months.

So, in conclusion, yes, it's about the platform. .NET, MSSQL 2003, etc aren't robust enough for this kind of job.

TradElect & InfoLect lasted longer... apk (0, Informative)

Anonymous Coward | more than 3 years ago | (#35231358)

"So, in conclusion, yes, it's about the platform. .NET, MSSQL 2003, etc aren't robust enough for this kind of job." - by mangu (126918) on Thursday February 17, @07:15AM (#35230620)

Oh, really? That's NOT what London Stock Exchange reps said:

"LSE denied that the collapse was TradElect's fault" - http://blogs.computerworld.com/london_stock_exchange_to_abandon_failed_windows_platform [computerworld.com] by Stephen J. Vaughn-Nichols

The other day here, while I was replying to other "Pro-*NIX" penguins here, I said this:

"TradElect &/or InfoLect being the Microsoft based system - & mind you also, it lasted just fine from 2007 until the crash that occurred, & from what I have been reading about it, it happened on the biggest volume day that exchange ever saw due to some takeover... it remains to be seen if Linux based MilleniumIT's system will handle THAT level of "load" too...!" - by Anonymous Coward on Tuesday February 15, @01:26PM (#35212722) from http://linux.slashdot.org/comments.pl?sid=1995760&cid=35207904 [slashdot.org]

Well, it appears I didn't have to wait long... lol, or even wait for a day of EXTREMELY HEAVY TRADING either!

APK

P.S.=> So far, under this Linux system MilleniumIT, the LSE's crashed TWICE ALREADY, and in less than 1 year!

By way of comparison?

The LSE under .NET systems, TradElect & InfoLect, ran fine from 2007 until it froze on the biggest trading volume day they ever had there (see the article above) due to some "takeover"... that's 2 yrs. or more worth of uptime out of it...

However, only THAT level of "stress on the system" collapsed the C#.NET/SQLServer 2000/Windows Server 2003 combination (not what's happening now to LSE under Linux, 2x now, no less).

(I bolded SQLServer 2000, because BETTER versions of it exist in 2005 & 2008, & I suspect IT was the "weakest link" in that mixture actually - but, more & more, it's looking like the IT Team handling things, actually, were I one to guess...) apk

Re:TradElect & InfoLect lasted longer... apk (1)

Americano (920576) | more than 3 years ago | (#35231858)

Remember folks, you heard it here first:

APK says Windows is better than Linux.

Linux @ LSE, down 2x in less than 1 yr. - FACT! (0)

Anonymous Coward | more than 3 years ago | (#35232890)

First - See subject-line, drink it in & digest it...

"Remember folks, you heard it here first: APK says Windows is better than Linux." - by Americano (920576) on Thursday February 17, @09:52AM (#35231858)

Secondly - For stability on a stock exchange? It appears that Windows DID do a better job @ LSE, because InfoLect + TradLect did stay "up & running" from 2007 until they replaced it w/ Linux, & this failing system... and they replaced it with WHAT that has crashed a 2nd time now, in LESS THAN 1 yr.'s time?? A LINUX BASED SYSTEM!

---

It's just that I posted facts & backing substantiating documetations that you "penguins" just DO NOT LIKE, and CANNOT HANDLE!

( ESPECIALLY ON THE NOTE OF STABILITY, because MilleniumIT is ALREADY CRASHING IN LESS THAN 1 yrs.' time (whereas InfoLect + TradLect, the MS' system LSE had, stayed up & running for what? 2 yrs., & only the HEAVIEST DAY EVER ON THAT EXCHANGE FLOORED IT, no less... )).

APK

P.S.=> However, on the note of "security" as well? Ok, some more backing proofs:

---

KNOWN Linux 2.6 security vulnerabilities, kernel ALONE, & not counting GUI shells ones too (02/15/2011) = 5% (12 of 247 Secunia advisories)

http://secunia.com/advisories/product/2719/?task=advisories

---

KNOWN Windows 7 security vulnerabilities, IN ITS ENTIRETY Gui shell & all (02/15/2011) = 11% (6 of 57 Secunia advisories)

http://secunia.com/advisories/product/2719/?task=advisories

---

Let's see:

---

1.) That's TWICE as many bugs still present in Linux' kernel ALONE, vs. Windows 7 in its entirety...

AND

2.) There were also by way of comparison, what? 2-3x++ as many bugs in Windows 7 patched as there were in Linux kernel 2.6 (which is a LOT older than Windows 7)??

---

NEED MORE? OK:

---

USB Autorun Attacks Against Linux:

http://linux.slashdot.org/story/11/02/07/1742246/USB-Autorun-Attacks-Against-Linux

(Where the "penguin crew" COPIED A FEATURE FROM Windows, one they bitch about no less that could be disabled via TweakUI a decade++ ago, or by registry edits - take your pick! They "blew it" right off the bat/outta the gate on it too... LOL!)

---

Security Warning Over Web-Based Android Market: (ANDROID is a Linux)

http://mobile.slashdot.org/story/11/02/04/181204/Security-Warning-Over-Web-Based-Android-Market

(Showing just "how secure" linux REALLY is... not!)

---

Die-hard bug bytes Linux kernel for second time:

http://www.theregister.co.uk/2010/09/15/linux_kernel_regression_bug/

(Where Linux bugs got "fixed", only to be shown to be EASILY broken thru again, & incomplete!)

---

These additional backing documentations of Linux' security issues? They're ALL VERY RECENT, mind you... & quite indicative of what's coming & yes, even currently HAPPENING on Linux, as far as its security!

So... read 'em, & weep boys, though you may not LIKE IT? It's "how it is"...

(& those backing documentations/evidences above? Heh - only the beginnings of what I saw happen to MacOS X once it got more "market share"! I.E. - it got hacked/cracked more, & exposed more for security weaknesses (lol, after the MacOS X commercials on TV implied "Mac is more secure than a PC"... b.s.!)... apk

Re:Linux @ LSE, down 2x in less than 1 yr. - FACT! (1)

Americano (920576) | more than 3 years ago | (#35233160)

Cool story, Bro.

Americano goes off topic? LMAO! (0)

Anonymous Coward | more than 3 years ago | (#35233392)

"Cool story, Bro." - by Americano (920576) on Thursday February 17, @11:27AM (#35233160)

See subject-line, you off-topic troll - & there's NO DENYING all you have is your puny little "meme" in response to facts I posted? Please...

APK

P.S.=> There you have it readers: When the "penguins of /." (pack of clucking hens is more like it, lol) can't dispute facts, such as the fact THIS IS THE 2nd TIME NOW THAT LSE HAS GONE DOWN UNDER LINUX BASED SYSTEMS IN LESS THAN 1 YR. (far less in fact, vs Windows & how long it lasted @ LSE since 2007 until it crashed on the HEAVIEST TRADING DAY THEY EVER SAW no less only (&with older DB engines too than what is current, for SQLServer too))... what do they do?

See Americano's reply quoted above - Well, as you can see/read? They just go off topic, like it can "save face" for them, or something... lol, & in doing so, they look even MORE foolish: Not that they mind, I do this to that type here, ALL the time (& I have yet to be proven wrong, for 8++ yrs. around here now - why? I use facts!)... apk

Cherry picking much, Mr. Ballmer? (1)

mangu (126918) | more than 3 years ago | (#35232010)

That's NOT what London Stock Exchange reps said:
"LSE denied that the collapse was TradElect's fault" - http://blogs.computerworld.com/london_stock_exchange_to_abandon_failed_windows_platform [computerworld.com] by Stephen J. Vaughn-Nichols

Let's quote from your own link. First of all, the line you put between quotes was entirely out of context, because you omitted the initial "while" and the following lines:

"they also refused to explain what the problem really was. Sources at the LSE tell me to this day that the problem was with TradElect.

Since then, the CEO that brought TradElect to the LSE, Clara Furse, has left without saying why she was leaving. Sources in the City-London's equivalent of New York City's Wall Street--tell me that TradElect's failure was the final straw for her tenure. The new CEO, Xavier Rolet, is reported to have immediately decided to put an end to TradElect."

You also failed to mention that the article's title was "London Stock Exchange to abandon failed Windows platform" and that the author concluded:

"So, might I suggest to the LSE that they consider Linux as the foundation for their next stock software infrastructure? After all, besides working well for Chi-X, Linux seems to be doing quite nicely for the CME (Chicago Mercantile Exchange), the NYSE (New York Stock Exchange), etc., etc."

Linux MilleniumIT system @ LSE down, 2x now? LOL! (1)

Anonymous Coward | more than 3 years ago | (#35232210)

"Sources at the LSE tell me to this day that the problem was with TradElect." - by mangu (126918) on Thursday February 17, @10:03AM (#35232010)

What 'sources' are those? The "horses' mouth" said otherwise:

"LSE denied that the collapse was TradElect's fault" - http://blogs.computerworld.com/london_stock_exchange_to_abandon_failed_windows_platform [computerworld.com] by Stephen J. Vaughn-Nichols

(OR, can't you read?)

---

"You also failed to mention that the article's title was "London Stock Exchange to abandon failed Windows platform" and that the author concluded: "So, might I suggest to the LSE that they consider Linux as the foundation for their next stock software infrastructure? After all, besides working well for Chi-X, Linux seems to be doing quite nicely for the CME (Chicago Mercantile Exchange), the NYSE (New York Stock Exchange), etc., etc." - by mangu (126918) on Thursday February 17, @10:03AM (#35232010)

LMAO - they abandoned a platform that ran SOLID for them for 2 yrs. time... and for what? A Linux one that has crashed 2x now in less than 1 yr.!

(Facts are FACTS, and today's article substantiates my statement above in reply to yours, easily!)

I have noted that Linux does the job elsewhere, just fine, myself -> http://linux.slashdot.org/comments.pl?sid=1995760&cid=35212722 , so, you can't pull the "you're not being fair/complete" b.s. on me...

(I just tell it how it is, and I use backing documentation... like the above quote, for example, FROM THE HORSES' MOUTH LSE (not some "sources" who remained unnamed - that's NOT "good enough" for me!))

APK

P.S.=> Still, the fact remains - LINUX WENT "DOWN FOR THE COUNT" IN LESS THAN 1/2 a YEAR ON THE LSE! and, it's NOT THE FIRST TIME EITHER boys, so... "read 'em & weep" + "argue with the facts" I put out, not myself... apk

Re:TradElect & InfoLect lasted longer... apk (0)

Anonymous Coward | more than 3 years ago | (#35233186)

Are you also the previous Microsoft appolgist that trolled previous articles on the Linux switch? Are you directly or indirectly hired by Microsoft? Just wondering since you're so desperate to do damage control on this.

Re:Teething problems (1)

UnknowingFool (672806) | more than 3 years ago | (#35233300)

To be fair we don't know the exact reasons TradElec went down. It was mentioned that an upgrade went wrong. It could have been the platform.

There was talk that the platform was slow and wasn't performing as expected. The outage was just the last straw. Like a poor performing employee, you'd like to replace him/her but until they screw up royally you don't have enough justification for HR.

Or it could have been working fine but a public embarasment like that needed a scape goat. Someone had to be fired; something had to be changed. Publicly.

Based on the fact that this was the first foray of MS using .NET into this type of application: high availability, low latency, high performance, it wasn't yet mature enough. Contrasted that to Linux which has been used for years in big iron my opinion is that it was a combination of platform and inexperience.

Re:Teething problems (1)

man_of_mr_e (217855) | more than 3 years ago | (#35233396)

You do realize that it's not .NET, MSSQL and Windows 2003 alone that made up the trading system, right? It was also a boat load of custom developed software.

There is no evidence to suggest the problems that TradeElect had were directly the problem with the technology it was based on, but more likely due to problems in the custom software developed for it. I'd venture a guess that Microsoft's own web sites, like Hotmail, MSN, etc.. generate more load and have more complexity than TradeElect, and they handle things just fine.

Likewise, the problem today was unlkely to be a problem with Linux, C++, or any other frameworks or databases they're using. And to claim it was, will be ridiculous.

"Oh well I guess Linux sucks then (1)

atomicbutterfly (1979388) | more than 3 years ago | (#35229572)

... should have stuck with Windows eh?"

I just know some Microsoft fanboys (they do exist, check out Neowin.net for examples) are gonna latch onto any and all problems that occur after the switchover, regardless of merit.

Re:"Oh well I guess Linux sucks then (0)

Anonymous Coward | more than 3 years ago | (#35229592)

Like the Linux fanboys did when they switched platforms?

Re:"Oh well I guess Linux sucks then (-1)

Anonymous Coward | more than 3 years ago | (#35229606)

you mean exactly the same way that Linux fanboys latched onto any and all problems that occurred before the switchover, regardless of merit?

Re:"Oh well I guess Linux sucks then (1)

atomicbutterfly (1979388) | more than 3 years ago | (#35229614)

you mean exactly the same way that Linux fanboys latched onto any and all problems that occurred before the switchover, regardless of merit?

Actually, yes, exactly like those Linux fanboys. I dislike fanboys of any camp.

What? You didn't think I was totally brainwashed by Slashdot did you? :)

Re:"Oh well I guess Linux sucks then (1)

ErroneousBee (611028) | more than 3 years ago | (#35230336)

I am not camp!

Also, I just think Linux is a better operating system, with reduced time spent managing the system, fixing errors and waiting for it to open applications.

Re:"Oh well I guess Linux sucks then (4, Insightful)

slimjim8094 (941042) | more than 3 years ago | (#35229636)

To be fair, if this was Windows you *know* the Linux fans (myself included) would be berating them for choosing such a crappy platform. And it was Windows, and we did berate them for it...

Of course, the truth is somewhere in the middle - isn't it always? The most important part of a trading computer system set up is, well, the trading computer system software. The system's biggest problems aren't going to be due to Coolwebsearch or a bluescreen, they'll be with the trading software itself. The OS doesn't really matter. Buggy software is buggy software.

That being said, Linux is just a better platform to build something like this on. Sure, you can do it with Windows and make it work, but it's just more and unnecessarily difficult.

Re:"Oh well I guess Linux sucks then (1, Interesting)

Anonymous Coward | more than 3 years ago | (#35229722)

That being said, Linux is just a better platform to build something like this on. Sure, you can do it with Windows and make it work, but it's just more and unnecessarily difficult.

I keep hearing this, but never see any technical details. Why is this so? From hearsay, the use of Linux in finance/trading shops is widespread (for servers... clients are typically Windows), it could be true... or it could just be to reduce licensing costs rather than for purely technical reasons.

Re:"Oh well I guess Linux sucks then (3, Interesting)

the_womble (580291) | more than 3 years ago | (#35230442)

Licensing costs are trivial in the context of these sorts of systems. TradeElect cost £40m, the new system was £50m - but that was to buy the company, not just the software.

Re:"Oh well I guess Linux sucks then (0)

Anonymous Coward | more than 3 years ago | (#35230556)

The reason is that Linux is so flexible: you can customize it easily. This is also the main reason why it's useful in supercomputers (see Top500 sorted by OS family [top500.org] as well as mobile phones (Android, Meego) and other places where you want a small and cheap computer [linuxfordevices.com] ).

You can try this out by compiling your own kernel: you can take everything out that is not crucial to the tasks at hand (i.e. processing transactions, for this stock exchange example). That also means your customized system isn't checking all the time if someone is diddling with the mouse or pressing on a key (what mouse? what keyboard?).

I remember a Windows NT system decades ago, where a whole multinational company depended on it, and you could make the entire company's workings slow down to a crawl by moving the mouse around (I'm not kidding!). Probably this has been fixed in more modern Windows systems though?

Per-CPU operating system licensing costs of 0 probably also play a rôle in case of the 294912-CPU Blue Gene/P supercomputer [wikipedia.org] :-)

The Unix Philosophy (1)

mangu (126918) | more than 3 years ago | (#35230894)

That being said, Linux is just a better platform to build something like this on. Sure, you can do it with Windows and make it work, but it's just more and unnecessarily difficult.

I keep hearing this, but never see any technical details. Why is this so?

I had extensive Windows programming experience before I switched to Linux about ten years ago and, basically, it boils down to the Unix philosophy [faqs.org] .

Linux is a system made by programmers for programmers. Windows is a system made following directions of the marketing department.

Re:"Oh well I guess Linux sucks then (2)

anonymous cupboard (446159) | more than 3 years ago | (#35231446)

I worked on a trading system back in the early days. We hit lots of "edge" performance cases. To take full advantage of what a system offers us and to code around problems we usually have source code to look at. We didn't change it, but we had to have the access. MS would gladly give their source code to major customers, but frankly there is more expertise around Linux kernels than Windows.

Re:"Oh well I guess Linux sucks then (0)

Anonymous Coward | more than 3 years ago | (#35231522)

From my previous experience the selection of the Linux systems for financials and/or other high throughput applications (geologic rendering, animation rendering and trade management) the selection criteria has been the wide array of tools, the ease of use managing "many" systems (no I am not talking about SMS or GPO), and the relatively short dev cycle for robust code. In the hiring process for several pipeline projects it seems that the UNIX people already have solved the hard problems, it then becomes a matter of phrasing the implementation.

Sadly... having spent a great deal of time working with the Winders guys on a Unisys project for massive email cluster for exchange servers it seems that the Windows platform does not already have the solution... they are still framing the question.

Re:"Oh well I guess Linux sucks then (1)

ADRA (37398) | more than 3 years ago | (#35229744)

That's a sad but true commentary. If I heard of a high profile Linux->Windows migration that failed in some way, I'm sure I would've jumped on it. The truth is that any new systems roll-out is hard, and the pressures on anything involving large amounts of money is even worse. That's why it takes forever to develop and deploy these systems. I hope their issue was something perceivable and easily addressable.

Re:"Oh well I guess Linux sucks then (1)

the_womble (580291) | more than 3 years ago | (#35230456)

The move to Windows happened three and a half years ago. That is hardly a new system: its mature and its still crap.

Re:"Oh well I guess Linux sucks then (1)

the_womble (580291) | more than 3 years ago | (#35230484)

Sorry, I misunderstood the GP.

Re:"Oh well I guess Linux sucks then (-1, Flamebait)

bjourne (1034822) | more than 3 years ago | (#35230590)

In this case, they didn't just move from Windows to Linux, they also changed the development environment from C# to C++ which definitely is a huge downgrade. They are trading a higher-level language with (builtin) garbage collection, huge standard library, runtime protection, linq and runtime protections for one which hasn't with the only advantage that C++ _may_ be 2-3% faster for some tasks.

Re:"Oh well I guess Linux sucks then (1)

Anonymous Coward | more than 3 years ago | (#35230716)

...changed ...from C# to C++ which definitely is a huge downgrade.

bozo

They are trading a higher-level language with (builtin) garbage collection, huge standard library, runtime protection, linq and runtime protections for one which hasn't ...

Yeah, because the non-deterministic garbage collection, the bloated standard library, 'runtime protections' and linq don't contibute to the solution at all, and if anything add possible failure points.

Re:"Oh well I guess Linux sucks then (0)

Anonymous Coward | more than 3 years ago | (#35230880)

sure .. because the main feature I want in a real-time constrained system is garbage collection....
Jesus, just use the right language for the right task. And get all those managed languages off my realtime lawn! :P

Re:"Oh well I guess Linux sucks then (0)

Anonymous Coward | more than 3 years ago | (#35231228)

Not using built-in garbage collection gives you 10% speedup off the bat.

Re:"Oh well I guess Linux sucks then (1)

Anonymous Coward | more than 3 years ago | (#35231346)

Of course, the difference is that Microsoft were hailing the LSE system as a great example of why you should use Microsoft's software stack, with the usual marketing dreck about how Microsoft software makes everything possible, blah blah blah.

It had problems because of the crap they built on top of it. We all knew that the underlying platform didn't really matter with these kinds of systems, but Microsoft wanted to publically take all the praise and positive publicity. So instead, they took the criticism and the blame.

It kind of was Microsoft's fault. They had a big hand in developing the software that ran the exchange, and they deliberately picked technologies based on what Microsoft could use as a showcase, rather than what actually made sense for the project.

After that, the LSE basically decided to build their own. It happens to use Linux. This doesn't really matter, because the underlying platform is irrelevant with these kinds of systems. It's hardly even worth mentioning, except for the history.

This time, it's their own system. It has a bug. They fix it. No blame, no complaining, and no scrapping the entire system and starting over.

Re:"Oh well I guess Linux sucks then (4, Funny)

michelcolman (1208008) | more than 3 years ago | (#35229668)

They should have just used a Mac. iStockExchange is a free download for MobileMe members.

Re:"Oh well I guess Linux sucks then (1)

tgetzoya (827201) | more than 3 years ago | (#35229732)

I can image them doing that, even after Apple stopped selling their server systems and told everyone to use their mac minis in the meantime.....

Re:"Oh well I guess Linux sucks then (3, Funny)

Anonymous Coward | more than 3 years ago | (#35229770)

But wont Apple keep 30% of the money that flows through it?

Re:"Oh well I guess Linux sucks then (1)

aliquis (678370) | more than 3 years ago | (#35230710)

iStockExchange is a free download for MobileMe members.

But Apple take 30% on every trade ...

42 seconds, that's eternity! (4, Insightful)

michelcolman (1208008) | more than 3 years ago | (#35229616)

Imagine telling a trader in the 1970s that we had a 42 second outage in the stock market, it was all over the news, and a few companies probably lost hundreds of millions in revenue.

Re:42 seconds, that's eternity! (2)

haploc (57693) | more than 3 years ago | (#35230066)

Imagine telling a trader in the 1970s that we had a 42 second outage in the stock market, it was all over the news, and a few companies probably lost hundreds of millions in revenue.

At least it would be The Answer to.. !

Re:42 seconds, that's eternity! (1)

Anonymous Coward | more than 3 years ago | (#35230622)

Imagine telling a trader in the 1970s that we had a 42 second outage in the stock market, it was all over the news, and a few companies probably lost hundreds of millions in revenue.

Now Imagine finding out that there really was not an outage of any length, all the trades completed, and nobody lost anything, but you had already made such claims in a very public forum.

Amazing how reading the actual article will prevent you from looking like a complete idiot.

Re:42 seconds, that's eternity! (1)

Americano (920576) | more than 3 years ago | (#35231944)

Man, I want you to come talk to my boss and tell him that 42 seconds isn't an "outage of any length."

When you work with market data and trading systems, a "significant outage" is anything with time > 0 that your functionality is unavailable. Business critical applications are, susprisingly, considered critical to the operation of the business.

This is why you have hot-spare High Availability / Disaster Recovery configurations, robust monitoring and load balancing, and a team of operations staff standing by to respond at the first sign of a problem. And that's also why it costs tens of millions of dollars to design, build, test and deploy a system like this, despite the script jockeys' assertions that "LOL 6 guys and 6 months could build that."

What I was wondering (2)

aliquis (678370) | more than 3 years ago | (#35230690)

Was the consequences of the loss of the automatic trade?

Everybody won? :D

42 seconds (2)

tal_mud (303383) | more than 3 years ago | (#35229676)

Liffe[sic] the universe and everything.

Re:42 seconds (2)

DamonHD (794830) | more than 3 years ago | (#35229700)

LIFFE != LSE

oops (0)

Anonymous Coward | more than 3 years ago | (#35229694)

The London Stock Exchange has taken steps to resolve a system problem...

Translation: "we noticed ntpd wasn't running and the time on the box was 42 seconds out"

Someone skimped on testing (4, Insightful)

PolygamousRanchKid (1290638) | more than 3 years ago | (#35229756)

I've seen this too often in my 25+ professional years in IT. The system test manager produces an excellent plan, that fully simulates the anticipated workload. But it requires X testers, Y test case developers and Z machines. The program manager rejects the plan, "because he is under pressure to reduce costs." The program manager says, "The testing that the developers do should be enough." He then moves on, before the system goes into production.

The result? It always ends in tears.

Re:Someone skimped on testing (0)

Anonymous Coward | more than 3 years ago | (#35230412)

Looks like they did quite a lot of testing to me...

http://www.londonstockexchange.com/products-and-services/millennium-exchange/technicalinformation/technicalinformation.htm

Re:Someone skimped on testing (4, Interesting)

daid303 (843777) | more than 3 years ago | (#35230760)

Testing is just hard. Even with a perfect plan, and a perfect execution you're still going to miss things.

I just spend 2 weeks finding a 6 year old bug, which only showed itself when the temperature of our product is somewhere at -5C, and then only on 75% of the total devices. Devices where tested at low as -40C, but those where lucky and kept working correctly, so nobody noticed before. Even after I found it I spend a week looking at the code to find what's really wrong.

Re:Someone skimped on testing (1)

pak9rabid (1011935) | more than 3 years ago | (#35231656)

Do you really think with a project this large and crucial, that the people in charge would willingly skimp on testing? I mean shit, this isn't BP we're talking about here. I'd imagine this is a pretty huge system. I'm willing to bet that whatever caused this outage was just simply overlooked when developing the test plan.

Re:Someone skimped on testing (1)

mvdwege (243851) | more than 3 years ago | (#35232140)

Do you really think with a project this large and crucial, that the people in charge would willingly skimp on testing?

Yes.

Re:Someone skimped on testing (0)

Anonymous Coward | more than 3 years ago | (#35232932)

You forgot the part that is usually a problem for me and any project I've ever been involved with for the last 5 years: "It will take X time to write and Y time to run the tests." "But that would push back the delivery date [again], so we can't do it." But the rest of your comment was spot on in either case.

No Suse for me (0)

LZLinuz (1978874) | more than 3 years ago | (#35229804)

Guess I won't download anything Suse.

great website (-1)

Anonymous Coward | more than 3 years ago | (#35229830)

http://www.erotiek-vergelijk.com

Slashdot Meta (-1, Offtopic)

Richard_at_work (517087) | more than 3 years ago | (#35229870)

What is up with the new commenting system? The article says there are 35 comments, the slider at the top of the page says there are 15 full, 0 abbreviated and 20 hidden - but the page only displays 4 comments. Wheres the rest of the "full" comments?!

Sort it out.

Re:Slashdot Meta (0)

Jeppe Salvesen (101622) | more than 3 years ago | (#35229956)

It's possible to revert to the old system. Bliss!

versace sunglasses (-1, Offtopic)

leiqiong (1976250) | more than 3 years ago | (#35230020)

I thought I would leave my first comment. I don’t know what to say except that I have enjoyed reading.Nice blog,I will keep visiting this blog very often. versace sunglasses [sunglasses-club.com]

LSE live in la la land (0)

Anonymous Coward | more than 3 years ago | (#35230186)

I used to work for a company that made a stock market analysis program, only left last November and my house mate still works there. Quite frankly, the LSE have been shit at this. I'm sure lots of people here know how, even at huge companies, they can still be shit at testing, and informing their customers. They where so unrealistic with dates and what their customers could do, their testing was awful and very basic. I've seen better testing done by a bunch of kids doing GCSE course work. Quite frankly, the fact they've only had these problems surprises me, I expected a lot more

Sometimes (0, Offtopic)

inpher (1788434) | more than 3 years ago | (#35230472)

Sometimes I just wish that there was some kind of levy on buying and selling the same stock within a short time frame (say 24 hours), it would probably never hurt long time investors (the ones that are actually important to the companies' whose stock is being traded) and would seldom affect hedge funds and similar accounts.

From my limited knowledge it always seems like this high speed trading is the worst part of stock exchange today. Especially since everything they do echoes in the job market and affect the companies whose stock they buy and sell without any actual interest in the companies. For all they care the high speed traders never even bother knowing the names of companies whose stocks they buy and sell.

I have a hard time believing this was the intent of creating a stock market.

Re:Sometimes (0)

Anonymous Coward | more than 3 years ago | (#35230782)

FWIW, there's a 0.5% levy on the purchase price. Details here. [direct.gov.uk]

Proposed patch (1)

Norsefire (1494323) | more than 3 years ago | (#35230866)

-- sleep(life_the_universe_and_everything);

What hour ? (1)

psergiu (67614) | more than 3 years ago | (#35231196)

4:30. What were the sys admins doing 10 minutes before this ? ;-)

Re:What hour ? (0)

Anonymous Coward | more than 3 years ago | (#35232622)

4:30. What were the sys admins doing 10 minutes before this ? ;-)

Being sysadmins of course :)

LSE - Software Problems Again - ROQ IT comment (0)

Anonymous Coward | more than 3 years ago | (#35231786)

We commented on this situation here: http://www.roqit.co.uk/news/roq-it-comment-on-lse-migration-issues/17/2/2011

Stephen Johnson – Director, at Quality Assurance and Testing specialist ROQ IT, said, “This has the hallmarks of a poorly controlled project lacking in appropriate process and quality gates. Large migration and integration projects are notoriously difficult and must be treated with respect. The only way to deliver these systems is to have clear process and communication and the Quality Assurance team having a respected place at the top table. It’s amazing to think that either these issues weren’t discovered in a year of testing or that the decision was made to go live despite these show stopping issues. This clearly shows the value of having an independent testing team or function that is willing to stand up and stop chaos prevailing”.

What's a closing auction ? (1)

doperative (1958782) | more than 3 years ago | (#35232468)

"Due to a technical problem, that system sent out the daily automatic message to clients - stating that the closing auction was beginning seconds late"
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?
or Connect with...

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>