Beta

Slashdot: News for Nerds

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Message Storm Knocks NYSE Offline

Zonk posted more than 9 years ago | from the fall-down-go-boom dept.

Businesses 163

ninjee writes "The New York Stock Exchange is re-examining its network after it was forced to close four minutes early at 3:56pm on Wednesday (1 June) because of a communications glitch. Trading opened on time (09:30 EDT) the following morning but the outage irked traders and raised questions about the reliability of a network described as 'ultra reliable' following improvements made in the wake the September 11 terrorist attacks. The outage stemmed from a fault in a system designed to distribute market data and operate computer trading systems. NYSE Chief Executive John Thain said that both the main system and its backup were swamped with error messages, Reuters reports. He added that the exchange would carry out remedial work designed to prevent any repetition of the problem."

cancel ×

163 comments

The Swear They've Got It Fixed (2, Funny)

geomon (78680) | more than 9 years ago | (#12752657)

They will begin beating the squirrels at precisely 3:55 EST from now on.

Re:The Swear They've Got It Fixed (1)

MyNymWasTaken (879908) | more than 9 years ago | (#12752740)

The squirrels must first be covered in hot butter sauce.

"Ultra Reliable".. heh.. (2, Funny)

Chordonblue (585047) | more than 9 years ago | (#12752756)

Why not just say, "Unsinkable" instead?

Re:The Swear They've Got It Fixed (0)

Anonymous Coward | more than 9 years ago | (#12752815)

There is an ancient African proverb that goes, "New York smells very bad, particularly when traders take their jackets off. It's always best to circumsize them before letting them near small animals."

Re:The Swear They've Got It Fixed (1)

geomon (78680) | more than 9 years ago | (#12752868)

Yes.... I've heard the immigrants from Africa saying this before.... It went something like this:

"Ubuntu.... Ubuntu..... Ubuntu......"

Re:The Swear They've Got It Fixed (1)

Stanistani (808333) | more than 9 years ago | (#12752860)

Here Try Again [nyse.com] - Poor Squirrels.

Re:The Swear They've Got It Fixed (1)

nametaken (610866) | more than 9 years ago | (#12753240)


Oops, someone forgot to shut off the Messenger Service.

Damnit! Who sent 40,000 print jobs to the laser printer in the pit?!

collisions, broadcast storms! (1)

ImaLamer (260199) | more than 9 years ago | (#12753314)

I told them to throw away all those hubs and upgrade to a nice set of Layer-3 switches, no one ever listens to me though...

SCO, of course (5, Funny)

Anonymous Coward | more than 9 years ago | (#12752667)

Immediately claimed the message storm to be the work of linux hackers

True story (0)

Anonymous Coward | more than 9 years ago | (#12753274)

PHB, about to leave for vacation, configures an automatic out-of-office response to any incoming e-mail. Then, not sure that he's done it correctly, he decides to send himself an e-mail...

That message storm brought down our network.

Re:True story (0)

Anonymous Coward | more than 9 years ago | (#12753987)

Sounds like a dogdy mail setup..

Any Linux role? (-1, Troll)

bogaboga (793279) | more than 9 years ago | (#12752682)

Was there any Linux role here? If not someone should be fired.

Re:Any Linux role? (1)

MasterDirk (659057) | more than 9 years ago | (#12752719)

Linux gives the sysadmins indemnity? That's nice.

Re:Any Linux role? (1)

v3xt0r (799856) | more than 9 years ago | (#12753146)

well, the difference between linux sysadmins and MS sysadmins, is when shit hits the fan, MS sysadmins blame windows, and linux sysadmins can only blame themselves.

We take responsibility for our systems, since we usually build them from scratch.

Windows sysadmins blame Microsoft, because they aren't smart enough to build their own customized system, and are too easy to replace if they take the responsibility.

Re:Any Linux role? (1)

superpulpsicle (533373) | more than 9 years ago | (#12752722)

I knew someone who worked in the tech area of the NYSE floor. They used to run alot of M$ windows clusters. Not sure if it's still true.

Re:Any Linux role? (0)

Anonymous Coward | more than 9 years ago | (#12753680)

not sure about NYSE, but lots of banks, even on their trading desks still run MS. Mostly (I think) because Bloomberg and perhaps Reuteurs only have Windows (Excel) APIs. Therefore, lots of traders, investment bankers do their modelling in Excel with VB. Imagine that!!

It's unbelievably but true.

Re:Any Linux role? (2)

taped2thedesk (614051) | more than 9 years ago | (#12752805)

Was there any Linux role here? If not someone should be fired.

And if Linux was involved, everything would be ok? Oh, that's right. I don't know what I was thinking.

Re:Any Linux role? (0, Funny)

Anonymous Coward | more than 9 years ago | (#12752941)

Yes, if you are using Linux you are incapable of wrong. Using Windows means you are probably a 50 IQ point cretin who licks his feet, drinks toilet water and eats raw poptarts. Using Linux means that you are inviolate, as pure as the Virgin Mary and a giant among men.

Re:Any Linux role? (0)

Anonymous Coward | more than 9 years ago | (#12752975)

But I like raw poptarts....

Re:Any Linux role? (1)

Eastree (719351) | more than 9 years ago | (#12753932)

> Using Linux means that you are ... as pure as the Virgin Mary ... Well, this *IS* Slashdot XD Of course, we're not pure *minded*, though.

Actually Linux is common at NYSE (1)

ad0gg (594412) | more than 9 years ago | (#12753221)

IBM is also supplying Linux workstations optimized for high-resolution graphics, but consume less power. Traders will use the workstations to pipe information from the exchange floor to trading desks upstairs in real-time.

Source [internetnews.com]

NYSE is an IBM shop, using DB2, websphere. Its competitor, NASDAQ is using a Microsoft solution [microsoft.com] . Not a good week to be IBM.

YACSF (-1, Flamebait)

Anonymous Coward | more than 9 years ago | (#12752698)

Yet Another Closed Source Failure. When will these jackasses learn that only FOSS can prevent this sort of thing.

Oh well, it's only money.

Re:YACSF (1)

coolsva (786215) | more than 9 years ago | (#12752737)

And, what is the point in making a very specialized system that they have into open source. Like, we would have a swarm of people trying to help NYSE. Please remember, OSS only applies when the software in question is needed by the many so that the some can derive value in enhancing it

There is the other spin of the story (2, Funny)

KZigurs (638781) | more than 9 years ago | (#12753302)

T-Shirt:
"I built NYSE IT backbone and all I got was this lousy t-shirt"

CV (Some poor teenager applying for underpaid software QA position):
Work Experience:
- Leading developer in development of mission critical heavily distributed and absolutely fault tolerant system that can handle 20m transactions per second (NYSE).

Re:YACSF (1)

DogDude (805747) | more than 9 years ago | (#12753048)

If you're gonna troll, you really should do it logged in, so that more people will see your post. Granted, it would've been modded down to -1 in about a nanosecond, but still... any yahoo can troll as an AC.

Re:YACSF (0)

Anonymous Coward | more than 9 years ago | (#12753946)

"When will these jackasses learn that only FOSS can prevent this sort of thing."

Put your money where your mouth is [microsoft.com] , Jackass.

Great! (-1, Troll)

Medieval (41719) | more than 9 years ago | (#12752707)

Now that we've discussed this to death for the last six days on NANOG and similar lists, Slashdot and its various clueless idiots can have at it.

Go!

Re:Great! (-1)

Doc Ruby (173196) | more than 9 years ago | (#12752750)

As usual, old news to geeks is new news to Nerds.

Re:Great! (4, Insightful)

Atzanteol (99067) | more than 9 years ago | (#12753013)

I never saw the slashdot editors claim they were 'late breaking' or 'first on the scene'. Where in hell did you people ever get that illusion?

Re:Great! (0, Offtopic)

Doc Ruby (173196) | more than 9 years ago | (#12753074)

Err, that's what I said. Why are you flaming me? Maybe your post is "Where whining and over-reacting is par for the course."

Re:Great! (1)

Atzanteol (99067) | more than 9 years ago | (#12753239)

If you *expect* things to be late, or at least not up-to-the-minute, then why in the hell are you complaining about it?

Frankly I'm rather sick of the "This was on foo.com last week - old news!" and "again slashdot is reporting old news!" posts.

Your post fit the "whining" portion of my sig.

Re:Great! (1)

Doc Ruby (173196) | more than 9 years ago | (#12753616)

You're really nuts. I posted only

" As usual, old news to geeks is new news to Nerds."

That's not a complaint. It's not even about Slashdot. It's a sarcastic comment about Nerds. What's your problem?

Microsoft's Fault (-1, Redundant)

Anonymous Coward | more than 9 years ago | (#12752710)

I blame MS, Gates and Ballmer. If they were 100% open source this would never happen.

Anyone notice (0)

Anonymous Coward | more than 9 years ago | (#12752718)

That the stock market rallied after GM announced 25,000 job cuts. Looks like high unemployment to drive down the cost of labour is the new in thing.

Oh and btw Slashdot nobody can read your f**king anti-script image half the time. God job discriminating against those with less than perfect eyesight...which is half your audience.

Re:Anyone notice (1)

EnronHaliburton2004 (815366) | more than 9 years ago | (#12752761)

Looks like high unemployment to drive down the cost of labour is the new in thing.

It's not new unfortunately. I remember saying something very similar when GM laid off a ton of people in the early 90s and their stock went up. And this sort of behavior predates my statement by a very long time.

The irony here: At the same time that GM & Ford were shutting down their factories in my state, Toyota & Honda were opening factories. So their whole argument about an "unfriendly business environment" doesn't hold much water.

If I want to be a patriot and buy American, I'm going to buy from the company who actually employ Americans thank you Mr. Ford & Mr. GM.

Re:Anyone notice (0)

Anonymous Coward | more than 9 years ago | (#12752848)

If I want to be a patriot and buy American, I'm going to buy from the company who actually employ Americans thank you Mr. Ford & Mr. GM.

What if you just want to be fucking asshole?

Re:Anyone notice (0)

Anonymous Coward | more than 9 years ago | (#12753151)

Move to California

Re:Anyone notice (1)

Sj0 (472011) | more than 9 years ago | (#12754224)

Buy a used car.

Re:Anyone notice (1)

Acid-Duck (228035) | more than 9 years ago | (#12752797)

Proof that democracy DOES exist on slashdot. Enough ppl complained and they removed it, hooray! I can tell you I have a good enough eye-sight (don't wear glasses, don't need 'em as of 3 months ago). Some images were tougher to read then others. At least 2 out of 6 times I just couldn't read the image

Re:Anyone notice (1, Informative)

Anonymous Coward | more than 9 years ago | (#12752878)

Proof that democracy DOES exist on slashdot. Enough ppl complained and they removed it, hooray! I can tell you I have a good enough eye-sight (don't wear glasses, don't need 'em as of 3 months ago). Some images were tougher to read then others. At least 2 out of 6 times I just couldn't read the image

I bet you say that to all the transexual prostitutes.

Re:Anyone notice (1)

Monf (783812) | more than 9 years ago | (#12753267)

i think the biggest problem was that they had images that contained uppercase and lowercase, but the answers had to be in lowercase...

Re:Anyone still post on-topic messages here? (0)

Anonymous Coward | more than 9 years ago | (#12754300)

Well?

This is messed (1)

CriminalNerd (882826) | more than 9 years ago | (#12752728)

Man, this is messed. Sooner or later, somebody will probably blame Windows or whatever excuse they can think of ATM. Also, I don't think it'll be just squirrels that will be beaten at 3:55. XDXDXD

Re:This is messed (1)

MightyMartian (840721) | more than 9 years ago | (#12752791)

Actually, I'm changing tact. I've decided to blame Alfred Hitchcock. I think it's healthy to spread blame around. Next week, I've decided everything will be your fault.

I used to monitor the NYSE for IBM. (www.nyse.com) (1)

Thaidog (235587) | more than 9 years ago | (#12752746)

It is "ultra reliable' but you've got to remember the amount of hits this site takes a day... it makes /. Trolling look like a fairy Godmother!

Re:I used to monitor the NYSE for IBM. (www.nyse.c (1)

BengalsUF (145009) | more than 9 years ago | (#12753768)

Except, of course, that we're not talking about the public website here, but some of the trading systems themselves. A little bot of a difference there.

Re:I used to monitor the NYSE for IBM. (www.nyse.c (1)

Thaidog (235587) | more than 9 years ago | (#12753855)

Yes those systems are not IBM's... they are SIAC.

no final print (1, Interesting)

gbasin (890301) | more than 9 years ago | (#12752751)

I as well as many others in my office got royally screwed here, getting stuck with quite sizeable unhedged positions overnight. It's bad enough that order routing went down, but they failed to open up for a final print (as originally proposed) later in the afternoon. Very bad.

Re:no final print (0)

Anonymous Coward | more than 9 years ago | (#12752933)

"I as well as many others in my office got royally screwed here, getting stuck with quite sizeable unhedged positions overnight." good

Re:no final print (0)

pthisis (27352) | more than 9 years ago | (#12753038)

I have no sympathy.

If you're making stock decisions that rely on 4-minute windows, you should expect to be screwed on a regular basis. Really, trying to time market fluctuations more closely than at least yearly terms--and more realistically, 5 year+ terms--is a crap shoot. Barring highly unusual situations, if a company was a good long-term prospect 4 minutes or 4 days ago then it's still a good prospect now. Really, how many of the positions you held made massive announcements that affected the company's prospects in those 4 minutes?

More to the point, whatever you were holding when trades stopped, you were still holding the next day, and the NYSE going down doesn't affect the value of the companies whose issues you held. Nor was your right to buy or sell shares affected, you just lost 4 minutes of service from the largest market for finding buyers/sellers.

Guys like Warren Buffett--or large fund managers-- don't speculate on minute-to-minute fluctations, and if the market was down for a couple of days it wouldn't really affect their ability to make wise investments. And tiny downtimes like this don't materially affect the fundamentals of the market system.

By nature, if you play with day-trading you're accepting a much higher risk, and when that risk materializes I don't feel that bad for you.

Re:no final print (4, Informative)

gbasin (890301) | more than 9 years ago | (#12753085)

Frankly, you have no clue what you are talking about nor to whom. I don't "invest", I'm a professional trader - mostly US treasury future triangular cash/future arbitrage but also quite a bit in stocks (mostly listed) doing M&A arbitrage and occasionally make markets. Technical difficulties are expected, however, if you knew what you were talking about you would understand that many on the street were expecting a final print (including specialists) and they failed at providing it. Your understanding of trading is that of an amateur, there is another side to this business buddy. When the final print didn't come, a large number of professional traders who get out of all or most of their positions by the end of the day got stuck with tied up capital, interest costs, and overnight risk that needed to be hedged as a result of no fault of their own.

Re:no final print (2, Interesting)

wayne606 (211893) | more than 9 years ago | (#12753215)

Was your problem a "fill on close" issue? I.e. did you have trades that were supposed to go through at the closing price but didn't because there was no close that day? If so I can see how this would screw a lot of traders.

Re:no final print (2, Interesting)

gbasin (890301) | more than 9 years ago | (#12753299)

Yes, on the average day I completely hedge myself using MOC (market on-close) orders, as well as open up some new positions against market imbalances on close to be closed off on the ECNs a few minutes after the close.

Re:no final print (0)

Anonymous Coward | more than 9 years ago | (#12753751)

presumably you are a technical trader as fundamentals do not change in days much less mins.

An aside:
Are these min-by-min trading executions automatically done now with minimal human intervention or not? I find it hard to believe it could be done any other way and still being able to watch many markets/instruments.

Re:no final print (1)

gbasin (890301) | more than 9 years ago | (#12754375)

I assume this question pertains to the NYSE (all other markets are fully electronic). The answer is yes and no, a normal order enters the specialists book where his job is to match it, usually quite quickly. You can also use an NX feature for round lots to bypass specialist intervention - used when speed is more important than price.

Re:no final print (2, Insightful)

krbvroc1 (725200) | more than 9 years ago | (#12753225)

While its true that minute to minute trading is speculation, there are many legitimate reasons that people dont want outstanding trades overnight. We are still talking trader/speculators, not investors. If something bad had happened overnight it could have been a disaster to the economy.

I'm not a day trader, but some (most?) day traders follow the rule that you close out all positions at the end of the day and don't leave yourself vunerable overnight.

By your logic it would be best to only open the market once a month. But that would make the market much less liquid and that does materially affect the fundamentals of the stock market system. Even if you are a day trader and accept a much higher risk, the risk of the market not working at all is still outside the norm.

I'm finding it tough to find anything to agree with in your post - my gut tells me even the statement of "large fund managers-- don't speculate on minute-to-minute fluctations" is wrong. People with huge risk try to minimize that risk by hedging short term (even minute-minute) flucations in stocks, options, and currency.

Re:no final print (1)

bluGill (862) | more than 9 years ago | (#12753733)

No, it would be best to open the stock market once every 5 years[1]! These are companies you are buying, not paper. They have value for fundamental reasons, not because the stars are stacked a different way today.

I suppose I should say anything, traders like you are what give students of Benjamin Graham the extra boost to our portfolio in the long run.

[1]Warren Buffet, second richest man in the world (these positions vary from time to time of course) said this. Though I don't recall the extact timeframe he said. He made all his money as a trader. He was Benjamin Graham's most successfull student.

Re:no final print (0)

Anonymous Coward | more than 9 years ago | (#12754288)

We are still talking trader/speculators, not investors.

That's exactly right. Gamblers.

If something bad had happened overnight it could have been a disaster to the economy.

Oh my!! Keep the NYSE running, our economy depends on it! Yeah, maybe the trading firms love the comissions. Ha ha ha. Hilarious.

I'm not a day trader

Good for you. And don't waste your time thinking about what "rules" day traders use. Wanna hear the "rules" for astrology or reading the future from coffee grounds?

By your logic it would be best to only open the market once a month.

Ha! How about once a YEAR? Actually, the only time the market needs to be open is when I personally want to buy something.

But that would make the market much less liquid and that does materially affect the fundamentals of the stock market system.

True, if you think it's important that liquidity be high, all the time (do you feel the same way about, say, real estate or collectibles?), then traders are the "useful idiots" of the stock market.

large fund managers-- don't speculate on minute-to-minute fluctations

Fund managers are mostly idiots (and not even useful ones), so whether or not they speculate is irrelevant.

People with huge risk try to minimize that risk by hedging short term (even minute-minute) flucations in stocks, options, and currency.

You don't minimize risk by taking on more.

The *professional* traders and market makers on Wall Street are simply profiting from their ability to work from within the system (they can do it faster than you can, every time.. they can *reorder* their trades in front of yours if they don't). If I were in their position, I would trade too. But I'm not, so I don't.

Re:no final print (1)

gbasin (890301) | more than 9 years ago | (#12754410)

I take very measured risks, I do not trade based on "technical analysis" and other crap. I do not work on wall street, I am private. I make a comfortable 7 figures, on track to do almost 8 this year - this is all from trading profits, not salary. People like you take for granted the value of real traders such as myself, who get paid by the market to make it more efficient (think arbitrage, market making). I love it how people shoot down others when they have no idea what they're talking about, how do you plan to buy your 1000 shares of GE once every 3 years when no one will sell it to you for a reasonable price? Oh yea, me. Idiots. http://themarkettruth.blogspot.com/ [blogspot.com]

Another high profile Linux failure (0)

Anonymous Coward | more than 9 years ago | (#12752752)

How embarassing. The NYSE should switch to OS X, a REAL OS for REAL computing.

Re:Another high profile Linux failure (0)

Anonymous Coward | more than 9 years ago | (#12752898)

How embarassing. The NYSE should switch to OS X, a REAL OS for REAL computing.

Good plan. "Honest, Bob, the window will open any time now..."

To NYSE (3, Informative)

mcguyver (589810) | more than 9 years ago | (#12752759)

1. If anything can go wrong, it will. (see Murphy's law)
2. Systems in general work poorly or not at all.
3. Complicated systems seldom exceed five percent efficiency.
4. In complex systems, malfunction and even total non-function may not be detectable for long periods (if ever).
5. A system can fail in an infinite number of ways.
6. Systems tend to grow, and as they grow, they encroach.
7. As systems grow in complexity, they tend to oppose their stated function.
8. As systems grow in size, they tend to lose basic functions.
9. The larger the system, the less the variety in the product.
10. The larger the system, the narrower and more specialized the interfaces between individual elements.
11. Control of a system is exercised by the element with the greatest variety of behavioral responses.
12. Loose systems last longer and work better.
13. Complex systems exhibit complex and unexpected behaviors.
14. Colossal systems foster colossal errors.
-KISS

Nice copyright violation (5, Informative)

Anonymous Coward | more than 9 years ago | (#12752849)

...of a wikipedia text [wikipedia.org] . (You didn't follow the terms of the GNU Free Documentation License.)

Re:Nice copyright violation (4, Insightful)

mcguyver (589810) | more than 9 years ago | (#12753236)

I never thought the day would come when someone posts a joke [answers.com] and the respone, on /. of all places, references copyright restrictions. How ironic, if not a sad sign of how times have changed.

Re:Nice copyright violation (1)

Detritus (11846) | more than 9 years ago | (#12753450)

Whether or not someone's copyright has been infringed, it is plagiarism.

Re:Nice copyright violation (1)

Lshmael (603746) | more than 9 years ago | (#12753461)

The GNU Free Documentation License is not about restrictions, but about giving the viewers of your post the same rights that Wikipedia gave you, thereby allowing anyone to continue to spread your "joke."

Re:To NYSE (1)

myov (177946) | more than 9 years ago | (#12754433)

Designing a system to not fail means that it will fail in a way that you didn't plan for.

Twice in the last year, for example, both my primary and backup systems have failed within a week of each other.

Big deal... (4, Funny)

Anonymous Coward | more than 9 years ago | (#12752768)

It's called "leaving early from work".....everyone does it.

Details? (2, Insightful)

CrackHappy (625183) | more than 9 years ago | (#12752772)

I haven't got the time or I would look myself - does anyone have any more informative sources on the specific information about the cause of the problem? And WTF is a "Message Storm"? God - another catchphrase - great!

Re:Details? (4, Informative)

afidel (530433) | more than 9 years ago | (#12752958)

A "message storm" is a storm of data that overwhelms a system, kind of like a DDOS, but legitimate traffic. In this case it sounds like a large number of error messages overwhelmed the message queueing system (probably MQ from IBM), which likely set off an even larger storm of error messages when backed up messages started to expire.

Re:Details? (1)

MightyMartian (840721) | more than 9 years ago | (#12753001)

Did you notice the ticker going "PR0N... V1AGRA... DIPLOMAS... PR0N... V1AGRA..."?

Re:Details? (1)

photon317 (208409) | more than 9 years ago | (#12753250)


I've never understood the corprorate programmers' obsession with purchasing message queue systems like MQSeries. You can design and code something that does the 90% that you actually need of what MQSeries does yourself in a week tops. MQSeries is so universal and overly complex for what most people want out of it, which is just reliable transactional networked message queueing with the options of in-order delivery, multiple queuers and dequeuers, and disk persistence.

Re:Details? (1)

conteXXt (249905) | more than 9 years ago | (#12753356)

well of course ... when you put it that way.

shoot, I'll just go and whip one up before bedtime.

Let me guess (1)

tripslash (683760) | more than 9 years ago | (#12752816)

They had to hire outside *nix coders when the in-house MS crew couldn't integrate the existing WinLAN into the (unsupported, shortsighted) Linux rollout last month.

What is the System? (2, Interesting)

putko (753330) | more than 9 years ago | (#12752827)

It sounds like a distributed systems failure, alright.

Here [nysedata.com] is something about the system that might have broken. I'm wondering if the thing that failed really is the thing mentioned here -- the stuff the stuff Birman [simc-inc.org] did. His new book on distributed systems is out [amazon.com] , by the way.

Somone will get flying ninja-kicked in the nuts for this, you can be sure.

how much of it was BT traffic ? (-1, Troll)

phpCypher (882779) | more than 9 years ago | (#12752829)

inquiring minds want to know .. ..

Oh no! (0, Flamebait)

alvinrod (889928) | more than 9 years ago | (#12752838)

It's black Tuesday all over again. Everybody sell, sell, sell! The market is about to collapse!

Seriously, who cares? No system is ever perfect. It's better that it happens now and gets fixed then sometime down the road where it can cause serious problems.

Re:Oh no! (1)

Monkeman (827301) | more than 9 years ago | (#12752904)

1000 "Huh? My computer must be messed up..."s in perfect syncronization. Then a single, "Woah, that was creepy" spoken from 1000 mouths.

I washed my shirt with Ultra Tide. (1)

ARRRLovin (807926) | more than 9 years ago | (#12752951)

That doesn't mean it will never get dirty.

My fault (0, Troll)

Willy on Wheels (889645) | more than 9 years ago | (#12752962)

I was performing page move vandalism on their website!

Transcript of the message storm (5, Funny)

Chairboy (88841) | more than 9 years ago | (#12752991)

JoeTrader: dood, chk out MSFT, 12m volume
XyxyZ: wtf i sold on margin
-- NASDUCK has entered the channel
JoeTrader: rofl!
NASDUCK: whatsup?
JoeTrader: sam sold msft on margin before the spike
NASDUCK: HAHAHA!
JoeTrader: werd
XyxyZ: screw you guys
JoeTrader: OMG roflrofldolololo!!!!!
NASDUCK: you are such a tool, sam
JoeTrader: brb, gotta tell the office
-SYSTEM- JoeTrader has left the channel (sam in a tool)
-SYSTEM-:NASDUCK has changed the subject to "XyxyZ sold MSFT before the spike today!!!:D:D:D"
XyxyZ: fu duck. i hope my boss isn't online
XyxyZ: ops
XyxyZ: +ops
-SYSTEM- Hot2Trade has joined the channel
NASDUCK: nice try, only way to erase that is to crash the server
Hot2Trade: Sam, I heard that you got the horns of the bull shoved up where the bear don't shine
XyxyZ: dude this sux hard
-SYSTEM- JOHN@MLYNCH has joined the channel
NASDUCK: nice one Hot2Trade. asl?
Hot2Trade: fu hippy, this is Jerry in at prudential
NASDUCK: fuc sorry, didn't recognize you :O
XyxyZ: So if I can down the server, I can erase the subject?
Hot2Trade: no worries I just changed my nic
NASDUCK: XyxyZ, you got pwned by the bull
JOHN@MLYNCH: SAM! HAHAHA I TOLDYOU NOT TO SELL!
JOHN@MLYNCH: YOU AER
JOHN@MLYNCH: SUCH A SP
XyxyZ: i got s cript
JOHN@MLYNCH: AZZZ!!!!!!!!!!!!!!!!!!!
XyxyZ: take this bitches
XyxyZ: THE C THE R THE I THE M THE I THE N THE A THE L
XyxyZ: THE C THE R THE I THE M THE I THE N THE A THE L
XyxyZ: THE C THE R THE I THE M THE I THE N THE A THE L
XyxyZ: THE C THE R THE I THE M THE I THE N THE A THE L
XyxyZ: THE C THE R THE I THE M THE I THE N THE A THE L
- SYSTEM - NASDUCK (quit(connection reset by peer))
- SYSTEM - JOHN@MLYNCH (quit(connection reset by peer))
- SYSTEM - Hot2Trade (quite(connection reset by peer))
- SYSTEM - error(91) - rebooting

Re:Transcript of the message storm (1)

udderly (890305) | more than 9 years ago | (#12753052)

Wish I had some mod points--that's too funny! Thanks for the giggle.

pain in the ass (5, Informative)

rcamera (517595) | more than 9 years ago | (#12753033)

as a trading engine developer/support guy for a financial firm in ny, i can't stress enough what a pain in the ass this was. the day after the nyse crash, it took hours upon hours of verifying (by hand) trades that the nyse says we were filled on that we never say (because all nyse trading lines were down).

this type of 'message flood' occurs from time to time, but not on the nyse in a while. it's generally the ecms trading otc stocks that have rouge programs blast orders in an infinite loop. when this happens to an ecm, they slow down but generally don't lose the ability to trade. the nyse, who toutes the importance of their rapists^H^H^H^H^H^H^Hspecialists because they add 'stability' to the system, was dead in the water. this crash goes to show how useless the specialists really are - without the technology working, they can do nothing. if this is the case, why not just replace them altogether with electronic trade matching?

interestingly enough, the nyse announced mere months ago that they are 'merging' with archipelago - a large ecm. perhaps this merger will be the beginning of the end of the specialists.

Re:pain in the ass (1)

Saeger (456549) | more than 9 years ago | (#12754246)

perhaps this merger will be the beginning of the end of the specialists.

Fewer unnecessary middlemen is always a good thing; now they can specialize in something that's more useful than being an overpaid floor trader. Mmm.. automation cuts from both sides.

Re:pain in the ass (0)

Anonymous Coward | more than 9 years ago | (#12754370)

okay acronym boy. wtf

Reuters? (1, Interesting)

ajs (35943) | more than 9 years ago | (#12753042)

"the main system and its backup were swamped with error messages, Reuters reports"

Which is kinda funny, since it was *probably* a reuters feed [reuters.com] that was spewing the errors in the first place....

At least... (1)

niteskunk (886685) | more than 9 years ago | (#12753081)

At least this didn't happen at the begining of the trading day, the very last thing we need is strain on the economy...

tibco? (1)

synx (29979) | more than 9 years ago | (#12753157)

I wonder if NYSE uses tibco rendevouz for their message transport "bus". My work uses this software, and our usage of it has stressed it to extremes and you can end up with message storm issues.

FYI this system is a multicast-based publish-subscribe system. The multicast thing tends to be a wash IMHO, especially since many people use it for queues, rather than true 1 to many messaging.

Re:tibco? (1)

beeblebrox (16781) | more than 9 years ago | (#12753425)

I worked at tibco, but not in the messaging side of engineering - it's only a smallish part of what they do. Here's what I remember, modulo time and lack of coffee:

- NASDAQ uses tibrv (formerly Rendezvous). This featured a lot in tibco's pitch about reliability/scalability etc. NYSE, I *think*, uses MQ Series.

- Rendezvous used to only run over IP broadcast, with software "router" daemons for crossing subnets. This has not been required for years now - remember, the history of Rendezvous stretches back to the late 80s.

- The/a key choking point for tibrv-over-multicast is (hardware) router multicast group table space.

Re:tibco? (1)

synx (29979) | more than 9 years ago | (#12754361)

Ah yes, we've seen the multicast group table problem. Basically you run into it and your system instantly crashes and suffers huge difficult to recover problems. I wasn't directly involved, but we have put in place migration paths away from tibco. Of course we were doing several things wrong:

- Dont have HUGE subject names - significantly reduces performance I have been told.
- dont use multicast for point to point RPC-like services.

Of course its intensely attractive since you can just run a program, not worry about sockets and stuff, and send and receive messages with no problems. But the very same attractiveness has allowed us to build a big pile of crappy middleware ontop of rv. arrrghh!

As for the software "router" daemons (rvrd), we use them to cross over WAN links. This in my experience has been the worst part, especially when you reach low-grade packet loss. What ends up happening is your entire message stream is sent on 1 TCP socket, and it ends up with a max thruput of about 20-30kB/sec so you end up with huge buffering which rvrd ends up unable to recover from. It has made me cry a few times.

Re:tibco? (0)

Anonymous Coward | more than 9 years ago | (#12753608)

I'm not sure what they're using now, but as of 2000, they were using Talarian's SmartSockets. That was back when I worked there, and they IPO'ed. A dude from the NYSE came out to the festivities and told us how important we were. Pretty crazy stuff, I couldn't believe our product was that stable.
Talarian's stock crashed shortly thereafter, rounds of layoffs (hi! :), and eventually, they got bought by the 800-lb gorilla of messaging--their only competitor--Tibco.
It's hard to imagine the NYSE converting all over to Tibco's software, but it's also hard to imagine Tibco keeping Talr's Smart Sockets for very long, as its not a strategy I usually see companies adopt.
Though, if you look up those terms in google, you get sites such as http://www.wallstreettechjobs.com/ [wallstreettechjobs.com] which list "Experience with sockets and messaging including Tibco EMS and
SmartSockets" for some developer job.
Anywho, that's my 5 pesos.. -Ajay [dhs.org]

(Reading slashdot since '97, and never got an account. Oh yeah..)

Re:tibco? (1)

fupeg (653970) | more than 9 years ago | (#12753739)

Tibco's implementation definitely leaves a lot to be desired. Their success has always came from their ties with Reuters (who used to own a huge stake in the company) and thus their use in high profile environments. Its never been because of their technology.

Multicast storm (1)

thelordx (588477) | more than 9 years ago | (#12753176)

This was actually most likely the result of a multicast or "slow consumer" storm. In a multicast network environment, often desktops are overloaded by all of the filtering they must perform (multicast sends nearly everything to everybody). Sometimes some desktops will miss a packet and ask for a retransmission. Often, this involves retransmitting in multicast-form - that is to all of the consumers. If this happens too many times, you get a storm. No matter what the NYSE does (unless they buy our technology - email for info if you'd like), they will not be able to avoid this happening in the future. It is a condition of middleware systems that were not meant to handle the volumes of modern stock markets, rather than a simple glitch in the network infrastructure.

and the irony of it is.... (0)

Anonymous Coward | more than 9 years ago | (#12753190)

... that they use linux [linuxinsider.com] . =P

TIBCO (1)

ManDude (231569) | more than 9 years ago | (#12753268)

"The outage stemmed from a fault in a system designed to distribute market data and operate computer trading systems."

I think they are using TIBCO for their data messaging bus.

. . . the reliability of a network described as "ultra reliable" . . .

The use of the word network doesn't seem to fit. They won't be calling cisco.

The Dude

Revenge of the Dick! (2, Interesting)

Joe Jarvis (712473) | more than 9 years ago | (#12753289)

Somewhere, in a secret underground lair wallpapered with 100 dollar bills, Dick Grasso is laughing maniacally.

This happens all the time (2, Insightful)

Ececheira (86172) | more than 9 years ago | (#12753291)

I work in Technology for a Wall Street firm (you've heard of them). Stuff like this happens all the time -- systems go down and are usually back up pretty quickly, some route to some exchange will bounce for a few min. This time it was worse in that it affected NYSE and not one of the smaller exchanges at the end of the trading day. If you look at any graph showing trading volumes, the last few minutes of trading are always the heaviest.

99.9% of the time, things bounce back very quickly and with the exception of a few internal emails, nobody cares, things go on.

Natural Selection (1)

handy_vandal (606174) | more than 9 years ago | (#12753584)

NYSE Chief Executive John Thain said that both the main system and its backup were swamped with error messages, Reuters reports. He added that the exchange would carry out remedial work designed to prevent any repetition of the problem.

No, the remedial work is designed to cull out less adaptive problems, thus preparing the digital ecosystem for the emergence of tougher problems.

-kgj

This is why... (0, Troll)

JeiFuRi (888436) | more than 9 years ago | (#12753767)

you should only drink natural breast milk

NYSE/OS (1)

poor_boi (548340) | more than 9 years ago | (#12754042)

Your NYSE/OS C: drive appears to be getting full. You have approximately 4% disk space free. NYSE/OS recommends having at least 15% free disk space to properly conduct Big Buck Stock Exchanges.

NYSE/OS has searched your drive for under-used and overly-large files and generated the following report.

Under-used
----------
c:\tickerlogs\SUNW.log 942 bytes

Over-sized
----------
c:\logs\NYSEerrors.log 543 Gb

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Create a Slashdot Account

Loading...