Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

FBI Scolds NASDAQ Over Out of Date Patches

samzenpus posted more than 2 years ago | from the someone-needs-a-time-out dept.

Businesses 66

DMandPenfold writes "NASDAQ's aging software and out of date security patches played a key part in the stock exchange being hacked last year, according to the reported preliminary results of an FBI investigation. Forensic investigators found some PCs and servers with out-of-date software and uninstalled security patches, Reuters reported, including Microsoft Windows Server 2003. The stock exchange had also incorrectly configured some of its firewalls. NASDAQ, which prides itself on running some of the fastest client-facing systems in the financial world, does have a generally sound PC and network architecture, the FBI reportedly found. But sources close to the investigation told Reuters that NASDAQ had been an 'easy target' because of the specific security problems found. Investigators had apparently expressed surprise that the stock exchange had not been more vigilant."

cancel ×

66 comments

Sorry! There are no comments related to the filter you selected.

In a alternate Universe (3, Insightful)

AftanGustur (7715) | more than 2 years ago | (#38164316)

If these had been Linux servers, Microsoft would now be making bold statements about "Linux Insecurity" and urging Everyone to get a complete Microsoft Solution with patch management.

Re:In a alternate Universe (5, Insightful)

Alwin Henseler (640539) | more than 2 years ago | (#38164428)

In an alternate Universe, software would be released not before it's done, bug-free, and not need updates other than to add functionality.

Software quality being what it is today, there's only 2 choices:

  1. If you don't want to patch all the time, disconnect from network so that you have a stand-alone installation (or only use on very strict managed local network).
  2. If internet-facing: patch, patch, patch, so that you have current software with known leaks fixed. In this respect, *nix or Windows doesn't make much difference, the important thing is that it's kept up-to-date.

Re:In a alternate Universe (2)

Nerdfest (867930) | more than 2 years ago | (#38164576)

There does seem to be a pretty large difference in the time between exploit and patch between the two platforms though. You can have Windows exploits go unpatched for months, although occasionally there is a workaround to mitigate the risk.

Re:In a alternate Universe (4, Insightful)

somersault (912633) | more than 2 years ago | (#38164696)

How are you going to guarantee that your software is bug-free? That's like trying to prove that God exists.

Software complexity being what it is today, it's very difficult to make sure that a system is bug free. Even if you didn't rely on other people's libraries, it would be very difficult to do anything non-trivial without introducing some kind of unanticipated behaviour.

Back to the summary, what's wrong with running Windows Server 2003 if it's still getting security updates? Wouldn't it be more likely to be secure than a newer version of Windows Server, which has new features that haven't had as much time to mature?

Re:In a alternate Universe (1)

Anonymous Coward | more than 2 years ago | (#38165044)

Back to the summary, what's wrong with running Windows Server 2003 if it's still getting security updates? Wouldn't it be more likely to be secure than a newer version of Windows Server, which has new features that haven't had as much time to mature?

Even more so, since win2003 doesn't have IPv6 by default. IPv6 software stacks have not been around that long, and many security flaws have been found (not in IPv6, in IPv6 software). Even OpenBSD was caught by an IPv6 flaw.

Unless you really need IPv6, you're much better off disabling it.

Re:In a alternate Universe (1)

WorBlux (1751716) | more than 2 years ago | (#38175888)

Formal proofs, redundant hardware pathways. But ya the proofs grow faster than the size of the code that they prove.

Re:In a alternate Universe (1)

somersault (912633) | more than 2 years ago | (#38176972)

Even if you prove that the code matches the proof - how do you know that the proof itself doesn't contain some false assumption? That you haven't misunderstood the problem that you're trying to solve? You then have to prove the proof. And so on.

Re:In a alternate Universe (1)

WorBlux (1751716) | more than 2 years ago | (#38211548)

The proof is about the code implicitly and proves that it follows a specification. Both are written in precise formal language though the specification less so. If you don't trust it read over it yourself. The proof itself is in a formal language, and the rules for manipulating them clear. There are several formal system, none of which have more than a handful of axioms, and such known systems are mathematically equivalent in what they can describe.

Re:In a alternate Universe (1)

somersault (912633) | more than 2 years ago | (#38212224)

Yes, but your specification may have bugs, which is what I was saying. You can prove that it matches the specification, but that doesn't mean that it's bug free.

Re:In a alternate Universe (1)

rrohbeck (944847) | more than 2 years ago | (#38168668)

In an alternate Universe, software would be released not before it's done, bug-free, and not need updates other than to add functionality.

Sounds like Debian stable to me. Almost.

Re:In a alternate Universe (1)

manu0601 (2221348) | more than 2 years ago | (#38172532)

If you don't want to patch all the time, disconnect from network"

And then you get owned by a USB key. This is how Stuxnet made its way into Iranian nuclear facilities

.

Re:In a alternate Universe (5, Insightful)

Anonymous Coward | more than 2 years ago | (#38164472)

If these had been Linux servers, Microsoft would now be making bold statements about "Linux Insecurity" and urging Everyone to get a complete Microsoft Solution with patch management.

If you think *nix servers in enterprise networks are more up to date then Windows servers, you clearly dont work in the industry.

Re:In a alternate Universe (-1, Offtopic)

Anonymous Coward | more than 2 years ago | (#38164490)

Work on your reading skills, he's talking not about software engineering and systems administration, but PR and marketing.

Re:In a alternate Universe (5, Insightful)

The Askylist (2488908) | more than 2 years ago | (#38164592)

More concerning is the poor firewall configuration. Badly patched servers can be put down to laziness, or unwillingness to fully regression test servers running bespoke software. Badly configured firewalls can only indicate incompetence.

Re:In a alternate Universe (3, Insightful)

Anonymous Coward | more than 2 years ago | (#38164616)

Laziness in following security bulletins and applying critical patches == incompetence of sysadmin.

You're basically saying "Their admins aren't just incompetent, they're incompetent as well!"

Re:In a alternate Universe (0)

Anonymous Coward | more than 2 years ago | (#38164964)

bullshit. More likely a badly written application needed so many open ports that the firewall was virtually open.

Re:In a alternate Universe (2)

Lumpy (12016) | more than 2 years ago | (#38165092)

Poor firewall configuration lies int he hands of the CTO. It's that man that is 100% at fault for that problem.

Been there done that. Had one of those 3 letter assholes demand we punch holes in a firewall that 24 months later a hacker used to get in.

Ditto. (4, Insightful)

khasim (1285) | more than 2 years ago | (#38166006)

It is impossible for a cynic (admin) to get certain concepts through to an optimist (management).

Every day that you are not cracked (or the crack go undetected) is "proof" for the optimist that he was right and you were just pushing unnecessary precautions to justify your job.

So, those 24 months ... that's over 700 times he was "proven" right and you were "proven" wrong.

The same with skipping patches. Every patch skipped multiplied by every day without a crack ... he's right thousands of times and you're chicken little ("the sky is falling, the sky is falling").

Re:Ditto. (2)

Pieroxy (222434) | more than 2 years ago | (#38166490)

This. This. A thousand times this.

Badly written systems are 99% the fault of the management. Do we need an architect? An expert? What for? The unqualified dude told me he can write the software on his own, and man, he's cheap!

Sure. It'll work. Most of the time. And whose fault is it? The unqualified dude?

Re:Ditto. (0)

Anonymous Coward | more than 2 years ago | (#38169528)

This. This.

Try to sound little more pretentious.

Re:Ditto. (1)

Pieroxy (222434) | more than 2 years ago | (#38175176)

I have nothing to say to AC.

Re:Ditto. (1)

Beryllium Sphere(tm) (193358) | more than 2 years ago | (#38169600)

That was the story of the financial crisis. A man jumps off a 100-story building, and after a while says "Look, I have 99 floors worth of data proving I'm safe. That's statistically significant!".

Thats America for you. (-1)

Anonymous Coward | more than 2 years ago | (#38164322)

Security? Whats that? Money isn't a thing to us, don't you guys watch the news? Who upgrades their security systems to protect investments these days, not no one. I say lets get more money stolen! Who's with me!?

Re:Thats America for you. (1)

Qzukk (229616) | more than 2 years ago | (#38164902)

Why bother? If someone breaks in and screws up the prices, they'll just roll back all the trades that hurt Goldman Sachs.

Yummy! (-1)

Anonymous Coward | more than 2 years ago | (#38164402)

Was I the only person who initially parsed the headline as "FBI Scolds NASDAQ Over Out of Date Peaches"?!

Friday! (4, Funny)

Anonymous Coward | more than 2 years ago | (#38164450)

Hey it's friday. let's just go get a beer and skip this patch testing. What could Possibly go wrong?

Go anything else (-1)

Anonymous Coward | more than 2 years ago | (#38164520)

why is it that they are running windows at all? It insecure slow and any other serious OS is fully customizable down to the kernel

Re:Go anything else (5, Funny)

Anonymous Coward | more than 2 years ago | (#38164558)

Your attempt to join our hive-mind is appreciated, but found to be lacking in zeal.

Re:Go anything else (2)

gmack (197796) | more than 2 years ago | (#38164578)

It's all about Marketing. MS Windows is has plenty of speed if you are willing to put the right hardware behind it and the brochure advertising their platform only mentions that their system has the lowest latency when processing stock data and not total cost.

Never underestimate the lazyness of managers (5, Insightful)

Viol8 (599362) | more than 2 years ago | (#38164652)

If they have 2 choices:

A) which is easy to set up and can be run by click-monkeys but is full of security vulnerabilities

or

B) harder to set up and requires people who know what they're doing but is very secure...

the BAs I'm afraid will will always go for A since people will usually trade effort now (setting up) for effort later (clearing up after a hack).

Re:Never underestimate the lazyness of managers (2)

sohmc (595388) | more than 2 years ago | (#38164754)

I see this all the time at my job. The third-tier support are click monkeys. They don't know how to actually manage a domain, but they know how to type and click. So some company makes a GUI for managing users, policies, etc that's as simple is "red light-green light". When we ask them what actually was changed, they have no clue.

It's cheaper to hire click monkeys than to actually hire a Windows Domain Engineer, but they figure that the cost-benefit is better.

Re:Never underestimate the lazyness of managers (0)

Anonymous Coward | more than 2 years ago | (#38164794)

That's because they have no incentive to do otherwise. Nobody actually sues these people and seek serious damages. If that would happen, then it would be WAY more cost-effective to hire competent people and use reliable software.

Re:Never underestimate the lazyness of managers (-1)

Anonymous Coward | more than 2 years ago | (#38164866)

.... hire a Windows Domain Engineer....

Best. Slashdot. Comment. Ever!

Re:Never underestimate the lazyness of managers (0)

tehcyder (746570) | more than 2 years ago | (#38166034)

Translation: I am highly self entitled 1337 UNIX hacker who thinks the world owes me a living.

Re:Go anything else (4, Informative)

SuricouRaven (1897204) | more than 2 years ago | (#38164698)

They run both. The actual trading system (I recall) runs some form of heavily modified real time linux, because the high-speed traders demand crazily fast speeds - they are trading on the microsecond level now, and growing frustrated by the time it takes for a signal to go down an ethernet cable. The Windows servers will be for things like the frontend interface used by the less-high-speed traders.

How?Where NASDAQ's using MS' stuff, & more... (0)

Anonymous Coward | more than 2 years ago | (#38171554)

The article I am reading it from, from E-Week, July 24th 2006 issue, lists it specifically so, quoted next:

"NASDAQ, the largest U.S. electronic stock market, lists companies from 37 countries. Their crucial trading and messaging systems use SQLServer 2005 to handle up to 64,000 transactions per second with 99.999% uptime"

(& it's probably more now, because I am basing that off of 2006 information (but it has stayed up & running 24x7 into the "fabled '5-9's" of uptime doing so).

* "RTOS" (real-time OS) doesn't mean speed, but rather guarantee of delivery (ala for example, no dropped packets)...

(You're making it sound like speed of ops, as far as what "real time" actually means... & though the term MAY sound that way? It's not really about speed, but about guarantee of info. delivery!)

APK

P.S.=> You said NASDAQ's using Linux? Hey - I'd like to see proof of that please - NYSE &/or LSE do, but NASDAQ?? ANYHOW... thank you for proof of your statements!

... apk

Re:Go anything else (0)

Anonymous Coward | more than 2 years ago | (#38169278)

why is it that they are running windows at all? It insecure slow and any other serious OS is fully customizable down to the kernel

Seriously? You think an organization that failed to properly point & click to configure a firewall would be *helped* by having something that's customizable down to the kernel?

Reuters is not much better (3, Informative)

Anonymous Coward | more than 2 years ago | (#38164562)

Reuters which is quoted in the article and which also provide feeds for the market are very slow at providing support updated Windows.

http://thomsonreuters.com/products_services/financial/financial_products/a-z/3000_xtra/#tab3
Reuters 3000 Software requirements:
        Windows Vista with Service Pack 1, Windows XP Pro with Service Pack 2 and Service Pack 3.
        Office 2007 and Office 2007 with Service Pack 1 (with restrictions on Excel 2007 Service Pack 1).
        IE 6.0, IE 6.0 with Service Pack 1 and Service Pack 2, IE 7.0.

Re:Reuters is not much better (0)

Anonymous Coward | more than 2 years ago | (#38164626)

Reuters aren't responsible for handling $bazillions in transactions a day. So your point is...?

Re:Reuters is not much better (1)

hedwards (940851) | more than 2 years ago | (#38165280)

That if they're not responsible for $bazillions in transactions and the NASDAQ is, then perhaps NASDAQ ought to be hiring somebody that knows what they're doing. The fact that there's a large volume of transactions only makes it more important that the machines be properly maintained and patched. What's more it's not like they can't take them down during the week end.

Re:Reuters is not much better (1)

Anne Thwacks (531696) | more than 2 years ago | (#38167118)

Reuters aren't responsible for handling $bazillions in transactions a day.

Are you sure about that? They always used to be, if indirectly. Then again, they used to use PDP8s.

Does my lawn look big in this?

Re:Reuters is not much better (0)

Anonymous Coward | more than 2 years ago | (#38164906)

Reuters does not "provide feeds" They take existing feeds and put that data into software for traders to use. They still use originating feeds from NASDAQ, SIAC, etc.. Comparing what NASDAQ provides and what Reuters provides is comparing apples to oranges.

Re:Reuters is not much better (0)

Anonymous Coward | more than 2 years ago | (#38170460)

Nasdaq is a market. All markets all over the world provide data on what is happening in their market...data feeds.
Reuters and others provide these feeds to everyone else. So Reuters does provide Nasdaq feeds. So for all intensive purposes security issues at the source or security issues at the market data feed handler are both equally disturbing for the consumer. The only difference is I can be a customer of multiple feed distro companies. When someone takes down Reuters I still get my market data. If someone takes down Nasdaq servers no one gets their market data.
So apples to apples....just different types of apples.

Re:Reuters is not much better (0)

Anonymous Coward | more than 2 years ago | (#38173746)

Reuters 3000 is a desktop application for consuming financial data. It is not a server process handling the transactions.

Scolds? (5, Insightful)

aikodude (734998) | more than 2 years ago | (#38164814)

Scolds? Really? What is this, kindergarten? How about a nice hefty fine to make them take security seriously? Oh, I forgot, can't be angering the real bosses. :/

Re:Scolds? (0)

Anonymous Coward | more than 2 years ago | (#38165206)

Why not wait and see what happens; before commenting on this. Oh wait this is Slashdot.

Re:Scolds? (0)

Anonymous Coward | more than 2 years ago | (#38165376)

Also, when did the FBI become the IT experts on systems. Correct me if I am wrong but isn't this the same agency that could not get emails across the country before 9/11?

What the Federal government of the USA needs is to have a Dept of Science and Technology. Under the tech branch would be the IT dept of the federal gov. They would be the ones "in the know" so to speak. You would also toss NOAA, NASA, all the labs from DOE (kill the DoE) in there as well as NSF, etc.

Wow (4, Insightful)

Dunbal (464142) | more than 2 years ago | (#38164844)

NASDAQ makes at least $0.001 in exchange fees for every single transaction that happens on that exchange, and yet they can't hire a competent IT department.

Re:Wow (0)

dintech (998802) | more than 2 years ago | (#38164898)

Valve makes a nice percentage for every single transaction that happens on Steam, and yet they can't hire a competent IT department.

Re:Wow (0)

Anonymous Coward | more than 2 years ago | (#38165586)

Except VALVe *does* have a competent IT department.

Re:Wow (0)

Anonymous Coward | more than 2 years ago | (#38164900)

Recently I did a contract with NYSE Euronext and all we got to work with was Windows XP, Office 2000, Outlook and file shares of some sort. I'm not saying they are insecure, because as far as I know they are, (and I couldn't go anywhere near the transaction side of the business). But if anyone was to try spear-phishing, the infrastructure couldn't be more 'classic'.

To me, in the age of the user-friendly linux desktop, the workstation setup I just described should be replaced to increase security and to avoid spearfishing. Open-Source support has value, blind-love for Microsoft does not.

Spear-phishing is what did Nasdaq in, and via that in-road other (client) corporations were also hacked.

http://www.reuters.com/article/2011/10/20/us-nasdaq-hacking-idUSTRE79J84T20111020 [reuters.com]

Re:Wow (1)

Anonymous Coward | more than 2 years ago | (#38165290)

How would using desktop linux make them safer from spear-phishing?

AFAIK spear-phishing = targetted. They will know what your desktop looks like, what browsers you use, maybe even your patch cycle.

Firefox, Chrome and Opera have had zero-day exploits. Hardly anyone uses stuff like apparmor/SELinux on desktops - when I last checked Ubuntu's default apparmor profile for Firefox was rather lax.

So given the typical Desktop Linux security model, once you're in, you can access everything the user can. Which is all most spear-phishers need - they don't need to tamper with your OS (which is as silly as bank robbers trying to knock down the bank's foundation and walls when they've already got the money and other "good stuff").

Re:Wow (0)

Anonymous Coward | more than 2 years ago | (#38165360)

More than likely they DO know about it.

"I am going to bring server XYZ down for maintenance"
"That server is the one I use to make our company millions and my transactions must go thru!!!"

After about 2-3 rounds of that an exception is created. As the old adage goes he who has the gold makes the rules...

The guy who screamed about it moves on but never tells the IT dept. So the server sits around probably not even being used anymore. But unpatched...

My bet is on that is what happened....

Re:Wow (1)

Anonymous Coward | more than 2 years ago | (#38165410)

I love reading quotes like that... You made me laugh. Financial services are a different beast and there are a couple of risks/issues that have to be understood that make patching very difficult...

First, you can't say the "IT department" as if its a simple thing that can make decisions. The IT department consists of several hundred to several thousand individuals working in different groups with different requirements. Looking at the org chart, most commonly the CTO or COO is the person that links the systems admins to the app dev teams followed by layers of managing directors and managers. Now throw in a team dedicated to information security and you get additional opinions on how to do patching. Its next to impossible to put 10 people in room and get a decision, and these conversations go on for years. In a firm that size, the best time to resolve them is when a new OS (such as moving from windows to linux, or sles to rhel, etc) is being adopted and you can start fresh and create patch management, config management, etc solutions. The only way a new app can use the new platform is to buy into the strategy, resource accordingly, and learn to love it.

Second, testing... As an app owner ( I am not one, but I'll do this in the first person) I test my application on a particular kernel and libraries. Putting a new kernel or library under my application means I need to retest my entire application because there is no way to know what has changed that may impact my application (can you prove API/ABI compatibility? Can you prove that a change won't cause performance regression?). To my management the most important thing is getting my business required releases out on time. Getting my projects finished gets me a bonus. Patching is not important to me and I'm going to ignore it. The end result is that management has to give more headcount dedicated to patching hosts which is next to impossible because its very hard to justify the cost.

 

Re:Wow (1)

bill_mcgonigle (4333) | more than 2 years ago | (#38165482)

Now throw in a team dedicated to information security and you get additional opinions on how to do patching. Its next to impossible to put 10 people in room and get a decision, and these conversations go on for years.

If that's the case, they're not a team dedicated to information security, they're dedicated to having easy jobs and like to call themselves 'information security professionals'.

Remember NASDAQ use gentoo... (0)

Anonymous Coward | more than 2 years ago | (#38164956)

....the updates will be complete when they finish compiling!

Intentional? (0)

Anonymous Coward | more than 2 years ago | (#38164990)

What I find interesting is the "and uninstalled security patches" does that mean that some employee did some sabotage? That could also explain the firewall part.

cap: unclosed

Re:Intentional? (1)

Joe_Dragon (2206452) | more than 2 years ago | (#38165258)

or that some software did not work with some of the security patches and the firewall needed some stuff opened up for that software to work.

Will work for Profits (1)

NetNinja (469346) | more than 2 years ago | (#38165442)

Dear WallStreet,

I will work for profits. Condo in Manhattan.

Thanks
  P.S. I am serious.

It's a culture issue (4, Insightful)

onyxruby (118189) | more than 2 years ago | (#38165560)

It's a culture issue on the concept of server up-time vs service up-time.

I developed the patch management process that is used on the servers of one of the largest trading companies in the world. I got started on this at the time after hearing one of the server admins brag about an up-time over five years. What he was really saying was that he hadn't patched his servers in over 5 years. Unless your running a mainframe or a certain flavors of Linux a reboot is required for many patches.

When one of those servers go down the cost is measured in the millions of dollars per minute. The culture took as a matter of pride to make sure that never happened. The best perceived way to avoid this was avoid anything that could affect server up-time. Since patching necessarily involved rebooting the server it simply wasn't done.

Changing this culture was a half year long internal political fight that boiled down to a single thing. I posited the argument that server up-time should no longer be tracked as a metric and should instead be replaced with service up-time.

During that half year period I developed the process (working with a lot of other teams) for patching these servers without affecting service up-time. Doing so involved creating a SLA that had server maintenance windows defined for specific times. It also explicitly defined that service availability would not be affected by having a server be unavailable during those very maintenance windows.

Ultimately the culture was so entrenched that it literally took upper management handing down orders from on high that server up-time was no longer allowed to be tracked as a metric. In the end we were patching our servers on a routine basis and doing so without impacting service availability.

Re:It's a culture issue (2)

elbles (516589) | more than 2 years ago | (#38166548)

Excellent point, and a practice I've already seen at my current job (tracking service availability instead of server uptime--in fact, since I started, we've tracked nothing but service availability).

That said, this has led us down the path of constantly increasing availability requirements, for things as (relatively) insignificant as an internal company blog. We're currently doing work between two new data centers, and one of the goals is to provide near 100% availability of all systems. It becomes very easy to sell such an idea to the business at little incremental cost (compared to the cost of building out two DCs in the first place), but the actual work involved in making it happen can be tricky at times. Not to mention the real incremental benefit is questionable at best, at least for a lot of the applications in question (IMHO, and given that many systems aren't tied to money-making endeavors).

Sure, it's theoretically possible to have two DCs, and when you want to do patching, you flip to your secondary site, patch your primary, flip back, and patch the secondary. It's a practice I'd certainly expect to see in an environment like NASDAQ. The business likes it, and the technical minutiae are workable (most of the time), but it is a substantial amount of added complexity (and work... and time) for little added benefit, in a lot of cases.

In short, I agree completely with what you said, but it can have the side effect of increasing the "required" availability numbers to the point where it becomes little different than simply looking at uptime (depending upon the environment).

Re:It's a culture issue (-1)

Anonymous Coward | more than 2 years ago | (#38166772)

When I'm patching my servers, I do it while naked. What is your opinion on server patching while wearing clothes?

Bernard L. Madoff (0)

Anonymous Coward | more than 2 years ago | (#38171392)

Any wonder that NASDAQ isn't trustworthy? Look at the guy that created it.

Must be really bad (1)

drinkypoo (153816) | more than 2 years ago | (#38174596)

Given that the FBI's security is shit [networkworld.com] , if they're shocked at how bad yours is, you know you're fucked.

Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?