Beta

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Heartbleed Disclosure Timeline Revealed

samzenpus posted about 4 months ago | from the when-did-you-know dept.

Security 62

bennyboy64 (1437419) writes "Ever since the Heartbleed flaw in OpenSSL was made public there have been various questions about who knew what and when. The Sydney Morning Herald has done some analysis of public mailing lists and talked to those involved with disclosing the bug to get the bottom of it. The newspaper finds that Google discovered Heartbleed on or before March 21 and notified OpenSSL on April 1. Other key dates include Finnish security testing firm Codenomicon discovering the flaw independently of Google at 23:30 PDT, April 3. SuSE, Debian, FreeBSD and AltLinux all got a heads up from Red Hat about the flaw in the early hours of April 7 — a few hours before it was made public. Ubuntu, Gentoo and Chromium attempted to get a heads up by responding to an email with few details about it but didn't, as the guy at Red Hat sending the disclosure messages out in India went to bed. By the time he woke up, Codenomicon had reported the bug to OpenSSL."

cancel ×

62 comments

Sorry! There are no comments related to the filter you selected.

http://www.linuxadvocates.com (-1)

Anonymous Coward | about 4 months ago | (#46751401)

Dear Linux Advocate,

Money doesn't grow on trees. And, Linux Advocates is growing. Naturally, we anticipate operating costs and hope to be able to meet them.

But, any amount you feel you are able to donate in support of our ongoing work will be most surely appreciated and put to very good use. Your contributions keep Linux Advocates growing.

Show your support by making a donation today.

Thank you.

Dieter T. Schmitz
Linux Advocates, Owner

http://www.linuxadvocates.com/... [linuxadvocates.com]

Re:http://www.linuxadvocates.com (-1, Flamebait)

Midnight_Falcon (2432802) | about 4 months ago | (#46751551)

Because spamming tech professionals as an AC has historically always created such goodwill for companies and urge for readers to donate to your cause.

Re:http://www.linuxadvocates.com (1)

Anonymous Coward | about 4 months ago | (#46751973)

It's almost as though the GP knows this and is deliberately setting out to harm the company. Could this be some kind of troll?

Re:http://www.linuxadvocates.com (2)

symbolset (646467) | about 4 months ago | (#46753013)

He knows we are going to talk about how Microsoftie Howard Schmidt is chairman of the board of codenomicon [techrights.org] .

End result: mass panic (0)

Anonymous Coward | about 4 months ago | (#46751453)

Almost nobody had a patch ready by the time the news was made public.

Re:End result: mass panic (-1)

Anonymous Coward | about 4 months ago | (#46751525)

That's what happens when you trust open sores software. The supposed million eyes constantly looking at and auditing the 100s of millions of lines of source code simply don't exist.

Re:End result: mass panic (1)

Anonymous Coward | about 4 months ago | (#46751573)

They exist; it's just that the vast majority of the people who belong to those eyes are really not qualified to be working on software that will be used in such important roles, and now we're paying the price. You don't use your Fisher Price tool set when you are building a real house. You just don't.

Re:End result: mass panic (2, Insightful)

Anonymous Coward | about 4 months ago | (#46751619)

And you also see this same type of thing in proprietary software, where tons of losers are hired to work on the code, with predictably terrible results. The thing about open source is that anyone can see the source code, and people not part of the group that wrote the code can check it, so you at least have some chance of understanding what's going on.

Anyone who claims that open source advocates claim that open source is 100% immune from all flaws is just spewing forth straw men.

Re:End result: mass panic (0)

Anonymous Coward | about 4 months ago | (#46751869)

Anyone who claims that open source advocates claim that open source is 100% immune from all flaws is just spewing forth straw men.

I suggest you spend more time reading open source forums and blogs because it is not a strawman.

Re:End result: mass panic (-1, Flamebait)

HairyNevus (992803) | about 4 months ago | (#46752213)

And since you clearly do, link? To someone who's a known open source contributor and can be quoted as saying (along the lines of ) "open source is 100% immune from flaws".

I mean, what are the odds that people anonymously post on the Internet pretending to be from one organization/ideology but intentionally use invalid logic and reason because they're actually an opponent of said organization/ideology? ...On the Internet!?

"Independent" discovery? (5, Interesting)

Red Herring (47817) | about 4 months ago | (#46751463)

> Google discovered Heartbleed on or before March 21 and notified OpenSSL on April 1. Other key dates include Finnish security testing firm Codenomicon discovering the flaw independently of Google at 23:30 PDT, April 2.

Doesn't it seem strange that the flaw has existed for a long, long time (years?) but Codenomicon happens to find it less than a day after Google notified OpenSSL, and, per the article, "some infrastructure providers under embargo"? That just seems... unlikely. Not impossible, but it kind of makes you wonder who is leaking information...

Re:"Independent" discovery? (5, Interesting)

Albanach (527650) | about 4 months ago | (#46751519)

Not necessarily. It may be that the bug was known to others and that Google and Codenomicon were both monitoring channels used by more nefarious types. Both organizations may have independently 'discovered' the bug after each becoming aware that an exploit existed without having full details of the exploit.

Re:"Independent" discovery? (2)

icebike (68054) | about 4 months ago | (#46752531)

Not necessarily. It may be that the bug was known to others and that Google and Codenomicon were both monitoring channels used by more nefarious types. Both organizations may have independently 'discovered' the bug after each becoming aware that an exploit existed without having full details of the exploit.

And the story should have been about WHEN those nefarious types first started mentioning it, not about when the white-hats actually found it.
Did those blackhats find it by reading the code, or accidentally stumbling upon it in some way?

I suspect it was the former, but I think that discussion is more important than when Google detected it. After all, the implication is that
google discovered nothing, but simply heard about it in the hallway or something.

Re:"Independent" discovery? (5, Interesting)

AdhSeidh (193409) | about 4 months ago | (#46751559)

perhaps you have already forgotten about CVE-2014-1266 the Apple SSL/TLS bug from Februrary this is why every security group on the planet was looking for other encryption related loopholes

Re:"Independent" discovery? (1)

OneAhead (1495535) | about 4 months ago | (#46751765)

Not to mention CVE-2014-0092.

Re:"Independent" discovery? (4, Funny)

JustOK (667959) | about 4 months ago | (#46751809)

Or NCC-1701-D

Re:"Independent" discovery? (1)

dave420 (699308) | about 4 months ago | (#46755771)

Not to mention LV-429. There was some talk of nuking that one, though.

Re:"Independent" discovery? (4, Interesting)

rmdingler (1955220) | about 4 months ago | (#46751607)

In all likelihood, there was a "discovery" by Google that led to a sharing of information with Codenomicon... someone told an old college buddy or former co-worker.

There were almost certainly folks who were aware of the vulnerability before Google.

Were these folks criminals or government employees? And yes, there's a small difference... generally found in the probability for prosecution.

Re:"Independent" discovery? (0)

Anonymous Coward | about 4 months ago | (#46755109)

It seems to me that this was a case of "insider trading" in the vulnerability market. Someone discovers Heartbleed and then secretly shares the information with Codenomicon so that they can capitalize on the publicity. It explains the delays and also the "independent discovery."

Re:"Independent" discovery? (1)

BVis (267028) | about 4 months ago | (#46755509)

Were these folks criminals or government employees? And yes, there's a small difference... generally found in the probability for prosecution.

There's always one, isn't there.

Re:"Independent" discovery? (0)

Anonymous Coward | about 4 months ago | (#46755819)

I happen to know that this is the most probable case. I know it doesn't mean much coming from an AC who can't provide evidence, but effectively it appeared to be a case of someone CC'ing an email to various parties, among them a college acquaintance, describing an OpenSSL exploit, which resulted in this friend investigating and discovering it shortly after. The original guy was frustrated he didn't get credit because he spent more than a working week on it (he had some kind of slideshow or something planned).

Re:"Independent" discovery? (4, Funny)

briancox2 (2417470) | about 4 months ago | (#46751685)

That's what Newton said to Leibniz.

Re:"Independent" discovery? (1)

Anonymous Coward | about 4 months ago | (#46751731)

No, this is not uncommon at all in research. The idea that two groups are both looking into how [X] works, and how [Y] responds to [X] is quite common. Being a security researcher myself (slightly different sub-field, but still reason for anon posting) I can say that it is quite an easy possibility that both teams were checking the ENTIRE ISO~TCP/IP stack from lvl 0 up to lvl (whatever 'top' is in your outlook/naming scheme) And that they both found it around the same time.

Until I see _any_ evidence to point to something else, I'm going to just accept this as fact for now. (I'm not saying other things "didn't" happen, but that we would need proof first) b/c as far as I can tell, the entire 'bug' seems fairly easy to accidentally include. So I'll err on the side of incompetence instead of malice for now. lol

Re:"Independent" discovery? (2)

ameen.ross (2498000) | about 4 months ago | (#46754369)

Thank you. I've been saying this from the beginning and am very annoyed that every time people write about Heartbleed, it links to Codenomicon's site. Even if it was an independent discovery (which it wasn't) then it's still too much credit. People should just link to the official CVE...

Negligence (3, Interesting)

Daniel Ellard (799842) | about 4 months ago | (#46751507)

Why did Google wait ten days before notifying OpenSSL? (even if they didn't trust OpenSSL to handle it responsibly, it couldn't have taken ten days for Google to patch their systems)

Re:Negligence (3, Insightful)

batrick (1274632) | about 4 months ago | (#46751569)

Negligence? They don't owe you a fucking thing.

Also, the flaw has also existed for over two years. What does one more week hurt?

Re:Negligence (-1)

Anonymous Coward | about 4 months ago | (#46751769)

Negligence? They don't owe you a fucking thing.

Simmer down, Ayn Rand.

One can be neglectful (i.e. negligent) without having the legal obligation to do otherwise.

I think that Google just might agree that it is at least in their best interest to have a significant vulnerability in OpenSSL be fixed. Even if it were neutral to their specific interests, it's still of interest to the internet-using community as a whole, which I would bet they are smart enough to know. Though not always a legal or ethical one, it is a moral responsibility to be a worthwhile participant within a society that you expect to be able to participate in.

Based on the information available, though, I'd have to say that Google is strongly leaning towards not having been negligent, though. At least they actually reported it. But could they have had malicious or fully selfish intentions for not reporting it? Yeah, they could have. But it seems implausible, and there's certainly not enough information here to get to that judgment.

Re:Negligence (1)

swillden (191260) | about 4 months ago | (#46753131)

I think that Google just might agree that it is at least in their best interest to have a significant vulnerability in OpenSSL be fixed.

Of course, but it's even more in their interest to make sure their own systems are fixed before they take any action which could result in the information spreading to potential attackers. Of course, attackers may already have had it, but if so that's water under the bridge. If not, the moment you disclose it to anyone you've increased dramatically the odds that someone who might want to exploit you will hear of it, even if you're just telling the dev team. For that matter, even disclosing it internally is a risk, so you'd want to keep that as tight as possible as well. I work in security at Google and didn't hear about Heartbleed until it was publicly disclosed.

Given the severity of the damage that could be caused by Heartbleed once information about it is widely disclosed, particularly given how trivial the exploit code is to write, Google would (IMNSHO) have been foolish to disclose it to the OpenSSL team before getting all of its key systems patched.

(Disclaimer: I work for Google but don't speak for Google. The above represents only my personal opinions.)

Re:Negligence (5, Insightful)

Anonymous Coward | about 4 months ago | (#46751585)

Simple, to fully test and develop the patch (see: https://bugzilla.redhat.com/at... [redhat.com] ). It's much better if someone who knows of both a problem and has the ability to fix it to sit on the announcement to keep from wider exposure. This helps keep the common knowledge exploitation period to a minimum.

Re:Negligence (1)

rahvin112 (446269) | about 4 months ago | (#46752331)

The problem is that we don't know how the discovery was made.

The NSA has apparently known about heartbleed since the start. And I would be surprised if Google and other major corps aren't monitoring criminal forums where these exploits are sold. Which makes me wonder if Google discovered it though monitoring the criminal channels or it's own audits.

Re:Negligence (1)

swillden (191260) | about 4 months ago | (#46753167)

And I would be surprised if Google and other major corps aren't monitoring criminal forums where these exploits are sold.

I think you would be surprised. I also think that the process one would have to go through to get vetted and get access to those forums probably requires actions that a major corp wouldn't take. FWIW, I work in security at Google and have never heard of any sort of monitoring of criminal forums.

Re:Negligence (1)

GryMor (88799) | about 4 months ago | (#46753225)

If it happens at all, it's using independent contractors.

Re:Negligence (1)

swillden (191260) | about 4 months ago | (#46755261)

If it happens at all, it's using independent contractors.

Which would amount to hiring criminals. Seems unlikely to me.

Re:Negligence (1)

Goose In Orbit (199293) | about 4 months ago | (#46755181)

The NSA has apparently known about heartbleed since the start

Source?

Ten days? Luxury! (0)

Anonymous Coward | about 4 months ago | (#46751643)

We sometimes get reports from researchers that are three or six months old.

Re:Negligence (5, Insightful)

freeze128 (544774) | about 4 months ago | (#46751675)

Also, April 1st is the *WORST* day to notify ANYONE that there is a severe security flaw..

Re:Negligence (1)

dkf (304284) | about 4 months ago | (#46754483)

Also, April 1st is the *WORST* day to notify ANYONE that there is a severe security flaw..

Major public holidays (e.g., Christmas) are much worse, as there's a really good chance nobody will even look at the warning, and may decide that their family time trumps fixing security problems.

April 1 is just the worst day to announce a major breakthrough or groundbreaking new product.

Re:Negligence (2, Insightful)

Anonymous Coward | about 4 months ago | (#46752075)

10 days to figure out a patch that was: 1) secure 2) stable 3) well tested??? 4) passed legal?

I mean... 10 days isn't a 'long' time for a big company like this to 'find' and then 'report' a big, especially of THIS magnitude

Re:Negligence (1)

slimjim8094 (941042) | about 4 months ago | (#46752247)

You don't think it could take 10 days to find a flaw, fix it, make sure you've fixed it, and roll the fixes out to prod? And then "notif[y] some infrastructure providers under embargo" and let them fix it and roll it out to prod?

You may disagree with Google looking out for themselves first here, but the fact is they'd be negligent (and foolish) to spread this more widely until they'd ensured it was fixed for themselves and (by extension) their customers/users.

Re:Negligence (2)

pedantic bore (740196) | about 4 months ago | (#46752515)

Yeah, if that's what happened. But that's not what the article says.

It says that on March 21st, Google had already fixed the flaw and rolled out the patches internally. Fine; they get to cover their own asses first. No argument.

Then a week went by.

Re:Negligence (1)

slimjim8094 (941042) | about 4 months ago | (#46752969)

You must be reading a different article than I am. I see "The patch is then progressively applied to Google services/servers across the globe." which implies to me that the 21st was the start of the clock. I could easily imagine that it would take several days to update everything.

Then the clock starts ticking for whoever the "infrastructure providers under embargo" are. I emphasized "then" in my original post - presumably they wouldn't share the flaw even with trusted partners until they'd fixed it themselves. Two sequential "several days" could hardly be shorter than 10 days.

Re:Negligence (1)

Anonymous Coward | about 4 months ago | (#46754433)

Why did Google wait ten days before notifying OpenSSL? (even if they didn't trust OpenSSL to handle it responsibly, it couldn't have taken ten days for Google to patch their systems)
Are you serious? They can see that there is a problem, but a patch or fix is not necessarily readily available. It would take a small team --very well versed in cryptography and networking-- several days to wade through all of the code. OpenSSL might sound like a nice little library, but is over 370,000 lines of source code, of which 70% is in C, and not nice straightforward 'hello world' type C, but ugly horrid, crufty 'winner of the obfuscated code contest' C. Wading through all that crap and coming up with a clean fix would have taken even the mighty Google a bit of time. Good on them for fixing it.

Re:Negligence (0)

Anonymous Coward | about 4 months ago | (#46754537)

To bait.

They fixed it but made the fix work like it wasn't patched and send out fake sensitive info and trigger a secret alert. After a week they were confident no one was exploiting it (so no new certs and new passwords for everyone on earth) and then started telling others.

Damn sleep... (3, Interesting)

Anonymous Coward | about 4 months ago | (#46751545)

Ubuntu, Gentoo and Chromium attempted to get a heads up by responding to an email with few details about it but didn't, as the guy at Red Hat sending the disclosure messages out in India went to bed.

I don't know why, but this reminded me of Cyril Evans [wikipedia.org] . Never go to bed.

Local LUG eats crow??? (0)

Anonymous Coward | about 4 months ago | (#46751667)

The folks in my local Linux User Group like to crow that Linux is more secure than Windows. I try to take such commentary with a grain of salt, but some of them can be a bit hostile toward folks who use multiple technologies, some of which are not open source (Windows, OS X, Oracle, iPhone, etc). Given the revelations of the Heartbleet bug and the coincidental revelation that Chrome is spying on your PC's microphone, do you think they will have to eat all that crow???

Re:Local LUG eats crow??? (1)

Malizar (553281) | about 4 months ago | (#46752421)

OpenSSL is in use on Linux as well as on Windows. Chrome is in use on Windows, the Linux version is Chromium, which may or may not have the same issue.

Re:Local LUG eats crow??? (1)

TheCycoONE (913189) | about 4 months ago | (#46758105)

Assuming he's referring to the speech to text exploit, the proof of concept works in Chromium as well. (http://guya.net/security/speech/)

I haven't tested the earlier mic keeps listening after enabled bug.

Re:Local LUG eats crow??? (0)

Anonymous Coward | about 4 months ago | (#46753589)

The 90s called. They want their LUGs back.

But when/if has it been exploited? (2)

queazocotal (915608) | about 4 months ago | (#46751733)

There are out there honeypot machines, which log all inbound and outbound packets.
They can run retrospective analysis of these packets to work out if undetected exploit probes have occurred.

Is anyone aware of this being done for heartbleed?

It would be interesting if - for example - it went from no exploits to most honeypots probed 3 months ago.

Re:But when/if has it been exploited? (2)

rainer_d (115765) | about 4 months ago | (#46752027)

There are various reports that efforts to exploit this vulnerability go back almost as far as the introduction of the bug to various distributions.

I wonder if someone discovered the bug and sold it to the "vulnerability assessment" industry (which in turn supplies spooks and other government agencies with their exploits so they can perform "lawful interception").
Such a bug would probably sell for a million these days. Or even more.

Re:But when/if has it been exploited? (1)

rmdingler (1955220) | about 4 months ago | (#46752361)

I don't know what the information about such a vulnerability would sell for.

Personally, I would recommend turning it into a multi-year deal as opposed a single large bonus check, but I'm old, boring, and practical.

We just don't know if it was discovered by a TLA or sold to a TLA, but because they would bid the highest at any auction you can conceive, they undoubtedly had it way, way before Google.

Back up contact in another time zone? (0)

ndogg (158021) | about 4 months ago | (#46751811)

I don't understand why Sidhpurwala didn't have a back up contact in another time zone that could have been contacted when he was asleep.

Re:Back up contact in another time zone? (0)

Anonymous Coward | about 4 months ago | (#46753519)

The original email offering details had the bug embargoed for a few days. There was no rush to reply... until there was...

Re:Back up contact in another time zone? (0)

Anonymous Coward | about 4 months ago | (#46755367)

I can.

I once spent a 12-hour day working on a problem, then went to bed. At 3am my phone went off - the guy wanted an update...

Re:Back up contact in another time zone? (0)

Xest (935314) | about 4 months ago | (#46756499)

He used to but they outsourced his job to India.

What about the small guys? (0)

Anonymous Coward | about 4 months ago | (#46753565)

Jeeze, what about the small guy that doesn't have access to the information? How are they expected to react?

April Fools! (1)

Max Threshold (540114) | about 4 months ago | (#46753569)

There's the trouble. Google's disclosure came on a day when nobody believes what they read on the Internet.

a fucked up rls at 11.59 P.M. on New Years Eve.. (0)

Anonymous Coward | about 4 months ago | (#46753873)

..spotted on April Fools Day, 2 years later..

Now, either *someones* at it here, or I hear an Alanis Morissette song playing. And I know which I believe.

Re:a fucked up rls at 11.59 P.M. on New Years Eve. (0)

Anonymous Coward | about 4 months ago | (#46762551)

Yeah, she's got one hand in your penguin!!!!!!!!!!!!!!!!!!

Is OpenSSL by [prev'ly DoD-funded] OpenBSD folks? (0)

Anonymous Coward | about 4 months ago | (#46754295)

Didn't OpenSSL come from the folks at OpenBSD, who
- some years back - brought backdoor[s] in OpenBSD,
ie, after receiving "funding" from folks like USA's DoD?

If so, who'd be surprised by such a discovery as HeartBleed?

Re:Is OpenSSL by [prev'ly DoD-funded] OpenBSD folk (2)

kevin lyda (4803) | about 4 months ago | (#46754555)

OpenSSL did not come from OpenBSD. So right from the start your theory is broken.

Class action against the NSA (0)

Anonymous Coward | about 4 months ago | (#46756015)

If the NSA did know about the heartbleed vulnerability and didn't disclose it, that makes them responsible for any and all financial losses, identity thefts, corporate data losses that occurred between when they found out about it and someone else released the fact that there was a vulnerability.

Cost to the NSA, billions if not trillions after you tack on penalties for willful negligence as they are tasked with improving the national computer security, a direct violation of their charter.

Let's get the ball rolling, seize all N.S.A. assets as well as their staff / management / superiors as they are all guilty.

Of course, we would then have to subpoena exactly when they knew about (developed) the vulnerability.

Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?
or Connect with...

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>