I created a database showing websites' vulnerability status - it's pretty convenient for multiple filtering:
http://www.ragic.com/heartb...

I think it's also worth considering what would have happened here if this were not an open source but instead a private product. First, there is surely an equal and possibly even greater possibly that this would have also been missed. Being part of a software test team at a large tech company (think twin bridges) for well over 8 years, trust me when i say that private company SW testing is geared almost solely around existing and new features. Regression and negative testing (what this would fall into) is rarely, if ever, performed. Why you ask??? Time and money. Features sell software and therefore, features must work. Negative testing, while invaluable, always takes a backset. Next, even if it were found, you can be guaranteed that it would not be made public unless leaked. And finally third - my very favorite - there is a belief in the encryption community that any software that serves to handle encryption should be open for the world to see because that is the only true way you will know if it is secure. If anyone can look at the source and it still holds up, then you have the best shot at it being the most secure. The natural counter to that is this very situation to which I would say, there are always corner cases. It is actually because this was open source why it was found at all - I believe to my core that if this had not been open source, that this would still be under the covers for many more years to come.

Ah yes, Kerckhoffs's principle:

"A cryptosystem should be secure even if everything about the system, except the key, is public knowledge."

Although I believe the intent of that is more about not falling into "security through obscurity" and ensuring the algorithm implementation can stand up on its own.

We can only speculate, but of course just because the source is closed doesn't mean that risky implementations won't be discovered by those with the resources to closely inspect it, but it does mean that it's harder for the masses to to find risks. I'm on the fence in terms of which is ultimately more or less secure, I can see both arguments.

I've rewritten this many times but I'm too tired to make it coherent. Meh.

Neither. Essentially: it is about managing risk. Security is not measured in absolutes.

I agree, very good that it was an Open Source case. I hope the SSL certificate I purchased from http://usa.weloveourhost.co... is really secure. It's not open source.

In a situation like this, it may simply boil down to a software flaw and from that alone we can't say "This is or is not secure because it's open source".

I’m not sure this is an accurate characterization of the bug and bugfix (I think you did an accurate summary of an inaccurate source). Summarizing the above explanation: the payload is only supposed to be 16 bytes -> this wasn’t checked -> over-allocation/exploit.

But a heartbeat’s payload and/or padding *can* be larger than 16 bytes (that wasn’t the problem). I’d sum up the bug as, “The stated size of payload was never verified with the actual size of the payload, and the code used it to determine how many bytes to read, reading past the end of the structure”. The code allocated a chunk of memory based on what the attacker says (which isn’t itself a problem), but then copies from a chunk of memory that might not be as big as the attacker implied.

Total Record Size (what came across the wire, how big it actually is) = “rrec.length”

Payload Size (size of a piece of the record, as stated by attacked) = “payload”

Before Fix:

buffer = OPENSSL_malloc(1 + 2 + payload + padding); // Could be bigger than what was actually sent

dtls1_write_bytes(s, TLS1_RT_HEARTBEAT, buffer, 3 + payload + padding); // Read too much from s and write it into our supersized buffer (to be sent back to attacker)

The fix added some checks to ensure 1) The record isn’t smaller than the the smallest possible size and 2) The stated payload size doesn’t make us exceed the record size:

if (1 + 2 + 16 > s->s3->rrec.length)

return 0; /* silently discard */

...

if (1 + 2 + payload + 16 > s->s3->rrec.length)

return 0; /* silently discard per RFC 6520 sec. 4 */

PS: SSL3_RT_MAX_PLAIN_LENGTH is defined as 16,384 (not 16 like the linked explanation implies).

This is clearly poorly written code on several counts!! How is it that someone that writes code like this has carte blanche access to a globally critical resource like OpenSSL in the first place??

Will anyone be held accountable, or is Open SSL a responsibility free zone?

It's still very early days, but I highly doubt it. It would require negligence to be demonstrated and it's especially hard when there's not a commercial entity to hold accountable. "Responsibility" in open source generally tends to be determined by people voting with their feet - if they don't like it, they don't use it, question in this case is whether the alternatives are any better.

Actually, line memcpy(bp, pl, payload) in the code is the point where access to private memory takes place; pl is a pointer to the original hearbeat request in memory; when hacking ocurrs memcpy copies the original string to be echoed and much more.

But yes, the fix is

if (1 + 2 + payload + 16 > s->s3->rrec.length)

return 0; /* silently discard per RFC 6520 sec. 4 */

and yes, either heartbeat request or heartbeat response should be larger than 1+2+16 .

"The code allocated a chunk of memory based on what the attacker says
(which isn’t itself a problem), but then copies from a chunk of memory
that might not be as big as the attacker implied." - shouldn't we stop using programming languages that allow such stuff to happen or fix them?

Good point. It might be possible to have a programming language that doesn't allow something like that - but it is based on other programming languages and some of those will always allow it since that's how computing works. But I see what you're saying. If this was created in a high level language that runs in a VM for example, then it could prevent odd memory access like this and might have prevented the bug... as long as the underlying techs are sound as well which is a lot better chance.

no need to have a runtime (VM) that protects us dynamically, if the compiler and language are clever enough to protect us statically (at compilation time). Of course if we get any binary code (native or VM machine bytecode) that may have been tampered and want to run it, then we do need dynamic protection for memory accesses, but most CPUs do provide low level facilities for such things that the OS can use

I like the VM way better because someone could build their own compiler, or modify one, to do whatever they wanted. I don't know what compiler was used in the software I downloaded, but I do know I went to java.com to get my Java VM.

Awesome feedback, thanks for the analysis! Amazing how days into this there's still so much speculation on what can or can't be done with the bug.

One thing that hasn't been stressed enough yet, is that clents are equally vulnerable. The articles out there give people the risk landscape from the server side's perspective. Troy's article is one of the very few that has a slight reference to clients being vulnerable too.
In the same way you can have a malicious client causing memory leaks to a vulnerable server, one can set up a malicious server that steals bits of the client's memory. This could include a bunch of very interesting information related to the victim.
You don't have to own some sort of server infrastructure that might be vulnerable in order to check if you need an update - everyone that has a dependency to openssl must do it!

That's something that's worrying me and I was intentionally vague about the possible risk because frankly, I'm not yet sure of the extent of it. On the one hand, clients aren't quite the central hub of data that servers are yet on the other hand, if you could orchestrate an attack against multiple connected devices then that could be pretty serious business.

Then there's patching: for clients, that's only going to happen via updates pushed from providers, mum and dad aren't going to be checking for vulnerable OpenSSL versions on all their devices! Add to that the fact that the device landscape is arguably more diverse than the server landscapes - obviously there are PCs and phones and tablets, but now now we've got "things" too. TVs, scales and even a freakin' fork are now internet connected (no really, Google "HAPIfork"), what's their risk surface?

This could well be the vuln that just keeps on giving, I have a feeling we're only just scratching the surface at the moment.

A bunch of clients are vulnerable, check out https://security.stackexcha... for more information including a Metasploit module from HDMoore

@AlexP Sorry, I hadn't read your post raising the potential client-side vulnerability before I posted. I thought of it myself as soon as I read the description on the openssl site. Steer clear of sites that your grandmother would question... well forever I guess or at least until you know that your end of the SSL communication pipe is invulnerable to a server

Something I *really* don't understand is why they are using malloc to allocate the buffer instead of calloc (calloc overwrites the buffer with zeros before returning it). In security-sensitive applications, you should often clear allocated memory before handing it to the client for exactly this reason: you don't know what leftover garbage might be in that memory space. In the same way, it often makes sense to mangle sensitive data structures (like passwords) before freeing them for reuse so they *can't* end up in a memory allocation somewhere else. Otherwise it's like handing your confidential business documents to your kid to color on instead of putting them through the shredder: who knows where it might end up?

I don't see how the SSL secret key could be compromised with this exploit unless a copy of the secret key was freed and put on the heap sometime before the heartbeat exploit was run. If the memory containing the key had been zeroed before being freed, there would be no issue. It's not hard to automate data mangling in a security-sensitive application.

Short answer: calloc in this case would not have helped, nor would zeroing memory before free.

Long answer: The bug isn't that the destination buffer is unsanitised, it is that data beyond the end of the source buffer is copied into the destination buffer. This data could well be valid, unfreed and actively used memory (which obviously can't be zeroed). calloc does solve some problems, just not this one. For more, see the explanation by colithium.

Regarding Troy's reply: The issue of key accessibility is orthogonal to cleaning memory on free and allocation. I've yet to see a quote that says that the keys are inaccessible, only that they are unlikely to be in memory locations that the attacker can read through this attack. The actual details are highly dependant on the memory allocator implementation used by the C library on the machine. However, unlikely is not impossible, hence you get some people reporting that it worked.

Me: "Of course, if an attacker did try that, there should be some increase in SEGFAULTs which might show up in logs. At least *some* of the bad copies are going to hit a page boundary and dump core."

I take that back actually. There might be some *small* increase in SEGFAULTs, but nowhere near as many as I had assumed. Apparently OpenSSL sidesteps most of the now common allocation strategies designed to force mismatches like this to fault out. From http://thread.gmane.org/gma... :

But around that time OpenSSL adds a wrapper around malloc & free so that the library will cache memory on it's own, and not free it to the protective malloc.

...

So then a bug shows up which leaks the content of memory mishandled by that layer. If the memoory[sic] had been properly returned via free, it would likely have been handed to munmap, and triggered a daemon crash
instead of leaking your keys.

So there might be no noticeable consequences to even a sustained attack. Combine that with EFF's evidence that this bug HAS been systematically exploited in the wild (from a botnet, no less, https://www.eff.org/deeplin..., and we are in *very deep doo*.

"The bug isn't that the destination buffer is unsanitised, it is that data beyond the end of the source buffer is copied into the destination buffer. "

That makes sense, and... ouch. So it is not the OPENSSL_malloc() call per say which is the issue but the strncpy()/memcpy()/bcopy() or whatever which immediately follows and uses the same tainted value in its bounds checking.

" I've yet to see a quote that says that the keys are inaccessible, only that they are unlikely to be in memory locations that the attacker can read through this attack."

Right, and, of course, you can game 'unlikely' by just repeating the hack a lot with different values and from different simultaneous connections. With zombies potentially available to do the dirty work, connections and computing time are cheap for the attacker.

Of course, if an attacker did try that, there should be some increase in SEGFAULTs which might show up in logs. At least *some* of the bad copies are going to hit a page boundary and dump core.

Yep, you got it.

A minor comment though. The typical page sizes on x86 is 4k, so a 64k read will span 16 or 17 pages. That is usually ok, as pages are in contiguous virtual addresses when the memory allocator uses sbrk() to grow the heap. So unless the buffer that the SSL request is in happens to end up in the last 64k of the heap, it probably wont segfault. Memory can often be allocated using mmap(), but that tends only to be done for large buffers (which the SSL struct in this case isn't).

@shopt "only that they are unlikely to be in memory locations that the attacker can read through this attack" if the same session requests another heartbeat or the same attacker comes in from another machine would the overrun get returned from another location in the server's memory? So long as a hypothetical traffic monitor never got suspicious you could probably mine a lot of a server's memory with a good chance that information of interest would be returned for analysis within a few hours. I would guess that the more important the information, the more likely that it would remain in memory. But... you raise a valid point about accessibility, if the 64K before the information of interest is always locked by the O/S then there is no chance that the memory would be allocated to record the heartbeat payload within that 64K and thus no chance that the read overrun would get to it. So strictly speaking, it's only data in un-cleared memory or the first 64Kb of a locked section of memory that is vulnerable?

Not quite. Accessible was probably a poor word for me to use.

You are correct that an attacker can keep trying, and get a different 64k slice each time. While it is different, it is not uniformly random though. The key is where the buffer that the attacker's SSL packet is allocated in is in relation to the interesting data. The private key buffer is likely to be allocated early and not be freed. So my understanding is that it will be hard for an attacker to get given a buffer in the 64k preceding the private key, as it's likely to be in use.

O/S locked memory is a red herring in this case. That just means that it wont be swapped to disk by the O/S, which is not relevant to this attack.

I don't profess to understand the intricacies of memory management at this level, but I do know that a bunch of very smart people have said the keys are accessible. Then other smart people have said they're not. Then even more smart people have posted screen grabs of keys pulled from memory. Who knows, it's open to speculation at this point, my view is to hope for the best and plan for the worst.

Thank you to Mr. Hunt for being one of the sanest people out there discussing this. The end of World rhetoric is so frickin' boring...I would estimate seriously that less than 5% of all Internet users would even begin to know how to leverage this exploit. And even then, many sites use layers of security to prevent just this type of situation. Anyone who understands security knows full well that success lies in a layered mentality - many points of distributed failure - not a single or many overlapping. Additionally, many are freaked about how to fix this and get back to normal business - this is the exact type of event that X.509 revocation was invented for - the widespread breach of critical key-based information. You revoke the certs per the design and new certs are issued. If root CAs were breached, then new root certs are also issued, integrated into browsers and then folks upgrade. People chg passwords - BOOM - back to normal.

Calm down folks - and BRAVO Mr. Hunt...

I'm one of the first to say that these security stories are usually beatups. And I agree that Troy's article is reasonable and balanced. Your comment on the other hand paints a very rosy world where all we have to do is *only* reissue thousands (probably more) of certificates, rely on a very flaky certificate revocation system, and get millions (billions?) of account passwords changed. All this is in addition to patching the vulnerability. This is the biggest vulnerability to have happened in terms of its attack surface and severity that I can think of. I'm open to suggestions of a bigger one.

You may estimate that mere millions of internet users can exploit this. That's hardly a small deal. Once the cyber criminals grab my credentials, I'm hardly comforted that John Citizen can't do it.

Which "layers" of security prevent this being exploited? Are you seriously saying that a password or session cookie compromise is no biggie, because of defence in depth? Some sites may use multi channel authentication where one of the channel is single use, but many don't (and many users don't turn it on if they do).

Maybe a few people need to calm down, but treating this like some mere annoyance is not accurate.

Mr. Cyberzeus thank you for pointing the facts as I too Thank Mr. Hunt for this open discussion.

Thank you for this useful, as always, article! I also found some good information on Vulnerability Notes Database website: http://www.kb.cert.org/vuls...

So far, resounding silence from banks (at least the handful that I deal with) regarding their exposure and mitigation...

I used three test sites against my bank which publicly claims to be buttoned down solid. One raised a warning on the SSL Cert date being within the period of vulnerability but could not assure me about the SSL version. The other two said that absolutely no problem (maybe they checked the SSL version that the first had not?) even though the SSL Cert date that the test site returned to me from the bank's server was clearly within the problematic date range.

Seconded, I've not heard anything from banks whilst I've received numerous messages from various other providers. I suspect they're still grappling with just how far-reaching this risk is and how to tackle it in what is inevitably a complex and very interdependent environment.

Over the next few weeks I think the biggest risk is phishing attacks. “We’ve fixed our systems. Click here to change your password.” And of course it goes to a scam site that collects your PayPal password.

That's always the risk with anything newsworthy, we've even seen events such as MH370 quickly jumped on by malicious parties looking to exploit the curiosity of the general public.

The difference in this case is that users are going to get a lot of legitimate requests to change their password. Not just the usual malware links – it’s easy for scammers to put up a change password form that collects your old and “new” password.

That's very true Nate, there's an expectation of receiving reset emails and a well known associated incident, certainly this would all add more legitimacy to phishing attempts.

This is the best article I've read so far in terms of coverage and adequate level of technicality (easy to understand). I really appreciate your efforts in clearly addressing developer, SysAdmins, end-users ...etc.

Thank you!

Big fan of your posts!
On an unrelated note...

I thought this might interest you, people are getting mass emails from twitch.tv telling them their usernames. Basically I think someone is scraping the twitch.tv email database because their "lost username" function tells the person whether or not they have that email in their database.

http://www.reddit.com/r/Twi...

http://help.twitch.tv/custo...

From what I can tell they're denying that anything is wrong.

Sorry if I'm wasting your time if you're not interested, I know you are a busy guy!

Keep the posts coming,

Regards.

Interesting, sounds like a lack of brute force protection on the recovery page. Often these sort of things aren't locked down until an incident like this occurs, let's see if anything changes now.

wow, what a nice and complex summary of the HeartBleed madness! Great work! I believe what CTO of Sticky Password said and I will change the passwords once everything is back in normal: http://blogen.stickypasswor...

Are you seriously suggesting that people check their sites for heartbleed by submitting urls to a web service that they don't control?

Absolutely, it's a URL, if you're not comfortable with it being publicly known then best just shut the site down until you feel comfortable with it being scrutinised.

Uhh, it's kind of already public. Like Troy says, if people knowing about your site is a bad thing, best shut the server down...

Dude - do you have any idea how much of your internet security is already being managed by sites you do not control? Just stay on SSL for a moment - how does it work? Well, you have an X.509 that is trusted but why? Because you told your browser to trust it or because it is signed and authenticated by a root cert which is managed by who - a company you do not control.

The Internet is a shared responsibility\trust model. If it weren't , it simply wouldn't exist...

Use the one on SSL Labs, click dont show result https://www.ssllabs.com/ssl... pretty sure http://www.qualys.com/ is legit ffs

This exploit is so random that it can't really be controlled to advantage. What would an attacker do to refine it and extract some metal from this raw ore?