Heartbleed Bug

Heartbleed is a security bug in some outdated versions of the OpenSSL cryptography library, which is a widely used implementation of the Transport Layer Security (TLS) protocol. It was introduced into the software in 2012 and publicly disclosed in April 2014. Heartbleed could be exploited regardless of whether the vulnerable OpenSSL instance is running as a TLS server or client. It resulted from improper input validation (due to a missing bounds check) in the implementation of the TLS heartbeat extension. Thus, the bug's name derived from ''heartbeat''. The vulnerability was classified as a buffer over-read, a situation where more data can be read than should be allowed.

Heartbleed was registered in the Common Vulnerabilities and Exposures database as . The federal Canadian Cyber Incident Response Centre issued a security bulletin advising system administrators about the bug. A fixed version of OpenSSL was released on 7 April 2014, on the same day Heartbleed was publicly disclosed.

TLS implementations other than OpenSSL, such as GnuTLS, Mozilla's Network Security Services, and the Windows platform implementation of TLS, were not affected because the defect existed in the OpenSSL's implementation of TLS rather than in the protocol itself.

System administrators were frequently slow to patch their systems. , 1.5% of the 800,000 most popular TLS-enabled websites were still vulnerable to the bug, and by , 309,197 public web servers remained vulnerable. According to a report from Shodan, nearly 180,000 internet-connected devices were still vulnerable to the bug, but by , the number had dropped to 144,000 according to a search performed on shodan.io for the vulnerability. Around two years later, , Shodan reported that 91,063 devices were vulnerable. The U.S. had the most vulnerable devices, with 21,258 (23%), and the 10 countries with the most vulnerable devices had a total of 56,537 vulnerable devices (62%). The remaining countries totaled 34,526 devices (38%). The report also broke the devices down by 10 other categories such as organization (the top 3 were wireless companies), product (Apache httpd, Nginx), and service (HTTPS, 81%).

History

The Heartbeat Extension for the Transport Layer Security (TLS) and Datagram Transport Layer Security (DTLS) protocols was proposed as a standard in February 2012 by . It provides a way to test and keep alive secure communication links without the need to renegotiate the connection each time. In 2011, one of the RFC's authors, Robin Seggelmann, then a Ph.D. student at the Fachhochschule Münster, implemented the Heartbeat Extension for OpenSSL. Following Seggelmann's request to put the result of his work into OpenSSL, his change was reviewed by Stephen N. Henson, one of OpenSSL's four core developers. Henson failed to notice a bug in Seggelmann's implementation, and introduced the flawed code into OpenSSL's source code repository on 31 December 2011. The defect spread with the release of OpenSSL version 1.0.1 on 14 March 2012. Heartbeat support was enabled by default, causing affected versions to be vulnerable.

Discovery

According to Mark J. Cox of OpenSSL, Neel Mehta of Google's security team privately reported Heartbleed to the OpenSSL team on 1 April 2014 11:09 UTC.

The bug was named by an engineer at Synopsys Software Integrity Group, a Finnish cyber security company that also created the bleeding heart logo, designed by a Finnish graphic designer Leena Kurjenniska, and launched an informational website, heartbleed.com. While Google's security team reported Heartbleed to OpenSSL first, both Google and Codenomicon discovered it independently at approximately the same time. Codenomicon reports 3 April 2014 as their date of discovery and their date of notification of for vulnerability coordination.

At the time of disclosure, some 17% (around half a million) of the Internet's secure web servers certified by trusted authorities were believed to be vulnerable to the attack, allowing theft of the servers' private keys and users' session cookies and passwords. The Electronic Frontier Foundation, Ars Technica, and Bruce Schneier all deemed the Heartbleed bug "catastrophic". ''Forbes'' cybersecurity columnist Joseph Steinberg wrote:

An unidentified UK Cabinet Office spokesman recommended that: On the day of disclosure, The Tor Project advised:

''The Sydney Morning Herald'' published a timeline of the discovery on 15 April 2014, showing that some organizations had been able to patch the bug before its public disclosure. In some cases, it is not clear how they found out.

Bugfix and deployment

Bodo Möller and Adam Langley of Google prepared the fix for Heartbleed. The resulting patch was added to Red Hat's issue tracker on 21 March 2014. Stephen N. Henson applied the fix to OpenSSL's version control system on 7 April. The first fixed version, 1.0.1g, was released on the same day. , 309,197 public web servers remained vulnerable. , according to a report from Shodan, nearly 180,000 internet-connected devices were still vulnerable. The number had dropped to 144,000 , according to a search on shodan.io for "vuln:cve-2014-0160".

Certificate renewal and revocation

According to Netcraft, about 30,000 of the 500,000+ X.509 certificates which could have been compromised due to Heartbleed had been reissued by 11 April 2014, although fewer had been revoked.

By 9 May 2014, only 43% of affected web sites had reissued their security certificates. In addition, 7% of the reissued security certificates used the potentially compromised keys. Netcraft stated:
''eWeek'' said, " eartbleed islikely to remain a risk for months, if not years, to come."

Cloudflare revoked all TLS certificates and estimated that publishing its certificate revocation list would cost the issuer, GlobalSign, $400,000 per month that year.

Exploitation

The Canada Revenue Agency reported a theft of social insurance numbers belonging to 900 taxpayers, and said that they were accessed through an exploit of the bug during a 6-hour period on 8 April 2014. After the discovery of the attack, the agency shut down its website and extended the taxpayer filing deadline from 30 April to 5 May. The agency said it would provide credit protection services at no cost to anyone affected. On 16 April, the RCMP announced they had charged a computer science student in relation to the theft with ''unauthorized use of a computer'' and ''mischief in relation to data''.

The UK parenting site Mumsnet had several user accounts hijacked, and its CEO was impersonated. The site later published an explanation of the incident saying it was due to Heartbleed and the technical staff patched it promptly.

Anti-malware researchers also exploited Heartbleed to their own advantage in order to access secret forums used by cybercriminals. Studies were also conducted by deliberately setting up vulnerable machines. For example, on 12 April 2014, at least two independent researchers were able to steal private keys from an experimental server intentionally set up for that purpose by CloudFlare. Also, on 15 April 2014, J. Alex Halderman, a professor at University of Michigan, reported that his honeypot server, an intentionally vulnerable server designed to attract attacks in order to study them, had received numerous attacks originating from China. Halderman concluded that because it was a fairly obscure server, these attacks were probably sweeping attacks affecting large areas of the Internet.

In August 2014, it was made public that the Heartbleed vulnerability enabled hackers to steal security keys from Community Health Systems, the second-biggest for-profit U.S. hospital chain in the United States, compromising the confidentiality of 4.5 million patient records. The breach happened a week after Heartbleed was first made public.

Possible prior knowledge and exploitation

Many major web sites patched the bug or disabled the Heartbeat Extension within days of its announcement, but it is unclear whether potential attackers were aware of it earlier and to what extent it was exploited.

Based on examinations of audit logs by researchers, it has been reported that some attackers may have exploited the flaw for at least five months before discovery and announcement. Errata Security pointed out that a widely used non-malicious program called ''Masscan'', introduced six months before Heartbleed's disclosure, abruptly terminates the connection in the middle of handshaking in the same way as Heartbleed, generating the same server log messages, adding "Two new things producing the same error messages might seem like the two are correlated, but of course, they aren't."

According to Bloomberg News, two unnamed insider sources informed it that the United States' National Security Agency had been aware of the flaw since shortly after its appearance butinstead of reporting itkept it secret among other unreported zero-day vulnerabilities in order to exploit it for the NSA's own purposes. The NSA has denied this claim, as has Richard A. Clarke, a member of the National Intelligence Review Group on Intelligence and Communications Technologies that reviewed the United States' electronic surveillance policy; he told Reuters on 11 April 2014 that the NSA had not known of Heartbleed. The allegation prompted the American government to make, for the first time, a public statement on its zero-day vulnerabilities policy, accepting the recommendation of the review group's 2013 report that had asserted "in almost all instances, for widely used code, it is in the national interest to eliminate software vulnerabilities rather than to use them for US intelligence collection", and saying that the decision to withhold should move from the NSA to the White House.

Behavior

The RFC 6520 Heartbeat Extension tests TLS/DTLS secure communication links by allowing a computer at one end of a connection to send a ''Heartbeat Request'' message, consisting of a payload, typically a text string, along with the payload's length as a 16-bit integer. The receiving computer then must send exactly the same payload back to the sender.

The affected versions of OpenSSL allocate a memory buffer for the message to be returned based on the length field in the requesting message, without regard to the actual size of that message's payload. Because of this failure to do proper bounds checking, the message returned consists of the payload, possibly followed by whatever else happened to be in the allocated memory buffer.

Heartbleed is therefore exploited by sending a malformed heartbeat request with a small payload and large length field to the vulnerable party (usually a server) in order to elicit the victim's response, permitting attackers to read up to 64 kilobytes of the victim's memory that was likely to have been used previously by OpenSSL. Where a Heartbeat Request might ask a party to "send back the four-letter word 'bird'", resulting in a response of "bird", a "Heartbleed Request" (a malicious heartbeat request) of "send back the 500-letter word 'bird'" would cause the victim to return "bird" followed by whatever 496 subsequent characters the victim happened to have in active memory. Attackers in this way could receive sensitive data, compromising the confidentiality of the victim's communications. Although an attacker has some control over the disclosed memory block's size, it has no control over its location, and therefore cannot choose what content is revealed.

Affected OpenSSL installations

The affected versions of OpenSSL are OpenSSL 1.0.1 through 1.0.1f (inclusive). Subsequent versions (1.0.1g and later) and previous versions (1.0.0 branch and older) are not vulnerable. Installations of the affected versions are vulnerable unless OpenSSL was compiled with ''-DOPENSSL_NO_HEARTBEATS''.

Vulnerable program and function

The vulnerable program source files are t1_lib.c and d1_both.c and the vulnerable functions are tls1_process_heartbeat() and dtls1_process_heartbeat().

Patch

The problem can be fixed by ignoring Heartbeat Request messages that ask for more data than their payload need, as required by the RFC.

Version 1.0.1g of OpenSSL adds some bounds checks to prevent the buffer over-read. The test listed below was one introduced to determine whether a heartbeat request would trigger Heartbleed; it silently discards malicious requests.

if (1 + 2 + payload + 16 > s->s3->rrec.length) return 0; /* silently discard per RFC 6520 sec. 4 */

Impact

The data obtained by a Heartbleed attack may include unencrypted exchanges between TLS parties likely to be confidential, including any form post data in users' requests. Moreover, the confidential data exposed could include authentication secrets such as session cookies and passwords, which might allow attackers to impersonate a user of the service.

An attack may also reveal private keys of compromised parties, which would enable attackers to decrypt communications (future or past stored traffic captured via passive eavesdropping, unless perfect forward secrecy is used, in which case only future traffic can be decrypted if intercepted via man-in-the-middle attacks).

An attacker having gained authentication material may impersonate the material's owner after the victim has patched Heartbleed, as long as the material is accepted (for example, until the password is changed or the private key revoked). Heartbleed therefore constitutes a critical threat to confidentiality. However, an attacker impersonating a victim may also alter data. Indirectly, Heartbleed's consequences may thus go far beyond a confidentiality breach for many systems.

A survey of American adults conducted in April 2014 showed that 60 percent had heard about Heartbleed. Among those using the Internet, 39 percent had protected their online accounts, for example by changing passwords or canceling accounts; 29 percent believed their personal information was put at risk because of the Heartbleed bug; and 6 percent believed their personal information had been stolen.

Client-side vulnerability

Although the bug received more attention due to the threat it represents for servers, TLS clients using affected OpenSSL instances are also vulnerable. In what ''The Guardian'' therefore dubbed ''Reverse Heartbleed'', malicious servers are able to exploit Heartbleed to read data from a vulnerable client's memory. Security researcher Steve Gibson said of Heartbleed that:

The stolen data could contain usernames and passwords. Reverse Heartbleed affected millions of application instances. Some of the vulnerable applications are listed in the "Software applications" section below.

Specific systems affected

Cisco Systems has identified 78 of its products as vulnerable, including IP phone systems and telepresence (video conferencing) systems.

Websites and other online services

An analysis posted on GitHub of the most visited websites on 8 April 2014 revealed vulnerabilities in sites including Yahoo!, Imgur, Stack Overflow, ''Slate'', and DuckDuckGo. The following sites have services affected or made announcements recommending that users update passwords in response to the bug:

* Akamai Technologies
* Amazon Web Services
* Ars Technica
* Bitbucket
* BrandVerity
* Freenode
* GitHub
* IFTTT
* Internet Archive
* Mojang
* Mumsnet
* PeerJ
* Pinterest
* Prezi
* Reddit
* Something Awful
* SoundCloud
* SourceForge
* SparkFun
* Stripe
* Tumblr
* All Wikimedia Foundation wikis (including Wikipedia in all languages)
* Wunderlist

The Canadian federal government temporarily shut online services of the Canada Revenue Agency (CRA) and several government departments over Heartbleed bug security concerns. Before the CRA online services were shut down, a hacker obtained approximately 900 social insurance numbers. Another Canadian Government agency, Statistics Canada, had its servers compromised due to the bug and also temporarily took its services offline.

Platform maintainers like the Wikimedia Foundation advised their users to change passwords.

The servers of LastPass were vulnerable, but due to additional encryption and forward secrecy, potential attacks were not able to exploit this bug. However, LastPass recommended that its users change passwords for vulnerable websites.

The Tor Project recommended that Tor relay operators and hidden service operators revoke and generate fresh keys after patching OpenSSL, but noted that Tor relays use two sets of keys and that Tor's multi-hop design minimizes the impact of exploiting a single relay. 586 relays later found to be susceptible to the Heartbleed bug were taken off-line as a precautionary measure.

Game-related services including Steam, ''Minecraft'', Wargaming, ''League of Legends'', GOG.com, Origin, Sony Online Entertainment, Humble Bundle, and ''Path of Exile'' were affected and subsequently fixed.

Software applications

Vulnerable software applications include:

* Several Hewlett-Packard server applications, such as HP System Management Homepage (SMH) for Linux and Windows.
* Some versions of FileMaker 13
* LibreOffice 4.2.0 to 4.2.2 (fixed in 4.2.3)
* LogMeIn claimed to have "updated many products and parts of our services that rely on OpenSSL".
* Multiple McAfee products, in particular some versions of software providing anti-viral coverage for Microsoft Exchange, software firewalls, and McAfee Email and Web Gateways
* Oracle MySQL Connector/C 6.1.0-6.1.3 and Connector/ODBC 5.1.13, 5.2.5-5.2.6, 5.3.2
* Oracle Big Data Appliance (includes Oracle Linux 6)
* Primavera P6 Professional Project Management (includes Primavera P6 Enterprise Project Portfolio Management)
* WinSCP (FTP client for Windows) 5.5.2 and some earlier versions (only vulnerable with FTP over TLS/SSL, fixed in 5.5.3)
* Multiple VMware products, including VMware ESXi 5.5, VMware Player 6.0, VMware Workstation 10 and the series of Horizon products, emulators and cloud computing suites

Several other Oracle Corporation applications were affected.

Operating systems/firmware

Several Linux distributions were affected, including Debian (and derivatives such as Linux Mint and Ubuntu) and Red Hat Enterprise Linux (and derivatives such as CentOS, Oracle Linux 6 and Amazon Linux), as well as the following operating systems and firmware implementations:

* Android 4.1.1, used in various portable devices. Chris Smith writes in Boy Genius Report that just this one version of Android is affected but that it is a popular version of Android (Chitika claim 4.1.1 is on 50 million devices; Google describe it as less than 10% of activated Android devices). Other Android versions are not vulnerable as they either have heartbeats disabled or use an unaffected version of OpenSSL.
* Firmware for some AirPort base stations
* Firmware for some Cisco Systems routers
* Firmware for some Juniper Networks routers
* pfSense 2.1.0 and 2.1.1 (fixed in 2.1.2)
* DD-WRT versions between and including 19163 and 23881 (fixed in 23882)
* Western Digital My Cloud product family firmware

Vulnerability testing services

Several services have been made available to test whether Heartbleed affects a given site. However, many services have been claimed to be ineffective for detecting the bug. The available tools include:

* Tripwire SecureScan
* AppCheck – static binary scan and fuzzing, from Synopsys Software Integrity Group (formerly Codenomicon)
* Arbor Network's Pravail Security Analytics
* Norton Safeweb Heartbleed Check Tool
* Heartbleed testing tool by a European IT security company
* Heartbleed test by Italian cryptographer Filippo Valsorda
* Heartbleed Vulnerability Test by Cyberoam
* Critical Watch Free Online Heartbleed Tester
* Metasploit Heartbleed scanner module
* Heartbleed Server Scanner by Rehmann
* Lookout Mobile Security Heartbleed Detector, an app for Android devices that determines the OpenSSL version of the device and indicates whether the vulnerable heartbeat is enabled
* Heartbleed checker hosted by LastPass
* Online network range scanner for Heartbleed vulnerability by Pentest-Tools.com
* Official Red Hat offline scanner written in the Python language
* Qualys SSL Labs' SSL Server Test which not only looks for the Heartbleed bug, but can also find other SSL/TLS implementation errors.
* Browser extensions, such as Chromebleed and FoxBleed
* SSL Diagnos
* CrowdStrike Heartbleed Scanner – Scans routers, printers and other devices connected inside a network including intranet web sites.
* Netcraft Site Report – indicates whether a website's confidentiality could be jeopardized due to a past exploitation of Heartbleed by checking data from Netcraft's SSL Survey to determine whether a site offered the heartbeat TLS Extension prior to the Heartbleed disclosure. The Netcraft Extensions for Chrome, Firefox and Opera also perform this check, whilst looking for potentially compromised certificates.

Other security tools have added support for finding this bug. For example, Tenable Network Security wrote a plugin for its Nessus vulnerability scanner that can scan for this fault. The Nmap security scanner includes a Heartbleed detection script from version 6.45.

Sourcefire has released Snort rules to detect Heartbleed attack traffic and possible Heartbleed response traffic. Open source packet analysis software such as Wireshark and tcpdump can identify Heartbleed packets using specific BPF packet filters that can be used on stored packet captures or live traffic.

Remediation

Vulnerability to Heartbleed is resolved by updating OpenSSL to a patched version (1.0.1g or later). OpenSSL can be used either as a standalone program, a dynamic shared object, or a statically-linked library; therefore, the updating process can require restarting processes loaded with a vulnerable version of OpenSSL as well as re-linking programs and libraries that linked it statically. In practice this means updating packages that link OpenSSL statically, and restarting running programs to remove the in-memory copy of the old, vulnerable OpenSSL code.

After the vulnerability is patched, server administrators must address the potential breach of confidentiality. Because Heartbleed allowed attackers to disclose private keys, they must be treated as compromised; key pairs must be regenerated, and certificates that use them must be reissued; the old certificates must be revoked. Heartbleed also had the potential to allow disclosure of other in-memory secrets; therefore, other authentication material (such as passwords) should also be regenerated. It is rarely possible to confirm that a system which was affected has not been compromised, or to determine whether a specific piece of information was leaked.

Since it is difficult or impossible to determine when a credential might have been compromised and how it might have been used by an attacker, certain systems may warrant additional remediation work even after patching the vulnerability and replacing credentials. For example, signatures made by keys that were in use with a vulnerable OpenSSL version might well have been made by an attacker; this raises the possibility integrity has been violated, and opens signatures to repudiation. Validation of signatures and the legitimacy of other authentications made with a potentially compromised key (such as client certificate use) must be done with regard to the specific system involved.

Browser security certificate revocation awareness

Since Heartbleed threatened the privacy of private keys, users of a website which was compromised could continue to suffer from Heartbleed's effects until their browser is made aware of the certificate revocation or the compromised certificate expires. For this reason, remediation also depends on users making use of browsers that have up-to-date certificate revocation lists (or OCSP support) and honour certificate revocations.

Root causes, possible lessons, and reactions

Although evaluating the total cost of Heartbleed is difficult, ''eWeek'' estimated US$500 million as a starting point.

David A. Wheeler's paper ''How to Prevent the next Heartbleed'' analyzes why Heartbleed wasn't discovered earlier, and suggests several techniques which could have led to a faster identification, as well as techniques which could have reduced its impact. According to Wheeler, the most efficient technique which could have prevented Heartbleed is a test suite thoroughly performing robustness testing, i.e. testing that invalid inputs cause failures rather than successes. Wheeler highlights that a single general-purpose test suite could serve as a base for all TLS implementations.

According to an article on ''The Conversation'' written by Robert Merkel, Heartbleed revealed a ''massive failure of risk analysis''. Merkel thinks OpenSSL gives more importance to performance than to security, which no longer makes sense in his opinion. But Merkel considers that OpenSSL should not be blamed as much as OpenSSL users, who chose to use OpenSSL, without funding better auditing and testing. Merkel explains that two aspects determine the risk that more similar bugs will cause vulnerabilities. One, the library's source code influences the risk of writing bugs with such an impact. Secondly, OpenSSL's processes affect the chances of catching bugs quickly. On the first aspect, Merkel mentions the use of the C programming language as one risk factor which favored Heartbleed's appearance, echoing Wheeler's analysis.

On the same aspect, Theo de Raadt, founder and leader of the OpenBSD and OpenSSH projects, has criticized the OpenSSL developers for writing their own memory management routines and thereby, he claims, circumventing OpenBSD C standard library exploit countermeasures, saying "OpenSSL is not developed by a responsible team." Following Heartbleed's disclosure, members of the OpenBSD project forked OpenSSL into LibreSSL. LibreSSL made a big code cleanup, removing more than 90,000 lines of C code just in its first week.

The author of the change which introduced Heartbleed, Robin Seggelmann, stated that he ''missed validating a variable containing a length'' and denied any intention to submit a flawed implementation. Following Heartbleed's disclosure, Seggelmann suggested focusing on the second aspect, stating that OpenSSL is not reviewed by enough people. Although Seggelmann's work was reviewed by an OpenSSL core developer, the review was also intended to verify functional improvements, a situation making vulnerabilities much easier to miss.

OpenSSL core developer Ben Laurie claimed that a security audit of OpenSSL would have caught Heartbleed. Software engineer John Walsh commented: The OpenSSL foundation's president, Steve Marquess, said "The mystery is not that a few overworked volunteers missed this bug; the mystery is why it hasn't happened more often." David A. Wheeler described audits as an excellent way to find vulnerabilities in typical cases, but noted that "OpenSSL uses unnecessarily complex structures, which makes it harder to both humans and machines to review." He wrote:

There should be a continuous effort to simplify the code, because otherwise just adding capabilities will slowly increase the software complexity. The code should be refactored over time to make it simple and clear, not just constantly add new features. The goal should be code that is "obviously right", as opposed to code that is so complicated that "I can't see any problems".

According to security researcher Dan Kaminsky, Heartbleed is sign of an economic problem which needs to be fixed. Seeing the time taken to catch this simple error in a simple feature from a "critical" dependency, Kaminsky fears numerous future vulnerabilities if nothing is done. When Heartbleed was discovered, OpenSSL was maintained by a handful of volunteers, only one of whom worked full time. Yearly donations to the OpenSSL project were about US$2,000. The Heartbleed website from Codenomicon advised money donations to the OpenSSL project. After learning about donations for the 2 or 3 days following Heartbleed's disclosure totaling US$841, Kaminsky commented "We are building the most important technologies for the global economy on shockingly underfunded infrastructure." Core developer Ben Laurie has qualified the project as "completely unfunded". Although the OpenSSL Software Foundation has no bug bounty program, the Internet Bug Bounty initiative awarded US$15,000 to Google's Neel Mehta, who discovered Heartbleed, for his responsible disclosure. Mehta later donated his reward to a Freedom of the Press Foundation fundraiser.

Paul Chiusano suggested Heartbleed may have resulted from failed software economics.

The industry's collective response to the crisis was the Core Infrastructure Initiative, a multimillion-dollar project announced by the Linux Foundation on 24 April 2014 to provide funds to critical elements of the global information infrastructure. The initiative intends to allow lead developers to work full time on their projects and to pay for security audits, hardware and software infrastructure, travel, and other expenses. OpenSSL is a candidate to become the first recipient of the initiative's funding.

After the discovery Google established Project Zero which is tasked with finding ''zero-day vulnerabilities'' to help secure the Web and society.

References

Bibliography

*
*
*

External links

Summary and Q&A about the bug
by Codenomicon Ltd

Information for Canadian organizations and individuals

List of all security notices

{{TLS/SSL

2014 in computing
Internet security
Software bugs
Transport Layer Security
Computer security exploits