In April 2014, the Heartbleed vulnerability was publicly disclosed, sending the information technology world into a panic and rushing to patch this critical vulnerability in OpenSSL, which was allowing the theft of information directly from the memory of vulnerable systems, including private keys and other secrets. This vulnerability featured extremely easy exploitation by attackers, leaving no trace of attacks. Heartbleed ultimately resulted in many late nights for most of the information technology industry, who worked to implement and validate patches for open and closed source products that have integrated the OpenSSL libraries – which accounts for an extremely large percentage of technologies connected to the internet.
While initial estimates of public-facing systems vulnerable to Heartbleed were at 600,000, the actual number of vulnerable systems, including those inside corporate networks and backend servers without direct exposure to the internet, were likely much higher. According to Errata Security, the number of publicly exposed unpatched systems was reduced by 50% to 300,000[1] by June 2014. Then, progress seems to have significantly decreased. Utilizing the security tool Shodan, Risk Based Security found 224,858 vulnerable devices accessible on the Internet in 2016,[2] and F5 Labs found that 199,594 devices were still vulnerable in January 2017.[3] As of January 2023, utilizing the same security tool, the number of vulnerable devices on the internet has only slightly decreased to 194,078.[4]
One of the most unsettling data points for current Heartbleed vulnerability statistics is the number of vulnerable systems hosted on Amazon AWS. In 2017, 6,375 vulnerable systems were hosted on AWS, and as of January 2023, that number has increased to 7,575. That means since 2017, organizations have deployed over 1,000 virtual machines in AWS that are vulnerable to Heartbleed. The number of vulnerable systems deployed with the vulnerability after patches are available is likely even higher, once we take into account that (we hope) a significant number of systems were patched for Heartbleed. Organizations are clearly struggling when it comes to vulnerability management, especially with cloud deployments that are not as easily scanned as on-premises servers.
One of the most common deployment methods in AWS is the AWS “marketplace.” This provides AWS users to select pre-created machine templates or even virtual appliances and deploy them immediately. While this greatly simplifies things for IT administrators, sometimes the administrators fail to perform required security updates after deploying the virtual machine. If a virtual machine template was created prior to patching Heartbleed, it’s very possible that the machine will contain the Heartbleed vulnerability when deployed, and unfortunately, EMA research shows that a small percentage of organizations believe their cloud service provider protects them from all attacks, completely disregarding the shared responsibility model.
Examining the top 10 vulnerable products also reveals some interesting data. Most of the vulnerable servers are running Apache httpd, which is commonly used not just for hosting web servers, but sometimes embedded applications. In fact, it’s likely that legacy and embedded applications make up the bulk of the remaining unpatched devices still vulnerable to Heartbleed. Interestingly, some vendors have still not issued patches for Heartbleed, or at least have not publicly announced a patch. In nearly all instances, OpenSSL itself cannot be updated directly because it has been incorporated into another product. A recent EMA research project found that it’s extremely common for organizations to use third-party code in their products, with almost 70% of respondents using open source third-party code in their products and another 30% utilizing closed source third-party code.[5] This research showed that organizations utilizing third-party code often experienced a tradeoff: lower software development costs, but at higher risk of vulnerabilities. This risk is typically acceptable, as long as third-party code is also maintained on a regular basis and frequently updated for security vulnerabilities. Unfortunately, what should be an industry standard practice of updating third-party libraries does not always happen.
One of the most serious issues with patching is the lack of patches for discontinued products. For example, Boa HTTPd and Boa Web Server were discontinued in 2005, meaning any devices using an embedded Boa Web server will never receive updates. Other products, such as nginx or lighttpd, are still maintained but are lightweight distributions intended for embedded applications, often creating logistical challenges in updating, and organizations using these vulnerable devices may have no idea that the devices need updates. Unfortunately, in some cases, even if updates to the core software are available, these embedded devices require a special build and update process from the manufacturer. Legacy devices that have been determined to be “end of life” by vendors may not even receive updates from the vendors, even if the original software has a patch available.
Unfortunately, the problem of unpatched vulnerabilities (especially in legacy and embedded devices utilizing third-party code) is not unique to Heartbleed. In fact, some vulnerabilities that received less press coverage than Heartbleed and are just as dangerous have significantly higher vulnerable device counts. Just a few examples include:
Many of the products about which the above vulnerabilities apply had patches available for years, but quite often, administrators are unaware of the risk posed by these unpatched devices or have not received an update from the vendor that utilized the vulnerable software as part of their own product. As long as vendors continue to utilize third-party code, we will continue to see far-reaching vulnerabilities, such as Heartbleed or Log4j.
Third-party code usage directly in applications and as part of infrastructure those applications rely upon will only continue to grow. Organizations and vendors need to start focusing not only on their own security, but also on the security of third-party code utilized in embedded and non-embedded applications.
It’s time for organizations to start investing in better vulnerability and attack surface management technologies to prevent malicious actors from exploiting these unknown risks.
EMA is currently planning follow-up research on DevSecOps and securing today’s enterprise, which should be available in Q3 2023. Vendors interested in sponsoring this research can contact our business development team for more information.
[1] https://blog.erratasec.com/2014/06/300k-vulnerable-to-heartbleed-two.html
[2] https://prev.riskbasedsecurity.com/2016/04/08/two-year-anniversary-for-heartbleed-still-many-vulnerable-devices/
[3] https://www.f5.com/labs/articles/threat-intelligence/friendly-reminder-app-security-in-the-cloud-is-your-responsibility-24894
[4] https://www.shodan.io/search/facet?query=vuln%3ACVE-2014-0160&facet=product
[5] https://www.enterprisemanagement.com/research/asset.php/4241/Secure-Coding-Practices-%96-Growing-Success-or-Zero-Day-Epidemic