Lessons From An Internet Outage - Issues Caused By Let’s Encrypt DST Root CA X3 Expiration

Updated
Published: October 1, 2021
8 mins read
By Sergey Katsev

In this blog post

As a monitoring and observability company, we have a lot of monitoring built into our systems. We have the standard monitoring to ensure that systems are performing properly, and data is flowing through our infrastructure. At the same time, we have monitoring for sudden changes to tests that our customers are running.

On September 29, 2021, 19:21:40 UTC, we started to see a tsunami of alerts at Catchpoint. They originated from some of our web tests from our synthetic nodes, occurring when our Let’s Encrypt “R3” certificate expired. These types of incidents are pretty rare.

The root cause of the crisis was not Catchpoint, our product, or any employee – but an issue with changes to the certificate path by a certificate issuer. As we work with many vendors, we’ve received updates from some which indicate that solving this problem is as easy as downloading the latest OS updates. While this is true for some, it does not solve the problem in general!

Below we explain why, and how to solve it on the server-side so that all of your clients can access your web service without issues. We also share our incident review of the event soe that the learnings will help others.

Do You Trust Let’s Encrypt?

Before we get into the weeds, I personally trust Let’s Encrypt. They’re a great company that’s made certificate management extremely accessible and developer-friendly.

Because of them, the number of websites using encryption has skyrocketed in recent years. Encryption is extremely important on the Internet. It’s the basis for secure communications. Whether you’re checking your bank balance, buying a new pair of socks from an e-tailer, or talking to your friends, you do so with the assumption that this transaction is secure.

In the last ~8 years (2013-2020), the percent of web pages using HTTPS has gone from 25% to over 84%! One of the reasons for this incredible growth is that Google has been gradually forcing sites to use HTTPS by making HTTP-based sites “not secure.”

At the same time, it doesn’t matter that I trust Let’s Encrypt if computers don’t. That’s exactly what happened on the evening of September 29 and again on the morning of September 30, 2021.

How Digital Certificate Trust Works

Here’s a quick summary of how certificate trust works on the Internet:

Root certificates: Root certificates are issued by major companies under scrutiny and installed in the Certificate Stores of computers worldwide. If you have a computer that can connect to an HTTPS website, you have such a certificate store.
Chain of Trust: When someone launches a website, they must support HTTPS and purchase a certificate from a provider, allowing a chain of trust from the website’s certificate to the root certificate.
Intermediate certificates: When you access an HTTPS website, the server sends the certificate during the SSL handshake with the client, which might send one or more intermediate certificates to connect the chain of trust from the server certificate to one of the root certificates.
Validation process: Your browser “walks” the chain of trust from the server certificate up to the root. If validated, the connection is allowed to proceed, otherwise, you get a Security Warning.

R3 Certificate Expiry And The Chain Of Trust…

Let’s go through the details of what happened:
As mentioned, the problems we saw with web tests from our synthetic nodes began at 19:21:40 UTC on September 29 when the Let’s Encrypt “R3” certificate expired. Here’s the certificate information for this intermediate certificate:
https://crt.sh/?id=3479778542
The expiration date of this certificate is crucial since it means that Let’s Encrypt used this certificate to sign other certificates for their customers.

The screenshot shows a website whose certificate was signed with this R3 certificate. After expiration, this website was no longer accessible! Browsers validating the website’s certificate would find the intermediate certificate expired, resulting in Security Warnings.

Let’s Encrypt published a new R3 certificate with an expiration date in 2025.

However, the certificate needs to be updated on your computer – usually through a Windows or MacOS update – before it’ll work. Many people don’t update their computers often enough, and some embedded devices rarely update their certificates.

September 30th, 10AM EST: DST Root CA X3 Certificate Expiry And The Consequences

At 10AM on September 30, the DST Root CA X3 certificate expired. This caused problems for systems relying on Let’s Encrypt certificates. Here’s Let’s Encrypt’s diagram of their certificate hierarchy:)

When the DST root certificate expired, it caused issues for two classes of systems:

Systems not updated with ISRT Root X1 certificate—failing to connect to Let’s Encrypt.
Systems with updated ISRT Root X1 certificate attempting to validate the expired DST Root certificate.

Fixes, Fixes

The first category was easy to fix: update the OS or download and install the new certificate. The second one is harder, especially for software relying on OpenSSL 1.0.2 or earlier, which have no fix.

If you run a website and want customers to be able to connect, regenerate the certificate that your site uses, to stop sending it to clients.

Fixing This On Your Server

Here’s how I fixed the issue on a test server. It was discovered that some clients ignore the certificate chain sent by the server; some certificates were signed by DST and the chain configured by Let’s Encrypt still included the ISRG certificate signed by DST.

On my system (Ubuntu 18.04 LTS with Apache 2 and Let’s Encrypt’s certbot installed), the certificates are in /etc/letsencrypt/live/www.mysite.com/fullchain.pem.

You can check this using openssl s_client using this command: openssl s_client -connect www.site.com:443 –showcerts

After replacing the incorrect ISRG certificate with the latest version and restarting the web server, the correct chain is now offered.

In Conclusion...

Here at Catchpoint, we have a huge footprint of test agents in every corner of the world. These agents have different configurations of hardware, software, firmware, etc.

However, the customers accessing your website probably don’t have the same operational support. The solution lies on the server side, and it is up to you to ensure your systems trust Let’s Encrypt once again.