On Monday, we experienced a number of issues related to expired certificates on humanitec.dev subdomains.
The issue, which began just before midnight CEST (technically, Sunday), was caused by conflicting ACME challenge records for these domains. Specifically, the domains were originally set up using a system that required TXT records for ACME challenge verification. Subsequently, as part of work to move our internal Kubernetes ingress away from ingress-nginx, a number of domains were updated to another system, which uses CNAME records for the same verifications. Although the initial certificate generation was not problematic, upon attempted renewal, the system errored out.
In our post-mortem, we discussed and have added to our roadmap the following items to mitigate this from happening in the future, across all of our domains:
Add monitoring and alerting on “time to expiration” (where it does not already exist), so that this issues are flagged well before certificate expiration.
Shorten expiration lengths in development and staging environments, so similar errors that could affect production would happen to those other environments first.
Review run book to ensure other systems migrating off of ingress-nginx do not experience similar issues.
As always, please reach out with any questions.
No components marked as affected
Resolved
On Monday, we experienced a number of issues related to expired certificates on humanitec.dev subdomains.
The issue, which began just before midnight CEST (technically, Sunday), was caused by conflicting ACME challenge records for these domains. Specifically, the domains were originally set up using a system that required TXT records for ACME challenge verification. Subsequently, as part of work to move our internal Kubernetes ingress away from ingress-nginx, a number of domains were updated to another system, which uses CNAME records for the same verifications. Although the initial certificate generation was not problematic, upon attempted renewal, the system errored out.
In our post-mortem, we discussed and have added to our roadmap the following items to mitigate this from happening in the future, across all of our domains:
Add monitoring and alerting on “time to expiration” (where it does not already exist), so that this issues are flagged well before certificate expiration.
Shorten expiration lengths in development and staging environments, so similar errors that could affect production would happen to those other environments first.
Review run book to ensure other systems migrating off of ingress-nginx do not experience similar issues.
As always, please reach out with any questions.
Resolved
The system is once again fully functional.
Monitoring
This morning, we have experienced a number of issues related to expired certificates on the humanitec.dev domain.
The issue is now resolved and propagating. We are now monitoring this will, and share a detailed summary of the cause of the issue after a post-mortem meeting later this week.
Identified
We are actively working on a fix for the certificate issue with humanitec.dev domains.
Identified
We are experiencing issues with Humanitec V2, due to issues with several TLS certificates. W are investigating
