On Wednesday, the Humanitec Platform Orchestrator was unable to deploy for customers using temporary credentials in AWS. I am reaching out to provide an overview of what happened, and steps we are taking to prevent future issues going forward.

Summary

At approximately 9:22 CEST, as part of an infrastructure clean-up project, the AWS account that hosts the trusted user used for temporary authentication to Humanitec was destroyed, alongside a number of other no-longer-used accounts.

Our team was notified of the error almost immediately, quickly identified its source, and pursued a strategy to restore the specific account through our third-party provider. (See below.) When it was clear that the response from said provider was causing undue delay, our team prepared a separate remedy, changing the trusted user associated with our temporary credentials to a new account. As there can only be one trusted user associated with temporary credentials via STS, we knew this was a backwards-incompatible operation.

At 15:10 CEST, after confirming the fix functioned as planned, we updated our production systems to reference the new user in our systems. Shortly thereafter, we began reaching out to affected customers with instructions on how to update. With this change, the issue was effectively resolved.

Humanitec works with a third-party reseller to simplify management for several cloud providers. While providing some benefits to cost reporting and support, it also adds a layer of indirection when performing less-common operations like restoring a deleted account. In the end, the process from contacting the vendor to having the accoung restored took approximately 24 hours.

Factors that contributed to the issue, and associated remedies

In review of the incident, our team identified a number of contributing factors, and associated remedies to mitigate future incidents:

Factor: The oversight was due to a misleading name for the account – which referenced the term “demo” – and insufficient checks during planning for the deletions.
Remedies: We will audit and rename all existing cloud accounts, to denote usage clearly.

Factor: Our ability to react to this issue, in particular, was clearly hampered by having an intermediary between us and AWS.
Remedy: We will shift to a direct relationship with AWS this quarter.

Factor: In supporting temporary credentials, our set-up for AWS differs from what we do in Azure and GCP, and is less resilient. Our set-ups for Azure and GCP both use OIDC.
Remedy: We have prioritized work to support using OIDC on AWS in our backlog, anticipating completion in the coming weeks. We will follow up with details on moving to the updated methodology at that time.

Factor: Simply put, we took too long to decide to address the issue by making a breaking change. This amount of downtime is hard to stomach, regardless of source.
Remedy: We are updating our incident run books to ensure they factor in time to resolution alongside impact and risk. The Incident Commander will be responsible for driving forward to resolution, alongside their existing responsibilities.

We are deeply sorry about the impact this incident may have had on your organization. Please reach out to me or your customer success team with any follow-up questions.

Louis Bennett
VP of Engineering @ Humanitec

We have confirmed that the change of trusted account has resolved the issue for a number of affected customers, so are marking this incident resolved.

As before, please reach out to your customer success team if you have any questions.

Earlier today, the AWS organization that hosts the trusted user used for temporary authentication to Humanitec was inadvertently destroyed. We have escalated with our AWS partner to restore the deleted organization, but rather than continue to wait – blocking deployments for customers like you – we are simply changing to a new trusted user on our end.

As a customer, you can find specifics on what you need to do in order to update things on your side here: https://www.notion.so/humanitec/Updating-AWS-temporary-credentials-34a000565fc68066a060ded9ca275708

We are deeply sorry about the downtime and any impact that it has had on your organization. We'll be in touch in the coming days with any relevant updates as well as a summary of our post-mortem for this incident.

We are currently experiencing an issue with temporary credentials use in AWS, as the AWS organization associated with the primary account used was inadvertently deleted. We have escalated with our AWS partner to restore the deleted organization, but are not able to provide a specific estimate for whether or when they will be able to restore the organization.

Accordingly, we are currently testing a remediation for the issue, which should be shared with customers by 16:00 CEST.

We are currently experiencing an issue with temporary credentials use in AWS, as the AWS organization associated with the primary account used was inadvertently deleted. :grimacing:

We are working with our AWS partner to restore the deleted organization, and are prepared to have customers switch to a new organization if restoring is not an option.

Thank for you for your patience as we address this issue, and continued apologies for the associated disruption in service.

We continue to pursue a remediation strategy for this issue. Our sincere apologies to those affected.

We have identified the probable cause of this incident, and are working to remediate it.

Another update will be provided before 11:00 CEST.