Postmortem -
Read details
May 9, 00:37 PDT
Resolved -
US-East Cloud Agents is now fully functional.
May 6, 02:41 PDT
Monitoring -
US-East recovered as of 08:55 UTC and has been serving traffic normally since. New and existing agent deployments are working as expected.
We are moving to monitoring while we confirm sustained stability. A full post-incident review will follow.
May 6, 02:17 PDT
Update -
We are continuing to work on restoring the Kubernetes API server in US-east. Starting at 08:15 UTC, we are observing impact to existing agent deployments in addition to new deployments. Mitigation is in progress.
May 6, 01:47 PDT
Update -
etcd service in our US East Kubernetes cluster is currently down, resulting in API server unresponsiveness and failures for new deployments and redeployments. We are actively working on mitigating this with our data center provider.
May 5, 22:34 PDT
Update -
Our team is actively working to resolve deployment failures in US East. Existing deployed agents should not be affected in any way.
May 5, 22:00 PDT
Update -
New agent deployments are temporarily unavailable in the US East region, and our team is actively working on this. Dispatches to previously deployed agents are not affected.
May 5, 21:22 PDT
Update -
We are currently investigating intermittent deployment failures on Cloud Agents in US East.
May 5, 21:06 PDT
Investigating -
We are currently investigating degraded performance on Agents Hosted on LiveKit Cloud in US East.
May 5, 21:01 PDT