Disconnections and failed connections in multiple regions

Incident Report for LiveKit

Resolved

This incident has been resolved. Between 07:10 and 07:43 UTC on October 31, a software bug caused media servers in several regions to restart repeatedly.

During this window:
- Participants connected to affected servers were disconnected. Impact was heaviest in US East, US Central, US West, Canada and Mexico, where about 18% of participants were disconnected at least once. Many were disconnected more than once, as automatic reconnects landed on servers that then restarted as well. A small share of participants in Europe Central, Europe South, UAE, Saudi Arabia and Israel were also disconnected. Most disconnected participants reconnected automatically, about half within 5 seconds, but each reconnect started a new session rather than resuming the existing one, so participants may have seen repeated audio and video interruptions. Some participants did not reconnect. Participants joining during this window may also have taken longer to connect or failed to join.
- Most agents reconnected and continued their conversations, but a small number of agent sessions ended or restarted. Some SIP call transfers also failed.
- About 1.3% of Room Service API requests failed during this period. ListParticipants and GetParticipant were the most affected. Some WHIP ingress update and delete requests also failed.

The bug was triggered when a specific type of data message was relayed between servers hosting the same multi-region room. We stopped the trigger at 07:43 UTC, and there were no further restarts after that. A fix was deployed to all regions by 08:49 UTC.

Apologies for the impact. We will follow up with a postmortem sharing more details on impact and RCA as soon as possible.
Posted Oct 31, 2026 - 00:43 PDT
This incident affected: Regional Ingress (US East - Ingress, US West - Ingress, US Central - Ingress) and Regional Real Time Communication (US West - Real Time Communication, US Central - Real Time Communication, UAE - Real Time Communication, Israel - Real Time Communication, Saudi Arabia - Real Time Communication, Europe South - Real Time Communication, US East - Real Time Communication, Europe Central - Real Time Communication).