Summary

Between 2026-07-16 and 2026-07-29, a caching bug caused webhooks to be intermittently not delivered for about one hour for projects whose configuration was updated during this period.

Root Cause

A bug in an internal configuration service caused webhook settings to be dropped from a project's cached configuration whenever that project's record was updated. Once triggered, our system stopped delivering webhooks until the cache refreshed from the database, up to one hour later.

Because of this caching behavior, the impact was intermittent: webhooks would stop, then resume on their own, then stop again after the next project update. Additionally, the "test webhook" feature in the dashboard reads webhook configuration through a different path and was unaffected, so test webhooks succeeded even while real event delivery was failing.

Timeline (UTC)

• 2026-07-16 23:48 - Change containing the bug reached production; intermittent impact begins
• 2026-07-24 - Some customer reports of missing webhooks received; investigation begins
• 2026-07-28 16:23 - Root cause identified; fix developed and rollout begins
• 2026-07-29 16:26 - Fix deployed to all regions; incident resolved

Scope of Impact

Customers using webhooks were affected across all regions between 2026-07-16 and 2026-07-29, with most impact occurring after 2026-07-22. A project was affected only if its configuration was updated during the window, and each occurrence lasted up to one hour. Affected projects saw room_started and participant_* webhooks silently not delivered, while room_finished webhooks and dashboard webhook tests often continued to work. If you observed missing webhooks and had recently made any settings changes to your project, you were likely impacted.

Mitigations and Follow-ups

• The bug has been fixed and a regression test has been added.
• We added logging of webhook delivery response codes to improve visibility into delivery failures.
• We are adding per-project webhook delivery monitoring and alerting that can detect missing deliveries independent of overall volume.
• We are unifying the configuration paths used by webhook delivery and the dashboard webhook test, so a passing test reflects actual delivery behavior.
• We are making the configuration cache lifetime adjustable so similar issues can be mitigated quickly in the future.