A full post-mortem for this issue is available here: https://hookdeck.notion.site/Bigtable-Write-Outage-Request-Processing-Disruption-36c783a05de280a2af03c530bda45246

The system is fully recovered, and the dashboard/API data should correctly represent the system state. We're sorry for the trouble caused, and we will publish a postmortem within a few days.

We identified the root cause of a GCP Bigtable outage in us-east-1, which has been confirmed by the GCP support team. However, the system should be able to absorb a Bigtable outage, and we're looking into why the issue had broader performance consequences.

The event delivery has recovered, and we've fully caught up on the backlog. However, the dashboard and API will not show complete data yet as we are still processing the data for the user-facing dataset.

We're experiencing a delay in the event delivery pipeline, which is also impacting other system operations like retries. Requests are still being ingested and no data is lost. We are working on stabilizing the system.