Delivery & API outage
Started 24 Nov at 03:15pm EST, last updated 25 Nov at 02:42am EST.
The reconciliation job is complete, all webhooks have been delivered and all audits have passed. Everything is back in order.
We've restored from a backup dating from 2 hours ago and have put all services back online. Some events and attempts will be missing from the dashboard until we've completed a reconciliation job.
We're running into unforeseen issues during a database upgrade that's preventing writes to the database. During that time any write operations on the API will fail and events will NOT be delivered nor shown in the UI. However, we are still ingestion events and no webhooks have been lost.