Incidents reported on status page for Hookdeck https://status.hookdeck.com/ https://d1lppblt9t2x15.cloudfront.net/logos/3d89b8fa7bbd9fd360d6f98868ea2116.png https://status.hookdeck.com/ en https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 22:50:00 -0000 https://status.hookdeck.com/incident/601752#349d203bb10a35327f7f2e1d631ab0a7d304166f364ff4e7a79f3084cc4a5346 Service is back to normal. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 22:50:00 -0000 https://status.hookdeck.com/incident/601752#349d203bb10a35327f7f2e1d631ab0a7d304166f364ff4e7a79f3084cc4a5346 Service is back to normal. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 22:50:00 -0000 https://status.hookdeck.com/incident/601752#349d203bb10a35327f7f2e1d631ab0a7d304166f364ff4e7a79f3084cc4a5346 Service is back to normal. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 22:50:00 -0000 https://status.hookdeck.com/incident/601752#349d203bb10a35327f7f2e1d631ab0a7d304166f364ff4e7a79f3084cc4a5346 Service is back to normal. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 22:50:00 -0000 https://status.hookdeck.com/incident/601752#349d203bb10a35327f7f2e1d631ab0a7d304166f364ff4e7a79f3084cc4a5346 Service is back to normal. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 22:50:00 -0000 https://status.hookdeck.com/incident/601752#349d203bb10a35327f7f2e1d631ab0a7d304166f364ff4e7a79f3084cc4a5346 Service is back to normal. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 22:50:00 -0000 https://status.hookdeck.com/incident/601752#349d203bb10a35327f7f2e1d631ab0a7d304166f364ff4e7a79f3084cc4a5346 Service is back to normal. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 22:50:00 -0000 https://status.hookdeck.com/incident/601752#349d203bb10a35327f7f2e1d631ab0a7d304166f364ff4e7a79f3084cc4a5346 Service is back to normal. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 20:21:00 -0000 https://status.hookdeck.com/incident/601752#3fa3abe8e6fd671d717b9d3ff8f6482fdd7c51c28b950fe09efc76cba1ffa7e0 GCP and Cloudflare have been gradually resolving their outages. Our backlogs are caught up, and we are processing traffic at this point. We'll keep monitoring the situation and providing updates. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 20:21:00 -0000 https://status.hookdeck.com/incident/601752#3fa3abe8e6fd671d717b9d3ff8f6482fdd7c51c28b950fe09efc76cba1ffa7e0 GCP and Cloudflare have been gradually resolving their outages. Our backlogs are caught up, and we are processing traffic at this point. We'll keep monitoring the situation and providing updates. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 20:21:00 -0000 https://status.hookdeck.com/incident/601752#3fa3abe8e6fd671d717b9d3ff8f6482fdd7c51c28b950fe09efc76cba1ffa7e0 GCP and Cloudflare have been gradually resolving their outages. Our backlogs are caught up, and we are processing traffic at this point. We'll keep monitoring the situation and providing updates. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 20:21:00 -0000 https://status.hookdeck.com/incident/601752#3fa3abe8e6fd671d717b9d3ff8f6482fdd7c51c28b950fe09efc76cba1ffa7e0 GCP and Cloudflare have been gradually resolving their outages. Our backlogs are caught up, and we are processing traffic at this point. We'll keep monitoring the situation and providing updates. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 20:21:00 -0000 https://status.hookdeck.com/incident/601752#3fa3abe8e6fd671d717b9d3ff8f6482fdd7c51c28b950fe09efc76cba1ffa7e0 GCP and Cloudflare have been gradually resolving their outages. Our backlogs are caught up, and we are processing traffic at this point. We'll keep monitoring the situation and providing updates. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 20:21:00 -0000 https://status.hookdeck.com/incident/601752#3fa3abe8e6fd671d717b9d3ff8f6482fdd7c51c28b950fe09efc76cba1ffa7e0 GCP and Cloudflare have been gradually resolving their outages. Our backlogs are caught up, and we are processing traffic at this point. We'll keep monitoring the situation and providing updates. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 20:21:00 -0000 https://status.hookdeck.com/incident/601752#3fa3abe8e6fd671d717b9d3ff8f6482fdd7c51c28b950fe09efc76cba1ffa7e0 GCP and Cloudflare have been gradually resolving their outages. Our backlogs are caught up, and we are processing traffic at this point. We'll keep monitoring the situation and providing updates. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 20:21:00 -0000 https://status.hookdeck.com/incident/601752#3fa3abe8e6fd671d717b9d3ff8f6482fdd7c51c28b950fe09efc76cba1ffa7e0 GCP and Cloudflare have been gradually resolving their outages. Our backlogs are caught up, and we are processing traffic at this point. We'll keep monitoring the situation and providing updates. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 19:11:00 -0000 https://status.hookdeck.com/incident/601752#bb591894507c1b43dc9ad83b526e0514952cd6369d0f68788c39a03be4b1d46d Traffic has started flowing again, and the majority backlog that we do have is caught up. We're currently investigating to see if we missed any traffic, but our providers are also still experiencing difficulties making the task a little more challenging. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 19:11:00 -0000 https://status.hookdeck.com/incident/601752#bb591894507c1b43dc9ad83b526e0514952cd6369d0f68788c39a03be4b1d46d Traffic has started flowing again, and the majority backlog that we do have is caught up. We're currently investigating to see if we missed any traffic, but our providers are also still experiencing difficulties making the task a little more challenging. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 19:11:00 -0000 https://status.hookdeck.com/incident/601752#bb591894507c1b43dc9ad83b526e0514952cd6369d0f68788c39a03be4b1d46d Traffic has started flowing again, and the majority backlog that we do have is caught up. We're currently investigating to see if we missed any traffic, but our providers are also still experiencing difficulties making the task a little more challenging. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 19:11:00 -0000 https://status.hookdeck.com/incident/601752#bb591894507c1b43dc9ad83b526e0514952cd6369d0f68788c39a03be4b1d46d Traffic has started flowing again, and the majority backlog that we do have is caught up. We're currently investigating to see if we missed any traffic, but our providers are also still experiencing difficulties making the task a little more challenging. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 19:11:00 -0000 https://status.hookdeck.com/incident/601752#bb591894507c1b43dc9ad83b526e0514952cd6369d0f68788c39a03be4b1d46d Traffic has started flowing again, and the majority backlog that we do have is caught up. We're currently investigating to see if we missed any traffic, but our providers are also still experiencing difficulties making the task a little more challenging. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 19:11:00 -0000 https://status.hookdeck.com/incident/601752#bb591894507c1b43dc9ad83b526e0514952cd6369d0f68788c39a03be4b1d46d Traffic has started flowing again, and the majority backlog that we do have is caught up. We're currently investigating to see if we missed any traffic, but our providers are also still experiencing difficulties making the task a little more challenging. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 19:11:00 -0000 https://status.hookdeck.com/incident/601752#bb591894507c1b43dc9ad83b526e0514952cd6369d0f68788c39a03be4b1d46d Traffic has started flowing again, and the majority backlog that we do have is caught up. We're currently investigating to see if we missed any traffic, but our providers are also still experiencing difficulties making the task a little more challenging. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 19:11:00 -0000 https://status.hookdeck.com/incident/601752#bb591894507c1b43dc9ad83b526e0514952cd6369d0f68788c39a03be4b1d46d Traffic has started flowing again, and the majority backlog that we do have is caught up. We're currently investigating to see if we missed any traffic, but our providers are also still experiencing difficulties making the task a little more challenging. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 18:00:00 -0000 https://status.hookdeck.com/incident/601752#56ff5d6dfccd549c3651e32903555cb8a445bde985e7f01cbd2b1db599435fc1 Services are currently degraded. We're investigating issues with service availabiity with our provider GCP. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 18:00:00 -0000 https://status.hookdeck.com/incident/601752#56ff5d6dfccd549c3651e32903555cb8a445bde985e7f01cbd2b1db599435fc1 Services are currently degraded. We're investigating issues with service availabiity with our provider GCP. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 18:00:00 -0000 https://status.hookdeck.com/incident/601752#56ff5d6dfccd549c3651e32903555cb8a445bde985e7f01cbd2b1db599435fc1 Services are currently degraded. We're investigating issues with service availabiity with our provider GCP. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 18:00:00 -0000 https://status.hookdeck.com/incident/601752#56ff5d6dfccd549c3651e32903555cb8a445bde985e7f01cbd2b1db599435fc1 Services are currently degraded. We're investigating issues with service availabiity with our provider GCP. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 18:00:00 -0000 https://status.hookdeck.com/incident/601752#56ff5d6dfccd549c3651e32903555cb8a445bde985e7f01cbd2b1db599435fc1 Services are currently degraded. We're investigating issues with service availabiity with our provider GCP. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 18:00:00 -0000 https://status.hookdeck.com/incident/601752#56ff5d6dfccd549c3651e32903555cb8a445bde985e7f01cbd2b1db599435fc1 Services are currently degraded. We're investigating issues with service availabiity with our provider GCP. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 18:00:00 -0000 https://status.hookdeck.com/incident/601752#56ff5d6dfccd549c3651e32903555cb8a445bde985e7f01cbd2b1db599435fc1 Services are currently degraded. We're investigating issues with service availabiity with our provider GCP. https://status.hookdeck.com/incident/601752 Thu, 12 Jun 2025 18:00:00 -0000 https://status.hookdeck.com/incident/601752#56ff5d6dfccd549c3651e32903555cb8a445bde985e7f01cbd2b1db599435fc1 Services are currently degraded. We're investigating issues with service availabiity with our provider GCP. https://status.hookdeck.com/incident/595621 Wed, 04 Jun 2025 00:00:00 -0000 https://status.hookdeck.com/incident/595621#b49e284bd362b4c15dbce34154a3b813ee0ff0139a9131f53812eee1fe9d851d The database issue that was causing latency spikes in delivery is now resolved. We will follow up with a full post mortem. https://status.hookdeck.com/incident/595621 Wed, 04 Jun 2025 00:00:00 -0000 https://status.hookdeck.com/incident/595621#b49e284bd362b4c15dbce34154a3b813ee0ff0139a9131f53812eee1fe9d851d The database issue that was causing latency spikes in delivery is now resolved. We will follow up with a full post mortem. https://status.hookdeck.com/incident/595621 Tue, 03 Jun 2025 12:28:00 -0000 https://status.hookdeck.com/incident/595621#7baee43edebf1a673117013fb160b5c4554445370a91877a1a6b95d8099b33a3 At this point we believe we identified the cause of the performance degradation in our databases. Our provider is working on building a new instance with the correct setting applied. This will likely take another few hours as it's a large database. We'll keep providing updates as we get new information. We'll also provide a full post mortem once this incident is resolved. https://status.hookdeck.com/incident/595621 Tue, 03 Jun 2025 12:28:00 -0000 https://status.hookdeck.com/incident/595621#7baee43edebf1a673117013fb160b5c4554445370a91877a1a6b95d8099b33a3 At this point we believe we identified the cause of the performance degradation in our databases. Our provider is working on building a new instance with the correct setting applied. This will likely take another few hours as it's a large database. We'll keep providing updates as we get new information. We'll also provide a full post mortem once this incident is resolved. https://status.hookdeck.com/incident/595621 Mon, 02 Jun 2025 11:05:00 -0000 https://status.hookdeck.com/incident/595621#01cd48d7fde77a1d1a8ae2f1cc9009709ff58b0c8d8c683854e77eb56bc1299b We're experiencing sporadic delay in delivering events. We tracked the root cause to an issue with a database upgrade. We are working on a fix. https://status.hookdeck.com/incident/595621 Mon, 02 Jun 2025 11:05:00 -0000 https://status.hookdeck.com/incident/595621#01cd48d7fde77a1d1a8ae2f1cc9009709ff58b0c8d8c683854e77eb56bc1299b We're experiencing sporadic delay in delivering events. We tracked the root cause to an issue with a database upgrade. We are working on a fix. https://status.hookdeck.com/incident/566295 Thu, 15 May 2025 18:29:00 -0000 https://status.hookdeck.com/incident/566295#530e416a489a874091515bcf50b6ed01f9003cc6bfcf24fa35aa523a3359d91e The issue has been resolved, no event delivery was impacted and all events will show in the dashboard https://status.hookdeck.com/incident/566295 Thu, 15 May 2025 18:14:00 -0000 https://status.hookdeck.com/incident/566295#cd7ca073a35cfb9e2c061f8003f5fe3e8240461d3399bdcea32ac4e874413f94 We're experiencing issues with events not loading properly in the dashboard. Events are still being delivered, this is only affecting the dashboard. We are investigating and will update as soon as we know more. https://status.hookdeck.com/incident/558269 Wed, 07 May 2025 19:27:00 -0000 https://status.hookdeck.com/incident/558269#15df633912deb989676d4db1bc9a0b24dd814e09a666ef7dc1f44584e5428f51 The issue has been resolved data show display correctly in the dashboard and API https://status.hookdeck.com/incident/558269 Wed, 07 May 2025 19:27:00 -0000 https://status.hookdeck.com/incident/558269#15df633912deb989676d4db1bc9a0b24dd814e09a666ef7dc1f44584e5428f51 The issue has been resolved data show display correctly in the dashboard and API https://status.hookdeck.com/incident/558269 Wed, 07 May 2025 19:25:00 -0000 https://status.hookdeck.com/incident/558269#095bbb89df3a2bc0fe95de022aa1a52c3eab519ec133902998a9cb92341b4b95 We're investigating an issue causing missing data in the dashboard and API. Requests and events are still being processed and delivered normally. https://status.hookdeck.com/incident/558269 Wed, 07 May 2025 19:25:00 -0000 https://status.hookdeck.com/incident/558269#095bbb89df3a2bc0fe95de022aa1a52c3eab519ec133902998a9cb92341b4b95 We're investigating an issue causing missing data in the dashboard and API. Requests and events are still being processed and delivered normally. https://status.hookdeck.com/incident/539393 Thu, 03 Apr 2025 15:02:00 -0000 https://status.hookdeck.com/incident/539393#3f7461bf21aca96f5539e3e76c457c681c4e9792c432aa3287c87b0092a799aa The issue has been resolved. Failed events can be manually retried but won't be automatically re-delivered since the CLI is not meant for production usage, and we don't want to risk causing request overload on individual developers' machines & connected clients. https://status.hookdeck.com/incident/539393 Thu, 03 Apr 2025 14:59:00 -0000 https://status.hookdeck.com/incident/539393#3be269621b67504a58771e204b091183d8e47b8b5d185615d529ef33e7d35166 We are aware of an issue causing CLI events to either be stuck as Pending or marked with the error "CLI_UNAVAILABLE". We've identified the root cause and publishing a fix. https://status.hookdeck.com/incident/538855 Wed, 02 Apr 2025 18:01:00 -0000 https://status.hookdeck.com/incident/538855#9a7e4ff94ffe64265b84fb825e5759be589dfbc3133349d6582170a3a58979d6 The issue has been resolved and the dashboard now loads properly https://status.hookdeck.com/incident/538855 Wed, 02 Apr 2025 17:59:00 -0000 https://status.hookdeck.com/incident/538855#df823027404c458ec1c54ddc11f6cf63e4384431815d4c46c4b689a794b9dffa We're investigating an issue preventing the dashboard from loading. https://status.hookdeck.com/incident/521920 Mon, 03 Mar 2025 14:38:00 -0000 https://status.hookdeck.com/incident/521920#c8a7e4c3fa225358e340332f862641426194c9a2dc4619b5df1f7b5350d31fac API and Dashboard data is fully caught up, all systems are back to normal https://status.hookdeck.com/incident/521920 Mon, 03 Mar 2025 14:38:00 -0000 https://status.hookdeck.com/incident/521920#c8a7e4c3fa225358e340332f862641426194c9a2dc4619b5df1f7b5350d31fac API and Dashboard data is fully caught up, all systems are back to normal https://status.hookdeck.com/incident/521920 Mon, 03 Mar 2025 14:38:00 -0000 https://status.hookdeck.com/incident/521920#c8a7e4c3fa225358e340332f862641426194c9a2dc4619b5df1f7b5350d31fac API and Dashboard data is fully caught up, all systems are back to normal https://status.hookdeck.com/incident/521920 Mon, 03 Mar 2025 13:42:00 -0000 https://status.hookdeck.com/incident/521920#780babe2321ace911f14939f4098fbd85df687ee5ae852fbef37668f21893bbb All services are restored, but events and requests data in the dashboard and API are lagging by a few minutes. https://status.hookdeck.com/incident/521920 Mon, 03 Mar 2025 13:42:00 -0000 https://status.hookdeck.com/incident/521920#780babe2321ace911f14939f4098fbd85df687ee5ae852fbef37668f21893bbb All services are restored, but events and requests data in the dashboard and API are lagging by a few minutes. https://status.hookdeck.com/incident/521920 Mon, 03 Mar 2025 13:42:00 -0000 https://status.hookdeck.com/incident/521920#780babe2321ace911f14939f4098fbd85df687ee5ae852fbef37668f21893bbb All services are restored, but events and requests data in the dashboard and API are lagging by a few minutes. https://status.hookdeck.com/incident/521920 Mon, 03 Mar 2025 13:34:00 -0000 https://status.hookdeck.com/incident/521920#04803c94d668925589533c37ca31e544fd823eb42b590ac461e560490ac15e1d Delivery latency was incorrectly identified as being impacted, we are investing issues with the /events and /requests API along with the events and requests UI in the dashboard https://status.hookdeck.com/incident/521920 Mon, 03 Mar 2025 13:34:00 -0000 https://status.hookdeck.com/incident/521920#04803c94d668925589533c37ca31e544fd823eb42b590ac461e560490ac15e1d Delivery latency was incorrectly identified as being impacted, we are investing issues with the /events and /requests API along with the events and requests UI in the dashboard https://status.hookdeck.com/incident/521920 Mon, 03 Mar 2025 13:34:00 -0000 https://status.hookdeck.com/incident/521920#04803c94d668925589533c37ca31e544fd823eb42b590ac461e560490ac15e1d Delivery latency was incorrectly identified as being impacted, we are investing issues with the /events and /requests API along with the events and requests UI in the dashboard https://status.hookdeck.com/incident/521920 Mon, 03 Mar 2025 13:26:00 -0000 https://status.hookdeck.com/incident/521920#a58aae0eee6f8aed831ff06923ca34f6a70edb0c5ae14d4b8cc0e14d1dbd2f2e We're investigating a delivery latency issue as well as some API errors returning 401's and 500's error codes https://status.hookdeck.com/incident/521920 Mon, 03 Mar 2025 13:26:00 -0000 https://status.hookdeck.com/incident/521920#a58aae0eee6f8aed831ff06923ca34f6a70edb0c5ae14d4b8cc0e14d1dbd2f2e We're investigating a delivery latency issue as well as some API errors returning 401's and 500's error codes https://status.hookdeck.com/incident/521920 Mon, 03 Mar 2025 13:26:00 -0000 https://status.hookdeck.com/incident/521920#a58aae0eee6f8aed831ff06923ca34f6a70edb0c5ae14d4b8cc0e14d1dbd2f2e We're investigating a delivery latency issue as well as some API errors returning 401's and 500's error codes https://status.hookdeck.com/incident/501809 Sat, 25 Jan 2025 20:51:00 -0000 https://status.hookdeck.com/incident/501809#9398f60f99e0d3eed1a7ae2bbc24e1c9e95634dd0a61b6f171d62ab787c7f2af Service has been restored https://status.hookdeck.com/incident/501809 Sat, 25 Jan 2025 20:03:00 -0000 https://status.hookdeck.com/incident/501809#90fe56f036c3e6b867b0ae2cce6e7785c3d258b77b66446e5ee00f528dd233bd On January 25, 2025, between 19:03-19:51 UTC, we experienced elevated delivery latency peaking at 45 seconds, with intermittent fluctuations throughout the incident window. The service has since returned to normal operation. https://status.hookdeck.com/incident/472517 Thu, 05 Dec 2024 00:14:00 -0000 https://status.hookdeck.com/incident/472517#80f8030acf36488e0445567ff79ef7b1079cf31a63ce59a00979ed9e41228850 We're investigating an issue causing a delivery backlog. At the moment, the delay can be up to a few minutes depending on your destination. https://status.hookdeck.com/incident/430141 Mon, 16 Sep 2024 20:11:00 -0000 https://status.hookdeck.com/incident/430141#a39e2cd1e44393d60f028bc5da6ae1a95935befc034c78b2d2fab7cebfb1a2cc We are now fully caught up on the backlog. Delivery is back to normal. Thank you for your patience. https://status.hookdeck.com/incident/430141 Mon, 16 Sep 2024 19:51:00 -0000 https://status.hookdeck.com/incident/430141#90d5a5a4cc0d94b04ef1d4123974a222c47f1db01c876c86306e4ad51d1ce214 We've rolled out a fix, but accumulated a backlog. We're working on processing it. https://status.hookdeck.com/incident/430141 Mon, 16 Sep 2024 19:26:00 -0000 https://status.hookdeck.com/incident/430141#1bf3820d87451246b48063ce905d58d0ab70dbb8029b0c532a38c5a9db0543e5 Event delivery is down. We are working on a fix, it should only take a few more minutes. We will update when delivery is back to normal. https://status.hookdeck.com/incident/430141 Mon, 16 Sep 2024 19:05:00 -0000 https://status.hookdeck.com/incident/430141#4dbc1268b54b85592cf0337fc30235313acf746df3ea85c4aba4786bbe268ef1 We are experiencing degraded performance with ingestion. We are investigating. https://status.hookdeck.com/incident/424619 Thu, 05 Sep 2024 12:01:00 -0000 https://status.hookdeck.com/incident/424619#b832e0a2f555948523869cd4bcc448dc99c76a21b0054d4720df28f1607184af The maintenance update is now complete. The Dashboard Requests list now loads up to 4X faster. https://status.hookdeck.com/incident/424619 Thu, 05 Sep 2024 10:52:00 -0000 https://status.hookdeck.com/incident/424619#baba9ba16e6abd29492b1158452fb867e22ddcd0790351a79be30dc986fdc809 We are currently running an update to improve the performance of the Requests list in the Dashboard and related API. Requests in the Dashboard will not be visible while the update is running. It should take about 2 hours. THis does not impact the processing and delivery of your events. https://status.hookdeck.com/incident/420779 Fri, 30 Aug 2024 14:46:00 -0000 https://status.hookdeck.com/incident/420779#a007541d9cba02995b725be47080552e964712ce7849ea0f5e3852186e1cddc4 All services are now fully restored, and the backlog has been processed. The system is back to normal operator conditions. We are continuing to monitor. We will provide a full post-mortem and have started implementation remediation to prevent similar issues from occurring again. More communication to follow. Thank you for your patience https://status.hookdeck.com/incident/420779 Fri, 30 Aug 2024 14:46:00 -0000 https://status.hookdeck.com/incident/420779#a007541d9cba02995b725be47080552e964712ce7849ea0f5e3852186e1cddc4 All services are now fully restored, and the backlog has been processed. The system is back to normal operator conditions. We are continuing to monitor. We will provide a full post-mortem and have started implementation remediation to prevent similar issues from occurring again. More communication to follow. Thank you for your patience https://status.hookdeck.com/incident/420779 Fri, 30 Aug 2024 14:46:00 -0000 https://status.hookdeck.com/incident/420779#a007541d9cba02995b725be47080552e964712ce7849ea0f5e3852186e1cddc4 All services are now fully restored, and the backlog has been processed. The system is back to normal operator conditions. We are continuing to monitor. We will provide a full post-mortem and have started implementation remediation to prevent similar issues from occurring again. More communication to follow. Thank you for your patience https://status.hookdeck.com/incident/420779 Fri, 30 Aug 2024 14:46:00 -0000 https://status.hookdeck.com/incident/420779#a007541d9cba02995b725be47080552e964712ce7849ea0f5e3852186e1cddc4 All services are now fully restored, and the backlog has been processed. The system is back to normal operator conditions. We are continuing to monitor. We will provide a full post-mortem and have started implementation remediation to prevent similar issues from occurring again. More communication to follow. Thank you for your patience https://status.hookdeck.com/incident/419515 Fri, 30 Aug 2024 04:41:00 -0000 https://status.hookdeck.com/incident/419515#5a092a81947dc47464ba589ddc3aaf37080c64fab595e3bb5fbc6e34887a6060 The backfill of events is now complete. The Dashboard now displays up to 30 days of data depending on your plan's retention period. https://status.hookdeck.com/incident/419515 Fri, 30 Aug 2024 04:41:00 -0000 https://status.hookdeck.com/incident/419515#5a092a81947dc47464ba589ddc3aaf37080c64fab595e3bb5fbc6e34887a6060 The backfill of events is now complete. The Dashboard now displays up to 30 days of data depending on your plan's retention period. https://status.hookdeck.com/incident/420779 Fri, 30 Aug 2024 02:29:00 -0000 https://status.hookdeck.com/incident/420779#f3a009485d8f9ff180c1951a9c80c025816afac483b06b8ac25b4aa961f8e9c3 The interruption in delivery has been restored and delivery is back to normal. Also the backfilling of the 30 days of events in the Dashboard is near completion. We will update once it's finished. https://status.hookdeck.com/incident/420779 Fri, 30 Aug 2024 02:29:00 -0000 https://status.hookdeck.com/incident/420779#f3a009485d8f9ff180c1951a9c80c025816afac483b06b8ac25b4aa961f8e9c3 The interruption in delivery has been restored and delivery is back to normal. Also the backfilling of the 30 days of events in the Dashboard is near completion. We will update once it's finished. https://status.hookdeck.com/incident/420779 Fri, 30 Aug 2024 02:29:00 -0000 https://status.hookdeck.com/incident/420779#f3a009485d8f9ff180c1951a9c80c025816afac483b06b8ac25b4aa961f8e9c3 The interruption in delivery has been restored and delivery is back to normal. Also the backfilling of the 30 days of events in the Dashboard is near completion. We will update once it's finished. https://status.hookdeck.com/incident/420779 Fri, 30 Aug 2024 02:29:00 -0000 https://status.hookdeck.com/incident/420779#f3a009485d8f9ff180c1951a9c80c025816afac483b06b8ac25b4aa961f8e9c3 The interruption in delivery has been restored and delivery is back to normal. Also the backfilling of the 30 days of events in the Dashboard is near completion. We will update once it's finished. https://status.hookdeck.com/incident/420779 Fri, 30 Aug 2024 01:11:00 -0000 https://status.hookdeck.com/incident/420779#1ec92f8b4f457e8e36c2233866f83ee2dd22670654bfe253945489862bf6f438 We are experiencing an interruption in delivery currently. We are looking into it and will provide another update shortly. https://status.hookdeck.com/incident/420779 Fri, 30 Aug 2024 01:11:00 -0000 https://status.hookdeck.com/incident/420779#1ec92f8b4f457e8e36c2233866f83ee2dd22670654bfe253945489862bf6f438 We are experiencing an interruption in delivery currently. We are looking into it and will provide another update shortly. https://status.hookdeck.com/incident/420779 Fri, 30 Aug 2024 01:11:00 -0000 https://status.hookdeck.com/incident/420779#1ec92f8b4f457e8e36c2233866f83ee2dd22670654bfe253945489862bf6f438 We are experiencing an interruption in delivery currently. We are looking into it and will provide another update shortly. https://status.hookdeck.com/incident/420779 Fri, 30 Aug 2024 01:11:00 -0000 https://status.hookdeck.com/incident/420779#1ec92f8b4f457e8e36c2233866f83ee2dd22670654bfe253945489862bf6f438 We are experiencing an interruption in delivery currently. We are looking into it and will provide another update shortly. https://status.hookdeck.com/incident/420779 Thu, 29 Aug 2024 14:59:00 -0000 https://status.hookdeck.com/incident/420779#a88fc5400520570698d1c90b3c4dba927194d02a1bdd3a1af60162337313cd9e The delay in ingestion has been cleared. Ingestion is now back to normal. All systems are back online. There is still a significant backlog of messages to process. We are scaling services now to handle the backlog as fast as possible. We will update again shortly. https://status.hookdeck.com/incident/420779 Thu, 29 Aug 2024 14:59:00 -0000 https://status.hookdeck.com/incident/420779#a88fc5400520570698d1c90b3c4dba927194d02a1bdd3a1af60162337313cd9e The delay in ingestion has been cleared. Ingestion is now back to normal. All systems are back online. There is still a significant backlog of messages to process. We are scaling services now to handle the backlog as fast as possible. We will update again shortly. https://status.hookdeck.com/incident/420779 Thu, 29 Aug 2024 14:59:00 -0000 https://status.hookdeck.com/incident/420779#a88fc5400520570698d1c90b3c4dba927194d02a1bdd3a1af60162337313cd9e The delay in ingestion has been cleared. Ingestion is now back to normal. All systems are back online. There is still a significant backlog of messages to process. We are scaling services now to handle the backlog as fast as possible. We will update again shortly. https://status.hookdeck.com/incident/420779 Thu, 29 Aug 2024 14:59:00 -0000 https://status.hookdeck.com/incident/420779#a88fc5400520570698d1c90b3c4dba927194d02a1bdd3a1af60162337313cd9e The delay in ingestion has been cleared. Ingestion is now back to normal. All systems are back online. There is still a significant backlog of messages to process. We are scaling services now to handle the backlog as fast as possible. We will update again shortly. https://status.hookdeck.com/incident/420779 Thu, 29 Aug 2024 14:14:00 -0000 https://status.hookdeck.com/incident/420779#cace3780be0b85eb63ba8b2239a829370e93aa251151a0a5b1cd4f654dbc96c8 The current ingestion is experiencing up to a 15 minute delay. Regarding the events page in the dashboard, reloading of the remaining 27 days for organizations with 30 days of retention is now 35% complete. https://status.hookdeck.com/incident/420779 Thu, 29 Aug 2024 14:14:00 -0000 https://status.hookdeck.com/incident/420779#cace3780be0b85eb63ba8b2239a829370e93aa251151a0a5b1cd4f654dbc96c8 The current ingestion is experiencing up to a 15 minute delay. Regarding the events page in the dashboard, reloading of the remaining 27 days for organizations with 30 days of retention is now 35% complete. https://status.hookdeck.com/incident/420779 Thu, 29 Aug 2024 14:14:00 -0000 https://status.hookdeck.com/incident/420779#cace3780be0b85eb63ba8b2239a829370e93aa251151a0a5b1cd4f654dbc96c8 The current ingestion is experiencing up to a 15 minute delay. Regarding the events page in the dashboard, reloading of the remaining 27 days for organizations with 30 days of retention is now 35% complete. https://status.hookdeck.com/incident/420779 Thu, 29 Aug 2024 14:14:00 -0000 https://status.hookdeck.com/incident/420779#cace3780be0b85eb63ba8b2239a829370e93aa251151a0a5b1cd4f654dbc96c8 The current ingestion is experiencing up to a 15 minute delay. Regarding the events page in the dashboard, reloading of the remaining 27 days for organizations with 30 days of retention is now 35% complete. https://status.hookdeck.com/incident/420779 Thu, 29 Aug 2024 07:17:00 -0000 https://status.hookdeck.com/incident/420779#1cb6624fc493c9615d16bab6fc588c3da6e4d039d707c8e3ab0f9912ce1b0fbc Regarding the events page in the dashboard, repopulating of data for the last 3 days is complete. The remaining 27 days for organizations with 30 days of retention is now repopulating. https://status.hookdeck.com/incident/420779 Thu, 29 Aug 2024 07:17:00 -0000 https://status.hookdeck.com/incident/420779#1cb6624fc493c9615d16bab6fc588c3da6e4d039d707c8e3ab0f9912ce1b0fbc Regarding the events page in the dashboard, repopulating of data for the last 3 days is complete. The remaining 27 days for organizations with 30 days of retention is now repopulating. https://status.hookdeck.com/incident/420779 Thu, 29 Aug 2024 07:17:00 -0000 https://status.hookdeck.com/incident/420779#1cb6624fc493c9615d16bab6fc588c3da6e4d039d707c8e3ab0f9912ce1b0fbc Regarding the events page in the dashboard, repopulating of data for the last 3 days is complete. The remaining 27 days for organizations with 30 days of retention is now repopulating. https://status.hookdeck.com/incident/420779 Thu, 29 Aug 2024 07:17:00 -0000 https://status.hookdeck.com/incident/420779#1cb6624fc493c9615d16bab6fc588c3da6e4d039d707c8e3ab0f9912ce1b0fbc Regarding the events page in the dashboard, repopulating of data for the last 3 days is complete. The remaining 27 days for organizations with 30 days of retention is now repopulating. https://status.hookdeck.com/incident/420779 Wed, 28 Aug 2024 21:29:00 -0000 https://status.hookdeck.com/incident/420779#1fb80f114e54587aecb2607fb7b77c93bbc69ef6e741f2e33d00994cb5a56ee7 Requests and events continue to be processed as normal within a few seconds, and we've restored all platform features, such as retries. Some historical events may not have been processed yet, and only a few projects, particularly those that routinely exceed their project throughput, should be missing events. Those are still being processed. Lastly, the events page in the dashboard is re-populating the data for the last 3 days. As of now, 1 day out of 3 has fully populated. Once the last 3 days is completely restored, we'll restore the remaining 27 days for organizations with 30 days of retention. https://status.hookdeck.com/incident/420779 Wed, 28 Aug 2024 21:29:00 -0000 https://status.hookdeck.com/incident/420779#1fb80f114e54587aecb2607fb7b77c93bbc69ef6e741f2e33d00994cb5a56ee7 Requests and events continue to be processed as normal within a few seconds, and we've restored all platform features, such as retries. Some historical events may not have been processed yet, and only a few projects, particularly those that routinely exceed their project throughput, should be missing events. Those are still being processed. Lastly, the events page in the dashboard is re-populating the data for the last 3 days. As of now, 1 day out of 3 has fully populated. Once the last 3 days is completely restored, we'll restore the remaining 27 days for organizations with 30 days of retention. https://status.hookdeck.com/incident/420779 Wed, 28 Aug 2024 21:29:00 -0000 https://status.hookdeck.com/incident/420779#1fb80f114e54587aecb2607fb7b77c93bbc69ef6e741f2e33d00994cb5a56ee7 Requests and events continue to be processed as normal within a few seconds, and we've restored all platform features, such as retries. Some historical events may not have been processed yet, and only a few projects, particularly those that routinely exceed their project throughput, should be missing events. Those are still being processed. Lastly, the events page in the dashboard is re-populating the data for the last 3 days. As of now, 1 day out of 3 has fully populated. Once the last 3 days is completely restored, we'll restore the remaining 27 days for organizations with 30 days of retention. https://status.hookdeck.com/incident/420779 Wed, 28 Aug 2024 21:29:00 -0000 https://status.hookdeck.com/incident/420779#1fb80f114e54587aecb2607fb7b77c93bbc69ef6e741f2e33d00994cb5a56ee7 Requests and events continue to be processed as normal within a few seconds, and we've restored all platform features, such as retries. Some historical events may not have been processed yet, and only a few projects, particularly those that routinely exceed their project throughput, should be missing events. Those are still being processed. Lastly, the events page in the dashboard is re-populating the data for the last 3 days. As of now, 1 day out of 3 has fully populated. Once the last 3 days is completely restored, we'll restore the remaining 27 days for organizations with 30 days of retention. https://status.hookdeck.com/incident/420779 Wed, 28 Aug 2024 20:29:00 -0000 https://status.hookdeck.com/incident/420779#1db30c08a0d6a70dc84e557787f5fa5c348f2bbe6b16d5eaa4a748d108dc9e63 We've managed to stabilize a subset of the traffic. Most sources should process requests within a few seconds, but we have roughly 10% of sources that are still impacted by the delay. We're still working on the issue. https://status.hookdeck.com/incident/420779 Wed, 28 Aug 2024 20:29:00 -0000 https://status.hookdeck.com/incident/420779#1db30c08a0d6a70dc84e557787f5fa5c348f2bbe6b16d5eaa4a748d108dc9e63 We've managed to stabilize a subset of the traffic. Most sources should process requests within a few seconds, but we have roughly 10% of sources that are still impacted by the delay. We're still working on the issue. https://status.hookdeck.com/incident/420779 Wed, 28 Aug 2024 20:29:00 -0000 https://status.hookdeck.com/incident/420779#1db30c08a0d6a70dc84e557787f5fa5c348f2bbe6b16d5eaa4a748d108dc9e63 We've managed to stabilize a subset of the traffic. Most sources should process requests within a few seconds, but we have roughly 10% of sources that are still impacted by the delay. We're still working on the issue. https://status.hookdeck.com/incident/420779 Wed, 28 Aug 2024 14:16:00 -0000 https://status.hookdeck.com/incident/420779#f0dc6fa3b0b347e6ce2ad3482b1b9740447427578fe20a5f05e8cacf879b80c5 We're experiencing a delay in processing requests and delivering events. No data has been lost. https://status.hookdeck.com/incident/420779 Wed, 28 Aug 2024 14:16:00 -0000 https://status.hookdeck.com/incident/420779#f0dc6fa3b0b347e6ce2ad3482b1b9740447427578fe20a5f05e8cacf879b80c5 We're experiencing a delay in processing requests and delivering events. No data has been lost. https://status.hookdeck.com/incident/419515 Tue, 27 Aug 2024 20:30:00 -0000 https://status.hookdeck.com/incident/419515#aa2d885538a72d55119dd54c701a4224b1fed40ea31fd3e56273d7561c5ebb99 All events since 8:30 UTC are shown correctly and with the right status. The backfill is still in progress and is estimated to take about 8-12 hours. https://status.hookdeck.com/incident/419515 Tue, 27 Aug 2024 20:30:00 -0000 https://status.hookdeck.com/incident/419515#aa2d885538a72d55119dd54c701a4224b1fed40ea31fd3e56273d7561c5ebb99 All events since 8:30 UTC are shown correctly and with the right status. The backfill is still in progress and is estimated to take about 8-12 hours. https://status.hookdeck.com/incident/419515 Mon, 26 Aug 2024 21:30:00 -0000 https://status.hookdeck.com/incident/419515#fc115408b2dad5f19654886b5b2f159bf64f8458b7d356875f28eefe4b1f5fed A fix has been implemented and newly updated events should show the appropriate status. We're now backfilling missing data from Friday night until now. https://status.hookdeck.com/incident/419515 Mon, 26 Aug 2024 21:30:00 -0000 https://status.hookdeck.com/incident/419515#fc115408b2dad5f19654886b5b2f159bf64f8458b7d356875f28eefe4b1f5fed A fix has been implemented and newly updated events should show the appropriate status. We're now backfilling missing data from Friday night until now. https://status.hookdeck.com/incident/419515 Sat, 24 Aug 2024 02:00:00 -0000 https://status.hookdeck.com/incident/419515#954d40aa6250513ffdb1cfaff5541c2956b03f0282ddcb0a54eee48f6217c184 We're experiencing an issue with our Clickhouse pipeline which is causing roughly 60% of all events to not show in the dashboard or to show an incorrect "Pending" state. Clicking the event should show the correct status. This issue impacts the /events API used to populate the dashboard. We have identified the root issue and are working on a fix. No data has been lost and this does not impact the event delivery. https://status.hookdeck.com/incident/419515 Sat, 24 Aug 2024 02:00:00 -0000 https://status.hookdeck.com/incident/419515#954d40aa6250513ffdb1cfaff5541c2956b03f0282ddcb0a54eee48f6217c184 We're experiencing an issue with our Clickhouse pipeline which is causing roughly 60% of all events to not show in the dashboard or to show an incorrect "Pending" state. Clicking the event should show the correct status. This issue impacts the /events API used to populate the dashboard. We have identified the root issue and are working on a fix. No data has been lost and this does not impact the event delivery. https://status.hookdeck.com/incident/417196 Wed, 21 Aug 2024 19:08:00 -0000 https://status.hookdeck.com/incident/417196#e3c442b7f9da2d685ddc6682cfe3b4487b407d57dac54ab490c54937e17674a4 We have identified the issue, dashboard latency is now back to normal https://status.hookdeck.com/incident/417196 Wed, 21 Aug 2024 18:39:00 -0000 https://status.hookdeck.com/incident/417196#dd1a4d0f0fe26e6872b2e6b73e55f7fca4b3c308583af753029f48808ee4251c The data in the dashboard is currently experiencing latency. This happened after one of our database servers crashed, and a replica took over. It affected event delivery for about 2-3 minutes, but event delivery is back to normal now. https://status.hookdeck.com/incident/409878 Wed, 07 Aug 2024 03:55:00 -0000 https://status.hookdeck.com/incident/409878#184cd15cf4197dcae740cb1eab95984c8ea9da16a0b2090489c8db3511a48e6c We have identified the issue that was causing delivery latency and applied a fix. Delivery latency is back to its normal range. https://status.hookdeck.com/incident/409878 Wed, 07 Aug 2024 03:55:00 -0000 https://status.hookdeck.com/incident/409878#184cd15cf4197dcae740cb1eab95984c8ea9da16a0b2090489c8db3511a48e6c We have identified the issue that was causing delivery latency and applied a fix. Delivery latency is back to its normal range. https://status.hookdeck.com/incident/409878 Tue, 06 Aug 2024 16:05:00 -0000 https://status.hookdeck.com/incident/409878#dc7bf0f4b8b48889554c6cf030aa0c675a13d1f8464f0d011b5df2f61c780058 We are currently experiencing increased latency in delivery. Our team is working on resolving the issue. https://status.hookdeck.com/incident/409878 Tue, 06 Aug 2024 16:05:00 -0000 https://status.hookdeck.com/incident/409878#dc7bf0f4b8b48889554c6cf030aa0c675a13d1f8464f0d011b5df2f61c780058 We are currently experiencing increased latency in delivery. Our team is working on resolving the issue. https://status.hookdeck.com/incident/349146 Tue, 02 Apr 2024 19:07:00 -0000 https://status.hookdeck.com/incident/349146#f7283845f5d324c0c13bd460926956affb1b436b66dffec6eeae05027f35b9be We have identified the cause of the lag and have implemented a fix. We are monitoring to ensure proper operation. https://status.hookdeck.com/incident/349146 Mon, 01 Apr 2024 15:00:00 -0000 https://status.hookdeck.com/incident/349146#03ce93a047b92a2578e8bb12c50f161bfb54c3377872fbbd4edf12e7097c32a6 The dahsboard is currently experiencing lag for some transactions. Event delivery us unaffected, all events are being delivered normally. https://status.hookdeck.com/incident/347518 Thu, 28 Mar 2024 20:16:00 +0000 https://status.hookdeck.com/incident/347518#de7c107f9eb08b3fabbdc60617b4983dce8503c83ecffc60e3a012e41799f3e6 Maintenance completed https://status.hookdeck.com/incident/347518 Thu, 28 Mar 2024 19:16:00 -0000 https://status.hookdeck.com/incident/347518#aa6b97d3b7e2cc500569218e84eeb8413db442cbc395f9adaa423e883f97cd60 The dashboard is current disabled while we perform planned migration work https://status.hookdeck.com/incident/342436 Mon, 18 Mar 2024 13:45:00 -0000 https://status.hookdeck.com/incident/342436#589f704cc25cf8019e716dc8ee23a0eac2d5aa37ba1906e846d334e5498ab4a6 We have identified the root cause of the issue. Ingestion rate is back to normal. https://status.hookdeck.com/incident/342436 Mon, 18 Mar 2024 12:05:00 -0000 https://status.hookdeck.com/incident/342436#9cc9f9122dfde3aa3dc8f50a2d3b74c2a455bac57877764bf01b27aa7f139296 Ingestion performance is currently degraded. We're investigating. Requests are still ingested and delivered but are experiencing high latency https://status.hookdeck.com/incident/331033 Fri, 23 Feb 2024 16:32:00 -0000 https://status.hookdeck.com/incident/331033#7413359edc7faa46e34c571e32a707ebcf201d63750bb1249ae15264ac574198 We have identified the root cause of the replication lag. The dashboard data is now caught up and up to date. https://status.hookdeck.com/incident/331033 Fri, 23 Feb 2024 15:15:00 -0000 https://status.hookdeck.com/incident/331033#f50e0815ea43af5acad7eb74511c784e4ea0ef04d140c63028efa3c86a839c29 Dashboard data is currently delayed across Hookdeck. This was caused by a Kafka cluster upgrade that should not have had any impact. We're working with our provider to fix the issue, and will update when we have more information. https://status.hookdeck.com/incident/326333 Tue, 13 Feb 2024 18:44:00 -0000 https://status.hookdeck.com/incident/326333#b8445dfa594915b38d3f7a988477e08efce5990878703832e7ada6dc0dea7d7c The issue has been resolved, and the backpressure cleared. https://status.hookdeck.com/incident/326333 Tue, 13 Feb 2024 16:48:00 -0000 https://status.hookdeck.com/incident/326333#d74d8a19e487092abf2a5ff4c8e7a0c4adf1f1e3a6e5f9351eb084d9d2029bb1 Some connections are seeing delayed delivery of events to destination URLs. Events are still being ingested, stored, and queued. But there may be a delay in delivery. https://status.hookdeck.com/incident/322626 Mon, 05 Feb 2024 17:45:00 -0000 https://status.hookdeck.com/incident/322626#7aa76d7e21e07b637a305a7594555f1c2fb906f34a4d75c162162406b1b18511 The dashboard lag issue is resolved, the maintenance update has been applied successfully. https://status.hookdeck.com/incident/322626 Mon, 05 Feb 2024 17:03:00 -0000 https://status.hookdeck.com/incident/322626#fd2be8e66ee4b77bffdd920872f079d2719323c85a1e9c6e40c281b9d0f9a4cd Dashboard data is currently experiencing delays in updating data. A maintenance applied by our provider is causing this delay. Ingestion and delivery are not impacted. https://status.hookdeck.com/incident/316818 Wed, 24 Jan 2024 03:08:00 -0000 https://status.hookdeck.com/incident/316818#39682d654f4d10215f6916f2f134e46d8de3c34d803fcb3ddd14188d85a84a54 Resolved: Issue causing the dashboard not to display recent events and requests https://status.hookdeck.com/incident/316818 Tue, 23 Jan 2024 22:47:00 -0000 https://status.hookdeck.com/incident/316818#46941eb051e1da9d95ce63d1576ca88bedaf0a9f7eefe563dbdeb58fe9556320 The issue causing the dashboard not to display recent events and requests has been resolved, and we’re fully caught up on the backlog. https://status.hookdeck.com/incident/291243 Sat, 18 Nov 2023 17:40:00 -0000 https://status.hookdeck.com/incident/291243#cf8eac75dace67a2a76923ad7cadabe727d3ff6058b7a78b824e71fcab0d8e54 On November 18th, 2023, we experienced two instances of degraded performance. The first one started at 12:13:00 UTC and ended at 12:40:00 UTC; the second began at 19:07:00 UTC and ended at 20:21:00 UTC. For full details, see https://hookdeck.notion.site/2023-11-18-Degraded-ingestion-performance-7add915071404f2da991c6ffd4607d29 https://status.hookdeck.com/incident/293279 Fri, 03 Nov 2023 13:09:00 -0000 https://status.hookdeck.com/incident/293279#7759482b5d91b8a889bd1f80f07e3c695cb5b46b02e5c1ad77fcf6e48a5122ac Problem: Missing payload data We made two changes related to how we store data in Hookdeck, resulting in us introducing an edge case scenario where payload data could be lost. Impact Between October 21st, 01:54:00AM UTC, and November 3rd, 20:26:00 UTC, 2023, resulting in 0.00039% of event payloads were impacted. Of these, 0.00037% were delivered at least once, and 0.00002% of the total payloads were lost and not delivered. Resolution The issue has been resolved as of November 3rd, 20:26:00 UTC, 2023. Full postmortem See the Postmortem - 2023-11-09: Lost payload data for full details: https://hookdeck.notion.site/2023-11-09-Missing-payload-data-298a6764666b4ea6899009c885e47929 https://status.hookdeck.com/incident/234407 Tue, 18 Jul 2023 15:34:00 -0000 https://status.hookdeck.com/incident/234407#f423f7ec6124613f34a7afc7cee03531c2ff836b225b64e7284e4a8e37b60681 The issue has been resolved by releasing a hotfix bypassing KV when unavailable. Post morterm: https://hookdeck.notion.site/2023-07-18-Increased-500-Errors-at-Ingestion-due-to-Cloudflare-Workers-KV-Store-Outage-e4c61aaff4004f8da12fbd2bbda254c2?pvs=4 https://status.hookdeck.com/incident/234407 Tue, 18 Jul 2023 14:42:00 -0000 https://status.hookdeck.com/incident/234407#3055921a36f5d200cf6e48dd19b32aa5d7e49e3c517c015c5dc309d27e6476f6 Cloudflare is currently experiencing issues with their KV product (https://www.cloudflarestatus.com/incidents/k5nmwlwzntcj), which impacts our ability to process requests. We have seen peak failure rates of around 1% and roughly 0.0001% ongoing failure rates. Failures are not uniformly distributed over data centers, which means depending on where your requests originate from, you may not see any error at all or an error rate above 1%. We are working on a hotfix to bypass KV in the meantime. We'll also publish a postmortem once we've managed to bring down the error rate to 0. https://status.hookdeck.com/incident/218152 Fri, 09 Jun 2023 21:16:00 -0000 https://status.hookdeck.com/incident/218152#6b9e8c0dd00d13550f67c3ebd8f464359e53d9c1e8e14bc9812c3d5ff225244b The issue has been resolved and was related to a specific node the API have network issue with the DB https://status.hookdeck.com/incident/218152 Fri, 09 Jun 2023 20:59:00 -0000 https://status.hookdeck.com/incident/218152#529c8c7ecc000799c4a846ab3c99a897fe1fcab3067c10470a5cb4fe5d2ff885 We're investigating an issue with the session on the dashboard. Some users are being logged-out automatically. https://status.hookdeck.com/incident/217322 Wed, 07 Jun 2023 19:01:00 -0000 https://status.hookdeck.com/incident/217322#fa508eb295aebc77798e279959c902b31901f371f7125fd45124430e6428c474 All service is back to normal, and we are no longer returning HTTP 500. The downtime lasted roughly 10 minutes. The issue seems related to the Cloudflare Worker platform failing to handle the requests. We are following up with Cloudflare support for me details. During that time, it's likely that webhooks were not received. If your provider has a retry logic, those webhooks will be delivered. If you need help or support following this downtime, please reach out to our team, and we'll assist you in the recovery. https://status.hookdeck.com/incident/217322 Wed, 07 Jun 2023 19:01:00 -0000 https://status.hookdeck.com/incident/217322#1d4e75baf471c7e73dbd8299b48b1aa06768f55c56393bf692bec90d1dc0d14e We are investigating an issue with Cloudflare Workers returning HTTP 500 to inbound requests. https://status.hookdeck.com/incident/192464 Tue, 14 Mar 2023 22:33:00 -0000 https://status.hookdeck.com/incident/192464#6cdf36f01b91a3b7e18294608661d069a784434ac6910cdfdcc9a8af0a9711b8 The configuration change has been reverted, and the redirect no longer occurs. Full postmortem here: https://www.notion.so/hookdeck/2023-03-14-SSL-Configuration-Change-6e1151229ff1480a9cb818f2afd652a5 https://status.hookdeck.com/incident/192464 Tue, 14 Mar 2023 22:32:00 -0000 https://status.hookdeck.com/incident/192464#95b19bacc8e4fe745db8789f15b01511971875bd8d8c1bee72b7be066d12c452 We modified an SSL configuration which caused our ingestion endpoint (events.hookdeck.com) to incorrectly return a 301 status code. This lasted for about 2 minutes, at which point we reverted the configuration. However, during that time we failed to ingest webhooks sent to our platform. For most, if the calling party followed redirects, hookdeck will have responded with a 405, the result of following the redirect with a GET. https://status.hookdeck.com/incident/167669 Tue, 24 Jan 2023 12:30:00 -0000 https://status.hookdeck.com/incident/167669#0ac85588960929749f8c3edcda09663c49715df3d4670def41c6c8478ea1bcee We have resolved the issue regarding the database failover. https://status.hookdeck.com/incident/167669 Tue, 24 Jan 2023 12:30:00 -0000 https://status.hookdeck.com/incident/167669#0ac85588960929749f8c3edcda09663c49715df3d4670def41c6c8478ea1bcee We have resolved the issue regarding the database failover. https://status.hookdeck.com/incident/167669 Tue, 24 Jan 2023 11:14:00 -0000 https://status.hookdeck.com/incident/167669#ffe2481787bae4cd3229a9bfba9bfc2369178a043860ba273d33f766af0fc3ff We're running into unforeseen issues during a database upgrade that's preventing writes to the database. During that time any write operations on the API will fail and events will NOT be delivered nor shown in the UI. However, we are still ingestion events and no webhooks have been lost. https://status.hookdeck.com/incident/167669 Tue, 24 Jan 2023 11:14:00 -0000 https://status.hookdeck.com/incident/167669#ffe2481787bae4cd3229a9bfba9bfc2369178a043860ba273d33f766af0fc3ff We're running into unforeseen issues during a database upgrade that's preventing writes to the database. During that time any write operations on the API will fail and events will NOT be delivered nor shown in the UI. However, we are still ingestion events and no webhooks have been lost. https://status.hookdeck.com/incident/95394 Thu, 16 Jun 2022 18:40:00 -0000 https://status.hookdeck.com/incident/95394#efb879a86625d10cbd1300926abafb3fffd9f9f450c1121191a2ac3a1f9fa4b1 We have identified and fixed the root cause of the issue. Webhooks are now being delivered successfully. We are continuing to monitor the situation to ensure everything is operational before marking the issue as fully resolved. https://status.hookdeck.com/incident/95394 Thu, 16 Jun 2022 18:30:00 -0000 https://status.hookdeck.com/incident/95394#e0d148c9d5cbfb80f5a2a36db867af9c60561521779425824e266906609d54e3 Hookdeck is currently experiencing an outage. Your webhooks are safe, and no data has been lost. Delivery is temporarily paused until we resolve the issue. Thank you for your patience! https://status.hookdeck.com/incident/60145 Thu, 25 Nov 2021 07:42:45 -0000 https://status.hookdeck.com/incident/60145#0f63c9b2c47ed3724aa2cf6619ee71a8976d92c8724802d8da0e1ce346f1396a The reconciliation job is complete, all webhooks have been delivered and all audits have passed. Everything is back in order. https://status.hookdeck.com/incident/60145 Wed, 24 Nov 2021 20:33:05 -0000 https://status.hookdeck.com/incident/60145#17dc6f335bf3f6a4c841bfddd4811d08df25ec36075c1321bbbfcaeb0ca434e9 We've restored from a backup dating from 2 hours ago and have put all services back online. Some events and attempts will be missing from the dashboard until we've completed a reconciliation job. https://status.hookdeck.com/incident/60145 Wed, 24 Nov 2021 20:15:46 -0000 https://status.hookdeck.com/incident/60145#1206fe5fd2a49fadf31e2e5402d11c5010b6f5ce96d1a97f7fb135fbdf75993d We're running into unforeseen issues during a database upgrade that's preventing writes to the database. During that time any write operations on the API will fail and events will NOT be delivered nor shown in the UI. However, we are still ingestion events and no webhooks have been lost. https://status.hookdeck.com/incident/59383 Tue, 16 Nov 2021 18:11:37 -0000 https://status.hookdeck.com/incident/59383#63b018746daa1107051cbe8e6b15a495b4fbba11c5eacde92875f89e7cae6425 API, Dashboard and retries services are back online. We're keeping an eye on it in case of further issues. https://status.hookdeck.com/incident/59383 Tue, 16 Nov 2021 18:11:37 -0000 https://status.hookdeck.com/incident/59383#63b018746daa1107051cbe8e6b15a495b4fbba11c5eacde92875f89e7cae6425 API, Dashboard and retries services are back online. We're keeping an eye on it in case of further issues. https://status.hookdeck.com/incident/59383 Tue, 16 Nov 2021 18:10:08 -0000 https://status.hookdeck.com/incident/59383#c5c912183e9938651544d9d39cada3f37c944d8a4776dffc4b9d192273c2d302 Automatic retries are also impacted, failed events scheduled for retry won't be retried until GCP recovers. https://status.hookdeck.com/incident/59383 Tue, 16 Nov 2021 18:10:08 -0000 https://status.hookdeck.com/incident/59383#c5c912183e9938651544d9d39cada3f37c944d8a4776dffc4b9d192273c2d302 Automatic retries are also impacted, failed events scheduled for retry won't be retried until GCP recovers. https://status.hookdeck.com/incident/59383 Tue, 16 Nov 2021 17:57:25 -0000 https://status.hookdeck.com/incident/59383#1e7024aa86a17aac73b711ccb475ab0ca147bce0cfab30b501499c1656fc5563 Hookdeck API and Dashboard are currently down because of the Google Cloud outage and Netlify outage. Webhooks are still being ingested and delivered. https://status.hookdeck.com/incident/59383 Tue, 16 Nov 2021 17:57:25 -0000 https://status.hookdeck.com/incident/59383#1e7024aa86a17aac73b711ccb475ab0ca147bce0cfab30b501499c1656fc5563 Hookdeck API and Dashboard are currently down because of the Google Cloud outage and Netlify outage. Webhooks are still being ingested and delivered.