{
  "page": {
    "id": "0mggy4p2v1q4",
    "name": "Upstash Status",
    "url": "https://status.upstash.com"
  },
  "status": {
    "indicator": "none",
    "description": "All Systems Operational"
  },
  "components": [{"status":"operational","name":"Redis Global","created_at":"2024-11-11T10:14:48.699Z","updated_at":"2024-11-11T10:26:24.194Z","position":1,"description":null,"showcase":false,"start_date":null,"id":"61bgyd0tzh22","group_id":null,"page_id":"0mggy4p2v1q4","components":["qszfl72zt4lm","zm64hy1n4996","nnfskcftmn5n","w4rzz67ctqtd","9v35zx451z1s","0rqw8cjxshf9","828b0qmpl5f1","33wdd01pp8n5","wg94b7byvr8f","9bmgx0plc8bc","xj2m103dbpst","pbwwc2pwm6ts","xxbbxhg382gx","986tvdgn7zvb","g9ktdwzl9hj7","xh0yj1xgp6d8"]},{"status":"operational","name":"Redis Regional","created_at":"2021-08-25T14:10:16.289Z","updated_at":"2024-11-11T10:26:24.204Z","position":2,"description":null,"showcase":false,"start_date":null,"id":"3wszb9kxy0n8","group_id":null,"page_id":"0mggy4p2v1q4","components":["l1z43434lqnb","x20f3jcqg1vq","hd01j6rwg9tf","4k8sv8q07w0g","3yb32ccfk6qt","pmnxm5wf6s83","03lj1p8n41q8"]},{"status":"operational","name":"Vector","created_at":"2024-06-03T22:21:40.921Z","updated_at":"2024-11-11T10:26:24.221Z","position":5,"description":null,"showcase":false,"start_date":null,"id":"y3fn3qrjgngg","group_id":null,"page_id":"0mggy4p2v1q4","components":["bp28p3qjhr1b","5wptztwvxk4v","l78b6h7482ly"]},{"status":"operational","name":"Upstash Console","created_at":"2025-06-26T06:51:35.154Z","updated_at":"2025-12-05T09:18:53.495Z","position":6,"description":"console.upstash.com","showcase":true,"start_date":"2025-01-01T00:00:00.000Z","id":"scyjjlxngg7k","group_id":null,"page_id":"0mggy4p2v1q4"},{"status":"operational","name":"Context7","created_at":"2025-12-05T08:58:57.443Z","updated_at":"2025-12-05T08:58:57.443Z","position":7,"description":null,"showcase":true,"start_date":"2025-12-05T00:00:00.000Z","id":"89wt497r42f4","group_id":null,"page_id":"0mggy4p2v1q4"},{"status":"operational","name":"Context7 Console","created_at":"2025-12-05T08:59:22.506Z","updated_at":"2025-12-05T09:18:53.512Z","position":8,"description":null,"showcase":true,"start_date":"2025-12-05T00:00:00.000Z","id":"00crz9f5ls9x","group_id":null,"page_id":"0mggy4p2v1q4"},{"status":"operational","name":"QStash","created_at":"2026-04-27T15:29:16.024Z","updated_at":"2026-04-27T15:29:16.024Z","position":9,"description":null,"showcase":false,"start_date":null,"id":"4gq098729864","group_id":null,"page_id":"0mggy4p2v1q4","components":["y15c5bt1tkwx","0kp4ybhh9srx"]},{"status":"operational","name":"Box","created_at":"2026-06-14T20:33:40.127Z","updated_at":"2026-06-14T20:44:37.698Z","position":10,"description":null,"showcase":true,"start_date":"2026-06-01T00:00:00.000Z","id":"376v79kl4kvv","group_id":null,"page_id":"0mggy4p2v1q4"}],
  "incidents": [{"name":"Brief connection proxy degradation in us-east-1","status":"resolved","created_at":"2026-09-29T15:12:00.000Z","updated_at":"2026-09-29T15:36:35.535Z","monitoring_at":null,"resolved_at":"2026-09-29T15:12:00.000Z","impact":"minor","shortlink":"https://stspg.io/f1kyl12t9l0q","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":"minor","scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2026-09-29T15:36:30.439Z","reminder_intervals":null,"id":"zmw7rg5r2qxw","page_id":"0mggy4p2v1q4","incident_updates":[{"status":"resolved","body":"Between 15:12 and 15:16 UTC, we observed a brief increase in load on the connection proxy layer in us-east-1, which may have affected some existing and new connections.\n\nThe team responded quickly and resolved the issue. All systems are now fully operational.","created_at":"2026-09-29T15:36:30.501Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-09-29T15:36:30.501Z","display_at":"2026-09-29T15:12:00.000Z","deliver_notifications":false,"tweet_id":null,"id":"26cymnzl50yp","incident_id":"zmw7rg5r2qxw","custom_tweet":null,"affected_components":null}],"postmortem_body":null,"postmortem_body_last_updated_at":null,"postmortem_ignored":false,"postmortem_published_at":null,"postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[]},{"name":"Intermittent DNS Resolution Errors for Upstash Vector in US East (us-east-1)","status":"resolved","created_at":"2026-09-09T12:00:00.000Z","updated_at":"2026-09-09T13:01:03.143Z","monitoring_at":null,"resolved_at":"2026-09-09T12:00:00.000Z","impact":"minor","shortlink":"https://stspg.io/yfkkt5363477","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":"minor","scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2026-09-09T13:00:58.510Z","reminder_intervals":null,"id":"p3wdmtwyy8zw","page_id":"0mggy4p2v1q4","incident_updates":[{"status":"resolved","body":"Upstash Vector experienced a brief DNS resolution issue in the US East (us-east-1) region lasting approximately 20 minutes. During this period, new DNS resolution attempts may have failed.\n\nExisting connections and clients with cached DNS records were not affected.\n\nThe issue has been resolved, and DNS resolution is operating normally.","created_at":"2026-09-09T13:00:58.564Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-09-09T13:00:58.564Z","display_at":"2026-09-09T12:00:00.000Z","deliver_notifications":false,"tweet_id":null,"id":"cqg7pxdc86yp","incident_id":"p3wdmtwyy8zw","custom_tweet":null,"affected_components":null}],"postmortem_body":null,"postmortem_body_last_updated_at":null,"postmortem_ignored":false,"postmortem_published_at":null,"postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[]},{"name":"Upstash Console login issue","status":"resolved","created_at":"2026-09-03T08:00:00.000Z","updated_at":"2026-09-03T08:29:36.492Z","monitoring_at":null,"resolved_at":"2026-09-03T08:00:00.000Z","impact":"none","shortlink":"https://stspg.io/dmdmfrm8lycz","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":null,"scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2026-09-03T08:29:36.412Z","reminder_intervals":null,"id":"mrsy2jmjlzj9","page_id":"0mggy4p2v1q4","incident_updates":[{"status":"resolved","body":"We experienced a brief issue affecting access to the Upstash Console. Our team quickly identified the cause and resolved the issue within minutes.\n\nThe console is now operating normally.","created_at":"2026-09-03T08:29:36.481Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-09-03T08:29:36.481Z","display_at":"2026-09-03T08:00:00.000Z","deliver_notifications":false,"tweet_id":null,"id":"qdvbmgpc6mfz","incident_id":"mrsy2jmjlzj9","custom_tweet":null,"affected_components":null}],"postmortem_body":null,"postmortem_body_last_updated_at":null,"postmortem_ignored":false,"postmortem_published_at":null,"postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[]},{"name":"Message Persistence Issue — QStash us-east-1","status":"resolved","created_at":"2026-08-28T22:20:00.000Z","updated_at":"2026-08-28T22:57:24.342Z","monitoring_at":null,"resolved_at":"2026-08-28T22:20:00.000Z","impact":"major","shortlink":"https://stspg.io/69bd84bqrpcq","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":"major","scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2026-08-28T22:56:58.127Z","reminder_intervals":null,"id":"z2vs4hjr81g5","page_id":"0mggy4p2v1q4","incident_updates":[{"status":"resolved","body":"Between 22:20 and 22:33 UTC, QStash clients in us-east-1 experienced disruptions related to message persistence. The team applied a fix, and service has been restored.","created_at":"2026-08-28T22:56:58.181Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-08-28T22:56:58.181Z","display_at":"2026-08-28T22:20:00.000Z","deliver_notifications":false,"tweet_id":null,"id":"swgk4xw5596g","incident_id":"z2vs4hjr81g5","custom_tweet":null,"affected_components":null}],"postmortem_body":null,"postmortem_body_last_updated_at":null,"postmortem_ignored":false,"postmortem_published_at":null,"postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[]},{"name":"Fly.io infrastructure disruption affecting some Upstash Redis databases on Fly.io DFW Region","status":"resolved","created_at":"2026-07-22T08:50:12.940Z","updated_at":"2026-07-22T10:05:17.932Z","monitoring_at":"2026-07-22T09:01:01.631Z","resolved_at":"2026-07-22T10:05:17.914Z","impact":"none","shortlink":"https://stspg.io/0l5sf44ng7bn","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":null,"scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2026-07-22T08:50:12.934Z","reminder_intervals":null,"id":"d6fvfsj3gkxm","page_id":"0mggy4p2v1q4","incident_updates":[{"status":"resolved","body":"This incident has been resolved.","created_at":"2026-07-22T10:05:17.914Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-07-22T10:05:17.914Z","display_at":"2026-07-22T10:05:17.914Z","deliver_notifications":false,"tweet_id":null,"id":"f2s1vch2t7sn","incident_id":"d6fvfsj3gkxm","custom_tweet":null,"affected_components":null},{"status":"monitoring","body":"The affected Fly.io-hosted machines recovered at 08:48 UTC, and service availability has been restored. We are monitoring the systems to confirm continued stability.","created_at":"2026-07-22T09:01:01.631Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-07-22T09:01:01.631Z","display_at":"2026-07-22T09:01:01.631Z","deliver_notifications":false,"tweet_id":null,"id":"3z3fgqwxycnl","incident_id":"d6fvfsj3gkxm","custom_tweet":null,"affected_components":null},{"status":"identified","body":"We are investigating connectivity issues affecting a subset of Redis databases hosted on Fly.io. The issue is related to an ongoing Fly.io infrastructure disruption in the DFW region (https://status.flyio.net/incidents/n4z2my6qw4sb). We are working to restore availability and will provide updates soon.","created_at":"2026-07-22T08:50:12.980Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-07-22T08:50:12.980Z","display_at":"2026-07-22T08:50:12.980Z","deliver_notifications":false,"tweet_id":null,"id":"gxy8ccn7mc97","incident_id":"d6fvfsj3gkxm","custom_tweet":null,"affected_components":null}],"postmortem_body":null,"postmortem_body_last_updated_at":null,"postmortem_ignored":false,"postmortem_published_at":null,"postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[]},{"name":"QStash us-east-1 - URL Publish Errors","status":"resolved","created_at":"2026-07-16T06:30:00.000Z","updated_at":"2026-07-16T08:35:07.559Z","monitoring_at":null,"resolved_at":"2026-07-16T06:30:00.000Z","impact":"none","shortlink":"https://stspg.io/9qvjxdfprs57","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":null,"scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2026-07-16T08:35:07.487Z","reminder_intervals":null,"id":"d9xmrx1kx7wp","page_id":"0mggy4p2v1q4","incident_updates":[{"status":"resolved","body":"Partial outage due to http parsing errors. Problem was resolved couple minutes later. If problem continues, refreshing the DNS cache is recommended.","created_at":"2026-07-16T08:35:07.544Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-07-16T08:35:07.544Z","display_at":"2026-07-16T06:30:00.000Z","deliver_notifications":false,"tweet_id":null,"id":"3605kyr63c08","incident_id":"d9xmrx1kx7wp","custom_tweet":null,"affected_components":null}],"postmortem_body":null,"postmortem_body_last_updated_at":null,"postmortem_ignored":false,"postmortem_published_at":null,"postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[]},{"name":"Upstash Redis Partial Service Disruption","status":"resolved","created_at":"2026-06-25T15:23:44.611Z","updated_at":"2026-06-25T16:36:00.871Z","monitoring_at":"2026-06-25T15:31:12.572Z","resolved_at":"2026-06-25T16:36:00.855Z","impact":"major","shortlink":"https://stspg.io/9gynqrnvcpyj","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":"major","scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2026-06-25T15:23:44.597Z","reminder_intervals":null,"id":"bs21y4fbyh4n","page_id":"0mggy4p2v1q4","incident_updates":[{"status":"resolved","body":"This incident has been resolved, we will publish RCA soon.","created_at":"2026-06-25T16:36:00.855Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-06-25T16:36:00.855Z","display_at":"2026-06-25T16:36:00.855Z","deliver_notifications":true,"tweet_id":null,"id":"0t2qplg3vg2r","incident_id":"bs21y4fbyh4n","custom_tweet":null,"affected_components":[{"code":"qszfl72zt4lm","name":"Redis Global - N. Virginia, USA (us-east-1)","old_status":"operational","new_status":"operational"},{"code":"zm64hy1n4996","name":"Redis Global - N. California, USA (us-west-1)","old_status":"operational","new_status":"operational"},{"code":"w4rzz67ctqtd","name":"Redis Global - Frankfurt, Germany (eu-central-1)","old_status":"operational","new_status":"operational"}]},{"status":"monitoring","body":"A fix has been implemented and we are monitoring the results.","created_at":"2026-06-25T15:31:12.572Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-06-25T15:31:12.572Z","display_at":"2026-06-25T15:31:12.572Z","deliver_notifications":true,"tweet_id":null,"id":"04wy99g1c3jl","incident_id":"bs21y4fbyh4n","custom_tweet":null,"affected_components":[{"code":"qszfl72zt4lm","name":"Redis Global - N. Virginia, USA (us-east-1)","old_status":"operational","new_status":"operational"},{"code":"zm64hy1n4996","name":"Redis Global - N. California, USA (us-west-1)","old_status":"operational","new_status":"operational"},{"code":"w4rzz67ctqtd","name":"Redis Global - Frankfurt, Germany (eu-central-1)","old_status":"operational","new_status":"operational"}]},{"status":"investigating","body":"We are currently experiencing a partial service disruption affecting Upstash Redis in the following regions:\n\n* eu-central-1\n* us-east-1\n* us-west-1\n\nOur team is investigating the issue and working to restore full service as soon as possible. We will share updates as more information becomes available.","created_at":"2026-06-25T15:23:44.669Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-06-25T15:23:44.669Z","display_at":"2026-06-25T15:23:44.669Z","deliver_notifications":true,"tweet_id":null,"id":"xppshdw85n6g","incident_id":"bs21y4fbyh4n","custom_tweet":null,"affected_components":[{"code":"qszfl72zt4lm","name":"Redis Global - N. Virginia, USA (us-east-1)","old_status":"operational","new_status":"operational"},{"code":"zm64hy1n4996","name":"Redis Global - N. California, USA (us-west-1)","old_status":"operational","new_status":"operational"},{"code":"w4rzz67ctqtd","name":"Redis Global - Frankfurt, Germany (eu-central-1)","old_status":"operational","new_status":"operational"}]}],"postmortem_body":null,"postmortem_body_last_updated_at":null,"postmortem_ignored":false,"postmortem_published_at":null,"postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[{"status":"operational","name":"N. Virginia, USA (us-east-1)","created_at":"2024-11-11T10:14:48.726Z","updated_at":"2025-06-11T09:20:09.163Z","position":1,"description":null,"showcase":true,"start_date":"2024-08-01T00:00:00.000Z","id":"qszfl72zt4lm","page_id":"0mggy4p2v1q4","group_id":"61bgyd0tzh22"},{"status":"operational","name":"N. California, USA (us-west-1)","created_at":"2024-11-11T10:18:23.922Z","updated_at":"2025-06-11T09:20:09.179Z","position":2,"description":null,"showcase":true,"start_date":"2024-08-01T00:00:00.000Z","id":"zm64hy1n4996","page_id":"0mggy4p2v1q4","group_id":"61bgyd0tzh22"},{"status":"operational","name":"Frankfurt, Germany (eu-central-1)","created_at":"2024-11-11T10:19:10.801Z","updated_at":"2025-09-09T15:49:13.294Z","position":4,"description":null,"showcase":true,"start_date":"2024-08-01T00:00:00.000Z","id":"w4rzz67ctqtd","page_id":"0mggy4p2v1q4","group_id":"61bgyd0tzh22"}]},{"name":"QStash EU Region — Degraded Performance","status":"resolved","created_at":"2026-06-23T16:22:25.859Z","updated_at":"2026-06-24T12:30:46.041Z","monitoring_at":"2026-06-23T16:32:26.385Z","resolved_at":"2026-06-23T17:14:46.066Z","impact":"minor","shortlink":"https://stspg.io/r5k8bdfqm35c","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":null,"scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2026-06-23T16:22:25.848Z","reminder_intervals":null,"id":"45rxy9q1zzc2","page_id":"0mggy4p2v1q4","incident_updates":[{"status":"resolved","body":"This incident has been resolved.","created_at":"2026-06-23T17:14:46.066Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-06-23T17:14:46.066Z","display_at":"2026-06-23T17:14:46.066Z","deliver_notifications":true,"tweet_id":null,"id":"q6bc0sh7ck1g","incident_id":"45rxy9q1zzc2","custom_tweet":null,"affected_components":[{"code":"y15c5bt1tkwx","name":"QStash - EU-CENTRAL-1","old_status":"degraded_performance","new_status":"operational"}]},{"status":"monitoring","body":"A fix has been implemented and we are monitoring the results.","created_at":"2026-06-23T16:32:26.385Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-06-23T16:32:26.385Z","display_at":"2026-06-23T16:32:26.385Z","deliver_notifications":true,"tweet_id":null,"id":"r5c4b1q9ctzs","incident_id":"45rxy9q1zzc2","custom_tweet":null,"affected_components":[{"code":"y15c5bt1tkwx","name":"QStash - EU-CENTRAL-1","old_status":"degraded_performance","new_status":"degraded_performance"}]},{"status":"investigating","body":"We are currently investigating this issue.","created_at":"2026-06-23T16:22:25.945Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-06-23T16:22:25.945Z","display_at":"2026-06-23T16:22:25.945Z","deliver_notifications":true,"tweet_id":null,"id":"4bzdvcnp9hwj","incident_id":"45rxy9q1zzc2","custom_tweet":null,"affected_components":[{"code":"y15c5bt1tkwx","name":"QStash - EU-CENTRAL-1","old_status":"operational","new_status":"degraded_performance"}]}],"postmortem_body":"### **Summary**\n\nOn June 23, 2026, QStash users in the EU-CENTRAL-1 region experienced elevated latency and degraded performance. The issue was caused by a regression introduced in a recent enhancement to Flow Control scheduling logic. The change increased resource consumption under load, leading to reduced performance on affected shards. The incident was resolved by rolling back to the previous stable version.\n\n### **Impact**\n\n* Service: QStash \\(EU-CENTRAL-1\\)\n* Start: 2026-06-23 16:22 UTC\n* Resolved: 2026-06-23 17:14 UTC\n* Duration: ~52 minutes\n* Impact: Increased latency and degraded performance for workloads routed to the affected instance in the EU. Multiple customers in the region experienced slower request processing during the incident window.\n\n### **Root Cause**\n\nWe recently deployed an enhancement to our Flow Control feature, designed to improve fairness between independent flow controls when unused global parallelism capacity was available.\n\nUnder production load, the new behavior introduced a regression that significantly increased resource utilization. The elevated resource consumption caused performance degradation on the affected QStash instamce, impacting all workloads sharing that instance.\n\n### **Resolution**\n\nAfter identifying the regression as the source of the slowdown, we rolled back the deployment to the last known stable version.\n\n* Rollback completed: 2026-06-23 19:18:46.84 UTC\n\nFollowing the rollback, system performance returned to normal levels and service stability was restored.\n\n### **Preventive Actions**\n\nTo reduce the likelihood of similar incidents in the future, we are taking the following actions:\n\n* Expand performance and load testing coverage for Flow Control changes.\n* Add resource utilization regression checks to the deployment pipeline.\n* Improve monitoring and alerting for abnormal CPU and memory consumption patterns.\n* Introduce additional canary validation before wider production rollout of scheduler-related changes.\n\nWe apologize for the disruption and appreciate our customers’ patience while we resolved the issue.","postmortem_body_last_updated_at":"2026-06-24T12:30:46.022Z","postmortem_ignored":false,"postmortem_published_at":"2026-06-24T12:30:46.022Z","postmortem_notified_subscribers":true,"postmortem_notified_twitter":false,"components":[{"status":"operational","name":"EU-CENTRAL-1","created_at":"2022-08-10T13:30:07.031Z","updated_at":"2026-06-23T17:14:46.035Z","position":1,"description":null,"showcase":true,"start_date":"2022-08-10T00:00:00.000Z","id":"y15c5bt1tkwx","page_id":"0mggy4p2v1q4","group_id":"4gq098729864"}]},{"name":"Box Access and Management Operations Unavailable","status":"resolved","created_at":"2026-06-23T06:05:57.207Z","updated_at":"2026-06-23T06:05:57.261Z","monitoring_at":null,"resolved_at":"2026-06-23T06:05:57.252Z","impact":"none","shortlink":"https://stspg.io/y6kf1kvffvjj","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":null,"scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2026-06-23T06:05:57.202Z","reminder_intervals":null,"id":"ms9sqp71kyc9","page_id":"0mggy4p2v1q4","incident_updates":[{"status":"resolved","body":"Between 01:18 and 04:03 UTC, customers were unable to access running boxes or perform create, read, update, or delete operations. Running containers were not affected and continued operating throughout. The issue has been fully resolved and all box operations are functioning normally.\n\nWe are taking all necessary measures to prevent this from happening again. We apologize for the disruption.","created_at":"2026-06-23T06:05:57.252Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-06-23T06:05:57.252Z","display_at":"2026-06-23T06:05:57.252Z","deliver_notifications":true,"tweet_id":null,"id":"p71h32hhwprx","incident_id":"ms9sqp71kyc9","custom_tweet":null,"affected_components":[{"code":"376v79kl4kvv","name":"Box","old_status":"operational","new_status":"operational"}]}],"postmortem_body":null,"postmortem_body_last_updated_at":null,"postmortem_ignored":false,"postmortem_published_at":null,"postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[{"status":"operational","name":"Box","created_at":"2026-06-14T20:33:40.127Z","updated_at":"2026-06-14T20:44:37.698Z","position":10,"description":null,"showcase":true,"start_date":"2026-06-01T00:00:00.000Z","id":"376v79kl4kvv","page_id":"0mggy4p2v1q4","group_id":null}]},{"name":"QStash Service Disruption in the EU Region","status":"resolved","created_at":"2026-06-19T09:11:00.000Z","updated_at":"2026-06-19T13:59:02.732Z","monitoring_at":null,"resolved_at":"2026-06-19T09:11:00.000Z","impact":"major","shortlink":"https://stspg.io/5m39sbnstm49","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":"major","scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2026-06-19T09:11:00.000Z","reminder_intervals":null,"id":"4rshtsqjhn3c","page_id":"0mggy4p2v1q4","incident_updates":[{"status":"resolved","body":"We identified an outbound networking issue affecting a subset of QStash server instances in the EU region between 2026-06-19 09:11 UTC and 2026-06-19 09:45 UTC.\n\nDuring this period, some requests dispatched by QStash may have failed.\n\nThe issue has been resolved, and QStash is currently operating normally. We will publish a postmortem with additional details as soon as it is available.\n\nUpdate: We identified that one QStash server had been assigned an outbound IP address outside of the configured QStash IP range. This remained the case until 2026-06-19 11:24 UTC.\n\nEndpoints that restrict access based on QStash outbound IP allowlists may have rejected requests from this server during this period.","created_at":"2026-06-19T10:07:25.073Z","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-06-19T11:38:11.340Z","display_at":"2026-06-19T09:11:00.000Z","deliver_notifications":false,"tweet_id":null,"id":"xwkwkf5z7jq7","incident_id":"4rshtsqjhn3c","custom_tweet":null,"affected_components":null}],"postmortem_body":"## What happened\n\nA configuration change applied to QStash's EU networking layer introduced an outbound connectivity fault. Between **09:11 and 09:45 UTC**, a subset of QStash instances had degraded egress connectivity, and some dispatched messages may have failed on their initial attempt. A related side effect caused some servers to egress from IP addresses outside QStash's advertised outbound range until **11:24 UTC**; during that window, endpoints enforcing QStash IP allowlists may have rejected requests from those servers.\n\nQStash's automatic retries delivered many affected messages on subsequent attempts once connectivity was restored, and QStash is now operating normally.\n\n## Customer impact\n\nImpact was limited to the EU region and to two effects within the window above:\n\n* **Delivery delays / failures** _\\(09:11 – 09:45 UTC\\)_ — messages dispatched through affected instances may have failed on their first attempt. Thanks to automatic retries with backoff, most were re-delivered once connectivity recovered; messages that exhausted their retry policy during the window followed their configured failure path \\(e.g., DLQ / failure callback\\).\n* **Allowlist rejections** _\\(09:11 – 11:24 UTC\\)_ — customers who restrict inbound traffic to QStash's outbound IP ranges may have seen requests from the affected servers rejected. Customers who do not enforce source-IP allowlisting were unaffected by this.\n\n## Root cause\n\nA routine networking configuration change contained an error that:\n\n1. disrupted outbound connectivity on the affected nodes, and\n2. caused affected nodes to acquire outbound IPs outside the advertised range.\n\nThe underlying gap was systemic: the change procedure had no automated validation gate to catch the faulty state before it reached production.\n\n## What we've done\n\n* **Fail-fast pre-checks** — automated validation now halts the change procedure _before it runs_ if a configuration would degrade connectivity or violate expected state.\n* **Egress IP-range enforcement** — outbound IP assignments are now checked against the advertised QStash range, so a node can no longer come online with an unadvertised IP.\n* **Tighter post-change verification** — completion now confirms outbound reachability and egress-IP conformance across affected nodes.\n\n## Customer action\n\nNone required. All QStash traffic again originates from the advertised outbound IP ranges.\n\nWe apologize for the disruption.","postmortem_body_last_updated_at":"2026-06-19T13:59:02.710Z","postmortem_ignored":false,"postmortem_published_at":"2026-06-19T13:59:02.710Z","postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[]}]
}