QStash EU Region — Degraded Performance

Summary

On June 23, 2026, QStash users in the EU-CENTRAL-1 region experienced elevated latency and degraded performance. The issue was caused by a regression introduced in a recent enhancement to Flow Control scheduling logic. The change increased resource consumption under load, leading to reduced performance on affected shards. The incident was resolved by rolling back to the previous stable version.

Impact

• Service: QStash (EU-CENTRAL-1)
• Start: 2026-06-23 16:22 UTC
• Resolved: 2026-06-23 17:14 UTC
• Duration: ~52 minutes
• Impact: Increased latency and degraded performance for workloads routed to the affected instance in the EU. Multiple customers in the region experienced slower request processing during the incident window.

Root Cause

We recently deployed an enhancement to our Flow Control feature, designed to improve fairness between independent flow controls when unused global parallelism capacity was available.

Under production load, the new behavior introduced a regression that significantly increased resource utilization. The elevated resource consumption caused performance degradation on the affected QStash instamce, impacting all workloads sharing that instance.

Resolution

After identifying the regression as the source of the slowdown, we rolled back the deployment to the last known stable version.

• Rollback completed: 2026-06-23 19:18:46.84 UTC

Following the rollback, system performance returned to normal levels and service stability was restored.

Preventive Actions

To reduce the likelihood of similar incidents in the future, we are taking the following actions:

• Expand performance and load testing coverage for Flow Control changes.
• Add resource utilization regression checks to the deployment pipeline.
• Improve monitoring and alerting for abnormal CPU and memory consumption patterns.
• Introduce additional canary validation before wider production rollout of scheduler-related changes.

We apologize for the disruption and appreciate our customers’ patience while we resolved the issue.