What happened
A configuration change applied to QStash's EU networking layer introduced an outbound connectivity fault. Between 09:11 and 09:45 UTC, a subset of QStash instances had degraded egress connectivity, and some dispatched messages may have failed on their initial attempt. A related side effect caused some servers to egress from IP addresses outside QStash's advertised outbound range until 11:24 UTC; during that window, endpoints enforcing QStash IP allowlists may have rejected requests from those servers.
QStash's automatic retries delivered many affected messages on subsequent attempts once connectivity was restored, and QStash is now operating normally.
Customer impact
Impact was limited to the EU region and to two effects within the window above:
- Delivery delays / failures (09:11 – 09:45 UTC) — messages dispatched through affected instances may have failed on their first attempt. Thanks to automatic retries with backoff, most were re-delivered once connectivity recovered; messages that exhausted their retry policy during the window followed their configured failure path (e.g., DLQ / failure callback).
- Allowlist rejections (09:11 – 11:24 UTC) — customers who restrict inbound traffic to QStash's outbound IP ranges may have seen requests from the affected servers rejected. Customers who do not enforce source-IP allowlisting were unaffected by this.
Root cause
A routine networking configuration change contained an error that:
- disrupted outbound connectivity on the affected nodes, and
- caused affected nodes to acquire outbound IPs outside the advertised range.
The underlying gap was systemic: the change procedure had no automated validation gate to catch the faulty state before it reached production.
What we've done
- Fail-fast pre-checks — automated validation now halts the change procedure before it runs if a configuration would degrade connectivity or violate expected state.
- Egress IP-range enforcement — outbound IP assignments are now checked against the advertised QStash range, so a node can no longer come online with an unadvertised IP.
- Tighter post-change verification — completion now confirms outbound reachability and egress-IP conformance across affected nodes.
Customer action
None required. All QStash traffic again originates from the advertised outbound IP ranges.
We apologize for the disruption.