tag:status.upstash.com,2005:/historyUpstash Status Status - Incident History2024-03-29T00:06:52ZUpstash Statustag:status.upstash.com,2005:Incident/202321002024-03-13T08:30:00Z2024-03-13T09:30:56ZMaintenance - Global Ap-Southeast-1<p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>08:30</var> UTC</small><br><strong>Resolved</strong> - We have taken actions to increase the capacity of the region. During the operation, clients might have experienced higher than usual latencies for about 15 minutes.</p>tag:status.upstash.com,2005:Incident/202233872024-03-12T09:49:52Z2024-03-12T09:49:52ZConnectivity Issues - Global eu-central-1<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>09:49</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>09:42</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.upstash.com,2005:Incident/202219652024-03-12T08:06:36Z2024-03-12T08:06:36ZConnectivity Issues - Global Ap-Southeast-1<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>08:06</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>07:40</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>06:59</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>06:53</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.upstash.com,2005:Incident/201783612024-03-06T19:42:10Z2024-03-06T19:42:10ZDegraded Availability in Global Us-East-1<p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>19:42</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>18:37</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.upstash.com,2005:Incident/184219092023-09-07T16:22:45Z2023-09-07T16:22:45ZDegraded Availability on Redis<p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>16:22</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>15:04</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>13:27</var> UTC</small><br><strong>Investigating</strong> - Some databases are not reachable. We are currently investigating issue.</p>tag:status.upstash.com,2005:Incident/182950602023-08-28T20:57:32Z2023-08-28T20:57:32ZClient connections are intermittently dropping<p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>20:57</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>15:46</var> UTC</small><br><strong>Monitoring</strong> - We found that small number of database connections are intermittently dropping.<br /><br />We have applied a fix and observing now.</p>tag:status.upstash.com,2005:Incident/181659252023-08-16T19:57:57Z2023-08-16T19:57:57ZDegraded Availability on QStash<p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>19:57</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>19:56</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>19:56</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>19:45</var> UTC</small><br><strong>Identified</strong> - An issue in persistence layer causes degradation in QStash availability. We have identified the cause and working on a fix right now.</p>tag:status.upstash.com,2005:Incident/178999152023-07-19T11:45:00Z2023-07-19T13:07:49ZPartial Downtime for REST Clients on Kafka EU-WEST-1 Clusters<p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>11:45</var> UTC</small><br><strong>Resolved</strong> - We regret to inform you that we experienced a partial downtime during our recent maintenance. This downtime specifically impacted our REST clients. We apologize for any inconvenience caused and assure you that our team has fixed connectivity issues.</p>tag:status.upstash.com,2005:Incident/173352422023-05-23T00:30:00Z2023-05-23T00:43:18ZDegraded performance on QStash<p><small>May <var data-var='date'>23</var>, <var data-var='time'>00:30</var> UTC</small><br><strong>Resolved</strong> - The performance of QStash experienced degradation, after getting alert from our monitoring system, our team intervened and restored its stability.</p>tag:status.upstash.com,2005:Incident/172107912023-05-10T21:46:41Z2023-05-10T21:46:41ZElevated latency in us-east-1<p><small>May <var data-var='date'>10</var>, <var data-var='time'>21:46</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>21:36</var> UTC</small><br><strong>Monitoring</strong> - We are monitoring the status now.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>21:27</var> UTC</small><br><strong>Identified</strong> - We have identified the issue and applied the fix.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>20:04</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing elevated latency in us-east-1 and are investigating the issue. We will share updates as they become available.</p>tag:status.upstash.com,2005:Incident/172000662023-05-09T22:17:24Z2023-05-09T22:17:24ZDegraded performance on QStash<p><small>May <var data-var='date'> 9</var>, <var data-var='time'>22:17</var> UTC</small><br><strong>Resolved</strong> - Surge in user activity resulted in unusually high traffic, causing temporary service disruption in QStash service.</p>tag:status.upstash.com,2005:Incident/171928062023-05-09T13:41:23Z2023-05-09T13:41:23ZDegraded performance on QStash<p><small>May <var data-var='date'> 9</var>, <var data-var='time'>13:41</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 9</var>, <var data-var='time'>13:32</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'> 9</var>, <var data-var='time'>13:27</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.upstash.com,2005:Incident/169216042023-04-20T15:05:04Z2023-04-20T15:05:04ZDegraded performance on Stash<p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>15:05</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>14:56</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>14:39</var> UTC</small><br><strong>Investigating</strong> - Degregeded performance on QStash</p>tag:status.upstash.com,2005:Incident/168183422023-04-11T15:00:23Z2023-04-11T22:48:38ZQStash Maintenance<p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>15:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>10:21</var> UTC</small><br><strong>Update</strong> - We have implemented some performance enhancements and are currently monitoring their effectiveness.</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>07:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>06:46</var> UTC</small><br><strong>Scheduled</strong> - We are currently experiencing high load on our QStash instances. To improve performance and accommodate the increasing load, we will be conducting some maintenance work. As a result, there may be some disruptions to the availability of QStash throughout the day. However, we will make every effort to minimize any impact on our users.</p>tag:status.upstash.com,2005:Incident/167705372023-04-06T14:47:30Z2023-04-06T14:49:45ZDegraded performance on AWS EU-WEST-1 region<p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>14:47</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>12:37</var> UTC</small><br><strong>Monitoring</strong> - We have identified a heavy system load on some of the database servers. We are adding new machines to the pool to prevent this happening again. Some databases may have observed high latencies or short disconnections during the event.</p>tag:status.upstash.com,2005:Incident/166490252023-03-26T06:00:00Z2023-03-26T06:35:26ZQStash Unavailable<p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>06:00</var> UTC</small><br><strong>Resolved</strong> - QStash had a short period of unavailability. More resource has been automatically allocated to the instance during this time. We are taking steps to optimize the resource availability by allocating further resources.</p>tag:status.upstash.com,2005:Incident/164579612023-03-12T14:49:40Z2023-03-12T14:49:40ZQStash Unavailable<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>14:49</var> UTC</small><br><strong>Resolved</strong> - We have allocated more resources to the instance. QStash is stable.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>14:08</var> UTC</small><br><strong>Monitoring</strong> - QStash had a short period of unavailability. More resource has been automatically allocated to the instance during this time. We are taking steps to optimize the resource availability by allocating further resources.</p>tag:status.upstash.com,2005:Incident/163131822023-03-01T20:41:37Z2023-03-01T20:41:37ZDegraded performance on Kafka eu-west-1 cluster<p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>20:41</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>19:19</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the cluster</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>11:15</var> UTC</small><br><strong>Identified</strong> - We have identified the problem, waiting for a resolution from our cloud provider.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>10:44</var> UTC</small><br><strong>Investigating</strong> - We are experiencing a cloud provider related problem at this moment. Investigating</p>tag:status.upstash.com,2005:Incident/148519712022-12-13T08:48:43Z2022-12-13T08:48:43ZDatabase Backup/Restore Performance Degradation on AWS US-EAST-1 Cluster<p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>08:48</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>19:40</var> UTC</small><br><strong>Investigating</strong> - We have observed degraded performance on database backup/restore operations on AWS US-EAST-1 cluster. The functionality is disabled temporarily</p>tag:status.upstash.com,2005:Incident/130634342022-11-16T08:30:30Z2022-11-16T08:30:31ZQStash Scheduled Maintenance<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>08:30</var> UTC</small><br><strong>Update</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>08:30</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>07:31</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>15:31</var> UTC</small><br><strong>Scheduled</strong> - We will be undertaking scheduled maintenance of our QStash Cluster on November 16th 07:30 to 08:30 UTC. We apologize for any disruption, and will endeavor to keep it to a minimum.</p>tag:status.upstash.com,2005:Incident/117913362022-10-11T08:30:38Z2022-10-11T08:30:38ZUS Kafka Scheduled Maintenance<p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>08:30</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>07:30</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>08:48</var> UTC</small><br><strong>Scheduled</strong> - We will be undertaking scheduled maintenance of our US Kafka Cluster on October 11th 07:30 to 08:30 UTC. We apologize for any disruption, and will endeavor to keep it to a minimum. If you have a single zone cluster deployment you might experience unresponsiveness for a short period of time (a few seconds).</p>tag:status.upstash.com,2005:Incident/116190962022-10-10T08:31:28Z2022-10-10T08:31:28ZEU Kafka Scheduled Maintenance<p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>08:31</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>07:30</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>16:42</var> UTC</small><br><strong>Scheduled</strong> - We will be undertaking scheduled maintenance of our EU Kafka Cluster on October 10th 07:30 to 08:30 UTC. We apologize for any disruption, and will endeavor to keep it to a minimum. If you have a single zone cluster deployment you might experience unresponsiveness for a short period of time (a few seconds). There will be no downtime with multi-zone deployments.</p>tag:status.upstash.com,2005:Incident/77192922021-08-05T11:13:00Z2021-08-09T11:13:38ZUS-EAST-1 Free Tier Rest Service Outage<p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>11:13</var> UTC</small><br><strong>Resolved</strong> - There was a partial outage in the REST service in one of the us-east free tier clusters</p>tag:status.upstash.com,2005:Incident/77192222021-08-04T11:02:00Z2021-08-09T11:02:44ZUS-EAST-1 Free Tier Rest Service Outage<p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>11:02</var> UTC</small><br><strong>Resolved</strong> - There was a partial outage in the REST service in one of the us-east free tier clusters</p>tag:status.upstash.com,2005:Incident/66298042021-03-27T21:22:34Z2021-05-23T20:04:52ZThis is an example incident<p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>21:22</var> UTC</small><br><strong>Resolved</strong> - Empathize with those affected and let them know everything is operating as normal.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>21:12</var> UTC</small><br><strong>Monitoring</strong> - Let your users know once a fix is in place, and keep communication clear and precise.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>21:18</var> UTC</small><br><strong>Identified</strong> - As you continue to work through the incident, update your customers frequently.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>21:48</var> UTC</small><br><strong>Investigating</strong> - When your product or service isn’t functioning as expected, let your customers know by creating an incident. Communicate early, even if you don’t know exactly what’s going on.</p>