[closed] At approximately 9:40AM EDT StatusCast engineers were alerted to errors on the application that were preventing users from accessing both their status page as well as the administrative portal. StatusCast’s engineers have determined a potential issue with its service provider Azure and is currently working with Microsoft to diagnose and resolve the issue. 

At this time services have been restored and should be operating as normal. If you continue to have any issues please contact support@statuscast.com to open a ticket. We will follow to this event with an RCA detailing what occurred and how we will handle this moving forward. 

Describe the full incident details below:

On July 21st, 2023 at approximately 9:40 EDT StatusCast’s engineers received alerts that the application was displaying a  HTTP Error 500.30 error when attempting to access any *.status.page status page or admin portal. During this period any notifications in progress or from schedule maintenance would have continued to work as expected. Additionally, during this period anyone using StatusCast’s legacy(*.statuscast.com) version of the application was not impacted. 

Describe action taken by StatusCast to mitigate issue:

Engineers immediately began to investigate the cause of the problem. StatusCast’s service provider, Azure, indicated that it was undergoing maintenance in the region that StatusCast’s is primarily hosted on(US East). Engineers got in contact with Microsoft to confirm and to get additional insight as the issue itself was impacting the failover region(US West). During this process StatusCast deployed an additional instance to another Azure region which experienced the same errors as both East and West.

The root cause of the problem ultimately was related to Azure’s maintenance and the availability of one of StatusCast’s databases used for managing connections to the application. Leading up to the outage StatusCast’s operations team was preparing for its monthly penetration test which regularly involves a fresh test database for a reserved test application. The updated connection was not properly propagated to all of StatusCast’s application servers and traffic manager which unfortunately caused the subsequent errors. 

Once the issue had been identified StatusCast’s engineers were quickly able to restore service. StatusCast development team will be performing an emergency patch today(July 21st, 2023) to ensure that an issue like this can be caught without the application going unavailable. 

 

[closed]

StatusCast engineers have detected a possible performance impacting event affecting status pages and the admin application. This event is not impacting notification processing. We apologize for this inconvenience and will provide an update shortly. 

This event has been resolved. 

[closed] StatusCast replaced its vendor for managing support chats within the administrative application. In an effort to improve the efficiency of StatusCast's Suppor team, we have replaced Intercom with Zendesk which we had been previously using to log support tickets. Just as with Intercom, Zendesk does not have access to your subscribers or to your incident data, but does retain some general usage details to help Support in diagnosing and reviewing accounts. For an updated copy of StatusCast's DPA please reach out to support@statuscast.com

[closed]

The StatusCast team will be performing a maintenance on February 17, 6:00am EST, the estimated duration is 60. We do not expect any impact to your service but in some cases there may be a brief interruption.

StatusCast's maintenance has been completed. All services should be operational, if you encounter any issues please contact StatusCast's support team at support@statuscast.com.

[closed]

StatusCast's engineers have been alerted that some users while attempting to access their status page and/or administrative portal experienced significant delays in response time and in some cases the application would not load at all. We are working to diagnose and resolve this issue ASAP and will provide updates as available. 

Engineers have confirmed that this morning StatusCast experienced an unexpected significant spike in traffic that effected response time for many users. In some cases, occasional timeouts were reported when loading status pages as well. We have scaled our servers temporarily while we investigate the root cause of the spike and will be mitigating for long term scalability.

At this time all services should be operating as expected and we will follow-up with a detailed RCA once our investigation is concluded

On November 17th at approximately 9:45AM EST StatusCast experienced a tremendous spike in inbound traffic(over 3x our historical max) which caused the primary caching mechanism for the application to become overloaded. This caused many connection requests to the application to experience either major delays in page loads or complete time outs.  During this time StatusCast’s own status page was also affected; not allowing for customers to check in regarding the status of the service and the actions being taken.

Engineers mitigated the issue by scaling out the service and performing an emergency flush of the caching system in order to restore service while investigating the source of traffic spike. 

Once the system had been fully restored engineers continued their investigation into the traffic spike and determined that it was not malicious in nature. The engineering and development teams have spent the last 24 hours making and preparing the following changes to StatusCast’s service offering:

1. Permanently scaled up the resource baseline for all of StatusCast’s servers

2. Added additional servers into the pool of servers used to maintain the application 

3. Revisited auto-scaling rules around resource baselines for auto-mitigation purposes

4. Planned caching updates for StatusCast’s December release that will aid in caching resource constraints

5. Migrated StatusCast’s own page to an environment that is totally separate from the production space that clients are deployed to.

StatusCast’s team will continue to monitor both the health of its service offerings and analyze traffic patterns in order to gauge if additional changes to its infrastructure are necessary.

[closed]

StatusCast engineers were alerted to an issue affecting some users access to the status.page and admin version of the application resulting in slow load times or pages to time out.

Engineers identified certain servers within its rotation had encountered memory issues and were able to resolve the issue. An RCA will follow this update. An RCA will follow this update.

At this point service should be operating as expected for all users, however if you continue to experience any issues please contact support@statuscast.com.

At 2:17pm EST, StatusCast engineers were alerted to an issue that resulted in some customers experiencing slow load times and page time outs when accessing status pages and admin portals. Engineers discovered we had an major spike in traffic and our servers existing scaling logic was not able to keep up which resulted in maxing out a couple of our resources. Once the engineers have determined the caused, the influx had resolved itself at 2:42pm EST, and returned to normal. As a temporary solution, we have scaled out the service to handle additional traffic spikes. However as a more permanent solution, additional automated scaling rules have been implemented to allow the application to handle its traffic spikes such as the one experienced today.

[closed]

StatusCast's engineers have been alerted that some users are experiencing latency when attempting to access their status page as well as their administrative portal. At this time this latency does not appear to be impacting all users.

Engineers are working to resolve this now and we will post an update shortly when more information is available. 

At this time all services should be operating as expected. If you continue to experience any further issues please reach out to StatusCast support at support@statuscast.com.

We will follow up with additional information related to the root cause of this latency at a later time. 

[closed] StatusCast’s engineers were alerted that our hosting provider; Azure, starting at approximately 21:30 UTC on 01 Apr 2021 started experiencing global issues with certain downstream services that StatusCast utilize.  As a result, some users were unable to connect directly to their status page or administrative portal.   Our engineers implemented fail-over procedures and at this time the application is available for all users. We will continue to monitor the situation with Azure, for more information related to Azure specifically please visit their status page here. 

Microsoft remediated their DNS issues fairly quickly and StatusCast customers had minimal impact.

[closed] StatusCast’s engineers were alerted that some users were experiencing time conversion issues after daylight savings had become active, resulting in some posts being scheduled 1 hour off from the intended date/time. After a thorough investigation it was determined that the local browser cache for affected users was not being invalidated as expected and as a result the time submissions were being evaluated with a different time offset for ‘Standard’ time.  After discovering this cause our engineers pushed out a cache invalidation to force all users connecting to the admin portal to completely reload the appropriate bundles. Most users were unaffected by this issue as it was localized to an individual’s browser but we are re-evaluating how caching of certain libraries work in the administrative portal to ensure this does not happen again in the future.

[closed] The StatusCast team will be performing a maintenance on Sunday December 27th, 2020 6:00:00 AM EDT, the estimated duration is 1 hour. We do not expect any impact to your service but in some cases there may be a brief interruption.

This maintenance has been completed. StatusCast has added 28 new cloud service providers, for a full list of providers that have been added please access your account's changelog from the admin portal.