On November 17th at approximately 9:45AM EST StatusCast experienced a tremendous spike in inbound traffic(over 3x our historical max) which caused the primary caching mechanism for the application to become overloaded. This caused many connection requests to the application to experience either major delays in page loads or complete time outs.  During this time StatusCast’s own status page was also affected; not allowing for customers to check in regarding the status of the service and the actions being taken.

Engineers mitigated the issue by scaling out the service and performing an emergency flush of the caching system in order to restore service while investigating the source of traffic spike. 

Once the system had been fully restored engineers continued their investigation into the traffic spike and determined that it was not malicious in nature. The engineering and development teams have spent the last 24 hours making and preparing the following changes to StatusCast’s service offering:

1. Permanently scaled up the resource baseline for all of StatusCast’s servers

2. Added additional servers into the pool of servers used to maintain the application 

3. Revisited auto-scaling rules around resource baselines for auto-mitigation purposes

4. Planned caching updates for StatusCast’s December release that will aid in caching resource constraints

5. Migrated StatusCast’s own page to an environment that is totally separate from the production space that clients are deployed to.

StatusCast’s team will continue to monitor both the health of its service offerings and analyze traffic patterns in order to gauge if additional changes to its infrastructure are necessary.

At this time all services should be operating as expected, if you continue to experience any further issues please reach out to StatusCast support at support@statuscast.com.

We will follow up with additional information related to the root cause of this latency at a later time. 

StatusCast’s engineers were alerted to an issue affecting some customers accessing the status.page version of the application. Engineers confirmed that a certificate renewal was not properly propagated to all servers. This did not impact customers utilizing the statuscast.com domain or those utilizing a custom domain name.

Once we were made aware of this issue the updated certificate was pushed out directly to all instances. At this point service should be operating as expected for all users, however if you continue to experience any issues please contact support@statuscast.com. 

StatusCast's engineers were alerted that schedule maintenance events created from StatusCast's legacy application("V2") were not properly auto-closing after their estimated duration had been reached. After an initial investigation engineers have confirmed the cause on the service responsible and a patch was performed to correct the error. Any maintenance that was overdue for closure should have been resolved and StatusCast's engineers will continue to monitor the legacy process for this to ensure no other issues occur.