Users experiencing timeouts on Status Pages

Incident
October 24, 6:55pm EDT

Users experiencing timeouts on Status Pages

Status: closed
Start: October 24, 2:17pm EDT
End: October 24, 2:42pm EDT
Duration: 25 minutes
Affected Components:
Status pages Admin application
Update

October 24, 2:17pm EDT

October 24, 2:17pm EDT

StatusCast engineers were alerted to an issue affecting some users access to the status.page and admin version of the application resulting in slow load times or pages to time out.

Resolved

October 24, 2:42pm EDT

October 24, 2:42pm EDT

Engineers identified certain servers within its rotation had encountered memory issues and were able to resolve the issue. An RCA will follow this update. An RCA will follow this update.

At this point service should be operating as expected for all users, however if you continue to experience any issues please contact support@statuscast.com.

Root Cause

October 24, 6:55pm EDT

October 24, 6:55pm EDT

At 2:17pm EST, StatusCast engineers were alerted to an issue that resulted in some customers experiencing slow load times and page time outs when accessing status pages and admin portals. Engineers discovered we had an major spike in traffic and our servers existing scaling logic was not able to keep up which resulted in maxing out a couple of our resources. Once the engineers have determined the caused, the influx had resolved itself at 2:42pm EST, and returned to normal. As a temporary solution, we have scaled out the service to handle additional traffic spikes. However as a more permanent solution, additional automated scaling rules are being evaluated to allow the application to handle its traffic spikes such as the one experienced today.