StatusCast’s engineers were alerted that some SMS and email notifications had an unusual delay in their delivery. The team immediately began investigating and determined that one of the notification processors was experiencing a failure that resulted in a queued backup of notifications. This did not impact all notifications sent through StatusCast, only a subset that were assigned to the instance in question.

Once the instance was isolated the backup of requests were offloaded and the instance itself taken out of rotation. Even though the instance itself did not enter an unhealthy state our engineers have re-evaluated certain health checks to account for queue backups as well as other performance attributes. Additionally, our engineers have scaled out the number of instances to help reduce the load on any single processor at a given time. At this time all notification services are running as expected but we will continue to monitor the system closely.