StatusCast’s engineers were alerted that some SMS and email
notifications had an unusual delay in their delivery. The team immediately
began investigating and determined that one of the notification processors was
experiencing a failure that resulted in a queued backup of notifications. This
did not impact all notifications sent through StatusCast, only a subset that
were assigned to the instance in question.
Once the instance was isolated the backup of requests were
offloaded and the instance itself taken out of rotation. Even though the
instance itself did not enter an unhealthy state our engineers have
re-evaluated certain health checks to account for queue backups as well as other
performance attributes. Additionally, our engineers have scaled out the number
of instances to help reduce the load on any single processor at a given time. At
this time all notification services are running as expected but we will
continue to monitor the system closely.
