No issues have been observed since the overnight background service queue delays on 2/6, and all overnight queues on 2/7 performed as expected. We will continue to monitor background services to ensure ongoing operation within desired ranges.
Posted Feb 07, 2024 - 19:57 EST
Update
As of earlier this afternoon, fewer than 1% of databases had ongoing overnight jobs still executing. The remaining databases have jobs that typically take hours overnight to complete, and with the accumulation of daytime jobs and blocking processes on their databases, are continuing to run. Separately, we have adjusted the background process monitoring to better recover from a scenario in which the Redis servers become temporarily unavailable due to hardware or software updates to those nodes.
Posted Feb 06, 2024 - 17:14 EST
Monitoring
An AWS-initiated overnight update to a series of Redis servers used for background service queue management has delayed background queue processing as the queues entered a stalled state. Delayed queues are processing now and will complete shortly.