Increased latency and connection stability

Incident Report for Technolutions

Resolved

On Wednesday afternoon, beginning around 3:40pm Eastern Time, a routine load balancer re-synchronization resulted in an unbalanced traffic distribution that increased the latency of some connections and returned a TCP RST status for some others, while still successfully processing and handling yet other connections depending upon their original node affinity. We have determined that this unbalanced traffic distribution was the result of an issue in the load balancers that reset node statuses during a re-synchronization. We have resolved the issue with the load balancers to prevent this from reoccurring. By 4:00pm Eastern Time, traffic had been fully redistributed across nodes and connection performance had returned to expected levels.

Posted Jan 29, 2025 - 16:00 EST