How Uber Protects Against Retry Storms

Uber has implemented various measures to protect against retry storms, which occur when a large number of requests are made to a system in a short amount of time, overwhelming it and causing downtime. To mitigate this issue, Uber uses a circuit breaker pattern to detect and prevent excessive requests, limits the rate of requests, and uses a technique called 'retry backoff' to space out retries. Additionally, Uber's system is designed to handle retry storms by detecting when a service is under attack and temporarily blocking the offending IP addresses. The company also utilizes a 'rate limiter' to limit the number of requests from a single IP address within a certain time window.