Why we need exponential backoff with jitter when retrying

PublishedDecember 27, 2024

•4 min read•View as Markdown

Why we need exponential backoff with jitter when retrying

Tiger Abrodi

Just a guy who loves to write code and watch anime.

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog

Become a cracked product engineer today

Qualities that make outstanding builders.

Jun 1, 20264 min read

Bipedal, humanoid, and the words for creature shapes in games

Bipedal vs quadrupedal: how many legs This is about the number of legs the creature walks on. Bipedal. Two legs. Humans, ostriches, T-rex, kangaroos, most fantasy humanoids. Quadrupedal. Four legs. Do

May 18, 20263 min read

Bipedal, humanoid, and the words for creature shapes in games

Lerp and smoothstep, what they actually do

Intro You hear "lerp" and "smoothstep" everywhere in game dev. They sound like math jargon. They're not. Both are small tools that do the same job: smoothly move from one value to another. The problem

May 18, 20264 min read

Lerp and smoothstep, what they actually do

IK and FK, what they actually are

What is Kinematics Kinematics is the math field. It's the study of how things move without worrying about forces (which would be dynamics). FK and IK are the two branches: forward kinematics and inver

May 10, 20267 min read

How to make your textures fast on the GPU

Intro Textures are usually the biggest cost in a 3D scene. Memory, bandwidth, and load time all get eaten by them. Resizing them is the obvious lever. There's more. This post is about the less obvious

May 10, 20268 min read

How to make your textures fast on the GPU

Tiger's Place

497 posts

The problem

You want to retry an operation e.g. a network request. If you spam requests, you will overload the server and end up getting rate limited (429 too many requests).

This leads us to exponential backoff first. For every retry, we wait longer.

Note: You shouldn't retry for every type of error. If it's a 4xx error, it's a bad request, so retrying the same request won't help.

You can think of this as multiplying the wait time by a constant factor.

1st retry: 1 second

2nd retry: 2 seconds

3rd retry: 4 seconds

4th retry: 8 seconds

However, this is not enough. It's good, but we could do better.

If you think about it, if we have 1000 clients all make API requests at the same time, they will all retry at the same time.

1st retry: 1 second - 1000 requests

2nd retry: 2 seconds - 1000 requests

3rd retry: 4 seconds - 1000 requests

4th retry: 8 seconds - 1000 requests

This creates a new traffic spike which can overwhelm the server. Mind you, it could easily be more than 1000 requests at the same time in the real world.

This problem is called the thundering herd problem. It's a weird naming but stems from nature where herds stampede when they hear thunder. Stampede means a large crowd rushes to a place. Anyways, the naming isn't important here lol.

The solution to not retry at the same time is to add jitter. Jitter means add an element of randomness to the retry times. One request might retry after 1.5 seconds, while another might retry after 2.5 seconds.

Code

// Types for our retry configuration interface RetryConfig { // Base delay in milliseconds e.g. 1000ms = 1 second baseDelay: number; // Maximum delay in milliseconds e.g. 30000ms = 30 seconds maxDelay: number; // Maximum number of retry attempts e.g. 5 maxRetries: number; // Factor to determine jitter range (0-1) e.g. 0.2 jitterFactor: number; } // Calculate delay with exponential backoff and jitter function calculateBackoffDelay(attempt: number, config: RetryConfig): number { // Calculate exponential backoff // Here Math.min is used to ensure we don't exceed maxDelay // This is where we calculate the exponential delay without jitter // config.baseDelay * Math.pow(2, attempt) is the key // e.g. attempt 2: 1000 * 2^2 = 4000ms = 4 seconds // Math.pow will be 2^{attempt} // For 4th attempt: 2^4 = 16 -> 2 * 2 * 2 * 2 = 16 // This ensures the delay grows exponentially const exponentialDelay = Math.min( config.maxDelay, config.baseDelay * Math.pow(2, attempt) ); // Add random jitter // e.g. 4000ms * 0.2 = 800ms // Math.random() generates a random floating point number between 0 and 1 (will never be 1) // e.g. 0.5 * 800ms = 400ms // This ensures that the jitter is a random amount between 0 and 800ms const jitterRange = exponentialDelay * config.jitterFactor; const jitter = Math.random() * jitterRange; // Add the jitter to the exponential delay // e.g. 4000ms + 400ms = 4400ms // For every retry this would be different return Math.floor(exponentialDelay + jitter); } async function retryWithBackoff<T>( operation: () => Promise<T>, config: RetryConfig ): Promise<T> { // Keep track of how many attempts we've made let attempt = 0; while (true) { try { // To be clear, return stops the loop return await operation(); } catch (error) { attempt++; // If we've made too many attempts, throw an error if (attempt >= config.maxRetries) { throw new Error(`Failed after ${attempt} attempts: ${error}`); } // Calculate the delay const delay = calculateBackoffDelay(attempt, config); console.log(`Attempt ${attempt} failed. Retrying in ${delay}ms...`); // Wait for the delay // Here we can't use .then or just setTimeout // Because they don't "wait", mimicing a sync flow // "await" makes it wait before we dive into the next retry await new Promise((resolve) => setTimeout(resolve, delay)); } } } // Example usage async function main() { const retryConfig: RetryConfig = { baseDelay: 1000, // Start with 1 second delay maxDelay: 30000, // Max 30 seconds delay maxRetries: 5, // Try up to 5 times // Moderate jitter factor is around 0.2-0.3 // Depending on your use case, you might want to tweak this // Higher jitter factor means more randomization jitterFactor: 0.2, // Add up to 20% random jitter }; try { const result = await retryWithBackoff(makeApiRequest, retryConfig); console.log("Final result:", result); } catch (error) { console.error("All retry attempts failed:", error); } }

Command Palette

Comments

More from this blog

The problem

Code