Fortifying your code with the Backoff Python package
Recently, I have been spending lots of time working with both commercial and internal APIs as part of larger systems.
One issue I kept running into was temporary failures. A network call can fail because the service is temporarily down, a VPN disconnects, a quota is exceeded, or for no obvious reason. In long-running processes, one of these failures can stop the whole job.
The backoff Python package has been useful discovery for handling these failures without repeating the same retry logic in every API call.
Why Backoff?
Before discovering backoff, I frequently found myself writing awkward loops to handle repeated API call attempts. These loops looked something like this:
while True:
try:
response = make_api_call()
if response.status_code == 200:
break
except NetworkError:
continue
except QuotaError:
sleep(60)
continue
except ServerError:
logger.error()This loop would endlessly attempt the call until it succeeded, offering support for some simple error management. This was effective but inelegant, verbose and hard to maintain. Moreover, it lacked any form of exponential delay, which meant it could overwhelm the network with retries during an outage or temporary failure.
backoff automates retries while incorporating exponential backoff and jitter without the clunky, error-prone logic.
It replaced my loop with much cleaner logic and only required a few lines of code.
Integrating Backoff with API Calls
Let’s dive into some examples illustrating how backoff enhances API call reliability.
Example #1: Basic API Call Handling
Consider a simple API call to a weather service, where we might occasionally encounter a failure due to rate limits or connectivity issues.
Here’s how you might wrap that call using backoff:
import backoff
import requests
@backoff.on_exception(backoff.expo, requests.exceptions.RequestException, max_tries=5)
def get_weather_data(city):
response = requests.get(
f'http://api.weather.com/v3/wx/forecast/daily/5day?city={city}',
timeout=10,
)
response.raise_for_status()
return response.json()
weather_data = get_weather_data("New York")With the @backoff decorator, the call to get_weather_data will automatically implement exponential backoff on encountering a RequestException, retrying up to 5 times.
Example #2: Handling Outages and API Limits
In this scenario, we’ll implement a constant wait strategy to handle HTTP error statuses, such as 429 (Too Many Requests) or 503 (Service Unavailable):
@backoff.on_predicate(
backoff.constant,
predicate=lambda response: response.status_code in {429, 503},
interval=5,
max_tries=8,
jitter=None,
)
def fetch_user_data(user_id):
return requests.get(f'http://api.example.com/users/{user_id}', timeout=10)
response = fetch_user_data(12345)
response.raise_for_status()
user_data = response.json()Here, we use a constant delay of 5 seconds between retries, making it well-suited for managing rate limits or temporary service unavailability.
Example #3: Usage with asyncio
Now, let’s explore a more real-world example using asyncio to perform 50 concurrent API calls, each with its own backoff strategy:
import asyncio
import backoff
import aiohttp
@backoff.on_exception(backoff.expo, aiohttp.ClientError, max_tries=5)
async def async_get_user_data(session, user_id):
async with session.get(f'http://api.example.com/users/{user_id}', timeout=10) as response:
response.raise_for_status()
return await response.json()
async def main():
async with aiohttp.ClientSession() as session:
tasks = [async_get_user_data(session, user_id) for user_id in range(50)]
results = await asyncio.gather(*tasks)
return results
# Running the coroutine
user_data_list = asyncio.run(main())This example demonstrates how to integrate backoff in an asyncio context to handle multiple concurrent API calls.
By utilizing aiohttp for asynchronous requests, we efficiently manage potential network issues without blocking the entire program, retrying each failed request up to 5 times with exponential backoff.
We can also achieve this without using backoff by setting return_exceptions=True in asyncio.gather. However, this would require us to write custom code to retry the functions that encountered errors. Using backoff makes this much simpler.
Additional features
The backoff package offers several decorators for handling retries and backoff strategies.
These are well documented on the backoff GitHub repository, but to sum them up briefly:
@backoff.on_exceptionretries a function when a specified exception is raised.@backoff.on_predicateretries a function based on its return value.@backoff.runtimeuses the return value or thrown exception of the decorated method to determine backoff behavior.
backoff also allows the use of multiple decorators on a single function, providing more precise control over error handling behavior.
Jitter
backoff supports adding randomness to backoff intervals to prevent thundering herd problems.
Default jitter function backoff.full_jitter implements the ‘Full Jitter’ algorithm, which is nicely explained in this AWS blog post
Conclusion
The backoff package has been useful for making long-running processes less likely to fail because of temporary API or network issues.
Instead of maintaining retry loops in every API call, the retry policy stays next to the function that needs it. The decorators make it easy to choose which errors to retry, how long to wait, and when to stop.
This is especially useful when making many requests, either synchronously or asynchronously. A temporary failure in one request does not have to interrupt the whole process.
Retries should still be configured carefully. The right delay, limit, and retry conditions depend on the API and the operation being performed.
Notes
Best Practices in Using Backoff
- Exponential Backoff & Jitter: While constant waits might make sense in some cases, exponential backoff with added jitter (randomness) is usually the best practice for network operations to prevent thundering herd problems. Be careful when configuring this, it should be taken care of in a per-function basis, because the backoff needed for a network error is not the same needed for a rate-limit error.
- Logging: Make sure to log backoff retries and integrate them into your monitoring tools to catch systemic issues early.
- Understand Max Retries: Opt for a reasonable balance between retry aggressiveness and the user experience, as excessive retries could mask real issues or delay error notification.
- Retry Safe Operations: Retrying
GETrequests is usually safe. For requests that modify data, such asPOSTor payments, use idempotency keys or an API-specific strategy to avoid performing the operation twice.