Skip to content

Retry

retry calls the function again when it raises, so transient failures (a dropped connection, a 503) do not reach your caller. The last exception is re-raised unchanged, with its original traceback, once attempts run out.

from python_wrap import retry

calls = []

@retry(max_attempts=4, delay=0.01, backoff=2, on=ConnectionError)
def fetch() -> str:
    calls.append(1)
    if len(calls) < 3:
        raise ConnectionError("down")
    return "payload"

assert fetch() == "payload"
assert len(calls) == 3

import asyncio
from python_wrap import retry

calls = []

@retry(max_attempts=4, delay=0.01, backoff=2, on=ConnectionError)
async def fetch() -> str:
    calls.append(1)
    if len(calls) < 3:
        raise ConnectionError("down")
    return "payload"

assert asyncio.run(fetch()) == "payload"
assert len(calls) == 3

retry works bare (@retry), with options, and as a plain call: retry(func, max_attempts=5).

When to use it

  • Errors that are likely to disappear on their own: dropped connections, 429 and 5xx responses, lock contention.
  • Operations that are safe to repeat.
  • Not for a dependency that stays down: add a circuit breaker so retries stop piling onto it.
  • Not for a call that is slow rather than failing: use timeout, or hedge for tail latency.

Retry only what is safe to repeat

Retrying a non-idempotent call (charging a card, sending an email) can do it twice. Narrow on= to errors that mean "it did not happen", and use retry_if to inspect the exception.

Options

The wait before attempt n + 1 is min(delay * backoff ** (n - 1), max_delay) + random.uniform(0, jitter).

Option Default Meaning
max_attempts 3 Total calls allowed. None retries forever.
delay 1.0 Base wait in seconds.
backoff 1.0 Multiplier applied after each failure (use 2 for exponential).
max_delay None Cap on the backoff wait (before jitter).
jitter 0.0 Up to this many random extra seconds, so many clients do not retry in lockstep.
max_time None Total time budget in seconds from the first attempt.
on Exception Exception type(s) that trigger a retry. Anything else propagates immediately.
retry_if None Extra predicate on the exception; retry only if it returns True. May be async def for async functions.
retry_if_result None Predicate on the return value; True means "retry". The last value is returned when retries run out. May be async def for async functions.
on_retry None Called with a RetryState before each wait. May be async def for async functions.
delay_from None exc -> seconds or None: exact wait for this attempt (e.g. Retry-After).
budget None A RetryBudget shared with other retried code, so retries stay a fraction of traffic.
logger package logger Logger or logger name for the per-retry message.
log_level WARNING Level of the per-retry message; None disables it.
log_exc_info False Attach the traceback to the per-retry log record.

Choosing what to retry

on picks exception types; retry_if looks inside the exception. on_retry receives a RetryState with attempt, exception, next_delay and elapsed, which is enough to feed a metric or a custom log line.

from python_wrap import RetryState, retry

class HttpError(Exception):
    def __init__(self, status: int) -> None:
        super().__init__(f"HTTP {status}")
        self.status = status

seen: list[RetryState] = []
responses = iter([503, 502, 200])

@retry(
    max_attempts=5,
    delay=0.01,
    on=HttpError,
    retry_if=lambda exc: exc.status >= 500,
    on_retry=seen.append,
)
def get() -> int:
    status = next(responses)
    if status != 200:
        raise HttpError(status)
    return status

assert get() == 200
assert [s.attempt for s in seen] == [1, 2]
assert isinstance(seen[0].exception, HttpError)

Retrying on a returned value

Some APIs report "not ready yet" in the return value instead of raising: a job status of "pending", an empty page, None. retry_if_result receives each returned value; when it returns True the attempt counts as failed and is retried with the usual backoff.

from python_wrap import retry

statuses = iter(["pending", "pending", "done"])

@retry(max_attempts=5, delay=0.01, retry_if_result=lambda status: status == "pending")
def job_status() -> str:
    return next(statuses)

assert job_status() == "done"

When no retry is left (attempts, max_time, deadline or budget), the last value is returned, not an exception: the function did not fail, it just never gave the answer you hoped for. Check the value if it matters:

from python_wrap import retry

@retry(max_attempts=3, delay=0.01, retry_if_result=lambda status: status == "pending")
def job_status() -> str:
    return "pending"

assert job_status() == "pending"  # three attempts, then the last value

retry_if_result combines with on/retry_if: exceptions and bad values share the same attempt count. In the RetryState passed to on_retry, exception is None and result holds the rejected value; the retry.attempt_failed and retry.gave_up events carry exception=None. An exception raised by the predicate itself propagates at once.

Why not raise?

tenacity raises RetryError when the result predicate never passes; the backoff library returns the last value. python-wrap returns it, so the decorated function keeps its return type and you never have to unwrap a wrapper exception. If you prefer an error, raise one yourself in the function instead of using retry_if_result.

Async hooks

For async def functions, retry_if, retry_if_result and on_retry may be async def callables: they are awaited, so they can look something up or report without blocking the loop. Plain functions keep working.

import asyncio
from python_wrap import RetryState, retry

reported: list[int] = []

async def is_transient(exc: BaseException) -> bool:
    await asyncio.sleep(0)  # e.g. ask a feature flag service
    return isinstance(exc, ConnectionError)

async def report(state: RetryState) -> None:
    await asyncio.sleep(0)  # e.g. send a metric
    reported.append(state.attempt)

errors = iter([ConnectionError("down")])

@retry(max_attempts=3, delay=0.01, retry_if=is_transient, on_retry=report)
async def fetch() -> str:
    err = next(errors, None)
    if err is not None:
        raise err
    return "payload"

assert asyncio.run(fetch()) == "payload"
assert reported == [1]

Nothing can await an async hook around a plain function, so decorating a sync function with one raises TypeError straight away:

from python_wrap import retry

async def is_transient(exc: BaseException) -> bool:
    return True

try:
    @retry(retry_if=is_transient)
    def fetch() -> str:
        return "payload"
except TypeError as exc:
    assert "retry_if= is an async function" in str(exc)

Retry budgets

max_attempts=3 on every call looks harmless until the dependency goes down: every request now makes three, and the retries keep it down. A RetryBudget caps retries at a fraction of the traffic, shared by every function that uses it (the Finagle and Envoy approach):

  • every call's first attempt deposits into the budget, every retry withdraws;
  • a retry is allowed while the retries in the last window seconds stay within ratio times the requests in that window, plus min_per_second * window so low-traffic code can still retry;
  • when the budget refuses, retry stops at once: no sleep, the last exception is re-raised (or the last value returned with retry_if_result), and retry.budget_exhausted is emitted.
from python_wrap import retry
from python_wrap.retry import RetryBudget, retry_stats

# Retries may add 10 % to traffic, and at least 1 retry per second is always allowed.
budget = RetryBudget(ratio=0.1, min_per_second=1, window=10)
calls = []

@retry(max_attempts=5, delay=0, on=ConnectionError, budget=budget)
def get_user() -> dict[str, str]:
    calls.append(1)
    raise ConnectionError("users service is down")

@retry(max_attempts=5, delay=0, on=ConnectionError, budget=budget)
def get_orders() -> list[str]:
    calls.append(1)
    raise ConnectionError("orders service is down")

for _ in range(10):
    for func in (get_user, get_orders):
        try:
            func()
        except ConnectionError:
            pass

# 20 calls; without the budget they would have made 100 attempts.
assert len(calls) == 20 + budget.stats.retries
assert budget.stats.retries <= 0.1 * 20 + 1 * 10
stats = retry_stats(get_user)
assert stats is not None and stats.budget_exhausted > 0
Argument Default Meaning
ratio 0.1 Retries allowed per request in the window.
min_per_second 10 Retries per second always allowed, whatever the traffic.
window 10.0 Length of the sliding window in seconds.

budget.stats returns a RetryBudgetStats with requests and retries (in the current window), rejected (refused since creation or the last reset()) and allowed (retries it would grant right now). One budget is thread-safe and can be shared across threads, event loops, retry decorators and retrying blocks; it never blocks or sleeps. To give every retry in the project the same budget, set it once with configure(retry={"budget": RetryBudget()}) before decorating; pass budget=None to opt a function out.

One budget per dependency

Share a budget between the calls that hit the same service. A budget shared by unrelated services lets a failing one spend the retries the others need.

Honouring Retry-After

Servers that answer 429 or 503 often say how long to wait. Pass delay_from a function that maps the exception to a number of seconds, or to None to fall back to the normal backoff. A number is the exact wait for that attempt (no backoff, no jitter), still capped by max_delay and max_time.

retry_after_header reads exc.retry_after or exc.response.headers["Retry-After"] (seconds or an HTTP-date) and never raises, so it works with httpx.HTTPStatusError, requests.HTTPError and similar without importing them. The example uses a stand-in exception with the same shape:

from python_wrap import retry
from python_wrap import retry_after_header

class Response:
    def __init__(self, headers: dict[str, str]) -> None:
        self.headers = headers

class HTTPStatusError(Exception):  # shaped like httpx / requests errors
    def __init__(self, headers: dict[str, str]) -> None:
        super().__init__("429 Too Many Requests")
        self.response = Response(headers)

replies = iter([{"Retry-After": "0.01"}, {}, None])

@retry(on=HTTPStatusError, delay=0.01, delay_from=retry_after_header, max_delay=30)
def fetch() -> str:
    headers = next(replies)
    if headers is not None:
        raise HTTPStatusError(headers)
    return "payload"

assert fetch() == "payload"

Any function works as delay_from, e.g. delay_from=lambda exc: getattr(exc, "wait", None). If it raises, the normal backoff is used.

Time budget

max_time is a total budget in seconds, measured from the first attempt. retry never sleeps past it: if the next wait would end after the deadline, the last exception is re-raised at once.

from python_wrap import retry

calls = []

@retry(max_attempts=None, delay=0.01, backoff=2, max_time=0.05)
def flaky() -> None:
    calls.append(1)
    raise ConnectionError("down")

try:
    flaky()
except ConnectionError:
    pass
assert len(calls) >= 2

retry also honours an enclosing deadline: it never sleeps past remaining() (from python_wrap.deadline). If the next wait would outlast the deadline, or none is left, the last exception is re-raised at once (the same rule as max_time).

from python_wrap import retry
from python_wrap.deadline import deadline

calls = []

@retry(max_attempts=None, delay=0.02)
def flaky() -> None:
    calls.append(1)
    raise ConnectionError("down")

with deadline(0.05):
    try:
        flaky()
    except ConnectionError:
        pass
assert 1 <= len(calls) <= 4

Retrying a block of code

retrying(...) takes the same options as retry and lets you retry code that is not a function of its own. Put the risky part in with attempt:; a retryable exception is swallowed and the loop waits and continues, success ends the loop, and the last failure propagates. attempt.number is the 1-based attempt number.

from python_wrap.retry import retrying

tries = []

for attempt in retrying(max_attempts=3, delay=0.01, on=ConnectionError):
    with attempt:
        tries.append(attempt.number)
        if len(tries) < 3:
            raise ConnectionError("down")

assert tries == [1, 2, 3]

import asyncio
from python_wrap.retry import retrying

async def main() -> list[int]:
    tries: list[int] = []
    async for attempt in retrying(max_attempts=3, delay=0.01, on=ConnectionError):
        with attempt:
            tries.append(attempt.number)
            if len(tries) < 3:
                raise ConnectionError("down")
    return tries

assert asyncio.run(main()) == [1, 2, 3]

Use async for inside coroutines: a plain for would block the event loop while waiting. With async hooks (retry_if, retry_if_result or on_retry defined with async def), also write async with attempt: so they can be awaited; a plain with attempt: raises TypeError for them.

For retry_if_result, store the block's value in attempt.result. A rejected value starts another attempt; when no retry is left the loop ends normally and your variable holds the last value:

from python_wrap.retry import retrying

pages = iter([[], [], ["row"]])
rows: list[str] = []

for attempt in retrying(max_attempts=5, delay=0.01, retry_if_result=lambda r: not r):
    with attempt:
        rows = next(pages)
        attempt.result = rows

assert rows == ["row"]

Statistics

retry_stats(func) returns a RetryStats with thread-safe counters: calls, attempts_total, retries, gave_up (including calls that returned a rejected value), budget_exhausted (the part of gave_up caused by a RetryBudget), total_sleep and last_exception. Counters are per decorated function (shared by all instances for methods); snapshot() copies them and reset() clears them.

from python_wrap import retry
from python_wrap.retry import retry_stats

n = []

@retry(delay=0.01)
def fetch() -> str:
    n.append(1)
    if len(n) < 3:
        raise OSError("flaky")
    return "ok"

assert fetch() == "ok"
stats = retry_stats(fetch)
assert stats is not None
assert (stats.calls, stats.attempts_total, stats.retries, stats.gave_up) == (1, 3, 2, 0)

Project-wide defaults

Set max_attempts, delay, backoff, max_delay, jitter, max_time, log_level and budget once for the whole project with python_wrap.config.configure(retry={...}). See Configuration.

Logging

Each retry logs one message at log_level (default WARNING) on logger. Use log_level=logging.INFO to quiet it, log_level=None to turn it off, and log_exc_info=True to attach the traceback.

Events

Event Fields
retry.attempt_failed attempt (1-based), exception (None for a rejected result), next_delay
retry.succeeded attempts
retry.gave_up attempts, exception (None for a rejected result)
retry.budget_exhausted attempts, exception; emitted just before retry.gave_up when a RetryBudget refuses

Subscribe with python_wrap.events; see Metrics.

Testing

fast_mode() makes the backoff waits instant, and fake_clock() turns them into clock advances you can assert on. A RetryBudget reads time from the same clock, so fake_clock.advance() slides its window. disable_retries() makes retry a single attempt (a budget still receives its per-call deposit, but nothing is withdrawn because no retry happens), and the max_attempts override from python_wrap.testing caps the attempts of every retry and retrying at call time. See Testing.

Gotchas

Cancellation and shutdown are never retried

asyncio.CancelledError, KeyboardInterrupt, SystemExit and GeneratorExit are never retried, even with on=BaseException or when you list them in on: retrying them would make tasks and processes impossible to stop. Cancelling an async call while it waits between attempts cancels it immediately.

python_wrap.retry is the function

The package attribute python_wrap.retry is the decorator, not the module. Import helpers explicitly: from python_wrap.retry import retrying, retry_stats.

See also