Resilience

So far Snip has been about going faster and handling more — caching, replicas, a queue. This chapter is about something different: staying up when a dependency misbehaves. Postgres gets slow. Redis restarts. The network hiccups. A resilient service treats those not as catastrophes but as Tuesday.

:::note Honesty up front The base project keeps the dependency calls simple — it awaits Postgres and Redis directly, with no timeout or retry wrapper. That’s deliberate: it keeps the teaching code readable. This chapter explains the patterns and shows illustrative code you’d add as the next hardening step. Where the code is an addition (not in the project yet), it’s clearly labelled. :::

Why resilience is its own concern

A fast service that falls over the moment Postgres is slow isn’t actually fast — it’s brittle. The recurring question of this book is “what does this buy us, and what does it cost?” Resilience patterns buy you predictable failure: instead of requests piling up and the whole service grinding to a halt, failures stay small, fast, and contained. The cost is more code and a few new failure modes to reason about.

The deeper idea: a dependency being slow is worse than a dependency being down. A down dependency fails instantly and you move on. A slow one holds your request — and the connection, the task, the memory — hostage. One slow dependency can exhaust your whole connection pool and take down requests that didn’t even need it. Most resilience patterns exist to convert “slow” into “fast failure.”

Timeouts: the foundation

Every call that crosses a boundary — to Postgres, to Redis, to the network — needs a deadline. Without one, a hung dependency hangs you. With one, a hung dependency just gives you an error you can handle.

The base code does the equivalent of this — a bare await:

// What the base project does today (no deadline):
let row: Option<(String,)> = sqlx::query_as("SELECT url FROM links WHERE code = $1")
    .bind(&code)
    .fetch_optional(&state.db)
    .await
    .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?;

Here’s the hardening step — wrap it in a deadline with tokio::time::timeout:

// An addition you could make (NOT in the base project yet):
use std::time::Duration;
use tokio::time::timeout;

let link = match timeout(
    Duration::from_millis(500),
    sqlx::query_as::<_, Link>("SELECT code, url FROM links WHERE code = $1")
        .bind(&code)
        .fetch_optional(&state.db),
)
.await
{
    Ok(Ok(link)) => link,            // query finished in time
    Ok(Err(e)) => return Err(e.into()), // query errored
    Err(_elapsed) => {                // the 500ms deadline blew
        tracing::warn!("db lookup timed out");
        return Err(AppError::Timeout);
    }
};

Notice the nested Result: the outer Ok/Err is the timeout, the inner is the query result. That’s the shape tokio::time::timeout always gives you.

:::tip Pick deadlines from your latency budget, not from a hat If your p99 for a DB lookup is 20ms, a 500ms timeout is generous — it fires only when something is genuinely wrong, not on a normal slow day. A timeout tighter than your real p99 just turns healthy requests into errors. Measure first (you have the load tester), then set the deadline above your p99. :::

For the full theory of layered deadlines and budgets, see Timeouts, Retries & Backoff.

Retries — but only for idempotent operations

A timeout tells you a call failed. Sometimes the right next move is to try again — a blip, a dropped packet, a leader election in the database. But retries are a loaded gun:

• Only retry idempotent operations. Reading a link by code? Safe — do it as many times as you like. Inserting a click event? Not safe if a naive retry could double-count. Snip’s redirect read is a great retry candidate; a “create link” write needs more care.
• Back off between attempts. Hammering a struggling dependency with instant retries is how you turn a slow database into a down one. Wait longer each time: 50ms, 100ms, 200ms.
• Add jitter. If a thousand requests all fail at once and all retry after exactly 100ms, they stampede the recovering dependency in lockstep. Randomizing the delay spreads the load out.

// An addition you could make (NOT in the base project yet) — retry an
// IDEMPOTENT read with exponential backoff + jitter:
use std::time::Duration;
use rand::Rng;

async fn lookup_with_retry(state: &AppState, code: &str) -> Result<Option<Link>, AppError> {
    let mut attempt = 0;
    loop {
        match timeout(Duration::from_millis(500), fetch_link(state, code)).await {
            Ok(Ok(link)) => return Ok(link),
            Ok(Err(e)) if attempt >= 2 => return Err(e.into()), // out of retries
            Err(_) if attempt >= 2 => return Err(AppError::Timeout),
            _ => {
                // base delay doubles each attempt: 50ms, 100ms
                let base = 50u64 << attempt;
                // full jitter: random point in [0, base)
                let jitter = rand::thread_rng().gen_range(0..base);
                tokio::time::sleep(Duration::from_millis(jitter)).await;
                attempt += 1;
            }
        }
    }
}

:::caution Retries multiply load Three retries means one slow moment can become 3× the traffic to a dependency that’s already struggling. Cap your attempts, keep them small, and never retry a non-idempotent write blindly. When the whole fleet retries together you can amplify a small wobble into an outage — this is exactly what circuit breakers (below) exist to stop. :::

Graceful degradation — and Snip already has one

Here’s the happy surprise: Snip is already gracefully degraded by design. Look at the read path you built in Caching with Redis. A redirect checks the cache first; only on a miss does it touch Postgres.

So consider what happens if Postgres is slow or down but Redis is up:

• A redirect for an already-cached code is served entirely from Redis — Postgres is never consulted, so the user never notices the database is unwell. Redirects keep working.
• A redirect for an uncached code, or a new shorten (which must write to Postgres), will fail.

That’s textbook graceful degradation: the core, high-value, read-heavy path stays up on cache alone, while the lower-frequency write path is the part that sheds. The service doesn’t go dark — it gets smaller. You’d make this explicit by serving cached redirects even when the DB lookup times out, and returning a clean 503 for writes rather than hanging.

This is the most important resilience property a read-heavy service can have, and Snip gets it almost for free from the cache-aside architecture. For the broader menu — serving stale data on purpose, dropping non-essential work, shedding load before you fall over — see Graceful Degradation & Load Shedding.

Circuit breakers and bulkheads (at a high level)

Two more patterns belong in your vocabulary, even if Snip doesn’t implement them yet.

Circuit breaker. If Postgres has failed the last N calls in a row, stop calling it for a while. A breaker has three states:

CLOSED   → calls flow normally, failures are counted
OPEN     → too many failures: fail fast WITHOUT calling the dependency
HALF-OPEN → after a cooldown, let one trial call through to test recovery

The point is to stop wasting time and threads on a dependency you already know is broken. While the breaker is open, redirect requests fail instantly (and, for Snip, fall back to cache) instead of each one waiting out a 500ms timeout. It also gives the sick dependency room to recover instead of being pounded.

Bulkhead. Partition your resources so one overloaded path can’t sink the whole ship — named after the watertight compartments that stop a hull breach from flooding an entire vessel. Concretely: give the write path its own small connection pool so a flood of slow writes can’t consume every connection and starve the read path. The reads stay afloat in their own compartment.

Both are explained in depth — with the state machine and pool-partitioning math — in Circuit Breakers & Bulkheads.

Where this leaves Snip

Pattern	In the base project?	Notes
Graceful degradation (cache serves redirects)	Yes — by design	Falls out of cache-aside
Timeouts on DB/Redis calls	No	Shown above as the next step
Retries + backoff + jitter	No	Read path is the safe candidate
Circuit breaker	No	Conceptual here; see linked chapter
Bulkhead (separate pools)	No	Conceptual here; see linked chapter

The honest summary: Snip’s architecture is resilient where it matters most (the hot read path), and the hardening (timeouts, retries, breakers) is a well-understood next layer you now know how to add.

Try it

You can watch graceful degradation happen with the stack you already have:

# Bring up the stack, then shorten and hit a URL so it lands in the cache.
docker compose up --build -d
CODE=$(curl -s -X POST localhost:8080/shorten -H 'content-type: application/json' \
  -d '{"url":"https://example.com"}' | grep -o '"code":"[^"]*"' | cut -d'"' -f4)
curl -sI localhost:8080/$CODE     # first hit: populates the cache

# Now STOP Postgres but leave Redis running:
docker compose stop postgres

# The already-cached redirect STILL WORKS (served from Redis):
curl -sI localhost:8080/$CODE     # → 307, no Postgres needed

# But a NEW shorten FAILS — it must write to Postgres:
curl -s -X POST localhost:8080/shorten -H 'content-type: application/json' \
  -d '{"url":"https://nope.example"}'   # → error / 5xx

The service didn’t go dark — it got smaller. That’s the goal. Bring Postgres back with docker compose start postgres.

Check your understanding

1. Why is a slow dependency often more dangerous than a down one?
2. tokio::time::timeout returns a nested Result. What does each layer (Ok(Ok), Ok(Err), Err) mean?
3. Why must you only retry idempotent operations? Give a Snip operation that’s safe to retry and one that isn’t.
4. What is “jitter” and what failure does it prevent when many requests retry at once?
5. Snip’s redirects survive Postgres being down (if the code is cached) but new shortens don’t. What is that property called, and which architectural choice gives it to us for free?

Show answers

1. A down dependency fails instantly, so you move on. A slow one holds your request — and the connection, the task, the memory — hostage; one slow dependency can exhaust your whole connection pool and take down requests that didn’t even need it. Most resilience patterns exist precisely to convert “slow” into “fast failure.”
2. Ok(Ok(link)) — the query finished in time and succeeded. Ok(Err(e)) — it finished in time but the query itself errored. Err(_elapsed) — the deadline blew before the query returned (the outer layer is the timeout, the inner is the query result). That nested Result is the shape tokio::time::timeout always gives you.
3. Because a retry runs the operation again, and a non-idempotent one would produce a wrong result if the first attempt actually succeeded but its response was lost. Safe in Snip: reading a link by code (a SELECT) — do it as many times as you like. Not safe: inserting a click event or creating a link — a naive retry could double-count or double-insert.
4. Jitter is randomizing the retry delay (e.g. a random point in [0, base)) instead of waiting a fixed interval. It prevents the thundering herd: if a thousand requests all fail at once and all retry after exactly the same delay, they stampede the recovering dependency in lockstep; spreading the delays out lets it recover.
5. Graceful degradation — the service doesn’t go dark, it gets smaller: cached redirects keep working on Redis alone while only the lower-frequency write path sheds. Snip gets it almost for free from the cache-aside architecture built in Caching with Redis, where a redirect checks the cache before ever touching Postgres. What resilience buys: predictable, contained failure. What it costs: more code and a few new failure modes.