Async Work with a Queue

Every time Snip redirects someone, we’d like to count the click — analytics, dashboards, “which links are popular.” The naive way is to do an UPDATE links SET clicks = clicks + 1 right there in the redirect handler. It works. It’s also a mistake, and seeing why is the whole point of this chapter.

WHY: the redirect path is sacred

A redirect is the hottest, most latency-sensitive thing Snip does. A human clicked a link and is staring at a blank tab waiting for the page to load. Everything we add to that path — every database round-trip, every lock — adds milliseconds the user feels.

A click count, by contrast, is not urgent. Nobody is waiting on it. If the dashboard shows 1,041 clicks instead of 1,043 for a few seconds, no one is harmed. So we have two pieces of work with wildly different priorities glued into one request. The fix is to unglue them.

:::note The core trade you’re making You’re trading strong consistency (the count is exact the instant the redirect returns) for low latency on the path that matters. The count becomes eventually consistent — correct soon, not instantly. For analytics, that’s a fantastic deal. :::

This is the textbook case for a message queue: a buffer that lets a fast producer hand work to a slower consumer without either waiting on the other. It’s also the heart of sync vs async messaging — the redirect stops synchronously waiting for the write and instead asynchronously fires it off.

HOW: enqueue on the way out

Here’s the only change to the redirect handler. After we’ve resolved the URL, just before returning, we push the code onto a Redis list named clicks:

// Fire-and-forget: enqueue a click event for the worker to count.
let _: Result<(), _> = conn.rpush("clicks", &code).await;

Ok(Redirect::temporary(&url))

Three things to notice, because each one is deliberate:

• rpush appends to the right end of a Redis list. We’re using a plain list as a queue: producers push on the right, the consumer pops from the left (FIFO). Redis is already in our stack for caching and rate limiting, so we get a queue for free — no new infrastructure.
• let _: Result<(), _> = ... — we bind the result to _, throwing it away. This is fire-and-forget. If the push fails, we do not fail the redirect. The user still gets their page; we just lose one click count. That’s the right priority: a redirect is worth more than a tally. The : Result<(), _> type annotation is there to tell Rust what kind of result rpush returns, so it knows what we’re discarding.
• The push is the last thing before Ok(...). The redirect doesn’t depend on it at all.

The redirect handler is now back to doing only what’s on the critical path: look up the URL (cache, then DB), fire the event, return. The expensive write moved somewhere else.

The worker: a separate program

That “somewhere else” is a second binary — src/bin/worker.rs. It’s a completely separate process from the web server. They share data (Redis and Postgres) but not code execution, which is exactly the decoupling we want: you can deploy them separately, restart one without the other, and scale them independently.

Setup looks just like the app — connect to Postgres and Redis from env vars:

#[tokio::main]
async fn main() {
    let database_url = env::var("DATABASE_URL")
        .unwrap_or_else(|_| "postgres://snip:snip@localhost:5432/snip".to_string());
    let redis_url =
        env::var("REDIS_URL").unwrap_or_else(|_| "redis://localhost:6379".to_string());

    let db = PgPoolOptions::new()
        .max_connections(5)
        .connect(&database_url)
        .await
        .expect("connect to postgres");

    let client = redis::Client::open(redis_url).expect("valid redis url");
    let mut conn = redis::aio::ConnectionManager::new(client)
        .await
        .expect("connect to redis");

Notice the worker asks for only max_connections(5) to Postgres — it’s one process doing one slow thing in a loop, so it doesn’t need the pool the web server uses for concurrent requests.

The worker loop: block, pop, write, repeat

The actual work is a single loop:

    println!("worker started — waiting for click events…");
    loop {
        // Block until an event arrives (timeout 0.0 = wait forever).
        let popped: redis::RedisResult<(String, String)> = conn.blpop("clicks", 0.0).await;
        match popped {
            Ok((_list, code)) => {
                if let Err(e) = sqlx::query("UPDATE links SET clicks = clicks + 1 WHERE code = $1")
                    .bind(&code)
                    .execute(&db)
                    .await
                {
                    eprintln!("failed to record click for {code}: {e}");
                }
            }
            Err(e) => {
                eprintln!("blpop error: {e}; retrying in 1s");
                tokio::time::sleep(std::time::Duration::from_secs(1)).await;
            }
        }
    }

The key call is blpop — blocking left pop. The b matters. A plain lpop would return immediately with “nothing here” if the list is empty, so a worker using it would have to poll — ask again and again in a tight loop, burning CPU and hammering Redis. blpop instead parks the connection until an item is available (or the timeout elapses). We pass 0.0 as the timeout, which in Redis means wait forever. So the worker sleeps, costing nothing, until the instant a redirect pushes a click — then Redis wakes it up.

blpop returns a tuple of two strings: the name of the list it popped from and the value. That’s why the type is (String, String) — we already know the list is "clicks", so we bind it to _list and use the code. Then we do the slow write the redirect refused to do, and loop back to block again.

The two branches of the match are the worker’s whole error story:

• Ok — we got a code; run the UPDATE. If that fails, we log to stderr and move on. One lost click beats a crashed worker.
• Err — the blpop itself failed, which almost always means Redis hiccuped. We log, sleep one second to avoid a hot retry loop, and try again. The worker is self-healing: if Redis comes back, so does the worker.

The trap: an unbounded queue

There’s a quiet danger here. The clicks list has no size limit. If redirects pour in faster than the worker can drain them, the list grows… and grows. In a real incident — a link goes viral, or the worker crashes and nobody notices — that list can eat all of Redis’s memory and take the whole cache down with it.

That’s why a queue is a buffer, not a bottomless pit. The discipline of watching how full a queue is, and slowing or shedding producers when a consumer can’t keep up, is backpressure & flow control. A production version of Snip would cap the list length (Redis can trim it), alert when it’s growing, and maybe run more than one worker to drain faster. For our learning build, the single worker keeps up easily — but now you know where the cliff is.

:::tip The whole pattern in one breath Producer (redirect) pushes fast and never blocks → queue (Redis list) absorbs the burst → consumer (worker) drains at its own pace. Latency stays on the fast path; durability and counting happen off it. :::

Try it

Run Postgres and Redis (the Compose file in rust/scalable-service/ has them), then start the app and the worker in two terminals:

# Terminal 1 — the web server
cargo run --bin snip

# Terminal 2 — the click worker
cargo run --bin worker

You’ll see worker started — waiting for click events…. Now shorten a link and hit it a few times:

# create a link
curl -s -X POST localhost:8080/shorten \
  -H 'content-type: application/json' \
  -d '{"url":"https://example.com"}'

# follow the redirect a few times (use the code you got back)
curl -s localhost:8080/abc1234 -o /dev/null
curl -s localhost:8080/abc1234 -o /dev/null

Each redirect returns instantly — it doesn’t wait for any counting. Watch the worker terminal stay quiet (it’s just doing fast writes), then check the count in Postgres:

psql postgres://snip:snip@localhost:5432/snip \
  -c "SELECT code, clicks FROM links;"

The count climbs. Now stop the worker (Ctrl-C), hit the link a few more times, and check again — the count doesn’t move, but the redirects still work perfectly. Restart the worker and watch it drain the backlog that piled up in Redis. That’s eventual consistency you can see with your own eyes.

Check your understanding

1. Why do we bind rpush’s result to _ instead of using ? to propagate errors? What would break if a failed enqueue could fail the redirect?
2. What’s the difference between lpop and blpop, and why does choosing blpop matter for the worker’s CPU and Redis load?
3. After moving click-counting to the worker, the count is no longer exact the instant a redirect returns. What is this property called, and why is it an acceptable trade for this data?
4. The clicks list has no maximum length. Describe a scenario where that becomes a real outage, and name the technique used to prevent it.
5. The app and the worker are separate binaries that share Postgres and Redis but not code execution. Name two operational benefits of that separation.

Show answers

1. Counting a click is fire-and-forget — it’s worth less than the redirect itself. Binding to _ discards the result so a failed enqueue silently loses one click instead of failing the redirect. Using ? would propagate the error and break the user’s redirect just because a tally couldn’t be written; that inverts the priorities — a redirect is sacred, a click count is not.
2. lpop returns immediately even when the list is empty, so a worker using it must poll in a tight loop, burning CPU and hammering Redis. blpop (blocking left pop) parks the connection until an item arrives (timeout 0.0 = wait forever), so an idle worker costs nothing and Redis wakes it the instant a click is pushed.
3. It’s eventual consistency — the count is correct soon, not instantly. That’s an acceptable trade for click analytics because nobody is waiting on the number: a dashboard briefly showing 1,041 instead of 1,043 harms no one, so trading exactness for low latency on the redirect path is a great deal. What it buys: a fast hot path. What it costs: the count lags reality for a few seconds.
4. If redirects pour in faster than the worker drains them (a link goes viral, or the worker crashes unnoticed), the unbounded clicks list grows until it eats all of Redis’s memory and takes the whole cache down. The technique that prevents it is backpressure & flow control — cap the list length (Redis can trim it), alert when it grows, and run more workers to drain faster.
5. You can deploy them separately (ship a worker fix without touching the web server), restart one without the other, and scale them independently (add workers to drain faster without adding web replicas). Any two of these — the point is that sharing data but not code execution decouples their lifecycles.