Scaling Out

The load test in the last chapter gave us a baseline. Now we make Snip faster under load — not by buying a bigger machine, but by running more copies of it. This is the moment the whole architecture pays off, and it pays off for one reason: Snip is stateless.

WHY before HOW

You can only run many identical copies of a service if the copies don’t disagree. The instant a request’s outcome depends on which replica handled it, scaling out breaks. So before we add replicas, the real question is: where does Snip keep its state?

Two ways to make a service handle more

There are exactly two levers:

vertical scaling    →  one bigger box   (more CPU/RAM on a single instance)
horizontal scaling  →  more boxes       (many instances behind a load balancer)

Vertical is the easy one: you change an instance type and you’re done — no code, no coordination. But it has a ceiling (the biggest box money can buy), it’s a single point of failure, and price grows faster than power. Horizontal has no hard ceiling and gives you redundancy for free (one replica dies, the others serve), but it has a precondition that vertical scaling doesn’t: the replicas must be interchangeable.

For the full trade-off, see Vertical vs Horizontal.

The precondition: statelessness

Look at what a Snip replica actually holds in memory. Here’s AppState from src/main.rs:

#[derive(Clone)]
struct AppState {
    db: PgPool,
    cache: redis::aio::ConnectionManager,
    base_url: String,
    metrics: Arc<Metrics>,
}

Read that carefully. db and cache are connection handles — pipes to Postgres and Redis. base_url is config read from an env var. There is no user data, no session table, no per-client counter living inside the process. Every fact that matters — the links, the cache, the rate-limit counters, the click queue — lives in Postgres or Redis, outside the app.

(The one in-memory thing, metrics, is just per-process counters for the /metrics endpoint. Each replica reports its own slice; nothing depends on those numbers being shared. We make metrics global in the Observability chapter.)

That’s what stateless means in practice: any replica can serve any request, because the request’s outcome doesn’t depend on anything stored in that replica. Two requests from the same user can land on two different replicas and behave identically. That is the property that makes the copies interchangeable — and interchangeable copies are what a load balancer needs.

Stateless ≠ no state

The system is enormously stateful — millions of links live in Postgres. It’s the app process that’s stateless. We pushed the state down into shared backends so the layer we want to replicate carries none of it. More on this in Statelessness & Sessions.

The app service in Compose

Here’s the app service from docker-compose.yml:

  # The stateless app. Scale it with:  docker compose up --scale app=3
  app:
    build: .
    environment:
      DATABASE_URL: postgres://snip:snip@postgres:5432/snip
      REDIS_URL: redis://redis:6379
      BIND_ADDR: 0.0.0.0:8080
      BASE_URL: http://localhost:8080
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_started

Notice there’s no ports: mapping on app. We don’t expose it to the host directly — clients never talk to a replica. They talk to nginx, and nginx talks to the replicas over Docker’s internal network. The app is configured entirely through environment variables (the same binary, pointed at the same Postgres and Redis), which is exactly why we can stamp out N identical copies.

Spinning up replicas

This one flag is the whole trick:

docker compose up --build --scale app=3

Compose starts three containers from the app service — app-1, app-2, app-3 — all identical, all pointed at the same Postgres and Redis. No code change. The reason this just works is everything above: stateless app, shared backends.

nginx: spreading requests across replicas

Now we need something in front to fan requests out across those three containers. That’s the load balancer’s job (see Load Balancers). Here’s the entire nginx.conf:

events {}

http {
    # Docker's embedded DNS. Using a variable in proxy_pass forces nginx to
    # re-resolve "app" per request, so it round-robins across all replicas.
    resolver 127.0.0.11 valid=5s;

    server {
        listen 80;

        location / {
            set $upstream http://app:8080;
            proxy_pass $upstream;

            # Pass the real client IP through so the app's rate limiter keys on it.
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }
}

Two pieces are doing real work here.

The resolver + variable proxy_pass trick. Docker runs an embedded DNS server at 127.0.0.11. When three app replicas are running, looking up the name app returns three IPs. Normally nginx resolves an upstream name once at startup and caches it forever — so it would pin to a single replica and ignore the others. By writing the upstream into a variable (set $upstream ...; proxy_pass $upstream;), we force nginx to re-resolve app through the resolver on (effectively) each request, picking up all the IPs and round-robining across them. valid=5s caps how long a DNS answer is cached, so scaling up or down is noticed within seconds.

Why not a normal upstream {} block?

The classic nginx pattern is an upstream block listing backends by name. That’s great when the backend list is fixed, but it’s resolved once and doesn’t follow Docker’s changing replica set. The variable-proxy_pass approach trades a little raw throughput for the ability to scale replicas up and down without editing nginx config. For Snip’s “scale with one flag” goal, that’s the right trade.

X-Forwarded-For. Once nginx sits in front, every request reaching the app comes from nginx’s IP — the app can no longer see who the real client is. proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; stamps the original client IP into a header so the app can recover it. That matters a lot for the next chapter: the rate limiter keys on the real client, and without this header every client would look like nginx. The app reads exactly this header — headers.get("x-forwarded-for") — which we’ll see in Rate Limiting.

The shape of the system now

client ──► nginx :8080 ──┬──► app-1 ─┐
                         ├──► app-2 ─┼──► Postgres (durable)
                         └──► app-3 ─┘──► Redis (cache + rate-limit + queue)

Three stateless app replicas, one load balancer, two shared backends. Adding a fourth replica is one number in one command. That’s horizontal scaling, and statelessness is what bought it.

Try it

Bring the stack up with three replicas:

docker compose up --build --scale app=3

In another terminal, confirm all three exist:

docker compose ps

Now load-test through nginx on :8080 (not the app directly — it isn’t even exposed):

# create a link
curl -s -X POST localhost:8080/shorten \
  -H 'content-type: application/json' \
  -d '{"url":"https://example.com"}'

# hammer the redirect path through the load balancer
hey -z 20s -c 50 localhost:8080/<code>

(Remember the load-testing chapter’s caution: raise the rate limiter’s LIMIT first, or most of these responses will be fast 429s.) Compare the throughput (requests/sec) and p95 against your single-replica baseline from the load-testing chapter. Then confirm every replica booted and is ready for its share:

docker compose logs app | grep "listening"

You should see a Snip listening line from each of app-1, app-2, and app-3. (Snip doesn’t log individual requests, so to see traffic spreading, curl localhost:8080/metrics a few times — nginx round-robins you across replicas, and each one reports its own counters.)

Check your understanding

1. AppState holds four fields. Which of them is state that must not be shared between replicas, and which are just handles or config? Why does that distinction make the app safe to replicate?
2. Why does docker compose up --scale app=3 require no code change? What property of Snip makes it possible?
3. What goes wrong if you write proxy_pass http://app:8080; directly instead of through the $upstream variable, once three replicas are running?
4. After nginx is in front, why can’t the app see the real client’s IP — and which line in nginx.conf fixes that?
5. Snip’s app service has no ports: mapping, but nginx maps 8080:80. Why expose only nginx and not the replicas?

Show answers

1. Only metrics is state, and crucially it’s per-process state that doesn’t need to be shared — each replica reports its own slice and nothing depends on the numbers being global (they’re made global in the Observability chapter). db, cache, and base_url are just connection handles and config. The app is safe to replicate because every fact that matters — links, cache, rate-limit counters, the click queue — lives in Postgres or Redis, outside the process, so any replica can serve any request identically.
2. Because the app is stateless: all state already lives in Postgres and Redis, so three identical copies pointed at the same backends can’t disagree. --scale app=3 just stamps out three containers from the same image with the same env config — interchangeable copies are exactly what a load balancer needs, and that’s the property statelessness buys.
3. nginx resolves an upstream name once at startup and caches it forever, so a literal proxy_pass http://app:8080; would pin to a single replica and ignore the other two. Writing the upstream into a variable (set $upstream ...; proxy_pass $upstream;) forces nginx to re-resolve app through Docker’s DNS per request, picking up all three IPs and round-robining across them.
4. Once nginx sits in front, every request reaching the app comes from nginx’s IP, so the app can no longer see who the real client is. The fix is proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;, which stamps the original client IP into a header the app can recover — which matters for the rate limiter that keys on the real client in Rate Limiting.
5. Because clients should reach the replicas only through the load balancer. nginx is the single front door that fans requests across however many app replicas exist; exposing the replicas directly would let clients bypass nginx, defeating the load balancing and breaking the clean front-door topology. What horizontal scaling buys: throughput and redundancy. What it costs: the app must be stateless.