Load Testing & Measuring

We’ve added a Postgres store and a Redis cache. Each one should have made Snip faster or more scalable — but “should” is not a number. From here on, every scaling chapter ends with a re-run of a load test, and we compare. This chapter builds the measuring stick.

:::tip WHY before HOW You can’t improve what you don’t measure, and you can’t measure what you don’t define. Before adding replicas, a queue, or a rate limiter, we need a baseline: this many requests per second, this much latency, on today’s setup. Every later change is judged against it. A win you can’t quantify is just a vibe. :::

The numbers we care about — throughput and tail latency — are exactly the ones from Latency, Throughput & the Numbers. This chapter is those ideas turned into a runnable tool.

Why percentiles, not averages

The single most important idea in this chapter: report percentiles, not the average.

The average latency hides the worst experiences. Suppose 99 requests take 5 ms and one takes 2,000 ms. The average is ~25 ms — sounds fine! But one real user waited two seconds. Averages get dragged around by outliers and smear over them at the same time; they tell you almost nothing about the experience at the edges.

Percentiles tell the truth:

p50 (median) — half of requests are faster than this. The “typical” experience.
p95 — 95% are faster; the slowest 1-in-20.
p99 — 99% are faster; the slowest 1-in-100. This is the tail.

At scale the tail is the experience: a user loading a page that makes 10 backend calls will hit your p99 on roughly 1 of those 10 calls almost every time. That’s why we obsess over p99, not the mean — see Tail Latency & p99 for why tails dominate and how they compound across services.

The tool: a concurrent load tester in Rust

It’s a single main in src/bin/loadtest.rs, so cargo run --bin loadtest builds it alongside the server. The shape: parse args → spawn N workers that hammer the URL → collect every latency → sort and compute percentiles.

Parsing the args

#[tokio::main]
async fn main() {
    let args: Vec<String> = env::args().collect();
    let url = args
        .get(1)
        .cloned()
        .unwrap_or_else(|| "http://localhost:8080/health".to_string());
    let total: usize = args.get(2).and_then(|s| s.parse().ok()).unwrap_or(2000);
    let concurrency: usize = args.get(3).and_then(|s| s.parse().ok()).unwrap_or(50);
    let per_task = (total / concurrency).max(1);

Three knobs: the URL, the total number of requests, and the concurrency (how many run at once). Each falls back to a sensible default via unwrap_or / unwrap_or_else. per_task divides the work evenly across workers — and .max(1) guards against asking each worker to do zero requests when total < concurrency.

concurrency is the dial that matters. One request at a time measures latency under no load; 100 at a time measures latency under contention — which is when the interesting failures show up. The gap between those two is the story of the rest of this book.

Shared, thread-safe counters

    let client = reqwest::Client::new();
    let latencies = Arc::new(Mutex::new(Vec::<f64>::with_capacity(total)));
    let done = Arc::new(AtomicU64::new(0));
    let errors = Arc::new(AtomicU64::new(0));

We have many concurrent tasks all writing to shared state, so everything shared is wrapped to make that safe:

Arc<...> — an atomic reference count. It lets multiple tasks own the same value; cloning an Arc bumps a counter instead of copying the data. This is how each spawned task gets its own handle to the same latencies / done / errors.
Mutex<Vec<f64>> — the latency list needs push, which mutates. A Mutex ensures only one task appends at a time so the Vec can’t be corrupted by a race.
AtomicU64 for the simple counters — incrementing a single integer doesn’t need a full lock; an atomic fetch_add is cheaper and lock-free. (Notice the same AtomicU64 pattern Snip’s /metrics uses.)

The rule of thumb: atomics for single numbers, a Mutex for richer structures like a growing Vec. We also share one reqwest::Client — it pools connections internally, so reusing it is both correct and much faster than building a client per request.

Spawning the workers

    let start = Instant::now();
    let mut handles = Vec::new();
    for _ in 0..concurrency {
        let client = client.clone();
        let url = url.clone();
        let latencies = Arc::clone(&latencies);
        let done = Arc::clone(&done);
        let errors = Arc::clone(&errors);
        handles.push(tokio::spawn(async move {
            for _ in 0..per_task {
                let t = Instant::now();
                match client.get(&url).send().await {
                    Ok(_) => latencies
                        .lock()
                        .unwrap()
                        .push(t.elapsed().as_secs_f64() * 1000.0),
                    Err(_) => {
                        errors.fetch_add(1, Ordering::Relaxed);
                    }
                }
                done.fetch_add(1, Ordering::Relaxed);
            }
        }));
    }
    for h in handles {
        let _ = h.await;
    }
    let elapsed = start.elapsed().as_secs_f64();

tokio::spawn launches an async task onto the runtime — think of it as a very lightweight thread. We spawn concurrency of them and they all run concurrently, so the server sees real simultaneous load.

Two Rust details that trip people up:

The clones before the closure. A tokio::spawned task must own everything it touches (it’s 'static — it may outlive this loop iteration). So before each spawn we clone the Arcs and the cheap-to-clone client/url, and the async move block takes ownership of those clones. The Arc::clone just bumps the refcount; everyone still points at the same shared data.
Timing each request individually. let t = Instant::now(); right before the call and t.elapsed() right after gives that request’s wall-clock latency. We convert to milliseconds as an f64 (.as_secs_f64() * 1000.0) and push it. Successes get timed; failures bump errors instead. Either way done increments, so it counts attempts.

Collecting the JoinHandles into handles and awaiting them all is the join: we start the clock before spawning and read elapsed only after every worker has finished, so elapsed is the true wall-clock duration of the whole run — what we divide by for throughput.

Sorting and computing percentiles

    let mut lat = latencies.lock().unwrap().clone();
    lat.sort_by(|a, b| a.partial_cmp(b).unwrap());
    let pct = |p: f64| -> f64 {
        if lat.is_empty() {
            return 0.0;
        }
        let idx = ((p / 100.0) * (lat.len() as f64 - 1.0)).round() as usize;
        lat[idx]
    };

The percentile trick is wonderfully simple: sort the latencies, then index into the sorted list. If the list is sorted ascending, the value at 95% of the way through is the p95 — 95% of samples sit at or below it. The index is (p/100) * (len - 1), rounded, so p = 100 lands on the last (slowest) element and p = 0 on the fastest.

A couple of notes. We .clone() the Vec out of the Mutex first so we hold the lock for the shortest possible time. We need sort_by with partial_cmp (not plain sort) because f64 doesn’t implement total ordering — floats can be NaN — so the standard sort won’t take them directly. The pct closure captures lat and turns “what percentile?” into one indexing operation.

Printing the verdict

    let count = done.load(Ordering::Relaxed);
    println!(
        "requests: {count}   errors: {}   time: {elapsed:.2}s",
        errors.load(Ordering::Relaxed)
    );
    println!("throughput: {:.0} req/s", count as f64 / elapsed.max(0.001));
    println!(
        "latency  p50 {:.1} ms   p95 {:.1} ms   p99 {:.1} ms",
        pct(50.0),
        pct(95.0),
        pct(99.0)
    );

Throughput = total requests ÷ wall-clock seconds (elapsed.max(0.001) avoids dividing by zero on an absurdly fast run). Then the three percentiles. That’s your baseline — write it down.

:::note A load tester is not the truth, it’s a ruler Running the test from the same machine as the server, over loopback, won’t reproduce production numbers — there’s no real network, no other tenants, no cold caches. That’s fine. The value here is relative: same tool, same args, before and after a change. If p99 drops from 40 ms to 12 ms after adding replicas, that delta is real even if the absolute numbers aren’t. For production-grade signal you’d reach for real metrics — see Metrics. :::

Establishing the baseline you’ll beat

Here’s the workflow for the rest of the book:

Start Snip (single instance, with Postgres + Redis).
Run the load test. Record throughput, p50, p95, p99.
Make one scaling change (replicas, a queue, …).
Run the exact same command again. Compare.

One change at a time, same ruler each time. That discipline is how you tell a real improvement from a lucky run — and how you catch a “scaling” change that quietly made things worse.

Try it

With Snip running, point the tester at /health (no DB or cache, so it isolates raw HTTP throughput) with 5000 requests at concurrency 100:

cargo run --release --bin loadtest -- http://localhost:8080/health 5000 100

requests: 5000   errors: 0   time: 0.42s
throughput: 11905 req/s
latency  p50 0.7 ms   p95 1.9 ms   p99 4.3 ms

(Your numbers will differ — that’s the point; they’re your baseline.) Always use --release: debug builds can be 10x+ slower and would give you a meaningless ruler. Now point it at a real redirect (http://localhost:8080/<code>) and watch how the cache from the last chapter changes the tail. Then re-run curl localhost:8080/metrics and look at the hit rate after a load test.

:::caution The rate limiter will photobomb your load test The router wraps every route — /health included — in the Redis-backed limiter we dissect in the rate-limiting chapter, which allows only 100 requests per client per minute. A 5000-request run blows through that immediately, and everything after is a fast 429 — which the tester still times as a “success”, quietly poisoning your numbers. For clean measurements, temporarily raise LIMIT in the rate_limit function (src/main.rs) to something huge and rebuild. The same caveat applies whenever you load-test in later chapters. :::

The tester’s two dials — concurrency and the latency it measures — aren’t independent of throughput. They’re locked together by Little’s Law: the number of requests in flight equals throughput × time in the system. For a closed-loop tester that holds exactly concurrency requests in flight:

concurrency = throughput × latency
⇒ throughput = concurrency ÷ latency

concurrency 100, latency 5 ms   → 100 ÷ 0.005 s ≈ 20,000 req/s   (the ceiling)
concurrency 100, latency 50 ms  → 100 ÷ 0.05 s  ≈  2,000 req/s   (10× worse)

This is why a throughput number is meaningless without the concurrency and latency that produced it — and why adding replicas helps: it lets you raise concurrency without each request waiting longer. When you re-run the test after a change, check whether throughput rose because latency fell or merely because you cranked concurrency. Only the first is a real win.

Check your understanding

Why does this tool report p50/p95/p99 instead of the average latency? Give a concrete example where the average misleads.
What’s the difference between latencies being an Arc<Mutex<Vec<f64>>> and the counters being Arc<AtomicU64> — why two different tools?
Why are the Arcs and the client cloned before each tokio::spawn?
How is the p95 actually computed from the list of latencies? Why must the list be sorted first, and why sort_by(partial_cmp) rather than sort?
Why does the chapter insist the absolute numbers don’t matter much, but the deltas do? And why --release?

Show answers

Because the average hides the worst experiences and gets dragged around by outliers. Concretely: 99 requests at 5 ms and one at 2,000 ms average to ~25 ms — “sounds fine” — yet one real user waited two seconds. Percentiles tell the truth: p50 is the typical experience, p95 the slowest 1-in-20, and p99 is the tail, which is the experience at scale (a page making 10 backend calls hits your p99 on roughly one of them almost every time).
latencies needs push, which mutates a growing Vec, so it needs a Mutex to serialize appends and stop a race from corrupting it. The counters are single integers, and incrementing one is a lock-free atomic fetch_add — much cheaper than a full lock. The rule of thumb: atomics for single numbers, a Mutex for richer structures.
A tokio::spawned task is 'static — it may outlive the loop iteration — so it must own everything it touches. Cloning the Arcs (a cheap refcount bump) and the cheap-to-clone client/url before each spawn gives the async move block its own handles while everyone still points at the same shared data.
Sort the latencies ascending, then index into the sorted list: the value at (95/100) * (len - 1), rounded, is the p95, because 95% of samples sit at or below it. It must be sorted first or the index is meaningless, and you need sort_by(partial_cmp) rather than plain sort because f64 doesn’t implement total ordering (floats can be NaN), so the standard sort won’t take them.
Because running over loopback on the same machine has no real network, no other tenants, and no cold caches — the absolute numbers don’t reproduce production, but the delta from the same tool with the same args before and after a change is real (p99 dropping 40 ms → 12 ms is a true win). --release because debug builds can be 10x+ slower, giving you a meaningless ruler. What the load test buys: a delta you can trust. What it costs: absolute numbers you can’t.