Building With LLMs

For most of this book a “dependency” meant something that, given the same input, returned the same output quickly and cheaply: a database read, a cache hit, an RPC. A large language model breaks all three assumptions at once. It is non-deterministic (the same prompt can yield different answers), slow (responses take seconds, not milliseconds), and metered per token (every call has a non-trivial dollar cost). Bolting one into your architecture is less like adding a function call and more like adding a brilliant, expensive, occasionally-wrong contractor to your request path. This page is about designing around that — and like everywhere else, the question is what does an LLM feature buy us, and what does it cost? This is a snapshot as of 2025; the tools move fast, so we’ll keep model names and prices qualitative and date the claims.

Scope and dating

“LLM” here means a hosted or self-hosted large language model accessed over an API — the dominant pattern in 2024–2025. Specific model families, context-window sizes, and per-token prices change every few months and are deliberately left vague below; the shape of the constraints is what’s durable.

The three new constraints

Every design decision in this page traces back to one of these.

NON-DETERMINISM  same input → possibly different output; it can be confidently wrong
LATENCY          first token in ~hundreds of ms; full answer in seconds; scales with
                 output length (the model emits one token at a time)
COST             billed per input + output token; long prompts and long answers add up

A traditional service treats a slow, flaky dependency as a bug to fix. With an LLM the slowness and the fuzziness are intrinsic — you can’t engineer them away, only budget and bound them. That reframing is the whole game.

Retrieval-augmented generation (RAG)

The model knows only what was in its training data, frozen at some cutoff, and it has no access to your private documents. The dominant pattern for grounding it in your facts — as of 2025 — is retrieval-augmented generation: don’t fine-tune the model on your data, retrieve the relevant snippets at request time and paste them into the prompt.

                   ┌──────────────────────────────────────────────┐
  user question ─► │ 1. embed the question → query vector          │
                   │ 2. ANN search a VECTOR DB for nearest chunks  │
                   │ 3. build prompt: [retrieved chunks] + question│
                   │ 4. call the LLM → stream the grounded answer  │
                   └──────────────────────────────────────────────┘

The offline half is an ingestion pipeline: chunk your documents, run each chunk through an embedding model to get a vector (a few hundred to a couple thousand floats capturing its meaning), and store those vectors in a vector database. The online half embeds the user’s question with the same model and asks the vector DB for the nearest chunks by cosine distance — semantic search, not keyword search. Those chunks go into the prompt as grounding context.

Why RAG instead of fine-tuning or a giant prompt

RAG buys you fresh, private, citable knowledge without retraining anything — update a document and the next query sees it. It costs you a retrieval pipeline, an embedding model, a vector store, and a new failure mode: retrieve the wrong chunks and the model confidently answers from garbage. Fine-tuning bakes knowledge in but is expensive to refresh and can’t cite sources; stuffing everything into one huge prompt is simple but blows your token budget and dilutes the model’s attention. For knowledge that changes, RAG is usually the right trade in 2025.

Under the hood — the vector DB and ANN

A vector database is, at heart, an index over high-dimensional vectors with an unusual query: “find the k nearest neighbours to this point.” Comparing the query against every stored vector (brute force) is exact but linear in the corpus size — fine for thousands of vectors, hopeless for hundreds of millions. So vector DBs use approximate nearest neighbour (ANN) indexes that trade a little recall for a massive speedup. The two canonical families:

• HNSW (Hierarchical Navigable Small World) — a multi-layer graph you greedily walk from sparse top layers down to dense bottom ones. Very fast, very high recall; memory-hungry because it stores the graph edges.
• IVF (Inverted File) — cluster the vectors with k-means, then at query time only search the few clusters nearest the query. Compact (especially combined with product quantization) and scales to billions; recall is tunable by how many clusters you probe.

These are covered in depth in The Data-Intensive Frontier. The point here: semantic retrieval rests on an approximate index, so “did we retrieve the right context?” is itself a tunable, fallible step — not a given.

The new latency and cost budget

LLM latency has a structure traditional services don’t: the model generates one token at a time, so total latency is roughly time-to-first-token plus (tokens out × per-token time). A longer answer literally takes longer to produce. That changes how you spend your latency budget.

By the numbers — a RAG latency and cost budget

A grounded-answer request, illustrative figures, as of 2026 (they will drift) (treat as orders of magnitude):

embed the question        ~20–50 ms
ANN search (vector DB)    ~5–30 ms
LLM time-to-first-token   ~150–600 ms
LLM streaming the rest    ~5–20 ms per token → a 400-token answer ≈ 2–8 s
-----------------------------------------------------------------
user sees first words in  under ~1 s   (because we STREAM)
full answer completes in  several seconds

The retrieval steps are cheap; the model dominates. And cost tracks tokens, not requests: a chatty prompt (big retrieved context) plus a long answer can cost an order of magnitude more than a terse one. Two levers fall straight out — stream so the user isn’t staring at a spinner for those seconds, and cache identical or near-identical requests so you don’t pay the model bill twice.

So the standard moves are: a timeout with retry-and-backoff around the model call (it will occasionally hang or rate-limit you), a cache on embeddings and on full responses for repeated questions, and aggressive control of prompt size — because every token you retrieve is a token you pay for and a token of the model’s attention you spend. What does a fatter retrieved context buy? Better grounding. What does it cost? Latency, dollars, and sometimes worse answers as the signal drowns in retrieved noise.

Streaming UX: designing for tokens-over-time

Because the answer arrives token-by-token, the interface changes shape. A request/response box that blocks for eight seconds feels broken; the same eight seconds feel fast if words start appearing in under a second and flow in like someone typing. So LLM features almost universally stream partial output to the client — typically over Server-Sent Events or a chunked HTTP response. This is the same insight as everywhere in this book: perceived latency is about time to first useful byte, not time to last byte. Streaming buys a dramatically better feel; it costs you a streaming transport, a UI that renders incrementally, and the awkward question of what to do when a guardrail needs to retract text you’ve already shown.

Non-determinism, guardrails, and evals

The deepest shift is psychological. You’re used to components that are correct or broken. An LLM is neither — it’s plausibly wrong, and it states wrong answers in the same confident voice as right ones. You cannot unit-test your way to “this function always returns the right string.” Two disciplines fill the gap.

Guardrails bound what the model is allowed to do, on both ends. On the input side: treat the prompt as an untrusted trust boundary — users will attempt prompt injection (“ignore your instructions and…”), and retrieved documents can carry injected instructions too. On the output side: validate structure (does it parse as the JSON you asked for?), filter unsafe content, and constrain actions (an LLM that can call tools must be sandboxed so a hallucinated call can’t do damage). Never let raw model output flow straight into a database write, a shell, or a customer promise without a checking layer.

Evals replace the deterministic test suite. Because there’s no single correct output, you build a dataset of representative inputs and graded expectations, then score the system on it — using heuristics, a held-out answer key, or another model as a judge. Evals are how you answer “did my prompt change make things better or worse?” in a world where you can’t just diff outputs. They are to LLM features what unit tests are to ordinary code: the thing that lets you change anything with confidence.

It actually happened — a chatbot’s made-up policy became the company’s liability

In February 2024, British Columbia’s Civil Resolution Tribunal ordered Air Canada to compensate a passenger who had been advised by the airline’s website chatbot that he could apply for a bereavement discount after booking — a policy that did not exist. Air Canada argued, remarkably, that the chatbot was a separate entity responsible for its own statements. The tribunal rejected that outright: the company was responsible for all information on its site, whether it came from a static page or a chatbot. The lesson for system designers is exact and sobering — a hallucinated answer is not a harmless glitch; it can be a binding commitment. That single ruling is the strongest possible argument for output guardrails, grounding via RAG, evals on high-stakes flows, and never letting an ungrounded, unchecked model speak for you on anything that carries legal or financial weight.

Key trade-offs

• LLM feature vs hand-coded logic: buys capabilities you couldn’t otherwise build; costs determinism, seconds of latency, a per-token bill, and a new abuse surface.
• RAG vs fine-tuning vs giant prompts: RAG buys fresh, citable, private knowledge; costs a retrieval pipeline and a “retrieved the wrong thing” failure mode.
• More retrieved context vs less: more grounding vs more latency, more cost, and possible signal dilution.
• Streaming vs wait-for-complete: far better perceived latency vs streaming transport, incremental UI, and harder mid-stream retraction.
• Guardrails/evals vs ship-it-raw: safety and the ability to change prompts confidently vs the engineering cost of building both.

The architect’s lens

Before you bolt an LLM into the request path, answer the five questions:

• Why does it exist? Because it buys capabilities you couldn’t otherwise hand-code — open-ended language understanding and generation — that no deterministic function could express.
• What problem does it solve? Tasks with no closed-form algorithm, grounded in your data via RAG (retrieve relevant chunks, paste into the prompt) so the model answers from fresh, private, citable facts instead of its frozen training data.
• What are the trade-offs? You inherit three intrinsic costs you can only budget, never engineer away: non-determinism (confidently wrong answers), latency (the model emits one token at a time — a 400-token answer is seconds, so you stream), and per-token cost (a fat retrieved context buys grounding but costs dollars, latency, and diluted attention).
• When should I avoid it? When hand-coded logic suffices — don’t add seconds of latency, a per-token bill, and a prompt-injection surface to a problem a deterministic function solves. And avoid letting raw output flow unchecked into a DB write, a shell, or a customer promise.
• What breaks if I remove the guardrails/evals? You ship a plausibly wrong component with no safety net: prompt injection goes unfiltered, hallucinations reach users, and you can’t tell whether a prompt change helped or hurt. The Air Canada ruling (2024) is the cost made concrete — a hallucinated policy became a binding legal commitment.

Check your understanding

1. Name the three new constraints an LLM introduces into a request path, and explain why you “budget and bound” them rather than engineer them away.
2. Walk through the four online steps of a RAG request. What does RAG buy over fine-tuning, and what new failure mode does it introduce?
3. Why does LLM latency scale with the length of the answer, and how does streaming change the user’s experience of that latency?
4. What is the difference between HNSW and IVF as ANN index families, and why does a vector DB use an approximate index at all?
5. Why can’t you unit-test an LLM feature the way you test ordinary code, and what two disciplines fill the gap? Tie your answer to the Air Canada ruling.

Show answers

1. Non-determinism (same input can give different, confidently-wrong output), latency (seconds, not milliseconds, growing with output length), and cost (billed per token). You budget and bound them — timeouts, caches, prompt-size limits, guardrails, evals — because, unlike a normal flaky dependency, these properties are intrinsic to the model and can’t be removed, only managed.
2. (1) Embed the question into a query vector with an embedding model; (2) ANN-search a vector DB for the nearest document chunks; (3) build a prompt of retrieved chunks + the question; (4) call the LLM and stream the grounded answer. RAG buys fresh, private, citable knowledge with no retraining (update a doc, the next query sees it), versus fine-tuning which bakes knowledge in and can’t cite. New failure mode: retrieve the wrong chunks and the model confidently answers from garbage.
3. The model generates one token at a time, so total time ≈ time-to-first-token + (tokens out × per-token time) — a longer answer literally takes longer. Streaming sends tokens as they’re produced, so the user sees words in under ~1 second and reads along as it generates, making several seconds of total generation feel fast: perceived latency is time-to-first-useful-byte, not time-to-last-byte.
4. HNSW is a multi-layer navigable-small-world graph you greedily traverse — very fast and high-recall but memory-hungry. IVF clusters vectors (k-means) and probes only the nearest few clusters at query time — compact and scales to billions, with recall tunable by how many clusters you probe. A vector DB uses an approximate index because exact brute-force search is linear in corpus size (fine for thousands, hopeless for hundreds of millions); ANN trades a little recall for a huge speedup.
5. There’s no single correct output, so you can’t assert one exact string — the model is plausibly wrong in a confident voice. Two disciplines fill the gap: guardrails (bound inputs against prompt injection and validate/constrain outputs before they act) and evals (a graded dataset that scores whether a change made the system better or worse). The Air Canada tribunal ruling (February 2024) shows why it matters: the airline was held liable for a refund policy its chatbot invented — a hallucination can become a binding commitment, so high-stakes flows need grounding, output checks, and evals.