The Data-Intensive Frontier

The data chapters of this book (SQL vs NoSQL, partitioning, replication, indexing) drew a tidy line: transactional databases on one side, analytics on the other, batch ETL shuffling data between them at night. That line has blurred. In the mid-2020s the data layer is streaming-first, stored in open lakehouse formats, fed by change data capture, and increasingly asked to answer semantic queries over vectors alongside the usual rows and columns. This page surveys that frontier and keeps asking the book’s question: what does each of these convergences buy us, and what does it cost? Snapshot as of 2025.

From batch to streaming

The old model moved data in nightly batches: dump yesterday’s transactions, transform them, load them into a warehouse, query in the morning. The new default is a continuous log of events that downstream systems consume in near-real-time. Two canonical pieces:

• Apache Kafka — a distributed, partitioned, durable append-only log. Producers append events to topics; consumers read at their own pace; the log is replayable and retained. It’s the message queue idea, scaled and made durable enough to be the system of record for events.
• Apache Flink — a stateful stream processor. It consumes streams (often from Kafka), maintains keyed state, reasons about event time (when something happened, not when it arrived), windows events, and joins streams — emitting results continuously.

  sources ──► [ Kafka: durable, partitioned, replayable log ] ──► consumers
                                  │
                                  ▼
                       [ Flink: stateful, event-time
                         windowing / joins / aggregation ] ──► sinks (DB, lake, index)

Why a log, not a queue

A traditional queue deletes a message once it’s consumed. Kafka retains the log, so the same events can feed many independent consumers, be replayed to rebuild a downstream store from scratch, and serve as an audit trail. That replayability buys you decoupling and recoverability — add a new consumer next year and it reads all of history — and costs you the storage to retain the log and the discipline of designing around an append-only, eventually-consistent stream instead of synchronous calls. This is the backbone of event-driven architecture.

Event-time and exactly-once

Streaming’s hard problems are time and duplicates. Events arrive out of order and late, so Flink tracks event time with watermarks to decide when a window is “complete enough” to emit. And because failures cause retries, naïve processing double-counts; Flink offers exactly-once processing semantics via periodic checkpoints (a distributed snapshot of all operator state) so it can recover to a consistent point after a crash. Note the careful wording — exactly-once state, not magic; the physics in Exactly-Once Semantics still apply, and end-to-end exactly-once requires idempotent or transactional sinks.

Change data capture (CDC)

How do events get into the stream in the first place, reliably? The dangerous way is the dual write: have the application write to its database and publish to Kafka. Those two writes aren’t atomic, so a crash between them leaves the log and the database disagreeing forever.

Change data capture sidesteps it by reading the database’s own transaction log — the same log it already writes for durability and replication (MySQL’s binlog, PostgreSQL’s write-ahead log via logical replication). A CDC connector (Debezium is the canonical open-source one) tails that log and emits one event per committed row change. Because it reads the log the database already commits to, every change is captured exactly as it was committed, in order, with no second write to go wrong.

Under the hood — why log tailing is the right primitive

  app ──► [ Primary DB ] ──writes──► transaction log (binlog / WAL)
                                          │  (already exists for crash recovery
                                          │   and replica catch-up)
                                          ▼
                              [ CDC connector tails the log ]
                                          │ one event per committed change
                                          ▼
                                  [ Kafka topic ] ──► search index, cache,
                                                       lake, analytics, ...

The transaction log is the database’s source of truth for what changed and in what order — it’s how replicas stay consistent. CDC reuses it, so there’s no extra write to keep in sync and no dual-write window. What does CDC buy? A reliable, ordered stream of every change with no application code change and no lost-update window. What does it cost? Operational coupling to the database’s log internals, schema-change handling, and the need for downstream consumers to be idempotent (the log can redeliver after a failure).

The lakehouse

Analytics used to force a choice: a data warehouse (structured, fast SQL, ACID, expensive, proprietary) or a data lake (cheap object storage full of files — Parquet — flexible but no transactions, no schema enforcement, no guarantees). The lakehouse is the convergence: keep data in cheap open object storage, but add a table format on top that brings warehouse guarantees — Delta Lake, Apache Iceberg, and Apache Hudi are the three canonical ones. They layer ACID transactions, schema evolution, and time travel (query the table as of an earlier version) over what are physically just Parquet files plus a metadata log.

  DATA WAREHOUSE   structured, ACID, fast SQL          but proprietary & pricey
  DATA LAKE        cheap object storage, open files    but no ACID, no schema, no guarantees
  LAKEHOUSE        cheap object storage + table format → ACID, schema, time travel on open files

The buy: one place for both your raw and curated data, in open formats, queryable by many engines, at object-storage prices. The cost: a younger, more complex stack and a metadata layer that is now itself load-bearing (corrupt the table-format metadata and the Parquet files underneath are just orphaned bytes).

It actually happened — the lakehouse got a name and a thesis in 2021

The term wasn’t always marketing. In 2021, Databricks researchers (Armbrust, Ghodsi, Xin, Zaharia) published “Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics” at the CIDR 2021 conference, arguing explicitly that the two-tier warehouse-plus-lake architecture — with its brittle nightly ETL between them — should collapse into a single tier: transactional metadata over open files in cheap storage. Whatever you think of any one vendor, that paper crystallized the direction the whole industry has moved since. It’s a clean example of the book’s thread stated as a research claim: unifying the two tiers buys you one system and kills the ETL seam between them; it costs you the maturity and specialization the dedicated warehouse had spent decades building.

Vector databases and ANN

The newest column in the data layer is the vector. Embedding models turn text, images, and audio into high-dimensional vectors where nearness means similarity, and a growing class of features — semantic search, recommendations, and the retrieval step of RAG — needs to find the nearest vectors to a query fast. Exact nearest-neighbour search compares the query to every stored vector: correct, but linear in the corpus, which is hopeless at hundreds of millions of vectors. So vector databases use approximate nearest neighbour (ANN) indexes that trade a sliver of recall for orders-of-magnitude speed. The two canonical families:

• HNSW (Hierarchical Navigable Small World) — build a multi-layer proximity graph: sparse upper layers for long “express” hops, dense lower layers for fine local search. A query greedily descends from the top, and search cost grows roughly logarithmically with corpus size. Excellent recall and latency; the cost is memory (it stores the graph’s edges) and trickier deletes.
• IVF (Inverted File) — cluster the vectors with k-means into many cells; at query time, probe only the few cells nearest the query (nprobe of them). Compact and scales to billions, especially combined with product quantization (IVF-PQ) that compresses each vector; recall is tunable by how many cells you probe, and it needs a training step to learn the clusters.

                       HNSW                          IVF (often + PQ)
structure              multi-layer graph             clustered cells (inverted lists)
query                  greedy graph walk, ~O(log N)  probe nearest `nprobe` cells
recall vs speed        high recall, low latency      tune via nprobe (recall ↔ speed)
memory                 high (stores edges)            low (PQ compresses vectors)
updates                inserts easy, deletes awkward  reclustering needed over time
sweet spot             low-latency, moderate scale    billion-scale, memory-constrained

By the numbers — why ANN, not brute force

Illustrative, as of 2025. Search 10 million vectors of 768 dimensions for the top-10 nearest:

brute force:  10,000,000 × 768 multiply-adds per query ≈ ~7.7 billion ops → tens of ms+, scales linearly
HNSW:         visits only ~hundreds–thousands of candidates (≈ grows with log N)
              → typically single-digit milliseconds, ~95–99% recall

The trade is explicit: ANN gives up exactness (you might miss a true neighbour now and then) to turn a linear scan into a near-logarithmic graph walk. What does the index buy? Sub-10-ms semantic search at scale. What does it cost? A few percent of recall and, for HNSW, a memory bill for the graph. You tune the dial — HNSW’s efSearch, IVF’s nprobe — to sit where your recall and latency budgets meet.

The convergence: OLTP, OLAP, and vectors

Step back and a pattern emerges: the same data increasingly needs three shapes at once. The app serves single-row reads and writes (OLTP — a row store), the business runs aggregations over billions of rows (OLAP — a column store), and the product does semantic search (a vector index). Historically each meant a separate system with its own copy of the data, stitched together by pipelines. The mid-2020s trend is convergence: blurring those boundaries so fewer systems serve more shapes.

• Transactional databases adding analytical and vector abilities (for example, PostgreSQL gaining approximate vector search through the popular pgvector extension, so one familiar database can do rows and vectors).
• Lakehouses serving low-latency queries that used to require a dedicated warehouse.
• “HTAP”-style systems trying to serve OLTP and OLAP from one engine, and embedded analytical engines (such as DuckDB) bringing column-store speed right next to the application.

        once: three systems, three copies, glued by pipelines
   OLTP (rows) ──etl──► OLAP (columns)        vector index (separate)

        trend (2025): convergence — fewer systems, more shapes each
   one platform: rows  +  columns  +  vectors  (with pipelines/CDC where it can't)

Convergence is a trade, not a free lunch

A single system that does OLTP and OLAP and vectors buys you radical simplicity — one copy of the data, one query language, no glue pipelines, no read-replica/CQRS split to operate. It costs you the specialization each dedicated system spent years perfecting: a jack-of-all-trades column store rarely matches a purpose-built warehouse on a giant aggregation, and a bolt-on vector index rarely matches a dedicated vector DB at billion-scale. The honest 2025 answer is “it depends on your scale”: converge while you can afford the generalist’s overhead, and split out the one workload that outgrows it — exactly the buy simplicity, cost specialization calculus that runs through this whole book.

Key trade-offs

• Streaming log vs request/response: replayability, decoupling, and audit vs eventual consistency, retained-log storage, and event-time/duplicate complexity.
• CDC vs dual writes: a reliable, ordered, no-extra-write change stream vs coupling to the DB’s log internals and idempotent downstream consumers.
• Lakehouse vs warehouse-plus-lake: one open, cheap tier with ACID and time travel vs a younger stack and load-bearing table-format metadata.
• HNSW vs IVF for ANN: low-latency, high-recall, memory-hungry graph vs compact, billion-scalable, nprobe-tunable clusters.
• Converged platform vs specialized systems: one copy and no glue vs the peak performance of purpose-built engines.

Check your understanding

1. What does Kafka’s retained, replayable log give you that a traditional delete-on-consume queue does not? Name two concrete benefits.
2. Why is change data capture safer than having the application write to both its database and Kafka? What primitive does CDC read, and why is that the right one?
3. What problem does the lakehouse solve, and how do table formats (Delta/Iceberg/Hudi) turn a pile of Parquet files into something with warehouse-like guarantees?
4. Compare HNSW and IVF as ANN indexes on structure, memory, and where each wins. Why use an approximate index instead of exact nearest-neighbour search at all?
5. Describe the OLTP/OLAP/vector convergence. What does collapsing them into one platform buy, and what does it cost?

Show answers

1. A queue deletes a message once consumed; Kafka retains the append-only log. That gives you (a) fan-out to many independent consumers reading at their own pace, and (b) replay — you can rebuild a downstream store from scratch or add a new consumer that reads all of history — plus it doubles as an audit trail. The cost is the storage to retain the log and designing around an append-only, eventually-consistent stream.
2. Writing to the DB and Kafka separately is a dual write: the two writes aren’t atomic, so a crash between them permanently desynchronizes them. CDC instead tails the database’s own transaction log (MySQL binlog / PostgreSQL WAL) — the log the DB already commits for durability and replication — and emits one event per committed change. Because it reuses the existing source-of-truth-for-changes, there’s no second write to go wrong and no lost-update window; the cost is coupling to log internals and needing idempotent consumers.
3. The lakehouse ends the warehouse-or-lake choice: cheap open object storage and warehouse guarantees. Table formats (Delta Lake, Iceberg, Hudi) add a transactional metadata log over the Parquet files, giving ACID transactions, schema evolution, and time travel (query an earlier version) — physically still Parquet, logically a governed table. The cost is a younger, more complex stack with load-bearing metadata.
4. HNSW is a multi-layer proximity graph searched by a greedy walk (~log N) — high recall, low latency, but memory-hungry (stores edges) and awkward to delete from; it wins at low-latency, moderate scale. IVF clusters vectors and probes the nearest nprobe cells — compact (especially IVF-PQ) and scales to billions, with recall tuned by nprobe, but needs a training step; it wins at large, memory-constrained scale. You use an approximate index because exact search is linear in the corpus (hopeless at hundreds of millions); ANN trades a few percent of recall for a near-logarithmic, orders-of-magnitude speedup.
5. The same data increasingly needs three shapes: single-row reads/writes (OLTP, row store), big aggregations (OLAP, column store), and semantic search (vector index) — historically three separate systems glued by pipelines. Convergence blurs them (e.g. Postgres + pgvector for rows and vectors; lakehouses serving warehouse queries; HTAP/embedded analytics). Collapsing them buys simplicity — one copy of the data, one query surface, no glue — and costs the specialization a purpose-built warehouse or vector DB achieves at extreme scale; converge while you can afford the generalist overhead, split out the workload that outgrows it.