Design Dropbox (File Sync & Storage)

A file-sync service (Dropbox, Google Drive, OneDrive, iCloud Drive) lets you drop a file into a folder on one device and have it appear, byte-for-byte, on every other device you own — and on your colleagues’ devices when you share it. It feels like magic and looks, on the whiteboard, like “upload a file, download a file.” The interesting engineering is hidden in three questions: how do you avoid re-uploading a 2 GB video because someone changed its title, how do you store the same file once when a thousand employees all have a copy, and how do you reconcile two people who edited the same file offline? The answers are chunking, content-addressed deduplication, and conflict resolution — and every one of them is a bet in the storage-versus-bandwidth trade-off. We’ll walk the six-step framework.

1. Requirements

Functional

• Upload a file and have it sync to all of a user’s devices and any shared collaborators.
• Download / read any file the user has access to.
• Edit a file and propagate only what changed.
• Detect and surface conflicts when two devices edit the same file independently.
• Share a folder; revoking access removes it.

Non-functional

• Durability above all — a sync service that loses your only copy of a file is worse than useless. Target the “eleven nines” range that object stores advertise.
• Bandwidth-efficient. Most edits touch a tiny fraction of a file; re-uploading the whole thing on every save is unacceptable on a laptop tethered to a phone.
• Eventually consistent across devices, with fast convergence — seconds, not minutes — once a device is online.
• Read-and-write mixed, but the byte volume is dominated by block transfers, while the request volume is dominated by tiny metadata operations. That split, as we’ll see, dictates the architecture.

2. Back-of-envelope estimation

Assume 500M registered users, 100M daily active, each syncing ~100 files/day averaging 1 MB, with a 10:1 read:write byte ratio.

Metadata ops:  100M users × 100 file events / 86,400s ≈ 116,000 ops/sec  (small rows)
Write bytes:   100M × 100 × 1 MB ≈ 10 PB ingested/day  (before dedup & delta)
Storage:       grows by petabytes/day → exabyte-scale within a few years
Block size:    files split into blocks of up to 4 MB

Two numbers jump out. The metadata path is high-QPS but moves bytes the size of a tweet; the block path is lower-QPS but moves the entire internet’s worth of bytes. What does this buy us? A clean architectural seam: serve metadata from a fast, transactional, replicated database and push the giant opaque blocks into a separate, cheap, massively-durable object store. They scale on completely different axes. See Back-of-Envelope Estimation.

3. API sketch

POST /files/commit
  body: { path, file_id, block_list: [hash1, hash2, ...], mtime }
  → reconciles metadata; returns which blocks the server is MISSING

PUT  /blocks/{block_hash}
  body: <up to 4 MB of bytes>
  → store a block by its content hash (idempotent; no-op if it already exists)

GET  /blocks/{block_hash}        → fetch one block by hash
GET  /delta?cursor=<token>       → "what changed since I last synced?" (long-poll)

The shape that matters: a client never says “upload this file.” It says “here is the list of block hashes that make up this file; tell me which ones you don’t already have.” The server replies with the missing set, the client uploads only those, then commits the metadata. That single indirection — files described as ordered lists of content-addressed blocks — is what makes dedup and delta sync fall out almost for free.

4. Data model — metadata versus block storage

Split the world in two.

METADATA STORE (transactional, indexed, replicated SQL)
  files
    file_id      UUID (pk)
    namespace_id (the shared folder / user root it lives in)
    path         "/Photos/trip.raw"
    block_list   ordered [block_hash, ...]   ← the file IS this list
    size, mtime, version
  namespaces / sharing / device cursors ...

BLOCK STORE (content-addressed object store)
  key   = SHA-256(block_bytes)               ← the address IS the content
  value = the (up to 4 MB) block bytes, replicated across zones

A file is not a blob — it’s a row of metadata pointing at an ordered list of block hashes. The metadata store is a classic relational workload: small rows, secondary indexes on path and namespace, transactional updates when you rename or move. The block store is the opposite: no queries, no joins, just get/put by an opaque key, optimized for throughput and eleven-nines durability. Keying blocks by their SHA-256 hash is the linchpin of the next two sections.

Why content-address the blocks?

Using SHA-256(content) as the storage key makes writes idempotent and storage self-deduplicating. Two users uploading the identical block produce the identical key, so the second PUT is a no-op. A file that hasn’t changed has the same hashes, so re-syncing it transfers nothing. The hash is simultaneously the integrity check (corruption changes the hash), the dedup key, and the cache key — one decision buys three properties.

5. Chunking and content-addressed dedup — the core decision

Splitting files into blocks is the engine room. The question is where to draw the block boundaries, and there are two families.

Fixed-size chunking

Cut the file every 4 MB, hash each block, done. Simple, fast, trivially parallel — and it’s the classic approach (Dropbox historically used fixed blocks of up to 4 MB). It has one nasty failure mode: the boundary-shift problem. Insert a single byte at the start of a file and every block boundary slides over by one byte, so every block gets a new hash and the whole file re-uploads — even though 99.999% of the bytes are unchanged.

Content-defined chunking (CDC)

Instead of cutting at fixed offsets, slide a rolling hash (a Rabin fingerprint) over the bytes and cut a boundary wherever the hash matches a pattern (e.g. the low 13 bits are zero, giving ~8 KB average chunks). Now boundaries are anchored to the content, not the offset. Insert a byte and only the one chunk containing it changes; every downstream boundary re-aligns. CDC is what backup tools (restic, borg) and many modern sync engines use precisely because it survives insertions.

                       Fixed 4 MB blocks        Content-defined chunking
boundary on insert     all blocks shift → bad   one chunk changes → good
cpu cost               cheap                     rolling hash over every byte
dedup across files     block-aligned only        finds shared spans anywhere
implementation         trivial                   trickier; tune avg chunk size

By the numbers — what dedup and chunking actually save

Cross-file dedup. A 15 MB onboarding PDF lands in the Dropbox of all 5,000 employees.

naïve storage:   15 MB × 5,000      = 75 GB
content-addressed: 15 MB (stored once) + 5,000 tiny metadata pointers ≈ 15 MB
saving:          ~99.98% of the bytes

Delta on edit. You change a chapter heading in a 100 MB manuscript split into 4 MB blocks.

blocks total:    100 MB / 4 MB = 25 blocks
blocks changed:  1
uploaded:        4 MB  (the changed block only)
bandwidth saved: 96 MB / 100 MB = 96%

The same hash list that powers dedup powers the delta: the server already knows 24 of your 25 hashes, so it asks for one block. Storage and bandwidth fall out of the same mechanism.

Under the hood — the commit handshake

The sync client and server negotiate using nothing but hashes:

1. client hashes the file → block_list = [h0, h1, h2, ...]
2. client → POST /files/commit { path, block_list }
3. server diffs block_list against what it already stores
       → replies "I'm missing [h2]"            (it had h0, h1 from dedup)
4. client → PUT /blocks/h2   (uploads ONLY the missing bytes)
5. server commits the new metadata row → file version bumps
6. server pushes a delta notification to the user's other devices

Steps 3–4 are where the storage-versus-bandwidth trade lives. Smaller blocks mean more dedup hits and smaller deltas (less bandwidth) but more metadata rows, more hashes to track, and more per-block overhead (more storage and CPU for the index). Bigger blocks mean a lean metadata store but coarse dedup and chunky deltas. What does each buy, and what does it cost? Small chunks buy bandwidth and pay in metadata; large chunks buy a simple index and pay in wasted transfer.

6. Sync, notifications, and conflict resolution

A device that’s online keeps a long-poll open against /delta (see polling vs WebSockets vs SSE); when anything in its namespaces changes, the notification server wakes it with a cursor and it pulls the changed metadata, then the missing blocks. On a LAN, clients can fetch blocks from each other directly instead of the cloud — same content-address, nearer source.

The hard part is concurrent edits. Two devices both edit report.docx while offline; both come back and commit. You cannot silently pick a winner — that’s a lost write, the cardinal sin of a sync product. File-sync services deliberately do not auto-merge arbitrary binary files (you can’t three-way-merge a .psd). Instead they detect the divergence via a version vector per file: if neither version descends from the other, it’s a true conflict, and the loser is preserved as a “conflicted copy” (report (Alice's conflicted copy 2026-06-26).docx). Ugly, but no edit is ever destroyed — the human resolves it.

Why not just last-write-wins?

Last-write-wins by wall-clock timestamp is tempting and almost always wrong here: clocks disagree across machines, and “newer timestamp” has nothing to do with “the edit the user wanted to keep.” A version vector answers the only question that matters — did these two versions share a common ancestor, or did they genuinely fork? — and a fork becomes a conflicted copy rather than a deletion.

                       ┌──────────────┐      ┌─────────────────────┐
  device A ──commit──► │ Metadata DB  │ ◄──► │ Notification server │
                       │ (SQL, repl.) │      │  (long-poll/push)   │
  device B ◄─delta──── └──────┬───────┘      └─────────────────────┘
                              │ block hashes
                       ┌──────┴────────────────────────────┐
                       │ Block store (content-addressed,    │
  PUT/GET /blocks/{h}  │ deduped, cross-zone replicated)    │
                       └────────────────────────────────────┘

Under the hood — keep blocks immutable, mutate only metadata

Because blocks are addressed by content, they are immutable — you never overwrite a block, you write a new block at a new hash and re-point the metadata. That makes versioning and undo nearly free (old block lists still reference old blocks), simplifies caching (a hash never goes stale), and lets garbage collection be a separate, lazy job that reaps blocks no live file references. The cost is storage drag: you keep historical blocks around, so a background GC and retention policy become load-bearing parts of the system, and “delete” is eventual, not immediate.

Key trade-offs

• Storage vs bandwidth (the whole story): smaller blocks and CDC slash transfer but inflate metadata and index cost; bigger blocks do the reverse. Dedup trades a little hashing CPU for enormous storage savings.
• Metadata store vs block store: splitting them lets each scale on its own axis (transactional QPS vs raw durable bytes) — at the cost of a two-system commit dance and the chance of metadata pointing at a not-yet-uploaded block.
• Conflicted copies vs auto-merge: preserving both edits never loses data but pushes resolution onto the user; auto-merge is friendlier but unsafe for opaque binaries.
• Immutable blocks vs prompt deletion: content-addressing buys free versioning and stale-proof caches; it costs you a garbage collector and delayed reclamation.

It actually happened — Dropbox built its own storage to escape the bandwidth-and-cost bill

For years Dropbox stored its blocks on Amazon S3 while running its own metadata tier. At exabyte scale the economics flipped: in 2016 Dropbox publicly migrated the vast majority of its user data off S3 onto its own custom storage infrastructure, “Magic Pocket.” The motivation was exactly the trade-off this page is about — at that volume, owning the durable-block layer (custom hardware, its own replication and erasure coding) was dramatically cheaper than renting it. Dropbox’s 2018 IPO filing later credited infrastructure optimization, including Magic Pocket, with tens of millions of dollars in savings and a meaningful gross-margin improvement. The lesson: the metadata/block split isn’t just tidy on a whiteboard — it’s what let one company swap out its entire storage backend without users ever noticing, because the metadata layer kept speaking the same content-addressed hashes the whole time.

Check your understanding

1. Why is splitting the system into a metadata store and a block store the first and most important architectural decision? What does each half optimize for?
2. Explain content-addressed storage. How does keying blocks by SHA-256(content) give you dedup, delta sync, and integrity checking from one mechanism?
3. What is the boundary-shift problem with fixed-size chunking, and how does content-defined chunking solve it? What does CDC cost in return?
4. Estimate the bandwidth saved when one 4 MB block changes in a 100 MB file, and the storage saved when a 15 MB file is shared with 5,000 users. Which trade-off do both illustrate?
5. Why do sync services create “conflicted copies” instead of using last-write-wins by timestamp? What data structure decides whether two versions truly conflict?

Show answers

1. The two halves move data on completely different axes: metadata is high-QPS but tiny rows (paths, versions, block lists) that need transactions and indexes — a relational store — while blocks are lower-QPS but petabyte-to-exabyte opaque bytes that need cheap, massively-durable get/put by key. Splitting them lets each scale independently and, as Magic Pocket showed, lets you replace the block backend without touching the metadata contract.
2. Content-addressing stores each block under a key equal to the hash of its bytes. Dedup: identical content yields an identical key, so duplicate PUTs are no-ops. Delta sync: an unchanged block keeps its hash, so the server already has it and only changed blocks transfer. Integrity: any corruption changes the bytes and therefore the hash, so a mismatch is detectable. One key, three properties — plus it doubles as a stale-proof cache key.
3. Boundary shift: with fixed 4 MB cuts, inserting one byte near the start slides every later boundary, changing every block’s hash and forcing a full re-upload though almost nothing changed. CDC cuts boundaries where a rolling (Rabin) hash matches a pattern, anchoring them to content, so an insertion changes only the affected chunk and downstream boundaries re-align. The cost: running a rolling hash over every byte (more CPU) and tuning the average chunk size.
4. Delta: a 100 MB file is 25 blocks of 4 MB; changing one uploads 4 MB, saving 96 MB = 96% of the bandwidth. Dedup: naïvely 15 MB × 5,000 = 75 GB, but content-addressed storage keeps it once (~15 MB plus tiny pointers), saving ~99.98%. Both illustrate the storage-versus-bandwidth trade-off — and that the same block-hash mechanism drives both savings.
5. Last-write-wins by timestamp is unsafe: clocks disagree across machines and “newer timestamp” has nothing to do with which edit the user wanted — it silently destroys a real edit. A version vector per file decides whether the two versions share a common ancestor (one descends from the other → fast-forward, no conflict) or genuinely forked (neither descends → true conflict). On a true fork, the loser is preserved as a conflicted copy so no edit is ever lost.