← All projects

Amazon · 2023–2025 · design author & lead engineer

Source: a deterministic lineage layer under 44 billion original images.

The data and ingestion architecture that stores, sanitizes and indexes every original upload at full fidelity, with two-way lineage to every variant made from it. It replaced eventually consistent index queries with deterministic key lookups inside a live upload pipeline.

original assets stored, sanitized and indexed
44B+
new originals per year
6–7B
database write and storage overhead
3× → 1×
failed storage requests per 90 days, resolved
~800K

01 / architecture

How it works

Source architecture: the upload pipeline above a new source layer and the existing platform UPLOAD PIPELINE · 8 STEPS · LIVE TRAFFIC Checkrequest Sourcededup · new Validateoriginal Clean + storesource · new Uploadrestrictions Copydedup Generatecopies Link copiesto source duplicate → skip all processing SOURCE LAYER · NEW Source records DynamoDB · strongly consistent reads key: SHA-1 fingerprint + file type 17 fields · copy links + history conditional-write lock per record Cleaned originals S3 · GPS + private headers stripped 65,536 prefix partitions + one bucket set per collection written once, by the lock holder key lookup write once lock + put Phase 2 · designed source groups: team-defined relationships, e.g. localized variants EXISTING PLATFORM Copies display, resized, … 50B+ assets platform-wide Old secondary indexes 3× storage, 3 writes eventually consistent now off the lookup path Legacy raw originals no record, no metadata inconsistent naming Image processing resize · compress rotate · optimize backfill 1 2 3 4 5 6
  1. 1Deterministic compound keyRecords are keyed on a 160-bit SHA-1 hash plus file type, in one strongly consistent lookup before any processing. Adding the file type resolves zero-byte and cross-format collisions.
  2. 2Sanitize once, inherit everywhereGPS and other sensitive EXIF data are scrubbed at upload. Fixed attributes (aspect ratio, moderation flags, AI-generation markers) are extracted once and inherited by every variant, with no repeated compute.
  3. 3Atomic distributed lockA conditional write acts as a per-record lock, so parallel uploads of the same file produce exactly one record and one stored object, with no race conditions.
  4. 4Prefix-partitioned storageThe first 4 hex characters of the hash spread files over 65,536 prefixes. This resolved the staging bottleneck behind ~800,000 failed requests in 90 days.
  5. 5Bidirectional lineageNested source-to-variant maps plus a transformation history let you trace both ways, from an original to every thumbnail, localized graphic and compressed web format, and back.
  6. 6Backfill billions of originalsEvery existing original is converted into a record, and duplicate copies left by older bugs are merged along the way.

02 / trade-offs

Trade-offs, measured

Choosing the fingerprint

Records needed before a 50% chance of a collision (log scale)

10⁰ 10¹⁰ 10²⁰ 10³⁰ 10⁴⁰ MD51.0× cost SHA-1 ✓1.2× cost SHA-2561.6× cost 1.4 × 10²⁴ 4 × 10³⁸ catalog size: 44B 5.1B, already exceeded

MD5 would already be past a coin-flip chance of a collision at this catalog size. SHA-1 removes that risk for 20% more compute; SHA-256's extra attack resistance doesn't matter for deduplication.

Where the processing went

Daily image-optimization requests, before Source

Duplicates were detected only after full processing. Moving deduplication to the second pipeline step stops them before any compute is spent.

Lookup cost per record

Storage footprint table copies kept
Index
3×
Source
1×
Writes per update per record change
Index
3
Source
1
Read freshness after an upload
Indexeventually consistent, so new uploads can be missed
Sourcestrongly consistent, so they're always found

03 / decisions

Key decisions

04 / impact

Impact

database write and storage overhead, by replacing eventually consistent secondary-index queries with deterministic key lookups
3× → 1×
failed storage requests per 90 days, resolved by prefix partitioning
~800K
of daily optimization requests were duplicates; they're now caught before any compute is spent
67%

What it unlocked