Amazon · 2023–2025 · design author & lead engineer
Source: a deterministic lineage layer under 44 billion original images.
The data and ingestion architecture that stores, sanitizes and indexes every original upload at full fidelity, with two-way lineage to every variant made from it. It replaced eventually consistent index queries with deterministic key lookups inside a live upload pipeline.
- original assets stored, sanitized and indexed
- 44B+
- new originals per year
- 6–7B
- database write and storage overhead
- 3× → 1×
- failed storage requests per 90 days, resolved
- ~800K
01 / architecture
How it works
- New
- Changed
- Unchanged
- Removed from the lookup path
- 1Deterministic compound keyRecords are keyed on a 160-bit SHA-1 hash plus file type, in one strongly consistent lookup before any processing. Adding the file type resolves zero-byte and cross-format collisions.
- 2Sanitize once, inherit everywhereGPS and other sensitive EXIF data are scrubbed at upload. Fixed attributes (aspect ratio, moderation flags, AI-generation markers) are extracted once and inherited by every variant, with no repeated compute.
- 3Atomic distributed lockA conditional write acts as a per-record lock, so parallel uploads of the same file produce exactly one record and one stored object, with no race conditions.
- 4Prefix-partitioned storageThe first 4 hex characters of the hash spread files over 65,536 prefixes. This resolved the staging bottleneck behind ~800,000 failed requests in 90 days.
- 5Bidirectional lineageNested source-to-variant maps plus a transformation history let you trace both ways, from an original to every thumbnail, localized graphic and compressed web format, and back.
- 6Backfill billions of originalsEvery existing original is converted into a record, and duplicate copies left by older bugs are merged along the way.
02 / trade-offs
Trade-offs, measured
Choosing the fingerprint
Records needed before a 50% chance of a collision (log scale)
MD5 would already be past a coin-flip chance of a collision at this catalog size. SHA-1 removes that risk for 20% more compute; SHA-256's extra attack resistance doesn't matter for deduplication.
Where the processing went
Daily image-optimization requests, before Source
Duplicates were detected only after full processing. Moving deduplication to the second pipeline step stops them before any compute is spent.
Lookup cost per record
03 / decisions
Key decisions
- SHA-1 fingerprints instead of MD5 or SHA-256MD5 reaches a 50% collision chance at around 5 billion records, well below this catalog's size. SHA-1 pushes that past 1024 while running only about 20% slower than MD5. SHA-256 cost about 60% more, and its extra attack resistance doesn't matter for deduplication. The old MD5 value is kept so existing customers don't break.
- A new record type, not a reused oneStoring originals in the existing copies table would have been less work up front. It would also have grown that table by 30%+ and kept the slow index in the critical path. A separate table keyed by fingerprint and file type removes the index from lookups entirely.
- Phase 2: customer-defined groups, not hard-coded linksFor teams that make derived images, such as one product photo localized into many languages, I designed "source groups" where each team defines its own relationship schema. I rejected a simpler option that would have locked every group to one parent image, because that choice would be hard to reverse later.
04 / impact
Impact
- database write and storage overhead, by replacing eventually consistent secondary-index queries with deterministic key lookups
- 3× → 1×
- failed storage requests per 90 days, resolved by prefix partitioning
- ~800K
- of daily optimization requests were duplicates; they're now caught before any compute is spent
- 67%
What it unlocked
- AI and computer-vision trainingFull-resolution originals, where before only compressed copies were available
- Storefront renderingStorefronts for multiple tenants can render from the original, not a resized copy
- Compute once, inherit everywhereModeration flags, AI-generation markers and aspect ratio are computed once per original
- Complete legal take-downsEvery thumbnail, localized graphic and web format can be reached from its source
- A base for later systemsThe perceptual compression engine records its source → variant links in this layer