Why Cache?

In any distributed system, the database is often the bottleneck. Disk I/O is slow (10-100ms). Caching solves this by keeping frequently accessed data in memory (RAM), which is orders of magnitude faster (1-5ms).

The Latency Hierarchy:

1. L1 Cache: ~1 ns
2. RAM (Memory): ~100 ns
3. SSD (Disk): ~100 µs (1,000x slower)
4. Network Call: ~50 ms (500,000x slower)

The Core Concepts

When implementing a cache, you must make three major architectural decisions.

1. Where does the cache live?

• Client Side: Browser cache, iOS local storage. Saves network calls entirely.
• CDN: Content Delivery Network. Geographic caching for static assets (images, CSS).
• Server Side: Redis/Memcached. sitting in front of your database.

2. How do we write to it? (Strategies)

Do you write to the cache first? Or the database? Where is the source of truth?

• Should you use Cache-Aside (safest)?
• Or Write-Back (fastest)?
• Read the Deep Dive on Caching Strategies →

3. What do we delete? (Eviction)

When your 16GB Redis instance is full, what gets thrown out?

• Do you delete the Oldest stuff?
• Or the Least Frequently Used stuff?
• Read the Deep Dive on Eviction Policies (LRU/LFU) →

Common Pitfalls

The "Stale Data" Problem

Caching introduces a new problem: Consistency. If you update a User in the DB, but the Cache still shows the old version, the user sees wrong data.

"There are only two hard things in Computer Science: cache invalidation and naming things."

Cache Stampede (Dog-piling)

If a popular key (e.g., "homepage_news") expires, 10,000 users might hit the database simultaneously to regenerate it.

• Solution: Locking (Mutex) or probabilistic early expiration.

Popular Caching Technologies

Real-World Examples

1. Netflix

Netflix caches movie metadata, user preferences, and thumbnails at multiple layers: CDN edge, regional data centers, and in-memory on servers.

2. Twitter Timeline

Twitter's timeline is pre-computed and cached using Redis. When you tweet, it's fanned out to followers' cached timelines.

3. E-commerce Product Pages

Amazon caches product details, reviews, and pricing. Inventory is read-through to ensure accuracy.

The Math: Why Hit Ratio Is Everything

The value of a cache is captured in one formula — the effective latency your users experience:

Plug in realistic numbers (cache: 1ms, database: 50ms):

Two non-obvious lessons hide in this table. First, the jump from 90% to 99% matters more than from 0% to 90% in database load terms — each "nine" cuts the miss traffic by 10x. Second, and more dangerous: a system running at 99% hit ratio has a database sized for 1% of true traffic. If the cache layer restarts cold, the database suddenly receives 100x its normal load and dies instantly. This is why mature systems warm caches before taking traffic and treat "cache cluster restart" as a serious operational event, not a routine one.

Sizing: The 80/20 Reality

Real access patterns are heavily skewed (Zipfian): a small fraction of keys receives most of the traffic. In practice this means you usually don't need RAM equal to your dataset — caching the hottest 20% of data often serves 80%+ of requests. Start by estimating: (number of hot objects) × (average object size) × ~1.5 overhead factor, then measure the actual hit ratio and adjust. Chasing the last few percent of hit ratio by doubling memory is often worse ROI than fixing TTLs or key design.

How Fresh Is Fresh Enough? (Invalidation, Simply)

Every caching conversation eventually becomes a freshness conversation. There are only three basic tools, usually combined:

1. TTL (time-to-live): every entry expires after N seconds. Simple, self-healing, and the right default. The question "what TTL?" is really a business question: how stale can a price / profile / article be before it causes harm? Seconds for inventory, minutes for social profiles, hours for a blog post.
2. Explicit invalidation: when data changes, delete the corresponding cache key. Precise but fragile — every write path must remember every affected key, and a missed one means indefinite staleness. (This is the "hard problem" the famous quote is about.)
3. Versioned keys: embed a version in the key (user:42:v7). Updates write a new version and change the pointer; old entries simply age out. Sidesteps invalidation races at the cost of extra memory.

A robust rule of thumb: use explicit invalidation for correctness, but always keep a TTL as the backstop — so any bug in invalidation heals itself within minutes instead of persisting forever.

Metrics That Tell You the Truth

A cache without instrumentation is a rumor. The four numbers to dashboard:

• Hit ratio (per key-family, not just global — a 95% global ratio can hide a 40% ratio on your most expensive query).
• Eviction rate: high evictions with a full cache means you're memory-starved; entries die before they can be re-used.
• p99 latency of misses: this is what your users actually feel when it matters.
• Origin traffic: the database QPS the cache is absorbing — this number is your blast radius if the cache disappears.

Interview Tips 💡

When discussing caching in system design interviews:

1. Identify read vs. write ratio: Caching benefits read-heavy systems most.
2. Estimate cache size: How much data will you cache? Can it fit in RAM?
3. Discuss invalidation strategy: How will you keep cache fresh?
4. Consider cache stampede: What happens when many requests hit an expired key simultaneously? (Solution: lock and single-fetch, or staggered TTLs)

Related Concepts

• Redis Internals — Deep dive into how Redis works
• CDN — Edge caching for global content delivery
• Database Replication — Another strategy for read scaling
• Rate Limiting — Often used alongside caching
• Consistent Hashing — Used to distribute cache keys across nodes