ExerciseProduction incident
Production incident
The product cache that quietly loses writes
45 minintermediate3–8 yrs
A real incident: symptom first, cause hidden, tradeoff at the end.
The incident
Your order service caches product lookups in a plain HashMap held in a
@Service singleton. It works perfectly in dev and in QA.
In production, under real concurrent load, two things happen:
1. The cache reports a size smaller than the number of successful puts.
2. Occasionally a lookup returns null for a product that was definitely
cached seconds earlier, causing a redundant DB hit.
There is no exception. Nothing appears in the logs. CPU is normal.
Diagnose the root cause and fix it. Then answer the design question:
is a Map even the right cache here?
What this teaches
- HashMap has no synchronization, so concurrent put() loses updates
- A lost update is silent — no exception, no log, just wrong numbers
- The Java 7 infinite-loop bug is fixed; unsafe concurrency is not
- ConcurrentHashMap vs Collections.synchronizedMap: lock scope matters
- An unbounded Map is a memory leak, not a cache — real caches need eviction and a TTL
Starter
Starter.java
import java.util.*;
import java.util.concurrent.*;
/**
* PRODUCTION EXERCISE — the product cache that quietly loses writes.
*
* Run this as-is. Tests.java hammers it from many threads.
* It will fail. Your job is to find out why, then fix ProductCache.
*
* Rules:
* - Do not change Tests.java.
* - Keep the same public method signatures.
* - Leave a comment explaining WHY you picked your fix over the
* alternatives. The reasoning is the exercise; the one-line fix is not.
*/
public class Starter {
static class ProductCache {
// ── THE BUG LIVES HERE ────────────────────────────────────────
// Looks harmless. Works flawlessly on one thread.
private final Map<String, String> cache = new HashMap<>();
// ──────────────────────────────────────────────────────────────
public void put(String sku, String name) {
cache.put(sku, name);
}
public String get(String sku) {
return cache.get(sku);
}
public int size() {
return cache.size();
}
}
public static void main(String[] args) throws Exception {
ProductCache cache = new ProductCache();
int threads = 8;
int perThread = 2_000;
int expected = threads * perThread;
ExecutorService pool = Executors.newFixedThreadPool(threads);
CountDownLatch start = new CountDownLatch(1);
for (int t = 0; t < threads; t++) {
final int id = t;
pool.submit(() -> {
try {
start.await();
for (int i = 0; i < perThread; i++) {
cache.put("SKU-" + id + "-" + i, "product-" + id + "-" + i);
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
});
}
start.countDown();
pool.shutdown();
pool.awaitTermination(30, TimeUnit.SECONDS);
System.out.println("expected entries = " + expected);
System.out.println("actual entries = " + cache.size());
System.out.println("lost writes = " + (expected - cache.size()));
int missing = 0;
for (int t = 0; t < threads; t++) {
for (int i = 0; i < perThread; i++) {
if (cache.get("SKU-" + t + "-" + i) == null) missing++;
}
}
System.out.println("unreadable keys = " + missing);
System.out.println(missing == 0 && cache.size() == expected ? "PASS" : "FAIL");
}
}Run it locally:
cd exercises/java/collections/hashmap-internals/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out StarterHints
Hint 1
Nothing here is synchronized. What do two threads writing to the same bucket array do to each other?
Hint 2
size is a plain int field. ++ on it is read-modify-write, not atomic.
Hint 3
Fixing the data race gets you correct. It does not get you a cache — what stops this map growing until the heap dies?
Done when
- Concurrent writes are not lost: final size equals the number of distinct keys
- No lookup returns null for a key that was successfully cached
- The chosen structure is justified against synchronizedMap in a comment
Solution
Show the solution — try it yourself first
Solution.java
import java.util.*;
import java.util.concurrent.*;
/**
* REFERENCE SOLUTION — shown only after the learner passes or gives up.
*
* ROOT CAUSE
* ----------
* HashMap has zero synchronization. Two threads calling put() concurrently race on:
*
* 1. the bucket array — thread A writes a Node into table[i], thread B
* overwrites the same slot with its own Node. A's entry is gone.
* 2. `size` — a plain int. `++size` is read-modify-write, so increments are lost.
* 3. resize() — two threads can build separate new tables; one is discarded
* along with everything written into it.
*
* All three are SILENT. No exception, no log, no CPU anomaly. The only symptom
* is that the numbers are wrong, which is why this class of bug reaches prod.
*
* WHY NOT Collections.synchronizedMap(new HashMap<>())?
* -----------------------------------------------------
* It is correct, but it wraps every operation in one lock over the whole map.
* A read-heavy cache — which is what this is — serialises every get() behind
* that single monitor. ConcurrentHashMap locks per-bin on write and is
* lock-free on read, so reads scale with cores instead of contending.
*
* WHY NOT just synchronized methods here?
* ---------------------------------------
* Same problem as synchronizedMap, plus you own the bug the next time someone
* adds a method and forgets the keyword.
*
* THE DEEPER POINT
* ----------------
* Fixing the race makes this CORRECT. It does not make it a CACHE. An unbounded
* map that is only ever written to is a memory leak with a friendly name.
* A real cache needs a size ceiling and an expiry policy — see BoundedCache below.
*/
public class Solution {
/** Fix 1 — correct, and scales for a read-heavy cache. */
static class ProductCache {
private final Map<String, String> cache = new ConcurrentHashMap<>();
public void put(String sku, String name) {
cache.put(sku, name);
}
public String get(String sku) {
return cache.get(sku);
}
public int size() {
return cache.size();
}
}
/**
* Fix 2 — the answer an interviewer actually wants at 4+ years.
*
* Bounded by capacity with LRU eviction. In real code reach for Caffeine,
* which adds TTL, hit-rate stats and a far better eviction policy. This
* version is here to show the mechanism rather than the dependency.
*
* Tradeoff being made explicitly: capping at maxEntries bounds memory, and
* costs hit rate whenever the working set is larger than the cap. That is
* a deliberate exchange — an unbounded cache trades an OutOfMemoryError for
* a hit rate you never measured.
*/
static class BoundedCache {
private final Map<String, String> cache;
BoundedCache(int maxEntries) {
this.cache = Collections.synchronizedMap(
new LinkedHashMap<>(16, 0.75f, true) { // true = access order
@Override
protected boolean removeEldestEntry(Map.Entry<String, String> eldest) {
return size() > maxEntries;
}
});
}
public void put(String sku, String name) { cache.put(sku, name); }
public String get(String sku) { return cache.get(sku); }
public int size() { return cache.size(); }
}
public static void main(String[] args) throws Exception {
ProductCache cache = new ProductCache();
int threads = 8, perThread = 2_000, expected = threads * perThread;
ExecutorService pool = Executors.newFixedThreadPool(threads);
CountDownLatch start = new CountDownLatch(1);
for (int t = 0; t < threads; t++) {
final int id = t;
pool.submit(() -> {
try {
start.await();
for (int i = 0; i < perThread; i++) {
cache.put("SKU-" + id + "-" + i, "product-" + id + "-" + i);
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
});
}
start.countDown();
pool.shutdown();
pool.awaitTermination(30, TimeUnit.SECONDS);
int missing = 0;
for (int t = 0; t < threads; t++)
for (int i = 0; i < perThread; i++)
if (cache.get("SKU-" + t + "-" + i) == null) missing++;
System.out.println("expected entries = " + expected);
System.out.println("actual entries = " + cache.size());
System.out.println("lost writes = " + (expected - cache.size()));
System.out.println("unreadable keys = " + missing);
System.out.println(missing == 0 && cache.size() == expected ? "PASS" : "FAIL");
}
}Stretch
Swap the fix for a bounded cache with expiry (Caffeine, or a
LinkedHashMap with removeEldestEntry). Explain the tradeoff you just
made between hit rate and memory ceiling.