Production incident

The product cache that quietly loses writes

45 minintermediate38 yrs

A real incident: symptom first, cause hidden, tradeoff at the end.

The incident

Your order service caches product lookups in a plain HashMap held in a @Service singleton. It works perfectly in dev and in QA. In production, under real concurrent load, two things happen: 1. The cache reports a size smaller than the number of successful puts. 2. Occasionally a lookup returns null for a product that was definitely cached seconds earlier, causing a redundant DB hit. There is no exception. Nothing appears in the logs. CPU is normal. Diagnose the root cause and fix it. Then answer the design question: is a Map even the right cache here?

What this teaches

  • HashMap has no synchronization, so concurrent put() loses updates
  • A lost update is silent — no exception, no log, just wrong numbers
  • The Java 7 infinite-loop bug is fixed; unsafe concurrency is not
  • ConcurrentHashMap vs Collections.synchronizedMap: lock scope matters
  • An unbounded Map is a memory leak, not a cache — real caches need eviction and a TTL

Starter

Starter.java
import java.util.*;
import java.util.concurrent.*;

/**
 * PRODUCTION EXERCISE — the product cache that quietly loses writes.
 *
 * Run this as-is. Tests.java hammers it from many threads.
 * It will fail. Your job is to find out why, then fix ProductCache.
 *
 * Rules:
 *   - Do not change Tests.java.
 *   - Keep the same public method signatures.
 *   - Leave a comment explaining WHY you picked your fix over the
 *     alternatives. The reasoning is the exercise; the one-line fix is not.
 */
public class Starter {

    static class ProductCache {

        // ── THE BUG LIVES HERE ────────────────────────────────────────
        // Looks harmless. Works flawlessly on one thread.
        private final Map<String, String> cache = new HashMap<>();
        // ──────────────────────────────────────────────────────────────

        public void put(String sku, String name) {
            cache.put(sku, name);
        }

        public String get(String sku) {
            return cache.get(sku);
        }

        public int size() {
            return cache.size();
        }
    }

    public static void main(String[] args) throws Exception {
        ProductCache cache = new ProductCache();

        int threads = 8;
        int perThread = 2_000;
        int expected = threads * perThread;

        ExecutorService pool = Executors.newFixedThreadPool(threads);
        CountDownLatch start = new CountDownLatch(1);

        for (int t = 0; t < threads; t++) {
            final int id = t;
            pool.submit(() -> {
                try {
                    start.await();
                    for (int i = 0; i < perThread; i++) {
                        cache.put("SKU-" + id + "-" + i, "product-" + id + "-" + i);
                    }
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
            });
        }

        start.countDown();
        pool.shutdown();
        pool.awaitTermination(30, TimeUnit.SECONDS);

        System.out.println("expected entries = " + expected);
        System.out.println("actual   entries = " + cache.size());
        System.out.println("lost writes      = " + (expected - cache.size()));

        int missing = 0;
        for (int t = 0; t < threads; t++) {
            for (int i = 0; i < perThread; i++) {
                if (cache.get("SKU-" + t + "-" + i) == null) missing++;
            }
        }
        System.out.println("unreadable keys  = " + missing);
        System.out.println(missing == 0 && cache.size() == expected ? "PASS" : "FAIL");
    }
}

Run it locally:

cd exercises/java/collections/hashmap-internals/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out Starter

Hints

  1. Hint 1

    Nothing here is synchronized. What do two threads writing to the same bucket array do to each other?

  2. Hint 2

    size is a plain int field. ++ on it is read-modify-write, not atomic.

  3. Hint 3

    Fixing the data race gets you correct. It does not get you a cache — what stops this map growing until the heap dies?

Done when

  • Concurrent writes are not lost: final size equals the number of distinct keys
  • No lookup returns null for a key that was successfully cached
  • The chosen structure is justified against synchronizedMap in a comment

Solution

Show the solution — try it yourself first
Solution.java
import java.util.*;
import java.util.concurrent.*;

/**
 * REFERENCE SOLUTION — shown only after the learner passes or gives up.
 *
 * ROOT CAUSE
 * ----------
 * HashMap has zero synchronization. Two threads calling put() concurrently race on:
 *
 *   1. the bucket array — thread A writes a Node into table[i], thread B
 *      overwrites the same slot with its own Node. A's entry is gone.
 *   2. `size` — a plain int. `++size` is read-modify-write, so increments are lost.
 *   3. resize() — two threads can build separate new tables; one is discarded
 *      along with everything written into it.
 *
 * All three are SILENT. No exception, no log, no CPU anomaly. The only symptom
 * is that the numbers are wrong, which is why this class of bug reaches prod.
 *
 * WHY NOT Collections.synchronizedMap(new HashMap<>())?
 * -----------------------------------------------------
 * It is correct, but it wraps every operation in one lock over the whole map.
 * A read-heavy cache — which is what this is — serialises every get() behind
 * that single monitor. ConcurrentHashMap locks per-bin on write and is
 * lock-free on read, so reads scale with cores instead of contending.
 *
 * WHY NOT just synchronized methods here?
 * ---------------------------------------
 * Same problem as synchronizedMap, plus you own the bug the next time someone
 * adds a method and forgets the keyword.
 *
 * THE DEEPER POINT
 * ----------------
 * Fixing the race makes this CORRECT. It does not make it a CACHE. An unbounded
 * map that is only ever written to is a memory leak with a friendly name.
 * A real cache needs a size ceiling and an expiry policy — see BoundedCache below.
 */
public class Solution {

    /** Fix 1 — correct, and scales for a read-heavy cache. */
    static class ProductCache {

        private final Map<String, String> cache = new ConcurrentHashMap<>();

        public void put(String sku, String name) {
            cache.put(sku, name);
        }

        public String get(String sku) {
            return cache.get(sku);
        }

        public int size() {
            return cache.size();
        }
    }

    /**
     * Fix 2 — the answer an interviewer actually wants at 4+ years.
     *
     * Bounded by capacity with LRU eviction. In real code reach for Caffeine,
     * which adds TTL, hit-rate stats and a far better eviction policy. This
     * version is here to show the mechanism rather than the dependency.
     *
     * Tradeoff being made explicitly: capping at maxEntries bounds memory, and
     * costs hit rate whenever the working set is larger than the cap. That is
     * a deliberate exchange — an unbounded cache trades an OutOfMemoryError for
     * a hit rate you never measured.
     */
    static class BoundedCache {

        private final Map<String, String> cache;

        BoundedCache(int maxEntries) {
            this.cache = Collections.synchronizedMap(
                new LinkedHashMap<>(16, 0.75f, true) {   // true = access order
                    @Override
                    protected boolean removeEldestEntry(Map.Entry<String, String> eldest) {
                        return size() > maxEntries;
                    }
                });
        }

        public void put(String sku, String name) { cache.put(sku, name); }
        public String get(String sku)            { return cache.get(sku); }
        public int size()                        { return cache.size(); }
    }

    public static void main(String[] args) throws Exception {
        ProductCache cache = new ProductCache();

        int threads = 8, perThread = 2_000, expected = threads * perThread;
        ExecutorService pool = Executors.newFixedThreadPool(threads);
        CountDownLatch start = new CountDownLatch(1);

        for (int t = 0; t < threads; t++) {
            final int id = t;
            pool.submit(() -> {
                try {
                    start.await();
                    for (int i = 0; i < perThread; i++) {
                        cache.put("SKU-" + id + "-" + i, "product-" + id + "-" + i);
                    }
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
            });
        }

        start.countDown();
        pool.shutdown();
        pool.awaitTermination(30, TimeUnit.SECONDS);

        int missing = 0;
        for (int t = 0; t < threads; t++)
            for (int i = 0; i < perThread; i++)
                if (cache.get("SKU-" + t + "-" + i) == null) missing++;

        System.out.println("expected entries = " + expected);
        System.out.println("actual   entries = " + cache.size());
        System.out.println("lost writes      = " + (expected - cache.size()));
        System.out.println("unreadable keys  = " + missing);
        System.out.println(missing == 0 && cache.size() == expected ? "PASS" : "FAIL");
    }
}

Stretch

Swap the fix for a bounded cache with expiry (Caffeine, or a LinkedHashMap with removeEldestEntry). Explain the tradeoff you just made between hit rate and memory ceiling.

← Back to How does HashMap work internally?