Production incident

The config that vanished from its own cache

45 minintermediate210 yrs

A real incident: symptom first, cause hidden, tradeoff at the end.

The incident

Pricing decisions are expensive to compute, so they are memoised. The cache key is a record — which is what records are for, and the reason the key type was changed to one last quarter. Since then: 1. The cache hit rate is zero. Not low — zero. Every lookup misses and recomputes, and nobody noticed, because the answers are still correct. 2. The cache grows without limit. It has an eviction policy; the policy evicts on size, and the size it sees is wrong. The record is declared correctly. The caller is doing something entirely ordinary. Nothing throws. Find the line that causes it, fix the record without touching main(), and then answer the design question: the entry is still in the map and still counted by size(). What exactly is unreachable, and why can the map not notice?

What this teaches

  • A record is only as immutable as its components
  • equals and hashCode are derived from the components' CURRENT contents
  • A key whose hash changes after insertion is unreachable but still stored
  • List.copyOf in a compact constructor severs the caller's reference
  • final on the field was never going to help — it freezes the reference, not the list

Starter

Starter.javaOpen in playground
import java.util.*;

/**
 * Incident reproduction: the config that vanished from its own cache.
 *
 * Pricing decisions are expensive to compute, so they are memoised. The
 * cache key is a record — which is exactly what records are for, and the
 * reason the key type was changed to one last quarter.
 *
 * Since then:
 *
 *   1. The cache hit rate is zero. Not low — zero. Every lookup misses and
 *      recomputes, and nobody noticed because the answers are still correct.
 *   2. The cache grows without limit. It has an eviction policy; the policy
 *      never fires, because it evicts on size and the size it sees is wrong.
 *
 * The record is declared correctly. The caller is doing something entirely
 * ordinary. Nothing throws.
 *
 * TASKS
 *   1. Run it. Confirm that a key which was just stored cannot be found.
 *   2. Find the line in main() that causes it. It is not in the cache and
 *      not in the record.
 *   3. Fix PriceRequest so this cannot happen, without changing main().
 *   4. In a comment: the entry is still in the map — size() counts it. Say
 *      what is now unreachable and why the map cannot notice.
 */
public class Starter {

    /**
     * DEFECT: a record component that is a mutable type. The record looks
     * immutable, and every generated method depends on the component's
     * current contents.
     */
    record PriceRequest(String tenant, List<String> skus) {}

    static final class PriceCache {
        private final Map<PriceRequest, Integer> computed = new HashMap<>();
        int hits = 0;
        int misses = 0;

        int priceFor(PriceRequest request) {
            Integer cached = computed.get(request);
            if (cached != null) {
                hits++;
                return cached;
            }
            misses++;
            int price = request.skus().size() * 1000;   // "expensive"
            computed.put(request, price);
            return price;
        }

        boolean knows(PriceRequest request) {
            return computed.containsKey(request);
        }

        int size() {
            return computed.size();
        }
    }

    public static void main(String[] args) {
        PriceCache cache = new PriceCache();

        // The caller builds a basket and prices it. Entirely ordinary.
        List<String> basket = new ArrayList<>(List.of("SKU-1"));
        cache.priceFor(new PriceRequest("acme", basket));

        // Control: a fresh, equal request finds the cached answer.
        boolean foundBefore = cache.knows(new PriceRequest("acme", List.of("SKU-1")));
        System.out.println("  before: an equal request is found : " + foundBefore);

        // The caller adds an item to the basket. One line, and nothing in
        // this file is touching the cache.
        basket.add("SKU-2");
        System.out.println("  caller added SKU-2 to its basket");

        // Now look for both the old contents and the new contents.
        boolean foundOld = cache.knows(new PriceRequest("acme", List.of("SKU-1")));
        boolean foundNew = cache.knows(new PriceRequest("acme", List.of("SKU-1", "SKU-2")));

        System.out.println("  after : old contents found        : " + foundOld);
        System.out.println("  after : new contents found        : " + foundNew);

        // Every subsequent request misses and stores another entry.
        for (int i = 0; i < 3; i++) {
            cache.priceFor(new PriceRequest("acme", List.of("SKU-1", "SKU-2")));
        }

        System.out.println();
        System.out.printf("  hits=%d misses=%d cacheSize=%d%n",
                cache.hits, cache.misses, cache.size());

        System.out.println();
        boolean entriesStayReachable = foundBefore && (foundOld || foundNew);
        boolean cacheWorks = cache.hits > 0;

        System.out.println("stored entries stay reachable : " + entriesStayReachable);
        System.out.println("cache is ever hit             : " + cacheWorks);
        System.out.println(entriesStayReachable && cacheWorks ? "PASS" : "FAIL");
    }
}

Run it locally:

cd exercises/java/oop/record-vs-class/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out Starter

Hints

  1. Hint 1

    The control lookup succeeds and the two after it fail. What happened in between, and who else holds that list?

  2. Hint 2

    Write down what the record's hashCode is computed from. Now ask when it was computed, and when it is computed again.

  3. Hint 3

    The entry is in the bucket chosen by the OLD hash. get() computes the new one. Where does it look?

  4. Hint 4

    The fix goes on the way in, not on the way out. By the time an accessor runs, the key is already filed in the wrong bucket.

Done when

  • A lookup with the original contents succeeds after the caller mutates its list
  • main() is unchanged
  • No mutable component can reach the record from outside
  • A comment explains why size() still counts the stranded entry

Solution

Show the solution — try it yourself first
Solution.javaOpen in playground
import java.util.*;

/**
 * Solution: the config that vanished from its own cache.
 *
 * The record was declared correctly and the caller did nothing unusual. The
 * defect is that a record is only as immutable as its components, and a
 * record with a List component is freely mutable through the list the caller
 * still holds.
 *
 * Everything a record generates — equals, hashCode, toString — is derived
 * from the components' CURRENT contents. So when the caller added an item to
 * its basket, the stored key's hashCode changed while the entry sat in the
 * bucket chosen by the old one. HashMap looks in the bucket the new hash
 * points at, finds nothing, and reports a miss.
 *
 * The entry is not gone. size() still counts it, iterating the entrySet
 * still yields it, and it still holds memory. It is simply unreachable by
 * key, forever, which is the same failure as any mutable HashMap key —
 * see /java/collections/hashmap-internals.
 *
 * THE FIX
 *   A compact constructor with List.copyOf. That severs the caller's list
 *   from the record, so nothing outside can change what the key hashes to.
 *   It also rejects nulls, which is a second small win.
 *
 *   Note what is NOT the fix: making the field final (it already is),
 *   copying in the accessor (too late — the damage is on the way in), or
 *   telling callers not to reuse lists (a convention, not a mechanism).
 *
 * THE DESIGN QUESTION
 *   A record whose components are all immutable is a genuinely safe map key.
 *   That is the shape to aim for, and it is worth checking every component
 *   type at the point you declare the record rather than at 3am.
 */
public class Solution {

    /**
     * FIX: copy the component in, so the record's contents cannot be changed
     * by anyone who kept a reference to the list they passed.
     *
     * List.copyOf returns the input unchanged when it is already immutable,
     * so callers passing List.of(...) pay nothing for this.
     */
    record PriceRequest(String tenant, List<String> skus) {
        PriceRequest {
            skus = List.copyOf(skus);
        }
    }

    static final class PriceCache {
        private final Map<PriceRequest, Integer> computed = new HashMap<>();
        int hits = 0;
        int misses = 0;

        int priceFor(PriceRequest request) {
            Integer cached = computed.get(request);
            if (cached != null) {
                hits++;
                return cached;
            }
            misses++;
            int price = request.skus().size() * 1000;   // "expensive"
            computed.put(request, price);
            return price;
        }

        boolean knows(PriceRequest request) {
            return computed.containsKey(request);
        }

        int size() {
            return computed.size();
        }
    }

    public static void main(String[] args) {
        PriceCache cache = new PriceCache();

        // The caller builds a basket and prices it. Entirely ordinary.
        List<String> basket = new ArrayList<>(List.of("SKU-1"));
        cache.priceFor(new PriceRequest("acme", basket));

        // Control: a fresh, equal request finds the cached answer.
        boolean foundBefore = cache.knows(new PriceRequest("acme", List.of("SKU-1")));
        System.out.println("  before: an equal request is found : " + foundBefore);

        // The caller adds an item to the basket. One line, and nothing in
        // this file is touching the cache.
        basket.add("SKU-2");
        System.out.println("  caller added SKU-2 to its basket");

        // Now look for both the old contents and the new contents.
        boolean foundOld = cache.knows(new PriceRequest("acme", List.of("SKU-1")));
        boolean foundNew = cache.knows(new PriceRequest("acme", List.of("SKU-1", "SKU-2")));

        System.out.println("  after : old contents found        : " + foundOld);
        System.out.println("  after : new contents found        : " + foundNew);

        // Every subsequent request misses and stores another entry.
        for (int i = 0; i < 3; i++) {
            cache.priceFor(new PriceRequest("acme", List.of("SKU-1", "SKU-2")));
        }

        System.out.println();
        System.out.printf("  hits=%d misses=%d cacheSize=%d%n",
                cache.hits, cache.misses, cache.size());

        System.out.println();
        boolean entriesStayReachable = foundBefore && (foundOld || foundNew);
        boolean cacheWorks = cache.hits > 0;

        System.out.println("stored entries stay reachable : " + entriesStayReachable);
        System.out.println("cache is ever hit             : " + cacheWorks);
        System.out.println(entriesStayReachable && cacheWorks ? "PASS" : "FAIL");
    }
}

Stretch

Add a second component that List.copyOf cannot freeze — a byte[] or a java.util.Date — and make the record safe anyway. Then argue the other side: at what point does the component list tell you this should not be a record at all, and what would you use instead?

← Back to When should you use a record instead of a class?