Production incident

The read-only job that ran out of memory

45 minintermediate215 yrs

A real incident: symptom first, cause hidden, tradeoff at the end.

The incident

A nightly repricing job. The ticket describes it as read-only. It loads two hundred thousand products, dies with an OutOfMemoryError about forty minutes in, and on the nights it does finish, the audit table gains a row for every product whether or not its price changed. What the review turned up: 1. Every product stays in the persistence context, with a snapshot beside it, until the single commit at the end. 2. The default UPDATE sets every column, so changing one price writes four — which is what the audit triggers are reacting to. Two defects, and one wrong fix. Three checks fail; the fourth passes right now and the instinctive response to a memory problem will break it. Fix the job so all four pass.

What this teaches

  • The context holds the entity and a snapshot, so a read-only job is not memory-free
  • flush() writes pending changes and releases nothing; clear() is what frees memory
  • Flushing without clearing re-compares everything on every flush
  • The default UPDATE sets every column, which is what column-level triggers react to
  • A job described as read-only in a ticket may not be read-only in the code

Starter

Starter.java
import java.util.*;

/**
 * Production: the read-only job that ran out of memory.
 *
 * A nightly repricing job. It is described as read-only in the ticket, it
 * loads two hundred thousand products, and it dies with an OutOfMemoryError
 * about forty minutes in. When it does complete, an audit table gains a row
 * for every product whether or not the price changed.
 *
 * Run this. Three checks fail. Fix RepricingJob so all four pass, without
 * weakening the checks — check 2 passes now and the obvious wrong fix breaks it.
 *
 * The persistence context is a model. It counts what a real one costs:
 * snapshots retained, field comparisons per flush, and columns written.
 */
public class Starter {

    static final class Product {
        final long id;
        String name;
        int priceMinor;
        int stock;
        String category;
        Product(long id, String name, int priceMinor, int stock, String category) {
            this.id = id; this.name = name; this.priceMinor = priceMinor;
            this.stock = stock; this.category = category;
        }
        Map<String, Object> state() {
            var s = new LinkedHashMap<String, Object>();
            s.put("name", name); s.put("priceMinor", priceMinor);
            s.put("stock", stock); s.put("category", category);
            return s;
        }
    }

    /* ── the persistence context (correct; do not change) ───────────────── */

    static final class Session {
        final Map<Long, Product> table = new LinkedHashMap<>();
        final Map<Long, Product> managed = new LinkedHashMap<>();
        final Map<Long, Map<String, Object>> snapshots = new LinkedHashMap<>();

        boolean readOnly, dynamicUpdate;
        int peakSnapshots, comparisons, columnsWritten, statements;

        Session seed(int n) {
            for (long i = 1; i <= n; i++)
                table.put(i, new Product(i, "p" + i, 1000 + (int) i, 5, i % 2 == 0 ? "A" : "B"));
            return this;
        }

        Product find(long id) {
            statements++;
            Product row = table.get(id);
            var entity = new Product(row.id, row.name, row.priceMinor, row.stock, row.category);
            managed.put(id, entity);
            if (!readOnly) snapshots.put(id, entity.state());
            peakSnapshots = Math.max(peakSnapshots, snapshots.size());
            return entity;
        }

        /** Compares every managed entity against its snapshot. */
        void flush() {
            for (var e : managed.entrySet()) {
                var snap = snapshots.get(e.getKey());
                if (snap == null) continue;
                var now = e.getValue().state();
                var changed = new ArrayList<String>();
                for (String f : now.keySet()) {
                    comparisons++;
                    if (!Objects.equals(now.get(f), snap.get(f))) changed.add(f);
                }
                if (changed.isEmpty()) continue;
                statements++;
                columnsWritten += dynamicUpdate ? changed.size() : now.size();
                snapshots.put(e.getKey(), now);
                table.put(e.getKey(), e.getValue());          // the UPDATE lands

            }
        }

        void clear() { managed.clear(); snapshots.clear(); }

        /** A bulk statement: one statement, no entities, no snapshots. */
        void bulkUpdate(String category, int columns, int rows) {
            statements++;
            columnsWritten += columns * rows;
        }

        void commit() { flush(); }
    }

    /* ── the job under review ───────────────────────────────────────────── */

    static final class RepricingJob {
        final Session session;
        RepricingJob(Session session) { this.session = session; }

        /** Raises every product in a category by ten percent. */
        void reprice(int productCount) {
            for (long id = 1; id <= productCount; id++) {
                Product p = session.find(id);
                if ("A".equals(p.category))
                    p.priceMinor = (int) (p.priceMinor * 1.1);
            }
            session.commit();
        }
    }

    /* ── checks ─────────────────────────────────────────────────────────── */

    static final int PRODUCTS = 20_000;
    static final int IN_CATEGORY_A = PRODUCTS / 2;

    public static void main(String[] args) {
        List<String> failures = new ArrayList<>();

        var session = new Session().seed(PRODUCTS);
        new RepricingJob(session).reprice(PRODUCTS);

        // 1. The job must not hold the whole table in the persistence context.
        if (session.peakSnapshots > 1_000)
            failures.add("1. the context held " + session.peakSnapshots + " snapshots at once for "
                       + PRODUCTS + " products — nothing is ever released before commit");

        // 2. Flushing more often must not mean comparing the same entities
        //    again. Each entity should be compared once, not once per flush.
        if (session.comparisons > PRODUCTS * 4L)
            failures.add("2. " + session.comparisons + " field comparisons for " + PRODUCTS
                       + " products — entities are being re-compared on every flush, which is what "
                       + "happens when you flush without clearing");

        // 3. Only the price column should be written.
        if (session.columnsWritten > IN_CATEGORY_A)
            failures.add("3. " + session.columnsWritten + " columns were written to change "
                       + IN_CATEGORY_A + " prices — every column is set on every update");

        // 4. And the job must actually do its work.
        long repriced = session.table.values().stream()
            .filter(p -> "A".equals(p.category) && p.priceMinor != 1000 + p.id).count();
        if (repriced != IN_CATEGORY_A)
            failures.add("4. repriced " + repriced + " products, expected " + IN_CATEGORY_A);

        /* ── report ─────────────────────────────────────────────────────── */
        if (failures.isEmpty()) {
            System.out.println("PASS");
        } else {
            failures.forEach(f -> System.out.println("  " + f));
            System.out.println("FAIL");
        }
    }
}

Run it locally:

cd exercises/java/jpa-hibernate/dirty-checking/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out Starter

Hints

  1. Hint 1

    Read check 2 before you start. It passes now. Whatever you do to fix check 1, run the checks again and see whether you broke it.

  2. Hint 2

    flush() and clear() do different things and only one of them frees memory. Write down which.

  3. Hint 3

    Check 3 is about the shape of the statement, not the number of them. One annotation changes it.

  4. Hint 4

    There is a fix that makes all four trivially pass by not loading any entities at all. Name it in a comment and say what it gives up.

Done when

  • Peak snapshots stay bounded regardless of how many products are processed
  • Each entity is compared once, not once per flush
  • Only the changed column is written
  • Every product in the category is repriced
  • A comment says why flush() alone makes the problem worse

Solution

Show the solution — try it yourself first
Solution.java
import java.util.*;

/**
 * Solution: the read-only job that ran out of memory.
 *
 * Two defects, and the second one is only visible because of a check that was
 * already passing. Flushing more often is the instinctive answer to a memory
 * problem and makes it worse — the context still holds everything, and now
 * compares all of it repeatedly.
 */
public class Solution {

    static final class Product {
        final long id;
        String name;
        int priceMinor;
        int stock;
        String category;
        Product(long id, String name, int priceMinor, int stock, String category) {
            this.id = id; this.name = name; this.priceMinor = priceMinor;
            this.stock = stock; this.category = category;
        }
        Map<String, Object> state() {
            var s = new LinkedHashMap<String, Object>();
            s.put("name", name); s.put("priceMinor", priceMinor);
            s.put("stock", stock); s.put("category", category);
            return s;
        }
    }

    /* ── the persistence context (correct; do not change) ───────────────── */

    static final class Session {
        final Map<Long, Product> table = new LinkedHashMap<>();
        final Map<Long, Product> managed = new LinkedHashMap<>();
        final Map<Long, Map<String, Object>> snapshots = new LinkedHashMap<>();

        boolean readOnly, dynamicUpdate;
        int peakSnapshots, comparisons, columnsWritten, statements;

        Session seed(int n) {
            for (long i = 1; i <= n; i++)
                table.put(i, new Product(i, "p" + i, 1000 + (int) i, 5, i % 2 == 0 ? "A" : "B"));
            return this;
        }

        Product find(long id) {
            statements++;
            Product row = table.get(id);
            var entity = new Product(row.id, row.name, row.priceMinor, row.stock, row.category);
            managed.put(id, entity);
            if (!readOnly) snapshots.put(id, entity.state());
            peakSnapshots = Math.max(peakSnapshots, snapshots.size());
            return entity;
        }

        /** Compares every managed entity against its snapshot. */
        void flush() {
            for (var e : managed.entrySet()) {
                var snap = snapshots.get(e.getKey());
                if (snap == null) continue;
                var now = e.getValue().state();
                var changed = new ArrayList<String>();
                for (String f : now.keySet()) {
                    comparisons++;
                    if (!Objects.equals(now.get(f), snap.get(f))) changed.add(f);
                }
                if (changed.isEmpty()) continue;
                statements++;
                columnsWritten += dynamicUpdate ? changed.size() : now.size();
                snapshots.put(e.getKey(), now);
                table.put(e.getKey(), e.getValue());          // the UPDATE lands

            }
        }

        void clear() { managed.clear(); snapshots.clear(); }

        /** A bulk statement: one statement, no entities, no snapshots. */
        void bulkUpdate(String category, int columns, int rows) {
            statements++;
            columnsWritten += columns * rows;
        }

        void commit() { flush(); }
    }

    /* ── the job under review ───────────────────────────────────────────── */

    static final class RepricingJob {
        final Session session;
        RepricingJob(Session session) { this.session = session; }

        static final int BATCH = 500;

        /*
         * Defect 1. Every product loaded stayed in the persistence context,
         * with a snapshot beside it, until the single commit at the end. Two
         * hundred thousand products meant two hundred thousand entities plus
         * two hundred thousand snapshots held simultaneously — which is the
         * OutOfMemoryError, in a job whose ticket said "read-only".
         *
         * flush() alone is not the fix and is the tempting one: it writes the
         * pending changes and keeps every entity managed, so the next flush
         * compares them all again. clear() is what releases them, and check 2
         * exists to catch the version of this fix that omits it.
         */
        void reprice(int productCount) {
            /*
             * Defect 2. The default UPDATE sets every column, so changing one
             * price wrote four columns — which is why the audit table gained a
             * row per product whether or not the price moved. @DynamicUpdate
             * builds the statement from the fields that actually differ.
             */
            session.dynamicUpdate = true;

            for (long id = 1; id <= productCount; id++) {
                Product p = session.find(id);
                if ("A".equals(p.category))
                    p.priceMinor = (int) (p.priceMinor * 1.1);

                if (id % BATCH == 0) {
                    session.flush();
                    session.clear();          // without this, flush makes it worse
                }
            }
            session.commit();
        }
    }

    /* ── checks ─────────────────────────────────────────────────────────── */

    static final int PRODUCTS = 20_000;
    static final int IN_CATEGORY_A = PRODUCTS / 2;

    public static void main(String[] args) {
        List<String> failures = new ArrayList<>();

        var session = new Session().seed(PRODUCTS);
        new RepricingJob(session).reprice(PRODUCTS);

        // 1. The job must not hold the whole table in the persistence context.
        if (session.peakSnapshots > 1_000)
            failures.add("1. the context held " + session.peakSnapshots + " snapshots at once for "
                       + PRODUCTS + " products — nothing is ever released before commit");

        // 2. Flushing more often must not mean comparing the same entities
        //    again. Each entity should be compared once, not once per flush.
        if (session.comparisons > PRODUCTS * 4L)
            failures.add("2. " + session.comparisons + " field comparisons for " + PRODUCTS
                       + " products — entities are being re-compared on every flush, which is what "
                       + "happens when you flush without clearing");

        // 3. Only the price column should be written.
        if (session.columnsWritten > IN_CATEGORY_A)
            failures.add("3. " + session.columnsWritten + " columns were written to change "
                       + IN_CATEGORY_A + " prices — every column is set on every update");

        // 4. And the job must actually do its work.
        long repriced = session.table.values().stream()
            .filter(p -> "A".equals(p.category) && p.priceMinor != 1000 + p.id).count();
        if (repriced != IN_CATEGORY_A)
            failures.add("4. repriced " + repriced + " products, expected " + IN_CATEGORY_A);

        /* ── report ─────────────────────────────────────────────────────── */
        if (failures.isEmpty()) {
            System.out.println("PASS");
        } else {
            failures.forEach(f -> System.out.println("  " + f));
            System.out.println("FAIL");
        }
    }
}

Stretch

This job loads two hundred thousand entities to change one column. A single bulk update statement would do it with no entities and no snapshots at all. Write it, then say precisely what it gives up — what happens to entities already in the context, to version columns, and to entity callbacks — and under what conditions those costs make it the wrong choice.

← Back to What is dirty checking, and why did my entity save without a save() call?