ExerciseProduction incident
Production incident
The read-only job that ran out of memory
45 minintermediate2–15 yrs
A real incident: symptom first, cause hidden, tradeoff at the end.
The incident
A nightly repricing job. The ticket describes it as read-only. It loads two
hundred thousand products, dies with an OutOfMemoryError about forty minutes
in, and on the nights it does finish, the audit table gains a row for every
product whether or not its price changed.
What the review turned up:
1. Every product stays in the persistence context, with a snapshot beside
it, until the single commit at the end.
2. The default UPDATE sets every column, so changing one price writes
four — which is what the audit triggers are reacting to.
Two defects, and one wrong fix. Three checks fail; the fourth passes right
now and the instinctive response to a memory problem will break it. Fix the
job so all four pass.
What this teaches
- The context holds the entity and a snapshot, so a read-only job is not memory-free
- flush() writes pending changes and releases nothing; clear() is what frees memory
- Flushing without clearing re-compares everything on every flush
- The default UPDATE sets every column, which is what column-level triggers react to
- A job described as read-only in a ticket may not be read-only in the code
Starter
Starter.java
import java.util.*;
/**
* Production: the read-only job that ran out of memory.
*
* A nightly repricing job. It is described as read-only in the ticket, it
* loads two hundred thousand products, and it dies with an OutOfMemoryError
* about forty minutes in. When it does complete, an audit table gains a row
* for every product whether or not the price changed.
*
* Run this. Three checks fail. Fix RepricingJob so all four pass, without
* weakening the checks — check 2 passes now and the obvious wrong fix breaks it.
*
* The persistence context is a model. It counts what a real one costs:
* snapshots retained, field comparisons per flush, and columns written.
*/
public class Starter {
static final class Product {
final long id;
String name;
int priceMinor;
int stock;
String category;
Product(long id, String name, int priceMinor, int stock, String category) {
this.id = id; this.name = name; this.priceMinor = priceMinor;
this.stock = stock; this.category = category;
}
Map<String, Object> state() {
var s = new LinkedHashMap<String, Object>();
s.put("name", name); s.put("priceMinor", priceMinor);
s.put("stock", stock); s.put("category", category);
return s;
}
}
/* ── the persistence context (correct; do not change) ───────────────── */
static final class Session {
final Map<Long, Product> table = new LinkedHashMap<>();
final Map<Long, Product> managed = new LinkedHashMap<>();
final Map<Long, Map<String, Object>> snapshots = new LinkedHashMap<>();
boolean readOnly, dynamicUpdate;
int peakSnapshots, comparisons, columnsWritten, statements;
Session seed(int n) {
for (long i = 1; i <= n; i++)
table.put(i, new Product(i, "p" + i, 1000 + (int) i, 5, i % 2 == 0 ? "A" : "B"));
return this;
}
Product find(long id) {
statements++;
Product row = table.get(id);
var entity = new Product(row.id, row.name, row.priceMinor, row.stock, row.category);
managed.put(id, entity);
if (!readOnly) snapshots.put(id, entity.state());
peakSnapshots = Math.max(peakSnapshots, snapshots.size());
return entity;
}
/** Compares every managed entity against its snapshot. */
void flush() {
for (var e : managed.entrySet()) {
var snap = snapshots.get(e.getKey());
if (snap == null) continue;
var now = e.getValue().state();
var changed = new ArrayList<String>();
for (String f : now.keySet()) {
comparisons++;
if (!Objects.equals(now.get(f), snap.get(f))) changed.add(f);
}
if (changed.isEmpty()) continue;
statements++;
columnsWritten += dynamicUpdate ? changed.size() : now.size();
snapshots.put(e.getKey(), now);
table.put(e.getKey(), e.getValue()); // the UPDATE lands
}
}
void clear() { managed.clear(); snapshots.clear(); }
/** A bulk statement: one statement, no entities, no snapshots. */
void bulkUpdate(String category, int columns, int rows) {
statements++;
columnsWritten += columns * rows;
}
void commit() { flush(); }
}
/* ── the job under review ───────────────────────────────────────────── */
static final class RepricingJob {
final Session session;
RepricingJob(Session session) { this.session = session; }
/** Raises every product in a category by ten percent. */
void reprice(int productCount) {
for (long id = 1; id <= productCount; id++) {
Product p = session.find(id);
if ("A".equals(p.category))
p.priceMinor = (int) (p.priceMinor * 1.1);
}
session.commit();
}
}
/* ── checks ─────────────────────────────────────────────────────────── */
static final int PRODUCTS = 20_000;
static final int IN_CATEGORY_A = PRODUCTS / 2;
public static void main(String[] args) {
List<String> failures = new ArrayList<>();
var session = new Session().seed(PRODUCTS);
new RepricingJob(session).reprice(PRODUCTS);
// 1. The job must not hold the whole table in the persistence context.
if (session.peakSnapshots > 1_000)
failures.add("1. the context held " + session.peakSnapshots + " snapshots at once for "
+ PRODUCTS + " products — nothing is ever released before commit");
// 2. Flushing more often must not mean comparing the same entities
// again. Each entity should be compared once, not once per flush.
if (session.comparisons > PRODUCTS * 4L)
failures.add("2. " + session.comparisons + " field comparisons for " + PRODUCTS
+ " products — entities are being re-compared on every flush, which is what "
+ "happens when you flush without clearing");
// 3. Only the price column should be written.
if (session.columnsWritten > IN_CATEGORY_A)
failures.add("3. " + session.columnsWritten + " columns were written to change "
+ IN_CATEGORY_A + " prices — every column is set on every update");
// 4. And the job must actually do its work.
long repriced = session.table.values().stream()
.filter(p -> "A".equals(p.category) && p.priceMinor != 1000 + p.id).count();
if (repriced != IN_CATEGORY_A)
failures.add("4. repriced " + repriced + " products, expected " + IN_CATEGORY_A);
/* ── report ─────────────────────────────────────────────────────── */
if (failures.isEmpty()) {
System.out.println("PASS");
} else {
failures.forEach(f -> System.out.println(" " + f));
System.out.println("FAIL");
}
}
}Run it locally:
cd exercises/java/jpa-hibernate/dirty-checking/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out StarterHints
Hint 1
Read check 2 before you start. It passes now. Whatever you do to fix check 1, run the checks again and see whether you broke it.
Hint 2
flush() and clear() do different things and only one of them frees memory. Write down which.
Hint 3
Check 3 is about the shape of the statement, not the number of them. One annotation changes it.
Hint 4
There is a fix that makes all four trivially pass by not loading any entities at all. Name it in a comment and say what it gives up.
Done when
- Peak snapshots stay bounded regardless of how many products are processed
- Each entity is compared once, not once per flush
- Only the changed column is written
- Every product in the category is repriced
- A comment says why flush() alone makes the problem worse
Solution
Show the solution — try it yourself first
Solution.java
import java.util.*;
/**
* Solution: the read-only job that ran out of memory.
*
* Two defects, and the second one is only visible because of a check that was
* already passing. Flushing more often is the instinctive answer to a memory
* problem and makes it worse — the context still holds everything, and now
* compares all of it repeatedly.
*/
public class Solution {
static final class Product {
final long id;
String name;
int priceMinor;
int stock;
String category;
Product(long id, String name, int priceMinor, int stock, String category) {
this.id = id; this.name = name; this.priceMinor = priceMinor;
this.stock = stock; this.category = category;
}
Map<String, Object> state() {
var s = new LinkedHashMap<String, Object>();
s.put("name", name); s.put("priceMinor", priceMinor);
s.put("stock", stock); s.put("category", category);
return s;
}
}
/* ── the persistence context (correct; do not change) ───────────────── */
static final class Session {
final Map<Long, Product> table = new LinkedHashMap<>();
final Map<Long, Product> managed = new LinkedHashMap<>();
final Map<Long, Map<String, Object>> snapshots = new LinkedHashMap<>();
boolean readOnly, dynamicUpdate;
int peakSnapshots, comparisons, columnsWritten, statements;
Session seed(int n) {
for (long i = 1; i <= n; i++)
table.put(i, new Product(i, "p" + i, 1000 + (int) i, 5, i % 2 == 0 ? "A" : "B"));
return this;
}
Product find(long id) {
statements++;
Product row = table.get(id);
var entity = new Product(row.id, row.name, row.priceMinor, row.stock, row.category);
managed.put(id, entity);
if (!readOnly) snapshots.put(id, entity.state());
peakSnapshots = Math.max(peakSnapshots, snapshots.size());
return entity;
}
/** Compares every managed entity against its snapshot. */
void flush() {
for (var e : managed.entrySet()) {
var snap = snapshots.get(e.getKey());
if (snap == null) continue;
var now = e.getValue().state();
var changed = new ArrayList<String>();
for (String f : now.keySet()) {
comparisons++;
if (!Objects.equals(now.get(f), snap.get(f))) changed.add(f);
}
if (changed.isEmpty()) continue;
statements++;
columnsWritten += dynamicUpdate ? changed.size() : now.size();
snapshots.put(e.getKey(), now);
table.put(e.getKey(), e.getValue()); // the UPDATE lands
}
}
void clear() { managed.clear(); snapshots.clear(); }
/** A bulk statement: one statement, no entities, no snapshots. */
void bulkUpdate(String category, int columns, int rows) {
statements++;
columnsWritten += columns * rows;
}
void commit() { flush(); }
}
/* ── the job under review ───────────────────────────────────────────── */
static final class RepricingJob {
final Session session;
RepricingJob(Session session) { this.session = session; }
static final int BATCH = 500;
/*
* Defect 1. Every product loaded stayed in the persistence context,
* with a snapshot beside it, until the single commit at the end. Two
* hundred thousand products meant two hundred thousand entities plus
* two hundred thousand snapshots held simultaneously — which is the
* OutOfMemoryError, in a job whose ticket said "read-only".
*
* flush() alone is not the fix and is the tempting one: it writes the
* pending changes and keeps every entity managed, so the next flush
* compares them all again. clear() is what releases them, and check 2
* exists to catch the version of this fix that omits it.
*/
void reprice(int productCount) {
/*
* Defect 2. The default UPDATE sets every column, so changing one
* price wrote four columns — which is why the audit table gained a
* row per product whether or not the price moved. @DynamicUpdate
* builds the statement from the fields that actually differ.
*/
session.dynamicUpdate = true;
for (long id = 1; id <= productCount; id++) {
Product p = session.find(id);
if ("A".equals(p.category))
p.priceMinor = (int) (p.priceMinor * 1.1);
if (id % BATCH == 0) {
session.flush();
session.clear(); // without this, flush makes it worse
}
}
session.commit();
}
}
/* ── checks ─────────────────────────────────────────────────────────── */
static final int PRODUCTS = 20_000;
static final int IN_CATEGORY_A = PRODUCTS / 2;
public static void main(String[] args) {
List<String> failures = new ArrayList<>();
var session = new Session().seed(PRODUCTS);
new RepricingJob(session).reprice(PRODUCTS);
// 1. The job must not hold the whole table in the persistence context.
if (session.peakSnapshots > 1_000)
failures.add("1. the context held " + session.peakSnapshots + " snapshots at once for "
+ PRODUCTS + " products — nothing is ever released before commit");
// 2. Flushing more often must not mean comparing the same entities
// again. Each entity should be compared once, not once per flush.
if (session.comparisons > PRODUCTS * 4L)
failures.add("2. " + session.comparisons + " field comparisons for " + PRODUCTS
+ " products — entities are being re-compared on every flush, which is what "
+ "happens when you flush without clearing");
// 3. Only the price column should be written.
if (session.columnsWritten > IN_CATEGORY_A)
failures.add("3. " + session.columnsWritten + " columns were written to change "
+ IN_CATEGORY_A + " prices — every column is set on every update");
// 4. And the job must actually do its work.
long repriced = session.table.values().stream()
.filter(p -> "A".equals(p.category) && p.priceMinor != 1000 + p.id).count();
if (repriced != IN_CATEGORY_A)
failures.add("4. repriced " + repriced + " products, expected " + IN_CATEGORY_A);
/* ── report ─────────────────────────────────────────────────────── */
if (failures.isEmpty()) {
System.out.println("PASS");
} else {
failures.forEach(f -> System.out.println(" " + f));
System.out.println("FAIL");
}
}
}Stretch
This job loads two hundred thousand entities to change one column. A single
bulk update statement would do it with no entities and no snapshots at all.
Write it, then say precisely what it gives up — what happens to entities
already in the context, to version columns, and to entity callbacks — and
under what conditions those costs make it the wrong choice.
← Back to What is dirty checking, and why did my entity save without a save() call?