Production incident

The report that dies on one tenant

45 minintermediate310 yrs

A real incident: symptom first, cause hidden, tradeoff at the end.

The incident

The nightly cost-allocation job walks a tenant's org chart, rolls each team's spend up to its parent, and writes one CSV row per team. It has run without incident for two years. Last night it produced two findings, and only one of them was filed as a bug: 1. For one tenant the job died with StackOverflowError. No code changed. The tenant's admin did reorganise their teams that afternoon. 2. Ops have been raising -Xmx on this job every few months. It holds the whole report in memory before writing a single row, so its peak usage tracks the largest tenant. Nobody calls it a bug, because it has never actually crashed — yet. One is a stack problem and one is a heap problem. Diagnose both, fix both, and then answer the design question: which one would a bigger flag have fixed, and for how long?

What this teaches

  • A StackOverflowError whose trace repeats one cycle of frames is a logic bug, not a depth problem
  • Cyclic data turns a correct recursive traversal into unbounded recursion
  • Moving frames to an explicit Deque moves the depth limit from the stack to the heap
  • Buffering a whole result set makes peak memory a function of the input
  • -Xss and -Xmx move different walls, and neither fixes a defect

Starter

Starter.javaOpen in playground
import java.util.*;

/**
 * Incident reproduction: the nightly cost-allocation report.
 *
 * The job walks a tenant's org chart, rolls each team's spend up to its
 * parent, and writes one CSV row per team. It has run fine for two years.
 *
 * Last night, for one tenant only:
 *
 *   1. The job died with StackOverflowError. Nothing about the code changed.
 *      The tenant's admin did reorganise their teams that afternoon.
 *
 *   2. Separately, ops have been raising -Xmx on this job every few months.
 *      It holds the entire report in memory before writing a single row, so
 *      its peak usage grows with the largest tenant rather than staying flat.
 *      Nobody has called it a bug because it has never actually crashed.
 *
 * One of these is a stack problem and one is a heap problem, and the fixes
 * have nothing in common.
 *
 * TASKS
 *   1. Run it. Find what the admin's reorganisation did to the data.
 *   2. Fix the traversal so it reports the bad data instead of dying on it.
 *      Catching StackOverflowError is not a fix.
 *   3. Fix the memory shape so peak buffered rows stays flat regardless of
 *      tenant size.
 *   4. In a comment: which of these would a bigger -Xmx have fixed, which
 *      would a bigger -Xss have fixed, and why is the honest answer
 *      "neither, for long"?
 */
public class Starter {

    /** Peak rows held in memory at once. The heap half of this incident. */
    static int peakBuffered = 0;

    record Team(int id, String name, int spendPaise) {}

    record Row(String team, int ownSpend, int rolledUpSpend) {}

    /** Pretends to be the CSV writer. Counts rows; holds nothing. */
    static final class ReportSink {
        int written = 0;

        void write(Row row) {
            written++;
        }
    }

    /** id -> child ids. The admin's reorganisation edited this. */
    static Map<Integer, List<Integer>> childrenOf = new HashMap<>();
    static Map<Integer, Team> teams = new HashMap<>();

    static void loadTenant() {
        int[][] chart = {
                {1, 0}, {2, 1}, {3, 1}, {4, 2}, {5, 2}, {6, 3}, {7, 3}, {8, 6}, {9, 6},
        };
        for (int[] row : chart) {
            int id = row[0], parent = row[1];
            teams.put(id, new Team(id, "team-" + id, id * 1000));
            childrenOf.computeIfAbsent(parent, k -> new ArrayList<>()).add(id);
        }

        // What the admin did yesterday afternoon: they moved team 3 under
        // team 8 to "tidy up the hierarchy". Team 8 is already under team 3.
        childrenOf.computeIfAbsent(8, k -> new ArrayList<>()).add(3);
    }

    /**
     * BUG 1 lives here: depth-first roll-up, written recursively, with no
     * memory of where it has already been.
     */
    static int rollUp(int teamId, List<Row> out) {
        Team team = teams.get(teamId);
        int total = team == null ? 0 : team.spendPaise();

        for (int child : childrenOf.getOrDefault(teamId, List.of())) {
            total += rollUp(child, out);
        }

        if (team != null) {
            out.add(new Row(team.name(), team.spendPaise(), total));
            peakBuffered = Math.max(peakBuffered, out.size());
        }
        return total;
    }

    /**
     * BUG 2 lives here: every row is accumulated before any row is written,
     * so peak memory is proportional to the tenant.
     */
    static void runReport(ReportSink sink) {
        List<Row> allRows = new ArrayList<>();
        rollUp(0, allRows);
        for (Row row : allRows) {
            sink.write(row);
        }
    }

    public static void main(String[] args) {
        loadTenant();

        boolean completed = false;
        ReportSink sink = new ReportSink();

        try {
            runReport(sink);
            completed = true;
        } catch (StackOverflowError e) {
            System.out.println("StackOverflowError — the job died on this tenant");
            System.out.println("  frames were all the same method, which is the clue");
        }

        boolean flatMemory = peakBuffered <= 2;

        System.out.println();
        System.out.println("rows written                 : " + sink.written);
        System.out.println("peak rows buffered in memory : " + peakBuffered);
        System.out.println();
        System.out.println("report completed             : " + completed);
        System.out.println("memory flat regardless of size: " + flatMemory);
        System.out.println(completed && flatMemory ? "PASS" : "FAIL");
    }
}

Run it locally:

cd exercises/java/jvm/heap-vs-stack/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out Starter

Hints

  1. Hint 1

    Read loadTenant() to the last line. What did the admin's edit do to the shape of childrenOf?

  2. Hint 2

    Catching StackOverflowError is not a fix — by the time it is thrown, how much of the report is trustworthy?

  3. Hint 3

    To walk a graph that might not be a tree you need to remember where you have already been. Where, specifically — anywhere you have ever been, or anywhere on the current path? They give different answers here.

  4. Hint 4

    For the heap half: at what moment does a row become final? Could it be written then, instead of collected?

Done when

  • The job completes on the reorganised tenant and reports the bad edge
  • StackOverflowError is not caught anywhere
  • Peak buffered rows is 1 or 2, not the tenant size
  • All nine teams are written, and the tenant total is correct
  • A comment answers what -Xss and -Xmx would each have bought

Solution

Show the solution — try it yourself first
Solution.javaOpen in playground
import java.util.*;

/**
 * Solution: the nightly cost-allocation report.
 *
 * Two defects, two different regions of memory, two unrelated fixes.
 *
 *   Bug 1 (stack) — the roll-up recursed over an org chart that the admin had
 *       accidentally made cyclic, so the recursion never terminated. A deeper
 *       stack only postpones that. The fix is to carry the current path and
 *       refuse to walk an edge that re-enters it: the bad data gets reported
 *       instead of crashing the job.
 *
 *   Bug 2 (heap) — every row was accumulated before any row was written, so
 *       peak memory scaled with the tenant. The fix is to write each row as
 *       soon as it is final, which makes peak memory a constant.
 *
 * Worth being explicit about why the traversal is now iterative. Cycle
 * detection alone would fix the crash. Moving the frames onto an explicit
 * ArrayDeque also moves the depth limit from the thread's stack — a fixed
 * reservation you cannot grow at runtime — onto the heap, which is where a
 * limit proportional to the data belongs.
 */
public class Solution {

    /** Peak rows held in memory at once. The heap half of this incident. */
    static int peakBuffered = 0;

    record Team(int id, String name, int spendPaise) {}

    record Row(String team, int ownSpend, int rolledUpSpend) {}

    static final class ReportSink {
        int written = 0;

        void write(Row row) {
            written++;
            // Exactly one Row is live at a time now, whatever the tenant size.
            peakBuffered = Math.max(peakBuffered, 1);
        }
    }

    static Map<Integer, List<Integer>> childrenOf = new HashMap<>();
    static Map<Integer, Team> teams = new HashMap<>();

    static void loadTenant() {
        int[][] chart = {
                {1, 0}, {2, 1}, {3, 1}, {4, 2}, {5, 2}, {6, 3}, {7, 3}, {8, 6}, {9, 6},
        };
        for (int[] row : chart) {
            int id = row[0], parent = row[1];
            teams.put(id, new Team(id, "team-" + id, id * 1000));
            childrenOf.computeIfAbsent(parent, k -> new ArrayList<>()).add(id);
        }
        // The admin's edit, left in place on purpose: the job must survive it.
        childrenOf.computeIfAbsent(8, k -> new ArrayList<>()).add(3);
    }

    /** One suspended call, made explicit. This is what a frame held. */
    private static final class Frame {
        final int teamId;
        final Iterator<Integer> children;
        int total;

        Frame(int teamId) {
            this.teamId = teamId;
            this.children = childrenOf.getOrDefault(teamId, List.<Integer>of()).iterator();
            Team team = teams.get(teamId);
            this.total = team == null ? 0 : team.spendPaise();
        }
    }

    /**
     * FIX 1 + FIX 2: post-order roll-up on an explicit stack, with the current
     * path tracked so a cycle is reported rather than followed, and each row
     * written the moment its subtotal is final.
     */
    static int rollUp(int rootId, ReportSink sink, List<String> problems) {
        Deque<Frame> stack = new ArrayDeque<>();
        Set<Integer> onPath = new HashSet<>();

        stack.push(new Frame(rootId));
        onPath.add(rootId);
        int rootTotal = 0;

        while (!stack.isEmpty()) {
            Frame frame = stack.peek();

            if (frame.children.hasNext()) {
                int child = frame.children.next();
                if (onPath.contains(child)) {
                    problems.add("cycle: team-" + frame.teamId + " -> team-" + child
                            + " re-enters the current path; edge skipped");
                    continue;
                }
                stack.push(new Frame(child));
                onPath.add(child);
                continue;
            }

            stack.pop();
            onPath.remove(frame.teamId);

            Team team = teams.get(frame.teamId);
            if (team != null) {
                sink.write(new Row(team.name(), team.spendPaise(), frame.total));
            }

            Frame parent = stack.peek();
            if (parent != null) {
                parent.total += frame.total;
            } else {
                rootTotal = frame.total;
            }
        }
        return rootTotal;
    }

    public static void main(String[] args) {
        loadTenant();

        boolean completed = false;
        ReportSink sink = new ReportSink();
        List<String> problems = new ArrayList<>();

        int total = rollUp(0, sink, problems);
        completed = true;

        for (String problem : problems) {
            System.out.println("data problem: " + problem);
        }

        boolean flatMemory = peakBuffered <= 2;

        System.out.println();
        System.out.println("rows written                 : " + sink.written);
        System.out.println("tenant total (paise)         : " + total);
        System.out.println("peak rows buffered in memory : " + peakBuffered);
        System.out.println();
        System.out.println("report completed             : " + completed);
        System.out.println("memory flat regardless of size: " + flatMemory);
        System.out.println(completed && flatMemory ? "PASS" : "FAIL");
    }
}

Stretch

The fix reports the cycle and skips the edge. That is a choice, not the only one: it could also fail the tenant loudly, or roll up the strongly connected component as a single unit. Pick the behaviour you would actually ship for a billing report, and write down what the finance team would say about each of the other two.

← Back to What lives on the heap and what lives on the stack?