Production incident

The suite nobody trusts

45 minintermediate212 yrs

A real incident: symptom first, cause hidden, tradeoff at the end.

The incident

Eight tests. The build is red about a third of the time, always naming a different test, and the team's response has become to re-run it. Two people now re-run without reading the failure at all. What the team found: 1. The failing test is never the same one twice. 2. It fails more often on Fridays at the end of a month. 3. Running a single test on its own always passes. 4. Someone proposed enabling retries and the reviewer asked for a day to look at it first. Findings 1 and 3 together identify one of the causes on their own. Finding 2 identifies a second. There is a third that neither points at — and one test in the suite is not flaky at all. Fix the flakiness. Do not fix the test that is genuinely failing; leave it failing, and be able to say why that is the right outcome.

What this teaches

  • The test that reports a failure is rarely the one that caused it
  • Static state outlives the test instance, so order decides the result
  • An assertion that is only true most of the month is wrong, not flaky
  • Waiting for work beats waiting for a duration
  • Flakiness hides real defects, which is its actual cost

Starter

Starter.java
import java.time.*;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;

/**
 * Production: the suite nobody trusts.
 *
 * Eight tests. The build is red about a third of the time, always with a
 * different test named, and the team's response has been to re-run it. Two
 * people now re-run without reading the failure at all.
 *
 * There are three distinct causes in here, and one test that is not flaky and
 * is failing for a real reason. Run it — four checks, and it fails.
 */
public class Starter {

    /* ─────────── the code under test ─────────── */

    static class Registry {
        /** DEFECT 1: static, so it outlives the test instance. */
        static final List<String> ENTRIES = new ArrayList<>();

        void add(String name) { ENTRIES.add(name); }
        int size() { return ENTRIES.size(); }
        boolean isEmpty() { return ENTRIES.isEmpty(); }
    }

    static class Subscription {
        /** DEFECT 2: reads the clock directly, so the result depends on today. */
        LocalDate renewalDate(LocalDate start) {
            return start.plusMonths(1);
        }

        boolean renewsOnTheSameDayOfMonth(LocalDate start) {
            return renewalDate(start).getDayOfMonth() == start.getDayOfMonth();
        }
    }

    static class Importer {
        private final ExecutorService pool = Executors.newSingleThreadExecutor();
        final AtomicInteger saved = new AtomicInteger();

        /** DEFECT 3: the test waits a fixed time instead of for the work. */
        void importAsync(int rows) {
            pool.submit(() -> {
                try { Thread.sleep(30); } catch (InterruptedException e) { return; }
                saved.addAndGet(rows);
            });
        }

        void shutdown() { pool.shutdown(); }
    }

    /* ─────────── the suite ─────────── */

    static List<Map.Entry<String, Runnable>> suite(LocalDate today, Importer importer) {
        Registry registry = new Registry();
        Subscription subs = new Subscription();

        return List.of(
            entry("addsFirstEntry", () -> {
                registry.add("ana");
                check(registry.size() == 1, "expected 1, was " + registry.size());
            }),
            entry("addsSecondEntry", () -> {
                registry.add("bo");
                check(registry.size() == 2, "expected 2, was " + registry.size());
            }),
            entry("startsEmptyForANewSession", () ->
                check(registry.isEmpty(), "registry was not empty")),
            entry("renewsSameDayOfMonth", () ->
                check(subs.renewsOnTheSameDayOfMonth(today), "renewal day moved")),
            entry("importSavesRows", () -> {
                importer.importAsync(100);
                try { Thread.sleep(5); } catch (InterruptedException e) { }
                check(importer.saved.get() == 100, "saved " + importer.saved.get());
            }),
            entry("arithmeticIsStable", () -> check(2 + 2 == 4, "arithmetic")),
            entry("stringsCompareByValue", () -> {
                String built = new StringBuilder("ab").toString();
                check(built.equals("ab"), "equals");
            }),
            // Not flaky. Genuinely broken, and hidden by the noise.
            entry("discountIsApplied", () -> {
                int price = 100, discounted = price - (price * 10 / 100);
                check(discounted == 80, "expected 80, was " + discounted);
            })
        );
    }

    public static void main(String[] args) {
        boolean ok = true;
        Importer importer = new Importer();

        LocalDate midMonth = LocalDate.parse("2026-03-15");
        LocalDate monthEnd = LocalDate.parse("2026-08-31");

        List<String> declared = run(suite(midMonth, importer));
        List<String> reversed = runReversed(suite(midMonth, importer));
        List<String> atMonthEnd = run(suite(monthEnd, importer));

        System.out.println("── the same suite, three ways ──");
        System.out.println("  declaration order, mid-month : " + describe(declared));
        System.out.println("  reversed order,    mid-month : " + describe(reversed));
        System.out.println("  declaration order, month end : " + describe(atMonthEnd));
        System.out.println();

        ok &= check2("the result does not depend on test order",
            new HashSet<>(declared).equals(new HashSet<>(reversed)));
        ok &= check2("the result does not depend on the date",
            new HashSet<>(declared).equals(new HashSet<>(atMonthEnd)));
        ok &= check2("no test depends on a sleep to pass",
            !declared.contains("importSavesRows") && !reversed.contains("importSavesRows"));
        ok &= check2("exactly one test fails, and it is the genuinely broken one",
            declared.equals(List.of("discountIsApplied")));

        importer.shutdown();
        System.out.println();
        System.out.println(ok ? "PASS" : "FAIL");
    }

    /* ─────────── the tiny runner ─────────── */

    static Map.Entry<String, Runnable> entry(String name, Runnable body) {
        return Map.entry(name, body);
    }

    static void check(boolean condition, String what) {
        if (!condition) throw new AssertionError(what);
    }

    static List<String> run(List<Map.Entry<String, Runnable>> tests) {
        List<String> failed = new ArrayList<>();
        for (var t : tests) {
            try { t.getValue().run(); } catch (Throwable e) { failed.add(t.getKey()); }
        }
        return failed;
    }

    static List<String> runReversed(List<Map.Entry<String, Runnable>> tests) {
        var copy = new ArrayList<>(tests);
        Collections.reverse(copy);
        return run(copy);
    }

    static String describe(List<String> failed) {
        return failed.isEmpty() ? "all pass" : failed.size() + " failed: " + failed;
    }

    static boolean check2(String what, boolean passed) {
        System.out.println((passed ? "  ok    " : "  FAIL  ") + what);
        return passed;
    }
}

Run it locally:

cd exercises/java/test-strategy/flaky-tests/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out Starter

Hints

  1. Hint 1

    Run the suite in reverse. If the set of failures changes, the cause is order or shared state.

  2. Hint 2

    One assertion is false on the 31st of a 31-day month. Work out what it should have asserted instead.

  3. Hint 3

    One test sleeps for less time than the work it is waiting for.

  4. Hint 4

    After the three fixes, exactly one test still fails. Read it carefully before assuming it is a fourth flake.

Done when

  • The set of failures is identical in declaration order and reversed
  • The set of failures is identical mid-month and at month end
  • No test depends on a sleep to pass
  • Exactly one test fails, and it is the genuine defect
  • A comment says why the genuine failure was left failing

Solution

Show the solution — try it yourself first
Solution.java
import java.time.*;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;

/**
 * Solution: the suite nobody trusts.
 *
 * Three causes, three different fixes, and one finding.
 *
 *   1. ORDER AND SHARED STATE. The registry was static, so it outlived the
 *      test instance and every test after the first saw whatever the previous
 *      one left. That is why the failing test kept changing name: the test
 *      that reports the failure is decided by the order, and it is never the
 *      test that caused it.
 *
 *      The fix is per-test fixtures — each test builds what it needs. Sharing
 *      one instance across the suite and clearing it in a setup method also
 *      works, and is weaker: it keeps the coupling and relies on nobody
 *      forgetting the reset.
 *
 *   2. TIME. The assertion was "renewal lands on the same day of the month",
 *      which is simply not true on the 31st of a 31-day month. It passed for
 *      twenty-eight days a month, which is why it looked flaky rather than
 *      wrong.
 *
 *      The fix is not a Clock here — it is a correct assertion. Two tests with
 *      fixed dates, one for the ordinary case and one for the clamping case,
 *      and the behaviour is now specified rather than sampled.
 *
 *   3. SLEEP. The import test waited 5ms for work that takes 30ms, so it
 *      passed only when the machine was slow enough to reorder things
 *      favourably. Raising the sleep would have made the suite slower and
 *      still flaky.
 *
 *      The fix is to wait for the WORK rather than for a DURATION: a callback
 *      and a latch with a deadline. It now finishes as soon as the import
 *      does, and a genuine hang fails at the deadline instead of passing.
 *
 * The finding: with the noise gone, exactly one test fails — and it is a real
 * bug in the discount calculation that had been invisible in a suite that was
 * red a third of the time anyway. That is the actual cost of flakiness.
 */
public class Solution {

    static class Registry {
        /** FIX 1: per instance, not static. */
        private final List<String> entries = new ArrayList<>();

        void add(String name) { entries.add(name); }
        int size() { return entries.size(); }
        boolean isEmpty() { return entries.isEmpty(); }
    }

    static class Subscription {
        LocalDate renewalDate(LocalDate start) { return start.plusMonths(1); }
    }

    static class Importer {
        private final ExecutorService pool = Executors.newSingleThreadExecutor();
        final AtomicInteger saved = new AtomicInteger();

        /** FIX 3: tell the caller when it is done. */
        void importAsync(int rows, Runnable onComplete) {
            pool.submit(() -> {
                try { Thread.sleep(30); } catch (InterruptedException e) { return; }
                saved.addAndGet(rows);
                onComplete.run();
            });
        }

        void shutdown() { pool.shutdown(); }
    }

    /** `today` is deliberately unused now — the suite no longer reads a clock. */
    static List<Map.Entry<String, Runnable>> suite(LocalDate today, Importer shared) {
        Subscription subs = new Subscription();

        return List.of(
            entry("addsFirstEntry", () -> {
                Registry registry = new Registry();          // its own fixture
                registry.add("ana");
                check(registry.size() == 1, "expected 1, was " + registry.size());
            }),
            entry("addsSecondEntry", () -> {
                Registry registry = new Registry();
                registry.add("ana");
                registry.add("bo");
                check(registry.size() == 2, "expected 2, was " + registry.size());
            }),
            entry("startsEmptyForANewSession", () ->
                check(new Registry().isEmpty(), "registry was not empty")),

            // FIX 2: fixed dates, and the clamping case stated on purpose.
            entry("renewalFromMidMonthKeepsTheDay", () ->
                check(subs.renewalDate(LocalDate.parse("2026-03-15"))
                          .equals(LocalDate.parse("2026-04-15")), "mid-month renewal")),
            entry("renewalFromMonthEndClampsToShorterMonth", () ->
                check(subs.renewalDate(LocalDate.parse("2026-08-31"))
                          .equals(LocalDate.parse("2026-09-30")), "month-end renewal")),

            entry("importSavesRows", () -> {
                Importer importer = new Importer();
                CountDownLatch done = new CountDownLatch(1);
                importer.importAsync(100, done::countDown);
                try {
                    check(done.await(5, TimeUnit.SECONDS), "import did not finish in 5s");
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                    throw new AssertionError("interrupted", e);
                } finally {
                    importer.shutdown();
                }
                check(importer.saved.get() == 100, "saved " + importer.saved.get());
            }),

            entry("arithmeticIsStable", () -> check(2 + 2 == 4, "arithmetic")),

            // Not flaky. Genuinely broken — and now it is the only failure.
            entry("discountIsApplied", () -> {
                int price = 100, discounted = price - (price * 10 / 100);
                check(discounted == 80, "expected 80, was " + discounted);
            })
        );
    }

    public static void main(String[] args) {
        boolean ok = true;
        Importer importer = new Importer();

        LocalDate midMonth = LocalDate.parse("2026-03-15");
        LocalDate monthEnd = LocalDate.parse("2026-08-31");

        List<String> declared = run(suite(midMonth, importer));
        List<String> reversed = runReversed(suite(midMonth, importer));
        List<String> atMonthEnd = run(suite(monthEnd, importer));

        System.out.println("── the same suite, three ways ──");
        System.out.println("  declaration order, mid-month : " + describe(declared));
        System.out.println("  reversed order,    mid-month : " + describe(reversed));
        System.out.println("  declaration order, month end : " + describe(atMonthEnd));
        System.out.println();

        ok &= check2("the result does not depend on test order",
            new HashSet<>(declared).equals(new HashSet<>(reversed)));
        ok &= check2("the result does not depend on the date",
            new HashSet<>(declared).equals(new HashSet<>(atMonthEnd)));
        ok &= check2("no test depends on a sleep to pass",
            !declared.contains("importSavesRows") && !reversed.contains("importSavesRows"));
        ok &= check2("exactly one test fails, and it is the genuinely broken one",
            declared.equals(List.of("discountIsApplied")));

        importer.shutdown();
        System.out.println();
        System.out.println(ok ? "PASS" : "FAIL");
    }

    static Map.Entry<String, Runnable> entry(String name, Runnable body) {
        return Map.entry(name, body);
    }

    static void check(boolean condition, String what) {
        if (!condition) throw new AssertionError(what);
    }

    static List<String> run(List<Map.Entry<String, Runnable>> tests) {
        List<String> failed = new ArrayList<>();
        for (var t : tests) {
            try { t.getValue().run(); } catch (Throwable e) { failed.add(t.getKey()); }
        }
        return failed;
    }

    static List<String> runReversed(List<Map.Entry<String, Runnable>> tests) {
        var copy = new ArrayList<>(tests);
        Collections.reverse(copy);
        return run(copy);
    }

    static String describe(List<String> failed) {
        return failed.isEmpty() ? "all pass" : failed.size() + " failed: " + failed;
    }

    static boolean check2(String what, boolean passed) {
        System.out.println((passed ? "  ok    " : "  FAIL  ") + what);
        return passed;
    }
}

Stretch

The suite is now trustworthy and one test is red. Write the bug report: what is wrong, what the correct discount is, and how far back it has been wrong. Then argue whether the fix belongs in the same change as the flakiness work or a separate one, and what that says about keeping a suite green as a policy.

← Back to What causes a flaky test, and how do you fix one?