Warm-up

Make the order matter

10 minjunior110 yrs

One concept, guided. Near-impossible to fail.

What this teaches

  • The same tests in a different order give a different result
  • The test that reports the failure is rarely the one that caused it
  • JUnit does not promise an order, and deliberately varies it
  • Per-test fixtures remove the coupling; a reset method only manages it

Starter

Starter.java
import java.util.*;

/**
 * Warm-up: the same four tests, three orders, three answers.
 *
 * A ten-line runner, so nothing is hidden behind a framework. The only thing
 * that changes between the runs below is the order.
 */
public class Starter {

    /** The state two of the tests both touch. */
    static final List<String> REGISTRY = new ArrayList<>();

    static Map.Entry<String, Runnable> test(String name, Runnable body) {
        return Map.entry(name, body);
    }

    static void check(boolean condition, String what) {
        if (!condition) throw new AssertionError(what);
    }

    static List<String> run(List<Map.Entry<String, Runnable>> tests) {
        List<String> failed = new ArrayList<>();
        for (var t : tests) {
            try { t.getValue().run(); } catch (Throwable e) { failed.add(t.getKey()); }
        }
        return failed;
    }

    static List<Map.Entry<String, Runnable>> suite() {
        REGISTRY.clear();
        return List.of(
            test("addsUser",   () -> { REGISTRY.add("ana"); check(REGISTRY.size() == 1, "size"); }),
            test("addsSecond", () -> { REGISTRY.add("bo");  check(REGISTRY.size() == 2, "size"); }),
            test("startsEmpty", () -> check(REGISTRY.isEmpty(), "expected empty")),
            test("independent", () -> check(1 + 1 == 2, "arithmetic"))
        );
    }

    public static void main(String[] args) {
        // TODO 1: predict which tests fail in declaration order, before running.
        System.out.println("declaration order : " + run(suite()));

        // TODO 2: reverse the list and run it again. Predict first — the
        // number of failures changes AND so do the names.

        // TODO 3: rotate the list by two (Collections.rotate) and run it. One
        // of the three orders passes completely. Say why that is the most
        // dangerous of the three results.

        // TODO 4: name the test that is BROKEN, as opposed to the tests that
        // FAILED. They are not the same, and this is the whole skill.

        // TODO 5: fix it so the order cannot matter. Two approaches — a reset
        // before each test, and a fresh fixture per test. Implement the
        // second, then say what the first still leaves in place.

        // TODO 6: `REGISTRY` is static. If it were an instance field of the
        // test class, would the bug exist? Look up how many instances JUnit 5
        // creates per test class by default, and why that default was chosen.
    }
}

Run it locally:

cd exercises/java/test-strategy/flaky-tests/01-warmup
javac Starter.java -d /tmp/out && java -cp /tmp/out Starter

Done when

  • You ran one suite in three orders and got three different results
  • You identified which test pollutes, not which one failed
  • You fixed it with per-test fixtures and confirmed order no longer matters
  • You can say why fixing the reporting test would have been wrong

← Back to What causes a flaky test, and how do you fix one?