Challenge

Five flaky tests, one cause each

25 minintermediate212 yrs

Edge cases. You have to reason, and two valid fixes differ.

What this teaches

  • Each cause has a different fix, and the wrong fix hides the symptom
  • Asserting an order the code never promised is a test bug, not a code bug
  • Unseeded randomness makes a failure unreproducible, which is the real cost
  • A sleep is a guess standing in for a guarantee
  • One of the five is reporting a genuine defect

Starter

Starter.java
import java.time.*;
import java.util.*;
import java.util.concurrent.*;

/**
 * Challenge: five failing tests, five different reasons.
 *
 * Four are flaky and one is not. For each, name the cause before you change
 * anything — the wrong fix makes every one of these go green while leaving
 * the problem exactly where it was.
 */
public class Starter {

    /* ─────────── 1 ─────────── */
    static Set<String> tags = new HashSet<>(List.of("beta", "alpha", "gamma"));

    static List<String> tagsInOrder() {
        return new ArrayList<>(tags);          // a HashSet has no order
    }

    /* ─────────── 2 ─────────── */
    static String pickWinner(List<String> entrants) {
        return entrants.get(new Random().nextInt(entrants.size()));
    }

    /* ─────────── 3 ─────────── */
    static LocalDate trialEnds() {
        return LocalDate.now().plusDays(14);
    }

    /* ─────────── 4 ─────────── */
    static int slowSum(List<Integer> values) throws Exception {
        var pool = Executors.newFixedThreadPool(2);
        var total = new java.util.concurrent.atomic.AtomicInteger();
        for (int v : values) pool.submit(() -> total.addAndGet(v));
        pool.shutdown();
        return total.get();                    // read before the work finished
    }

    /* ─────────── 5 ─────────── */
    static int applyDiscount(int price, int percent) {
        return price - (price * percent / 100);
    }

    public static void main(String[] args) throws Exception {
        System.out.println("1 tagsInOrder  : " + tagsInOrder());
        System.out.println("2 pickWinner   : " + pickWinner(List.of("ana", "bo", "cy")));
        System.out.println("3 trialEnds    : " + trialEnds());
        System.out.println("4 slowSum      : " + slowSum(List.of(1, 2, 3, 4, 5)) + " (expected 15)");
        System.out.println("5 discount     : " + applyDiscount(100, 10) + " (expected 80)");

        // TODO 1: for each of the five write the cause in one line.
        //
        //   1 : ____________________
        //   2 : ____________________
        //   3 : ____________________
        //   4 : ____________________
        //   5 : ____________________

        // TODO 2: number 1 fails only sometimes and only on some JDKs. Fix
        // the ASSERTION rather than the data — say what the test was really
        // asserting, and what it should assert instead.

        // TODO 3: number 2 cannot be reproduced when it fails. Make it
        // reproducible FIRST, before making it pass. That order matters and
        // is the point of this one.

        // TODO 4: number 3 passes today. Work out the exact date on which it
        // will fail, then make the date an input instead of a fact.

        // TODO 5: number 4 is a race. Fix it by waiting for the work rather
        // than by sleeping — and say why raising a sleep would have been the
        // worst available fix.

        // TODO 6: number 5 is not flaky. It fails every single time. Say what
        // it tells you about a suite where nobody noticed.
    }
}

Run it locally:

cd exercises/java/test-strategy/flaky-tests/02-challenge
javac Starter.java -d /tmp/out && java -cp /tmp/out Starter

Hints

  1. Hint 1

    For each, ask what differs between two runs. If nothing in the test differs, look at what it shares with the rest of the suite.

  2. Hint 2

    One asserts on the order of something whose order was never specified.

  3. Hint 3

    One would be reproducible if a single line recorded a number.

  4. Hint 4

    One is not flaky at all. It fails for a reason, and the noise hid it.

Done when

  • Each of the five has a named cause and a fix that addresses the cause
  • The order-dependent assertion is fixed at the assertion, not the data
  • The random test is made reproducible before it is made to pass
  • The genuine defect is identified and reported rather than patched

← Back to What causes a flaky test, and how do you fix one?