What causes a flaky test, and how do you fix one?

The Answer

Say this in the room. 45 seconds.

• A flaky test passes and fails without the code changing. The cost is not the test — it is that the team learns to re-run red builds, and then a real failure looks the same.
• Four causes, in the order worth checking: order dependence, shared state, time, real concurrency.
• Order and shared state are the same bug seen from two angles: one test leaves something behind and another depends on it.
• Time means anything reading the clock — month ends, leap years, daylight saving, timezone, and "now plus one day".
• Concurrency means a sleep standing in for a guarantee. Force the interleaving with a latch instead.
• Retrying is not a fix. It hides the signal, keeps the bug, and the bug is often in the production code rather than the test.

Understand It

Order dependence, which is the one people do not look for

Four tests. Nothing about them changes between the runs below — only the order they are executed in:

var declared = suite();
System.out.println("  declaration order : " + describe(run(declared)));

var reversed = new ArrayList<>(suite());
Collections.reverse(reversed);
System.out.println("  reversed order    : " + describe(run(reversed)));

var rotated = new ArrayList<>(suite());
Collections.rotate(rotated, 2);
System.out.println("  rotated by two    : " + describe(run(rotated)));

  declaration order : 1 failed: [registryIsEmptyForNewSession]
  reversed order    : 2 failed: [addsSecond, addsUser]
  rotated by two    : all pass

Three orders, three different answers — including one where everything passes, which is how this reaches production. The suite is green on the developer's machine and red in CI, and the only difference is that the runner chose a different order.

Two things follow.

The failing test is usually not the broken one. registryIsEmptyForNewSession fails because the two tests before it left data behind. Fixing it — by making it tolerate leftovers — makes the suite green and removes the only evidence.

JUnit 5 does not guarantee order and deliberately varies it. The default is deterministic but intentionally not declaration order, precisely so this surfaces. @TestMethodOrder exists for the rare suite that genuinely needs a sequence, and using it to stop a flake is suppressing the finding.

The diagnostic is cheap: run the suite in a random order a few times. JUnit will do it for you.

junit.jupiter.testinstance.lifecycle.default = per_method
junit.jupiter.execution.order.random.seed = 42

Time, which fails on a schedule

A test that reads the clock is a test whose result depends on the day it runs. This is the same calculation on four different start dates:

Function<LocalDate, Boolean> renewsSameDayNextMonth =
    start -> start.plusMonths(1).getDayOfMonth() == start.getDayOfMonth();

for (String date : List.of("2026-03-15", "2026-01-31", "2024-02-29", "2026-08-31")) {
    LocalDate d = LocalDate.parse(date);
    System.out.println("  start " + date + " -> "
        + (renewsSameDayNextMonth.apply(d) ? "passes" : "FAILS"));
}

  start 2026-03-15 -> passes
  start 2026-01-31 -> FAILS
  start 2024-02-29 -> passes
  start 2026-08-31 -> FAILS

Written with LocalDate.now(), that test passes for twenty-eight days a month and fails on the thirty-first — so it fails roughly seven times a year, always overnight, and nobody is watching. Add year-end, leap days, daylight saving and a CI runner in UTC while the developers are not, and "it only fails sometimes" is fully explained.

The fix is to stop reading the clock. Inject a Clock:

public class Subscriptions {
    private final Clock clock;                       // injected

    Subscriptions(Clock clock) { this.clock = clock; }

    public LocalDate renewalDate() {
        return LocalDate.now(clock).plusMonths(1);   // not LocalDate.now()
    }
}

// In the test, time is an input like any other.
var subs = new Subscriptions(Clock.fixed(Instant.parse("2026-01-31T00:00:00Z"), ZoneOffset.UTC));

Once the clock is a parameter, the month-end case becomes a test you can write deliberately rather than a failure you wait for.

Retrying makes it worse, and here is the arithmetic

The usual response to a flaky test is @RepeatedTest, a retry extension, or retries: 3 in the CI config. Thirty builds of a test that fails one attempt in three:

int seed = 0, failedAttempts = 0, redBuilds = 0, totalAttempts = 0;

for (int build = 1; build <= 30; build++) {
    boolean green = false;
    for (int attempt = 1; attempt <= 3; attempt++) {
        totalAttempts++;
        if (seed++ % 3 == 0) failedAttempts++;
        else { green = true; break; }
    }
    if (!green) redBuilds++;
}

System.out.println("  builds run             : 30");
System.out.println("  test attempts          : " + totalAttempts);
System.out.println("  attempts that failed   : " + failedAttempts);
System.out.println("  builds reported as red : " + redBuilds);

  builds run             : 30
  test attempts          : 45
  attempts that failed   : 15
  builds reported as red : 0

Fifteen failures, and the dashboard shows thirty green builds. The failure rate did not change by a single percent; the reporting did.

That is the case against retries stated precisely. Two further points make it worse:

The bug is often in the production code. A test that fails intermittently under a real database or real threads is frequently reporting a genuine race. Retrying discards the only evidence you had.

Trust is the real asset. Once a team re-runs a red build as a matter of routine, a genuine regression gets re-run too. The suite stops being a signal, which is a much larger loss than one deleted test.

If you must retry, make it visible: mark the test, record the flake rate, and put a deadline on it. A quarantined test with an owner is defensible; a global retry setting is how a suite dies quietly.

Concurrency, where sleep is the tell

service.processAsync(order);
Thread.sleep(100);                       // hope
assertThat(repository.findById(id)).isPresent();

That test passes on a fast machine and fails on a loaded CI runner, and the usual fix — raising it to 500 — makes the suite slower and still flaky.

A sleep is a guess about timing standing in for a guarantee. Replace it with one:

CountDownLatch done = new CountDownLatch(1);
service.processAsync(order, done::countDown);
assertThat(done.await(5, SECONDS)).isTrue();      // a deadline, not a delay

// Or make the boundary synchronous in the test.
var service = new OrderService(Runnable::run);    // same-thread executor

// Or poll for the condition with a timeout.
await().atMost(5, SECONDS).until(() -> repository.findById(id).isPresent());

The first two are better than the third, because they finish as soon as the work does rather than after a fixed wait.

And the honest limit: for a memory-visibility bug, no amount of running is evidence either way. A green concurrency test means the interleaving did not occur, not that it cannot.

Reference

How to find them, how to fix each cause, and what to configure. Copy from here.

Finding order dependence

# junit-platform.properties — random order, and a fresh instance per test
junit.jupiter.testinstance.lifecycle.default = per_method
junit.jupiter.execution.order.random.seed = 42

# Run one test alone. If it passes alone and fails in the suite, it is polluted.
mvn test -Dtest=RegistryTest#registryIsEmptyForNewSession

# Bisect a suite by running halves until the pair is found.
mvn test -Dtest='RegistryTest,UserTest'

Fixing each cause

// SHARED STATE — reset what you touch, in the right scope.
@BeforeEach void reset() { registry.clear(); }              // per test, preferred
@AfterEach  void cleanUp() { ... }                          // for external resources

@DirtiesContext                                             // Spring: rebuild the context
@Transactional                                              // Spring test: roll back after each

// TIME — inject a Clock and fix it.
@TestConfiguration
static class FixedTime {
    @Bean Clock clock() { return Clock.fixed(Instant.parse("2026-01-31T00:00:00Z"), UTC); }
}

// RANDOMNESS — seed it, and print the seed on failure.
var random = new Random(seed);                              // never new Random()

// ORDERING OF RESULTS — assert as a set, or sort first.
assertThat(result).containsExactlyInAnyOrder("a", "b");     // not containsExactly

// CONCURRENCY — a deadline, not a delay.
assertThat(latch.await(5, SECONDS)).isTrue();

// PORTS AND FILES — never hardcode.
int port = 0;                                               // 0 = the OS picks a free one
Path temp = Files.createTempDirectory("test");              // not /tmp/fixed-name

The usual sources, in the order to check them

Quarantine, if you must

@Tag("flaky")                                  // excluded from the gating build
@Disabled("FLAKY-482: fails at month end, owner @hemanth, remove by 2026-10-01")

An owner and a date. A @Disabled with neither is a deleted test that still costs you a file.

Scenarios

Real situations, with the decision and the argument.

1. A test fails in CI once a week and passes on every developer machine.

Before touching the test, collect what differs: CI runs in UTC, in a different order, possibly in parallel, on a slower machine with less memory. Each of those maps to one of the four causes.

Run the suite locally with the CI seed and TZ=UTC, and it usually reproduces immediately. The instinct to add a retry is strongest here precisely because the failure is rare — and rare-but-real is exactly the profile of the production bugs worth finding.

2. Someone proposes turning on retries: 2 for the whole pipeline.

The arithmetic above is the argument: it changes reporting rather than failure rate. It also applies to every test, so a genuine regression that fails intermittently — a race in new code — now gets retried into green.

If the pipeline is unusable today, a time-boxed retry with a tracked list and a removal date is a defensible compromise. What is not defensible is turning it on and moving the conversation to something else, because nothing will ever turn it off.

3. A test is flaky because the code under test is genuinely racy.

This is the outcome worth hoping for, and it is regularly thrown away. An intermittent failure against a real database or real threads is evidence of a production defect that no other test found.

Before changing the test, work out whether the assertion is wrong or the code is. If two threads really can interleave that way in production, the test is the only thing that noticed. Fixing the code and keeping the test is the right outcome; deleting the test converts a known bug into an unknown one.

4. Parallelising the suite to cut build time, and forty tests start failing.

They were always sharing state — static fields, a database, a fixed port, a temp file with a fixed name — and serial execution was hiding it.

The forty failures are a finding, not a regression. Fix the shared state where it is cheap, and where it is not, mark those classes @Execution(SAME_THREAD) and keep the rest parallel. Reverting to serial buys back the build time and keeps every one of the dependencies.

5. A test asserts on a list and fails only in production-like environments.

containsExactly on something whose order was never specified — a HashSet, a groupingBy result, a query with no ORDER BY. The order is stable enough locally to look deterministic and changes with the data, the JDK, or the plan.

Assert as a set, or sort before comparing. If the order genuinely matters to the application, the fix is upstream: add the ORDER BY, or use a type that promises an order. A test asserting an order the code does not guarantee is testing an accident.

Interviewer's Next Move

The question after the question. This is what the round is actually testing.

1. "What causes a flaky test?" Order dependence, shared state, time, and real concurrency — in that order of how often they turn up. The first two are the same bug seen from two sides: something is left behind and something else depends on it.

2. "A test passes alone and fails in the suite. Where do you look?" State left by another test — a static field, a database row, a cache, a system property. The failing test is usually not the broken one; the test before it is.

3. "How do you find which test is polluting?" Run in a random order with a fixed seed so the failure is reproducible, then bisect by running halves of the suite until the pair is isolated.

4. "Is it acceptable to retry a flaky test in CI?" It changes the reporting, not the failure rate — a test failing one attempt in three shows zero red builds over thirty with three retries. And it retries genuine regressions too. Defensible only as a time-boxed quarantine with an owner and a date.

5. "A test only fails at month end." Something calls now(). Inject a Clock, fix it in the test, and add the month-end case deliberately — the bug is often in the production code's date arithmetic rather than in the test.

6. "How do you test asynchronous code without a sleep?" A latch with a timeout, a same-thread executor so the boundary is synchronous, or polling for the condition with a deadline. A sleep is a guess about timing standing in for a guarantee, and it is both slow and still flaky.

7. "Parallelising the suite broke forty tests. What happened?" They shared state that serial execution hid — statics, a database, a fixed port. That is a finding rather than a regression, and reverting to serial keeps every one of the dependencies.

8. "When would you delete a flaky test rather than fix it?" When it tests something no longer worth the cost — a duplicate of a cheaper test, or an implementation detail. Deleting it deliberately is respectable; leaving it retried is not, because it still costs build time and still hides the next failure.

Code traps

Trap A — predict before you run:

private static final List<String> CACHE = new ArrayList<>();

@Test void addsEntry()    { CACHE.add("a"); assertThat(CACHE).hasSize(1); }
@Test void startsEmpty()  { assertThat(CACHE).isEmpty(); }

Trap B:

@Test
void expiresAfterThirtyDays() {
    var token = new Token(LocalDate.now());
    assertThat(token.expiresOn()).isEqualTo(LocalDate.now().plusDays(30));
}

Trap C:

@Test
void savesAllRecords() throws Exception {
    importer.importAsync(file);
    Thread.sleep(200);
    assertThat(repository.count()).isEqualTo(1000);
}

Common wrong answers

Check Yourself

Q1. A test passes alone and fails in the suite. Which test is broken?

Q2. Why is retrying a flaky test worse than leaving it red?

Q3. How would you test that an async import saved a thousand rows, without a sleep?

Practice