Interview replay
Full round replay — flaky tests
Timed verbal replay with pass/fail criteria per follow-up.
How to run this
The opener
“What causes a flaky test, and how do you fix one?”
Budget: 45 seconds. Going long here is itself a fail signal.
Follow-ups
1. “A test passes alone and fails in the suite. Where do you look?”
Testing: Do they suspect the failing test or the ones before it?
Scoring
Pass: State left by another test — a static field, a database row, a cache, a system property. The failing test is usually the one that noticed, not the one that broke.
Fail: Starts debugging the test that failed.
2. “How would you find which test is polluting?”
Testing: A method, not a guess.
Scoring
Pass: Run in a random order with a fixed seed so the failure reproduces, then bisect by running halves of the suite until the pair is isolated.
Fail: Reads the tests hoping to spot it.
3. “Is it acceptable to retry a flaky test in CI?”
Testing: Can they quantify the objection?
Scoring
Pass: It changes reporting, not the failure rate — a test failing one attempt in three shows zero red builds over thirty with three retries. It also retries genuine regressions. Defensible only as a time-boxed quarantine with an owner and a date.
Fail: Treats retries as a normal mitigation.
4. “A test fails only at month end.”
Testing: Time as an input.
Scoring
Pass: Something calls now(). Inject a Clock and fix it in the test — and check whether the production date arithmetic is actually wrong, because that assertion is often reporting a real bug.
Fail: Adjusts the assertion so it passes on the 31st.
5. “How do you test asynchronous code without a sleep?”
Testing: Condition versus duration.
Scoring
Pass: A latch with a timeout, a same-thread executor so the boundary is synchronous, or polling for the condition with a deadline. It finishes as soon as the work does, and a hang fails at the deadline.
Fail: Suggests a longer sleep.
6. “Parallelising the suite broke forty tests. What happened?”
Testing: Finding versus regression.
Scoring
Pass: They shared state that serial execution hid — statics, a database, a fixed port. That is a finding. Reverting to serial keeps every one of the dependencies.
Fail: Concludes the suite cannot be parallelised.
7. “Can a green concurrency test prove the code is thread-safe?”
Testing: The limit of testing.
Scoring
Pass: No. It proves the interleaving did not occur on that run, on that machine. Visibility and ordering are argued with happens-before; jcstress exists because ordinary tests cannot reach these interleavings.
Fail: Treats a passing run as evidence.
8. “When would you delete a flaky test rather than fix it?”
Testing: Judgement, not dogma.
Scoring
Pass: When it duplicates a cheaper test, or asserts an implementation detail. Deleting it deliberately is respectable; leaving it retried is not, because it still costs build time and still hides the next failure.
Fail: Says never delete a test.
9. “What is the actual cost of a flaky suite?”
Testing: The answer that shows seniority.
Scoring
Pass: Trust. Once the team re-runs red builds by reflex, a genuine regression looks identical to a flake — so the suite stops being a signal, which is a bigger loss than any one test.
Fail: Only mentions wasted build minutes.