Challenge

Make the numbers defensible

25 minintermediate312 yrs

Edge cases. You have to reason, and two valid fixes differ.

What this teaches

  • A failure rate over a window opens under partial failure; a run does not
  • The window must be full before the rate is judged, or a blip trips it
  • Retry belongs inside the breaker, never outside it
  • Without a timeout there is no failure to count
  • One breaker per dependency, or an unimportant failure blocks a critical call

Starter

Starter.java
import java.util.*;
import java.util.function.*;

/**
 * Challenge: the breaker works. Now make the configuration defensible.
 *
 * Every number here is a decision with a consequence in both directions.
 * Four experiments, each of which changes an answer you would give in an
 * interview.
 */
public class Starter {

    static class Clock {
        long now = 0;
        void advance(long millis) { now += millis; }
    }

    /** Fails a fixed percentage of calls, deterministically. */
    static class Flaky {
        private final int failPercent;
        int calls = 0;

        Flaky(int failPercent) { this.failPercent = failPercent; }

        String fetch() {
            calls++;
            // Deterministic: exactly failPercent failures in every hundred.
            if (calls % 100 < failPercent) throw new IllegalStateException("timed out");
            return "ok";
        }
    }

    // TODO 1: bring in your breaker from the warm-up, or write it again.

    public static void main(String[] args) {
        System.out.println("four experiments — predict each before running it");

        // ── 1: consecutive failures versus a failure rate ────────────────
        // TODO: run 200 calls against a dependency failing 50% of the time,
        // through a breaker counting 5 CONSECUTIVE failures. Count how many
        // reached the dependency.
        //
        // Then write a second breaker that counts "50% of the last 20 calls"
        // and run the same 200. Compare.
        //
        // State which one you would ship, and what the other one is good for.

        // ── 2: the empty window ──────────────────────────────────────────
        // TODO: judge the rate BEFORE the window is full and make one call
        // fail. What is the failure rate over one call? What does your
        // breaker do? Fix it, and say what the fix is protecting against.

        // ── 3: retry outside versus inside ───────────────────────────────
        // TODO: wrap 3 retries AROUND the breaker and run 100 user requests
        // during a total outage. Count:
        //   - calls that reached the dependency
        //   - open-circuit rejections consumed per user request
        // Then move the retry INSIDE and measure again. State the multiplier.

        // ── 4: one breaker or many ───────────────────────────────────────
        // TODO: two dependencies — a critical one that is healthy and a
        // reporting one that is dead — behind ONE breaker. Show that calls to
        // the healthy dependency get rejected. Then fix it, and state the rule
        // in one sentence.

        // TODO 5: write down your three numbers — failure rate, window size,
        // cool-down — with one sentence of justification each, and what goes
        // wrong if each is too low and too high.

        // TODO 6: what must exist on the underlying call for ANY of this to
        // work? Say what happens without it.
        System.out.println("run the four experiments, then delete this line");
    }
}

Run it locally:

cd exercises/java/resilience/circuit-breaker/02-challenge
javac Starter.java -d /tmp/out && java -cp /tmp/out Starter

Hints

  1. Hint 1

    Simulate a dependency failing 50% of the time and see which counting strategy opens.

  2. Hint 2

    What is the failure rate after the very first failed call, if you judge an empty window?

  3. Hint 3

    Put the retry outside and count how many rejections one user request consumes during an outage.

  4. Hint 4

    Two dependencies, one breaker: make the unimportant one fail and see what happens to the important one.

Done when

  • The rate-based breaker opens under 50% failure; the run-based one does not
  • A single failure does not open your breaker
  • You measured the retry-outside case and can state the multiplier
  • You can justify each of your three numbers in one sentence

← Back to What does a circuit breaker actually do?