Production incident

The breaker that never opened

45 minintermediate312 yrs

A real incident: symptom first, cause hidden, tradeoff at the end.

The incident

A pricing dependency degraded rather than died — roughly half its calls time out, the rest succeed normally. The service in front of it has a circuit breaker. It was configured deliberately, reviewed by two people, and it stayed CLOSED for the whole ninety-minute incident while the thread pool filled with timeouts and unrelated endpoints started failing. What the team found: 1. The breaker works. A test that kills the dependency completely opens it in five calls, exactly as configured. 2. During the incident the dependency was never completely down. 3. Raising the threshold, lowering it, and shortening the cool-down all changed nothing. 4. The dashboard showed a 50% error rate the entire time. Findings 1 and 2 together are the answer. Work out why a breaker that opens correctly for a dead dependency cannot see a sick one, and fix it.

What this teaches

  • Consecutive-failure counting only detects total failure
  • A success resets the run, so partial failure never trips it
  • Real outages are partial, which is the case the configuration must handle
  • A rate over a rolling window opens for both dead and degraded dependencies
  • The window must be full before it is judged, and cleared when the breaker closes

Starter

Starter.java
import java.util.*;
import java.util.function.*;

/**
 * Production: the breaker that never opened.
 *
 * A pricing dependency degraded rather than died — roughly half its calls
 * time out, the rest succeed. The service in front of it has a circuit
 * breaker, configured and reviewed, and it stayed closed for the whole
 * ninety-minute incident while its thread pool filled with timeouts.
 *
 * Time is injected, so every check below is exact rather than timed.
 *
 * Run it. Three checks, and it fails.
 */
public class Starter {

    /** Fails every other call — a degraded dependency, not a dead one. */
    static class Pricing {
        int calls = 0;
        boolean down = true;

        String fetch() {
            calls++;
            if (down && calls % 2 == 1) throw new IllegalStateException("read timed out");
            return "price";
        }
    }

    static class OpenCircuitException extends RuntimeException {
        OpenCircuitException() { super("circuit is open"); }
    }

    static class Clock {
        long now = 0;
        void advance(long millis) { now += millis; }
    }

    enum State { CLOSED, OPEN, HALF_OPEN }

    static class CircuitBreaker {
        private final int threshold;
        private final long openMillis;
        private final Clock clock;
        private State state = State.CLOSED;
        private long openedAt = 0;

        /** DEFECT: consecutive failures. One success resets the whole count. */
        private int consecutiveFailures = 0;

        CircuitBreaker(int threshold, long openMillis, Clock clock) {
            this.threshold = threshold;
            this.openMillis = openMillis;
            this.clock = clock;
        }

        State state() {
            if (state == State.OPEN && clock.now - openedAt >= openMillis) return State.HALF_OPEN;
            return state;
        }

        <T> T call(Supplier<T> action) {
            State current = state();
            if (current == State.OPEN) throw new OpenCircuitException();
            try {
                T result = action.get();
                state = State.CLOSED;
                consecutiveFailures = 0;
                return result;
            } catch (RuntimeException e) {
                if (current == State.HALF_OPEN || ++consecutiveFailures >= threshold) {
                    state = State.OPEN;
                    openedAt = clock.now;
                }
                throw e;
            }
        }
    }

    static final int REQUESTS = 200;

    public static void main(String[] args) {
        boolean ok = true;

        Clock clock = new Clock();
        Pricing pricing = new Pricing();
        CircuitBreaker breaker = new CircuitBreaker(5, 1000, clock);

        int rejected = 0, failed = 0, succeeded = 0;
        for (int i = 0; i < REQUESTS; i++) {
            try {
                breaker.call(pricing::fetch);
                succeeded++;
            } catch (OpenCircuitException e) {
                rejected++;
            } catch (RuntimeException e) {
                failed++;
            }
        }

        System.out.println("── " + REQUESTS + " requests against a dependency failing 50% ──");
        System.out.println("  reached the dependency   : " + pricing.calls);
        System.out.println("  rejected without calling : " + rejected);
        System.out.println("  timed out                : " + failed);
        System.out.println("  succeeded                : " + succeeded);
        System.out.println("  breaker state            : " + breaker.state());
        System.out.println();

        ok &= check("the breaker opened under a 50% failure rate", rejected > 0);
        ok &= check("fewer than half the requests reached the sick dependency",
            pricing.calls < REQUESTS / 2);

        System.out.println();
        System.out.println("── after recovery ──");
        pricing.down = false;
        clock.advance(1000);
        boolean closes;
        try {
            breaker.call(pricing::fetch);
            closes = breaker.state() == State.CLOSED;
        } catch (RuntimeException e) {
            closes = false;
        }
        System.out.println("  state : " + breaker.state());
        ok &= check("the breaker closes once the dependency recovers", closes);

        System.out.println();
        System.out.println(ok ? "PASS" : "FAIL");
    }

    static boolean check(String what, boolean passed) {
        System.out.println((passed ? "  ok    " : "  FAIL  ") + what);
        return passed;
    }
}

Run it locally:

cd exercises/java/resilience/circuit-breaker/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out Starter

Hints

  1. Hint 1

    Write out the sequence of outcomes at a 50% failure rate. How long is the longest run of failures?

  2. Hint 2

    What does a single success do to a consecutive counter?

  3. Hint 3

    What would you need to remember to answer 'what fraction of recent calls failed'?

  4. Hint 4

    Once you have a window: what is the failure rate after one failed call, and is that a reason to open?

Done when

  • The breaker opens under a 50% failure rate
  • Fewer than half the requests reach the sick dependency
  • The breaker still closes once the dependency recovers
  • A comment explains why the original passed its own test

Solution

Show the solution — try it yourself first
Solution.java
import java.util.*;
import java.util.function.*;

/**
 * Solution: the breaker that never opened.
 *
 * One defect, one line of configuration, and a ninety-minute outage.
 *
 * The breaker counted CONSECUTIVE failures. Against a dependency that is
 * fully dead that works — the failures arrive in an unbroken run. Against one
 * that is degraded, every other call succeeds and resets the counter, so five
 * in a row essentially never happens and the breaker stays closed while half
 * of all user requests time out.
 *
 * Total failure is the easy case. Partial failure is what a real outage looks
 * like, and it is the case consecutive counting cannot see.
 *
 * The fix is to count a failure RATE over a rolling window: "half of the last
 * twenty calls failed" opens correctly whether the dependency is dead or
 * merely sick. Every production breaker — Resilience4j, Polly, Hystrix before
 * them — is configured this way, and the option to count consecutive failures
 * mostly exists to catch total outages faster.
 *
 * Two details that matter as much as the window:
 *
 *   - The window must be FULL before the rate is judged, or the first failed
 *     call is a 100% failure rate and the breaker opens on a blip.
 *   - The window is cleared when the breaker closes, so a recovered
 *     dependency is not judged on history from before the outage.
 */
public class Solution {

    /** Fails every other call — a degraded dependency, not a dead one. */
    static class Pricing {
        int calls = 0;
        boolean down = true;

        String fetch() {
            calls++;
            if (down && calls % 2 == 1) throw new IllegalStateException("read timed out");
            return "price";
        }
    }

    static class OpenCircuitException extends RuntimeException {
        OpenCircuitException() { super("circuit is open"); }
    }

    static class Clock {
        long now = 0;
        void advance(long millis) { now += millis; }
    }

    enum State { CLOSED, OPEN, HALF_OPEN }

    static class CircuitBreaker {
        private final int threshold;
        private final long openMillis;
        private final Clock clock;
        private State state = State.CLOSED;
        private long openedAt = 0;

        /** FIX: the last N outcomes, judged as a rate rather than a run. */
        private final boolean[] window;
        private int index = 0;
        private int recorded = 0;

        CircuitBreaker(int threshold, long openMillis, Clock clock) {
            this.threshold = threshold;
            this.openMillis = openMillis;
            this.clock = clock;
            this.window = new boolean[WINDOW];
        }

        private void record(boolean failed) {
            window[index] = failed;
            index = (index + 1) % window.length;
            if (recorded < window.length) recorded++;
        }

        private void reset() {
            Arrays.fill(window, false);
            index = 0;
            recorded = 0;
        }

        /** Only judged once the window is full, so one bad call cannot trip it. */
        private boolean rateExceeded() {
            if (recorded < window.length) return false;
            int failures = 0;
            for (boolean failed : window) if (failed) failures++;
            return failures * 100 >= window.length * FAILURE_PERCENT;
        }

        State state() {
            if (state == State.OPEN && clock.now - openedAt >= openMillis) return State.HALF_OPEN;
            return state;
        }

        <T> T call(Supplier<T> action) {
            State current = state();
            if (current == State.OPEN) throw new OpenCircuitException();
            try {
                T result = action.get();
                record(false);
                // Only clear the history when the breaker actually CLOSES after
                // being open. Clearing on every success would mean the window
                // never fills, and the rate would never be judged at all.
                if (current == State.HALF_OPEN) reset();
                state = State.CLOSED;
                return result;
            } catch (RuntimeException e) {
                record(true);
                if (current == State.HALF_OPEN || rateExceeded()) {
                    state = State.OPEN;
                    openedAt = clock.now;
                    reset();
                }
                throw e;
            }
        }
    }

    static final int REQUESTS = 200;
    static final int WINDOW = 20;
    static final int FAILURE_PERCENT = 50;

    public static void main(String[] args) {
        boolean ok = true;

        Clock clock = new Clock();
        Pricing pricing = new Pricing();
        CircuitBreaker breaker = new CircuitBreaker(5, 1000, clock);

        int rejected = 0, failed = 0, succeeded = 0;
        for (int i = 0; i < REQUESTS; i++) {
            try {
                breaker.call(pricing::fetch);
                succeeded++;
            } catch (OpenCircuitException e) {
                rejected++;
            } catch (RuntimeException e) {
                failed++;
            }
        }

        System.out.println("── " + REQUESTS + " requests against a dependency failing 50% ──");
        System.out.println("  reached the dependency   : " + pricing.calls);
        System.out.println("  rejected without calling : " + rejected);
        System.out.println("  timed out                : " + failed);
        System.out.println("  succeeded                : " + succeeded);
        System.out.println("  breaker state            : " + breaker.state());
        System.out.println();

        ok &= check("the breaker opened under a 50% failure rate", rejected > 0);
        ok &= check("fewer than half the requests reached the sick dependency",
            pricing.calls < REQUESTS / 2);

        System.out.println();
        System.out.println("── after recovery ──");
        pricing.down = false;
        clock.advance(1000);
        boolean closes;
        try {
            breaker.call(pricing::fetch);
            closes = breaker.state() == State.CLOSED;
        } catch (RuntimeException e) {
            closes = false;
        }
        System.out.println("  state : " + breaker.state());
        ok &= check("the breaker closes once the dependency recovers", closes);

        System.out.println();
        System.out.println(ok ? "PASS" : "FAIL");
    }

    static boolean check(String what, boolean passed) {
        System.out.println((passed ? "  ok    " : "  FAIL  ") + what);
        return passed;
    }
}

Stretch

The fixed breaker judges only failures. A dependency that answers every call in eight seconds is not failing and is just as dangerous. Add slow-call detection — a call over a latency threshold counts toward the rate — and say how you would choose that threshold. Then argue whether a breaker should open on a 500 from the dependency but not on a 400.

← Back to What does a circuit breaker actually do?