Production incident

The config that was read half-updated

45 minintermediate312 yrs

A real incident: symptom first, cause hidden, tradeoff at the end.

The incident

A service reads its endpoint and its timeout from a settings object on every request. An operator can reload settings at runtime, which mutates that object in place. The pairing matters: each endpoint version has a matching timeout. A v2 endpoint with a v1 timeout is a request sent to the right place with the wrong deadline, and it shows up as a burst of spurious timeouts a few seconds after every reload. What the team found: 1. Reloads are rare. Reads are constant. The bursts line up exactly with reloads. 2. Making the two fields volatile did not help. 3. Synchronizing the readers was rejected on latency grounds before anyone checked whether it would even work. 4. It reproduces in a loop test within seconds. Finding 2 is the interesting one. Explain why volatile was never going to fix this, then fix it without putting a lock on the read path.

What this teaches

  • A reader cannot be made safe against an object that is briefly inconsistent
  • volatile publishes a reference; it does not make two field reads atomic
  • Replacing an immutable object is a single write, so it is all-or-nothing
  • A final field set in a constructor is visible, fully built, with no lock
  • One read of the shared reference into a local is part of the fix, not style

Starter

Starter.java
import java.lang.reflect.*;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;

/**
 * Production: the config that was read half-updated.
 *
 * A settings object is reloaded when an operator changes it. Readers call
 * endpoint() and timeoutMillis() on every request. The two must always
 * correspond — a v2 endpoint with a v1 timeout is a misrouted request with the
 * wrong deadline.
 *
 * Run it. Readers see mixed configurations, every time.
 */
public class Starter {

    static final int READERS = 6;
    static final int RELOADS = 40_000;

    /** Three valid configurations. endpoint and timeout must always match. */
    static final String[] ENDPOINTS = { "https://api/v1", "https://api/v2", "https://api/v3" };
    static final int[] TIMEOUTS = { 100, 200, 300 };

    /**
     * DEFECT: mutable, and reloaded by mutating the shared instance in place.
     * There is a window between the two writes in which the object holds one
     * version's endpoint and another version's timeout.
     */
    static class Settings {
        String endpoint = ENDPOINTS[0];
        int timeoutMillis = TIMEOUTS[0];

        void reload(int version) {
            this.endpoint = ENDPOINTS[version];
            this.timeoutMillis = TIMEOUTS[version];
        }
    }

    /** DEFECT: a plain field, so nothing publishes a new value across an edge. */
    static Settings settings = new Settings();

    public static void main(String[] args) throws Exception {
        boolean ok = true;

        AtomicLong reads = new AtomicLong();
        AtomicLong mixed = new AtomicLong();
        AtomicBoolean stop = new AtomicBoolean(false);

        Thread[] readers = new Thread[READERS];
        for (int i = 0; i < READERS; i++) {
            readers[i] = new Thread(() -> {
                while (!stop.get()) {
                    Settings s = settings;
                    String endpoint = s.endpoint;
                    int timeout = s.timeoutMillis;
                    reads.incrementAndGet();
                    if (!matches(endpoint, timeout)) mixed.incrementAndGet();
                }
            });
            readers[i].start();
        }

        for (int n = 0; n < RELOADS; n++) {
            settings.reload(n % ENDPOINTS.length);
        }
        stop.set(true);
        for (Thread t : readers) t.join();

        System.out.println("── " + RELOADS + " reloads, " + READERS + " concurrent readers ──");
        System.out.println("  reads observed    : " + reads.get());
        System.out.println("  mixed configs     : " + mixed.get());
        System.out.println();

        ok &= check("no reader ever saw a mixed configuration", mixed.get() == 0);
        ok &= check("the settings type is immutable — every field final",
            allFieldsFinal(Settings.class));
        ok &= check("the shared reference is published across a happens-before edge",
            publishedSafely(Starter.class, "settings"));

        System.out.println();
        System.out.println(ok ? "PASS" : "FAIL");
    }

    static boolean matches(String endpoint, int timeout) {
        for (int i = 0; i < ENDPOINTS.length; i++) {
            if (ENDPOINTS[i].equals(endpoint)) return TIMEOUTS[i] == timeout;
        }
        return false;
    }

    static boolean allFieldsFinal(Class<?> type) {
        for (Field f : type.getDeclaredFields()) {
            if (!Modifier.isStatic(f.getModifiers()) && !Modifier.isFinal(f.getModifiers())) {
                return false;
            }
        }
        return true;
    }

    /**
     * A shared reference that other threads read needs an edge. volatile is
     * the cheapest one; a final field or publication inside a lock also work.
     */
    static boolean publishedSafely(Class<?> owner, String field) throws Exception {
        int m = owner.getDeclaredField(field).getModifiers();
        return Modifier.isVolatile(m) || Modifier.isFinal(m);
    }

    static boolean check(String what, boolean passed) {
        System.out.println((passed ? "  ok    " : "  FAIL  ") + what);
        return passed;
    }
}

Run it locally:

cd exercises/java/concurrency/happens-before/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out Starter

Hints

  1. Hint 1

    During reload, is there a moment when the object itself is wrong? No reader-side change can help with that.

  2. Hint 2

    What would a reader have to hold for the two values to be guaranteed to correspond?

  3. Hint 3

    If the object is never modified after construction, what is left to protect?

  4. Hint 4

    Count the reads of the shared reference on the read path. Two is one too many.

Done when

  • No reader ever observes a mixed configuration, across the whole run
  • The settings type is immutable — every field final
  • The shared reference is published across a happens-before edge
  • There is no lock on the read path
  • A comment explains why volatile on the two fields was never the fix

Solution

Show the solution — try it yourself first
Solution.java
import java.lang.reflect.*;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;

/**
 * Solution: the config that was read half-updated.
 *
 * The defect was not a missing lock on the reader. It was that a reload
 * MUTATED the object every reader was holding, so there was a window in which
 * the object itself was inconsistent — a v2 endpoint beside a v1 timeout. No
 * amount of synchronizing the readers fixes an object that is briefly wrong.
 *
 * The fix is the one the entry keeps arriving at: never mutate shared state
 * in place. Build a complete, immutable replacement and publish it with a
 * single write across a happens-before edge.
 *
 *   1. Settings becomes immutable — every field final, set in the constructor.
 *      A final field written in a constructor is guaranteed visible, fully
 *      initialised, to any thread that sees the reference. That is the freeze
 *      action, and it is why no lock is needed on the read path.
 *
 *   2. The shared reference becomes volatile, so the write publishing the new
 *      object happens-before every later read of it. One write, one edge, and
 *      everything the constructor did comes across with it.
 *
 *   3. Readers take ONE read of the reference into a local and use that. Two
 *      reads could straddle a reload and reintroduce the mix — this is the
 *      part people leave out, and it is why the local variable is not a style
 *      choice.
 *
 * Cost: an object per reload. Reloads are rare and reads are constant, so this
 * is the right trade — and there is no lock anywhere on the hot path.
 */
public class Solution {

    static final int READERS = 6;
    static final int RELOADS = 40_000;

    static final String[] ENDPOINTS = { "https://api/v1", "https://api/v2", "https://api/v3" };
    static final int[] TIMEOUTS = { 100, 200, 300 };

    /** FIX 1: immutable. Nothing can observe it partly updated, ever. */
    static final class Settings {
        final String endpoint;
        final int timeoutMillis;

        Settings(int version) {
            this.endpoint = ENDPOINTS[version];
            this.timeoutMillis = TIMEOUTS[version];
        }
    }

    /** FIX 2: volatile, so one write publishes the whole object. */
    static volatile Settings settings = new Settings(0);

    public static void main(String[] args) throws Exception {
        boolean ok = true;

        AtomicLong reads = new AtomicLong();
        AtomicLong mixed = new AtomicLong();
        AtomicBoolean stop = new AtomicBoolean(false);

        Thread[] readers = new Thread[READERS];
        for (int i = 0; i < READERS; i++) {
            readers[i] = new Thread(() -> {
                while (!stop.get()) {
                    // FIX 3: one read of the reference, then use the local.
                    Settings s = settings;
                    String endpoint = s.endpoint;
                    int timeout = s.timeoutMillis;
                    reads.incrementAndGet();
                    if (!matches(endpoint, timeout)) mixed.incrementAndGet();
                }
            });
            readers[i].start();
        }

        for (int n = 0; n < RELOADS; n++) {
            settings = new Settings(n % ENDPOINTS.length);
        }
        stop.set(true);
        for (Thread t : readers) t.join();

        System.out.println("── " + RELOADS + " reloads, " + READERS + " concurrent readers ──");
        System.out.println("  reads observed    : " + reads.get());
        System.out.println("  mixed configs     : " + mixed.get());
        System.out.println();

        ok &= check("no reader ever saw a mixed configuration", mixed.get() == 0);
        ok &= check("the settings type is immutable — every field final",
            allFieldsFinal(Settings.class));
        ok &= check("the shared reference is published across a happens-before edge",
            publishedSafely(Solution.class, "settings"));

        System.out.println();
        System.out.println(ok ? "PASS" : "FAIL");
    }

    static boolean matches(String endpoint, int timeout) {
        for (int i = 0; i < ENDPOINTS.length; i++) {
            if (ENDPOINTS[i].equals(endpoint)) return TIMEOUTS[i] == timeout;
        }
        return false;
    }

    static boolean allFieldsFinal(Class<?> type) {
        for (Field f : type.getDeclaredFields()) {
            if (!Modifier.isStatic(f.getModifiers()) && !Modifier.isFinal(f.getModifiers())) {
                return false;
            }
        }
        return true;
    }

    static boolean publishedSafely(Class<?> owner, String field) throws Exception {
        int m = owner.getDeclaredField(field).getModifiers();
        return Modifier.isVolatile(m) || Modifier.isFinal(m);
    }

    static boolean check(String what, boolean passed) {
        System.out.println((passed ? "  ok    " : "  FAIL  ") + what);
        return passed;
    }
}

Stretch

This design allocates one object per reload. Argue when that is wrong — a reload rate approaching the read rate — and what you would use instead. Then say what changes if a reader needs several settings that must all come from the same generation, spread across three different objects.

← Back to What is the happens-before relationship?