Production incident

The balance that created money

45 minintermediate312 yrs

A real incident: symptom first, cause hidden, tradeoff at the end.

The incident

A ledger service moves money between accounts. Transfer is a synchronized method on Account, every field it touches is guarded, and it has been in production for two years. Then reconciliation starts failing. Not by much — a few hundred units a day across millions of transfers, sometimes positive, sometimes negative. Money is being created and destroyed. What the team found: 1. It only happens when two transfers between the same pair of accounts run in opposite directions at the same time. 2. Every method involved is synchronized. There is no unguarded field. 3. Adding volatile to the balance changed nothing. 4. The first fix — locking both accounts — made the service hang under load instead. Finding 2 is true and finding 3 is expected. Explain both, then fix it so that neither the totals drift nor the service hangs.

What this teaches

  • synchronized guards a monitor, so it protects `this` and not the argument
  • An operation over two objects needs both monitors, not two correct locks
  • Acquiring two locks in call order deadlocks when callers disagree on order
  • A consistent global acquisition order is what actually prevents deadlock
  • volatile cannot fix an atomicity defect, however many fields you add it to

Starter

Starter.java
import java.util.*;
import java.util.concurrent.*;

/**
 * Production: the balance that created money.
 *
 * Two accounts, eight threads, transfers in both directions. Every field that
 * matters is guarded by a synchronized method, and the code reads as though it
 * is safe.
 *
 * Run it. Money appears or disappears, every time.
 */
public class Starter {

    static final int THREADS = 8;
    static final int PER_THREAD = 100_000;
    static final long OPENING = 1_000_000L;

    static class Account {
        final int id;
        long balance;

        Account(int id, long balance) {
            this.id = id;
            this.balance = balance;
        }

        /**
         * DEFECT: this locks `this`, so it guards the debit and not the credit.
         * A transfer in the opposite direction holds the OTHER account's
         * monitor and writes this one's balance with no exclusion at all.
         */
        synchronized void transfer(Account to, long amount) {
            if (balance < amount) return;
            balance -= amount;
            to.balance += amount;
        }

        synchronized long balance() {
            return balance;
        }
    }

    public static void main(String[] args) throws Exception {
        boolean ok = true;

        Account a = new Account(1, OPENING);
        Account b = new Account(2, OPENING);
        long before = a.balance + b.balance;

        System.out.println("── " + THREADS + " threads x " + PER_THREAD + " transfers, both directions ──");

        Thread work = new Thread(() -> {
            try {
                hammer(a, b);
            } catch (Exception e) {
                throw new RuntimeException(e);
            }
        });
        work.start();
        work.join(30_000);

        boolean finished = !work.isAlive();
        if (!finished) {
            System.out.println("  still running after 30s — threads are stuck");
            work.interrupt();
        }

        long after = a.balance + b.balance;
        System.out.println("  opening total : " + before);
        System.out.println("  closing total : " + after);
        System.out.println("  difference    : " + (after - before));
        System.out.println("  account 1     : " + a.balance);
        System.out.println("  account 2     : " + b.balance);
        System.out.println();

        ok &= check("the run completed — nothing deadlocked", finished);
        ok &= check("total money is unchanged", after == before);
        ok &= check("no account ended negative", a.balance >= 0 && b.balance >= 0);

        System.out.println();
        System.out.println(ok ? "PASS" : "FAIL");
    }

    /** Half the threads move money one way, half the other. */
    static void hammer(Account a, Account b) throws Exception {
        Thread[] threads = new Thread[THREADS];
        CountDownLatch start = new CountDownLatch(1);
        for (int i = 0; i < THREADS; i++) {
            Account from = (i % 2 == 0) ? a : b;
            Account to = (i % 2 == 0) ? b : a;
            threads[i] = new Thread(() -> {
                try {
                    start.await();
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                    return;
                }
                for (int n = 0; n < PER_THREAD; n++) from.transfer(to, 1L);
            });
            threads[i].start();
        }
        start.countDown();
        for (Thread t : threads) t.join();
    }

    static boolean check(String what, boolean passed) {
        System.out.println((passed ? "  ok    " : "  FAIL  ") + what);
        return passed;
    }
}

Run it locally:

cd exercises/java/concurrency/volatile-vs-synchronized/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out Starter

Hints

  1. Hint 1

    Write out which fields the method writes, and which monitor covers each one. One of them is not covered.

  2. Hint 2

    Two transfers in opposite directions: which monitor does each hold, and which does each need?

  3. Hint 3

    Locking both in the order they were passed is the deadlock. What has to be true about the order instead?

  4. Hint 4

    Account id is a total order. Any total order works, as long as nothing ever deviates from it.

Done when

  • Total money is unchanged after the run, every run
  • The run completes — no deadlock under opposing transfers
  • No account ends negative
  • A comment explains why volatile on the balance was never going to help

Solution

Show the solution — try it yourself first
Solution.java
import java.util.*;
import java.util.concurrent.*;

/**
 * Solution: the balance that created money.
 *
 * The defect was lock scope, not a missing lock. `synchronized void transfer`
 * locks `this`, so it guarded the debit and left the credit — a write to
 * ANOTHER object's field — completely unguarded. A transfer in the opposite
 * direction holds the other account's monitor, so the two never exclude each
 * other, and both balances are written concurrently.
 *
 * The fix has two halves, and the second is the one people miss:
 *
 *   1. A transfer touches two accounts, so it must hold both monitors. Two
 *      separately-correct locks do not make one correct operation.
 *   2. Acquiring both in call order deadlocks the moment two transfers run in
 *      opposite directions — each holds what the other needs. So they must be
 *      acquired in a consistent GLOBAL order. Account id is a natural one, and
 *      any total order works as long as every caller uses the same one.
 *
 * Note what is NOT needed: volatile on the balance. A field only read and
 * written while holding a monitor gets its visibility from the monitor.
 */
public class Solution {

    static final int THREADS = 8;
    static final int PER_THREAD = 100_000;
    static final long OPENING = 1_000_000L;

    static class Account {
        final int id;
        long balance;

        Account(int id, long balance) {
            this.id = id;
            this.balance = balance;
        }

        /**
         * FIX: hold both monitors, always in id order, so no two transfers can
         * acquire them in opposite sequences.
         */
        void transfer(Account to, long amount) {
            Account first = this.id < to.id ? this : to;
            Account second = this.id < to.id ? to : this;

            synchronized (first) {
                synchronized (second) {
                    if (balance < amount) return;
                    balance -= amount;
                    to.balance += amount;
                }
            }
        }

        long balance() {
            synchronized (this) {
                return balance;
            }
        }
    }

    public static void main(String[] args) throws Exception {
        boolean ok = true;

        Account a = new Account(1, OPENING);
        Account b = new Account(2, OPENING);
        long before = a.balance + b.balance;

        System.out.println("── " + THREADS + " threads x " + PER_THREAD + " transfers, both directions ──");

        Thread work = new Thread(() -> {
            try {
                hammer(a, b);
            } catch (Exception e) {
                throw new RuntimeException(e);
            }
        });
        work.start();
        work.join(30_000);

        boolean finished = !work.isAlive();
        if (!finished) {
            System.out.println("  still running after 30s — threads are stuck");
            work.interrupt();
        }

        long after = a.balance + b.balance;
        System.out.println("  opening total : " + before);
        System.out.println("  closing total : " + after);
        System.out.println("  difference    : " + (after - before));
        System.out.println("  account 1     : " + a.balance);
        System.out.println("  account 2     : " + b.balance);
        System.out.println();

        ok &= check("the run completed — nothing deadlocked", finished);
        ok &= check("total money is unchanged", after == before);
        ok &= check("no account ended negative", a.balance >= 0 && b.balance >= 0);

        System.out.println();
        System.out.println(ok ? "PASS" : "FAIL");
    }

    static void hammer(Account a, Account b) throws Exception {
        Thread[] threads = new Thread[THREADS];
        CountDownLatch start = new CountDownLatch(1);
        for (int i = 0; i < THREADS; i++) {
            Account from = (i % 2 == 0) ? a : b;
            Account to = (i % 2 == 0) ? b : a;
            threads[i] = new Thread(() -> {
                try {
                    start.await();
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                    return;
                }
                for (int n = 0; n < PER_THREAD; n++) from.transfer(to, 1L);
            });
            threads[i].start();
        }
        start.countDown();
        for (Thread t : threads) t.join();
    }

    static boolean check(String what, boolean passed) {
        System.out.println((passed ? "  ok    " : "  FAIL  ") + what);
        return passed;
    }
}

Stretch

Lock ordering works but scales badly as the number of locks in one operation grows. Argue for the two alternatives: a single lock over the whole ledger (simple, contended) and an optimistic version-check retry (no locks, but the operation must be safe to repeat). Say which you would ship for a payments ledger and what you would need to measure first.

← Back to What does volatile guarantee, and what does it not?