ExerciseProduction incident
Production incident
The transfer service that stalled at month end
45 minintermediate2–15 yrs
A real incident: symptom first, cause hidden, tradeoff at the end.
The incident
Eleven months of the year this service is fine. At month end, when batch
transfers run in both directions at once, throughput collapses and the
ledger stops balancing by a few hundred rupees. Nobody can reproduce it in
staging, where the batch runs one direction at a time.
What the review turned up:
1. Locks are taken in argument order, so transfer(a, b) and transfer(b, a)
reach for the same two locks in opposite orders.
2. A failed lock acquisition returns true — the caller records a completed
transfer that never happened.
3. The slow audit write runs while both account locks are held, so every
other transfer on those accounts queues behind a remote call.
The three compound: the ordering causes contention, the slow call under the
lock makes every contended wait long enough to time out, and the swallowed
timeout turns those into silent successes. Fix all three.
This version uses tryLock with a timeout, so a mistake fails a check rather
than hanging your machine. The same bug under plain synchronized is a hard
deadlock with no timeout and no recovery.
What this teaches
- Inconsistent lock ordering is the cause; a global order on a stable key is the fix
- identityHashCode is not a valid ordering key, because it is not guaranteed unique
- A lock you could not acquire is a failure, and reporting it as success hides the incident
- Never hold a lock across a call you do not control
- Under synchronized these same three defects produce a hang instead of a wrong number
Starter
Starter.java
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;
import java.util.concurrent.locks.ReentrantLock;
/**
* Production: the transfer service that stalled at month end.
*
* Eleven months of the year this service is fine. At month end, when batch
* transfers run in both directions at once, transfers start failing, the
* ledger stops balancing, and throughput collapses.
*
* Run this. Four checks fail. Fix TransferService so all four pass, without
* weakening the checks.
*
* Note: this uses tryLock with a timeout throughout, so a mistake makes the
* checks fail rather than hanging your machine. The underlying bug is the one
* that produces a hard deadlock under plain synchronized.
*/
public class Starter {
static final class Account {
final String id;
final ReentrantLock lock = new ReentrantLock();
volatile long balance;
Account(String id, long balance) { this.id = id; this.balance = balance; }
}
/** Stands in for a remote audit write. Deliberately slow. */
static final AtomicInteger auditsInFlight = new AtomicInteger();
static final AtomicInteger maxAuditsInFlight = new AtomicInteger();
static void audit(String message) {
maxAuditsInFlight.accumulateAndGet(auditsInFlight.incrementAndGet(), Math::max);
try { Thread.sleep(20); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
auditsInFlight.decrementAndGet();
}
/** Counts transfers that actually moved money, as opposed to those reported as applied. */
static final AtomicInteger applied = new AtomicInteger();
/* ── the service under review ───────────────────────────────────────── */
static final class TransferService {
/** Returns true if the transfer was applied. */
boolean transfer(Account from, Account to, long amount) {
try {
if (!from.lock.tryLock(30, TimeUnit.MILLISECONDS)) return true;
try {
if (!to.lock.tryLock(30, TimeUnit.MILLISECONDS)) return true;
try {
from.balance -= amount;
to.balance += amount;
applied.incrementAndGet();
audit("moved " + amount + " from " + from.id + " to " + to.id);
return true;
} finally { to.lock.unlock(); }
} finally { from.lock.unlock(); }
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
return false;
}
}
}
/* ── checks ─────────────────────────────────────────────────────────── */
static final int ROUNDS = 40;
public static void main(String[] args) throws Exception {
List<String> failures = new ArrayList<>();
var service = new TransferService();
var a = new Account("account-A", 10_000);
var b = new Account("account-B", 10_000);
long before = a.balance + b.balance;
var pool = Executors.newFixedThreadPool(8);
var reportedOk = new AtomicInteger();
var futures = new ArrayList<Future<?>>();
for (int i = 0; i < ROUNDS; i++) {
boolean forward = i % 2 == 0;
futures.add(pool.submit(() -> {
if (service.transfer(forward ? a : b, forward ? b : a, 10)) reportedOk.incrementAndGet();
}));
}
pool.shutdown();
boolean finished = pool.awaitTermination(30, TimeUnit.SECONDS);
for (Future<?> f : futures) f.get();
// 1. Nothing may be reported as applied unless it moved money.
if (reportedOk.get() != applied.get())
failures.add("1. " + reportedOk.get() + " transfers reported success but only "
+ applied.get() + " moved money — a failed lock acquisition is being "
+ "reported as a completed transfer");
// 2. The ledger must balance regardless of how many transfers succeeded.
long after = a.balance + b.balance;
if (after != before)
failures.add("2. ledger does not balance: started with " + before + ", ended with " + after);
// 3. Equal numbers of transfers each way must return both balances exactly.
if (!finished)
failures.add("3. transfers did not finish within 30s");
else if (applied.get() != ROUNDS)
failures.add("3. only " + applied.get() + " of " + ROUNDS + " transfers were applied — "
+ "opposite-direction transfers are contending for locks in opposite order");
else if (a.balance != 10_000 || b.balance != 10_000)
failures.add("3. equal transfers both ways left " + a.balance + " and " + b.balance);
// 4. The slow audit call must not be holding the account locks.
if (maxAuditsInFlight.get() < 2)
failures.add("4. audits never overlapped (max " + maxAuditsInFlight.get()
+ " at a time) across " + ROUNDS + " transfers — the slow call is inside "
+ "the lock, so every transfer on these accounts waits for it");
/* ── report ─────────────────────────────────────────────────────── */
if (failures.isEmpty()) {
System.out.println("PASS");
} else {
failures.forEach(f -> System.out.println(" " + f));
System.out.println("FAIL");
}
}
}Run it locally:
cd exercises/java/concurrency/deadlock/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out StarterHints
Hint 1
Check 1 is the one to fix first — not because it is worst, but because until failures are reported honestly you cannot see what the other two are doing.
Hint 2
For the ordering, ask what both directions could agree on. It has to be stable for the object's lifetime and unique across accounts.
Hint 3
Check 4 measures how many audits overlap. Ask what has to be true of the critical section for two transfers on the same pair to audit at once.
Hint 4
After moving the audit out, ask whether it can still be attributed correctly if the transfer failed. Where exactly should it go?
Done when
- Locks are acquired in a consistent order derived from a stable key
- A failed acquisition returns false, and the caller's count matches reality
- The audit call happens outside both locks, and audits overlap
- All forty transfers apply and both balances return to their starting values
- A comment says what this bug would look like under synchronized instead of tryLock
Solution
Show the solution — try it yourself first
Solution.java
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;
import java.util.concurrent.locks.ReentrantLock;
/**
* Solution: the transfer service that stalled at month end.
*
* Three defects, and they compound. The lock order caused contention, the slow
* call under the lock made every contended wait long enough to time out, and
* the swallowed timeout meant the failures were reported as successes — so the
* ledger drifted and nothing in the logs said why.
*/
public class Solution {
static final class Account {
final String id;
final ReentrantLock lock = new ReentrantLock();
volatile long balance;
Account(String id, long balance) { this.id = id; this.balance = balance; }
}
static final AtomicInteger auditsInFlight = new AtomicInteger();
static final AtomicInteger maxAuditsInFlight = new AtomicInteger();
static void audit(String message) {
maxAuditsInFlight.accumulateAndGet(auditsInFlight.incrementAndGet(), Math::max);
try { Thread.sleep(20); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
auditsInFlight.decrementAndGet();
}
static final AtomicInteger applied = new AtomicInteger();
/* ── the service, fixed ─────────────────────────────────────────────── */
static final class TransferService {
boolean transfer(Account from, Account to, long amount) {
/*
* Defect 1. Locks were taken in argument order, so transfer(a, b)
* and transfer(b, a) reached for them in opposite orders. Under
* plain synchronized that is a hard deadlock; with tryLock it is a
* storm of timeouts, which is what month end looked like.
*
* Ordering by a stable, total key means both directions take the
* same lock first, so a cycle cannot form. The account id is the
* right key: it is unique and it does not change. identityHashCode
* would not be — it is not guaranteed unique, and a collision puts
* you back where you started.
*/
Account first = from.id.compareTo(to.id) < 0 ? from : to;
Account second = first == from ? to : from;
try {
if (!first.lock.tryLock(30, TimeUnit.MILLISECONDS)) return false;
try {
/*
* Defect 2. `return true` on a failed acquisition reported
* a transfer that never happened. The caller counted it as
* done, so money silently failed to move and the only
* evidence was a ledger that stopped balancing.
*
* A lock you could not take is a failure. Say so and let
* the caller decide whether to retry.
*/
if (!second.lock.tryLock(30, TimeUnit.MILLISECONDS)) return false;
try {
from.balance -= amount;
to.balance += amount;
applied.incrementAndGet();
} finally { second.lock.unlock(); }
} finally { first.lock.unlock(); }
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
return false;
}
/*
* Defect 3. The audit was a slow remote write performed while both
* account locks were held, so every transfer touching these
* accounts queued behind it and the 30ms acquisition timeout was
* guaranteed to expire. Moving it outside the locks shortens the
* critical section to a few field writes.
*
* This is the rule worth generalising: never hold a lock across a
* call you do not control. It causes more outages than deadlock.
*/
audit("moved " + amount + " from " + from.id + " to " + to.id);
return true;
}
}
/* ── checks (unchanged from the starter) ────────────────────────────── */
static final int ROUNDS = 40;
public static void main(String[] args) throws Exception {
List<String> failures = new ArrayList<>();
var service = new TransferService();
var a = new Account("account-A", 10_000);
var b = new Account("account-B", 10_000);
long before = a.balance + b.balance;
var pool = Executors.newFixedThreadPool(8);
var reportedOk = new AtomicInteger();
var futures = new ArrayList<Future<?>>();
for (int i = 0; i < ROUNDS; i++) {
boolean forward = i % 2 == 0;
futures.add(pool.submit(() -> {
if (service.transfer(forward ? a : b, forward ? b : a, 10)) reportedOk.incrementAndGet();
}));
}
pool.shutdown();
boolean finished = pool.awaitTermination(30, TimeUnit.SECONDS);
for (Future<?> f : futures) f.get();
if (reportedOk.get() != applied.get())
failures.add("1. " + reportedOk.get() + " transfers reported success but only "
+ applied.get() + " moved money — a failed lock acquisition is being "
+ "reported as a completed transfer");
long after = a.balance + b.balance;
if (after != before)
failures.add("2. ledger does not balance: started with " + before + ", ended with " + after);
if (!finished)
failures.add("3. transfers did not finish within 30s");
else if (applied.get() != ROUNDS)
failures.add("3. only " + applied.get() + " of " + ROUNDS + " transfers were applied — "
+ "opposite-direction transfers are contending for locks in opposite order");
else if (a.balance != 10_000 || b.balance != 10_000)
failures.add("3. equal transfers both ways left " + a.balance + " and " + b.balance);
if (maxAuditsInFlight.get() < 2)
failures.add("4. audits never overlapped (max " + maxAuditsInFlight.get()
+ " at a time) across " + ROUNDS + " transfers — the slow call is inside "
+ "the lock, so every transfer on these accounts waits for it");
if (failures.isEmpty()) {
System.out.println("PASS");
} else {
failures.forEach(f -> System.out.println(" " + f));
System.out.println("FAIL");
}
}
}Stretch
The fixed version still returns false when it cannot acquire in 30ms, and
the caller drops that transfer. Decide what should really happen: retry with
backoff and jitter, queue it, or surface it. Then explain why retrying in a
tight loop would turn this into a livelock, and what in your monitoring
would distinguish that from the deadlock you started with.