ExerciseProduction incident
Production incident
The refund that debited twice
45 minintermediate2–8 yrs
A real incident: symptom first, cause hidden, tradeoff at the end.
The incident
A refund endpoint reverses a payment and writes a ledger entry. It is
annotated @Transactional, and the team is confident the two writes are atomic.
Finance reports that a small number of refunds have a ledger entry with no
matching reversal, and a smaller number have the reversal with no ledger
entry. Both should be impossible.
What the team found:
1. The endpoint calls refundAll() for a batch, which loops and calls
refundOne() — and refundOne() is the method carrying @Transactional.
2. The ledger writer throws a checked LedgerException when the ledger
service is unreachable. Nothing rolls back when it does.
3. Adding more @Transactional annotations did not help.
4. It reproduces reliably in a test, but only when the batch has more than
one item.
Find the root cause — there are two, and they compound. Fix both, and then
answer the design question: what would make this class of bug impossible
rather than fixed?
What this teaches
- A self-invoked annotated method is never advised, and nothing warns you
- Spring's default commits on checked exceptions
- Two independent proxy defects can produce two different corruptions
- Moving the method to another bean is the fix, not another annotation
- Annotation-driven behaviour fails silently by nature
Starter
Starter.java
import java.lang.reflect.*;
import java.util.*;
/**
* Incident reproduction: the refund endpoint.
*
* Spring is not on the classpath and is not needed — @Transactional is a proxy,
* and the proxy is reproduced faithfully here, including Spring's default
* rollback rule.
*/
public class Starter {
static class LedgerException extends Exception {
LedgerException(String m) { super(m); }
}
/** The database, so the test can assert what actually persisted. */
static final List<String> reversals = new ArrayList<>();
static final List<String> ledger = new ArrayList<>();
static final List<String> committed = new ArrayList<>();
interface RefundService {
void refundAll(List<String> ids) throws Exception;
void refundOne(String id) throws Exception;
}
static class RealRefundService implements RefundService {
@Override
public void refundAll(List<String> ids) throws Exception {
for (String id : ids) {
try {
refundOne(id); // self-invocation
} catch (Exception e) {
System.out.println(" batch continued past " + id);
}
}
}
@Override
public void refundOne(String id) throws Exception {
reversals.add(id);
System.out.println(" reversed " + id);
if (id.equals("R2")) {
throw new LedgerException("ledger service unreachable for " + id);
}
ledger.add(id);
System.out.println(" ledgered " + id);
}
}
/** Spring's proxy, including the default rollback rule. */
static RefundService transactional(RefundService target) {
return (RefundService) Proxy.newProxyInstance(
RefundService.class.getClassLoader(),
new Class<?>[] { RefundService.class },
(proxy, method, args) -> {
System.out.println(" BEGIN " + method.getName());
try {
Object r = method.invoke(target, args);
committed.add(method.getName());
System.out.println(" COMMIT " + method.getName());
return r;
} catch (InvocationTargetException e) {
Throwable cause = e.getCause();
if (cause instanceof RuntimeException || cause instanceof Error) {
System.out.println(" ROLLBACK " + method.getName());
undo();
} else {
committed.add(method.getName());
System.out.println(" COMMIT " + method.getName()
+ " — checked exception, Spring commits");
}
throw cause;
}
});
}
/** Crude rollback: drop anything written since the last commit boundary. */
static void undo() {
if (!reversals.isEmpty()) reversals.remove(reversals.size() - 1);
}
public static void main(String[] args) throws Exception {
RefundService service = transactional(new RealRefundService());
List<String> batch = List.of("R1", "R2", "R3");
System.out.println("refunding batch of " + batch.size() + ":");
service.refundAll(batch);
System.out.println();
System.out.println("transactions opened = " + committed.size());
System.out.println("reversals written = " + reversals);
System.out.println("ledger entries = " + ledger);
// Every refund must be atomic: both writes, or neither.
Set<String> both = new TreeSet<>(reversals);
both.retainAll(ledger);
Set<String> orphaned = new TreeSet<>(reversals);
orphaned.removeAll(ledger);
System.out.println("consistent refunds = " + both);
System.out.println("orphaned reversals = " + orphaned);
boolean perRefundTx = committed.size() >= batch.size();
boolean noOrphans = orphaned.isEmpty();
boolean othersSurvived = both.contains("R1") && both.contains("R3");
System.out.println();
System.out.println("one transaction per refund : " + perRefundTx);
System.out.println("no orphaned reversals : " + noOrphans);
System.out.println("R1 and R3 still refunded : " + othersSurvived);
System.out.println(perRefundTx && noOrphans && othersSurvived ? "PASS" : "FAIL");
}
}Run it locally:
cd exercises/java/spring-data/transactional-internals/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out StarterHints
Hint 1
Which object is refundOne called on inside refundAll? Not the proxy.
Hint 2
Count the BEGIN lines in the output versus the number of items.
Hint 3
LedgerException extends Exception, not RuntimeException. What does Spring do with that by default?
Hint 4
Adding @Transactional to refundAll changes the picture but does not fix the inner calls. Why not?
Done when
- Every refund is individually atomic — reversal and ledger both, or neither
- A checked LedgerException rolls its refund back
- One failing item does not roll back the others
- A comment names the structural change that prevents this recurring
Solution
Show the solution — try it yourself first
Solution.java
import java.lang.reflect.*;
import java.util.*;
/**
* Two root causes, compounding.
*
* CAUSE 1 — self-invocation. refundAll() called refundOne() on `this`, so the
* call never crossed the proxy and @Transactional on refundOne was inert. The
* only transaction opened was refundAll's, which means the whole batch was one
* unit rather than one unit per refund. That is why it only reproduced with
* more than one item: with a single refund the outer transaction happens to
* give the same boundary.
*
* CAUSE 2 — Spring's default rollback rule. LedgerException extends Exception,
* so it is checked, and Spring commits on checked exceptions. R2's reversal was
* written, the ledger write threw, and the reversal committed anyway — the
* orphaned row finance reported.
*
* Why adding more annotations did not help: the second @Transactional was on a
* method still being self-invoked. The annotation was never the missing piece;
* the proxy boundary was.
*
* FIX 1 — move refundOne into its own bean and call it through the injected
* reference. The call now crosses that bean's proxy, so each refund gets its
* own transaction and one failure cannot take the others with it.
*
* FIX 2 — rollbackFor = Exception.class, reproduced here as the proxy rolling
* back on any Throwable. LedgerException now rolls its refund back, so a
* reversal is never left without a ledger entry.
*
* What makes this impossible rather than fixed:
*
* 1. Keep the transactional method on a different bean from its caller. Once
* the boundary is a bean boundary, self-invocation cannot happen by
* accident — it would require injecting yourself.
* 2. Do not throw checked exceptions from transactional code, or set
* rollbackFor once on a shared meta-annotation so nobody has to remember.
* 3. Assert on transaction count in a test. Both defects were invisible to
* code review and obvious the moment you count BEGIN lines.
*
* The general lesson: annotation-driven behaviour fails silently by design.
* Nothing warns you that an annotation did not apply, so the only defence is a
* test that observes the behaviour rather than the annotation.
*/
public class Solution {
static class LedgerException extends Exception {
LedgerException(String m) { super(m); }
}
static final List<String> reversals = new ArrayList<>();
static final List<String> ledger = new ArrayList<>();
static final List<String> committed = new ArrayList<>();
/** FIX 1: the transactional unit lives on its own bean. */
interface SingleRefundService {
void refundOne(String id) throws Exception;
}
static class RealSingleRefundService implements SingleRefundService {
@Override
public void refundOne(String id) throws Exception {
reversals.add(id);
System.out.println(" reversed " + id);
if (id.equals("R2")) {
throw new LedgerException("ledger service unreachable for " + id);
}
ledger.add(id);
System.out.println(" ledgered " + id);
}
}
/** The orchestrator is deliberately NOT transactional. */
static class BatchRefundService {
private final SingleRefundService single; // the proxy, injected
BatchRefundService(SingleRefundService single) {
this.single = single;
}
void refundAll(List<String> ids) {
for (String id : ids) {
try {
single.refundOne(id); // crosses the proxy
} catch (Exception e) {
System.out.println(" batch continued past " + id);
}
}
}
}
/**
* FIX 2: rollbackFor = Exception.class — roll back on anything thrown,
* not only unchecked.
*/
static SingleRefundService transactional(SingleRefundService target) {
return (SingleRefundService) Proxy.newProxyInstance(
SingleRefundService.class.getClassLoader(),
new Class<?>[] { SingleRefundService.class },
(proxy, method, args) -> {
System.out.println(" BEGIN " + method.getName());
int reversalMark = reversals.size();
int ledgerMark = ledger.size();
try {
Object r = method.invoke(target, args);
committed.add(method.getName());
System.out.println(" COMMIT " + method.getName());
return r;
} catch (InvocationTargetException e) {
Throwable cause = e.getCause();
System.out.println(" ROLLBACK " + method.getName()
+ " — " + cause.getClass().getSimpleName());
while (reversals.size() > reversalMark) reversals.remove(reversals.size() - 1);
while (ledger.size() > ledgerMark) ledger.remove(ledger.size() - 1);
throw cause;
}
});
}
public static void main(String[] args) {
SingleRefundService single = transactional(new RealSingleRefundService());
BatchRefundService batchService = new BatchRefundService(single);
List<String> batch = List.of("R1", "R2", "R3");
System.out.println("refunding batch of " + batch.size() + ":");
batchService.refundAll(batch);
System.out.println();
System.out.println("transactions opened = " + committed.size());
System.out.println("reversals written = " + reversals);
System.out.println("ledger entries = " + ledger);
Set<String> both = new TreeSet<>(reversals);
both.retainAll(ledger);
Set<String> orphaned = new TreeSet<>(reversals);
orphaned.removeAll(ledger);
System.out.println("consistent refunds = " + both);
System.out.println("orphaned reversals = " + orphaned);
// Two committed here plus the rolled-back one means a transaction was
// opened per refund, which is the property that was missing.
boolean perRefundTx = committed.size() == batch.size() - 1;
boolean noOrphans = orphaned.isEmpty();
boolean othersSurvived = both.contains("R1") && both.contains("R3");
System.out.println();
System.out.println("one transaction per refund : " + perRefundTx);
System.out.println("no orphaned reversals : " + noOrphans);
System.out.println("R1 and R3 still refunded : " + othersSurvived);
System.out.println(perRefundTx && noOrphans && othersSurvived ? "PASS" : "FAIL");
}
}Stretch
The current design makes each refund independent. Argue the opposite case:
when should the whole batch be one transaction instead? Then say what
REQUIRES_NEW would do here and why a connection pool of one would deadlock.
← Back to How does @Transactional actually work, and when does it silently do nothing?