Production incident

The refund that debited twice

45 minintermediate28 yrs

A real incident: symptom first, cause hidden, tradeoff at the end.

The incident

A refund endpoint reverses a payment and writes a ledger entry. It is annotated @Transactional, and the team is confident the two writes are atomic. Finance reports that a small number of refunds have a ledger entry with no matching reversal, and a smaller number have the reversal with no ledger entry. Both should be impossible. What the team found: 1. The endpoint calls refundAll() for a batch, which loops and calls refundOne() — and refundOne() is the method carrying @Transactional. 2. The ledger writer throws a checked LedgerException when the ledger service is unreachable. Nothing rolls back when it does. 3. Adding more @Transactional annotations did not help. 4. It reproduces reliably in a test, but only when the batch has more than one item. Find the root cause — there are two, and they compound. Fix both, and then answer the design question: what would make this class of bug impossible rather than fixed?

What this teaches

  • A self-invoked annotated method is never advised, and nothing warns you
  • Spring's default commits on checked exceptions
  • Two independent proxy defects can produce two different corruptions
  • Moving the method to another bean is the fix, not another annotation
  • Annotation-driven behaviour fails silently by nature

Starter

Starter.java
import java.lang.reflect.*;
import java.util.*;

/**
 * Incident reproduction: the refund endpoint.
 *
 * Spring is not on the classpath and is not needed — @Transactional is a proxy,
 * and the proxy is reproduced faithfully here, including Spring's default
 * rollback rule.
 */
public class Starter {

    static class LedgerException extends Exception {
        LedgerException(String m) { super(m); }
    }

    /** The database, so the test can assert what actually persisted. */
    static final List<String> reversals = new ArrayList<>();
    static final List<String> ledger = new ArrayList<>();
    static final List<String> committed = new ArrayList<>();

    interface RefundService {
        void refundAll(List<String> ids) throws Exception;
        void refundOne(String id) throws Exception;
    }

    static class RealRefundService implements RefundService {
        @Override
        public void refundAll(List<String> ids) throws Exception {
            for (String id : ids) {
                try {
                    refundOne(id);              // self-invocation
                } catch (Exception e) {
                    System.out.println("    batch continued past " + id);
                }
            }
        }

        @Override
        public void refundOne(String id) throws Exception {
            reversals.add(id);
            System.out.println("    reversed " + id);
            if (id.equals("R2")) {
                throw new LedgerException("ledger service unreachable for " + id);
            }
            ledger.add(id);
            System.out.println("    ledgered " + id);
        }
    }

    /** Spring's proxy, including the default rollback rule. */
    static RefundService transactional(RefundService target) {
        return (RefundService) Proxy.newProxyInstance(
            RefundService.class.getClassLoader(),
            new Class<?>[] { RefundService.class },
            (proxy, method, args) -> {
                System.out.println("  BEGIN   " + method.getName());
                try {
                    Object r = method.invoke(target, args);
                    committed.add(method.getName());
                    System.out.println("  COMMIT  " + method.getName());
                    return r;
                } catch (InvocationTargetException e) {
                    Throwable cause = e.getCause();
                    if (cause instanceof RuntimeException || cause instanceof Error) {
                        System.out.println("  ROLLBACK " + method.getName());
                        undo();
                    } else {
                        committed.add(method.getName());
                        System.out.println("  COMMIT  " + method.getName()
                            + " — checked exception, Spring commits");
                    }
                    throw cause;
                }
            });
    }

    /** Crude rollback: drop anything written since the last commit boundary. */
    static void undo() {
        if (!reversals.isEmpty()) reversals.remove(reversals.size() - 1);
    }

    public static void main(String[] args) throws Exception {
        RefundService service = transactional(new RealRefundService());

        List<String> batch = List.of("R1", "R2", "R3");
        System.out.println("refunding batch of " + batch.size() + ":");
        service.refundAll(batch);

        System.out.println();
        System.out.println("transactions opened = " + committed.size());
        System.out.println("reversals written   = " + reversals);
        System.out.println("ledger entries      = " + ledger);

        // Every refund must be atomic: both writes, or neither.
        Set<String> both = new TreeSet<>(reversals);
        both.retainAll(ledger);
        Set<String> orphaned = new TreeSet<>(reversals);
        orphaned.removeAll(ledger);

        System.out.println("consistent refunds  = " + both);
        System.out.println("orphaned reversals  = " + orphaned);

        boolean perRefundTx = committed.size() >= batch.size();
        boolean noOrphans = orphaned.isEmpty();
        boolean othersSurvived = both.contains("R1") && both.contains("R3");

        System.out.println();
        System.out.println("one transaction per refund : " + perRefundTx);
        System.out.println("no orphaned reversals      : " + noOrphans);
        System.out.println("R1 and R3 still refunded   : " + othersSurvived);
        System.out.println(perRefundTx && noOrphans && othersSurvived ? "PASS" : "FAIL");
    }
}

Run it locally:

cd exercises/java/spring-data/transactional-internals/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out Starter

Hints

  1. Hint 1

    Which object is refundOne called on inside refundAll? Not the proxy.

  2. Hint 2

    Count the BEGIN lines in the output versus the number of items.

  3. Hint 3

    LedgerException extends Exception, not RuntimeException. What does Spring do with that by default?

  4. Hint 4

    Adding @Transactional to refundAll changes the picture but does not fix the inner calls. Why not?

Done when

  • Every refund is individually atomic — reversal and ledger both, or neither
  • A checked LedgerException rolls its refund back
  • One failing item does not roll back the others
  • A comment names the structural change that prevents this recurring

Solution

Show the solution — try it yourself first
Solution.java
import java.lang.reflect.*;
import java.util.*;

/**
 * Two root causes, compounding.
 *
 * CAUSE 1 — self-invocation. refundAll() called refundOne() on `this`, so the
 * call never crossed the proxy and @Transactional on refundOne was inert. The
 * only transaction opened was refundAll's, which means the whole batch was one
 * unit rather than one unit per refund. That is why it only reproduced with
 * more than one item: with a single refund the outer transaction happens to
 * give the same boundary.
 *
 * CAUSE 2 — Spring's default rollback rule. LedgerException extends Exception,
 * so it is checked, and Spring commits on checked exceptions. R2's reversal was
 * written, the ledger write threw, and the reversal committed anyway — the
 * orphaned row finance reported.
 *
 * Why adding more annotations did not help: the second @Transactional was on a
 * method still being self-invoked. The annotation was never the missing piece;
 * the proxy boundary was.
 *
 * FIX 1 — move refundOne into its own bean and call it through the injected
 * reference. The call now crosses that bean's proxy, so each refund gets its
 * own transaction and one failure cannot take the others with it.
 *
 * FIX 2 — rollbackFor = Exception.class, reproduced here as the proxy rolling
 * back on any Throwable. LedgerException now rolls its refund back, so a
 * reversal is never left without a ledger entry.
 *
 * What makes this impossible rather than fixed:
 *
 *   1. Keep the transactional method on a different bean from its caller. Once
 *      the boundary is a bean boundary, self-invocation cannot happen by
 *      accident — it would require injecting yourself.
 *   2. Do not throw checked exceptions from transactional code, or set
 *      rollbackFor once on a shared meta-annotation so nobody has to remember.
 *   3. Assert on transaction count in a test. Both defects were invisible to
 *      code review and obvious the moment you count BEGIN lines.
 *
 * The general lesson: annotation-driven behaviour fails silently by design.
 * Nothing warns you that an annotation did not apply, so the only defence is a
 * test that observes the behaviour rather than the annotation.
 */
public class Solution {

    static class LedgerException extends Exception {
        LedgerException(String m) { super(m); }
    }

    static final List<String> reversals = new ArrayList<>();
    static final List<String> ledger = new ArrayList<>();
    static final List<String> committed = new ArrayList<>();

    /** FIX 1: the transactional unit lives on its own bean. */
    interface SingleRefundService {
        void refundOne(String id) throws Exception;
    }

    static class RealSingleRefundService implements SingleRefundService {
        @Override
        public void refundOne(String id) throws Exception {
            reversals.add(id);
            System.out.println("    reversed " + id);
            if (id.equals("R2")) {
                throw new LedgerException("ledger service unreachable for " + id);
            }
            ledger.add(id);
            System.out.println("    ledgered " + id);
        }
    }

    /** The orchestrator is deliberately NOT transactional. */
    static class BatchRefundService {
        private final SingleRefundService single;   // the proxy, injected

        BatchRefundService(SingleRefundService single) {
            this.single = single;
        }

        void refundAll(List<String> ids) {
            for (String id : ids) {
                try {
                    single.refundOne(id);          // crosses the proxy
                } catch (Exception e) {
                    System.out.println("    batch continued past " + id);
                }
            }
        }
    }

    /**
     * FIX 2: rollbackFor = Exception.class — roll back on anything thrown,
     * not only unchecked.
     */
    static SingleRefundService transactional(SingleRefundService target) {
        return (SingleRefundService) Proxy.newProxyInstance(
            SingleRefundService.class.getClassLoader(),
            new Class<?>[] { SingleRefundService.class },
            (proxy, method, args) -> {
                System.out.println("  BEGIN   " + method.getName());
                int reversalMark = reversals.size();
                int ledgerMark = ledger.size();
                try {
                    Object r = method.invoke(target, args);
                    committed.add(method.getName());
                    System.out.println("  COMMIT  " + method.getName());
                    return r;
                } catch (InvocationTargetException e) {
                    Throwable cause = e.getCause();
                    System.out.println("  ROLLBACK " + method.getName()
                        + " — " + cause.getClass().getSimpleName());
                    while (reversals.size() > reversalMark) reversals.remove(reversals.size() - 1);
                    while (ledger.size() > ledgerMark) ledger.remove(ledger.size() - 1);
                    throw cause;
                }
            });
    }

    public static void main(String[] args) {
        SingleRefundService single = transactional(new RealSingleRefundService());
        BatchRefundService batchService = new BatchRefundService(single);

        List<String> batch = List.of("R1", "R2", "R3");
        System.out.println("refunding batch of " + batch.size() + ":");
        batchService.refundAll(batch);

        System.out.println();
        System.out.println("transactions opened = " + committed.size());
        System.out.println("reversals written   = " + reversals);
        System.out.println("ledger entries      = " + ledger);

        Set<String> both = new TreeSet<>(reversals);
        both.retainAll(ledger);
        Set<String> orphaned = new TreeSet<>(reversals);
        orphaned.removeAll(ledger);

        System.out.println("consistent refunds  = " + both);
        System.out.println("orphaned reversals  = " + orphaned);

        // Two committed here plus the rolled-back one means a transaction was
        // opened per refund, which is the property that was missing.
        boolean perRefundTx = committed.size() == batch.size() - 1;
        boolean noOrphans = orphaned.isEmpty();
        boolean othersSurvived = both.contains("R1") && both.contains("R3");

        System.out.println();
        System.out.println("one transaction per refund : " + perRefundTx);
        System.out.println("no orphaned reversals      : " + noOrphans);
        System.out.println("R1 and R3 still refunded   : " + othersSurvived);
        System.out.println(perRefundTx && noOrphans && othersSurvived ? "PASS" : "FAIL");
    }
}

Stretch

The current design makes each refund independent. Argue the opposite case: when should the whole batch be one transaction instead? Then say what REQUIRES_NEW would do here and why a connection pool of one would deadlock.

← Back to How does @Transactional actually work, and when does it silently do nothing?