Production incident

The retry that never retried

45 minintermediate312 yrs

A real incident: symptom first, cause hidden, tradeoff at the end.

The incident

A payment service charges a batch of orders through a flaky gateway. The gateway times out on the first couple of attempts and then succeeds, so the team added retry advice — the framework equivalent of @Retryable — and considered the problem solved. Support then reports two things that should not happen together: 1. Some batches return no receipts at all, and the caller stores a null. 2. Finance can see money moving for orders that have no receipt. What the team found: 1. The batch method chargeAll() loops and calls charge() itself, and charge() is the method the retry advice was meant to protect. 2. When the attempts run out, the advice logs and returns null rather than rethrowing. Nothing upstream notices. 3. Adding the retry annotation to chargeAll() as well made it worse, not better. 4. It only reproduces when the batch has more than one item. Two defects, and they compound into a third thing nobody predicted: the retry is happening at the wrong granularity, so work that already succeeded is being repeated. Find all three, fix the first two, and answer the design question the third one raises.

What this teaches

  • Retry advice on a self-invoked method never runs
  • Retrying a batch is not the same as retrying its items, and is rarely safe
  • Around-advice that swallows the final failure turns an error into a null
  • The fix is a second bean called through its proxy, not another annotation
  • Only an assertion on observed behaviour catches any of this

Starter

Starter.java
import java.lang.reflect.*;
import java.util.*;

/**
 * Production: the retry that never retried.
 *
 * Spring is not on the classpath and does not need to be — @Retryable is
 * around-advice on a proxy, which is exactly what withRetry() builds here.
 *
 * Run it. It prints FAIL. Two defects, both invisible to code review, and a
 * third consequence that only shows up when you count.
 */
public class Starter {

    /** Fails the first `failFirst` attempts per id, then succeeds. */
    static class Gateway {
        private final Map<String, Integer> attempts = new HashMap<>();
        private final int failFirst;
        int calls = 0;

        Gateway(int failFirst) {
            this.failFirst = failFirst;
        }

        String charge(String id) {
            calls++;
            int n = attempts.merge(id, 1, Integer::sum);
            if (n <= failFirst) throw new IllegalStateException("gateway timeout for " + id);
            System.out.println("      money moved for " + id + " on attempt " + n);
            return "receipt-" + id;
        }
    }

    interface PaymentService {
        String charge(String id);
        List<String> chargeAll(List<String> ids);
    }

    static class RealPaymentService implements PaymentService {
        private final Gateway gateway;

        RealPaymentService(Gateway gateway) {
            this.gateway = gateway;
        }

        @Override
        public String charge(String id) {
            return gateway.charge(id);
        }

        @Override
        public List<String> chargeAll(List<String> ids) {
            List<String> receipts = new ArrayList<>();
            for (String id : ids) receipts.add(charge(id));
            return receipts;
        }
    }

    /** The retry advice. This is the only aspect in the system. */
    static PaymentService withRetry(PaymentService target, int maxAttempts) {
        return (PaymentService) Proxy.newProxyInstance(
            PaymentService.class.getClassLoader(),
            new Class<?>[] { PaymentService.class },
            (proxy, method, args) -> {
                RuntimeException last = null;
                for (int attempt = 1; attempt <= maxAttempts; attempt++) {
                    try {
                        return method.invoke(target, args);
                    } catch (InvocationTargetException e) {
                        last = (RuntimeException) e.getCause();
                        System.out.println("  retry: " + method.getName()
                            + " attempt " + attempt + " failed — " + last.getMessage());
                    }
                }
                return null;
            });
    }

    public static void main(String[] args) {
        boolean ok = true;

        System.out.println("── charging a batch of three, gateway fails twice per id ──");
        Gateway gateway = new Gateway(2);
        PaymentService payments = withRetry(new RealPaymentService(gateway), 3);
        List<String> receipts = payments.chargeAll(List.of("A", "B", "C"));

        System.out.println();
        System.out.println("receipts      : " + receipts);
        System.out.println("gateway calls : " + gateway.calls);
        System.out.println();

        ok &= check("every item produced a receipt",
            receipts != null && receipts.size() == 3 && !receipts.contains(null));
        ok &= check("each item was attempted exactly 3 times — 9 calls, no more",
            gateway.calls == 9);

        System.out.println();
        System.out.println("── a gateway that never recovers ──");
        PaymentService doomed = withRetry(new RealPaymentService(new Gateway(99)), 3);
        boolean threw = false;
        try {
            doomed.charge("Z");
        } catch (RuntimeException e) {
            threw = true;
        }
        ok &= check("an exhausted retry throws instead of returning null", threw);

        System.out.println();
        System.out.println(ok ? "PASS" : "FAIL");
    }

    static boolean check(String what, boolean passed) {
        System.out.println((passed ? "  ok    " : "  FAIL  ") + what);
        return passed;
    }
}

Run it locally:

cd exercises/java/spring-aop/aop-proxies/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out Starter

Hints

  1. Hint 1

    Count the gateway calls. Compare that number to attempts × items.

  2. Hint 2

    Which object does chargeAll call charge on? It is not the proxy.

  3. Hint 3

    The advice catches, remembers the exception, and then never uses it.

  4. Hint 4

    When the retry sits outside the loop, what happens to the items that already succeeded on the previous attempt?

  5. Hint 5

    Splitting the class is the fix. Ask what the two halves are actually for.

Done when

  • Every item is retried individually, and the gateway call count proves it
  • An exhausted retry throws instead of returning null
  • No item is charged more times than the retry policy allows
  • A comment says why retrying a batch is unsafe even when retrying an item is

Solution

Show the solution — try it yourself first
Solution.java
import java.lang.reflect.*;
import java.util.*;

/**
 * Solution: the retry that never retried.
 *
 * Two defects, and neither is fixed by an annotation.
 *
 *   Defect 1 — self-invocation. chargeAll() called charge() on `this`, so the
 *   retry advice never saw the individual charges. The retry that DID run was
 *   the one wrapped around chargeAll itself, which is the wrong granularity:
 *   retrying a batch repeats the items that already succeeded, and this
 *   gateway is not idempotent, so that is duplicate money.
 *
 *   Defect 2 — the advice swallowed the final failure and returned null.
 *   An exhausted retry has to rethrow, or every caller has to null-check a
 *   value the signature says can never be null.
 *
 * The fix for the first is structural: the unit that is safe to repeat lives
 * in its own bean, and the batch calls it through its injected reference —
 * which is the proxy. That is also the honest design. "Charge one payment,
 * retrying transient failures" and "walk a list" are two different jobs, and
 * the proxy limitation only made that obvious.
 */
public class Solution {

    static class Gateway {
        private final Map<String, Integer> attempts = new HashMap<>();
        private final int failFirst;
        int calls = 0;

        Gateway(int failFirst) {
            this.failFirst = failFirst;
        }

        String charge(String id) {
            calls++;
            int n = attempts.merge(id, 1, Integer::sum);
            if (n <= failFirst) throw new IllegalStateException("gateway timeout for " + id);
            System.out.println("      money moved for " + id + " on attempt " + n);
            return "receipt-" + id;
        }
    }

    /** The unit that is safe to repeat, and nothing else. */
    interface ChargeService {
        String charge(String id);
    }

    static class RealChargeService implements ChargeService {
        private final Gateway gateway;

        RealChargeService(Gateway gateway) {
            this.gateway = gateway;
        }

        @Override
        public String charge(String id) {
            return gateway.charge(id);
        }
    }

    /** FIX 2: exhausting the attempts is a failure, so it throws. */
    static ChargeService withRetry(ChargeService target, int maxAttempts) {
        return (ChargeService) Proxy.newProxyInstance(
            ChargeService.class.getClassLoader(),
            new Class<?>[] { ChargeService.class },
            (proxy, method, args) -> {
                RuntimeException last = null;
                for (int attempt = 1; attempt <= maxAttempts; attempt++) {
                    try {
                        return method.invoke(target, args);
                    } catch (InvocationTargetException e) {
                        if (!(e.getCause() instanceof RuntimeException re)) throw e.getCause();
                        last = re;
                        System.out.println("  retry: " + method.getName()
                            + " attempt " + attempt + " failed — " + re.getMessage());
                    }
                }
                throw last;
            });
    }

    /**
     * FIX 1: a separate bean. `charges` holds the proxy, so every call crosses
     * the proxy boundary and every item is retried on its own.
     *
     * This class is deliberately not retryable itself. Retrying the batch
     * would re-run items that already succeeded, and a charge is not safe to
     * repeat. Retry belongs at the granularity of the thing that can be
     * repeated safely — never above it.
     */
    static class BatchPaymentService {
        private final ChargeService charges;

        BatchPaymentService(ChargeService charges) {
            this.charges = charges;
        }

        List<String> chargeAll(List<String> ids) {
            List<String> receipts = new ArrayList<>();
            for (String id : ids) receipts.add(charges.charge(id));
            return receipts;
        }
    }

    public static void main(String[] args) {
        boolean ok = true;

        System.out.println("── charging a batch of three, gateway fails twice per id ──");
        Gateway gateway = new Gateway(2);
        ChargeService charges = withRetry(new RealChargeService(gateway), 3);
        BatchPaymentService payments = new BatchPaymentService(charges);
        List<String> receipts = payments.chargeAll(List.of("A", "B", "C"));

        System.out.println();
        System.out.println("receipts      : " + receipts);
        System.out.println("gateway calls : " + gateway.calls);
        System.out.println();

        ok &= check("every item produced a receipt",
            receipts != null && receipts.size() == 3 && !receipts.contains(null));
        ok &= check("each item was attempted exactly 3 times — 9 calls, no more",
            gateway.calls == 9);

        System.out.println();
        System.out.println("── a gateway that never recovers ──");
        ChargeService doomed = withRetry(new RealChargeService(new Gateway(99)), 3);
        boolean threw = false;
        try {
            doomed.charge("Z");
        } catch (RuntimeException e) {
            threw = true;
        }
        ok &= check("an exhausted retry throws instead of returning null", threw);

        System.out.println();
        System.out.println(ok ? "PASS" : "FAIL");
    }

    static boolean check(String what, boolean passed) {
        System.out.println((passed ? "  ok    " : "  FAIL  ") + what);
        return passed;
    }
}

Stretch

The gateway here is not idempotent — every call moves money. Rewrite the charge so that repeating it is safe, then say what the client has to send for that to work and who is responsible for generating it. Then argue whether the retry belongs in this service at all, or one layer down.

← Back to How does Spring AOP work, and what are its limits?