Production incident

The report that slowed down everything else

45 minintermediate312 yrs

A real incident: symptom first, cause hidden, tradeoff at the end.

The incident

A nightly report was made faster by adding .parallelStream() in three places. It did get faster. Then three things started happening: 1. Unrelated endpoints time out while the report runs. Not slow — timing out, and only during that window. 2. The settled-charge total disagrees with the database, by a different amount every night. 3. The average charge is wrong. Not slightly wrong; wrong in a way that nobody can reproduce on their laptop, and that changed again when the service moved to a bigger container. What the team found: 1. Benchmarking the report alone shows no problem at all. 2. Reverting any one of the three parallelStream calls fixes exactly one symptom. 3. The developer who added them says the code is thread-safe because streams are functional. Three defects, three different mistakes. Fix all three, and be able to say which one would have been caught by a test and which two would not.

What this teaches

  • parallelStream submits to one JVM-wide pool, so it is not a local decision
  • Blocking IO on that pool starves every other parallel stream in the process
  • A shared accumulator in a parallel forEach loses elements and can throw
  • A non-associative reduction changes its answer with the core count
  • Functional-looking code is not automatically thread-safe

Starter

Starter.java
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;
import java.util.stream.*;

/**
 * Production: the report that slowed down everything else.
 *
 * A nightly report was made "faster" by adding .parallelStream() in three
 * places. It did get faster. Then unrelated endpoints started timing out
 * while it ran, the totals stopped matching, and the average was wrong in a
 * way nobody could reproduce on their laptop.
 *
 * Three defects, one per parallelStream. Run it — three failures.
 */
public class Starter {

    record Charge(String id, double amount, boolean settled) { }

    static final List<String> IDS =
        IntStream.range(0, 40).mapToObj(i -> "C-" + i).toList();

    /** Every thread that ran a remote call, so we can see whose pool it was. */
    static final Set<String> fetchThreads = ConcurrentHashMap.newKeySet();

    /** Stands in for the remote call: slow, and not CPU work. */
    static Charge fetch(String id) {
        fetchThreads.add(Thread.currentThread().getName());
        try {
            Thread.sleep(5);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
        int n = Integer.parseInt(id.substring(2));
        return new Charge(id, 10.0 + n, n % 3 != 0);
    }

    /** DEFECT 1: blocking IO on the JVM-wide common pool. */
    static List<Charge> fetchAll(List<String> ids) {
        return ids.parallelStream().map(Starter::fetch).toList();
    }

    /** DEFECT 2: a shared mutable accumulator in a parallel forEach. */
    static List<Charge> settledOnly(List<Charge> charges) {
        List<Charge> out = new ArrayList<>();
        charges.parallelStream().filter(Charge::settled).forEach(c -> {
            try {
                out.add(c);
            } catch (Throwable ignored) {
                // an ArrayList resized by two threads at once can throw
            }
        });
        return out;
    }

    /** DEFECT 3: pairwise averaging is not associative — or even correct. */
    static double averageAmount(List<Charge> charges) {
        return charges.parallelStream()
            .mapToDouble(Charge::amount)
            .reduce(0.0, (a, b) -> (a + b) / 2);
    }

    public static void main(String[] args) {
        boolean ok = true;

        List<Charge> all = fetchAll(IDS);
        List<Charge> settled = settledOnly(all);
        long expectedSettled = all.stream().filter(Charge::settled).count();
        double expectedAverage = all.stream().mapToDouble(Charge::amount).average().orElseThrow();
        double average = averageAmount(all);

        System.out.println("── the nightly report ──");
        System.out.println("  fetched        : " + all.size() + " of " + IDS.size());
        System.out.println("  fetch threads  : " + describe(fetchThreads));
        System.out.println("  settled found  : " + settled.size() + " of " + expectedSettled);
        System.out.printf ("  average        : %.4f (correct: %.4f)%n", average, expectedAverage);
        System.out.println();

        ok &= check("no blocking call ran on the shared common pool",
            fetchThreads.stream().noneMatch(n -> n.startsWith("ForkJoinPool.commonPool")));
        ok &= check("every settled charge was collected", settled.size() == expectedSettled);
        ok &= check("the average is correct",
            Math.abs(average - expectedAverage) < 0.0001);

        System.out.println();
        System.out.println(ok ? "PASS" : "FAIL");
    }

    /** Thread names carry numbers, so summarise rather than print them. */
    static String describe(Set<String> names) {
        boolean common = names.stream().anyMatch(n -> n.startsWith("ForkJoinPool.commonPool"));
        boolean caller = names.contains("main");
        boolean own = names.stream().anyMatch(n -> n.startsWith("pool-"));
        return (common ? "common pool" : own ? "our own pool" : "unknown")
            + (caller ? " + the calling thread" : "");
    }

    static boolean check(String what, boolean passed) {
        System.out.println((passed ? "  ok    " : "  FAIL  ") + what);
        return passed;
    }
}

Run it locally:

cd exercises/java/java8/parallel-stream-pitfalls/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out Starter

Hints

  1. Hint 1

    Print the thread names inside the remote call. The prefix names the pool and the pool is shared with everything.

  2. Hint 2

    For the totals, ask what two threads calling add on one ArrayList do.

  3. Hint 3

    For the average, compute the pairwise version by hand for four numbers. Is it even right sequentially?

  4. Hint 4

    The fix for the first is not a faster pool — it is a pool this job owns, sized for the remote service rather than for the CPU.

Done when

  • No blocking call runs on the common ForkJoinPool
  • Every settled charge is collected, every run
  • The average matches the correct value
  • A comment says which defect a unit test would have caught, and why the other two need a different kind of check

Solution

Show the solution — try it yourself first
Solution.java
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;
import java.util.stream.*;

/**
 * Solution: the report that slowed down everything else.
 *
 * Three defects, one per parallelStream, and they are three different
 * mistakes rather than three instances of one.
 *
 *   Defect 1 — blocking IO on the common ForkJoinPool. parallelStream does
 *   not create threads; it submits to a JVM-wide singleton whose parallelism
 *   is cores minus one. Forty calls sleeping for 5ms each occupy every worker,
 *   and any other parallel stream in the process — including inside a library
 *   — queues behind them. That is why unrelated endpoints timed out while a
 *   report ran, and why no benchmark of the report alone showed it.
 *
 *   The fix is a pool this job owns, sized for the remote service rather than
 *   for the CPU. The concurrency limit becomes a decision instead of a side
 *   effect of the core count, and nothing else in the JVM is affected. On 21
 *   a virtual-thread executor is the better answer again, since the work is
 *   waiting rather than computing.
 *
 *   Defect 2 — a shared ArrayList in a parallel forEach. Elements are lost and
 *   the list can throw from inside add. Collectors exist for this: each worker
 *   accumulates into its own container and the results are merged, which is
 *   both correct and faster than contending on one list.
 *
 *   Defect 3 — pairwise averaging. It is not associative, so in parallel the
 *   answer depends on how the stream was split and therefore on the core
 *   count. It is also simply wrong sequentially, weighting later elements far
 *   more heavily. average() is correct, and its parallel form is correct too
 *   because summing IS associative.
 */
public class Solution {

    record Charge(String id, double amount, boolean settled) { }

    static final List<String> IDS =
        IntStream.range(0, 40).mapToObj(i -> "C-" + i).toList();

    static final Set<String> fetchThreads = ConcurrentHashMap.newKeySet();

    static Charge fetch(String id) {
        fetchThreads.add(Thread.currentThread().getName());
        try {
            Thread.sleep(5);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
        int n = Integer.parseInt(id.substring(2));
        return new Charge(id, 10.0 + n, n % 3 != 0);
    }

    /** FIX 1: a pool this job owns, sized for the remote service. */
    static List<Charge> fetchAll(List<String> ids) throws Exception {
        ExecutorService io = Executors.newFixedThreadPool(8);
        try {
            List<Future<Charge>> futures = ids.stream()
                .map(id -> io.submit(() -> fetch(id)))
                .toList();
            List<Charge> out = new ArrayList<>(futures.size());
            for (Future<Charge> f : futures) out.add(f.get());
            return List.copyOf(out);
        } finally {
            io.shutdown();
        }
    }

    /** FIX 2: a collector, so each worker accumulates separately. */
    static List<Charge> settledOnly(List<Charge> charges) {
        return charges.parallelStream().filter(Charge::settled).toList();
    }

    /** FIX 3: summing is associative, so this is correct in parallel too. */
    static double averageAmount(List<Charge> charges) {
        return charges.parallelStream().mapToDouble(Charge::amount).average().orElseThrow();
    }

    public static void main(String[] args) throws Exception {
        boolean ok = true;

        List<Charge> all = fetchAll(IDS);
        List<Charge> settled = settledOnly(all);
        long expectedSettled = all.stream().filter(Charge::settled).count();
        double expectedAverage = all.stream().mapToDouble(Charge::amount).average().orElseThrow();
        double average = averageAmount(all);

        System.out.println("── the nightly report ──");
        System.out.println("  fetched        : " + all.size() + " of " + IDS.size());
        System.out.println("  fetch threads  : " + describe(fetchThreads));
        System.out.println("  settled found  : " + settled.size() + " of " + expectedSettled);
        System.out.printf ("  average        : %.4f (correct: %.4f)%n", average, expectedAverage);
        System.out.println();

        ok &= check("no blocking call ran on the shared common pool",
            fetchThreads.stream().noneMatch(n -> n.startsWith("ForkJoinPool.commonPool")));
        ok &= check("every settled charge was collected", settled.size() == expectedSettled);
        ok &= check("the average is correct",
            Math.abs(average - expectedAverage) < 0.0001);

        System.out.println();
        System.out.println(ok ? "PASS" : "FAIL");
    }

    static String describe(Set<String> names) {
        boolean common = names.stream().anyMatch(n -> n.startsWith("ForkJoinPool.commonPool"));
        boolean caller = names.contains("main");
        boolean own = names.stream().anyMatch(n -> n.startsWith("pool-"));
        return (common ? "common pool" : own ? "our own pool" : "unknown")
            + (caller ? " + the calling thread" : "");
    }

    static boolean check(String what, boolean passed) {
        System.out.println((passed ? "  ok    " : "  FAIL  ") + what);
        return passed;
    }
}

Stretch

The fix uses a fixed pool sized 8. Justify that number, and say what you would measure to choose it — then write the virtual-thread version for Java 21 and say why waiting work no longer needs a pool size at all. Finally, argue whether the report should have been parallel in the first place.

← Back to When is parallelStream() a mistake?