Production incident

The refund path that failed once a week

45 minintermediate215 yrs

A real incident: symptom first, cause hidden, tradeoff at the end.

The incident

A dependency gate that runs in CI and is supposed to stop version conflicts reaching production. It has been green for months. Last Tuesday a refund threw NoSuchMethodError, and the classpath turned out to have been wrong since March. What the review turned up: 1. Versions are compared with String.compareTo, so 2.9.0 resolves above 2.17.1 and every library past its ninth minor version silently downgrades. 2. The dependencyManagement pins are read into a map and never used — including the ones added after previous incidents. 3. The convergence check returns an empty list, so no build has ever failed on a version disagreement. 4. Exclusions are applied without checking whether anything still needs the artifact, which turns a version conflict into a missing class. Two defects resolve the wrong version and two keep the gate quiet about conflicts it exists to catch. Fix all four.

What this teaches

  • Version strings are ordered numerically per component, never lexicographically
  • A dependencyManagement pin must override the graph, because depth moves and a pin does not
  • Resolving a conflict silently is the default the gate exists to replace
  • An exclusion is a claim about someone else's code, and it should be verified rather than trusted
  • A gate that cannot fail is indistinguishable from no gate

Starter

Starter.java
import java.util.*;
import java.util.stream.*;

/**
 * Production: the refund path that failed once a week.
 *
 * A dependency gate that runs in CI and is supposed to stop version conflicts
 * reaching production. It has been green for months. Last Tuesday a refund
 * threw NoSuchMethodError.
 *
 * Run this. Four checks fail. Fix the gate so all four pass, without
 * weakening the checks.
 */
public class Starter {

    /** One reachable version of one artifact: how deep, and through which path. */
    record Dep(String artifact, String version, int depth, String via) {}

    /** What a path needs from an artifact, so an exclusion can be checked. */
    record Requirement(String artifact, String neededBy) {}

    /* ── the gate under review ──────────────────────────────────────────── */

    /** Compares two version strings. */
    static int compareVersions(String a, String b) {
        return a.compareTo(b);
    }

    /**
     * Picks the version that ends up on the classpath.
     * `managed` holds versions pinned in dependencyManagement.
     */
    static Dep resolve(List<Dep> candidates, Map<String, String> managed) {
        return candidates.stream()
            .max((x, y) -> compareVersions(x.version(), y.version()))
            .orElseThrow();
    }

    /**
     * Reports artifacts where two paths disagree on the version, so the build
     * can fail instead of resolving one silently.
     */
    static List<String> convergenceViolations(List<Dep> all) {
        return List.of();
    }

    /**
     * Applies exclusions, then reports any artifact that is still required by
     * some path but is no longer on the classpath.
     */
    static List<String> applyExclusions(List<Dep> all, Set<String> excluded,
                                        List<Requirement> requirements) {
        all.removeIf(d -> excluded.contains(d.artifact()));
        return List.of();
    }

    /* ── checks ─────────────────────────────────────────────────────────── */

    public static void main(String[] args) {
        List<String> failures = new ArrayList<>();

        // 1. Version ordering must be numeric per component, not lexicographic.
        {
            var candidates = List.of(
                new Dep("jackson-databind", "2.9.0",  2, "legacy-api"),
                new Dep("jackson-databind", "2.17.1", 3, "spring-boot-starter-web"));
            var picked = resolve(new ArrayList<>(candidates), Map.of());
            if (!picked.version().equals("2.17.1"))
                failures.add("1. highest-wins picked " + picked.version()
                           + " over 2.17.1 — versions are being compared as strings");
        }

        // 2. A version pinned in dependencyManagement must win at any depth.
        {
            var candidates = List.of(
                new Dep("commons-lang3", "3.4",  2, "report-service"),
                new Dep("commons-lang3", "3.12", 3, "audit-client -> http-toolkit"));
            var picked = resolve(new ArrayList<>(candidates), Map.of("commons-lang3", "3.8.1"));
            if (!picked.version().equals("3.8.1"))
                failures.add("2. dependencyManagement pinned 3.8.1 and the gate resolved "
                           + picked.version() + " — the pin is being ignored");
        }

        // 3. Disagreement between paths must be reported, not silently resolved.
        {
            var all = new ArrayList<>(List.of(
                new Dep("commons-lang3", "3.4",  2, "report-service"),
                new Dep("commons-lang3", "3.12", 3, "audit-client"),
                new Dep("guava",         "32.1", 2, "billing-client"),
                new Dep("slf4j-api",     "2.0.7",  2, "http-client"),
                new Dep("slf4j-api",     "2.0.13", 3, "metrics-agent")));
            var violations = convergenceViolations(all);
            boolean sawLang = violations.stream().anyMatch(v -> v.contains("commons-lang3"));
            boolean sawSlf4j = violations.stream().anyMatch(v -> v.contains("slf4j-api"));
            boolean sawGuava = violations.stream().anyMatch(v -> v.contains("guava"));
            if (!sawLang || !sawSlf4j)
                failures.add("3. convergence check reported " + violations.size()
                           + " violations; commons-lang3 and slf4j-api both have two versions");
            else if (sawGuava)
                failures.add("3. convergence check flagged guava, which only has one version");
        }

        // 4. An exclusion must not remove an artifact a path still needs.
        {
            var all = new ArrayList<>(List.of(
                new Dep("commons-lang3", "3.4",  2, "report-service"),
                new Dep("commons-lang3", "3.12", 3, "audit-client")));
            var requirements = List.of(
                new Requirement("commons-lang3", "report-service"),
                new Requirement("commons-lang3", "audit-client"));
            var orphaned = applyExclusions(all, Set.of("commons-lang3"), requirements);
            if (orphaned.size() != 2)
                failures.add("4. excluding commons-lang3 left " + orphaned.size()
                           + " unmet requirements reported; two paths still need it");
        }

        /* ── report ─────────────────────────────────────────────────────── */
        if (failures.isEmpty()) {
            System.out.println("PASS");
        } else {
            failures.forEach(f -> System.out.println("  " + f));
            System.out.println("FAIL");
        }
    }
}

Run it locally:

cd exercises/java/dependencies/transitive-conflicts/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out Starter

Hints

  1. Hint 1

    Start with the comparator and write the failing case first: assert that 2.17.1 sorts above 2.9.0, then watch String.compareTo disagree.

  2. Hint 2

    The managed map is a parameter that is never read. Ask what a pin is for — if depth could override it, it would not be worth writing.

  3. Hint 3

    Convergence is not about which version wins. Group by artifact, and any artifact with more than one version is the report.

  4. Hint 4

    For exclusions, the question is not what was removed but what still needed it. The requirements list is right there in the signature.

Done when

  • 2.17.1 resolves above 2.9.0, and your comparator handles 2.17 versus 2.17.0
  • A pinned version wins regardless of depth
  • Artifacts with two versions are reported, and artifacts with one are not
  • Excluding an artifact some path requires is reported as an unmet requirement
  • A comment says which two defects produced a wrong classpath and which two produced silence

Solution

Show the solution — try it yourself first
Solution.java
import java.util.*;
import java.util.stream.*;

/**
 * Solution: the refund path that failed once a week.
 *
 * Four defects. Two of them made the gate resolve the wrong version, and two
 * made it stay quiet about conflicts it was built to catch — which is why it
 * had been green for months while the classpath was wrong the whole time.
 */
public class Solution {

    record Dep(String artifact, String version, int depth, String via) {}
    record Requirement(String artifact, String neededBy) {}

    /* ── the gate, fixed ────────────────────────────────────────────────── */

    /*
     * Defect 1. String.compareTo put 2.9.0 above 2.17.1, because '9' sorts
     * after '1'. Every version that reached double digits resolved backwards,
     * and the gate reported success while downgrading the library.
     *
     * Compare numeric components, padding the shorter one with zeros so
     * 2.17 and 2.17.0 compare equal rather than by length.
     */
    static int compareVersions(String a, String b) {
        List<Integer> x = numericParts(a), y = numericParts(b);
        for (int i = 0; i < Math.max(x.size(), y.size()); i++) {
            int c = Integer.compare(i < x.size() ? x.get(i) : 0, i < y.size() ? y.get(i) : 0);
            if (c != 0) return c;
        }
        return 0;
    }

    /** "2.17.1-RELEASE" -> [2, 17, 1]. Qualifiers are dropped, not compared. */
    static List<Integer> numericParts(String version) {
        var out = new ArrayList<Integer>();
        for (String p : version.split("[.\\-]")) {
            var digits = p.replaceAll("\\D", "");
            if (!digits.isEmpty()) out.add(Integer.parseInt(digits));
        }
        return out;
    }

    /*
     * Defect 2. `managed` was accepted and never read, so every pin written in
     * dependencyManagement did nothing — including the ones added to fix
     * earlier incidents. A pin has to override the graph entirely: that is the
     * whole reason it is the right tool for this, since depth can move when an
     * unrelated dependency is added and a pin cannot.
     */
    static Dep resolve(List<Dep> candidates, Map<String, String> managed) {
        String artifact = candidates.get(0).artifact();
        String pinned = managed.get(artifact);
        if (pinned != null) {
            return new Dep(artifact, pinned, 0, "dependencyManagement");
        }
        return candidates.stream()
            .max((x, y) -> compareVersions(x.version(), y.version()))
            .orElseThrow();
    }

    /*
     * Defect 3. This returned an empty list, so the gate never failed a build.
     * Resolving a conflict silently is the default behaviour it was written to
     * replace — the point of the gate is to make disagreement visible while
     * someone can still choose deliberately.
     */
    static List<String> convergenceViolations(List<Dep> all) {
        Map<String, Set<String>> byArtifact = new TreeMap<>();
        for (Dep d : all)
            byArtifact.computeIfAbsent(d.artifact(), k -> new TreeSet<>()).add(d.version());

        return byArtifact.entrySet().stream()
            .filter(e -> e.getValue().size() > 1)
            .map(e -> {
                String paths = all.stream()
                    .filter(d -> d.artifact().equals(e.getKey()))
                    .map(d -> d.version() + " via " + d.via())
                    .collect(Collectors.joining("; "));
                return e.getKey() + " has " + e.getValue().size() + " versions: " + paths;
            })
            .toList();
    }

    /*
     * Defect 4. Exclusions were applied and nothing checked whether anything
     * still needed the artifact. That is how a version conflict gets "fixed"
     * into a NoClassDefFoundError — a strictly worse failure, because there is
     * no longer a version to compare against.
     *
     * An exclusion is a claim that a path does not need a dependency. This
     * verifies the claim instead of trusting it.
     */
    static List<String> applyExclusions(List<Dep> all, Set<String> excluded,
                                        List<Requirement> requirements) {
        all.removeIf(d -> excluded.contains(d.artifact()));
        Set<String> remaining = all.stream().map(Dep::artifact).collect(Collectors.toSet());

        return requirements.stream()
            .filter(r -> !remaining.contains(r.artifact()))
            .map(r -> r.neededBy() + " still needs " + r.artifact()
                    + ", which an exclusion removed from the classpath")
            .toList();
    }

    /* ── checks (unchanged from the starter) ────────────────────────────── */

    public static void main(String[] args) {
        List<String> failures = new ArrayList<>();

        {
            var candidates = List.of(
                new Dep("jackson-databind", "2.9.0",  2, "legacy-api"),
                new Dep("jackson-databind", "2.17.1", 3, "spring-boot-starter-web"));
            var picked = resolve(new ArrayList<>(candidates), Map.of());
            if (!picked.version().equals("2.17.1"))
                failures.add("1. highest-wins picked " + picked.version()
                           + " over 2.17.1 — versions are being compared as strings");
        }

        {
            var candidates = List.of(
                new Dep("commons-lang3", "3.4",  2, "report-service"),
                new Dep("commons-lang3", "3.12", 3, "audit-client -> http-toolkit"));
            var picked = resolve(new ArrayList<>(candidates), Map.of("commons-lang3", "3.8.1"));
            if (!picked.version().equals("3.8.1"))
                failures.add("2. dependencyManagement pinned 3.8.1 and the gate resolved "
                           + picked.version() + " — the pin is being ignored");
        }

        {
            var all = new ArrayList<>(List.of(
                new Dep("commons-lang3", "3.4",  2, "report-service"),
                new Dep("commons-lang3", "3.12", 3, "audit-client"),
                new Dep("guava",         "32.1", 2, "billing-client"),
                new Dep("slf4j-api",     "2.0.7",  2, "http-client"),
                new Dep("slf4j-api",     "2.0.13", 3, "metrics-agent")));
            var violations = convergenceViolations(all);
            boolean sawLang = violations.stream().anyMatch(v -> v.contains("commons-lang3"));
            boolean sawSlf4j = violations.stream().anyMatch(v -> v.contains("slf4j-api"));
            boolean sawGuava = violations.stream().anyMatch(v -> v.contains("guava"));
            if (!sawLang || !sawSlf4j)
                failures.add("3. convergence check reported " + violations.size()
                           + " violations; commons-lang3 and slf4j-api both have two versions");
            else if (sawGuava)
                failures.add("3. convergence check flagged guava, which only has one version");
        }

        {
            var all = new ArrayList<>(List.of(
                new Dep("commons-lang3", "3.4",  2, "report-service"),
                new Dep("commons-lang3", "3.12", 3, "audit-client")));
            var requirements = List.of(
                new Requirement("commons-lang3", "report-service"),
                new Requirement("commons-lang3", "audit-client"));
            var orphaned = applyExclusions(all, Set.of("commons-lang3"), requirements);
            if (orphaned.size() != 2)
                failures.add("4. excluding commons-lang3 left " + orphaned.size()
                           + " unmet requirements reported; two paths still need it");
        }

        if (failures.isEmpty()) {
            System.out.println("PASS");
        } else {
            failures.forEach(f -> System.out.println("  " + f));
            System.out.println("FAIL");
        }
    }
}

Stretch

The gate reports conflicts. It cannot tell you whether a conflict actually breaks anything — commons-lang 3.4 versus 3.12 is usually harmless, and sometimes is not. Sketch what it would take to answer that: which call sites the application has, which signatures each resolved jar provides, and where those two sets disagree. Then say why the JVM finding this at runtime, one call site at a time, is not the same as finding it in a build.

← Back to Two libraries need different versions of the same dependency. What happens?