ExerciseProduction incident
Production incident
The incident with no cause in the logs
45 minintermediate3–8 yrs
A real incident: symptom first, cause hidden, tradeoff at the end.
The incident
The nightly reconciliation job started failing. It runs against a partner
SFTP endpoint, downloads a file, and imports it.
What on-call had to work with:
1. The alert says the job failed.
2. The only stack trace in the logs is an IllegalStateException from the
connection pool's close() — "connection already released".
3. That exception's stack trace shows the pool, not the job. Nothing in it
mentions the file, the partner, or the import.
4. Retrying the job by hand sometimes works, which sent the first
investigation toward the network.
Three engineers spent two days on the connection pool. The connection pool was
never the problem.
Find out what the logs are not telling you, fix the code so the next incident
reports the real failure, and then answer the design question: what should the
import layer throw, and why.
What this teaches
- A close() that throws inside finally replaces the in-flight exception
- The replacement destroys the real cause — it is not chained anywhere
- try-with-resources keeps the body's exception primary and suppresses the close failure
- Suppressed exceptions are the correctness reason to use try-with-resources
- Wrapping without passing the cause produces the same blindness one layer up
Starter
Starter.java
import java.io.*;
/**
* Incident reproduction: the reconciliation job that reports the wrong error.
*
* The real failure is a malformed row in the partner file. What on-call sees is
* an IllegalStateException from the connection pool.
*/
public class Starter {
/** Stands in for a pooled SFTP/JDBC connection. */
static final class PooledConnection implements Closeable {
private final String partner;
PooledConnection(String partner) {
this.partner = partner;
}
String fetch(String file) throws IOException {
// The actual failure: the partner sent a truncated file.
throw new IOException("malformed row 4128 in " + file + " from " + partner);
}
/**
* The pool double-releases under load. Annoying, but not the incident —
* and it is what destroys the evidence.
*/
@Override
public void close() {
throw new IllegalStateException("connection already released");
}
}
/** The import step. Note the finally block. */
static String download(String partner, String file) throws IOException {
PooledConnection conn = new PooledConnection(partner);
try {
return conn.fetch(file);
} finally {
conn.close();
}
}
/** One layer up, where the exception is wrapped for the scheduler. */
static void reconcile(String partner, String file) {
try {
download(partner, file);
} catch (Exception e) {
// Wrapped without the cause.
throw new RuntimeException("reconciliation failed for " + partner);
}
}
public static void main(String[] args) {
String partner = "acme-bank";
String file = "settlements-2026-08-24.csv";
Throwable surfaced = null;
try {
reconcile(partner, file);
} catch (Throwable t) {
surfaced = t;
}
System.out.println("what on-call sees");
System.out.println(" exception : " + surfaced.getClass().getSimpleName());
System.out.println(" message : " + surfaced.getMessage());
System.out.println(" cause : " + surfaced.getCause());
Throwable root = surfaced;
while (root.getCause() != null) root = root.getCause();
System.out.println(" root cause : " + root.getClass().getSimpleName()
+ ": " + root.getMessage());
System.out.println(" suppressed : " + root.getSuppressed().length);
// What a usable failure report has to contain.
String all = describe(surfaced);
boolean namesRealCause = all.contains("malformed row");
boolean keepsCloseFailure = all.contains("already released");
System.out.println();
System.out.println("mentions the real cause (malformed row) : " + namesRealCause);
System.out.println("close failure still reachable : " + keepsCloseFailure);
System.out.println(namesRealCause && keepsCloseFailure ? "PASS" : "FAIL");
}
/** Flattens an exception chain, including suppressed exceptions, to text. */
static String describe(Throwable t) {
StringBuilder sb = new StringBuilder();
for (Throwable c = t; c != null; c = c.getCause()) {
sb.append(c.getClass().getSimpleName()).append(": ").append(c.getMessage()).append('\n');
for (Throwable s : c.getSuppressed()) {
sb.append(" suppressed ").append(s.getClass().getSimpleName())
.append(": ").append(s.getMessage()).append('\n');
}
}
return sb.toString();
}
}Run it locally:
cd exercises/java/exceptions/checked-vs-unchecked-exceptions/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out StarterHints
Hint 1
Read the finally block and ask what happens if the statement inside it throws while an exception is already propagating.
Hint 2
There is no chaining involved. The second exception does not wrap the first; it replaces it.
Hint 3
try-with-resources does not just close the resource. Look up Throwable.getSuppressed().
Hint 4
There is a second instance of the same mistake in this file, one layer up, where an exception is wrapped.
Done when
- The failure the caller sees identifies the real cause
- The close() failure is still reachable, not discarded
- Every wrap in the file passes the original as the cause
- A comment states what the import layer should throw and why
Solution
Show the solution — try it yourself first
Solution.java
import java.io.*;
/**
* Root cause: two independent instances of "the real exception was thrown away".
*
* 1. download() closes the connection in a finally block. When close() throws
* while the IOException from fetch() is already propagating, the new
* exception does not wrap the old one — it REPLACES it. The IOException
* naming the malformed row is destroyed at that moment. Nothing logs it,
* nothing chains it, it is simply gone.
*
* That is why the only stack trace in the logs belongs to the connection
* pool: it is the last exception standing, and its stack shows the pool
* because that is where it was created. The pool was never the problem — it
* was the messenger, and it overwrote the message.
*
* try-with-resources fixes exactly this. The body's exception stays primary
* and the close() failure is attached via addSuppressed(), so both survive.
* This is the correctness reason to use it. Closing the resource for you is
* the convenience reason.
*
* 2. reconcile() catches and rethrows without passing the cause. Even with
* fix #1 in place, that second bug would still discard everything below it.
* Both had to be fixed; fixing either one alone leaves on-call blind.
*
* Why manual retries sometimes worked: the pool only double-releases under
* concurrent load. Run the job by hand off-peak and close() succeeds, so the
* real IOException survives and the job fails with a readable error — or the
* partner has since re-sent a valid file and it passes outright. Either way the
* evidence pointed away from the actual defect.
*
* What the import layer should throw: an unchecked, domain-specific exception —
* ReconciliationFailed — carrying the partner, the file and the cause. Unchecked
* because no caller up this stack can recover from a malformed partner file;
* the only sane responses are alert and retry, and forcing every intermediate
* layer to declare `throws` buys nothing. Domain-specific because the scheduler
* should not need to know that this particular partner is reached over SFTP.
* This is the same reasoning Spring applies when it wraps SQLException into
* unchecked DataAccessException.
*/
public class Solution {
static final class PooledConnection implements Closeable {
private final String partner;
PooledConnection(String partner) {
this.partner = partner;
}
String fetch(String file) throws IOException {
throw new IOException("malformed row 4128 in " + file + " from " + partner);
}
@Override
public void close() {
throw new IllegalStateException("connection already released");
}
}
/** Unchecked and domain-specific, carrying context as fields. */
static final class ReconciliationFailed extends RuntimeException {
private final String partner;
private final String file;
ReconciliationFailed(String partner, String file, Throwable cause) {
super("reconciliation failed for " + partner + " file " + file, cause);
this.partner = partner;
this.file = file;
}
String partner() {
return partner;
}
String file() {
return file;
}
}
/**
* FIX 1: try-with-resources. The IOException from fetch() stays primary and
* the close() failure rides along as a suppressed exception.
*/
static String download(String partner, String file) throws IOException {
try (PooledConnection conn = new PooledConnection(partner)) {
return conn.fetch(file);
}
}
/** FIX 2: wrap WITH the cause, and add the context on-call needs. */
static void reconcile(String partner, String file) {
try {
download(partner, file);
} catch (IOException e) {
throw new ReconciliationFailed(partner, file, e);
}
}
public static void main(String[] args) {
String partner = "acme-bank";
String file = "settlements-2026-08-24.csv";
Throwable surfaced = null;
try {
reconcile(partner, file);
} catch (Throwable t) {
surfaced = t;
}
System.out.println("what on-call sees");
System.out.println(" exception : " + surfaced.getClass().getSimpleName());
System.out.println(" message : " + surfaced.getMessage());
Throwable root = surfaced;
while (root.getCause() != null) root = root.getCause();
System.out.println(" root cause : " + root.getClass().getSimpleName()
+ ": " + root.getMessage());
System.out.println(" suppressed : " + root.getSuppressed().length);
for (Throwable s : root.getSuppressed()) {
System.out.println(" " + s.getClass().getSimpleName() + ": " + s.getMessage());
}
String all = describe(surfaced);
boolean namesRealCause = all.contains("malformed row");
boolean keepsCloseFailure = all.contains("already released");
System.out.println();
System.out.println("mentions the real cause (malformed row) : " + namesRealCause);
System.out.println("close failure still reachable : " + keepsCloseFailure);
System.out.println(namesRealCause && keepsCloseFailure ? "PASS" : "FAIL");
}
static String describe(Throwable t) {
StringBuilder sb = new StringBuilder();
for (Throwable c = t; c != null; c = c.getCause()) {
sb.append(c.getClass().getSimpleName()).append(": ").append(c.getMessage()).append('\n');
for (Throwable s : c.getSuppressed()) {
sb.append(" suppressed ").append(s.getClass().getSimpleName())
.append(": ").append(s.getMessage()).append('\n');
}
}
return sb.toString();
}
}Stretch
The job is retried by hand and sometimes succeeds, which made this look like a
network flake. Add enough context to the thrown exception — partner, file,
attempt number — that the next on-call does not need to reproduce it. Then
decide whether that context belongs in the message or in fields on a custom
exception, and say why.
← Back to What is the difference between checked and unchecked exceptions?