ExerciseProduction incident
Production incident
The auth check that passed every test and failed every user
45 minintermediate2–8 yrs
A real incident: symptom first, cause hidden, tradeoff at the end.
The incident
An internal service authenticates callers against a small allow-list of API
tokens loaded at startup. The check is a tight loop over the allow-list,
written for speed.
Every unit test passes. It passed code review twice. In the staging
environment, hand-tested with tokens pasted from the config file, it works.
In production, every single request is rejected as unauthorised. The tokens
are correct — operators have diffed them character by character against the
config. The service logs no error, because being rejected is not an error.
Then someone "fixed" it by calling intern() on the incoming token, and it
started working. Six weeks later that instance began spending noticeable time
in GC and its memory footprint would not come back down under load.
Find the original root cause. Fix it properly. Then explain why intern() made
the symptom disappear and what it cost.
What this teaches
- == compares identity, and only pooled strings share identity
- Literals in tests are interned, so == passes tests it should fail
- Runtime-built strings (parsed headers, JSON, DB rows) are never interned
- intern() makes == work and is still the wrong fix
- Interning attacker-controlled input is unbounded growth you cannot evict
Starter
Starter.java
import java.util.*;
/**
* Incident reproduction: the auth check that passes tests and rejects users.
*
* The two halves of main() run the SAME check against the SAME token text.
* One is shaped like the unit test. One is shaped like production.
*/
public class Starter {
/** Loaded from config at startup. These are literals in the class file. */
private static final String[] ALLOW_LIST = {
"tok_live_a91f", "tok_live_b7c2", "tok_live_c503"
};
/**
* "Written for speed": no allocation, no method call, just a pointer
* comparison per entry.
*/
static boolean authorised(String token) {
for (String allowed : ALLOW_LIST) {
if (allowed == token) return true;
}
return false;
}
/**
* How the token actually arrives: sliced out of a request header. Built at
* runtime, character by character, like anything from a socket or a parser.
*/
static String fromHeader(String headerValue) {
StringBuilder sb = new StringBuilder();
for (char c : headerValue.substring("Bearer ".length()).toCharArray()) {
sb.append(c);
}
return sb.toString();
}
public static void main(String[] args) {
System.out.println("--- as the unit test calls it (literal token) ---");
int devOk = 0;
for (String t : new String[] { "tok_live_a91f", "tok_live_b7c2", "tok_live_c503" }) {
boolean ok = authorised(t);
devOk += ok ? 1 : 0;
System.out.println(" " + t + " -> " + (ok ? "authorised" : "REJECTED"));
}
System.out.println(" passed " + devOk + "/3");
System.out.println("--- as production calls it (token parsed from a header) ---");
int prodOk = 0;
int requests = 0;
for (String t : ALLOW_LIST) {
String parsed = fromHeader("Bearer " + t);
requests++;
boolean ok = authorised(parsed);
prodOk += ok ? 1 : 0;
System.out.println(" " + parsed + " -> " + (ok ? "authorised" : "REJECTED")
+ " (equals says " + Arrays.asList(ALLOW_LIST).contains(parsed) + ")");
}
System.out.println(" passed " + prodOk + "/" + requests);
System.out.println();
System.out.println("dev authorised = " + devOk + "/3");
System.out.println("prod authorised = " + prodOk + "/" + requests);
System.out.println(devOk == 3 && prodOk == requests ? "PASS" : "FAIL");
}
}Run it locally:
cd exercises/java/strings/string-immutability-and-pool/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out StarterHints
Hint 1
Look at the comparison operator in the loop, then ask where the token in the test came from versus where the production token comes from.
Hint 2
A string parsed out of a header is constructed at runtime. Nothing interns it.
Hint 3
intern() forces the pool lookup, which makes == work again. Ask what the pool now contains after a million distinct requests, and how you would remove an entry from it.
Done when
- Both the dev-shaped and prod-shaped tokens authenticate
- The comparison no longer depends on how the string was constructed
- A comment explains why intern() worked and why it was not shipped
Solution
Show the solution — try it yourself first
Solution.java
import java.util.*;
/**
* Root cause: `allowed == token` compares object identity, not text.
*
* The allow-list entries are string literals, so javac interned them into the
* JVM-wide pool. The unit test also passes literals — the very same pooled
* objects — so == is true and the test passes. Production parses the token out
* of a request header, which builds a brand new String at runtime that was
* never interned. Same characters, different object, == is false, every request
* rejected.
*
* The test could not have caught this, because the test's inputs were literals.
* That is the whole lesson: == on strings is a bug that passes its own tests.
*
* Why intern() "fixed" it: intern() returns the pooled instance for that text,
* so the parsed token becomes the same object as the literal and == starts
* working again. It is still the wrong fix, for two reasons:
*
* 1. It leaves a correctness landmine in place. The next comparison someone
* writes against a non-interned string breaks the same way.
* 2. It interns attacker-controlled input. Every distinct token ever
* presented — including garbage from a scanner — is added to the string
* pool. Before Java 7 that filled PermGen and produced
* "OutOfMemoryError: PermGen space". Since Java 7 the pool is on the heap,
* so instead you get a table that grows without bound, is expensive to
* resize, and has no eviction API. You cannot remove an entry from the
* string pool. That is the GC behaviour they saw six weeks later.
*
* The fix is to compare the text.
*/
public class Solution {
private static final String[] ALLOW_LIST_RAW = {
"tok_live_a91f", "tok_live_b7c2", "tok_live_c503"
};
/**
* A Set does the comparison with equals() and hashCode(), so identity never
* enters into it — and it turns a linear scan into one hash lookup, which
* is what "written for speed" should have meant in the first place.
*/
private static final Set<String> ALLOW_LIST = Set.of(ALLOW_LIST_RAW);
static boolean authorised(String token) {
return token != null && ALLOW_LIST.contains(token);
}
static String fromHeader(String headerValue) {
StringBuilder sb = new StringBuilder();
for (char c : headerValue.substring("Bearer ".length()).toCharArray()) {
sb.append(c);
}
return sb.toString();
}
public static void main(String[] args) {
System.out.println("--- as the unit test calls it (literal token) ---");
int devOk = 0;
for (String t : new String[] { "tok_live_a91f", "tok_live_b7c2", "tok_live_c503" }) {
boolean ok = authorised(t);
devOk += ok ? 1 : 0;
System.out.println(" " + t + " -> " + (ok ? "authorised" : "REJECTED"));
}
System.out.println(" passed " + devOk + "/3");
System.out.println("--- as production calls it (token parsed from a header) ---");
int prodOk = 0;
int requests = 0;
for (String t : ALLOW_LIST_RAW) {
String parsed = fromHeader("Bearer " + t);
requests++;
boolean ok = authorised(parsed);
prodOk += ok ? 1 : 0;
System.out.println(" " + parsed + " -> " + (ok ? "authorised" : "REJECTED")
+ " (identical object? " + (parsed == t) + ")");
}
System.out.println(" passed " + prodOk + "/" + requests);
// The token that must still be rejected.
System.out.println("unknown token = "
+ (authorised("tok_live_dead") ? "authorised" : "REJECTED"));
System.out.println();
System.out.println("dev authorised = " + devOk + "/3");
System.out.println("prod authorised = " + prodOk + "/" + requests);
System.out.println(devOk == 3 && prodOk == requests
&& !authorised("tok_live_dead") ? "PASS" : "FAIL");
}
}Stretch
The allow-list check is also vulnerable to a timing side channel: equals()
returns as soon as two characters differ. Swap in
MessageDigest.isEqual or a constant-time comparison and explain what
attack that closes. Then decide whether it matters for this threat model.
← Back to Why is String immutable, and what is the string pool?