ExerciseProduction incident
Production incident
The dedupe job that stopped deduping
45 minintermediate2–8 yrs
A real incident: symptom first, cause hidden, tradeoff at the end.
The incident
Billing ingests invoice rows from two upstream feeds every night. The feeds
overlap by design — the job's whole purpose is to collapse the overlap to one
row per account and period before anything is charged.
It ran correctly for eight months. Then a release went out that added a
`period` field to the invoice key so the same account could be billed in
more than one month.
Since that release:
1. Accounts in the overlap window are being charged twice.
2. The job's own log line says it deduplicated zero rows.
3. No exception, no stack trace, no failed health check. The job reports
success every night.
Finance found it, not monitoring.
Find the root cause, fix it, and then answer the design question: what would
have caught this before the release?
What this teaches
- Overriding equals() without hashCode() silently disables hash-based dedupe
- A HashSet consults the hash first, so equals() is never even called
- The failure is invisible: no exception, and size() reports confidently
- Identity hash codes differ per instance, so every row looks unique
- A record makes the whole class of bug unrepresentable
Starter
Starter.java
import java.util.*;
/**
* Incident reproduction: the nightly billing dedupe.
*
* Two feeds overlap by 2,500 accounts. Collapsing them should leave 7,500
* distinct account+period rows out of 10,000 ingested. It leaves 10,000, so
* 2,500 accounts get billed twice.
*
* Run it. Then find out why, and fix InvoiceKey.
*/
public class Starter {
/**
* The release that broke billing added the `period` field here, and updated
* equals() to compare it.
*/
static final class InvoiceKey {
private final String account;
private final String period;
InvoiceKey(String account, String period) {
this.account = account;
this.period = period;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (!(o instanceof InvoiceKey other)) return false;
return account.equals(other.account) && period.equals(other.period);
}
@Override
public String toString() {
return account + "@" + period;
}
}
private static final String PERIOD = "2026-08";
/** Feed A covers ACC0..ACC4999, feed B covers ACC2500..ACC7499. */
static List<InvoiceKey> ingest() {
List<InvoiceKey> rows = new ArrayList<>();
for (int i = 0; i < 5000; i++) rows.add(new InvoiceKey("ACC" + i, PERIOD));
for (int i = 2500; i < 7500; i++) rows.add(new InvoiceKey("ACC" + i, PERIOD));
return rows;
}
public static void main(String[] args) {
List<InvoiceKey> ingested = ingest();
// The dedupe step. One line, and it is doing nothing.
Set<InvoiceKey> distinct = new HashSet<>(ingested);
int expected = 7500; // ACC0..ACC7499, one period
int billed = distinct.size();
System.out.println("rows ingested = " + ingested.size());
System.out.println("expected distinct = " + expected);
System.out.println("actually billed = " + billed);
System.out.println("duplicate charges = " + (billed - expected));
System.out.println("job's own log : deduplicated "
+ (ingested.size() - billed) + " rows");
// The part that makes this invisible in review: equals() is correct.
InvoiceKey a = new InvoiceKey("ACC3000", PERIOD);
InvoiceKey b = new InvoiceKey("ACC3000", PERIOD);
System.out.println("a.equals(b) = " + a.equals(b));
System.out.println("set finds a via b = " + distinct.contains(b));
System.out.println(billed == expected ? "PASS" : "FAIL");
}
}Run it locally:
cd exercises/java/oop/hashcode-equals-contract/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out StarterHints
Hint 1
The release added a field to the key class. Look at which methods were updated when it did — and which were not.
Hint 2
HashSet.add() computes the hash, picks a bucket, and only then compares with equals(). If the hash is wrong the comparison never happens.
Hint 3
Object.hashCode() is identity-based. Two separately constructed rows with identical contents get different hashes.
Hint 4
Once it is correct, ask what test would have failed on the release commit. 'It compiled' is not a test.
Done when
- Distinct account+period pairs collapse to one row each
- equals() and hashCode() read exactly the same fields
- A comment names the test that would have caught this on the release commit
Solution
Show the solution — try it yourself first
Solution.java
import java.util.*;
/**
* Root cause: the release added `period` to InvoiceKey and to equals(), but the
* class never had a hashCode() at all. It inherited Object.hashCode(), which is
* identity-based — so two rows with identical contents hash to different
* buckets, HashSet never compares them with equals(), and every row looks
* unique.
*
* Why nothing failed loudly: HashSet.add() computes the hash, picks a bucket,
* and only then calls equals(). A wrong hash means the comparison never
* happens. There is nothing to throw. size() answers confidently and wrongly.
*
* Why it worked for eight months: it did not. The eight-month-old version had
* the same missing hashCode(). It only became visible when the overlap window
* grew — before the release the two feeds were deduplicated upstream by
* account alone, so this set was never load-bearing.
*
* The fix is one method, and the rule it enforces is: equals() and hashCode()
* must read exactly the same fields.
*
* What would have caught it on the release commit: a unit test asserting
* two separately constructed keys with identical contents are interchangeable
* in a Set —
*
* var a = new InvoiceKey("ACC1", "2026-08");
* var b = new InvoiceKey("ACC1", "2026-08");
* assertEquals(a, b);
* assertEquals(a.hashCode(), b.hashCode()); // this is the one that fails
* assertEquals(1, new HashSet<>(List.of(a, b)).size());
*
* assertEquals(a, b) alone passes, which is exactly why "we have tests for the
* key class" was not enough. The hashCode assertion is the test that matters,
* and EqualsVerifier (nl.jqno.equalsverifier) writes it for you.
*/
public class Solution {
static final class InvoiceKey {
private final String account;
private final String period;
InvoiceKey(String account, String period) {
this.account = account;
this.period = period;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (!(o instanceof InvoiceKey other)) return false;
return account.equals(other.account) && period.equals(other.period);
}
/**
* THE FIX. Same two fields equals() reads, in the same order.
*
* Objects.hash allocates a varargs array and boxes nothing here, since
* both fields are already references. On a hot path you would write
* `31 * account.hashCode() + period.hashCode()` and skip the array;
* this job runs nightly, so readability wins.
*/
@Override
public int hashCode() {
return Objects.hash(account, period);
}
@Override
public String toString() {
return account + "@" + period;
}
}
/*
* The structural fix, which is what I would actually ship:
*
* record InvoiceKey(String account, String period) {}
*
* Both methods are generated from all components, so they cannot drift when
* the next field is added — which is precisely the failure above.
*
* What you give up by using a record:
* - the class is final, so no subclass can extend the key
* - every component is final and public via an accessor
* - you cannot hash a subset of fields (legal, and occasionally wanted:
* hashing fewer fields than equals() compares is contract-safe, since
* equal objects still agree on the hash)
*/
private static final String PERIOD = "2026-08";
static List<InvoiceKey> ingest() {
List<InvoiceKey> rows = new ArrayList<>();
for (int i = 0; i < 5000; i++) rows.add(new InvoiceKey("ACC" + i, PERIOD));
for (int i = 2500; i < 7500; i++) rows.add(new InvoiceKey("ACC" + i, PERIOD));
return rows;
}
public static void main(String[] args) {
List<InvoiceKey> ingested = ingest();
Set<InvoiceKey> distinct = new HashSet<>(ingested);
int expected = 7500;
int billed = distinct.size();
System.out.println("rows ingested = " + ingested.size());
System.out.println("expected distinct = " + expected);
System.out.println("actually billed = " + billed);
System.out.println("duplicate charges = " + (billed - expected));
System.out.println("job's own log : deduplicated "
+ (ingested.size() - billed) + " rows");
InvoiceKey a = new InvoiceKey("ACC3000", PERIOD);
InvoiceKey b = new InvoiceKey("ACC3000", PERIOD);
System.out.println("a.equals(b) = " + a.equals(b));
System.out.println("same hash = " + (a.hashCode() == b.hashCode()));
System.out.println("set finds a via b = " + distinct.contains(b));
System.out.println(billed == expected ? "PASS" : "FAIL");
}
}Stretch
Convert InvoiceKey to a record and delete both methods. Then explain what you
gave up: what can a hand-written key class still do that a record cannot?
(Think about mutability, subclassing, and hashing a subset of fields.)
← Back to What is the contract between hashCode() and equals()?