Production incident

The table that was salted and still broken

45 minintermediate212 yrs

A real incident: symptom first, cause hidden, tradeoff at the end.

The incident

A password table written by someone who had read half of the advice. Every row has a unique salt, so rainbow tables are useless against it — and the auditor still marked the finding critical. What the review turned up: 1. The team's defence was "we salt every password", which is true. 2. Salts generated in a loop during a data migration came out identical. 3. Login response times differ measurably between a wrong password that shares a prefix with the real hash and one that does not. 4. Nobody can raise the cost factor, because there is no way to tell an old row from a new one. Four separate controls, four separate defects, and the one everybody talks about — the salt — is the one they got right. Fix all four.

What this teaches

  • Salting defeats precomputation; it does nothing about guessing speed
  • java.util.Random is predictable, so salts from it can be regenerated
  • String comparison leaks how much of a hash matched
  • A stored hash must record its own algorithm and cost, or it can never be migrated
  • The four controls are independent; getting one right proves nothing about the others

Starter

Starter.java
import java.util.*;

/**
 * Production: the table that was salted and still broken.
 *
 * A password table written by someone who had read half of the advice. Every
 * row has a unique salt, so rainbow tables are useless — and the auditor
 * still marked it critical.
 *
 * Four checks. All four fail, and each one is a different mistake.
 */
public class Starter {

    record Account(String user, String salt, String hash) { }

    /** DEFECT 1: a fast hash. Salting fixes precomputation, not speed. */
    static String hash(String password, String salt) throws Exception {
        var md = java.security.MessageDigest.getInstance("SHA-256");
        return HexFormat.of().formatHex(md.digest((salt + password).getBytes("UTF-8")));
    }

    /** DEFECT 2: a predictable PRNG. An attacker can regenerate every salt. */
    static String newSalt() {
        Random random = new Random(System.currentTimeMillis());
        byte[] bytes = new byte[16];
        random.nextBytes(bytes);
        return HexFormat.of().formatHex(bytes);
    }

    /** DEFECT 3: short-circuits, so the response time leaks the prefix length. */
    static boolean verify(String password, Account account) throws Exception {
        return account.hash().equals(hash(password, account.salt()));
    }

    public static void main(String[] args) throws Exception {
        boolean ok = true;

        Account ana = new Account("ana", newSalt(), null);
        ana = new Account("ana", ana.salt(), hash("hunter2", ana.salt()));

        System.out.println("── the stored row ──");
        System.out.println("  salt : " + ana.salt());
        System.out.println("  hash : " + ana.hash().substring(0, 32) + "...");
        System.out.println();

        // 1. How fast can an attacker guess against this scheme?
        //    Measured against a one-second budget rather than a fixed count,
        //    so it takes the same time whatever the hash costs.
        long start = System.nanoTime();
        int guesses = 0;
        do {
            hash("guess" + guesses, ana.salt());
            guesses++;
        } while (System.nanoTime() - start < 1_000_000_000L);
        double perSecond = guesses / ((System.nanoTime() - start) / 1_000_000_000.0);
        System.out.printf("  offline guess rate : roughly %.1f per second%n", perSecond);
        ok &= check("guessing is slower than 10,000 per second", perSecond < 10_000);

        // 2. Are the salts unpredictable?
        Set<String> salts = new LinkedHashSet<>();
        for (int i = 0; i < 5; i++) salts.add(newSalt());
        ok &= check("five salts generated in a loop are all different", salts.size() == 5);

        // 3. Is the comparison constant time?
        ok &= check("hash comparison is constant time", usesConstantTimeCompare());

        // 4. Does the scheme record how the hash was made?
        ok &= check("the stored value records its algorithm and cost",
            ana.hash().startsWith("{") || ana.hash().startsWith("$"));

        System.out.println();
        System.out.println(ok ? "PASS" : "FAIL");
    }

    /** Reads the verify method's own source-level choice, declared here. */
    static boolean usesConstantTimeCompare() {
        return CONSTANT_TIME;
    }

    /** Set this to true only when verify() actually uses MessageDigest.isEqual. */
    static final boolean CONSTANT_TIME = false;

    static boolean check(String what, boolean passed) {
        System.out.println((passed ? "  ok    " : "  FAIL  ") + what);
        return passed;
    }
}

Run it locally:

cd exercises/java/authentication/password-storage/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out Starter

Hints

  1. Hint 1

    Measure the guess rate before changing anything. That number is the finding.

  2. Hint 2

    Generate five salts in a tight loop and print them. What seeds java.util.Random by default?

  3. Hint 3

    How long does String.equals take when the first character differs, versus when thirty do?

  4. Hint 4

    Look at a bcrypt hash. Why does it start with $2a$12$ rather than just the digest?

Done when

  • Offline guessing is under ten thousand attempts per second
  • Five salts generated in a loop are all different
  • Comparison is constant time
  • The stored value records its algorithm and cost
  • A comment names which of the four the original got right

Solution

Show the solution — try it yourself first
Solution.java
import java.util.*;

/**
 * Solution: the table that was salted and still broken.
 *
 * The salt was the part they got right, and it is the part that gets the
 * attention. Four things were still wrong, and each is a separate control:
 *
 *   1. SPEED. Salting defeats precomputed tables; it does nothing about
 *      guessing rate. SHA-256 lets an attacker with the leaked table try
 *      hundreds of thousands of passwords a second per core, and far more on
 *      a GPU. A password hash has to be deliberately slow — that is what a
 *      work factor is, and it is the only control that makes a leak
 *      survivable.
 *
 *   2. SALT SOURCE. java.util.Random is a predictable PRNG seeded from the
 *      clock. Salts generated in a loop within the same millisecond are
 *      IDENTICAL, which quietly undoes the one thing the scheme had right.
 *      SecureRandom, always.
 *
 *   3. COMPARISON. String.equals returns at the first differing character, so
 *      how long a login takes leaks how much of the hash matched.
 *      MessageDigest.isEqual compares the full length regardless. The same
 *      rule applies to tokens, API keys and HMAC signatures.
 *
 *   4. FORMAT. Storing a bare hex string records nothing about how it was
 *      made, so the cost factor can never be raised and the algorithm can
 *      never be migrated — there is no way to tell an old row from a new one.
 *      Encoding the algorithm, the cost and the salt into the stored value is
 *      what makes future migration possible at all, and it is why bcrypt and
 *      Argon2 hashes look the way they do.
 *
 * PBKDF2 is used here because it is in the JDK, so the fix needs no
 * dependency. In a Spring service, use BCryptPasswordEncoder through
 * PasswordEncoderFactories.createDelegatingPasswordEncoder(), which produces
 * the {bcrypt}$2a$12$... form and handles all four of these for you.
 */
public class Solution {

    record Account(String user, String salt, String hash) { }

    static final int ITERATIONS = 210_000;
    static final String ALGORITHM = "pbkdf2-sha256";

    /** FIX 1: a deliberately slow, tunable KDF. */
    static String hash(String password, String salt) throws Exception {
        var spec = new javax.crypto.spec.PBEKeySpec(
            password.toCharArray(), HexFormat.of().parseHex(salt), ITERATIONS, 256);
        byte[] key = javax.crypto.SecretKeyFactory
            .getInstance("PBKDF2WithHmacSHA256").generateSecret(spec).getEncoded();
        spec.clearPassword();
        // FIX 4: the stored value records how it was made.
        return "$" + ALGORITHM + "$" + ITERATIONS + "$" + HexFormat.of().formatHex(key);
    }

    /** FIX 2: a cryptographic source, so salts cannot be regenerated. */
    static String newSalt() throws Exception {
        byte[] bytes = new byte[16];
        java.security.SecureRandom.getInstanceStrong().nextBytes(bytes);
        return HexFormat.of().formatHex(bytes);
    }

    /** FIX 3: constant time, so the response leaks no prefix length. */
    static boolean verify(String password, Account account) throws Exception {
        byte[] expected = account.hash().getBytes("UTF-8");
        byte[] actual = hash(password, account.salt()).getBytes("UTF-8");
        return java.security.MessageDigest.isEqual(expected, actual);
    }

    static final boolean CONSTANT_TIME = true;

    public static void main(String[] args) throws Exception {
        boolean ok = true;

        Account ana = new Account("ana", newSalt(), null);
        ana = new Account("ana", ana.salt(), hash("hunter2", ana.salt()));

        System.out.println("── the stored row ──");
        System.out.println("  salt : " + ana.salt());
        System.out.println("  hash : " + ana.hash().substring(0, 40) + "...");
        System.out.println();

        long start = System.nanoTime();
        int guesses = 0;
        do {
            hash("guess" + guesses, ana.salt());
            guesses++;
        } while (System.nanoTime() - start < 1_000_000_000L);
        double perSecond = guesses / ((System.nanoTime() - start) / 1_000_000_000.0);
        System.out.printf("  offline guess rate : roughly %.1f per second%n", perSecond);
        ok &= check("guessing is slower than 10,000 per second", perSecond < 10_000);

        Set<String> salts = new LinkedHashSet<>();
        for (int i = 0; i < 5; i++) salts.add(newSalt());
        ok &= check("five salts generated in a loop are all different", salts.size() == 5);

        ok &= check("hash comparison is constant time", CONSTANT_TIME);

        ok &= check("the stored value records its algorithm and cost",
            ana.hash().startsWith("{") || ana.hash().startsWith("$"));

        System.out.println();
        System.out.println("  correct password verifies? " + verify("hunter2", ana));
        System.out.println("  wrong password rejected?   " + !verify("hunter3", ana));

        System.out.println();
        System.out.println(ok ? "PASS" : "FAIL");
    }

    static boolean check(String what, boolean passed) {
        System.out.println((passed ? "  ok    " : "  FAIL  ") + what);
        return passed;
    }
}

Stretch

The fix uses PBKDF2 because it needs no dependency. Write the Spring Security version with a DelegatingPasswordEncoder, and describe the migration for a table that now holds both formats — including what happens to accounts that never log in again, and who decides the deadline.

← Back to How should a password be stored?