Challenge

Five deadlock reports, five different fixes

25 minintermediate115 yrs

Edge cases. You have to reason, and two valid fixes differ.

What this teaches

  • Lock ordering is the fix for some deadlocks and a distraction for others
  • A sequential scan inside a write transaction locks rows it does not change
  • One of the five is a lock wait timeout, which wants the opposite response
  • A single hot row cannot be ordered, so it needs a different shape of fix

Starter

Starter.java
import java.util.*;

/**
 * Challenge: five reports from production. Diagnose each and fix it.
 *
 * They arrive looking the same — an exception, two statements, a rolled-back
 * transaction. They have five different causes, and one of them is not a
 * deadlock at all.
 *
 * Reuse the lock manager from the warm-up: heldBy, waitingFor, and a walk of
 * the waits-for graph.
 */
public class Starter {

    // ── report 1 ──────────────────────────────────────────────────────────
    // Two transfers between the same pair of accounts, in opposite directions.
    //   T1: UPDATE account SET ... WHERE id = 7
    //       UPDATE account SET ... WHERE id = 3
    //   T2: UPDATE account SET ... WHERE id = 3
    //       UPDATE account SET ... WHERE id = 7

    // ── report 2 ──────────────────────────────────────────────────────────
    // Two nightly batch jobs, each archiving a list of order ids built by a
    // different query. The lists overlap. Neither is sorted.
    //   T1: UPDATE orders SET archived = true WHERE id = ?   (ids 4, 1, 9)
    //   T2: UPDATE orders SET archived = true WHERE id = ?   (ids 9, 1, 4)

    // ── report 3 ──────────────────────────────────────────────────────────
    // Two statements whose matched rows do not overlap at all.
    //   T1: UPDATE orders SET priority = 1 WHERE id = 500        (primary key)
    //   T2: UPDATE orders SET status = 'x' WHERE region = 'APAC' (no index on region,
    //                                                            matches only id 1)

    // ── report 4 ──────────────────────────────────────────────────────────
    // Reported as a deadlock by the on-call engineer. The log says:
    //   T1 holds inventory:42 and is running a 4-second call to a payment API
    //   T2 waits for inventory:42
    //   T3 waits for inventory:42
    //   T4 waits for inventory:42
    // Error surfaced to users: lock wait timeout exceeded.

    // ── report 5 ──────────────────────────────────────────────────────────
    // A single counter row, updated by every request during a flash sale.
    //   T1: SELECT stock FROM inventory WHERE id = 1 FOR UPDATE
    //       UPDATE inventory SET stock = ? WHERE id = 1
    //   ...and two hundred other transactions doing exactly the same thing.

    public static void main(String[] args) {

        // TODO 1: model each report with your lock manager and confirm which
        // ones actually produce a cycle. Two of the five do not.

        // TODO 2: for each, answer the three diagnostic questions in order:
        //   a. do the transactions take rows in different orders?
        //   b. does either statement scan rather than seek?
        //   c. is either transaction holding locks longer than it needs to?
        // Exactly one question explains each report.

        // TODO 3: fix report 1. One line. Say why it works without reducing
        // how much is locked or for how long.

        // TODO 4: fix report 2. The lists come from different queries, so you
        // cannot fix it at the call site. Where does the fix belong, and what
        // is the smallest change that makes both jobs agree?

        // TODO 5: fix report 3. Lock ordering will not help here — show why by
        // trying it. Then say what you would have run before touching any code.

        // TODO 6: report 4 is not a deadlock. Name what it is, say how you can
        // tell from the log alone, and explain why retrying makes it worse.

        // TODO 7: report 5 cannot be fixed by ordering, because there is only
        // one row to order. Give two fixes of different shapes, and name the
        // cost of each. One of them changes what the application can promise
        // the user — say which and how.

        // TODO 8: a retry policy on deadlocks would hide reports 1, 2 and 3
        // from users entirely. Say why you would still alert on the rate, and
        // what the rate climbing would be telling you.
    }
}

Run it locally:

cd exercises/java/transactions/database-deadlocks/02-challenge
javac Starter.java -d /tmp/out && java -cp /tmp/out Starter

Hints

  1. Hint 1

    For each report, ask three questions in order: do they take rows in different orders, does either do a sequential scan, and is either transaction longer than it needs to be.

  2. Hint 2

    One report has no cycle in it at all. Read the waits carefully before assuming the error name is accurate.

  3. Hint 3

    One is between two statements whose matched rows do not overlap. The answer is in the access path, not the ordering.

  4. Hint 4

    One is on a single row that everything updates. Sorting one id does nothing — say what actually helps and what it costs.

Done when

  • Each report is classified and has a specific fix
  • The lock wait timeout is identified and is not answered with a retry
  • The missing-index case is answered with an index, not with lock ordering
  • The hot-row case is answered with a different shape of fix, and its cost is named
  • You said which of the five would be hidden by a retry policy, and why that is still worth alerting on

← Back to What causes a database deadlock, and how do you prevent it?