Production incident

The index that was one column out

40 minintermediate312 yrs

A real incident: symptom first, cause hidden, tradeoff at the end.

The incident

A dashboard query times out. It filters an events table on three columns, and there is an index naming all three of them. index : (tenant_id, created_at, event_type) query : WHERE tenant_id = ? AND event_type = ? AND created_at >= ? What the team found: 1. The execution plan shows the index being used, which is why two reviewers approved it. 2. The plan also shows far more rows read than returned — a number nobody looked at. 3. Adding a second index on event_type made writes slower and the query no faster. 4. The columns are all there. Only the order is wrong. Work out which of the three columns actually narrow the scan and which one is being checked row by row, then fix it without adding an index.

What this teaches

  • Seeking stops at the first range column
  • A range column in the middle wastes every column after it
  • A plan naming your index does not mean the index is doing the work
  • Rows-read versus rows-returned is the number that exposes this
  • Reordering costs nothing; adding an index costs a write per insert

Starter

Starter.java
import java.util.*;
import java.util.stream.*;

/**
 * Production: the index that was one column out.
 *
 * An events table, one index, and a dashboard query that times out. The index
 * names all three columns the query filters on, so the review approved it and
 * the plan does show the index being used.
 *
 * It is still reading a hundred times more entries than it returns, because
 * the columns are in the wrong order.
 *
 * A model, not a database. It applies the two rules that decide everything: a
 * composite index can only be entered from the left, and only up to and
 * including the first range column.
 */
public class Starter {

    record Index(String name, List<String> columns) {
        static Index of(String name, String... c) { return new Index(name, List.of(c)); }
    }

    /** equality: compared with =. range: compared with > or BETWEEN, or null. */
    record Query(String name, List<String> equality, String range, List<String> selected) { }

    static final List<Index> INDEXES = List.of(
        Index.of("idx_events", "tenant_id", "created_at", "event_type"));

    static final Query DASHBOARD = new Query(
        "logins for a tenant, last 24h",
        List.of("tenant_id", "event_type"),
        "created_at",
        List.of("id", "user_id"));

    /* ─────────── the rules ─────────── */

    /**
     * How many leading index columns actually narrow the scan.
     * Equality columns count while they keep matching from the left; the first
     * range column counts and then everything stops.
     */
    static int usefulPrefix(Index index, Query query) {
        int used = 0;
        for (String column : index.columns()) {
            if (query.equality().contains(column)) { used++; continue; }
            if (column.equals(query.range())) { used++; }
            break;
        }
        return used;
    }

    /** True when every column the query reads is in the index. */
    static boolean covers(Index index, Query query) {
        Set<String> have = new HashSet<>(index.columns());
        have.add("id");
        return have.containsAll(query.selected());
    }

    /** Entries the scan must examine, given how much of the index narrowed it. */
    static long entriesExamined(Index index, Query query) {
        long total = 1_000_000;
        for (int i = 0; i < usefulPrefix(index, query); i++) {
            String column = index.columns().get(i);
            total /= column.equals(query.range()) ? 24 : selectivityOf(column);
        }
        return Math.max(total, matchingRows());
    }

    static int selectivityOf(String column) {
        return switch (column) {
            case "tenant_id" -> 50;
            case "event_type" -> 20;
            default -> 1;
        };
    }

    static long matchingRows() { return 1_000_000 / 50 / 24 / 20; }

    public static void main(String[] args) {
        boolean ok = true;
        Index index = INDEXES.get(0);

        long examined = entriesExamined(index, DASHBOARD);
        long returned = matchingRows();
        int prefix = usefulPrefix(index, DASHBOARD);

        System.out.println("── " + DASHBOARD.name() + " ──");
        System.out.println("  index            : " + index.name() + " " + index.columns());
        System.out.println("  equality columns : " + DASHBOARD.equality());
        System.out.println("  range column     : " + DASHBOARD.range());
        System.out.println("  columns that narrow the scan : " + prefix + " of " + index.columns().size()
            + " " + index.columns().subList(0, prefix));
        System.out.println("  entries examined : " + examined);
        System.out.println("  rows returned    : " + returned);
        System.out.println("  read amplification : " + (examined / returned) + "x");
        System.out.println("  covering?        : " + covers(index, DASHBOARD));
        System.out.println();

        ok &= check("every filtered column narrows the scan",
            prefix == DASHBOARD.equality().size() + 1);
        ok &= check("read amplification is under 5x", examined / returned < 5);
        ok &= check("the query is covered by the index", covers(index, DASHBOARD));

        System.out.println();
        System.out.println(ok ? "PASS" : "FAIL");
    }

    static boolean check(String what, boolean passed) {
        System.out.println((passed ? "  ok    " : "  FAIL  ") + what);
        return passed;
    }
}

Run it locally:

cd exercises/java/indexing/composite-index-order/03-production
javac Starter.java -d /tmp/out && java -cp /tmp/out Starter

Hints

  1. Hint 1

    Walk the index columns left to right and stop at the first one that is not an equality predicate. What is left?

  2. Hint 2

    Within one tenant and one time window, are the entries grouped by event_type, or ordered by time?

  3. Hint 3

    The fix does not add a column or an index. It moves one.

  4. Hint 4

    Once all three narrow the scan, ask what the query still has to fetch.

Done when

  • All three filtered columns narrow the scan
  • Read amplification is under 5x
  • The query is covered, so no row is fetched
  • The index count is unchanged, and a comment says why that matters

Solution

Show the solution — try it yourself first
Solution.java
import java.util.*;
import java.util.stream.*;

/**
 * Solution: the index that was one column out.
 *
 * The original index named all three filtered columns and the plan showed it
 * being used, which is why the review passed it. It was still wrong:
 *
 *     (tenant_id, created_at, event_type)
 *                 ^^^^^^^^^^ a RANGE, in the middle
 *
 * Seeking stops at the first range column. tenant_id narrows, created_at
 * narrows, and then the entries within that time window are ordered by time
 * rather than grouped by event_type — so every entry in the window is read
 * and event_type is checked one at a time. The index "uses" three columns and
 * only two of them do any narrowing.
 *
 * Moving the range column last puts both equality columns in front of it:
 *
 *     (tenant_id, event_type, created_at)
 *
 * Now all three narrow the scan, and the entries read are the entries
 * returned. That reordering is the entire performance fix — no new index, no
 * extra write cost, the same three columns.
 *
 * Then user_id is appended so the query never fetches a row. A wider index
 * costs the same ONE write per insert as a narrow one, because write cost is
 * per index and not per column, which is why widening usually beats adding.
 *
 * The rule, in the order to apply it:
 *   1. every equality column first, in any order among themselves
 *   2. the range or ORDER BY column next
 *   3. the selected columns after that, to cover
 */
public class Solution {

    record Index(String name, List<String> columns) {
        static Index of(String name, String... c) { return new Index(name, List.of(c)); }
    }

    /** equality: compared with =. range: compared with > or BETWEEN, or null. */
    record Query(String name, List<String> equality, String range, List<String> selected) { }

    static final List<Index> INDEXES = List.of(
        Index.of("idx_events", "tenant_id", "event_type", "created_at", "user_id"));

    static final Query DASHBOARD = new Query(
        "logins for a tenant, last 24h",
        List.of("tenant_id", "event_type"),
        "created_at",
        List.of("id", "user_id"));

    /* ─────────── the rules ─────────── */

    /**
     * How many leading index columns actually narrow the scan.
     * Equality columns count while they keep matching from the left; the first
     * range column counts and then everything stops.
     */
    static int usefulPrefix(Index index, Query query) {
        int used = 0;
        for (String column : index.columns()) {
            if (query.equality().contains(column)) { used++; continue; }
            if (column.equals(query.range())) { used++; }
            break;
        }
        return used;
    }

    /** True when every column the query reads is in the index. */
    static boolean covers(Index index, Query query) {
        Set<String> have = new HashSet<>(index.columns());
        have.add("id");
        return have.containsAll(query.selected());
    }

    /** Entries the scan must examine, given how much of the index narrowed it. */
    static long entriesExamined(Index index, Query query) {
        long total = 1_000_000;
        for (int i = 0; i < usefulPrefix(index, query); i++) {
            String column = index.columns().get(i);
            total /= column.equals(query.range()) ? 24 : selectivityOf(column);
        }
        return Math.max(total, matchingRows());
    }

    static int selectivityOf(String column) {
        return switch (column) {
            case "tenant_id" -> 50;
            case "event_type" -> 20;
            default -> 1;
        };
    }

    static long matchingRows() { return 1_000_000 / 50 / 24 / 20; }

    public static void main(String[] args) {
        boolean ok = true;
        Index index = INDEXES.get(0);

        long examined = entriesExamined(index, DASHBOARD);
        long returned = matchingRows();
        int prefix = usefulPrefix(index, DASHBOARD);

        System.out.println("── " + DASHBOARD.name() + " ──");
        System.out.println("  index            : " + index.name() + " " + index.columns());
        System.out.println("  equality columns : " + DASHBOARD.equality());
        System.out.println("  range column     : " + DASHBOARD.range());
        System.out.println("  columns that narrow the scan : " + prefix + " of " + index.columns().size()
            + " " + index.columns().subList(0, prefix));
        System.out.println("  entries examined : " + examined);
        System.out.println("  rows returned    : " + returned);
        System.out.println("  read amplification : " + (examined / returned) + "x");
        System.out.println("  covering?        : " + covers(index, DASHBOARD));
        System.out.println();

        ok &= check("every filtered column narrows the scan",
            prefix == DASHBOARD.equality().size() + 1);
        ok &= check("read amplification is under 5x", examined / returned < 5);
        ok &= check("the query is covered by the index", covers(index, DASHBOARD));

        System.out.println();
        System.out.println(ok ? "PASS" : "FAIL");
    }

    static boolean check(String what, boolean passed) {
        System.out.println((passed ? "  ok    " : "  FAIL  ") + what);
        return passed;
    }
}

Stretch

The fixed index is now four columns wide. Say what that costs — entry size, fanout, cache residency — and at what point you would stop adding covering columns. Then argue the case for leaving the range column in the middle: there is one, and it involves a second query.

← Back to Does an index on (a, b) help a query filtering only on b?