How do you size a thread pool, and what does an unbounded queue cost you?

The Answer

• A ThreadPoolExecutor has three dials — corePoolSize, maximumPoolSize, and the queue — and they interact in an order almost nobody guesses.
• The rule: fill core threads, then queue, and only create threads up to maximumPoolSize when the queue is full. Reject once both are full.
• So with an unbounded queue, maximumPoolSize is dead configuration. It is never reached, and the rejection policy never fires.
• Executors.newFixedThreadPool(n) gives you exactly that — a LinkedBlockingQueue with room for Integer.MAX_VALUE tasks.
• The consequence is not a slow service, it is a memory leak with a queue in front of it. Latency climbs, the heap fills, and nothing sheds load.
• newCachedThreadPool() fails the other way: maximumPoolSize is Integer.MAX_VALUE, so a burst creates a thread per task.
• Size it from what the tasks do: CPU-bound ≈ cores + 1; I/O-bound is higher and should be derived from measured wait time, not guessed.

Understand It

What the factory methods actually build

Executors is a convenience class, and every one of its shortcuts hides the dial that matters:

System.out.println(describePool("newFixedThreadPool(2)", Executors.newFixedThreadPool(2)));
System.out.println(describePool("newCachedThreadPool()", Executors.newCachedThreadPool()));
System.out.println(describePool("newSingleThreadExecutor", Executors.newSingleThreadExecutor()));

newFixedThreadPool(2)    core=2  max=2           queue=LinkedBlockingQueue (unbounded)
newCachedThreadPool()    core=0  max=unbounded   queue=SynchronousQueue (holds 0)
newSingleThreadExecutor  wrapped — not a ThreadPoolExecutor, so it cannot be reconfigured

Two different ways to have no limit. newFixedThreadPool bounds the threads and leaves the queue unbounded; newCachedThreadPool bounds the queue at zero and leaves the threads unbounded. Neither can refuse work, and refusing work is the only thing that keeps a service up when it is overloaded.

The third line is a different lesson and worth a moment. newSingleThreadExecutor() deliberately returns a wrapper rather than the pool itself, so the cast fails and the pool cannot be reconfigured later. newFixedThreadPool(1) looks identical from the outside and is not wrapped, so this works:

ThreadPoolExecutor pool = (ThreadPoolExecutor) Executors.newFixedThreadPool(1);
pool.setMaximumPoolSize(5);     // order matters — core may never exceed max,
pool.setCorePoolSize(5);        //   so raising core first throws IllegalArgumentException
// no longer single-threaded, and every caller relying on that is now wrong

Two factory methods, the same apparent behaviour, opposite intent: one lets you take the guarantee away and one does not. If a caller depends on "exactly one thread" for ordering or for single-threaded confinement, that wrapper is the thing protecting them.

This is why Effective Java says to prefer ThreadPoolExecutor directly. Not because the factories are wrong, but because they make the decision for you and then hide it.

The growth rule, and why maximumPoolSize is usually a lie

Here is the ordering, and it is the single most useful thing on this page:

The emphasised steps are where the intuition breaks. Most people read new ThreadPoolExecutor(1, 10, …) as "between one and ten threads, as load demands". It means "one thread, until the queue is full, then up to ten". If the queue never fills, the pool never grows.

Watch it happen. Fifty tasks, core 1, maximum 10, unbounded queue:

ThreadPoolExecutor unbounded = new ThreadPoolExecutor(
        1, 10, 60, TimeUnit.SECONDS, new LinkedBlockingQueue<>());
System.out.println("unbounded queue : " + saturate(unbounded, 50));

ThreadPoolExecutor bounded = new ThreadPoolExecutor(
        1, 10, 60, TimeUnit.SECONDS, new ArrayBlockingQueue<>(5));
System.out.println("queue of 5      : " + saturate(bounded, 50));

unbounded queue : threads=1   queued=49  rejected=0
queue of 5      : threads=10  queued=5   rejected=35

Same core, same maximum, same fifty tasks. The only difference is the queue, and it decides everything: one thread or ten, and nothing rejected or thirty-five rejected.

The first line is what a newFixedThreadPool does all day. The second is a pool that can actually say no.

"Rejected" is not the failure. It is the feature.

Thirty-five rejections looks like the worse outcome, which is exactly the instinct that produces the first line in production.

Think about what the two rows mean at 3am. The bounded pool refused 35 tasks immediately, at the point where a caller can still do something about it: retry elsewhere, fail the request fast, return a 503 the load balancer understands. The unbounded pool accepted all fifty and now owes fifty answers it cannot deliver. Every queued task holds its arguments alive on the heap, and the queue is a GC root by way of the pool.

So the queue has a second meaning nobody writes down: queue depth is latency you have already promised. A thousand queued tasks on a pool that completes ten a second is a hundred seconds of debt, and the clients waiting on them have almost certainly timed out already — so the pool is doing work whose results nobody will read, while new work waits behind it.

That is the argument for a small queue, and it is the same reason connection-pool-sizing argues for a small pool.

submit() swallows exceptions and execute() does not

The second thing that bites, and it is silent:

ExecutorService pool = Executors.newFixedThreadPool(1);

Future<?> submitted = pool.submit(() -> {
    throw new IllegalStateException("the task failed");
});

Thread.sleep(200);
System.out.println("after submit(), the stack trace printed : no");

try {
    submitted.get();
} catch (ExecutionException e) {
    System.out.println("and Future.get() finally reveals it   : " + e.getCause());
}

pool.shutdown();

after submit(), the stack trace printed : no
and Future.get() finally reveals it   : java.lang.IllegalStateException: the task failed

submit() wraps the task in a FutureTask, which catches everything and stores it in the Future. If you never call get() — and for a fire-and-forget task you never do — the exception is gone. No log, no stack trace, no uncaught-exception handler. The task simply did not happen.

execute() has no Future to hide it in, so the exception reaches the thread's uncaught-exception handler and gets printed.

The practical rule: use execute() for fire-and-forget, submit() only when you will actually consume the Future. If you must use submit() for a task nothing waits on, wrap the body in a try/catch that logs — nothing else will.

Sizing, without the folklore

The formula people quote — threads = cores × (1 + wait/compute) — is real, from Java Concurrency in Practice, and useless without measuring wait and compute. What the formula is actually telling you is that there are only two kinds of task and they want opposite things:

The last row is the one that matters and the one sizing discussions skip. If eighty threads all call the same database with a connection pool of ten, the answer is not eighty threads — seventy of them are queued on the connection pool instead of on yours, and you have moved the queue rather than removed it. Size the pool to the narrowest downstream limit, and put the queue where you can see it.

Runtime.getRuntime().availableProcessors() is the starting point for the CPU case, with one caveat: inside a container it reports the cgroup CPU limit, not the host's cores. That is correct behaviour and has been since Java 10, but it means the number changes when someone edits a Kubernetes manifest.

Java 19+For I/O-bound work the question is going away. Executors.newVirtualThreadPerTaskExecutor() is not a pool — it creates a virtual thread per task, and a parked virtual thread costs a few hundred bytes of heap rather than a megabyte of stack. Pool sizing exists to ration platform threads; where the tasks block rather than compute, virtual threads remove the thing being rationed. It does not help CPU-bound work, where the ceiling is the CPU and always was.

Reference

A pool you would actually ship

// Named threads, a bounded queue, and a rejection policy that is a decision
// rather than a default.
ThreadPoolExecutor pool = new ThreadPoolExecutor(
        8, 8,                                   // core == max: a fixed pool,
        0L, TimeUnit.MILLISECONDS,              //   so keep-alive is moot
        new ArrayBlockingQueue<>(100),          // the number that decides everything
        new ThreadFactory() {                   // names show up in every thread dump
            private final AtomicInteger n = new AtomicInteger(1);
            public Thread newThread(Runnable r) {
                Thread t = new Thread(r, "orders-" + n.getAndIncrement());
                t.setUncaughtExceptionHandler((thread, e) ->
                        log.error("uncaught in {}", thread.getName(), e));
                return t;
            }
        },
        new ThreadPoolExecutor.CallerRunsPolicy());

// Shutdown, in the order that actually works.
pool.shutdown();                                 // stop accepting, finish what is queued
if (!pool.awaitTermination(30, TimeUnit.SECONDS)) {
    pool.shutdownNow();                          // interrupt the stragglers
    pool.awaitTermination(10, TimeUnit.SECONDS); // and give them a moment
}

Naming the threads is the cheapest operational win available. pool-1-thread-7 in a thread dump tells you nothing; orders-7 tells you which subsystem is stuck.

The four rejection policies

CallerRunsPolicy is the underrated one. It turns rejection into throttling: the thread that submitted the work gets conscripted into doing it, so the producer cannot outrun the pool. For a request thread accepting uploads, that is precisely the behaviour you want.

Choosing an executor

// Fixed work, CPU-bound.
Executors.newFixedThreadPool(cores + 1)            // fine, but see the queue caveat

// Anything user-facing where overload is possible: build it yourself.
new ThreadPoolExecutor(...)                        // so the queue is a decision

// Scheduled work. Note it is core-sized only and the queue is unbounded.
Executors.newScheduledThreadPool(2)

// Blocking I/O, Java 21+. Not a pool; no sizing question.
Executors.newVirtualThreadPerTaskExecutor()

// Divide-and-conquer over a data structure. Work stealing, not a queue.
new ForkJoinPool(parallelism)

What to watch in production

// ThreadPoolExecutor exposes everything worth alerting on. Export these.
pool.getActiveCount();        // threads currently running a task
pool.getQueue().size();       // THE metric — queue depth is promised latency
pool.getPoolSize();           // grows only when the queue fills
pool.getCompletedTaskCount();
pool.getLargestPoolSize();    // did it ever reach maximumPoolSize? often: no

Alert on queue depth, not on pool size. Pool size is a lagging indicator that with an unbounded queue never moves at all.

Scenarios

A service gets slower under load and then OOMs, with no thread count change. The classic shape of an unbounded queue: getPoolSize() sits flat at the core size while getQueue().size() climbs into the hundreds of thousands, each queued task holding its request object alive. The heap dump shows a huge LinkedBlockingQueue. The fix is a bounded queue and a rejection policy, and the uncomfortable part of that fix is that the service will now visibly reject requests it previously accepted and quietly failed to answer in time.

Someone proposes raising maximumPoolSize because the pool seems stuck at its core size. It will change nothing, and knowing why is the whole entry: threads beyond the core are only created when the queue is full, and the queue is unbounded. Raising the maximum is a no-op until the queue is bounded. Bound the queue first; then the maximum starts meaning something, and you can decide whether you wanted it.

Tasks submitted with submit() are vanishing. No errors, no logs, work simply not done. FutureTask caught the exception and put it in a Future nobody reads. Switch to execute() for fire-and-forget, or keep submit() and wrap the body in try/catch. This one is worth a lint rule: an ignored return value from submit() is nearly always a bug, and it is silent in exactly the cases you care about.

A pooled task submits to the same pool and waits. A fixed pool of eight, where each task submits a subtask to that pool and calls get(). Under load, all eight threads are blocked waiting on subtasks that can never start, because every thread is occupied waiting. This is thread-pool deadlock, and no pool size fixes it — eight becomes eighty and it deadlocks at eighty. Use a separate pool for the nested stage, or compose without blocking via CompletableFuture. See deadlock.

Interviewer's Next Move

1. "What is wrong with Executors.newFixedThreadPool?" Its queue is a LinkedBlockingQueue with capacity Integer.MAX_VALUE. It can never reject, so under sustained overload it accumulates tasks until the heap is gone — and because threads beyond the core are only created when the queue fills, maximumPoolSize is unreachable too. Build a ThreadPoolExecutor with a bounded queue instead.

2. "In what order does ThreadPoolExecutor use its three settings?" Core threads first, then the queue, then growth up to maximumPoolSize, then rejection. The step people get wrong is the second: it queues before creating threads beyond the core, so an unbounded queue means the pool never grows and the rejection handler never runs.

3. "What is the difference between submit() and execute()?" execute() takes a Runnable and lets exceptions reach the thread's uncaught handler. submit() returns a Future and stores the exception inside it, so a task that throws is silent unless someone calls get(). Use execute() for fire-and-forget; if you use submit(), consume the Future or catch inside the task.

4. "How would you size a pool for an endpoint that calls two services and a database?" By measuring, and by finding the narrowest downstream limit first. The work is I/O-bound so it wants more threads than cores, but if the database connection pool is ten then more than about ten concurrent database calls just moves the queue somewhere I cannot see. I would start from the connection pool size, measure wait versus compute, and alert on queue depth rather than tune the number once and forget it.

5. "When would you choose CallerRunsPolicy?" When I want backpressure rather than failure. It makes the submitting thread run the rejected task, so the producer is slowed by exactly the amount the pool is behind — which for a request thread reading uploads is self-limiting in the right direction. It is a poor choice when the calling thread must not block, such as an event-loop or a scheduler thread.

6. "Do virtual threads make this obsolete?" For blocking I/O, largely yes — newVirtualThreadPerTaskExecutor() is not a pool, so there is no size to choose, and a parked virtual thread costs heap rather than a megabyte of stack. For CPU-bound work, no: the ceiling was the CPU, not the thread, and a million virtual threads does not add cores. You also still need to bound admission somewhere, or you have moved the unbounded queue rather than removed it.

Code traps

ThreadPoolExecutor pool = new ThreadPoolExecutor(
        2, 50, 60, TimeUnit.SECONDS, new LinkedBlockingQueue<>());

for (int i = 0; i < 10_000; i++) {
    pool.execute(this::slowTask);
}
// How many threads does this pool have?

ExecutorService pool = Executors.newFixedThreadPool(4);
pool.submit(() -> riskyCleanup());
pool.shutdown();

Common wrong answers

Check Yourself

Q1. new ThreadPoolExecutor(1, 10, …, new LinkedBlockingQueue<>()) is given fifty slow tasks. How many threads run, and why?

Q2. Why is a rejected task better news than a queued one?

Q3. A fire-and-forget task throws and nothing appears in the logs. What is the likely cause?

Practice