Intro
Every time I sit down to size a server, I ask myself one question: "How many CPU cores does this thing actually need?"
You can't just eyeball it. I've seen people divide thread count by two, I've seen people buy the biggest instance they can afford "just in case," and I've seen people match CPU cores to Tomcat's threads.max—then wonder why utilization never breaks 20%.
The root of this problem is simple: thread count and CPU core count are not the same thing. A Tomcat instance with 800 worker threads will run just fine on 4 cores. Why? Because most of those threads are waiting, not computing. I'll share the capacity planning method I use myself to make the relationship between threading models and compute requirements clear.
1. The Counterintuitive Truth: More Threads ≠ More CPU
In the JVM, a thread spends most of its life in one of these states:
In a typical Spring Boot web app, threads spend the vast majority of their time in WAITING—waiting for the database, waiting for disk I/O, waiting for a downstream API. The time actually spent RUNNABLE might be 10–20%.
What this means: Of those 800 threads, only a few dozen are competing for CPU at any given moment. The rest are parked. CPU core count should match the number of simultaneously runnable threads, not the total thread pool size.
Think of it like a restaurant. 800 waiters sounds like a lot, but most of them are standing around waiting for the kitchen or taking orders. Only a handful are actually running food at any given second. How many cooks (CPU cores) you need depends on how many dishes are being prepared at once—not how many waiters you employ.
2. IO-Bound vs CPU-Bound: Two Completely Different Calculations
IO-Bound (most Spring Boot apps)
CPU spends most of its time waiting on IO.
Thread count can far exceed core count. The classic formula:
Optimal threads ≈ cores × (1 + wait time / compute time)
Example: a request takes 100ms total, 80ms waiting on the database, 20ms computing.
wait/compute = 4
On 4 cores: 4 × (1 + 4) = 20 threads
In practice, Tomcat often runs 200–800 threads because context switching is cheap and more threads improve throughput. The key point: core count doesn't need to scale with thread count.
CPU-Bound (encryption, image processing, heavy computation)
CPU is always busy. Thread count should equal core count, maybe +1. More threads just add context-switching overhead.
3. Reverse-Engineering Compute Needs from Tomcat Config
Tomcat's three core parameters:
server:
tomcat:
max-connections: 10000 # max simultaneous connections
threads:
max: 800 # max worker threads
min-spare: 100 # minimum idle threads
accept-count: 200 # backlog queue lengthWhat they actually mean:
max-connections: how many connections can be open at once (including queued)threads.max: how many requests can be processed concurrentlyaccept-count: how long the queue gets once connections are maxed
The real question: how many cores do 800 threads need?
Answer: it depends on the compute ratio per request.
Assume:
Peak QPS: 1000
Average response time: 100ms
Compute time per request: 20ms
Then the number of requests simultaneously computing ≈ 1000 × 0.02 = 20. In theory, 20 cores would suffice—but accounting for spikes and GC, 8–16 cores is a safer bet.
If compute time is higher (say 50ms), the simultaneous compute count rises, and so does your core requirement.
4. A Practical Sizing Formula
Here's a simplified formula:
Cores needed ≈ peak QPS × CPU time per request (seconds) ÷ target utilization
Example:
Peak QPS = 500
CPU time per request = 20ms = 0.02s
Target utilization = 70%
500 × 0.02 ÷ 0.7 ≈ 14.3 cores
Round up: 16 cores. If budget is tight, start with 8 and monitor. Scale when CPU utilization consistently exceeds 70%.
Caveats: this gives you the average requirement. You still need headroom for:
GC pauses (non-negotiable for JVM apps)
Traffic spikes (30–50% buffer)
Other processes (system, monitoring, logging)
5. Connection Pools and Async Thread Pools Also Eat CPU
Two more places where CPU gets consumed indirectly:
1. HikariCP connection pool
The pool itself doesn't burn CPU, but too many connections cause:
Frequent database context switches
Lock contention
GC pressure (every connection holds objects)
Rule of thumb: 30–50 connections on 4 cores, 50–80 on 8 cores. Don't just max it out.
2. Async thread pool (@Async)
spring:
task:
execution:
pool:
core-size: 8
max-size: 16
queue-capacity: 100Async pool sizing should also respect core count. IO-bound async tasks can be over-provisioned; CPU-bound async tasks should equal core count.
6. Cloud vs Bare Metal: The Compute Stability Gap
Same core count, very different real-world behavior.
The thing high-concurrency systems fear most isn't low average performance—it's P99 jitter. When a cloud neighbor gets busy, CPU steal time rises and tail latency spikes. Bare metal doesn't have this problem. Compute is predictable.
If your system is P99-sensitive, dedicated bare metal compute is worth more than a few extra cloud cores.
7. When Does GPU Enter the Picture?
Not every high-concurrency workload needs a GPU. GPUs make sense for:
AI inference (model forward passes)
Video transcoding
Image processing
Batch scientific computing
Decision rule: if your workload is "lots of simple parallel operations," a GPU wins. If it's "complex logic with lots of branching," CPU is the better fit.
For AI inference in production, GPU bare metal guarantees dedicated VRAM and stable compute—exactly what production workloads need.
8. TL;DR
Back to the original question: how much compute does a high-concurrency system actually need?
Three rules:
Thread count ≠ core count. IO-bound apps can run hundreds of threads on a handful of cores.
Size cores by compute load. Peak QPS × CPU time per request ÷ target utilization.
Stability beats peak performance. P99 jitter hurts user experience more than average latency.
Thank you for reading to the end！
