Hypothetical system studies that reason from first principles. Scale figures and design choices are illustrative — the aim is rigorous engineering analysis, not marketing.
Hundreds of models, ten thousand GPUs, a nine-figure compute bill — and almost every design decision turns out to be one dial: how much do you pay, in money or complexity, to protect latency?
Hundreds of models, thousands of GPUs, a multi-million-dollar compute bill — and yet the scheduler dispatches about once a second. The hard part isn't throughput. It's trusting the number.
It looks read-heavy. It isn't. A per-user RAG at 220M-user scale is write-dominated, the LLM bill dwarfs the infrastructure, and one partitioning decision quietly solves four problems at once.
A single H100, a 32B model, billions of short answers a day — and GPUs that looked busy but weren't fast. The bottleneck wasn't compute. It was memory bandwidth.
Why adding GPUs stopped improving throughput in a 235B-parameter MoE platform — and how the bottleneck turned out to live in the network, not the GPU.