Architecture Studies
Hypothetical system studies that reason from first principles. Scale figures and design choices are illustrative — the aim is rigorous engineering analysis, not marketing.
Serving Hundreds of LLMs at Scale: Architecting a Multi-Model Inference Platform Across Heterogeneous GPUs
Hundreds of models, ten thousand GPUs, a nine-figure compute bill — and almost every design decision turns out to be one dial: how much do you pay, in money or complexity, to protect latency?
Benchmarking LLMs at Scale: Architecting a Reproducible, GPU-Efficient Platform for Hundreds of Models a Day
Hundreds of models, thousands of GPUs, a multi-million-dollar compute bill — and yet the scheduler dispatches about once a second. The hard part isn't throughput. It's trusting the number.
Per-User RAG at Scale: Architecting a Private AI Inbox for 220M Users and Billions of Queries a Day
It looks read-heavy. It isn't. A per-user RAG at 220M-user scale is write-dominated, the LLM bill dwarfs the infrastructure, and one partitioning decision quietly solves four problems at once.
Kernel Fusion for Large-Scale LLM Inference: When GPU Memory Bandwidth, Not Compute, Is the Bottleneck
A single H100, a 32B model, billions of short answers a day — and GPUs that looked busy but weren't fast. The bottleneck wasn't compute. It was memory bandwidth.
Networking Bottlenecks in Large-Scale Mixture-of-Experts (MoE) Inference
Why adding GPUs stopped improving throughput in a 235B-parameter MoE platform — and how the bottleneck turned out to live in the network, not the GPU.