Every verified cell, side by side
The cards below tell you how to run one model. This tells you which one to reach for — and where a config that wins on latency loses on throughput.
Model serving recipes
Pick a model, then a verified cell in the coverage matrix (target × strategy). Cells marked — aren't benchmarked yet — we publish measured numbers only.
What's missing — and how to fill it
Every gap below is a config or metric we haven't measured. Each comes with the exact command to run
against a live server — the result drops straight into models.js as a verified row.
This is the to-do list; ready-made harness scripts live in bench/.
Before you launch
Shared setup for every recipe. The model-specific Docker image, weights path, and flags live in each recipe card above.
Disable NUMA balancing
Kernel NUMA auto-balancing migrates pages under the GPU and costs throughput. Turn it off before serving.
sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'
cat /proc/sys/kernel/numa_balancing # expect: 0Mount the HuggingFace cache
Bind-mount the host weights cache into the container so models load offline. Recipes assume the cache is visible at /hf-cache.
# host HF cache -> /hf-cache inside the container
-v /data/hf-cache:/hf-cache -e HF_HOME=/hf-cachePull the model's verified image
Each recipe pins the exact rocm/sgl-dev (or bespoke) image it was measured on — copy it from the recipe's provenance footnote. ROCm 7.2 / MI35x images also cover gfx942.
How the numbers are made
Latency — offline
sglang.bench_one_batch runs a single batch×input_len forward unchunked. Reports prefill tok/s and per-token decode (TPOT) at BS=1 — the single-request floor.
Throughput — online
sglang.bench_serving drives the live server with chunked prefill across concurrency levels. Reports TTFT, TPOT, and tok/s/GPU — the apples-to-apples basis for the NVIDIA comparison.
Roofline
Decode ceiling = aggregate HBM bandwidth ÷ active-parameter bytes (memory-bound, BS=1). The gauge shows measured ÷ ceiling. See the full formula.
Accuracy
GSM8K through in-tree sglang.test.run_eval at --temperature 0. AIME25 through sgl-eval, not in-tree — its strict Answer: first-match regex under-reports reasoning models badly (62.5% vs 90.6% on the same GLM-5.2 server). Reported as pass@1 averaged over repeats, with the SEM.