Explore inference engines, container images, launch parameters, GPU requirements, and reports comparing baseline and tuned configurations.
glm · Single node
4 × B300
Tuned by LLM AutoTune: TP4 on four B300 GPUs, single node, modelopt_fp4 weights, EAGLE speculative decoding, hierarchical KV cache. Normalized throughput (8 GPUs) within the SLO vs the baseline variant: +195.3% (50k + 1.5k), +30.6% (8k + 1k).
50k + 1.5k tokens · Measured with v1.0.4
deepseek · Single node
8 × H100
Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node. +89% on the tuning benchmark vs the baseline variant.
50k + 1.5k tokens · Measured with v1.0.0
laguna · Single node
8 × H100
Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node, fp8 KV cache. +26% on the tuning benchmark vs the baseline variant.
8k + 1k tokens · Measured with v1.0.0
hunyuan · Single node
8 × H100
Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node, NEXTN speculative decoding. Normalized throughput within the SLO vs the baseline variant: +64.2% (50k + 1.5k), +7.6% (8k + 1k).
50k + 1.5k tokens · Measured with v1.0.0
deepseek · Single node
8 × H100
Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node, DSPARK speculative decoding. Normalized throughput within the SLO vs the baseline variant: +58.0% (50k + 1.5k), +23.4% (8k + 1k).
50k + 1.5k tokens · Measured with v1.0.0
kimi · Single node
8 × B300
Tuned by LLM AutoTune: TP8 on eight B300 GPUs, single node, fp8 KV cache, DSPARK speculative decoding, hierarchical KV cache. Normalized throughput within the SLO vs the baseline variant: +57.6% (50k + 1.5k), +21.4% (8k + 1k).
50k + 1.5k tokens · Measured with v1.0.5
mistral · Single node
2 × H100
Tuned by LLM AutoTune: TP2 on two H100 GPUs, single node. Normalized throughput within the SLO vs the baseline variant: +52.5% (50k + 1.5k), +32.1% (8k + 1k).
50k + 1.5k tokens · Measured with v1.0.0
deepseek · Single node
8 × H100
Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node, EAGLE speculative decoding. +51% on the tuning benchmark vs the baseline variant.
50k + 1.5k tokens · Measured with v1.0.2
gemma · Single node
2 × H100
Tuned by LLM AutoTune: TP2 on two H100 GPUs, single node, fp8 KV cache. Normalized throughput within the SLO vs the baseline variant: +38.8% (50k + 1.5k), +45.6% (8k + 1k).
8k + 1k tokens · Measured with v1.0.0
gpt · Single node
8 × H100
Tuned by LLM AutoTune: TP2 DP4 on eight H100 GPUs, single node, fp8 KV cache, EAGLE3 speculative decoding (lmsys/EAGLE3-gpt-oss-120b-bf16). +44% on the tuning benchmark vs the baseline variant.
50k + 1.5k tokens · Measured with v1.0.0
google · Single node
4 × H100
Tuned by LLM AutoTune: TP4 on four H100 GPUs, single node. Normalized throughput within the SLO vs the baseline variant: +32.4% (50k + 1.5k), +1.6% (8k + 1k).
50k + 1.5k tokens · Measured with v1.0.0
qwen · Single node
2 × H100
Tuned by LLM AutoTune: TP2 on two H100 GPUs, single node. Normalized throughput within the SLO vs the baseline variant: +30.1% (50k + 1.5k), +12.1% (8k + 1k).
50k + 1.5k tokens · Measured with v1.0.0
nemotron · Single node
8 × H100
Tuned by LLM AutoTune: TP1 DP8 on eight H100 GPUs, single node. +27% on the tuning benchmark vs the baseline variant.
50k + 1.5k tokens · Measured with v1.0.0
nex · Single node
8 × H100
Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node, fp8 KV cache. +24% on the tuning benchmark vs the baseline variant.
8k + 1k tokens · Measured with v1.0.0
ling · Single node
8 × H100
Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node. +15% on the tuning benchmark vs the baseline variant.
8k + 1k tokens · Measured with v1.0.0
ling · Single node
8 × H100
Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node. Normalized throughput within the SLO vs the baseline variant: +15.5% (50k + 1.5k), +21.3% (8k + 1k).
8k + 1k tokens · Measured with v1.0.0
glm · Single node
8 × B300
Tuned by LLM AutoTune: TP8 on eight B300 GPUs, single node, fp8 KV cache. Normalized throughput within the SLO vs the baseline variant: +12.5% (50k + 1.5k), +8.2% (8k + 1k).
50k + 1.5k tokens · Measured with v1.0.1
qwen · Single node
8 × H100
Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node. Normalized throughput within the SLO vs the baseline variant: +4.3% (50k + 1.5k), +10.0% (8k + 1k).
8k + 1k tokens · Measured with v1.1.0
qwen · Single node
8 × H100
Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node. +6% on the tuning benchmark vs the baseline variant.
8k + 1k tokens · Measured with v1.0.0
gpt · Single node
1 × H100
Tuned by LLM AutoTune: TP1 on one H100 GPU, single node, fp8 KV cache. Normalized throughput within the SLO vs the baseline variant: +6.2% (50k + 1.5k), +2.2% (8k + 1k).
50k + 1.5k tokens · Measured with v1.0.0
glm · Single node
8 × H100
Tuned by LLM AutoTune: TP8 on eight H100 GPUs, single node. +5% on the tuning benchmark vs the baseline variant.
50k + 1.5k tokens · Measured with v1.0.1
mistral · Single node
1 × H100
Tuned by LLM AutoTune: TP1 on one H100 GPU, single node, fp8 KV cache. Normalized throughput within the SLO vs the baseline variant: +1.4% (50k + 1.5k), +5.3% (8k + 1k).
8k + 1k tokens · Measured with v1.0.0
deepseek · Single node
4 × RTX-6000D
TP4 on four GPUs. 149 GiB of weights do not need eight cards: halving them still leaves 61% of the KV pool, because the weights are a fixed cost that does not shrink with the card count. Measured 38.57 GiB of weights per card and max_total_num_tokens 4,002,048, against 6,524,672 at TP8. Those two numbers imply ~34 and ~68 KiB per token respectively, so neither can be used to extrapolate another TP width -- the reason for the gap is not known.
deepseek · Single node
8 × RTX-6000D
TP8 on one eight-GPU node, default MoE runner. Measured max_total_num_tokens 6,524,672.
deepseek · Single node
8 × H100
Baseline: TP8 on eight H100 GPUs, single node.
deepseek · Single node
4 × RTX-6000D
TP4 on four GPUs, the vLLM counterpart of sglang-tp4-6000d. Measured 39.42 GiB per card for weights plus non-torch memory, 33.07 GiB left for KV, GPU KV cache size 3,447,225 tokens -- 3.29x the 1,048,576-token maximum request length. The engine's own suggestion is to raise the KV allocation to 40.08 GiB with --kv-cache-memory, which is left off here because it has not been load-tested.
deepseek · Single node
8 × RTX-6000D
TP8 on one eight-GPU RTX 6000D node. Single-node, so no RDMA and no host networking.
deepseek · Single node
8 × H100
Baseline: TP8 on eight H100 GPUs, single node.
deepseek · Single node
4 × RTX-6000D
NVIDIA's NVFP4 build with TP4 on four RTX 6000D GPUs. Measured: ready in ~6.5 min, 40.94 GB of weights per card, KV ~3.6M tokens. Two fit one node. --moe-runner-backend=marlin is required: the default trtllm backend has no sm120 kernels, and flashinfer_cutlass serves but silently miscomputes (identical temperature-0 requests answer 23*47 as 161, 1721, 1551); marlin was correct and stable. /root/.cache is a hostPath so compiled kernels survive a restart.
deepseek · Single node
8 × RTX-6000D
NVIDIA's NVFP4 build with TP8 on one RTX 6000D node. Measured: ready in ~7 min on a cold cache, KV 6,274,304 tokens, deployed with this chart. --moe-runner-backend=marlin is required: the default trtllm backend has no sm120 kernels, and flashinfer_cutlass serves but silently miscomputes (identical temperature-0 requests answer 23*47 as 161, 1721, 1551); marlin was correct and stable. /root/.cache is a hostPath so compiled kernels survive a restart.
deepseek · Single node
4 × RTX-6000D
NVIDIA's NVFP4 build (NVFP4 routed experts, FP8 elsewhere) with TP4 on four RTX 6000D GPUs, single node. FlashInfer autotune is off: with it on, vLLM v0.30.0 hangs on these cards at the whole-model autotune pass. /root/.cache is a hostPath so compiled kernels survive a restart.
deepseek · Single node
8 × RTX-6000D
NVIDIA's NVFP4 build with TP8 on one RTX 6000D node. Measured in docker: ready in ~10 min on a cold cache, KV 5,222,157 tokens, answers correct; thinking on by default. FlashInfer autotune is off: with it on, vLLM v0.30.0 hangs on these cards at the whole-model autotune pass. /root/.cache is a hostPath so compiled kernels survive a restart.
deepseek · Prefill / Decode disaggregation
RTX-6000D
NVIDIA's NVFP4 build, prefill/decode disaggregated: TP8 per instance on RTX 6000D nodes, NIXL transfer backend, 2 prefill + 1 decode, with L2 KV-offload to mooncakeStore (global segment 100GB, local buffer 4GB). Production-shape values pinned to the harbor v0.30.0-pd-fix2 image, 0.85 GPU memory utilization and 850Gi shm for the large KV offload; node affinity pins pods to RTX-6000D. ModelRoute deepseek-v4-flash publishes the service via the llm-route openresty gateway.
deepseek · Multi-node LWS
8 × RTX-6000D × 2 nodes
TP8 per node, PP2 across two nodes over RoCE. The fp8 e4m3 weights need no MoE backend override. Measured max_total_num_tokens 2,833,152.
deepseek · Multi-node LWS
8 × RTX-6000D × 2 nodes
TP8 per node, PP2 across two nodes over RoCE.
deepseek · Single node
8 × H100
Baseline: TP8 on eight H100 GPUs, single node.
gemma · Single node
2 × H100
Baseline: TP2 on two H100 GPUs, single node.
glm · Single node
8 × B300
Baseline: TP8 on eight B300 GPUs, single node.
glm · Single node
8 × H100
Baseline: TP8 on eight H100 GPUs, single node.
glm · Multi-node LWS
8 × RTX-6000D × 2 nodes
TP8 per node, PP2 across two nodes over RoCE.
glm · Single node
8 × B300
Baseline: TP8 on eight B300 GPUs, single node, modelopt_fp4 weights.
glm · Multi-node LWS
8 × RTX-6000D × 2 nodes
TP8 per node with expert parallel, PP2 across two nodes over RoCE. EP is required, not a tuning choice: plain TP8 cannot load this NVFP4 build. KV pool 1,180,288 tokens, max_req_input_len 1,048,570, and a real 1M-token request completes in 428 s. At 50k/1.5k and concurrency 10: 4,341 input and 126 output tok/s; at 80k/600: 6,722 and 47.6, ahead of the vLLM variant while running 10 concurrent requests against its 17. TTFT is 1.7-3x worse throughout, because prefill queues in 8,192-token chunks.
glm · Prefill / Decode disaggregation
RTX-6000D
Prefill/decode-disaggregated DisaggregatedSet, TP8 PP2 EP8 per role on RTX 6000D over RoCE. EP is required, not a tuning choice: plain TP8 cannot load this NVFP4 build. Default 2 prefill + 1 decode; prefill.replicas is configurable at deploy time (xPyD).
glm · Prefill / Decode disaggregation
RTX-6000D
Prefill/decode-disaggregated DisaggregatedSet, TP8 PP2 EP8 per role on RTX 6000D over RoCE, with local KV offload via hicache (size 100). EP is required, not a tuning choice: plain TP8 cannot load this NVFP4 build. Default 2 prefill + 1 decode; prefill.replicas is configurable at deploy time (xPyD).
glm · Multi-node LWS
8 × RTX-6000D × 2 nodes
TP8 per node, PP2 across two nodes over RoCE, on the same NVFP4 weights as the B300 variant.
glm · Prefill / Decode disaggregation
RTX-6000D
Prefill/decode-disaggregated DisaggregatedSet, TP8 PP2 per role on RTX 6000D, mooncake transfer backend. Default 2 prefill + 1 decode; prefill.replicas is configurable at deploy time (xPyD).
glm · Prefill / Decode disaggregation
RTX-6000D
Prefill/decode-disaggregated DisaggregatedSet, TP8 PP2 per role on RTX 6000D, mooncake transfer backend + local KV offload via mooncakeStore (8 GPUs x 100 GB global segment + 8 x 4 GB local buffer per engine). Default 2 prefill + 1 decode; prefill.replicas is configurable at deploy time (xPyD).
google · Single node
4 × H100
Baseline: TP4 on four H100 GPUs, single node.
gpt · Single node
2 × H100
Baseline: TP2 on two H100 GPUs, single node.
gpt · Single node
1 × H100
Baseline: TP1 on one H100 GPU, single node.
hunyuan · Single node
8 × H100
Baseline: TP8 on eight H100 GPUs, single node.
kimi · Multi-node LWS
8 × H100 / H800 × 2 nodes
TP8 per node, PP2 across two nodes over IB.
kimi · Single node
8 × B300
Baseline: TP8 on eight B300 GPUs, single node.
kimi · Multi-node LWS
8 × RTX-6000D × 4 nodes
TP8 per node, PP4 across four nodes over RoCE. Marlin is the only correct MoE runner here: the default upcasts the MXFP4 experts to bf16 and needs ~6000 GiB, triton_kernel fails in the weight loader, and flashinfer_mxfp4 loads, serves and computes wrong answers. Measured 47.81 GiB of weights per card and max_total_num_tokens 1,237,195.
kimi · Single node
8 × B300
Kimi K3 with SGLang and vLLM on a single eight-GPU B300 node, or vLLM with TP8 and PP4 across four RTX 6000D nodes. The tuned SGLang B300 variant uses DCP8, an fp8 KV cache, DSPARK speculative decoding, and hierarchical KV caching. DSPARK requires engine-compatible Kimi-K3-DSpark draft weights on every eligible node, mounted beside the target model.
kimi · Multi-node LWS
8 × RTX-6000D × 4 nodes
TP8 per node, PP4 across four nodes over RoCE. A Marlin MoE path is required on sm120, to keep the 4-bit weights 4-bit; the default runner upcasts them to bf16 and does not fit. sglang reaches the same shape with --moe-runner-backend marlin.
laguna · Single node
4 × H100
Baseline: TP4 on four H100 GPUs, single node.
ling · Single node
8 × H100
Baseline: TP8 on eight H100 GPUs, single node.
ling · Single node
8 × H100
Baseline: TP8 on eight H100 GPUs, single node.
mimo · Single node
8 × H100
Baseline: TP8 on eight H100 GPUs, single node.
mistral · Single node
1 × H100
Baseline: TP1 on one H100 GPU, single node.
mistral · Single node
2 × H100
Baseline: TP2 on two H100 GPUs, single node.
nemotron · Single node
1 × H100
Baseline: TP1 on one H100 GPU, single node.
nex · Single node
8 × H100
Baseline: TP8 on eight H100 GPUs, single node.
qwen · Single node
1 × Ascend-910B3
Baseline: one Ascend 910B card, single node. vLLM's Ascend backend (CANN), not a CUDA build. Not yet tuned -- no AutoTune run on this hardware.
qwen · Single node
1 × A100
Baseline: bf16, TP1 on one NVIDIA A100 SXM4 80GB GPU, single node. Not yet tuned on this hardware.
qwen · Single node
2 × H100 / H800
TP2 on two H100 or H800 GPUs, single node.
qwen · Single node
8 × H100
Baseline: TP8 on eight H100 GPUs, single node.
qwen · Single node
1 × H100
Baseline: TP1 on one H100 GPU, single node.
qwen · Single node
8 × H100
Baseline: TP8 on eight H100 GPUs, single node.
