Benchmarking vLLM on 2× RTX 3090 — Pre-NVLink: What PCIe Costs You

The PCIe Baseline

The Six Pre-NVLink Sweeps

#	Date	Sweep	Models	Key finding
1	2026-06-23	TP=2 anchor	8 models	qwen3-coder-30b winner (150.6 tok/s, 256K ctx)
2	2026-06-25	Ampere salvage	glm47-flash, gpt-oss	fp8e4nv fix → GLM viable (110 tok/s), gpt-oss dead (11 tok/s)
3	2026-06-28	Concurrency scaling	main models	c4 aggregate throughput reported
4	2026-06-29	GLM single-GPU	glm47-flash	Fits 1 GPU (107 tok/s, 89K ctx)
5	2026-06-29	gpt-oss nightly	gpt-oss-20b	Needs newer image; original runs eager-only at ~11 tok/s
6	2026-06-29	qwen36-27b TP=2	qwen36-27b	TTFT floor discovery (~4.2s at TP=2, dominated by PCIe)

The Winning Tier: Models Under 2500ms TTFT at TP=2

Model	Decode tok/s	TTFT p50 ms	GPUs	Max usable ctx	Type
qwen3-coder-30b	152.94	629	2	256K	MoE coding specialist
qwen36-35b-a3b	136.31	990	2	192K	MoE multimodal
qwen3-coder-next-60b-ream	118.61	1203	2	256K	MoE coding specialist
glm47-flash (TP=2)	110.95	1181	2	192K	MoE coding agent
glm47-flash (1 GPU)	107.05	795	1	89K	MoE coding agent
gpt-oss-20b-nightly	64.81	855	2	128K	Dense (nightly image)

Models That Failed the TTFT Budget

Model	Best decode tok/s	Best TTFT p50 ms	GPUs	Why it failed
qwen36-27b (TP=2)	~59.7	~4215	2	NCCL all-reduce over PCIe dominates every prefill
qwen36-27b (1 GPU)	~40	~3700	1	Dense model, not coding-specialized; TTFT still over budget
devstral-24b-gptq	54.90	3239	2	General multimodal assistant; TTFT over budget
devstral-24b-fp8	53.43	3131	2	Same as GPTQ variant; FP8-native weights
gpt-oss-20b (original)	11.11	—	2	--enforce-eager required on Ampere → ~14× slower than Qwen winners
qwen3-coder-next-80b	—	—	2	OOM in all attempts — too large for 2× 24 GiB

The PCIe TTFT Floor

Ampere (sm_86) Compatibility Gotchas

Model	Override	Why
glm47-flash	--kv-cache-dtype auto	fp8e4nv unsupported in grouped-attention KV kernel on sm_86
gpt-oss-20b (original)	--enforce-eager --kv-cache-dtype auto	(1) torch.compile/Inductor autotune failure on MXFP4 MoE kernel; (2) fp8e4nv in reshape_and_cache_kernel_flash

Concurrency Scaling: What Happens When Two Agents Ask at Once

Model	c1 decode tok/s	c2 agg tok/s	c4 agg tok/s	Note
qwen36-35b-a3b	136	271	542	Highest aggregate; DeltaNet produces 1.88M-token KV pool
qwen3-coder-30b	150	253	399	Best single-user + strong batching
qwen3-coder-next-60b-ream	119	191	368	Confirmed in retest
glm47-flash (TP=2)	98	—	—	Crashes under sustained concurrent load
glm47-flash (1 GPU)	100	—	—	Same crash behaviour

Maximum Usable Context: What Each Model Fits

Model	Max usable ctx	KV cache tokens	Decode tok/s at ceiling	GPUs
qwen3-coder-30b	256K	535632	150.04	2
qwen3-coder-next-60b-ream	256K	645438	118.49	2
qwen36-35b-a3b	192K	1877606	135.90	2
glm47-flash (TP=2)	192K	236768	99.59	2
qwen36-27b (TP=2)	192K	801326	59.68	2
devstral-24b-gptq	128K	199952	54.62	2
gpt-oss-20b-nightly	128K	1101438	42.96	2
glm47-flash (1 GPU)	89K	120448	82.92	1
qwen36-27b (1 GPU)	32K	70390	39.84	1

Pre-NVLink: What You’d Actually Deploy

Tier 1 — Daily driver:

• qwen3-coder-30b (TP=2, 153 tok/s, 256K ctx, non-thinking mode)
• qwen36-35b-a3b (TP=2, 136 tok/s, 192K ctx, multimodal)

Tier 2 — Diversity / fallback:

• glm47-flash (single GPU, 107 tok/s, 89K ctx) — co-locate with qwen30b on GPU 1
• qwen3-coder-next-60b-ream (TP=2, 119 tok/s, 256K ctx) — quality upgrade, slower
• gpt-oss-20b-nightly (TP=2, 65 tok/s, 128K ctx) — requires nightly image

Tier 3 — Skip on PCIe:

• devstral-24b (TTFT over budget at both 3.1–3.2s)
• qwen36-27b (TTFT over budget, not a coding specialist)
• gpt-oss-20b original image (11 tok/s eager-only)
• qwen3-coder-next-80b (OOM)

What Changes with NVLink