Multi-GPU llama.cpp benchmarks — 2× RTX 5060 Ti 16GB: pp/tg ladder, max context up to 393k, parallel tests up to 64k (UD-Q4_K_XL)

#24
by rafaelsobek - opened

Inkling 2 Mini on llama.cpp — multi-GPU benchmark results (2× RTX 5060 Ti 16 GB)

We benchmarked Inkling-2-Mini-UD-Q4_K_XL (Mainline_Support variant) on mainline
llama.cpp across two 16 GB consumer GPUs and published the full reproducible setup
(docker compose, patched image, scripts, raw data).

Setup: 2× RTX 5060 Ti 16 GB (Blackwell sm_120, layer-split + auto-fit) · Ryzen 5 7500F ·
llama.cpp @ 2a28df0f9bd5 (inkling2_loop layout #32694 + scale-reshare fix #32704 — the fix is
required, otherwise loop passes 2/3 lose their per-expert NVFP4 scales) · flash-attn on ·
KV q8_0 · ubatch 256 · 12 CPU threads · alias inkling-2-mini

Depth ladder (fresh requests, temp 0, 128 gen)

Prompt depth pp (t/s) tg (t/s)
2.6k 475 22.5
8k 459 23.0
16k 419 22.0
32k 360 23.0
64k 280 20.2
96k 225 19.7
128k 190 17.2

Max context (per-slot, np=1)

ctx KV cache ubatch VRAM GiB/GiB tg (t/s) Note
131,072 q8_0 256 13.3 / 14.6 24.7 comfortable default
262,144 q4_0 128 13.3 / 14.6 24.7 full speed at 256k!
393,216 q4_0 128 14.8 / 14.8 17.6 tight split, still runs
524,288 q4_0 ≤128 — — OOM

Parallel requests (np=4 → per-slot KV, 4× 32k slots)

  • 4 concurrent generations (~2.6k prompt each): tg fairly time-sliced at ~9 t/s per stream,
    aggregate ≈ 22.5 tok/s ≈ single-stream — decode is routing/loop-bound, not bandwidth-bound
  • 4× ~14.5k prompts (64k combined prefill): aggregate PP 485 tok/s, +23 % vs sequential
    (batched prefill + MoE activation pay off — unlike our dense 27B baseline)

Multi-GPU findings worth knowing

  1. -sm tensor is not implemented for inkling2_loop — layer-split is the only multi-GPU path
  2. Setting TENSOR_SPLIT or N_GPU_LAYERS aborts the auto-fit → the loader tries to allocate
    the whole 16 GB model on CUDA0 → OOM. Leave both unset.
  3. The MoE compute buffer scales with context: ubatch 512 only fits up to ~64k ctx;
    131k needs 256, 262k needs 128
  4. llama-bench has the same monolithic-allocation problem on multi-GPU → we benchmarked via
    the server API (scripts included)
  5. tg ≈ 22–23 t/s is on par with a dense 27B on the same rig: the loop runs the middle 12
    layers 3× (48 applications/token, ~80 % routing/NVFP4-dequant overhead), so VRAM is saved but
    compute isn't. DSpark speculative decoding (PR #32218) should be the big lever once it lands.

Links

Questions, corrections or results from other multi-GPU rigs are very welcome!

Sign up or log in to comment