LakoMoor/Nanbeige4.3-33B-IQ3_XXS-NInfer-v3

Nanbeige4.3-33B IQ3_XXS — NInfer v3

Native .ninfer conversion of Nanbeige4.3-33B, using AtomicChat's dynamic Nanbeige4.3-33B-AD-IQ3_XXS.gguf. Tested on a Tesla V100-SXM2-16GB with CUDA 12.9. The recommended interactive profile uses an 8192-token window. The weights retain the original model's configuration; the profile is a runtime memory/latency choice.

This is an NInfer v3 artifact, not a GGUF or Transformers checkpoint. Tokenizer, generation config and official chat template are embedded. No original safetensors or n-gram table are needed. Use the engine revision below; an older NInfer build without the explicit Llama family cannot load this model.

Provenance and representation

• Original model: Nanbeige/Nanbeige4.3-33B, config/tokenizer revision df919e1e2cf077a22204c78b6c826bf3db18bc08.
• Quantization: AtomicChat/Nanbeige4.3-33B-GGUF, revision 560e151e11dfdd4b008d394f6e16bf196a952bd0.
• Source file: Nanbeige4.3-33B-AD-IQ3_XXS.gguf.
• Converter recipe: llama_gguf; model conversion/qualification by LakoMoor.
• Engine: NInfer Llama/Nanbeige implementation, reproducible source revision 807a99ec065f47b29635834e2d13620aad607562.

The source's encoded matrix blocks are retained without requantization. GGUF Llama Q/K rotary pairs are converted to NeoX half-head order by moving complete encoded rows. Norms widen to FP32. The dynamic IQ3_XXS label does not mean every tensor has one uniform three-bit codec: this artifact mixes IQ3_S, IQ3_XXS, IQ4_XS, Q5_K, Q6_K and FP32.

Architecture: dense LlamaForCausalLM, 33,118,977,024 stored model parameters, 52 layers, hidden size 7168, FFN width 22016, 56 query heads / 8 KV heads, head width 128, unscaled RoPE theta 120,000,000, no Q/K norm, attention gate or sink. The source config declares 524288 native positions; this is not a claim that full-prompt inference or quality has been tested there.

Download

hf download LakoMoor/Nanbeige4.3-33B-IQ3_XXS-NInfer-v3 --local-dir ./models/nanbeige
(cd models/nanbeige && sha256sum -c SHA256SUMS)

Nanbeige4.3-33B-IQ3_XXS.ninfer is a single self-contained file of 14,958,925,568 bytes (about 13.93 GiB). This repository also includes its conversion report, provenance, license, checksums and a portable launch script. It contains no CUDA executable.

Build the compatible engine for V100

Install the dependencies described in the engine's Linux build guide. Use CUDA 12.8 or 12.9 for V100; CUDA 13 cannot compile sm_70. For example, with CUDA 12.9 selected in your environment:

git clone --branch codex/nanbeige https://github.com/LakoMoor/ninfer-all.git
cd ninfer-all
git checkout 807a99ec065f47b29635834e2d13620aad607562
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=70 \
  -DNINFER_BUILD_APPS=ON -DBUILD_TESTING=OFF -DNINFER_BUILD_BENCHMARKS=OFF
cmake --build build --target ninfer-serve -j 8

Fast interactive profile

From the engine checkout created above (the download folder is beside the checkout):

NINFER_BIN="$PWD/build/apps/ninfer-serve" sh ../models/nanbeige/run-v100-fast.sh

Or invoke the engine directly:

CUDA_VISIBLE_DEVICES=0 build/apps/ninfer-serve \
  ../models/nanbeige/Nanbeige4.3-33B-IQ3_XXS.ninfer \
  --model-id nanbeige4.3-33b --host 0.0.0.0 --port 8000 \
  --max-context 8192 --kv-capacity 8192 --max-concurrency 1 \
  --prefill-chunk 512 --host-cache-mib 2048 --no-cuda-graph \
  --no-thinking --default-max-tokens 2048 --lookup-ngram 4

The API is OpenAI-compatible at /v1; the model id is nanbeige4.3-33b. Thinking is off by default in this profile and can be enabled explicitly per request. Context lookup uses three automatic drafts. Do not add --draft-tokens without a separate --spec backend: the existing CLI rejects that combination.

The executor fits resident F16 KV pages to free VRAM while retaining 256 MiB of planning headroom, then transfers only populated rows from host KV. On the tested V100 it kept the first 4096 positions in VRAM (832 MiB KV); startup reported 377.5 MiB free. Extra processes can reduce the selected prefix. The bounded host prefix-image budget is 2 GiB, separate from active KV, which can reach 1.625 GiB at 8k. All attention arithmetic runs on CUDA. Execution is eager; this implementation does not capture decode CUDA Graphs.

Only one GPU and one active sequence are supported by this Llama path. No vision, scaled RoPE, separate draft-model backend or multi-GPU execution is implemented.

Measured speed and limits

Tesla V100-SXM2-16GB, CUDA 12.9, sm_70 Release, same artifact, greedy sampling, cache reuse off. Short outputs were truncated at 160 tokens, with two runs per workload:

Workload	Original fixed-1024 resident KV, no lookup	Fast 8k profile
LRU-cache code, 57 input tokens	32.32 / 32.32 tok/s	32.30 / 32.32 tok/s
Russian HTTP-cache explanation, 70 input tokens	31.79 / 31.78 tok/s	31.49 / 31.50 tok/s
Repeated square functions	31.92 / 31.90 tok/s	51.30 / 51.35 tok/s
2048-token reference continuation, 32 output tokens	4.44 tok/s	36.86 tok/s
6144-token reference continuation, 32 output tokens	2.56 tok/s	13.28 tok/s

Every measured greedy output matched its baseline exactly. The reference continuations are repetitive and favor lookup: do not generalize their rates to arbitrary text at the same context length. Ordinary code was essentially unchanged, and ordinary prose slowed about 0.9%. Different rows have different prompt/output shapes. No stable 60+ tok/s claim is supported. These are throughput checks, not broad model-quality evaluations; disabling thinking also changes the default answering policy.

Long contexts trade throughput for memory capacity. The original fixed-1024 implementation backed a full 256k F16 cache (52 GiB of host RAM) and executed the last legal token with finite logits, but that synthetic full-cache token took 19.35 seconds, about 0.052 tok/s. That capacity result predates the expanded-residency optimization. A complete real 256k prompt and broad long-context quality remain untested. The recommended fast profile intentionally caps the window at 8k.

Validation

• 15 Llama/Flint converter tests and affected Python 3.11 compilation checks passed.
• Synthetic and real Metaspace tokenizers each passed 20 independent Hugging Face cases.
• Independent FP64 numerical tests covered RMSNorm at width 7168, RoPE through position 262143, F16 K/V storage, causal GQA 56/8 and segmented attention over 262144 keys.
• Full-model snapshot/image/branch continuations matched exactly across cache boundaries at positions 1024, 3072 and 4096.
• Against llama.cpp on identical 20-token and 1537-token inputs: same top token, 19/20 top-token overlap, reference-to-native KL 5.66e-5 and 2.50e-6. Raw logit RMS differences were 0.0842 and 0.3714 (max 0.3304 and 0.9098); arithmetic paths are not bit-identical.
• The optimized 8k synthetic capacity check executed position 8191 with finite logits and rejected the next token. Its last-position plain decode took 0.550 s.
• API tool calls and a real server OpenCode read call passed under the fast 8k profile.

See PORT_NOTES.md for the implementation evidence and qualification boundaries.

License and attribution

Apache-2.0, following the original model and AtomicChat's GGUF release. See LICENSE and NOTICE. Credit for model training belongs to Nanbeige; credit for the source quantization belongs to AtomicChat. This release packages their weights for the explicit NInfer Llama runtime and does not claim to be an official Nanbeige or AtomicChat release.