Arete-27B-FP8

FP8 (e4m3, 128×128 block) checkpoint of NousResearch/Arete-27B, laid out tensor-for-tensor like Alibaba's own Qwen/Qwen3.8-27B-FP8. It loads in vLLM with no flags and no load-time quantization, and halves the download against the BF16 original (28.3 GiB vs ~54 GiB).

Arete-27B is Nous Research's supervised fine-tune of Qwen3.8-27B. This repo changes nothing about the model's behaviour on purpose: it is the same weights, stored in 8 bits where Qwen's team stored theirs in 8 bits.

Why this exists

Arete ships in BF16 only. Serving it on two 48 GB cards means either --quantization fp8 at load time, which quantizes every linear layer (including the tiny DeltaNet gate projections and the vision merger that Qwen's checkpoint deliberately keeps in BF16) and triggers Marlin thread-tile padding warnings on Ampere, or a pre-quantized checkpoint. None existed in standard FP8, so this is one.

How it was made

arete_fp8.py (in this repo) walks Qwen3.8-27B-FP8's safetensors headers and applies one rule per tensor:

• If Qwen stores the tensor as F8_E4M3, block-quantize Arete's tensor of the same name: 128×128 blocks, scale_inv = amax / 448 stored in BF16 as <name>_scale_inv, exactly the shapes Qwen uses.
• Otherwise copy Arete's tensor unchanged (norms, embeddings, lm_head, the vision tower, in_proj_a/b/ba, and everything else in Qwen's modules_to_not_convert).

config.json is Arete's, with Qwen's quantization_config block inserted. Tokenizer, chat template and generation config are copied from Arete as-is. No calibration data; this is static weight quantization, the same math as the Qwen checkpoint.

The script fails if any shape or name disagrees with Qwen's layout, and its last step compares the output headers to Qwen's. Result on this build:

plan: quantize 400  copy 784  skip qwen-only 15 (all mtp.*)
rel-err model.language_model.layers.0.linear_attn.in_proj_qkv.weight: 0.0264
rel-err model.language_model.layers.0.linear_attn.in_proj_z.weight:   0.0264
rel-err model.language_model.layers.0.linear_attn.out_proj.weight:    0.0263
output tensors 1584  total 28.3 GiB
LAYOUT MATCH: identical to Qwen3.8-27B-FP8 (minus mtp.*)

rel-err is the relative Frobenius error between the dequantized block and the BF16 original, measured on the first three quantized tensors. ~2.6% is the expected figure for e4m3's 3-bit mantissa.

MTP head: Arete was trained with the multi-token-prediction head disabled and does not ship mtp.* weights, so neither does this checkpoint. Speculative decoding via MTP is not available; everything else is unaffected.

Serving

Tested with vllm/vllm-openai 0.33.0 (CUDA 13) on 2× RTX A6000 (Ampere, sm_86) over NVLink:

vllm serve processreap/Arete-27B-FP8 --served-model-name arete-27b \
  --tensor-parallel-size 2 --max-model-len 65536 --gpu-memory-utilization 0.90 \
  --enable-prefix-caching --language-model-only \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking": false}'

No --quantization flag: vLLM reads it from config.json. On that hardware: Selected MarlinFP8ScaledMMLinearKernel for Fp8LinearMethod, zero padding warnings, 14.6 GiB weights per GPU, 806k-token KV cache at 65k context (12.3× concurrency). Ampere has no native FP8 compute, so Marlin runs weight-only FP8; on Hopper or later the same checkpoint should use native FP8 GEMMs.

Drop --language-model-only to load the vision tower too. It is copied unquantized from Arete, but I have not exercised the vision path on this checkpoint.

What was and was not checked

• Loads and matches layout: yes, as above.

• Agentic tool use: 11/11 passes on an internal threat-hunting agent benchmark (OpenAI-compatible tool calling through vLLM, 4–6 tool calls per run, strict output grader), identical to the load-time-quantized Arete and within noise of Qwen3.8-27B-FP8 on the same bench. That is one narrow task, not a general evaluation.

• Nous's value-alignment results: not re-run. If the framing behaviour Arete was trained for matters to you, verify it against NousResearch/Propaganda_Eval rather than assuming quantization preserved it exactly. Block FP8 at this error level normally does, but I measured weights, not that eval.

• Throughput: identical to Qwen3.8-27B-FP8 on the same hardware and settings, as expected for the same layout through the same kernel. vllm bench serve, random 3000-token prompts / 256-token outputs, 16 requests at concurrency 4, 2× RTX A6000 TP=2:

  	Qwen3.8-27B-FP8	Arete-27B-FP8 (this repo)
  Mean TTFT	2318.6 ms	2318.8 ms
  Mean TPOT	32.75 ms	32.69 ms
  Output throughput	95.8 tok/s	95.9 tok/s

  Not measured: the BF16 original, or --quantization fp8 at load time, so this card makes no claim about speed relative to those.

Reproducing

python3 arete_fp8.py --qwen <Qwen3.8-27B-FP8 snapshot> --arete <Arete-27B snapshot> --out ./Arete-27B-FP8

Runs on CPU (torch + safetensors, no GPU), ~90 s for the full model with the shards in page cache. Needs ~30 GB free disk. --dry-run prints the plan and validates shapes without writing.

Credits and license

• Base model and all of the training: Nous Research, Arete-27B, Apache-2.0.
• Starting model and FP8 recipe: Alibaba's Qwen team, Qwen3.8-27B and Qwen3.8-27B-FP8, Apache-2.0.
• Quantization: this repo. Apache-2.0, same as upstream.

This is a community quantization, not affiliated with or endorsed by Nous Research or Alibaba.