akhaliq/nemotron-3.5-transcribe-mini-64e

Nemotron-3.5-Transcribe-Mini (REAP-pruned)

The first pruned derivative of NVIDIA's Nemotron-3.5-Transcribe-30B-A4B — the #1 open-weight speech LLM — shrunk with zero retraining. From 67.5GB to 10GB, and faster than the original.

Routed MoE experts were pruned one-shot with REAP (Router-weighted Expert Activation Pruning, arXiv:2510.13999), calibrated on multilingual speech transcripts (FLEURS, 8 languages). No recovery training, no distillation — the audio encoder, speaker encoder, and CTC timestamp head are untouched.

🎙️ Demo: akhaliq/nemotron-transcribe-mini

WER vs the full 30B teacher (300 clips: FLEURS en/de/es + LibriSpeech test-clean, greedy decoding)

Variant	Experts	Size	fleurs-en	fleurs-de	fleurs-es	librispeech	Mean
Teacher	128	67.5GB	5.79%	4.59%	4.37%	1.73%	4.12%
96e	96	~53GB	5.61%	4.65%	4.53%	1.81%	4.15% (+0.7% rel)
64e	64	~38GB	6.73%	5.67%	5.17%	2.75%	5.08% (+23% rel)
64e-W4A16	64	10.0GB	6.29%	6.35%	5.70%	2.28%	5.16% (+25% rel)

96e is within noise of the teacher with 25% of its routed experts removed. The 10GB 4-bit variant costs just +0.08 WER over its BF16 version — quantization is essentially free on top of pruning. A quarter of the experts were dead weight.

Speed (H200, vLLM, greedy, FLEURS clips)

Model	Mean latency	Single-stream RTFx	~90s clip	Burst RTFx ×8
Teacher 67.5GB	0.24s	39.3	0.49s / 107.1	142.1
64e BF16 38GB	0.24s	39.6	0.47s / 112.2	147.8
64e-W4A16 10GB	0.19s	50.4	0.50s / 103.8	171.3

The 10GB model is the fastest configuration: +27% single-stream throughput and +21% burst vs the teacher, while being 6.7× smaller. (RTFx = seconds of audio processed per second. Raw data: eval/speed_results.json.)

Transformers usage 🤗 (verified)

The fused checkpoints load in-process with Hugging Face Transformers, exactly like the teacher:

pip install git+https://github.com/huggingface/transformers librosa av

from transformers import AutoModelForMultimodalLM, AutoProcessor

# download the fused variant first (subdir load)
#   hf download akhaliq/nemotron-3.5-transcribe-mini-64e --include "fused-96e/*"
model_dir = "~/.cache/huggingface/hub/models--akhaliq--nemotron-3.5-transcribe-mini-64e/snapshots/*/fused-96e"

processor = AutoProcessor.from_pretrained(model_dir)
model = AutoModelForMultimodalLM.from_pretrained(model_dir, device_map="auto")

# custom instructions are supported — this is an instruction-following speech LLM
inputs = processor.apply_transcription_request(
    "audio.wav",
    prompt="Transcribe this audio with punctuation and capitalization. "
           "Put speaker tags <spk:0> <spk:1> on the transcription.",
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1024)
print(processor.decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True))

Verified on fused-64e end-to-end. For fast Mamba kernels, also install mamba-ssm and causal-conv1d (otherwise transformers uses slower reference implementations).

The standalone quantized text backbone loads the standard way (text-only — no audio encoder):

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "akhaliq/nemotron-3.5-transcribe-mini-64e",
    subfolder="backbone-64e-w4a16-g64", device_map="auto")
tok = AutoTokenizer.from_pretrained(
    "akhaliq/nemotron-3.5-transcribe-mini-64e", subfolder="backbone-64e-w4a16-g64")

Serving (vLLM, same stack as the teacher)

pip install vllm==0.32.0
pip install "nemo-toolkit[asr-only,core,common-only] @ git+https://github.com/NVIDIA-NeMo/Speech.git"
hf download akhaliq/nemotron-3.5-transcribe-mini-64e --include "fused-64e-w4a16-g64/*"

export VLLM_PLUGINS=nemo_speechlm
vllm serve ~/.cache/huggingface/hub/models--akhaliq--nemotron-3.5-transcribe-mini-64e/snapshots/*/fused-64e-w4a16-g64 \
  --served-model-name nemotron-mini \
  --trust-remote-code --dtype bfloat16 \
  --gpu-memory-utilization 0.9 --max-model-len 8192

The 10GB W4A16 fused checkpoint serves via vLLM's compressed-tensors Marlin kernels — it fits a 12GB consumer GPU. Transcribe with the OpenAI chat API (audio as base64 data URI), exactly as in the teacher's model card.

Layout

• fused-64e-w4a16-g64/ — 10GB servable 4-bit speech model (W4A16 backbone + BF16 audio encoder), WER + speed verified
• fused-96e/ — servable 96-expert checkpoint (BF16, near-lossless)
• fused-64e/ — servable 64-expert checkpoint (BF16)
• backbone-64e-w4a16-g64/ — 10.0GB W4A16 INT4 quantized 64e backbone (text-only)
• backbone-96e/, backbone-64e/ — standalone HF NemotronH text backbones
• eval/ — raw transcripts and speed data from every eval run
• scripts/ — the exact eval/release/quantization scripts used

Capabilities preserved

40 languages, recordings up to 1 hour, speech translation, speaker-attributed transcription, word timestamps (CTC head), contextual biasing, instruction-controlled transcription style, audio Q&A — all inherited unchanged; only the LLM's routed experts were pruned.

Method

1. Extract the NemotronH backbone from the fused NeMo checkpoint.
2. REAP saliency per expert (router weight × activation norm) over ~2,500 multilingual FLEURS transcripts, one calibration pass, layer-wise.
3. Drop the lowest-saliency experts per MoE layer (128→96 / 128→64), slice router rows + correction bias, keep shared experts, MTP head, and all dense layers.
4. Reassemble the fused SALM checkpoint with the original audio encoder.
5. Model-free W4A16 quantization of the pruned backbone — zero calibration, router/embeddings/lm_head/Mamba scalars kept at full precision.

Limitations

• 64e loses ~1 WER point vs the teacher; 96e is the recommended default for datacenter GPUs, 64e-W4A16 for consumer GPUs.
• Evaluated on short clips (≤30s) plus one ~90s clip; long-form behavior is expected to track the teacher but is not yet measured at full 1-hour length.
• License: OpenMDW-1.1 (inherited from the teacher).