Duckets/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-MLX-4bit

NVIDIA Nemotron 3.5 Bolt 74B-A9B — MLX 4-bit

An MLX 4-bit conversion of NVIDIA's Nemotron 3.5 Bolt 74B-A9B (BF16), built so the model runs natively on Apple Silicon with mlx-vlm and in LM Studio.

This is the full omni model, vision included — the Nemotron-H hybrid language model plus its C-RADIO vision encoder, converted from the BF16 source (not from a GGUF). Bolt was compressed/distilled from Nemotron 3.5 Super VL by NVIDIA using Star Elastic. The source checkpoint has no audio tower (sound_config is null), and neither does this one.

Conversion details

Source	nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16
Quantization	MLX affine 4-bit, group size 64, ~4.61 bits/weight
Size on disk	40.2 GiB, 9 safetensors shards
Total / active parameters	74B / 9B
Architecture	Nemotron-H hybrid (Mamba-2 hybrid LatentMoE) + C-RADIO vision encoder
MoE geometry	Elastic (Star Elastic): 40 MoE layers with per-layer expert counts of 240–496 experts per layer (240×12, 304×6, 368×9, 432×10, 496×3), intermediate sizes 1536–2560, top-9 — the true geometry is recorded in block_configs in config.json; the flat expert fields in llm_config are Megatron export artifacts and do not describe it
Context length	262,144 tokens
Inputs	Text and images (video frames are supported upstream; verified here for text + image)
Output	Text

Config fixes applied versus a raw conversion:

• Removed the redundant _downsample_factor key from the processor configs (the value is derived from downsample_ratio; the literal key breaks NVIDIA's own processor code, which exposes it as a read-only property).
• Normalized time_step_limit's Infinity sentinel to a plain number (1e30) so strict JSON parsers accept config.json.
• Added a top-level max_position_embeddings (262144) so front-ends report the true context length instead of a 4096 default.
• The shipped chat_template.jinja is null-hardened so strict jinja renderers (LM Studio's) don't fail on tool schemas containing null values. Output is unchanged for non-null input.
• Elastic MoE handling: this conversion was produced with a patched mlx-vlm nemotron_h_omni package that builds and stacks each MoE layer with its own per-layer geometry from block_configs. Stock mlx-lm/mlx-vlm nemotron_h code assumes uniform MoE layers.

Vision projector note: Bolt's source checkpoint ships without vision final-layernorm (vision_projector) weights, so that module is absent here by design — the model code guards the call (projector is skipped when absent), and vision is verified working regardless.

Verified

• Generates in mlx-vlm — text and image description both tested.
• Loads and serves in LM Studio at the full 262,144 context, with text and vision answers verified through LM Studio's own API.
• Loads on a 128 GB Apple Silicon Mac in about 10 seconds.

Usage

With mlx-vlm:

pip install -U mlx-vlm

python -m mlx_vlm.generate \
  --model Duckets/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-MLX-4bit \
  --prompt "Describe this image." \
  --image /path/to/image.png

With LM Studio: download the repo into your LM Studio models folder and load it from the MLX models list. Two things to know:

• Load with KV Cache Quantization OFF — with it on, LM Studio's mlx-vlm batched-vision path rejects the load / ignores the context-length setting.
• This architecture is newer than some LM Studio backends: at upload time, a stock backend was missing the nemotron_h_omni mlx-vlm model package and two small processor dependencies, so a plain load could fail on engine grounds alone. That is an engine-side gap, not a problem with these weights — the same files run as-is under the patched mlx-vlm described above. Newer LM Studio builds may already include what's needed.

Reasoning is configurable through the chat template (enable_thinking), as in the upstream release.

Limitations

• 4-bit quantization can differ slightly from the BF16 original on borderline outputs; use the upstream BF16 release for full-precision evaluation.
• Vision was verified with images. Video input follows the upstream design but was not part of this port's verification.
• No audio input or output — the source model has no audio tower.
• Re-converting from the BF16 source requires the elastic-MoE-patched nemotron_h_omni package described above; stock converters assume uniform MoE geometry.

License

Use is governed by the upstream license: the OpenMDW License Agreement, version 1.1. This is a derivative conversion of NVIDIA's model; all credit for the model itself, its training data, and its capabilities belongs to NVIDIA. See the upstream model card for the full description, benchmarks, and training details.

---

Conversion and packaging by Juno for Duckets, 2026-10-15.