flackzz/Ternary-Bonsai-2-9B-ONNX-WebGPU

Ternary Bonsai 2 9B — local Transformers.js export

Text and image Transformers.js export prepared for local inference in hellaw-gpt. hellaw-export.json records every source revision, checksum, modification and output file hash.

Sources and precision

• ONNX graph and ternary weights: danyalmalik/Ternary-Bonsai-2-9B-ONNX-WebGPU.
• Official tokenizer, chat template, generation settings and FP32 gate weights: prism-ml/Ternary-Bonsai-2-9B-mlx-2bit.
• Projection weights retain their exact 2-bit MatMulNBits representation. Token embeddings retain the source's exact 4-bit GatherBlockQuantized representation of the ternary values, including its inverse Hadamard transform.
• All 48 alpha/beta projections are restored from the official FP32 safetensors. For every tensor, transposing and casting to FP16 must reproduce the source ONNX bytes exactly; a mismatched source is rejected.
• Gate arithmetic and all 24 recurrent attention states use FP32. Attention activations cross the surrounding graph boundary in FP16; normal KV and convolution caches retain FP16. Original quantized tensor bytes are unchanged.
• Image features are merged by the application after text embedding; the unused original image ScatterND is removed. Cache names and external-data filenames match Transformers.js, and the missing embedding contrib-opset is added.
• The vision graph derives from the pinned onnx-community/Qwen3.5-4B-ONNX topology. Its geometry is adapted to Bonsai's 27 layers, width 1152 and output width 4096. All 333 learned tensors come from the official Bonsai snapshot; no Qwen 4B weights are retained. Weights remain stored in FP16, while vision arithmetic uses FP32 because the native WebGPU FP16 compute path failed numerical equivalence checks. Constant-folded FP32 vision weights may double GPU weight memory; the fp16 filename/dtype selects stored weights, not FP16 arithmetic. The source pins, tensor mapping and hashes are in vision-export-proof.json.

The official processor metadata omits defaults supplied by the Python Qwen image processor. This export explicitly materializes normalization, rescaling, RGB conversion, resizing and bicubic sampling for Transformers.js. The official source hash and the derived output hash are both recorded in the manifest.

Required application compatibility

Use the pinned Transformers.js 4.3.1 and its matching native WebGPU ONNX Runtime (asyncify) assets. Use AutoModelForImageTextToText with AutoProcessor for images, and AutoModelForCausalLM for text-only loads. Use subfolder: 'onnx' and per-session dtype: embed_tokens and decoder_model_merged q2f16, vision_encoder fp16. Shard counts are in config.json. Keep the vision session unloaded for text-only requests.

The application must apply the fixes in src/app/core/llm/workers/webgpu-text-only-runtime.ts after loading: configureQwen35TextOnlyPositions(model) and stripNullPixelValuesAfterPrepare(model). Version 4.3.1 otherwise adds the prompt width to the text RoPE delta, advancing subsequent positions incorrectly; cached text generation can also insert pixel_values: null and trigger image encoding. Image inference also requires the version-bound configureQwen35ImageFeatureMerge helper in webgpu-image-merge.ts: FP32 image features must be converted to the FP16 text embedding's numeric/storage representation before scatter. This path requires native JavaScript Float16Array support when converting FP32 to FP16. The application also processes Qwen3VL images independently before concatenating patches and grids; Transformers.js 4.3.1 otherwise groups images as video frames. These are local application fixes, not model-hosted executable code.

Render the supplied chat template with enable_thinking set explicitly and use the official tokenizer. Tool calls use the Qwen XML function/parameter protocol. With this tokenizer, skip_special_tokens: true preserves the ordinary XML and thinking markers while omitting the special EOS tokens. Verify token metadata before applying that setting to other tokenizers or model families. The application keeps its normal local-only load and explicit-download policy. ORT may execute mask/position/shape control nodes on the local CPU, while model matrix, Hadamard, embedding and attention computation uses WebGPU.

Scope and qualification

A vision encoder with FP16 weight storage and explicit image processor defaults is included. No audio, video or MTP/speculative-decoding head is provided. A short local Chrome smoke does not establish long-context quality, German legal-domain reliability, memory headroom on other devices, or Edge/Firefox support. See the project's matching integration evidence for the exact tested checks and remaining gaps.

Apache-2.0 applies; original LICENSE and NOTICE.txt are retained. These files are modified exports; the exact changes are documented above and in the manifest.