flackzz/gemma-4-E2B-it-qat-mobile-SRQ-ONNX

Gemma 4 E2B Mobile-QAT with optimized decoder SRQ (ONNX)

This derivative restores and optimizes calibrated Static Range Quantization (SRQ) around the quantized decoder linears. It retains the original mixed 2/4/8-bit weights and FP16 activation format.

Experimental. E2B remains experimental after failing task-specific proofreading and formatting checks. This arithmetic optimization does not constitute a quality fix. No claim of MLX output parity is made.

This revision

• Previous published revision: d65e5c50805defdda2685b24df01d4d338a584df.
• SRQ now uses 15 standard ONNX operations per chain, with explicit nearest-even decisions and signed-zero preservation. It does not use the hardware-dependent 8-operation large-offset rounding prototype.
• Identical input tensor/effective FP16 scale combinations share their SRQ result: 30 duplicate input chains removed, leaving 450 chains across 240 calibrated linears.
• Decoder graph nodes change from 9,900 to 8,490. Node counts are not measured GPU-dispatch counts or a performance claim.
• Original decoder initializer bytes, packed MatMulNBits weights and every external weight blob remain unchanged. Embedding, vision/audio encoders, tokenizer, processor, configuration and chat template remain unchanged.
• Original Google F32 calibration is cast directly to FP16; there is no BF16 intermediate conversion or new calibration. Fused gate/up projections and shared-KV behavior are preserved. The uncalibrated LM head is unchanged.

The q2f16 suffix is the upstream filename convention; this package is not uniformly 2-bit. Source model context metadata remains 131,072 tokens. That maximum is not a browser-memory or task-quality guarantee. Text, image/audio, thinking and function-call capabilities require separate application paths and qualification.

Rounding contract

q = clip(FP16(x / FP16(scale)), -128, 127)
b = floor(q)
f = FP16(q - b)
even = (b * 0.5 == floor(b * 0.5))
increment = f > (0.5 if even else previous_float16(0.5))
r = b + FP16(increment)
r = q * 0 if r == 0 else r
result = FP16(r * FP16(scale))

For a materialized FP16 fraction, f > previous_float16(0.5) is exactly f >= 0.5. Even integer bases therefore retain ties; odd bases increment them. This preserves the explicit nearest-even decision without assuming a particular GPU arithmetic rounding mode for a large-offset trick. Division and rescaling retain the same FP16 boundaries as the preceding SRQ version. ONNX Round is avoided because it fell back to CPU in the tested WebGPU runtime.

Provenance

Component	Immutable source
ONNX source	onnx-community/gemma-4-E2B-it-qat-mobile-ONNX, a7f19b0ab28252b77cea830f52b8290b2479c87d
Original calibration	google/gemma-4-E2B-it-qat-mobile-transformers, dd693ff40353f057ca5f07e945ad867f4afbf2ec
SRQ reference	Transformers gemma_quant.py

See srq-manifest.json for all runtime file hashes, source revisions, scalar mappings and shared-input provenance. The Google checkpoint file hash comes from pinned public LFS metadata; only the scalar ranges and header were downloaded and hashed locally. The included extracted calibration hash is b041ccf9aeee9a95b06427ebd154d366ec6c542f5efb0e280a68c042a5a63eb0.

Validation

• Ten numerical/integrity tests passed. For E2B, all 63,488 finite FP16 bit patterns at all 397 effective calibrated decoder scales produced 25,204,736 bit-identical ONNX-reference results, including signed zero and saturation.
• The same exhaustive FP16 input set was checked at three representative scales on actual WebGPU with CPU fallback prohibited. All 190,464 final outputs matched bitwise. All 45 test nodes ran on WebGPU. This covers the isolated SRQ graph, not provider placement of the complete model.
• Static structure checks pass. Existing ORT custom operators are not independently schema-verified.
• The production app worker completed two synthetic proofreading cases per variant (with and without tool definitions). The old and optimized graphs produced identical response-text hashes, with no proposed tools, errors, warnings or external requests. These are bounded regression checks, not a task-quality qualification.
• Model regression and performance results for the selected revision are recorded in validation.json. Historical 8-operation prototype timings do not describe this 15-operation implementation.
• CPU execution is unqualified: ORT CPU can promote intermediate FP16 operations, altering rounding boundaries.

The prior published model checks and publisher feedback are separately labeled in validation metadata. They do not replace optimized-model regression checks, cross-device tests, long-context checks or multimodal qualification. No private prompts, documents or response transcripts are included.

Measured decode performance

On one Apple M5 with 32 GiB unified memory and Chrome 154, the selected 15-operation graph produced these warm-run median results versus the preceding 17-operation SRQ graph:

Input length	Prior SRQ	Optimized SRQ	Change
31 tokens (3 runs)	19.65 tokens/s	21.79 tokens/s	+10.9%
985 tokens (2 runs)	19.64 tokens/s	21.80 tokens/s	+11.0%

Each measured generation produced 96 tokens with greedy sampling, after one excluded 16-token warmup. Decode rate uses the 95 streamed token intervals after the first token; it excludes prefill and includes runtime/sampling/streamer overhead. Response-text hashes matched between variants. Variants were measured sequentially, using the same diagnostic worker without full app orchestration; a background public-model download continued. The small sample and single device do not establish a cross-device speed guarantee, a 16 GiB memory qualification or universal token/logit parity.

Reproduce offline

Prepare the complete pinned upstream ONNX snapshot in downloaded-onnx/. With onnx==1.23.2 and numpy==2.5.4, run from this repository root:

python conversion/restore-gemma4-mobile-onnx-srq.py \
  --model e2b \
  --source-root downloaded-onnx \
  --calibration-json conversion/original-calibration.json \
  --source-manifest conversion/source-artifacts.json \
  --output-root optimized-export \
  --optimize-srq

The converter verifies immutable source/calibration hashes and refuses an existing destination. Omit --optimize-srq only to reproduce the prior 17-operation graph. Conversion is offline and requires no GPU.

Loading

Target runtime: Transformers.js 4.3.1, WebGPU with FP16 shader support, and sufficient device memory. Retain the original dtype mapping: embed_tokens: q2f16, decoder_model_merged: q2f16, vision_encoder: fp16, audio_encoder: q2f16. This repository is public. Pin a verified commit for deployments instead of tracking main.

Downloading model/runtime assets requires network access initially. After complete installation, inference can run locally and offline; configure the application to use its local cache or local model directory. The original encoder assets remain available, but this decoder optimization alone does not qualify image/audio behavior.

License

Original model: Google DeepMind. Original ONNX conversion: onnx-community. Original F32 calibration: Google. SRQ derivative: flackzz. See LICENSE for Apache-2.0 and NOTICE for unchanged attribution and the additional modification record.