vladimir94/dory-1.0-4b-ane-coreml

Dory 1.0 4B — Core ML for the Apple Neural Engine (24,576-token context)

This is a conversion + 8-bit palettization (quantization, not a fine-tune and not a merge) of nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16, split into 102 Core ML programs so that a looped 4B model with gated attention can run on the Neural Engine of a Mac at a 24,576-token context.

Gated attention, the |α| nGPT residual and NEOX RoPE are all represented; nothing is distilled, pruned or retrained. The programs also avoid two places where the Neural Engine loses accuracy: a projection whose inputs are small, and silu/sigmoid. The forms used instead are mathematically equal to the originals.

Measured on an M-series laptop

Against the PyTorch model, positions 64-319 of the evaluation set, 2 loops, decode mode:

	8-bit (this repo)	earlier 6-bit build
mean KLD	0.0145	0.0979
median KLD	0.0039	0.0290
top-1 agreement	94.1 %	88.3 %
decode, 200 tokens incl. prefill,	8.7 tok/s	9.3 tok/s
size	7.6 GB	5.6 GB

Both answer ordinary chat correctly. The earlier 6-bit build is in this repository's history.

What is in here

• meta.yaml — the 102-program plan: dispatch order, per-program files, state declarations, and the gate chain that threads through the stack
• *.mlpackage — the programs themselves (infer and prefill variants)
• rope_tables.npz — RoPE rows as inputs
• tokenizer files and chat_template.jinja — the model's own template, including its thinking mode

Running it

git clone https://github.com/vladimir-voinea/dory-ane && cd dory-ane
python3.12 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt
huggingface-cli download vladimir94/dory-1.0-4b-ane-coreml --local-dir dory-1.0-4b-ane-coreml
python scripts/dory_serve.py --model-dir ./dory-1.0-4b-ane-coreml   # OpenAI API + chat UI

The runtime reads meta.yaml and needs nothing else: it loads one phase of packages at a time, keeps one MLState per stated program, and rejoins the split LM head.

Limits worth knowing before you download

• Prefill and decode cannot be resident together (204 packages fails on this hardware), so a prompt is fed token by token at ~10 tok/s unless you accept a 100–270 s switch to the batch-prefill packages.
• An int8 KV cache is impossible: Core ML states are fp16 only, and integer inputs are rejected, so 999 MiB of fp16 state is the floor for 24K.
• The embedding lookup runs on the GPU, not the Neural Engine, in this build.
• One conversation cache is 999 MiB, so a server holds a couple of sessions, not hundreds.

Built on Anemll. The code is MIT; the weights follow the source model's NVIDIA Open Model License.