rina-llama.cpp

A llama.cpp build for ternary GGUF models on x86 CPUs, maintained by Rinstitute. It is the PrismML llama.cpp fork (branch prism, which tracks upstream llama.cpp) plus tq16, a set of CPU kernels for the three ternary formats used by Rina Juvenile Exp: PQ2_0, PTQ1_0 and Q2_0.

  • 中文:面向 x86 CPU 的 llama.cpp 构建,为三值 GGUF(PQ2_0 / PTQ1_0 / Q2_0)加了 AVX2 / AVX-VNNI / AVX-512 内核。
  • 日本語:三値 GGUF(PQ2_0 / PTQ1_0 / Q2_0)向けに AVX2 / AVX-VNNI / AVX-512 カーネルを追加した、x86 CPU 用の llama.cpp ビルドです。

Release 2026.11.14

File Contents
rina-llama.cpp-2026.11.14-linux-x64.tar.gz Linux x86-64, statically linked (any distribution): llama-cli, llama-completion, llama-server, llama-bench
rina-llama.cpp-2026.11.14-src.tar.gz Full source of this release
source/0001-*.patch, source/0002-*.patch The tq16 changes as patches on top of PrismML prism 6447906
SHA256SUMS Checksums

Base: PrismML prism at 6447906bf96c1a023980d3453fb59daa6af3bb23; tq16 at de0d48e42aaaaae3bad1cc2d44e1bcf8a80a3867. Windows builds are not included yet.

Quick start (Linux)

tar xzf rina-llama.cpp-2026.11.14-linux-x64.tar.gz
cd rina-llama.cpp-2026.11.14-linux-x64
./llama-cli -m Rina-Juvenile-Exp-2B-PQ2_0.gguf
./llama-server -m Rina-Juvenile-Exp-2B-PQ2_0.gguf --host 127.0.0.1 --port 8080

The top-level llama-* files are small launchers. They read /proc/cpuinfo and run one of three builds in bin/:

Level Requires Typical CPUs
avx512 AVX-512 F/CD/BW/DQ/VL + VNNI Intel Cascade Lake, Ice Lake, Sapphire Rapids and later Xeons; AMD Zen 4, Zen 5
avxvnni AVX2, FMA, F16C, BMI2, AVX-VNNI Intel Alder Lake and later Core
avx2 AVX2, FMA, F16C, BMI2 Intel Haswell and later; AMD Zen 1–3

RINA_CPU_LEVEL=avx2 (or avxvnni, avx512) forces a level. CPUs without AVX2 are not supported by this package; use an upstream llama.cpp build there.

What tq16 changes

  • At load time the weights of all three formats are rearranged into one layout of 16-column tiles. Each 64-byte line holds 2-bit codes such that (line >> 2s) & 3 is a 16-column × 4-input operand for vpdpbusd (AVX-512 VNNI, AVX-VNNI) or pmaddubsw (plain AVX2). Every output column accumulates in its own SIMD lane.
  • PTQ1_0 trits are expanded to 2-bit codes when the model loads.
  • Activations are quantized to int8 with one scale per 32 values.
  • The output of a token does not depend on how many tokens are in the batch, so speculative decoding reproduces plain decoding bit for bit.

Environment variables:

Variable Effect
GGML_TQ16=0 Use the original PrismML kernels
GGML_TQ16_ACT=128 One activation scale per weight group instead of per 32 values: about 10% faster prompt processing on VNNI CPUs, slightly lower precision (KLD 1.46e-4 instead of 1.02e-4)

Speed

Rina Juvenile Exp 2B, llama-bench -p 512 -n 128, tokens per second. “PrismML” is the same source with GGML_TQ16=0. The avxvnni and avx2 rows are the same CPU running builds without AVX-512.

2-vCPU virtual machine (host Ryzen 9 9950X, about 20 GB/s memory bandwidth), 2 threads:

Format Level PrismML pp512 tq16 pp512 PrismML tg128 tq16 tg128
PQ2_0 avx512 195 377 49.3 54.0
PQ2_0 avxvnni 136 258 35.1 53.2
PQ2_0 avx2 92 163 30.2 55.9
PTQ1_0 avx512 58 387 28.6 53.2
PTQ1_0 avx2 40 177 22.8 55.0
Q2_0 avx512 56 371 27.0 49.9
Q2_0 avx2 53 167 25.5 49.4

On this machine, generation at 2 threads is close to the memory-bandwidth limit.

Ryzen 9 9950X, all cores (16 cores / 32 threads, dual-channel DDR5-6400, Linux under WSL 2, this release package, avx512 level):

Format PrismML pp512, 16 threads tq16 pp512, 16 threads PrismML tg128 tq16 tg128
PQ2_0 827 1575 94.9 98.5
PTQ1_0 325 1659 102.3 95.5
Q2_0 311 1759 91.0 95.7
  • tg128 is the median of 10 samples at the best thread count, which was 8 to 12 threads.
  • Generation is limited by memory bandwidth here (about 51 GB/s), so thread counts from 6 to 16 give similar results.
  • Using all 32 threads (SMT) is slower than 16 for both prompt processing and generation.
  • With a 4K context (-d 4096, PQ2_0, 16 threads), tq16 gives pp512 1447 and tg128 82.4.

Single core of the same CPU:

Format Level PrismML pp512 tq16 pp512 PrismML tg128 tq16 tg128
PQ2_0 avx512 111 229 37.7 64.9
PQ2_0 avx2 59 99 24.8 55.4
PTQ1_0 avx512 33 232 22.2 65.7
Q2_0 avx512 30 225 20.5 66.2

Accuracy

KL divergence against an F16 copy of the same weights (exact for ternary weights × fp16 scales), 40 chunks of 512 tokens of Chinese / Japanese / English chat text:

Kernel KLD Same top-1 token
PrismML, AVX-512 1.00e-4 99.45%
PrismML, AVX2, PQ2_0 1.66e-4 99.37%
tq16, all levels 1.02e-4 – 1.03e-4 99.43% – 99.53%

A unit test (tests/test-tq16.cpp in the source) checks matrix products against a double-precision reference: 300 shapes per format, each with two activation distributions.

Known limitations

  • Only x86-64 Linux binaries so far. tq16 does nothing on ARM; there, PrismML's own kernels are used.
  • PTQ1_0 uses as much memory as PQ2_0 after loading, because its trits are expanded to 2-bit codes. On machines with many cores and limited memory bandwidth, PTQ1_0 generation can be a few percent slower than with GGML_TQ16=0: about 5% on the 9950X at 8 or more threads.
  • llama-server is built without TLS. Put a reverse proxy in front of it if it must be reachable from other machines.

Building from source

tar xzf rina-llama.cpp-2026.11.14-src.tar.gz && cd rina-llama.cpp-2026.11.14-src
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON
cmake --build build -j --target llama-cli llama-server

To apply the patches to a newer PrismML prism checkout, run git am source/*.patch.

License

MIT, the same as llama.cpp. Copyright (c) 2023-2026 The ggml authors; the tq16 changes are by Rinstitute. The licenses of bundled third-party code are in licenses/ inside each archive.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support