fraQtl/fraqtl-sm80-runtime

fraQtl SM80 runtime: prebuilt verification wheel

Prebuilt binary wheel of the fraqtl vLLM runtime. It reproduces the fraQtl KV-cache receipts on a rented A100 for ≈$15, with no kernel or calibration source distributed.

What's inside

• The fraQtl CUDA kernel as a compiled SM80 (A100) binary (fraqtl/_native/libggml-cuda.so). No .cu/.c/header source ships; the build hard-fails if any leaks in (see wheel_build_receipt.json).
• The vLLM integration layer (attention backend, model adapters, sidecar readers) as Python source; it must stay torch.compile-traceable or the engine provisions a smaller KV pool than the receipts. It is orchestration glue: how eigenbases are computed is not in the wheel in any form (no calibration or sidecar-factory code; remaining runtime modules are Cython-compiled binaries).
• vLLM plugin auto-registration via the vllm.general_plugins entry point: pip install the wheel and vLLM's plugin loader finds the backend in every worker process.

Scope

• Hardware: A100 (SM80) only. The binary will not run on H100, L4, or consumer GPUs.
• vLLM 0.20.2, CUDA 12.4, Python 3.11, the exact receipt environment. Other versions are unsupported.
• Purpose: verification of the published receipts. It is not a general-purpose release.
• Published speed rows are batch-1 decode. Retrieval is NIAH-verified, not bit-identical. fp8-KV appears in every table.

Use it

Use the one-command repro script (fraqtl_repro_receipts.py, in this repo and in the sidecar repos):

pip install modal && modal setup
modal secret create huggingface HF_TOKEN=hf_...
modal run fraqtl_repro_receipts.py

It installs this wheel, pulls the calibrated sidecars from fraQtl/mistral-7b-instruct-v0.3-kv-sidecars, runs the three-arm receipt (fraQtl D2 vs fp16 vs fp8-KV) at 8K and 32K with a needle-in-a-haystack gate per arm, and prints the table.

Files

File	What
fraqtl_runtime-*.whl	The prebuilt wheel (binary-only)
wheel_build_receipt.json	Build provenance: sha256, source-leak audit result, module list
fraqtl_repro_receipts.py	The one-command repro script

License

Proprietary; free to install and run for verification and evaluation. No reverse engineering. Sidecars and receipts in the companion repos are Apache 2.0.

More from fraQtl

The on-device lane, calibration-aware Hi-Fi GGUFs: Gemma-4-E2B (runs an offline phone agent) · Llama-3.1-8B + Llama-3.2-1B draft pair · Qwen3.6-35B MoE. Org page: huggingface.co/fraQtl.