Qwen2.5-72B-Instruct-lecore-hrr — leCore gated-HRR attention gates (WORK IN PROGRESS)

Work in progress — not a working model yet. Passkey recall still fails; what is verified is the constant-memory claim (a fixed 3.3 MB attention state on the small proof). These are the trained gate tensors plus the code that produced them, published so the run is reproducible and the checkpoint is not lost.

What this is

A drop-in replacement for full attention in Qwen/Qwen2.5-72B-Instruct: the layer keeps a sliding-window softmax (fixed window W=256) plus a gated HRR (holographic reduced representation) recurrent state, so decode memory is O(1) in context length instead of a KV cache that grows with it. code/hrr_swa.py holds the module (HRRSWAAttention(old, cfg, window=256)), code/gated_scan_triton.py the Triton scan kernel, code/bigjob.py the training driver: it loads one copy of the model, toggles the original attention as the teacher, trains only the new HRR parameters (theta, beta, hnorm, mix) by distillation, and uploads gates + logs to the Hub as it goes.

Files

file	size	what it is
hrr_gates.pt	5.4 MB	trained gates (theta, beta, hnorm, mix) — load into the swapped modules
code/bigjob.py	8.8 KB	training driver: teacher = original attention, student = hybrid HRR
code/hrr_swa.py	8.2 KB	the attention module (HRRSWAAttention, convert, new_params)
code/gated_scan_triton.py	5.0 KB	Triton gated scan (10x faster than the Python loop, 75 → 42 GB)
train_log.json	3.3 KB	KD loss / tokens / tok-s, one record per 10 steps

Training (from train_log.json)

metric	value
steps	450
tokens seen	0.92 M
throughput	384 tok/s
wall clock	40.1 min
KD loss, median of last 10 logged steps	0.0683
KD loss, minimum	0.0364

The distillation loss is noisy step to step (it spikes on hard batches); the last-10 median is the number to compare.

Status and evidence

Gates are uploaded and load cleanly, and the swapped model runs — but passkey recall still fails, so this is not a usable model yet. What is verified: the attention state is constant (3.3 MB) regardless of context, and the swap is a pure distillation of the base model's own attention (no base weights changed).

The from-scratch evidence for the operator itself (softmax vs gated-HRR, matched pair) is in staccs/lecore-303m-rerun: at 303M / 1B tokens gated-HRR lands 41.5% worse in validation loss (4.5138 vs 3.1902; 6.5120 vs 4.6025 bits/byte). Cheap constant memory, a real quality price. Method write-ups, the tiny-twin benchmarks and the footprint measurements: staccs/lecore-hrr-gpt2.

How to use

import torch
from hrr_swa import convert, new_params   # code/hrr_swa.py
convert(model, 256)                    # swap attention for HRRSWAAttention, window 256
state = torch.load("hrr_gates.pt", map_location="cpu")
# load the trained theta / beta / hnorm / mix tensors into the swapped modules

License

Base model: Qwen/Qwen2.5-72B-Instruct. The base weights are untouched — only the HRR gate tensors here were trained.