GRT-30M-Curriculum: Multi-Round Symplectic Curriculum (MatterWaveAI/grt-30m-curriculum)

Multi-Round Symplectic Curriculum Release: Starting from the exact same 29.3M-parameter (9.74M non-embedding backbone across 12 rotational blocks) 24-hour Qwen 3 distilled checkpoint as MatterWaveAI/grt-30m, MatterWaveAI/grt-30m-curriculum applies multi-round curriculum optimization with HuygensSymplecticAdam. Across the 9-task SLM-Bench multiple-choice suite (13,144 held-out examples), macro-average accuracy rises from 37.56% (Zero-Shot Distilled Base) → 39.61% (PairAdam Baseline) → 43.86% (Single-Pass HuygensSymplecticAdam) → 48.07% (Multi-Round Curriculum, +8.46% vs. PairAdam), while GSM8K 4-way numerical verification reaches 99.20% (PPL = 7.60, down 3.68× from 27.99).

Model Details

Property Value
Architecture GRTForCausalLM (model_type: "grt", grt30m_kd)
Base model MatterWaveAI/grt-30m (24-Hour Qwen 3 Distillation)
Total stored parameters 29,305,498 (29.31M)
Non-embedding backbone parameters 9,740,197 (9.74M across 12 blocks)
Dense-equivalent parameters 55,680,446 (55.68M)
hidden_size (d_model) 256
num_hidden_layers (n_blocks) 12
num_attention_heads (n_heads) 4 (head_dim = 64)
intermediate_size (d_ff) 1,024
n_qkv_rotations / n_mixer_rotations / ffn_n_rotations 11 / 1 / 21
Vocabulary 151,669 (Qwen 3 BPE, tied polar cosine head)
Context length (max_position_embeddings) 2,048
Weight files model.safetensors (116 MB; base FP32 GGUF in MatterWaveAI/grt-30m)

Because grt-30m shares the full 151,669-token Qwen 3 vocabulary at hidden_size = 256 for direct token-aligned distillation from Qwen 3, the tied polar embedding table (phase_angles [151669, 128] + log_magnitude [151669]) accounts for 19.57M parameters (66.8% of stored weights), leaving 9.74M stored parameters for the 12-layer rotational transformer backbone.


24-Hour Qwen 3 Base Distillation & Curriculum Summary

Metric Value
Teacher family / Tokenizer Qwen 3 (151,669-token Qwen 3 BPE vocabulary)
Student config grt30m_kd (d_model = 256, 12 rotational blocks, 9.74M non-embedding params)
Base training corpus mix fineweb-edu (45%), cosmopedia-v2 (20%), wikipedia (15%), finemath (12%), stack-edu (8%)
Base distillation hardware 8x NVIDIA A100-SXM4-80GB (DDP + Fused CUDA Givens Kernels)
Base wall-clock / tokens 24.0 hours (86,969s) / 10,789,404,672 (10.79B tokens, 20,580 steps at 123,900 tok/s)
Base distillation optimizer PairAdam (Peak LR 1e-3, WSD schedule; val_loss = 4.4776, val_ppl = 88.02)
Multi-round curriculum optimizer HuygensSymplecticAdam (Two-stage Huygens splay torque → Symplectic non-squeezing curriculum)

4-Backend Exact Greedy Generation Parity (Base Checkpoint Reference)

On the base MatterWaveAI/grt-30m checkpoint, model.safetensors (transformers) and grt-30m-f32.gguf (llama.cpp) produce identical token-for-token greedy continuations (temperature = 0, 16 new tokens) across all four supported backends on AMD Strix Halo (gfx1151):

  • PyTorch ROCm (gfx1151, fused HIP kernels)
  • llama.cpp CPU (x86_64 AVX2)
  • llama.cpp Vulkan (RADV gfx1151)
  • llama.cpp ROCm (HIP gfx1151)
Prompt Greedy continuation on grt-30m base (16 tokens, identical on all 4 backends)
The capital of France is The capital of France is the first to be the first to be the first to be the first to be
Once upon a time Once upon a time, you might be wondering what exactly is the way to do with the right point
In the beginning In the beginning of the world, the world is the most common form of the world. The

Benchmarks: Distilled Base (GRT-30M) → PairAdam → Single-Pass HuygensSymplecticAdam → Multi-Round Curriculum (grt-30m-curriculum)

Evaluated across the 10-task SLM-Bench small-language-model suite (training/slm_bench.py / grt-slm-bench) starting from the exact same 29.3M-parameter (9.74M non-embedding) 24-hour Qwen 3 distilled checkpoint:

1. Multiple-Choice Accuracy (9 Core SLM-Bench Tasks, 13,144 Held-Out Examples)

Task (SLM-Bench) Split (n) Random Zero-Shot (GRT-30M) PairAdam Baseline Single-Pass (HuygensSymplectic) Multi-Round Curriculum Total Gain (Curriculum vs. PairAdam)
BoolQ (google/boolq) val (3,270) 50.00% 51.41% 60.55% 62.23% 65.72% +5.17% (+3.49% vs. Single-Pass)
ARC-Easy (allenai/ai2_arc) test (2,376) 25.00% 29.59% 28.96% 36.32% 40.57% +11.61% (+4.25% vs. Single-Pass)
ARC-Challenge (allenai/ai2_arc) test (1,172) 25.00% 20.82% 23.55% 25.68% 30.80% +7.25% (+5.12% vs. Single-Pass)
OpenBookQA (allenai/openbookqa) test (500) 25.00% 28.20% 30.00% 35.80% 42.80% +12.80% (+7.00% vs. Single-Pass)
PIQA (baber/piqa) val (1,838) 50.00% 55.28% 54.08% 55.28% 56.47% +2.39% (+1.19% vs. Single-Pass)
WinoGrande (allenai/winogrande) val (1,267) 50.00% 50.28% 50.99% 51.22% 51.70% +0.71% (+0.48% vs. Single-Pass)
HellaSwag (Rowan/hellaswag) val (1,000) 25.00% 29.20% 31.60% 31.60% 37.40% +5.80% (+5.80% vs. Single-Pass)
QNLI (nyu-mll/glue) val (1,000) 50.00% 51.60% 54.00% 72.00% 77.90% +23.90% (+5.90% vs. Single-Pass)
CommonsenseQA (tau/commonsense_qa) val (1,221) 20.00% 21.70% 22.77% 24.65% 29.24% +6.47% (+4.59% vs. Single-Pass)
9-Task Multiple-Choice Macro Average 13,144 total 35.56% 37.56% 39.61% 43.86% 48.07% +8.46% (+4.21% vs. Single-Pass)

2. Mathematical Reasoning Held-Out Perplexity & Verification (GSM8K, test split, n = 500)

Metric (SLM-Bench) Zero-Shot (GRT-30M) PairAdam Baseline Single-Pass (HuygensSymplectic) Multi-Round Curriculum Total Improvement (Curriculum vs. Zero-Shot / PairAdam)
GSM8K Held-Out Perplexity (PPL ↓) 27.99 10.65 9.04 7.60 3.68× lower vs. Zero-Shot (-28.6% vs. PairAdam)
GSM8K 4-Way Numerical Verification (Acc ↑) 34.20% 39.20% 98.60% 99.20% +60.00% vs. PairAdam (+0.60% vs. Single-Pass)

Overview of Multi-Round Curriculum Optimization (HuygensSymplecticAdam)

While single-pass task adaptation with HuygensSymplecticAdam already improves substantially over first-generation PairAdam, a compact 9.74M-parameter rotational backbone (12 Givens butterfly blocks on SO(D)) benefits further from multi-round curriculum training using HuygensSymplecticAdam (transformers.models.grt.optimization_grt):

  1. Stage-1 Huygens Circular Order Parameter Splay Torque (_apply_huygens_splay_torque): Across the K Givens cascade stages of each projection (dim=0), HuygensSymplecticAdam computes the classical Huygens circular synchronization order parameter $$R_p = \left|\frac{1}{K}\sum_{k=0}^{K-1} e^{i\theta_{k,p}}\right|$$ During high-gradient alignment (step <= splay_anneal_steps and high normalized momentum energy $m_{\text{norm}}^2$), it applies a curriculum gate $\gamma_{\text{step}} = \frac{1}{1 + (\text{step} / \text{splay_anneal_steps})^2}$ and momentum activity factor $\gamma_{\text{mom}} = \frac{m_{\text{norm}}^2}{m_{\text{norm}}^2 + \text{zpe_floor}}$ to inject a strictly zero-mean repulsive splay torque $\operatorname{relu}(R_p - \tau_{\text{splay}})\sin(\theta_{k,p} - \Psi_p)$ whenever $R_p > \tau_{\text{splay}}$ (splay_threshold = 0.35). This prevents cascade stages from bunching into redundant angles without altering mean stage drift.
  2. Stage-2 Gromov–Liouville Symplectic Non-Squeezing Floor & Helmholtz–Casimir Wave Cavity: As step advances past splay_anneal_steps (annealing the splay gate to splay_stage2_ratio = 0.2) and angular momentum settles toward zpe_floor, the Huygens splay perturbation smoothly decays while the Dupin cyclide conic metric $r_{\text{conic}}^2$, reciprocal symplectic stiffness floor $$c_{\text{symp}} = \sqrt{c_{\text{base}}^2 + c_{\text{rms}}^2 \frac{\text{zpe_floor}^2}{m_{\text{norm}}^2 + \text{zpe_floor}}}$$ and Helmholtz–Casimir wave-cavity boundary Laplacian (_apply_casimir_cavity_cascade, mixing deviatoric projection deviatoric_mix and Dirichlet boundary Laplacian casimir_coupling) govern fine-grained phase calibration.
  3. Cayley Isometric Rational Retraction & Progressive Multi-Round Schedule: Tangent angular steps $u_{\text{total}}$ are retracted onto $\operatorname{SO}(2)^P$ via the smooth rational Cayley map $\Delta\theta = -2\theta_{\max}\arctan!\left(\frac{\eta, u_{\text{total}}}{2\theta_{\max}}\right)$ bounded by max_angle_step and wrapped to $[-\pi, \pi)$. Because every projection is norm-preserving (||R(theta) x||_2 = ||x||_2), the backbone is adapted across successive curriculum rounds without norm explosion or catastrophic forgetting.
from transformers.models.grt import GRTForCausalLM, build_grt_optimizer

model = GRTForCausalLM.from_pretrained("MatterWaveAI/grt-30m-curriculum")
optimizer = build_grt_optimizer(
    model,
    lr=5e-3,
    weight_decay=0.01,
    optimizer_type="huygens_symplectic_adam",
    max_angle_step=0.08,
    zpe_floor=0.015,
    casimir_coupling=0.08,
    splay_weight=0.05,
    splay_anneal_steps=250,
)

Usage

PyTorch (transformers)

import torch
from transformers import AutoTokenizer, GRTForCausalLM

# Optional: register fused CUDA/ROCm Givens kernels if grt-kernels is installed
try:
    from grt_kernels.torch_ops import ensure_grt_ops
    ensure_grt_ops()
except ImportError:
    pass

model_id = "MatterWaveAI/grt-30m-curriculum"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = GRTForCausalLM.from_pretrained(model_id, dtype=torch.float32).eval()

prompt = "The history of the city"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
with torch.no_grad():
    output = model.generate(input_ids, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Verify file integrity against SHA256SUMS:

sha256sum --ignore-missing -c SHA256SUMS

Architecture Overview

GRT replaces dense weight matrices (nn.Linear) in both attention and FFN blocks with cascades of learned 2×2 Givens rotations arranged in a butterfly topology. At each stage k, pairs of coordinates (x0, x1) are rotated by an angle theta_k modulated by 4-component Cl(2,0) multivector features:

feat(x0, x1) = [x0^2 + x1^2,  x0,  x1,  x0 * x1_next - x1 * x0_next]
theta_k      = alpha_k + tanh(norm(feat(x0, x1)) @ C_k)
r0           = x0 * cos(theta_k) - x1 * sin(theta_k)
r1           = x0 * sin(theta_k) + x1 * cos(theta_k)
  • GRTEmbedding & GRTCosineHead — Token embeddings are stored in polar form (phase_angles + log_magnitude) and tied to a scaled cosine-similarity language modeling head (lm_head.output_scale = 40.23).
  • GRTAttention — 11-stage content-modulated Givens butterfly cascades for Q, K, V, O, paired with standard scaled dot-product attention and learnable RoPE frequencies.
  • GRTMixer & GRTFFN — Cross-head Givens mixing stage followed by 21-stage rotational FFN expansion/contraction and phase-preserving GRTRadialGELU.
  • GRTPearlNorm — Pair-aware RMS normalization that normalizes pair radii sqrt(x_{2i}^2 + x_{2i+1}^2) while preserving phase angles.

Intended Use & Limitations

  • Research preview. Intended for studying geometric/rotational neural architectures, symplectic/synchronization optimization, kernel benchmarking, and multi-backend (PyTorch / llama.cpp / vLLM) verification.
  • Curriculum causal LM. Not chat/instruction-tuned (see MatterWaveAI/grt-30m-chat for conversational formatting), no safety filtering, and prone to repetition under greedy decoding at small parameter scale.

Complete MatterWaveAI/grt-30m* Research Suite on Hugging Face

  1. Base & Curriculum Causal LMs: MatterWaveAI/grt-30m | MatterWaveAI/grt-30m-curriculum
  2. Chat & Code Specialists: MatterWaveAI/grt-30m-chat | MatterWaveAI/grt-30m-code
  3. 4-Expert Mixture-of-Rotations (MoR) & Toroidal LoRA: MatterWaveAI/grt-30m-mor | MatterWaveAI/grt-30m-lora
  4. Process Reward Model (PRM), Extractive QA & NER: MatterWaveAI/grt-30m-prm | MatterWaveAI/grt-30m-qa | MatterWaveAI/grt-30m-ner
  5. Circular-Pooled Sentence Embedding & Reranker: MatterWaveAI/grt-30m-embedding | MatterWaveAI/grt-30m-reranker
  6. Canonical Tokenizer & Polar Embedding Bank: MatterWaveAI/grt-30m-tokenizer
  7. Vision-Language (GRTVL) & Text-to-Image (GRTImage): MatterWaveAI/grt-30m-vl | MatterWaveAI/grt-30m-image
  8. Speech-to-Text (GRTASR) & Text-to-Speech (GRTTTS): MatterWaveAI/grt-30m-asr | MatterWaveAI/grt-30m-tts
  9. Multimodal Vision-Audio-Language-Speech (GRTOmni): MatterWaveAI/grt-30m-omni
  10. System-1 Agent Reflex & Decision Scorer (open-s1): MatterWaveAI/open-s1-grt-30m

License

Apache 2.0


Citation

@techreport{householter2026grt,
  title       = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
  author      = {Householter, David},
  institution = {MatterWave AI},
  year        = {2026},
  url         = {https://matterwave.ai}
}
Downloads last month
903
Safetensors
Model size
29.8M params
Tensor type
I64
·
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for MatterWaveAI/grt-30m-curriculum

Finetuned
(15)
this model

Datasets used to train MatterWaveAI/grt-30m-curriculum