uncoalesced/parentheses-v0.9-130M-v3

parentheses v0.9-130M-v3 (Project Tanzinite)

parentheses v0.9-130M-v3 is a recurrent language model built from scratch by uncoalesced for Dravidian languages and English. It was trained on a single 8 GB laptop GPU.

Languages

• Strong: Kannada, Tamil, Telugu, Malayalam, English.
• Low-resource, also in training (weaker): Tulu, Koya, Kuvi, Kolami (Naikpod).

Architecture Highlights

• Recurrent State Space Duality (SSD): Linear $O(T)$ chunked associative linear recurrence in parallel training, and $O(1)$ constant memory updates in inference: $S_t = \alpha_t S_{t-1} + \beta_t (k_t \otimes v_t)$ and $y_t = q_t S_t$. Zero sequence Softmax, zero growing KV-cache.
• Scale: 148.38M parameters (16 layers, $d_{\text{model}}=768$, 12 heads, $d_{\text{head}}=64$). The Hub's parameter count is higher because it counts the tied input/output embedding twice.
• Tokenization: Custom BPE 32k (v3_32k) tokenizer (32,000 vocab + 19 special tokens: <think>, </think>, language tags). Its regex keeps Indic vowel signs and viramas attached to their consonant, so aksharas are not split.
• Recurrent Innovations: $\beta$-gated dynamic write selectivity, Token-Shift ($\mu$-mix), and $V_0$ value highway.
• Evaluation (released checkpoint): Native-script output in 40/40 runs across the five strong languages (53/72 across all nine), 0 control-token leaks, 0/19 refusals on legitimate technical prompts, and 5/24 on a decontaminated set of fresh arithmetic, logic and multilingual prompts. Per-token decode latency stays flat (last/first ratio about 1.0).
• Edge Deployment: Static-shape ONNX single-step graph (DirectML), matching PyTorch to a max logits error of about 2e-5.

Model Details

• Base Pretraining: 1.50B tokens seen (about 1.9 passes over the 808M-token processed_130m_v1 corpus: Sangraha Dravidian core, STEM reasoning, conversational text, English).
• Clean SFT v2: 131.07M tokens on processed_sft_stem_open_v2 with zero truncated documents.
• Calibrated DPO v2: Preference optimization over 900 balanced reasoning and epistemic abstention pairs (whole-set margin +0.35, 84.2% pair accuracy).

Limitations

At this size the model makes factual errors, often fails multi-step arithmetic, can repeat itself in longer outputs, and does not yet reliably answer "I don't know". Treat it as a research model.

License

Model weights: CC BY-NC 4.0, research and non-commercial use only. Some training sources are non-commercial or carry no license; see docs/TRAINING_DATA.md in the GitHub release. Code is MIT.