YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

AGIFORMER: Byte-Level Language Model with Neuroplasticity

Status: Phase 7 - Curriculum Learning βœ… Complete
Latest Achievement: 20K curriculum training with 77% BPC reduction

A research implementation of a byte-level language model featuring:

  • 🧠 Hebbian Memory with dynamic neuroplasticity
  • πŸ“š Curriculum Learning (3-stage developmental approach)
  • πŸ”„ System 2 Reasoning (iterative thinking loop)
  • πŸš€ Linear Complexity attention mechanism

Quick Start

Installation

pip install torch datasets tqdm

Training (Curriculum Learning)

python train_curriculum.py  # 20K steps, 3 curriculum stages

Inference

python generate.py best_model_curriculum.pth

Testing

python test_recall.py best_model_curriculum.pth  # Memory test
python inspect_reasoning.py                        # System 2 diagnostics

Architecture

Bytes β†’ Encoder (RoPE) β†’ Hebbian Memory β†’ Reasoning Loop β†’ Local RNN β†’ Bytes
         (Patches)        (Dynamic Ξ»)       (3 steps)        (Autoregressive)

Core Components

  • ByteLatentEncoder: Patches bytes into latent vectors with RoPE
  • HebbianMemory: Fast weights with learnable decay + neuroplasticity (Ξ±)
  • RecurrentReasoningBlock: 3-step iterative thinking loop (System 2)
  • LocalAutoregressiveHead: GRU-based byte decoder

See docs/architecture.md for technical details.

Features

βœ… No Tokenization - Universal byte-level processing
βœ… Linear Complexity - O(N) attention with Hebbian memory
βœ… Neuroplasticity - Dynamic memory consolidation (Ξ±: 0.1 β†’ 0.99)
βœ… Curriculum Learning - 3-stage developmental training
βœ… Active Reasoning - Verified thinking loop (Ξ”z = 12.7)
βœ… AMP Compatible - Mixed precision training with stability fixes

Curriculum Learning (Phase 7)

Training Stages

Stage Steps Plasticity (Ξ±) Data Purpose
1. Childhood 0-3K 0.10 Dictionary Lexical grounding
2. Youth 3K-8K 0.50 Stories Syntactic scaffolding
3. Adulthood 8K-20K 0.99 Wikipedia Semantic expansion

Results (20K Steps - Turkish Training)

Metrics:

  • Final BPC: 1.85 (↓77% from initialization)
  • Best Val BPC: 1.78
  • Training Time: ~50 minutes (CUDA GPU)
  • Stability: 0 NaN occurrences across 20K steps

Progress:

Step 0:     BPC = 8.04  (Random initialization)
Step 5K:    BPC = 2.23  (Initial curriculum complete)
Step 10K:   BPC = 1.98  (Mid-training)
Step 20K:   BPC = 1.85  (Final)

Improvement: 6.19 BPC reduction (77% improvement)

Critical Fix: AMP Stability

Problem: Float16 overflow in Hebbian Memory with low plasticity (Ξ±=0.1)
Solution: Force float32 computation for memory module

@torch.amp.autocast('cuda', enabled=False)
def forward(self, x):
    x = x.float()  # Bypass AMP for numerical stability
    # ... Hebbian computation ...
    return out.to(input_dtype)

This fix enables stable 20K+ step training with AMP enabled.

Documentation

Model Files

  • best_model_curriculum.pth - Best checkpoint (Val BPC: 1.78)
  • last_model_curriculum.pth - Final model state (20K steps)
  • metrics_curriculum.json - Full training metrics

Next Steps

Recommended Improvements

  1. Extended Training: 30K-50K steps for further convergence
  2. Larger Model: Increase d_model=768, n_layers=8
  3. Longer Context: Extend to 2048 token window
  4. Fine-tuning: Domain-specific Turkish datasets

Research Directions

  • Adaptive plasticity scheduling
  • Multi-stage curriculum optimization
  • Cross-lingual transfer learning
  • Sparse Hebbian memory

Citation

@software{agiformer2025,
  title={AGIFORMER: Byte-Level Language Model with Hebbian Memory and Neuroplasticity},
  author={inkbytefo},
  year={2025},
  note={Phase 7: Curriculum Learning with Dynamic Plasticity},
  url={https://github.com/inkbytefo/agi-former}
}

License

MIT License - see LICENSE file for details.

Acknowledgments

  • Built with PyTorch
  • Turkish Wikipedia dataset (trwiki)
  • Turkish Dictionary dataset (TDK)
  • Inspired by Fast Weights, Linear Transformers, and developmental neuroscience

Developer: inkbytefo
Phase: 7 (Curriculum Learning & Neuroplasticity)
Status: Production Ready βœ…
Last Updated: 2025-11-23

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support