Instructions to use el4/Sage-A1.5-TOFFEE-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use el4/Sage-A1.5-TOFFEE-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf el4/Sage-A1.5-TOFFEE-GGUF
# Run inference directly in the terminal:
llama cli -hf el4/Sage-A1.5-TOFFEE-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf el4/Sage-A1.5-TOFFEE-GGUF
# Run inference directly in the terminal:
llama cli -hf el4/Sage-A1.5-TOFFEE-GGUF

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf el4/Sage-A1.5-TOFFEE-GGUF
# Run inference directly in the terminal:
./build/bin/llama cli -hf el4/Sage-A1.5-TOFFEE-GGUF

Use Docker

docker model run hf.co/el4/Sage-A1.5-TOFFEE-GGUF

Llama new
LM Studio
Jan

vLLM

How to use el4/Sage-A1.5-TOFFEE-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "el4/Sage-A1.5-TOFFEE-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "el4/Sage-A1.5-TOFFEE-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/el4/Sage-A1.5-TOFFEE-GGUF

Ollama
How to use el4/Sage-A1.5-TOFFEE-GGUF with Ollama:
```
ollama run hf.co/el4/Sage-A1.5-TOFFEE-GGUF
```
Unsloth Desktop

How to use el4/Sage-A1.5-TOFFEE-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf el4/Sage-A1.5-TOFFEE-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "el4/Sage-A1.5-TOFFEE-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use el4/Sage-A1.5-TOFFEE-GGUF with Docker Model Runner:
```
docker model run hf.co/el4/Sage-A1.5-TOFFEE-GGUF
```

Lemonade

How to use el4/Sage-A1.5-TOFFEE-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull el4/Sage-A1.5-TOFFEE-GGUF

Run and chat with the model

lemonade run user.Sage-A1.5-TOFFEE-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use el4/Sage-A1.5-TOFFEE-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf el4/Sage-A1.5-TOFFEE-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default el4/Sage-A1.5-TOFFEE-GGUF

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use el4/Sage-A1.5-TOFFEE-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf el4/Sage-A1.5-TOFFEE-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "el4/Sage-A1.5-TOFFEE-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

🍬 TOFFEE

Tensor-Optimized Quantization Engine

This repo contains TOFFEE quants for InternScience/Sage-A1.5

⚙️ The TOFFEE Architecture

🔍 Activation-Driven Imatrix

Captures per-tensor activation variance during forward passes and uses it to assign precision per tensor. Critical layers get more bits; redundant layers get aggressively crushed.

🎯 MoE Router-Weighted Scaling

Mixes MoE router probabilities into activation scales, so "hot" experts are fiercely protected while "cold" experts are quantized down hard within the same tensor block.

🏗️ Architecture-Agnostic Mapping

Dynamically reads HuggingFace modules and the F16 GGUF to map tensors — no hardcoded regex. Works on Llama, DeepSeek, Qwen hybrids, and custom SSM/MoE architectures.

🖥️ Consumer VRAM Safe

Enforces a per-process VRAM fraction at the CUDA allocator level, so imatrix harvesting runs on 12 GB laptop GPUs without OOMing mid-pass.

Tier	Target Quality	Target Size	Middle Layer Strategy
🍬 `quality`	Q8 Match	~21 GB	IQ4_XS
⚖️ `balanced`	Q6 Match	~24 GB	Q5_K
📦 `compact`	Q4 Match	~16 GB	Q3_K
🚀 `mini`	Q2 Match	~12 GB	IQ2_XXS
💎 `nano`	Q2 Match	~8 GB	IQ2_XXS

🚀 Usage

Quantize a model with an activation-driven imatrix:

$ python -m toffee quant --model ./Sage-A1.5 --output-dir ./out \
  --tier quality --samples 2075 --gguf-quant Q4_K_M

📚 Credits & Foundations

👉 APEX Quantization Method
Ettore Di Giacinto & Richard Palethorpe (LocalAI Team). TOFFEE builds on APEX's layer-wise precision gradients and MoE-aware tensor classification with dynamic, data-driven imatrix calibration.

👉 Bartowski and Lamim
For the semantic imatrix calibration dataset that powers TOFFEE's activation scaling.

👉 llama.cpp
Georgi Gerganov and contributors for the foundational inference and quantization engine.

👉 HuggingFace Accelerate
For the init_empty_weights() context manager that makes the 0-RAM "Ghost Model" possible on consumer hardware.

👉 ONYX
TOFFEE ports ONYX's imatrix passes and Conv-aware reduction logic for architecture-agnostic quantization.

Support the Project

Fuel the TOFFEE — send some crypto my way! No coffee here either, but a pink lemonade would be fire! 🔥

github.com/damien1141/toffee

recommended sampling parameters:

--temp 0.85 --top-p 0.95 --top-k 40 --min-p 0.05 --repeat-penalty 1.1 --presence-penalty 0.0

Downloads last month: 1,378

GGUF

Model size

36B params

Architecture

qwen35moe

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Model tree for el4/Sage-A1.5-TOFFEE-GGUF

Base model

InternScience/Sage-A1.5

Quantized

(41)

this model

Collection including el4/Sage-A1.5-TOFFEE-GGUF

TOFFEE

Collection

1 item • Updated 7 days ago • 1