LFM3-Omni-2B-98K-FP16

Community long-context fine-tune of LiquidAI/LFM3-Omni-2B, extended to a verified ~98k+ token context via LoRA SFT and merged into a single checkpoint. **## Usage (transformers)

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor

model = AutoModelForMultimodalLM.from_pretrained(
    "LFM3-Omni-2B-98K-FP16", dtype=torch.bfloat16, device_map="auto",
    trust_remote_code=True)
proc = AutoProcessor.from_pretrained("LFM3-Omni-2B-98K-FP16", trust_remote_code=True)

**

Verification

Check	Result
Needle-in-haystack (HF, depths 10/50/90%)	15/15 HIT, 15k–109k tokens
Needle-in-haystack (GGUF @ 32k, all quants)	Pass on every quant
Needle-in-haystack (GGUF @ 64k, Q6_K)	Pass
Omni mmproj (audio in, llama.cpp mtmd)	Working
llama.cpp generation (Vulkan, RX 6750 XT)	~70–180 tok/s

Training run: LoRA r=32/α=64 SFT on 65k–98k synthetic long-context data, merged into the base model. Effective trained length ≈ 94k (the last ~4 optimizer steps of 94.5k–98.3k examples were skipped on OOM); retrieval verified beyond that regardless.

Architecture note (important)

This fine-tune converted the 5 sliding_attention layers to full_attention (theta 1e6, partial rotary 0.5, no window) during training. The HF checkpoints here carry the correct layer_types already.

llama.cpp requires at least one sliding-window layer, so the GGUFs emulate the same behavior: the former sliding layers are tagged SWA with sliding_window=131072, freq_base_swa=1e6, and n_rot_swa=64 — numerically identical to full attention. This is load-bearing: don't "fix" the metadata.

License

Built on LiquidAI/LFM3-Omni-2B. Weights are subject to the base model's license (LFM Open License).