cicaba/Unlimited-OCR-2-GGUF

Unlimited-OCR-2 GGUF (F16 / Q8_0 / Q4_K_M)

GGUF builds of baidu/Unlimited-OCR-2 — Baidu's 3.3B mixture-of-experts unified OCR model — for llama.cpp. It turns document page images (scans, photos, PDF pages) into text with layout structure.

Mainline llama.cpp cannot load these files yet. Unlimited-OCR-2 uses DASH routed sparse attention (SR-SWA) and page-aware 2D RoPE, which are not in upstream llama.cpp. Build llama.cpp with the patch shipped in this repo — see Build below.

Files

File	Size	Description
Unlimited-OCR-2-F16.gguf	5.5 GB	Full precision (source weights are BF16, exported as F16)
Unlimited-OCR-2-Q8_0.gguf	3.0 GB	Recommended — measured output identical to F16
Unlimited-OCR-2-Q4_K_M.gguf	1.9 GB	Smallest (5.32 BPW)
mmproj-Unlimited-OCR-2-F16.gguf	788 MB	Vision encoder (SAM + CLIP) — required with every text file above
llama.cpp-patch/unlimited-ocr2.patch	62 KB	llama.cpp v2 support patch (shipped in this repo)

Build patched llama.cpp

The patch is based on upstream commit 3c809f443 (2026-10-30) and includes a server robustness fix: malformed media-only requests return HTTP 400 instead of crashing the process.

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout 3c809f443
# download llama.cpp-patch/unlimited-ocr2.patch from this repo, then:
git apply unlimited-ocr2.patch
cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j \
      --target llama-mtmd-cli llama-server

Drop -DGGML_CUDA=ON if you don't have an NVIDIA GPU (Metal is used on Mac automatically).

Run

./build/bin/llama-mtmd-cli \
  -m Unlimited-OCR-2-Q8_0.gguf \
  --mmproj mmproj-Unlimited-OCR-2-F16.gguf \
  --jinja --temp 0 -c 32768 -n 8000 -ngl 99 \
  --image page.png -p "<PAGE><__media__>Multi page parsing."

OpenAI-compatible server:

./build/bin/llama-server \
  -m Unlimited-OCR-2-Q8_0.gguf \
  --mmproj mmproj-Unlimited-OCR-2-F16.gguf \
  --jinja -np 1 -c 32768 -ngl 99 --host 0.0.0.0 --port 8080

Image-only requests (supported out of the box)

The GGUFs in this repo embed an auto-injecting chat template: when a message contains only images and no text, the template appends the default instruction Multi page parsing. (equivalent to what the official transformers/sglang runtime does), so clients can send bare image requests:

{"messages":[{"role":"user","content":[
  {"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}
]}], "temperature": 0}

• Requests that already carry text (including custom instructions) are untouched — no double injection.
• Concurrent requests are queued and serialized, never dropped.
• On memory-tight GPUs add --image-max-tokens 273 (base mode, the official multi-page default); the default single-page gundam tiling mode is more accurate but needs ~1.3 GB extra VRAM for the vision encoder.

Limitations

• One request at a time (-np 1): page numbers and positions are derived from a single sequence's token history; concurrent requests queue up.
• On 16 GB GPUs without flash-attention support (e.g. Tesla P100), the F16 build needs reduced offload (-ngl 12 -c 8192); Q8_0 / Q4_K_M offload fully.
• No no-repeat n-gram guard (the official server has one); cap output with -n if a page loops.

Quantization notes

• Converted from the official weights (HF baidu/Unlimited-OCR-2, ModelScope mirror PaddlePaddle/Unlimited-OCR-2) with the patched converter, quantized with llama-quantize. No weight modifications.
• The DASH router (indexer.* tensors) stays F16/F32 in every build — routing matches the original model exactly.
• In the Q4_K_M build, 12 ffn_down_exps tensors (row width 896, not a multiple of 256, so K-quants don't apply) fall back to Q8_0 — standard llama.cpp behavior for that shape.
• All three text builds were verified with llama-mtmd-cli on a Tesla P100 16G (a logo image and a text-dense chart page): F16 / Q8_0 / Q4_K_M produce identical output with complete layout blocks.

License

MIT, inherited from baidu/Unlimited-OCR-2. The llama.cpp v2 support patch shipped in llama.cpp-patch/ is based on the llama.cpp community's Unlimited-OCR-2 work, rebased onto the 2026-10-30 upstream master with additional server robustness fixes.