gatilin/Youtu-LLM-A7B-Base-GGUF

Youtu-LLM-A7B-Base — GGUF (llama.cpp)

GGUF conversion of tencent/Youtu-LLM-A7B-Base @ 6b0d02b2912c2975806db5a4922a1e091fa2f5d3, with a llama.cpp patch that adds the youtu-moe architecture (text and vision).

This arch is not in upstream llama.cpp. You must apply the patch from gatilin/llama.cpp-youtu-moe before these files will load.

A vision-language MoE: ~120B total / ~7B active parameters per token, 48 layers, hidden 3072, hybrid attention (KDA linear attention interleaved with GatedMLA), 256 routed experts (top-8) + 1 shared expert, Qwen3-VL-MoE vision tower, 262144-token context.

Files

Weights are split into shards because Hugging Face rejects any single file above 50 GB. Every shard is a normal .gguf fragment in llama.cpp's standard name-NNNNN-of-NNNNN.gguf scheme.

Shards	Total size	Notes
youtu-moe-a7b-Q4_K_M.gguf-0000{1..8}-of-00008.gguf	66 GiB	recommended — runs on a 128 GiB unified-memory machine
youtu-moe-a7b-bf16.gguf-0000{1..25}-of-00025.gguf	220 GiB	reference weights, same dtype as the source
youtu-vl-mmproj-f16.gguf	867 MiB	vision tower + merger, required for image input

All GGUF sets carry 1090 tensors; youtu-moe-a7b-bf16.gguf is the direct conversion of the 24 safetensors shards, and Q4_K_M was quantized from it (not from safetensors).

# both shard sets, plus mmproj — about 286 GiB
hf download gatilin/Youtu-LLM-A7B-Base-GGUF --local-dir .

# or just the quantized set
hf download gatilin/Youtu-LLM-A7B-Base-GGUF --include "youtu-moe-a7b-Q4_K_M.gguf-*" \
  --include "youtu-vl-mmproj-f16.gguf" --local-dir .

Quick start

# 1. build patched llama.cpp
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout e0f142d5ddc1461ce721644d92af33db23545105
git apply /path/to/llama.cpp-youtu-moe.patch
cmake -B build -DGGML_METAL=ON -DLLAMA_BUILD_SERVER=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-cli llama-server llama-mtmd-cli

Point -m at shard 1 and nothing else — llama.cpp reads the -of-NNNNN counter and pulls in the remaining shards from the same directory by itself. Do not concatenate or rename the shards.

Text

./build/bin/llama-cli -m youtu-moe-a7b-Q4_K_M.gguf-00001-of-00008.gguf --jinja \
  -p "用一句话解释什么是光合作用。" -st --simple-io -ngl 99 -n 128

Output (with the leading </think> stripped — see Known issues):

光合作用是植物、藻类和某些细菌利用光能将二氧化碳和水转化为有机物（如葡萄糖）并释放氧气的过程。

Measured on an M3 Max 128 GiB (Q4_K_M, Metal): model load ≈ 20 s, prefill ≈ 38 t/s, decode ≈ 14 t/s.

Image

./build/bin/llama-mtmd-cli -m youtu-moe-a7b-Q4_K_M.gguf-00001-of-00008.gguf \
  --mmproj youtu-vl-mmproj-f16.gguf \
  --image frontpage.jpg \
  -p "<|vision_start|><__media__><|vision_end|>描述这张图片的内容。" \
  -n 200 -ngl 99

The <|vision_start|>…<|vision_end|> wrapper is mandatory, and you must pass the raw prompt (not --jinja): chat_template.jinja upstream has no image branch, so the template cannot place the image placeholder. <__media__> is mtmd's media marker — do not additionally write <|image_pad|>. For dense documents add --image-min-tokens 1024 (default is 300, which limits small-text OCR).

Verified on the 1969-07-21 New York Times front page: masthead, date MONDAY, JULY 21, 1969, LATE CITY EDITION, 10 CENTS, VOL. CXVIII. NO. 46,731, headline MEN WALK ON MOON, subhead ASTRONAUTS LAND ON PLAIN; COLLECT ROCKS, PLANT FLAG — all correct.

Server

./build/bin/llama-server -m youtu-moe-a7b-Q4_K_M.gguf-00001-of-00008.gguf \
  --mmproj youtu-vl-mmproj-f16.gguf -ngl 99 -c 8192 --host 127.0.0.1 --port 8080

Merging the shards (optional)

Only needed if you want a single file — llama.cpp does not require it.

llama-gguf-split --merge youtu-moe-a7b-Q4_K_M.gguf-00001-of-00008.gguf youtu-moe-a7b-Q4_K_M.gguf
llama-gguf-split --merge youtu-moe-a7b-bf16.gguf-00001-of-00025.gguf youtu-moe-a7b-bf16.gguf

Known issues

The model always emits a leading </think>. This is a base model — upstream recommends raw text continuation, and chat_template.jinja is only provided "for further purpose". It emits </think> (token 128014, token type USER_DEFINED) as its first assistant-side token even when the prompt contains no think markers at all. llama.cpp's auto parser does not classify it, so it lands in message.content. Strip it on the output side — the patch repo ships strip_think.py / youtu_chat.sh for exactly this. Banning token 128014 via logit_bias makes it worse: the model then believes it is still inside the think block and emits quasi-reasoning prose.

Other notes:

• Q4_K_M and BF16 only; other quant types are untested.
• Only validated on Metal (macOS). CUDA / CPU should work but have not been run.
• llama-mtmd-cli prints generated text via LOG(), so --log-disable swallows the answer too.

How it was converted

Full reproduction recipe (weight download, mmproj export, BF16 conversion, metadata self-check, quantization, smoke tests) is in gatilin/llama.cpp-youtu-moe — convert_youtu.sh is the 7-step pipeline. That repo also documents the three non-obvious bits of the conversion: the RoPE weight permutation needed to reconcile ggml's NEOX-style IMROPE with the official adjacent-pair rotation, why mrope_section must be [11, 10, 10, 0] instead of the official [12, 10, 10], and how the modality-aware MoE expert bias (qb_beta, two rows) is routed per token.

License

Weights and architecture follow the upstream Youtu-LLM license — see LICENSE.txt. This repository is an unofficial conversion; all credit for the model belongs to the Tencent Youtu team.