luluw/embeddinggemmatext2-65k-en-ne-trim-GGUF

EmbeddingGemma 2 (English + Nepali, 65k vocabulary) - GGUF

GGUF files of luluw/embeddinggemmatext2-65k-en-ne-trim, a vocabulary-trimmed EmbeddingGemma 2 for English and Nepali. They run in llama.cpp (llama-server, llama-embedding) and tools built on it.

The vocabulary was reduced from 262,144 to 65,536 tokens, which removes about 100.7M of the 134.2M embedding parameters. The rest of the model (24 transformer layers, 512→768 projection, mean pooling) is unchanged and the weights were not retrained. Output embeddings are 768-dimensional.

Files

File	Quantization	Size	Notes
embeddinggemma2-65k-en-ne-trim-BF16.gguf	BF16	345 MB	Full-precision reference (bfloat16, not float16: activations overflow float16).
embeddinggemma2-65k-en-ne-trim-Q8_0.gguf	Q8_0	192 MB	8-bit, every weight tensor quantized to Q8_0. Near-lossless, about half the size of BF16.
embeddinggemma2-65k-en-ne-trim-Q4_K_M.gguf	Q4_K_M	114 MB	4-bit K-quant (llama.cpp's standard Q4_K_M recipe keeps a few sensitive tensors at 6-bit). Smallest.

Use Q8_0 unless you need the smallest file. Every file uses the same trimmed 65,536-token vocabulary.

Supported languages

Language	Status
English (en)	Supported
Nepali (ne, Devanagari)	Supported, with a dedicated share of the vocabulary
Other languages	Not targeted. Text still encodes through byte fallback but is split into many small pieces and quality drops. Use the original model for them.

Usage

llama-server (OpenAI-compatible API)

llama-server -hf luluw/embeddinggemmatext2-65k-en-ne-trim-GGUF:Q8_0 --embeddings --pooling mean -c 2048

curl http://127.0.0.1:8080/v1/embeddings -H "Content-Type: application/json" -d '{
  "input": ["task: search result | query: What causes the northern lights?"]
}'

llama-embedding (command line)

llama-embedding -hf luluw/embeddinggemmatext2-65k-en-ne-trim-GGUF:Q8_0 --pooling mean -p "title: none | text: The northern lights are caused by charged particles from the sun."

Prompts

The model is trained with short task prefixes. Always add them, the same way for every runtime:

Use	Text to embed
Search query	task: search result | query: {query}
Document / passage	title: none | text: {passage} (or title: {title} | text: {passage})
Question answering	task: question answering | query: {question}
Classification	task: classification | query: {text}
Clustering	task: clustering | query: {text}
Sentence similarity	task: sentence similarity | query: {text}

Normalize the vectors (L2) before cosine similarity. For smaller vectors you can keep the first 512, 256 or 128 dimensions of the 768 and re-normalize (Matryoshka).

Quality of the quantized files

Question → passage retrieval on Belebele (about 900 questions over about 488 passages per language, English eng_Latn and Nepali npi_Deva), maximum context 2048 tokens, with the prompts above. R@k is the share of questions whose correct passage is in the top k; MRR is the mean reciprocal rank. "Cosine to trimmed HF" is how close each file's passage embeddings are to the full-precision trimmed model (1.0 is identical).

English

Model	R@1	R@5	R@10	MRR	Cosine to trimmed HF (passages)
Original EmbeddingGemma 2 (HF, 262k vocab)	0.9056	0.9756	0.9822	0.9357	0.9870
Trimmed 65k (HF)	0.9067	0.9756	0.9811	0.9355	1.0000
Trimmed 65k GGUF BF16	0.9100	0.9756	0.9811	0.9371	1.0000
Trimmed 65k GGUF Q4_K_M	0.9078	0.9711	0.9800	0.9353	0.9887
Trimmed 65k GGUF Q8_0	0.9067	0.9744	0.9822	0.9356	0.9997

Nepali

Model	R@1	R@5	R@10	MRR	Cosine to trimmed HF (passages)
Original EmbeddingGemma 2 (HF, 262k vocab)	0.7911	0.9056	0.9311	0.8433	0.9992
Trimmed 65k (HF)	0.7933	0.9056	0.9278	0.8438	1.0000
Trimmed 65k GGUF BF16	0.7889	0.9044	0.9289	0.8415	1.0000
Trimmed 65k GGUF Q4_K_M	0.7944	0.9044	0.9356	0.8438	0.9895
Trimmed 65k GGUF Q8_0	0.7911	0.9056	0.9289	0.8432	0.9998

Belebele questions often say "according to the passage", so it is easier than open retrieval. Use it to compare these files against each other and the original, not as an absolute accuracy figure.

Vocabulary trimming

	Original	This model
Vocabulary size	262,144	65,536
Embedding parameters	134.2M	33.6M

The vocabulary was chosen from token counts on FineWeb-2 web text (70,000 English and 30,000 Nepali documents): all special and byte-fallback tokens are kept, the 12,288 most-used Nepali tokens come first, then English tokens by frequency, and every kept token keeps the smaller tokens it is merged from. A removed token is not lost: the tokenizer splits it into smaller pieces, so text decodes back exactly, but rare English words and uncommon Nepali words cost a few more tokens. Only file size and memory shrink; speed is unchanged. See luluw/embeddinggemmatext2-65k-en-ne-trim for the fidelity numbers against the original model.

Limitations

• Trimmed for English and Nepali; other languages are not targeted.
• Embeddings differ slightly from the original model, and the quantized files differ slightly from BF16 (see the table above).
• Benchmark scores published for the original EmbeddingGemma 2 were not re-measured for this model.
• Use BF16 or F32 inference with the original weights, never float16.

License

Apache 2.0, following the base model. The base model is by Google DeepMind. See the EmbeddingGemma 2 model card for the Gemma terms and prohibited use policy.