Groot-65B-A6B-261023-NVFP4-GGUF

Groot-65B-A6B-261023 is an open-weight (qwen-community-1.0), natively multimodal Mixture-of-Experts (MoE) reasoning model engineered for complex real-world tasks, agentic coding, and native tool calling. Derived from Qwen3.8-Flash-Next and post-trained on fewer than 0.1 billion open-source tokens, it comprises roughly 65B total model parameters with ~6B activated per token across 48 hybrid layers (12 Qwen Sparse Attention and 36 Gated DeltaNet) and 256 routed experts (10 activated plus 1 shared), paired with a 1.4B Multi-Token Prediction (MTP) layer and a 51B n-gram embedding lookup table that adds zero activated compute and can be offloaded to host memory. Despite having 46.7% fewer parameters than Qwen3.5-122B-A10B and 48.0% fewer than Qwen3.8-Flash-Next, it averages 51.51 across 16 benchmarks—outperforming Qwen3.5-122B-A10B on 14 of 16 benchmarks (+12.89 points overall) and beating Qwen3.8-Flash-Next on HLE and NL2Repo-Bench—while generating 23.9% and 28.3% fewer output tokens, respectively. The model operates with thinking and preserved-thinking enabled by default, offers a tunable reasoning_effort parameter (xhigh, medium, low) as well as non-thinking mode, and supports a native 262,144-token (256K) context window extensible to ~1M tokens via YaRN. Released in BF16, FP8, NVFP4, and GGUF (Q4_K_M) formats for SGLang, vLLM, KTransformers, and Hugging Face Transformers, it runs the full 256K context in GGUF on a single 96GB GPU (RTX PRO 6000 Blackwell or H20), in GGUF or NVFP4 on 128GB unified-memory systems (DGX Spark, AMD Strix Halo, Mac M5 Max), in FP8 on one H200 or two H100s, and in BF16 across four 80GB GPUs.

MTP — Speculative Decoding: Yes

Multi-Token Prediction (MTP) GGUF is a specialized GGUF model file format extension that integrates speculative decoding directly into the model weights to significantly accelerate local inference. Unlike traditional speculative decoding which requires a separate, smaller "draft" model, MTP GGUF files include additional output heads within the main model architecture that predict multiple future tokens in a single forward pass.

Model Files

File Name	Quant Type	File Size	File Link	Description
Groot-65B-A6B-261023-NVFP4.gguf	NVFP4	149 GB	Link	NVFP4 quantization. Highly optimized for NVIDIA Blackwell hardware accelerators (e.g., RTX Pro 6000, RTX 50-series, B200, B300), offering native hardware-accelerated FP4 Tensor Core speeds and a strong balance of size and performance for this 65B parameter MoE model.
Groot-65B-A6B-261023-NVFP4.mmproj.gguf	mmproj	908 MB	Link	Multimodal projection file. Used for vision/language models.

llama.cpp

LLM inference in C/C++ — https://github.com/ggml-org/llama.cpp

---

Quantized using llama.cpp by prithivMLmods