amd/Mistral-Large-4-1T-A52B-Quark-MXFP4

Model Overview

• Model Name: amd/Mistral-Large-4-1T-A52B-Quark-MXFP4
• Model Architecture: PixtralForConditionalGeneration
  • Input: Text, Image
  • Output: Text
• Supported Hardware Microarchitecture: AMD MI350/MI355
• ROCm: 10.0 (HIP runtime 7.15.26333)
• PyTorch: 2.13.0+rocm10.0.0
• Transformers: 5.20.2
• Operating System(s): Linux
• Inference Engine: vLLM
• Container base image: This model was tested using the docker container vllm/vllm-openai-rocm:nightly-rocm100-00b6c4c565a8732c131e1f452d4a17803bb0842b (vLLM nightly 26/10/2026) on a development branch for Mistral Large 4.
• Model Optimizer: upcoming AMD Quark 0.14
  • Routed experts weight quantization: OCP MXFP4, Static
  • Routed experts activation quantization: OCP MXFP4, Dynamic
  • All other layers: unchanged precision compared to base mistralai/Mistral-Large-4-1T-A52B

Model deployment

Please refer to vLLM recipe and SGLang cookbook.

This model was tested with vLLM on 4x MI355X. The Mistral tokenizer (tekken.json v15, with low / medium / high reasoning efforts) requires mistral-common 1.13.0 or later, which supports these reasoning effort levels:

pip install "mistral-common[image]>=1.13.0"

Example vLLM server:

OMP_NUM_THREADS=1 VLLM_ROCM_USE_AITER=1 \
vllm serve amd/Mistral-Large-4-1T-A52B-Quark-MXFP4 \
    --tokenizer-mode mistral \
    --config-format mistral \
    --load-format mistral \
    --tensor-parallel-size 4 \
    --monokernel \
    --kv-cache-dtype fp8 \
    --max-model-len 262144 \
    --max-num-batched-tokens 8192 \
    --gpu-memory-utilization 0.85 \
    --reasoning-parser mistral \
    --speculative-config '{"method": "dspark", "num_speculative_tokens": 3}'

Notes:

• --monokernel is only supported on MI350X / MI355X (gfx950, 256 CUs) at tensor parallelism TP=4. It supports dspark speculative decoding, and fp8 and bf16 KV cache.
• --speculative-config enables DSpark speculative decoding with the checkpoint's draft head, which is kept out of the quantization (see mtp_dspark.* in EXCLUDE). num_speculative_tokens must be 3, the depth of the draft block; drop the flag for a non-speculative baseline.

Model Quantization

The model was quantized from mistralai/Mistral-Large-4-1T-A52B with AMD-Quark, through the file-to-file flow: the safetensors shards are quantized one at a time, so the 1T-parameter model is never materialized.

from quark.torch import ModelQuantizer
from quark.torch.quantization.config.config import QConfig
from quark.torch.quantization.config.template import MXFP4Scheme

SRC = "/path/to/Mistral-Large-4-1T-A52B"
OUT = "/path/to/Mistral-Large-4-1T-A52B-Quark-MXFP4"

# Pre-quantized layers kept in their source format. Native Mistral module names,
# which differ from the Hugging Face layout (layers.N.attention.wq rather than
# model.layers.N.self_attn.q_proj).
EXCLUDE = [
    "*attention.*",
    "*shared_experts.*",
    "*feed_forward.*",
    "*.gate",
    "output",
    "vision_encoder.*",
    "patch_merger.*",
    "vision_language_adapter.*",
    "mtp_dspark.*",
]

quant_config = QConfig(global_quant_config=MXFP4Scheme().config, exclude=EXCLUDE)
ModelQuantizer(quant_config).direct_quantize_checkpoint(
    pretrained_model_path=SRC,
    save_path=OUT,
    keep_excluded_layers_as_original_model_state=True,
    device="cuda",
)

Evaluation

All tasks are evaluated with a single lm-evaluation-harness command, each task with its own maximum generation length (--max_gen_toks_per_task) and number of repeats (--repeats_per_task); a group name such as mmlu_pro_chat applies to all its subtasks:

lm_eval --model local-chat-completions \
    --model_args "model=<server-model-id>,base_url=http://<server-ip>/v1/chat/completions,num_concurrent=16,timeout=99999,tokenized_requests=False,think_end_token='[/THINK]',tokenizer=<tokenizer-path>,tokenizer_backend=huggingface,server_sampling_defaults=True" \
    --tasks gpqa_diamond_cot_zeroshot,aime26,mmlu_pro_chat \
    --gen_kwargs 'reasoning_effort=medium,temperature=1.0,top_p=0.95,until=["[INST]"]' \
    --log_samples --output_path results \
    --apply_chat_template \
    --max_gen_toks_per_task gpqa_diamond_cot_zeroshot=96000 aime26=130000 mmlu_pro_chat=16384 \
    --repeats_per_task gpqa_diamond_cot_zeroshot=8 aime26=8 mmlu_pro_chat=1 \
    --generation-length

Evaluation results correctness are validated by an agent.

Benchmarks:

• GPQA Diamond: task gpqa_diamond_cot_zeroshot (198 questions), maximum generation length 96k tokens.
• AIME26: task aime26 (30 problems), maximum generation length 130k tokens.
• MMLU-Pro (chat): task mmlu_pro_chat (14 subjects, 12,032 questions, 5-shot as multi-turn chat), maximum generation length 16,384 tokens.

The baseline is evaluated with a BF16 KV cache, amd/Mistral-Large-4-1T-A52B-Quark-MXFP4 with an FP8 KV cache (--kv-cache-dtype fp8); see Evaluation settings.

Summary

mean@8: accuracy averaged over 8 independent repeats (GPQA Diamond, AIME26). mean@1: accuracy of a single repeat (MMLU-Pro).

Benchmark	mistralai/Mistral-Large-4-1T-A52B baseline	This model (MXFP4)
GPQA Diamond (mean@8)	88.89	88.13
AIME26 (mean@8)	98.75	98.75
MMLU-Pro, chat (mean@1)	85.40	85.04

Detailed results

Benchmark	Metric	mistralai/Mistral-Large-4-1T-A52B baseline	This model (MXFP4)
GPQA Diamond	mean@8	88.89	88.13
	maj@8	89.39	89.90
	Mean tokens per answer	10,389	10,505
	% of answers hitting maximum generation length limit	0.82%	1.07%
AIME26	mean@8	98.75	98.75
	maj@8	100.00	100.00
	Mean tokens per answer	12,032	12,574
	% of answers hitting maximum generation length limit	0.00%	0.00%
MMLU-Pro, chat	mean@1	85.40	85.04
	Mean tokens per answer	2,147	2,098
	% of answers hitting maximum generation length limit	0.97%	0.96%

Evaluation settings

Details

mistralai/Mistral-Large-4-1T-A52B is evaluated with tensor parallelism TP=8 and DSpark speculative decoding (--speculative-config '{"method": "dspark", "num_speculative_tokens": 3}'), with BF16 KV cache.

amd/Mistral-Large-4-1T-A52B-Quark-MXFP4 is evaluated with tensor parallelism TP=4, with --monokernel and DSpark speculative decoding (--speculative-config '{"method": "dspark", "num_speculative_tokens": 3}'), with FP8 KV cache (--kv-cache-dtype fp8).

Other evaluation settings include:

• The DSA indexer key cache is always FP8.
• Sampling, for both models and all tasks: temperature=1.0, top_p=0.95, no top_k, no fixed seed (each repeat is an independent sample). These are the default sampling parameters of the base mistralai/Mistral-Large-4-1T-A52B checkpoint.
• Reasoning effort: medium
• The evaluation servers run without --reasoning-parser mistral (lm-eval reads the reasoning trace from content), with --max-model-len 163840 --gpu-memory-utilization 0.83.
• Evaluations use lm-evaluation-harness, branch lm-eval-fixes, commit 503ad16, against a running vllm serve OpenAI-compatible endpoint (chat API).

Scoring notes (all tasks):

• A generation that hits the maximum generation length, never closes its reasoning ([/THINK]), or gives an empty final answer is scored incorrect and does not vote in maj@8.
• maj@8 ties (no single most frequent answer among the 8 repeats) are broken by the answer that appears first in repeat order. On GPQA Diamond, 0 of 198 questions are ties for mistralai/Mistral-Large-4-1T-A52B and 1 of 198 for amd/Mistral-Large-4-1T-A52B-Quark-MXFP4 (3 of 198 with the fp8_ds_mla KV cache below), so maj@8 can move by up to about 1.5 points depending on the tie rule. AIME26 has no ties.
• Mean tokens per answer: mean generation length over all generations (8 per question for GPQA Diamond and AIME26, 1 for MMLU-Pro), reasoning trace included.
• One amd/Mistral-Large-4-1T-A52B-Quark-MXFP4 GPQA Diamond answer (fp8 KV cache) was manually re-scored as correct: its final line is The answer is (Bs)., the right letter with a stray character, which the extractor does not parse. There are no such cases for mistralai/Mistral-Large-4-1T-A52B, for AIME26, or with the fp8_ds_mla KV cache.

Reference only: results with fp8_ds_mla KV cache

Evaluation results with fp8_ds_mla KV cache (unreleased on Instinct, reference only):

Model	Task	KV cache dtype	Repeats	mean	maj@8	Mean tokens per answer	% of answers hitting maximum generation length limit
This model (MXFP4)	GPQA Diamond	fp8_ds_mla	8	87.50	88.38	10,410	1.14%
This model (MXFP4)	AIME26	fp8_ds_mla	8	99.17	100.00	11,637	0.00%
This model (MXFP4)	MMLU-Pro (chat)	fp8_ds_mla	1	85.26	N/A	2,087	0.96%

These results use TP=4 with DSpark speculative decoding and without --monokernel.

Difference with --kv-cache-dtype fp8:

• fp8 on AMD Instinct: the latent KV cache (512 latent + 64 RoPE values per token, 576 bytes) is stored as plain FP8 E4M3 with a single per-tensor scale of 1.0 (the checkpoint ships no KV scales).
• fp8_ds_mla: reference only, using the DeepSeek-V3.2 / FlashMLA fp8_ds_mla KV cache layout: 656 bytes per token, the 512 latent values in FP8 E4M3 with one FP32 power-of-two scale per block of 128 values, and the 64 RoPE values in BF16. The per-block scales keep small-magnitude latents (e.g. in the first layers) more precise than a single per-tensor scale.

License

Modifications Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.