Flint layer sensitivity and footprint–quality qualification
The expanded study supports keeping the 41.730 GB all-expert FP8 release as the current FP8 candidate. Its 47.56% storage saving is real: 95.1% of all model parameters are FP8 and only the 4.9% non-expert parameters remain BF16. Adding 5.636 GB of early-layer BF16 improves numerical fidelity but gives no consistent task advantage across the two fresh task samples. A 35.738 GB mixed FP8/NVFP4 policy is a promising smaller alternative, with lower numerical fidelity and uncertain task differences. It is an evaluated policy, not a newly released checkpoint or a proven improvement.
The study evaluates 72 complete policies under 28, 32, 36, 40, 44 and 48 decimal GB budgets. BF16 at 79.571 GB is the accuracy reference. The published FP8 weights remain unchanged. Public diagnostics and exact policies make the allocation and measured tradeoffs reviewable.
All routed-expert payloads use the existing recipes: E4M3 FP8 with 128×128 FP32 scales, or E2M1 NVFP4 with per-16 E4M3 scales and an FP32 matrix scale. Every non-expert weight remains the original BF16 bytes. This study improves the evidence for precision assignment; it does not optimize FP8 scales or establish the best achievable FP8 quantizer.
Why the earlier evidence was insufficient
The previous screen tested all 47 MoE layers individually, but on only six 1,024-token sequences. Projection-level end-to-end screening covered layers 41–48. Those experiments established a broad early/late effect. They did not establish an exact, general ranking for allocating precision inside a model whose other layers are also quantized.
The new corpus contains two disjoint 12-window ranking panels, each with four prose, four code and four math windows. Each window has 1,024 tokens, giving 24,552 scored targets altogether. Every source row used in previous test corpora is excluded, and no new window shares a source row with another. An additional 12 disjoint 4,096-token windows (49,140 targets) are reserved for complete-policy validation. These are text-fidelity fixtures, not MBPP/GSM8K coding or reasoning accuracy tests.
Conditional measurements
Every MoE layer, 2–48 inclusive, receives three full-model interventions:
| Baseline | Change at one layer | Question |
|---|---|---|
| BF16 | Routed experts become FP8 | How much harm can an isolated perturbation cause? |
| All routed experts FP8 | Restore that layer's experts to BF16 | What does extra precision buy in an already quantized model? |
| All routed experts FP8 | Change that layer's experts to NVFP4 | What does further compression cost in that model? |
There are 141 intervened models, plus BF16, repeated BF16, all-FP8 and all-NVFP4 controls. Every gate/up/down projection group at all 47 layers is also checked at matched inputs: FP8/NVFP4 substitution on the BF16 trajectory and BF16-restoration/NVFP4-substitution on the FP8 trajectory. This is 564 local projection interventions. Local measurements alone are not called complete model accuracy or a globally valid projection ranking.
For each whole-layer intervention, residual-stream relative error and selected expert-set changes are recorded immediately, after 1/4/8 layers where available, and at the final layer. Identical BF16 execution is bit-exact through all layers; matched-input interventions preserve router IDs. The instrumented step agrees with the existing fused reference at a dense and an MoE layer, and the complete BF16 and FP8 baselines agree between the qualification and joint-policy runners. Actual FlashInfer CUDA fused-add RMSNorm is used throughout; attention and expert GEMMs remain the matched PyTorch reference. All measured processes have zero swap.
What replicates, and what does not
Spearman correlations between the layer rankings in the two new panels are:
| Ranking | Correlation |
|---|---|
| Harm from introducing FP8 into one BF16 layer | 0.994 |
| Benefit from restoring one layer in the FP8 model | 0.145 |
| Harm from replacing one FP8 layer with NVFP4 | 0.399 |
The broad early/late ordering is strongly reproducible for isolated FP8 perturbations. The exact ordering inside the early band is less certain: layers 2–6 have overlapping bootstrap rank intervals. Most importantly, that ordering is not a reliable ranking of the benefit from BF16 restoration inside the FP8 model. For example, the leading restoration candidate, layer 6, has a bootstrap rank interval of 1–12; layer 10 spans 1–34.
Intervals use 2,000 bootstrap resamples of whole windows, stratified by panel and domain. Tokens are not treated as independent observations. Intervals are marginal, not simultaneous multiple-comparison bounds, and describe variability within these sampled domains rather than all prompts or runtime kernels.
Local distortion also differs from downstream harm. FP8 changes the matched-input MoE output by 1.58% relative RMS at layer 2 and 4.18% at layer 40. Their immediate residual-stream errors are similar: mean-window 0.85% and 0.94%. By the final layer they become 14.90% and 2.25%, respectively; final-layer expert sets differ for 46.50% and 13.11% of tokens. These controls support downstream amplification of early perturbations. They do not show that early weights are intrinsically harder to encode, or that each early layer individually requires BF16.
Footprint-constrained joint search
The first joint pass evaluates 36 distinct policies: conditional rankings, isolated-layer rankings, depth rules, fixed random allocations, and projection allocations based on local output error. Comparisons within each tier have exactly equal tensor payload sizes. A second pass evaluates 36 neighboring policies, swapping equal-size projection groups around the best policy at each tier. All 72 complete policies are actually evaluated; their errors are not estimated by summing single-layer errors.
The selected tier footprints are 27.982, 31.860, 35.738, 39.968, 43.341 and 47.366 GB. Fixed anchors are all-NVFP4 (25.162 GB), the previously published FP8/NVFP4 mix (30.802 GB), and all routed experts FP8 (41.730 GB). Precision counts, parameter fractions and tensor bytes include the preserved non-expert weights and quantization scales. These fixed-format payload counts are not a claim about total runtime GPU memory.
The finalist per tier is chosen by selection KL, before longer-context or answer-level validation. Depth and conditional-allocation controls are retained for independent same-size comparisons on the longer text holdout. The separate task check covers all 500 OpenBookQA test questions, without the supporting fact, and 1,024 HellaSwag validation questions excluded from both earlier subsets. It uses untemplated zero-shot continuation scoring with both raw and character-normalized accuracy. This is a custom runner, not an official harness or a generation/coding-agent benchmark. Task labels are not used to fit weights or choose the finalist policies.
Held-out footprint and quality
The table uses the disjoint 49,140-target text holdout and 1,524 fresh task questions. BF16 achieves 42.20% OpenBookQA acc_norm and 75.68% HellaSwag acc_norm. Agreement means next-token agreement with BF16, not task accuracy; KL is the mean BF16-to-candidate distribution divergence. Precision fractions count all parameters, including non-experts. Tensor GB includes quantization scales; fractions may differ by 0.1 percentage point from rounding.
| Policy | Tensor GB | NVFP4 / FP8 / BF16 (% of all parameters) | Agreement | KL | OpenBookQA acc_norm | HellaSwag acc_norm |
|---|---|---|---|---|---|---|
| all-nvfp4 | 25.162 | 95.1 / 0.0 / 4.9 | 89.25% | 0.153252 | 42.60% | 75.59% |
| budget28-swap1 | 27.982 | 78.9 / 16.2 / 4.9 | 89.79% | 0.138774 | 43.00% | 75.78% |
| published-mix | 30.802 | 62.7 / 32.4 / 4.9 | 90.66% | 0.112959 | 43.00% | 75.59% |
| budget32-conditional | 31.860 | 56.7 / 38.5 / 4.9 | 90.90% | 0.110788 | 41.60% | 76.07% |
| budget36-swap1 | 35.738 | 34.4 / 60.7 / 4.9 | 91.89% | 0.090109 | 43.00% | 76.37% |
| budget40-swap5 | 39.968 | 10.1 / 85.0 / 4.9 | 92.94% | 0.077981 | 43.00% | 76.27% |
| all-fp8 | 41.730 | 0.0 / 95.1 / 4.9 | 93.59% | 0.072432 | 41.60% | 75.78% |
| budget44-depth | 43.341 | 0.0 / 91.1 / 8.9 | 93.54% | 0.066195 | 42.60% | 76.27% |
| budget48-depth | 47.366 | 0.0 / 81.0 / 19.0 | 94.01% | 0.055908 | 43.40% | 75.39% |
budget44-depth restores routed experts in layers 2–3 to BF16;
budget48-depth restores layers 2–8. The policies below 41.730 GB retain no
additional BF16 expert weights. swap names identify complete projection-level
allocations from the equal-size neighborhood search, not checkpoint formats.
The published mix keeps layers 2–17 FP8 and layers 18–48 NVFP4.
Exact layer/projection maps
are supplied for every finalist and same-size control.
Raw accuracy and NLL are also retained, so conclusions do not depend on one normalization convention:
| Policy | OpenBookQA raw accuracy | HellaSwag raw accuracy | Text NLL | BF16 share of tensor bytes |
|---|---|---|---|---|
| BF16 reference | 32.60% | 59.77% | 2.214730 | 100% |
| all-nvfp4 | 33.00% | 60.16% | 2.219005 | 15.39% |
| budget28-swap1 | 33.00% | 59.96% | 2.225089 | 13.84% |
| published-mix | 34.80% | 59.77% | 2.203223 | 12.57% |
| budget32-conditional | 34.00% | 60.16% | 2.210039 | 12.15% |
| budget36-swap1 | 32.80% | 60.06% | 2.219417 | 10.83% |
| budget40-swap5 | 32.80% | 60.16% | 2.218666 | 9.69% |
| all-fp8 | 31.80% | 60.25% | 2.217737 | 9.28% |
| budget44-depth | 33.00% | 60.55% | 2.219923 | 16.37% |
| budget48-depth | 31.80% | 60.16% | 2.219438 | 31.98% |
All nine candidates' paired 95% bootstrap intervals for normalized accuracy versus BF16 include zero on both tasks. This does not establish equivalence or exclude smaller regressions. Intervals resample whole questions 10,000 times; they are marginal intervals without correction for comparing many candidates. Raw and normalized outcomes sometimes disagree. For example, the published 30.802 GB mix has higher raw OpenBookQA accuracy than BF16, but its normalized accuracy difference remains uncertain. These samples do not establish improved capability from quantization.
The 35.738 GB policy saves 14.36% relative to all-FP8, or 55.09% relative to BF16. Against all-FP8, its normalized accuracy changes are +1.40 percentage points on OpenBookQA (95% interval −0.40 to +3.40) and +0.59 points on HellaSwag (−0.39 to +1.56). Its lower text agreement and higher KL are clear tradeoffs; its positive task point estimates are not a demonstrated task improvement.
Increasing all-FP8 from 41.730 to 47.366 GB adds 13.51% storage, raises agreement by 0.42 percentage point, and reduces KL from 0.072432 to 0.055908. Normalized OpenBookQA improves by 1.80 points relative to all-FP8 (marginal 95% interval +0.20 to +3.40), but raw OpenBookQA is unchanged and normalized HellaSwag changes by −0.39 points (−1.37 to +0.59). This is evidence of a numerical-fidelity benefit and a task-specific result, not a consistent accuracy return for the extra BF16. There is no basis here for recommending a mostly-BF16 checkpoint.
Does a more elaborate layer ranking beat depth?
On the independent text holdout, the conditional-ranking allocation has higher KL than the same-size depth allocation at five of six budgets. Conditional minus depth KL is +0.00754 at 36 GB, +0.00359 at 40 GB, +0.00501 at 44 GB and +0.00704 at 48 GB; their marginal whole-window bootstrap intervals exclude zero. The 28 and 32 GB comparisons remain uncertain. Selection-set improvements from local swaps also do not reliably transfer to the holdout.
Consequently, an unstable per-layer restoration ranking should not be treated as a precise precision-allocation prescription. The robust finding is that isolated early perturbations propagate farther; it is not that the early expert matrices are intrinsically less representable in FP8. Joint evaluation at a fixed storage budget and independent validation are essential.
This is an observed footprint–quality tradeoff, not the globally optimal Pareto frontier. The search covers 72 policies rather than all possible assignments, keeps non-experts BF16, uses fixed quantization recipes, and has only two fresh multiple-choice task checks. It does not establish long-form generation, reasoning, coding-agent quality or native compressed-kernel behavior. Within those limits, the evidence favors retaining the current all-expert FP8 release and keeping the smaller mixed policies visible as explicit alternatives.
Reproduction
tools/flint_qualification_corpus.py freezes the disjoint text panels.
tools/flint_layer_qualification.py performs the full-model and matched-input
interventions. tools/flint_qualification_analysis.py computes bootstrap rankings
and proposes exact-budget policies. tools/flint_layer_policy.py evaluates
complete policies, while tools/flint_pareto_neighbors.py performs the
footprint-preserving search pass. tools/flint_pareto_select.py freezes finalists
and controls. tools/flint_pareto_tasks.py prepares and scores the new questions;
tools/flint_qualification_report.py combines the results and plots.
Source weights and dataset revisions are pinned. The frozen fixtures, policy maps, per-window results, choice likelihoods, precision counts, code hashes and plots are included with the release diagnostics. Run logs, checkpoints and generated profiles remain untracked. Native serving support and throughput are separate qualification tasks; this study makes no serving throughput claim.
The source revision is 2a247be0c0aec895ccb0ac7ca92229ec6df37288.
Evaluation used Torch 2.8.0+cu128, Triton 3.4.0 and FlashInfer 0.6.16.post1 on
an RTX PRO 6000 Blackwell Server GPU. The layer study peaked at 77.043 GB of
allocated GPU memory; the task run peaked at 41.324 GB. These are evaluator
allocations, not native compressed serving footprints. Each result records
the package versions, source and script hashes. The initial task-fixture
producer is preserved under reproduction/history/; the evaluation script's
later dataset-registry fix is recorded separately.
To repeat the independent checks, download the release's reproduction/ and
diagnostics/qualification/ directories, and the pinned source weights. Run
from the downloaded release directory with the versions above and a CUDA
toolchain available for FlashInfer. Set the two paths below to the source and
an empty output directory. The prepare commands build both expert formats
without using evaluation labels:
flint_source=/path/to/pinned-flint-source
qualification_work=/path/to/empty-qualification-work
mkdir -p "$qualification_work"
for precision in all-fp8 all-nvfp4; do
python reproduction/flint_quantize.py prepare \
--source "$flint_source" --work "$qualification_work" \
--corpus reproduction/corpus-layer-qualification.json \
--precision-policy "diagnostics/qualification/policies/$precision.json"
done
python reproduction/flint_layer_policy.py evaluate \
--source "$flint_source" --work "$qualification_work" \
--corpus reproduction/corpus-layer-qualification.json --split validation \
--policies diagnostics/qualification/frozen-policies.json --label validation
python reproduction/flint_pareto_tasks.py evaluate \
--source "$flint_source" --work "$qualification_work" \
--corpus reproduction/task-pareto-corpus.json \
--policies diagnostics/qualification/task-policies.json \
--output "$qualification_work/tasks.json"
The frozen fixtures reproduce this comparison; they are no longer unseen data for a subsequent search. Any further tuning should use another holdout before claiming an improved frontier.