Blame for diagnostics/qualification/report.md

# Flint layer sensitivity and footprint–quality qualification

The expanded study supports keeping the **41.730 GB all-expert FP8 release** as
the current FP8 candidate. Its 47.56% storage saving is real: 95.1% of all model
parameters are FP8 and only the 4.9% non-expert parameters remain BF16. Adding
5.636 GB of early-layer BF16 improves numerical fidelity but gives no consistent
task advantage across the two fresh task samples. A **35.738 GB mixed FP8/NVFP4
policy** is a promising smaller alternative, with lower numerical fidelity and
uncertain task differences. It is an evaluated policy, not a newly released
checkpoint or a proven improvement.

The study evaluates 72 complete policies under 28, 32, 36, 40, 44 and 48 decimal
GB budgets. BF16 at 79.571 GB is the accuracy reference. The published FP8
weights remain unchanged. [Public diagnostics and exact policies](https://huggingface.co/HawkBearPig/flint-FP8-BF16/tree/main/diagnostics/qualification)
make the allocation and measured tradeoffs reviewable.

All routed-expert payloads use the existing recipes: E4M3 FP8 with 128×128
FP32 scales, or E2M1 NVFP4 with per-16 E4M3 scales and an FP32 matrix scale.
Every non-expert weight remains the original BF16 bytes. This study improves
the evidence for precision assignment; it does not optimize FP8 scales or
establish the best achievable FP8 quantizer.

## Why the earlier evidence was insufficient

The previous screen tested all 47 MoE layers individually, but on only six
1,024-token sequences. Projection-level end-to-end screening covered layers
41–48. Those experiments established a broad early/late effect. They did not
establish an exact, general ranking for allocating precision inside a model
whose other layers are also quantized.

The new corpus contains two disjoint 12-window ranking panels, each with four
prose, four code and four math windows. Each window has 1,024 tokens, giving
24,552 scored targets altogether. Every source row used in previous test
corpora is excluded, and no new window shares a source row with another.
An additional 12 disjoint 4,096-token windows (49,140 targets) are reserved for
complete-policy validation. These are text-fidelity fixtures, not MBPP/GSM8K
coding or reasoning accuracy tests.

## Conditional measurements

Every MoE layer, 2–48 inclusive, receives three full-model interventions:

| Baseline | Change at one layer | Question |
|---|---|---|
| BF16 | Routed experts become FP8 | How much harm can an isolated perturbation cause? |
| All routed experts FP8 | Restore that layer's experts to BF16 | What does extra precision buy in an already quantized model? |
| All routed experts FP8 | Change that layer's experts to NVFP4 | What does further compression cost in that model? |

There are 141 intervened models, plus BF16, repeated BF16, all-FP8 and all-NVFP4
controls. Every gate/up/down projection group at all 47 layers is also checked
at matched inputs: FP8/NVFP4 substitution on the BF16 trajectory and
BF16-restoration/NVFP4-substitution on the FP8 trajectory. This is 564 local
projection interventions. Local measurements alone are not called complete
model accuracy or a globally valid projection ranking.

For each whole-layer intervention, residual-stream relative error and selected
expert-set changes are recorded immediately, after 1/4/8 layers where available,
and at the final layer. Identical BF16 execution is bit-exact through all layers;
matched-input interventions preserve router IDs. The instrumented step agrees
with the existing fused reference at a dense and an MoE layer, and the complete
BF16 and FP8 baselines agree between the qualification and joint-policy runners.
Actual FlashInfer CUDA fused-add RMSNorm is used throughout; attention and expert
GEMMs remain the matched PyTorch reference. All measured processes have zero swap.

## What replicates, and what does not

Spearman correlations between the layer rankings in the two new panels are:

| Ranking | Correlation |
|---|---:|
| Harm from introducing FP8 into one BF16 layer | 0.994 |
| Benefit from restoring one layer in the FP8 model | 0.145 |
| Harm from replacing one FP8 layer with NVFP4 | 0.399 |

The broad early/late ordering is strongly reproducible for isolated FP8
perturbations. The exact ordering inside the early band is less certain:
layers 2–6 have overlapping bootstrap rank intervals. Most importantly, that
ordering is not a reliable ranking of the benefit from BF16 restoration inside
the FP8 model. For example, the leading restoration candidate, layer 6, has a
bootstrap rank interval of 1–12; layer 10 spans 1–34.

Intervals use 2,000 bootstrap resamples of whole windows, stratified by panel
and domain. Tokens are not treated as independent observations. Intervals are
marginal, not simultaneous multiple-comparison bounds, and describe variability
within these sampled domains rather than all prompts or runtime kernels.

Local distortion also differs from downstream harm. FP8 changes the matched-input
MoE output by 1.58% relative RMS at layer 2 and 4.18% at layer 40. Their immediate
residual-stream errors are similar: mean-window 0.85% and 0.94%. By the final
layer they become 14.90% and 2.25%, respectively; final-layer expert sets differ
for 46.50% and 13.11% of tokens. These controls support downstream amplification
of early perturbations. They do not show that early weights are intrinsically
harder to encode, or that each early layer individually requires BF16.

## Footprint-constrained joint search

The first joint pass evaluates 36 distinct policies: conditional rankings,
isolated-layer rankings, depth rules, fixed random allocations, and projection
allocations based on local output error. Comparisons within each tier have
exactly equal tensor payload sizes. A second pass evaluates 36 neighboring
policies, swapping equal-size projection groups around the best policy at each
tier. All 72 complete policies are actually evaluated; their errors are not
estimated by summing single-layer errors.

The selected tier footprints are 27.982, 31.860, 35.738, 39.968, 43.341 and
47.366 GB. Fixed anchors are all-NVFP4 (25.162 GB), the previously published
FP8/NVFP4 mix (30.802 GB), and all routed experts FP8 (41.730 GB). Precision
counts, parameter fractions and tensor bytes include the preserved non-expert
weights and quantization scales. These fixed-format payload counts are not a
claim about total runtime GPU memory.

The finalist per tier is chosen by selection KL, before longer-context or
answer-level validation. Depth and conditional-allocation controls are retained
for independent same-size comparisons on the longer text holdout. The separate
task check covers all 500 OpenBookQA test questions, without the supporting fact,
and 1,024 HellaSwag validation questions excluded from both earlier subsets.
It uses untemplated zero-shot continuation scoring with both raw and
character-normalized accuracy. This is a custom runner, not an official harness
or a generation/coding-agent benchmark. Task labels are not used to fit weights
or choose the finalist policies.

## Held-out footprint and quality

The table uses the disjoint 49,140-target text holdout and 1,524 fresh task
questions. BF16 achieves **42.20% OpenBookQA acc_norm** and **75.68% HellaSwag
acc_norm**. Agreement means next-token agreement with BF16, not task accuracy;
KL is the mean BF16-to-candidate distribution divergence. Precision fractions
count all parameters, including non-experts. Tensor GB includes quantization
scales; fractions may differ by 0.1 percentage point from rounding.

| Policy | Tensor GB | NVFP4 / FP8 / BF16 (% of all parameters) | Agreement | KL | OpenBookQA acc_norm | HellaSwag acc_norm |
|---|---:|---:|---:|---:|---:|---:|
| all-nvfp4 | 25.162 | 95.1 / 0.0 / 4.9 | 89.25% | 0.153252 | 42.60% | 75.59% |
| budget28-swap1 | 27.982 | 78.9 / 16.2 / 4.9 | 89.79% | 0.138774 | 43.00% | 75.78% |
| published-mix | 30.802 | 62.7 / 32.4 / 4.9 | 90.66% | 0.112959 | 43.00% | 75.59% |
| budget32-conditional | 31.860 | 56.7 / 38.5 / 4.9 | 90.90% | 0.110788 | 41.60% | 76.07% |
| budget36-swap1 | 35.738 | 34.4 / 60.7 / 4.9 | 91.89% | 0.090109 | 43.00% | 76.37% |
| budget40-swap5 | 39.968 | 10.1 / 85.0 / 4.9 | 92.94% | 0.077981 | 43.00% | 76.27% |
| all-fp8 | 41.730 | 0.0 / 95.1 / 4.9 | 93.59% | 0.072432 | 41.60% | 75.78% |
| budget44-depth | 43.341 | 0.0 / 91.1 / 8.9 | 93.54% | 0.066195 | 42.60% | 76.27% |
| budget48-depth | 47.366 | 0.0 / 81.0 / 19.0 | 94.01% | 0.055908 | 43.40% | 75.39% |

`budget44-depth` restores routed experts in layers 2–3 to BF16;
`budget48-depth` restores layers 2–8. The policies below 41.730 GB retain no
additional BF16 expert weights. `swap` names identify complete projection-level
allocations from the equal-size neighborhood search, not checkpoint formats.
The published mix keeps layers 2–17 FP8 and layers 18–48 NVFP4.
[Exact layer/projection maps](https://huggingface.co/HawkBearPig/flint-FP8-BF16/tree/main/diagnostics/qualification/policies)
are supplied for every finalist and same-size control.

Raw accuracy and NLL are also retained, so conclusions do not depend on one
normalization convention:

| Policy | OpenBookQA raw accuracy | HellaSwag raw accuracy | Text NLL | BF16 share of tensor bytes |
|---|---:|---:|---:|---:|
| BF16 reference | 32.60% | 59.77% | 2.214730 | 100% |
| all-nvfp4 | 33.00% | 60.16% | 2.219005 | 15.39% |
| budget28-swap1 | 33.00% | 59.96% | 2.225089 | 13.84% |
| published-mix | 34.80% | 59.77% | 2.203223 | 12.57% |
| budget32-conditional | 34.00% | 60.16% | 2.210039 | 12.15% |
| budget36-swap1 | 32.80% | 60.06% | 2.219417 | 10.83% |
| budget40-swap5 | 32.80% | 60.16% | 2.218666 | 9.69% |
| all-fp8 | 31.80% | 60.25% | 2.217737 | 9.28% |
| budget44-depth | 33.00% | 60.55% | 2.219923 | 16.37% |
| budget48-depth | 31.80% | 60.16% | 2.219438 | 31.98% |

All nine candidates' paired 95% bootstrap intervals for normalized accuracy
versus BF16 include zero on both tasks. This does not establish equivalence or
exclude smaller regressions. Intervals resample whole questions 10,000 times;
they are marginal intervals without correction for comparing many candidates.
Raw and normalized outcomes sometimes disagree. For example, the published
30.802 GB mix has higher raw OpenBookQA accuracy than BF16, but its normalized
accuracy difference remains uncertain. These samples do not establish improved
capability from quantization.

The 35.738 GB policy saves **14.36% relative to all-FP8**, or **55.09% relative to
BF16**. Against all-FP8, its normalized accuracy changes are +1.40 percentage
points on OpenBookQA (95% interval −0.40 to +3.40) and +0.59 points on HellaSwag
(−0.39 to +1.56). Its lower text agreement and higher KL are clear tradeoffs;
its positive task point estimates are not a demonstrated task improvement.

Increasing all-FP8 from 41.730 to 47.366 GB adds **13.51% storage**, raises agreement
by **0.42 percentage point**, and reduces KL from 0.072432 to 0.055908.
Normalized OpenBookQA improves by 1.80 points relative to all-FP8 (marginal 95%
interval +0.20 to +3.40), but raw OpenBookQA is unchanged and normalized HellaSwag
changes by −0.39 points (−1.37 to +0.59). This is evidence of a numerical-fidelity
benefit and a task-specific result, not a consistent accuracy return for the
extra BF16. There is no basis here for recommending a mostly-BF16 checkpoint.

![Measured footprint and held-out quality](https://huggingface.co/HawkBearPig/flint-FP8-BF16/resolve/main/diagnostics/qualification/footprint-quality-pareto.png)

## Does a more elaborate layer ranking beat depth?

On the independent text holdout, the conditional-ranking allocation has higher
KL than the same-size depth allocation at five of six budgets. Conditional
minus depth KL is +0.00754 at 36 GB, +0.00359 at 40 GB, +0.00501 at 44 GB and
+0.00704 at 48 GB; their marginal whole-window bootstrap intervals exclude zero.
The 28 and 32 GB comparisons remain uncertain. Selection-set improvements from
local swaps also do not reliably transfer to the holdout.

Consequently, an unstable per-layer restoration ranking should not be treated
as a precise precision-allocation prescription. The robust finding is that
isolated early perturbations propagate farther; it is not that the early
expert matrices are intrinsically less representable in FP8. Joint evaluation
at a fixed storage budget and independent validation are essential.

![Local error and downstream propagation](https://huggingface.co/HawkBearPig/flint-FP8-BF16/resolve/main/diagnostics/qualification/local-vs-propagated-error.png)

This is an observed footprint–quality tradeoff, not the globally optimal Pareto
frontier. The search covers 72 policies rather than all possible assignments,
keeps non-experts BF16, uses fixed quantization recipes, and has only two fresh
multiple-choice task checks. It does not establish long-form generation,
reasoning, coding-agent quality or native compressed-kernel behavior. Within
those limits, the evidence favors retaining the current all-expert FP8 release
and keeping the smaller mixed policies visible as explicit alternatives.

## Reproduction

`tools/flint_qualification_corpus.py` freezes the disjoint text panels.
`tools/flint_layer_qualification.py` performs the full-model and matched-input
interventions. `tools/flint_qualification_analysis.py` computes bootstrap rankings
and proposes exact-budget policies. `tools/flint_layer_policy.py` evaluates
complete policies, while `tools/flint_pareto_neighbors.py` performs the
footprint-preserving search pass. `tools/flint_pareto_select.py` freezes finalists
and controls. `tools/flint_pareto_tasks.py` prepares and scores the new questions;
`tools/flint_qualification_report.py` combines the results and plots.

Source weights and dataset revisions are pinned. The frozen fixtures, policy
maps, per-window results, choice likelihoods, precision counts, code hashes and
plots are included with the release diagnostics. Run logs, checkpoints and generated
profiles remain untracked. Native serving support and throughput are separate
qualification tasks; this study makes no serving throughput claim.

The source revision is `2a247be0c0aec895ccb0ac7ca92229ec6df37288`.
Evaluation used Torch 2.8.0+cu128, Triton 3.4.0 and FlashInfer 0.6.16.post1 on
an RTX PRO 6000 Blackwell Server GPU. The layer study peaked at 77.043 GB of
allocated GPU memory; the task run peaked at 41.324 GB. These are evaluator
allocations, not native compressed serving footprints. Each result records
the package versions, source and script hashes. The initial task-fixture
producer is preserved under `reproduction/history/`; the evaluation script's
later dataset-registry fix is recorded separately.

To repeat the independent checks, download the release's `reproduction/` and
`diagnostics/qualification/` directories, and the pinned source weights. Run
from the downloaded release directory with the versions above and a CUDA
toolchain available for FlashInfer. Set the two paths below to the source and
an empty output directory. The prepare commands build both expert formats
without using evaluation labels:

```bash
flint_source=/path/to/pinned-flint-source
qualification_work=/path/to/empty-qualification-work
mkdir -p "$qualification_work"
for precision in all-fp8 all-nvfp4; do
  python reproduction/flint_quantize.py prepare \
    --source "$flint_source" --work "$qualification_work" \
    --corpus reproduction/corpus-layer-qualification.json \
    --precision-policy "diagnostics/qualification/policies/$precision.json"
done

python reproduction/flint_layer_policy.py evaluate \
  --source "$flint_source" --work "$qualification_work" \
  --corpus reproduction/corpus-layer-qualification.json --split validation \
  --policies diagnostics/qualification/frozen-policies.json --label validation

python reproduction/flint_pareto_tasks.py evaluate \
  --source "$flint_source" --work "$qualification_work" \
  --corpus reproduction/task-pareto-corpus.json \
  --policies diagnostics/qualification/task-policies.json \
  --output "$qualification_work/tasks.json"
```

The frozen fixtures reproduce this comparison; they are no longer unseen data
for a subsequent search. Any further tuning should use another holdout before
claiming an improved frontier.