# HANDOFF — flint Q4_1 CPU optimization (this machine)

Updated: 2026-10-31 (UTC+1) — periodic checkpoint (uploaded every ~5 min while working).

## Read this first

This is the live handoff for continuing the Q4_1 optimization of
`HelloSun/flint_cpu_opt`. The static docs are `README.md`,
`benchmarks/q41-summary.md` (Q4_1) and `benchmarks/summary.md` (Q4_K_XL).

## Machine (the one this work targets)

- Intel Xeon Platinum 8559C, 2 sockets, 192 logical CPUs online.
- **cgroup limit: 16 CPUs** (`cpu.max = 1600000/100000`), `nproc` = 16.
- 2 TiB RAM host; memory guard: agent-started processes must stay < 80 GiB RSS.
- Flags: AVX512-F/BW/VL/DQ, AVX512-VNNI, AVX512-FP16, AMX-INT8/AMX-BF16, F16C.
- Same CPU model as the numbers in `machine.txt` (16 threads), but the host is
  shared, so single runs vary ~±8 %.
- Repo base: llama.cpp `de5122d879d00d2895c36be7e0cee94105aeeabd` with
  `patches/flint-cpu-opt.patch` applied in `llama.cpp/`.

## Current model/server

- Model: `flint-int4-q4_1.gguf` (24.2 GiB, every weight Q4_1; token_embd +
  router F32), downloaded from `RecursiveSI/flint-int4-gguf`.
- `llama-server` running on `127.0.0.1:20000`, alias `flint-q4_1`,
  `-t 16 -fa on --jinja -c 32768` (started by `build.sh`).
- pi coding agent is configured to use it (`pi-local`).

## Current Q4_1 kernel state (already in the patch)

1. `ggml_vec_dot_q4_1_q8_1` (decode): AVX512-VNNI, 2 blocks per ymm, 4
   blocks/iteration, prefetch `GGML_Q41_PF` (default 192 blocks, T1 hint).
2. Tiled VNNI GEMM for Q4_1 prefill: per-32-block f32 `sd`/`sm`, `Q41`
   template flag, Q8_1 activations.
3. Reported on 8559C/16t: pp256 266, pp2048 297, tg128 55.7 t/s; stock
   183/189/48.3. KLD inside stock's own noise floor.

## Plan / next steps

- [x] Read existing docs and machine context.
- [ ] Build `llama-bench` + `llama-cli` (only `llama-server` was built).
- [ ] Baseline on **this** machine: interleaved A/B, `taskset -c 0-15`,
      `-t 16 -fa on -b 1024 -ub 1024`, Q4_1 file.
- [ ] Sweep `GGML_Q41_PF` (e.g. 128/192/256/320) and thread counts on the
      real model (not only microbench).
- [ ] Try one decode-side improvement (memory-bound path) and/or an AMX
      prefill variant; keep only if it beats noise.
- [ ] A/B compare, record raw CSV/log under `benchmarks/`.
- [ ] Update `README.md` + `benchmarks/q41-summary.md` only with measured,
      reproducible numbers.
- [ ] Restart `llama-server` with the final best settings.

## How to reproduce

```bash
cd /root/flint_cpu_opt
cmake --build llama.cpp/build -j16 --target llama-bench llama-cli
taskset -c 0-15 llama.cpp/build/bin/llama-bench -m flint-int4-q4_1.gguf \
  -t 16 -fa on -lm mlock -b 1024 -ub 1024 -p 256,2048 -n 128
```

## Notes / gotchas

- Do **not** run 100 parallel Tor/open code probes here; memory guard kills
  processes over 80 GiB / single child over 64 GiB.
- The llama-server is serving pi; stop it before a clean benchmark, restart
  afterwards (`bash build.sh` reuses an already-running server).
- Host is shared: use medians of >= 3 interleaved rounds; differences under
  ~8 % are not a win.
- Kernel changes must keep integer dot products exact; only f32 accumulation
  order may change. Check `benchmarks/graph_accuracy_test.cpp` and KLD if a
  numeric path changes.