locke1
locke1
AI & ML interests
None yet
Recent Activity
repliedto dipankarsarkar's post 4 days ago
I audited one of my own evaluations. The ranking did not hold up the way I expected.
Eight open models, one task: infer the structure of a prompt. Then ask again with the identical call. Caching off.
- Agreement between repeated identical calls (mean Jaccard) ranged from 0.39 to 0.96 across models.
- Only 35 of 127 prompt-model cells were perfectly reproducible on every run.
- I bootstrapped the reproducibility ranking over prompts. The two least reproducible models kept their rank in 99% and 86% of resamples. The middle four kept theirs in 27% to 48%.
So the table reliably finds the worst model. It does not reliably find the best.
Reproducible is also not the same as correct. F1 against gold annotations ran from 0.56 to 0.99.
By the audit date, 4 of the 8 model variants had been retired (HTTP 410). The study as specified can no longer be re-run. The saved outputs are what survives, so I published all of them: every inferred structure, every run, the prompts and the annotations.
Paper: https://huggingface.co/papers/2609.30074
Dataset: https://huggingface.co/datasets/dipankarsarkar/llm-evaluation-self-audit
Code: https://github.com/sarkar-dipankar/llm-evaluation-self-audit
How many of the leaderboard rankings you rely on would survive re-running the same calls? liked a dataset 5 days ago
figai/RIDGE published a dataset 5 days ago
figai/RIDGEOrganizations
