Ercin/synthetic-llm-red-teaming-findings-banking
Viewer • Updated • 60 • 91
Ercin Dedeoglu PRO
Ercin
AI & ML interests
AI Architect. Sovereign LLM, RAG and agentic AI for regulated banking. LLM red teaming before go-live: open scorer + checklist. ercin.info
Recent Activity
commentedon a paper 6 minutes ago
TenetBench: A Clause-Anchored Russian Corpus for Evaluating Language Models against Professional Codes of Ethics posted an update about 1 hour ago
One prompt in six. That is how often two labelling pipelines disagreed when they labelled the same ~53k training prompts under two definitions of "unsafe" (arXiv 2610.25060, "Policy Is Not Capability").
Now picture that gap sitting inside the single guardrail number a vendor shows your bank. Across ten guardrail arms with the same architecture, HarmBench-copyright spans 0 to 100. The authors also moved a benchmark score across its entire range with 0.03% of the parameters, at unchanged detection capability. Their words: "The move leaves no trace in a data card."
That is a Label-Fit Score. It measures agreement with somebody's labels, NOT what gets through.
My newest article on huggingface.co/Ercin lists the three fields a bank go-live file should demand before it trusts that number: contamination, a seed floor, and the grading definition.
If you publish guardrail evals, do you record which label definition your benchmark was graded against, or only the score? commentedon a paper about 17 hours ago
Exploring The Empirical Privacy Landscape of Synthetic DataOrganizations
