Jie Jin

Good idea, thank you. We'll track the skip rate per skill per round, next to the fixed-answer test scores.

Falling worries us more. Rising mostly means slower training. Falling is only good news if the fixed-answer scores rise with it; if more items pass while those scores stay flat, the judge is getting overconfident on generated audio.

Good catch, and it's already how the loop is set up:

1. We only train where the judge is strong. A skill starts only once DuplexJev is above 80% on held-out human-labeled speech for it. The first skill is word stress, where DuplexJev is above 90% on held-out speakers. For some skills we use a specialist judge: a DuplexJev trained for that one task, which scores higher than the balanced release.
2. No forced winner. A candidate is kept only if its target probability clears an absolute floor and beats the group median by a margin. If no candidate does, the item contributes nothing that round.
3. People check it before every release. Before a version ships, blind human ratings compare it with the previous one, alongside DuplexJev-vs-human agreement on sophon audio and on real speech. If the judge's picks don't hold up with people on generated audio, it shows up there.