Article
Coding agents that submit without running a test or script
serkanpeker
• Serkan Peker PRO
serkanpeker
AI & ML interests
Sovereign & Air-Gapped AI Architectures, Large-Scale Distributed Pre-training & Distillation (Slurm / InfiniBand HPC), Turkish LLM & Custom Tokenizer Engineering, High-Throughput Inference & AI Gateways (vLLM), End-to-End MLOps & Active Learning Computer Vision Pipelines, and Autonomous Multi-Agent Reasoning Systems with Enterprise RAG.
Recent Activity
posted an update 3 days ago
In 51,087 SWE-agent runs, the agent ran no test or script after its last edit in 39.4% of the submissions that changed product code. Within the same issue, those submissions had 0.68 times the odds of resolving it (95% CI 0.63 to 0.74) compared with submissions whose last run showed no failure.
A rerun with Claude Code driven by gpt-oss-120b on SWE-bench Verified gave 0.55 (0.37 to 0.82) on the 147 issues where the agent had skipped verification at least once; an earlier run with the same agent and model, before four changes to the environment, was inconclusive (1.04, 0.42 to 2.55). The rerun's estimate leaves out 136 trials that ended in an exception, 125 of them agent timeouts, and the direction of any bias from that is unknown.
In phase 1, unverified submissions whose closing message claimed success resolved slightly more often than the other unverified ones (20.8% against 17.6%), and among submissions whose last run showed no failure, running the test suite showed no clear advantage over running a script (0.99, 0.78 to 1.27). All comparisons are observational. Every audit was done by AI agents, the 80 phase 2b rows were re-checked by a jury of four models, and no human has read the audited rows.
Article (English): https://huggingface.co/blog/serkanpeker/done-claims
Makale (Türkçe): https://huggingface.co/blog/serkanpeker/kanitsiz-bitti
Data: https://huggingface.co/datasets/serkanpeker/done-claims
Code: https://github.com/speker/done-claims published an article 3 days ago
Coding agents that submit without running a test or script published a dataset 3 days ago
serkanpeker/turkish-decision-benchmarkOrganizations
published an article 3 days ago
published an article 4 days ago
Article
Test ya da betik çalıştırmadan "bitti" diyen kodlama ajanları ne kadar haklı?
serkanpeker
• • 1 published an article 5 days ago
Article
Türkçe vergisi: açık LLM'lerde Türkçenin servis maliyeti
serkanpeker
• • 3 published an article 6 days ago
Article
The Turkish tax: what serving Turkish costs on open LLMs
serkanpeker
•