Hugging Face's logo

Spaces:
Program-Lab
/
README
Running

App Files Community
README / README.md
YaoTang23's picture
YaoTang23
Add RL-oriented intro with hidden test counts
35c11bb verified
preview code
|
Raw History Blame Contribute Delete
1.95 kB
metadata 
title: README
emoji: 😻
colorFrom: purple
colorTo: purple
sdk: static
pinned: false

ProgramLab

Train single and multi agents to rebuild programs with RL

Blog Code Dataset

1,261 RL environments where an agent runs a compiled program it cannot read, then rebuilds the whole program in any language.

Built for RL

  • Graded reward: the reward is the pass rate over hidden behavioral tests. There are 187,070 in total, a median of 127 per task, and over 90% of tasks have at least 50, so partial progress earns partial reward.
  • Validated reward: tests that the original program fails, that an empty program passes, or that are flaky are removed.
  • Hard to hack: the agent never sees the source or the tests, and the reference program is execute-only and offline.
  • Ready to run: one prebuilt image per task, scored by the official ProgramBench evaluator.

RL on ProgramLab raises Qwen3.6-35B-A3B on ProgramBench from 13.7% to 24.0% (Claude Code, two-hour budget).

  • ProgramLab: tasks, metadata, and build files
  • ProgramLab-Tests: hidden tests
  • Prebuilt task images: ghcr.io/programlab-agent/programlab