Code generation & program synthesis — Benchmark Sources & Consensus
Generating or reconstructing working programs from specs, binaries, or natural-language descriptions. Frontier-difficulty benchmarks where state-of-the-art is still well under 10%.
Platforms tracked: Claude Cowork · Kilocode · Openclaw · Chatgpt
Consensus across 2 sources
Across 2 sources, the two available data points measure different things and neither separates platforms cleanly. ProgramBench (May 2026) sets a difficulty ceiling: 0% of evaluated models fully solve program reconstruction from binaries, with the best reaching 3% "almost resolved." A framework-level benchmark (August 2026, 21 Rails tasks, 8 models) reports 92% top accuracy but finds models reach for existing framework APIs only 8-35% of the time, and score higher when they do — suggesting idiomatic-code use, not raw task success, is where current models differ.
All Sources
We aggregate published benchmarks; we never run our own tests and never pick winners. Each row links back to the original publication.
| Source | Date | Finding | Methodology | Quality |
|---|---|---|---|---|
| ProgramBench | 2026-05-09 | Claude Opus 4.7 leads at 3% "almost resolved" on 200+ program-reconstruction tasks from compiled binaries; 0% fully solved by any model. Frontier-difficulty. | Reconstruct working source from compiled binary; automated pass/fail on hidden test suite. 200+ tasks across 12 languages. · 200+ tasks | high winner: cowork |
| Ruby on Rails | 2026-08-13 | 21 Rails tasks on a realistic app across 8 models: top accuracy 92%, cheapest full run 91 cents; models rarely reached for existing framework APIs (recall 8-35%), and success was higher when they did (92% vs 87%) | 21 tasks on Writebook, each targeting one Rails API without naming it; Claude Opus 5, Claude Fable 5, GPT-5.6 Sol, GPT-5.6 Luna, Muse Spark 1.2, Kimi K3, GLM 5.2, DeepSeek V4 Flash; accuracy plus API-recall and cost recorded per model | high |
How we work
OpenClawDatabase aggregates and links to published benchmarks. We don't run our own tests, and we don't pick winners. Our weekly benchmark-aggregator routine scans 7+ live leaderboards (OpenRouter, Aider, SWE-bench, GAIA, LMSYS, BigCodeBench, MMLU-Pro) plus relevant Reddit and Hacker News threads, then writes structured entries into /assets/benchmarks.json. Every row here links back to the original publication.
← Back to all benchmark tasks · See also: Decision guide · Cost calculator