LoopsBench: Testing Coding Agents on Long-Horizon Tasks
Melvin Vivas · X post · 2026-08-20 · Open on X
Topics: Evaluation (Evals) & Testing, AI Agents, Tool Use & MCP, AI Dev Tools & Productivity · Level: advanced
Summary
LoopsBench evaluates coding agents (a model plus its harness) on 112 long-horizon coding tasks built from real pull requests. Claude Opus 4.7 with Claude Code had the highest resolve rate at 25.00%. GPT-5.5 with Codex reached 21.43%. The low scores show long-horizon coding is still hard for agents.
Key points
- 112 long-horizon coding tasks built from real pull requests.
- The benchmark scores a model and its harness together (for example Opus 4.7 + Claude Code).
- Top resolve rate: Opus 4.7 with Claude Code, 25.00%.
- GPT-5.5 with Codex: 21.43%.
- The paper (10 Aug) frames this as moving from 'harness engineering' to 'loop engineering'.
- The creator hopes the benchmark will be updated with newer models and harnesses.
Resources mentioned
- LoopsBench Leaderboard · website · loopsbench.ai · free
Leaderboard of coding agent resolve rates on LoopsBench. - LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation · paper · alphaxiv.org · free
Paper introducing LoopsBench, a benchmark of 112 long-horizon coding tasks from real pull requests. - Claude Code · tool · code.claude.com · paid · recommended by both Bashiri Smith & Melvin Vivas
Build agents and pipelines from the terminal; the guide's main agentic coding tool.
Also in: First impressions: Claude Code in the cloud is good and fast (Melvin Vivas on X · notes), Why Claude Code feels faster: gzipped turns and instant cloud sessions (Melvin Vivas on X · notes), Watch What Fills Your Coding Agent's Context and When to Clear (Melvin Vivas on X · notes), Claude Code's Right Sidebar and Context Breakdown (Melvin Vivas on X · notes) and 124 more - OpenAI Codex · tool · openai.com · paid · recommended by both Bashiri Smith & Melvin Vivas
OpenAI's coding agent. In the diagram it writes code, fixes review findings and drives the build loop. The creator also used it to make this video.
Also in: Developers moving from ChatGPT/Codex to Claude (Melvin Vivas on X · notes), Creator Opinion: Codex Is Still the Best at Computer Use (Melvin Vivas on X · notes), OpenAI Codex Ambassador applications are open (Melvin Vivas on X · notes), GPT-Live-1.1: Multilingual Voice Update for ChatGPT and Codex (Melvin Vivas on X · notes) and 264 more
Try this
- Read the LoopsBench paper and check the full leaderboard.
- Run several model + harness pairs on a small set of real PR-based tasks and compare their resolve rates.
More in Evaluation (Evals) & Testing
- Idea: A Benchmark for Codex Usage-Limit Consumption
- Why Evaluation Is the Skill That Sets Top AI Engineers Apart
- AI-Written Code Still Needs Human Testing
- Agent Loops Need Real Evals and Feedback
- OCR Testing a Small Vision Model with LLM-Made Ground Truth
- Code Arena Fullstack Benchmark: Kimi K3 Ranks #1