Idea: A Benchmark for Codex Usage-Limit Consumption
Melvin Vivas · X post · 2026-08-24 · Open on X
Topics: Evaluation (Evals) & Testing, LLMOps, Deployment & Monitoring, AI Dev Tools & Productivity · Level: intermediate
Summary
Proposes a benchmark that gives Codex the exact same tasks over time and measures how much of the usage limit each run uses. The goal is to catch regressions in cost with data instead of guesses.
Key points
- Run a fixed set of identical tasks through Codex again and again.
- Measure how much of the usage limit each run consumes.
- Comparing results over time shows when token or usage cost gets worse.
- This applies regression-testing ideas to cost: rely on data, not vibes.
Resources mentioned
- OpenAI Codex · tool · openai.com · paid · recommended by both Bashiri Smith & Melvin Vivas
OpenAI's coding agent. In the diagram it writes code, fixes review findings and drives the build loop. The creator also used it to make this video.
Also in: Developers moving from ChatGPT/Codex to Claude (Melvin Vivas on X · notes), Creator Opinion: Codex Is Still the Best at Computer Use (Melvin Vivas on X · notes), OpenAI Codex Ambassador applications are open (Melvin Vivas on X · notes), GPT-Live-1.1: Multilingual Voice Update for ChatGPT and Codex (Melvin Vivas on X · notes) and 264 more
Try this
- Build a usage and cost regression benchmark: run a fixed task suite through a coding agent on a schedule and track how much of the usage limit or how many tokens each task consumes.
More in Evaluation (Evals) & Testing
- Running LLM evals with promptfoo on local llama.cpp models
- The 8 Layers of Evaluating Production RAG and Agent Systems
- Pipette: Open-Source Benchmarking for On-Device Models
- Why Evaluation Is the Skill That Sets Top AI Engineers Apart
- AI-Written Code Still Needs Human Testing
- LoopsBench: Testing Coding Agents on Long-Horizon Tasks