Evals Are Real Work: Use Coding Agents to Find and Build Datasets
Melvin Vivas · X post · 2026-08-27 · Open on X
Topics: Evaluation (Evals) & Testing, AI Dev Tools & Productivity · Level: beginner
Summary
Building evals takes real effort, mostly in finding existing datasets or generating synthetic ones. Coding agents like Claude Code or Codex can help with that work.
Key points
- Searching for eval datasets takes a lot of time.
- Generating synthetic eval data is another option, but it also takes time.
- Use Claude Code or Codex to help find or generate datasets.
Resources mentioned
- Claude Code · tool · code.claude.com · paid · recommended by both Bashiri Smith & Melvin Vivas
Build agents and pipelines from the terminal; the guide's main agentic coding tool.
Also in: First impressions: Claude Code in the cloud is good and fast (Melvin Vivas on X · notes), Why Claude Code feels faster: gzipped turns and instant cloud sessions (Melvin Vivas on X · notes), Watch What Fills Your Coding Agent's Context and When to Clear (Melvin Vivas on X · notes), Claude Code's Right Sidebar and Context Breakdown (Melvin Vivas on X · notes) and 124 more - OpenAI Codex · tool · openai.com · paid · recommended by both Bashiri Smith & Melvin Vivas
OpenAI's coding agent. In the diagram it writes code, fixes review findings and drives the build loop. The creator also used it to make this video.
Also in: Developers moving from ChatGPT/Codex to Claude (Melvin Vivas on X · notes), Creator Opinion: Codex Is Still the Best at Computer Use (Melvin Vivas on X · notes), OpenAI Codex Ambassador applications are open (Melvin Vivas on X · notes), GPT-Live-1.1: Multilingual Voice Update for ChatGPT and Codex (Melvin Vivas on X · notes) and 264 more
Try this
- Use a coding agent to help find or generate synthetic eval datasets.
More in Evaluation (Evals) & Testing
- Cohere Parse: Pricing vs Parse Bench Score
- Evaluation for AI Engineering: Free Full Guide (Resource Share)
- Local Data Studio: Inspect Private Datasets Without Hugging Face Pro
- Is OpenAI Evals Still Maintained?
- Running LLM evals with promptfoo on local llama.cpp models
- The 8 Layers of Evaluating Production RAG and Agent Systems