OpenAI Retires Its Evals Platform and Recommends promptfoo
Melvin Vivas · X post · 2026-10-06 · Open on X
Topics: Evaluation (Evals) & Testing, AI Safety, Security & Guardrails, LLMOps, Deployment & Monitoring · Level: intermediate
Summary
The post says OpenAI is retiring its evals platform and recommends promptfoo as the replacement. promptfoo is an open-source tool for testing prompts, agents and RAG pipelines. It also covers red teaming and vulnerability scanning, and it can compare models such as GPT, Claude, Gemini and DeepSeek.
Key points
- OpenAI is retiring its evals platform.
- OpenAI suggests moving to promptfoo instead.
- promptfoo is open source and tests prompts, agents and RAG pipelines.
- It also does red teaming, pentesting and vulnerability scanning for AI apps.
- It compares results across GPT, Claude, Gemini, DeepSeek and other models.
- Tests are written as simple declarative configs, so they fit into regression testing.
Resources mentioned
- promptfoo · repo · github.com · free · recommended by both Bashiri Smith & Melvin Vivas
Listed under eval frameworks for defining and running evaluations, offline and in CI.
Also in: Running LLM evals with promptfoo on local llama.cpp models (Melvin Vivas on X · notes) - OpenAI Evals platform · tool · developers.openai.com · check price
OpenAI's hosted platform for evaluating models, which is being retired.
Try this
- If you use OpenAI's evals platform, move your evals to promptfoo.
- Set up a promptfoo test suite that compares GPT, Claude and Gemini on your own prompts or RAG pipeline.
More in Evaluation (Evals) & Testing
- Decision Model Rankings by Spend and Token Share
- Local Evals v0.1.0: Run LLM Evals Locally
- local-evals: A Local Eval App with a Setup Wizard
- Benchmarks matter less than how a model does on your own task
- 5 Things AI Engineers Must Evaluate: RAG, Agents, Models, Data, Guardrails
- Test models on your own workflow, not public benchmarks