Local Evals: Open-Source App for Evaluating Local or OpenAI-Compatible Models
Melvin Vivas · X post · 2026-09-08 · Open on X
Topics: Evaluation (Evals) & Testing, LLM Fundamentals, AI Dev Tools & Productivity · Level: intermediate
Summary
Melvin Vivas released local-evals, an open-source evaluation app that runs locally. It works with local models or any OpenAI-compatible API. It evaluates document/image-to-JSON, text-to-JSON and tool calling. He built it entirely with Codex, using GPT-6 Astra as orchestrator and Luna as subagents, and tested it with LM Studio and OpenRouter.
Key points
- local-evals runs on your own machine and talks to any OpenAI-compatible endpoint, so you can compare local and hosted models with the same harness.
- Supported eval types: document/image → JSON extraction, text → JSON (structured output), and tool calling.
- It was tested with LM Studio for local models and OpenRouter for hosted models.
- He built the whole app with Codex using an Astra/Luna combo (Astra orchestrates, Luna subagents do the work).
- Structured-output and tool-calling accuracy are useful evals when you pick a small or local model for agent tasks.
Resources mentioned
- donvito/local-evals · repo · github.com · free
Eval app that runs locally and tests local models or any OpenAI-compatible API on JSON extraction and tool calling.
Also in: Local Evals v0.1.0: Run LLM Evals Locally (Melvin Vivas on X · notes), local-evals: A Local Eval App with a Setup Wizard (Melvin Vivas on X · notes), Melvin Vivas's open-source AI tools, built with Codex (Melvin Vivas on X · notes), local-evals: A Local LLM Eval App Built by Codex Subagents (Melvin Vivas on X · notes) and 1 more - LM Studio · tool · x.com · free
Desktop app for downloading and running LLMs locally, with a developer mode that serves models through an API.
Also in: Run Local LLMs on the Computer You Already Have: A Beginner's Path (Melvin Vivas on X · notes), Running LLMs Locally Without an Expensive Rig (Melvin Vivas on X · notes), Adding Vercel AI Gateway as a provider in AIBackends with Devin (Melvin Vivas on X · notes), LoRA Fine-Tune Qwen3.5-2B on Your Tweets with Unsloth Studio (Melvin Vivas on X · notes) and 32 more - OpenRouter · tool · openrouter.ai · free
A single OpenAI-compatible API that routes requests to hundreds of models from many providers, with one bill and model fallbacks.
Also in: Stealth Model 'Space Bunny' Revealed as MiniMax M3.1 Flash (Melvin Vivas on X · notes), NVIDIA Nemotron 3.5 Super VL now on OpenRouter (Melvin Vivas on X · notes), OpenRouter serves image models, including FLUX 3 Image Klein (Melvin Vivas on X · notes), Comparing Opus 5.5, GPT-6 Astra and Grok 4.7 on OpenRouter (Melvin Vivas on X · notes) and 53 more - OpenAI Codex · tool · openai.com · paid · recommended by both Bashiri Smith & Melvin Vivas
OpenAI's coding agent. In the diagram it writes code, fixes review findings and drives the build loop. The creator also used it to make this video.
Also in: Developers moving from ChatGPT/Codex to Claude (Melvin Vivas on X · notes), Creator Opinion: Codex Is Still the Best at Computer Use (Melvin Vivas on X · notes), OpenAI Codex Ambassador applications are open (Melvin Vivas on X · notes), GPT-Live-1.1: Multilingual Voice Update for ChatGPT and Codex (Melvin Vivas on X · notes) and 264 more
Try this
- Clone local-evals and run it against a local model in LM Studio or a hosted model via OpenRouter.
- Build a local eval harness that compares models on JSON extraction and tool-calling accuracy.
More in Evaluation (Evals) & Testing
- Evaluating Claude Code Plugins with `claude plugin eval`
- local-evals: A Local LLM Eval App Built by Codex Subagents
- GPT-6 Astra Tops Vending-Bench, Beating Claude Fable 5.1
- Building an OCR Evals App with Codex
- Be Skeptical of Model Leaderboards: Muse Spark vs Astra
- Personal Smoke Test for New Models: "Make a CRM App"