Why Evals Matter: Making AI Apps Survive More Than a Demo
Bashiri Smith · Facebook reel · 2026-10-13 · 0:49 · 5,192 views · Open on Facebook
Topics: Evaluation (Evals) & Testing, Retrieval-Augmented Generation (RAG), Resume, Job Search & Interviews · Level: beginner
Summary
Bashiri Smith argues that because LLM outputs are non-deterministic, getting one good answer doesn't prove your AI system is reliable. Evaluation is the key skill for engineers moving into AI roles. He quotes an unnamed AI expert who says hiring may come down to building a big project that holds up when someone tries to break it with tools like Codex. He lists five areas to evaluate (retrieval quality, groundedness, task success, guardrails, cost/latency) and shares his free AI evaluation guide.
Key points
- LLMs are non-deterministic: the same input can give different outputs, so one good answer doesn't prove reliability.
- Test your system with messy questions, irrelevant context and adversarial instructions meant to break it, not just the happy path.
- Retrieval quality: did the RAG system actually find the information needed to answer?
- Groundedness: is the answer supported by the retrieved information?
- Task success: did the agent actually complete the job?
- Guardrails: how does the app handle attempts to manipulate it (e.g. prompt injection)?
- Cost and latency: measure cost per successful request and how long each request takes.
- Evals help you find failures, measure improvements and catch regressions before users do. The creator says many engineers moving into AI roles have weak eval skills, and many job postings require them.
Resources mentioned
- Evaluation field guide (AI evaluation guide) · pdf · drive.google.com · free
The creator's guide to evaluating AI apps: retrieval quality, groundedness, task success, guardrails, and cost/latency. - OpenAI Codex · tool · openai.com · paid · recommended by both Bashiri Smith & Melvin Vivas
OpenAI's coding agent. In the diagram it writes code, fixes review findings and drives the build loop. The creator also used it to make this video.
Also in: Developers moving from ChatGPT/Codex to Claude (Melvin Vivas on X · notes), Creator Opinion: Codex Is Still the Best at Computer Use (Melvin Vivas on X · notes), OpenAI Codex Ambassador applications are open (Melvin Vivas on X · notes), GPT-Live-1.1: Multilingual Voice Update for ChatGPT and Codex (Melvin Vivas on X · notes) and 264 more
From the PDF shared here: Evaluation Field Guide (baswe.ai engineer accelerator – Ops and Evaluation module)
Open the original · 41 pages
Bashiri Smith's 41-page practitioner's guide to evaluating AI systems. It covers eval fundamentals (eval-driven development, golden datasets, metric types, basic statistics), deterministic and overlap metrics for LLM outputs, LLM-as-a-judge (rubric design, judge biases, checking judges against human labels), RAG evaluation (retrieval metrics and the RAG triad), agent trajectory evaluation, model benchmarking and fine-tune evaluation, and production evals (tracing, online signals, A/B tests, drift, guardrails, CI eval gates). It ends with a map of the eval tool stack, four portfolio projects and interview signals. Read it once end to end, then use it as a reference, and build one of the portfolio projects.
- Langfuse · tool · langfuse.com · free
Purpose-built tool for viewing traces, running evals and managing prompts.
Also in: Step-by-Step Roadmap to a $200K+ AI Engineering Role (Bashiri Smith on Facebook · notes), 7 Habits to Become an AI Engineer: Books, Tooling, Research & Shipping (Bashiri Smith on Facebook · notes), Taking a RAG App to Production: Evals, Guardrails, Cost and Tracing (Bashiri Smith on Facebook · notes) - LangSmith · tool · docs.smith.langchain.com · free
Purpose-built tool for viewing traces, running evals and managing prompts.
Also in: Step-by-Step Roadmap to a $200K+ AI Engineering Role (Bashiri Smith on Facebook · notes), 7 Habits to Become an AI Engineer: Books, Tooling, Research & Shipping (Bashiri Smith on Facebook · notes) - RAGAS · tool · github.com · free
Listed under RAG evaluation tools in the guide's evaluation stack.
Also in: Find Hidden AI Engineering Jobs by Searching for Skills, Not Job Titles (Bashiri Smith on Facebook · notes), SWE-to-AI Engineer Plan for 2027: LLMs, RAG, Agents, Evals, Job Search (Bashiri Smith on Facebook · notes), Taking a RAG App to Production: Evals, Guardrails, Cost and Tracing (Bashiri Smith on Facebook · notes) - promptfoo · repo · github.com · free · recommended by both Bashiri Smith & Melvin Vivas
Listed under eval frameworks for defining and running evaluations, offline and in CI.
Also in: OpenAI Retires Its Evals Platform and Recommends promptfoo (Melvin Vivas on X · notes), Running LLM evals with promptfoo on local llama.cpp models (Melvin Vivas on X · notes) - Arize Phoenix · tool · github.com · free
Purpose-built tool for viewing traces, running evals and managing prompts. - Braintrust · tool · braintrust.dev · free
Purpose-built tool for viewing traces, running evals and managing prompts. - OpenAI Evals · repo · github.com · free · recommended by both Bashiri Smith & Melvin Vivas
Listed under eval frameworks for defining and running evaluations.
Also in: Is OpenAI Evals Still Maintained? (Melvin Vivas on X · notes) - Weights & Biases Weave · tool · github.com · free
Purpose-built tool for viewing traces, running evals and managing prompts. - DeepEval · tool · github.com · free
Listed under eval frameworks for defining and running evaluations, offline and in CI. - Guardrails AI · tool · github.com · free
Listed under guardrails tools for enforcing quality and safety on every request at runtime. - NeMo Guardrails · tool · github.com · free
Listed under guardrails tools for enforcing quality and safety on every request at runtime. - TruLens · tool · trulens.org · free
Listed under RAG evaluation tools in the guide's evaluation stack.
Try this
- Comment 'evals' on the reel to get the link to the AI evaluation guide (it's also in the caption).
- Test your AI app with messy questions, irrelevant context and adversarial instructions, not just one good demo answer.
- Measure retrieval quality, groundedness, task success, guardrail behavior, and cost/latency for your app.
- Use evals to catch regressions before users do.
- Build and deploy a large AI project that holds up when others actively try to break it (e.g. with AI coding agents), and back it with an eval suite covering retrieval, groundedness, task success, guardrails, and cost/latency.