RAG Pre-Deployment Checklist: Evals, Grounding, Latency, Cost & Monitoring
Bashiri Smith · Facebook reel · 2026-09-27 · 0:54 · 4,364 views · Open on Facebook
Topics: Retrieval-Augmented Generation (RAG), Evaluation (Evals) & Testing, LLMOps, Deployment & Monitoring · Level: intermediate
Summary
A short dialogue covering what to check before shipping a RAG app to production. It goes through an eval set of golden questions, retrieval quality with reranking, answers grounded in sources with citations, using the same embedding model for queries and documents, latency and cost targets, and observability in production. It ends with an invitation to join the creator's AI engineering community.
Key points
- Build an eval set first: golden questions with expected answers, and run every change against it before it ships (regression testing).
- Check retrieval quality: make sure recall is good, then rerank the top hits so the model only sees the best chunks.
- Prevent hallucination: answer only from retrieved context, cite sources, and say 'I don't know' when nothing relevant is retrieved.
- Use the same embedding model and version for queries and documents. If they don't match, retrieval breaks.
- Latency target: P95 under 2 seconds.
- Control cost per query with a semantic cache so repeated or similar questions aren't paid for twice.
- Production observability: trace every query, logging retrieval scores, tokens and latency, and set alerts or reports for faithfulness drops and cost spikes.
Resources mentioned
- BASWE.Ai Engineer (Skool community) · community · skool.com · paid
The creator's paid community and program, with an AI learning roadmap (including the full ops and evaluation track), daily calls with engineers and recruiters, resume and portfolio help, and a job-search pipeline.
Also in: Basic vs Production AI Engineering: RAG, Agents, and Context over Prompts (Bashiri Smith on Facebook · notes), ChatGPT Prompt to Generate Standout Portfolio Project Ideas (Bashiri Smith on Facebook · notes), Vector Database vs. Graph Database: When to Use Each (and Both for RAG) (Bashiri Smith on Facebook · notes), The 5 Levels of RAG: From Naive Retrieval to Agentic RAG (Bashiri Smith on Facebook · notes) and 93 more
Try this
- Build a golden-question eval set with expected answers and run it on every change before deploying.
- Add reranking on top of retrieval and measure recall.
- Make the model answer only from retrieved context, cite sources, and refuse when the context isn't relevant.
- Check that query and document embeddings use the same model and version.
- Measure P95 latency (target under 2s) and cost per query, and add a semantic cache for repeated questions.
- Trace every production query (retrieval scores, tokens, latency) and alert on faithfulness drops and cost spikes.
- Comment "Deploy" to get the link to the creator's community.
More in Retrieval-Augmented Generation (RAG)
- Basic RAG Pipeline in 60 Seconds: From Documents to Grounded Answers
- How to Evaluate a RAG Pipeline: Retrieval vs. Generation (Interview Answer)
- LangChain vs. LangGraph Explained with One RAG Chatbot
- Wannabe vs $200K+ AI Engineer: RAG, Agents, Context and Fine-Tuning Mistakes
- How RAG Works: Chunking, Embedding, Vector Storage, and Retrieval
- Using Jev for RAG as a Similarity Metric and Reranker