AI Engineer Mistakes: RAG Evaluation, Hybrid Search and Cutting Cost
Bashiri Smith · Facebook reel · 2026-10-06 · 0:47 · 4,694 views · Open on Facebook
Topics: Retrieval-Augmented Generation (RAG), Evaluation (Evals) & Testing, LLMOps, Deployment & Monitoring · Level: intermediate
Summary
The reel compares beginner habits with senior habits across three common RAG and production mistakes. Don't judge a RAG app by whether its answers sound right. Evaluate the retrieval and grounding instead. Use hybrid keyword and vector search with a reranker rather than vector search alone. Measure where latency and cost come from before you switch models. It ends with a pitch for the creator's paid Skool community.
Key points
- A convincing answer can still be wrong, so don't ship a RAG app because the answers 'sound right'.
- Evaluate RAG in layers. Check whether retrieval found the right evidence and how much irrelevant context came back. Then check whether the answer is supported, complete and correctly cited from the source data.
- Vector (semantic) search can miss exact matches like error codes and product IDs, which keyword search handles well.
- Test keyword search and vector search, combine them (hybrid search) when it helps, and add a reranker to move the strongest matches to the top.
- For a slow or expensive app, first measure where the latency and cost come from.
- Then cut unnecessary LLM calls and cache results that are safe to reuse.
- Only after that, send simpler requests to a smaller, cheaper model instead of switching everything to one.
- Saving tutorials and roadmaps without building and practicing won't get you a job.
Resources mentioned
- BASWE.Ai Engineer (Skool community) · community · skool.com · paid
The creator's paid community and program, with an AI learning roadmap (including the full ops and evaluation track), daily calls with engineers and recruiters, resume and portfolio help, and a job-search pipeline.
Also in: Basic vs Production AI Engineering: RAG, Agents, and Context over Prompts (Bashiri Smith on Facebook · notes), ChatGPT Prompt to Generate Standout Portfolio Project Ideas (Bashiri Smith on Facebook · notes), Vector Database vs. Graph Database: When to Use Each (and Both for RAG) (Bashiri Smith on Facebook · notes), The 5 Levels of RAG: From Naive Retrieval to Agentic RAG (Bashiri Smith on Facebook · notes) and 93 more
Try this
- Evaluate your RAG app's retrieval quality: is the right evidence found and how much irrelevant context comes back?
- Check whether generated answers are supported, complete and correctly cited from the source data.
- Benchmark keyword search against vector search on your data, especially for exact IDs and error codes.
- Try hybrid search and add a reranker on top.
- Profile latency and cost before optimizing: remove unnecessary calls, cache reusable results, and send only simple requests to a smaller model.
- Build a RAG evaluation harness that scores retrieval relevance, noise in the results, and whether answers are faithful, complete and correctly cited.
- Build a hybrid search pipeline (keyword + vector + reranker) and compare it with vector-only search on queries that contain exact product IDs and error codes.
- Add cost and latency measurement, caching and routing to a smaller model for simple requests in an existing LLM app.
More in Retrieval-Augmented Generation (RAG)
- Entity Linking: Connecting Text Mentions to Structured Records
- 5 RAG Metrics: Top-k, Recall@k, Precision@k, Top-p and Cost per Request
- AI Engineer Interview: RAG Evals, Hybrid Search & Cost/Latency Optimization
- Production RAG Interview Answers: Design, Debugging, and Scaling
- Using Jev as a Reranker to Augment RAG Retrieval
- Basic RAG Pipeline in 60 Seconds: From Documents to Grounded Answers