MLflow 3.17.0: Jev models as LLM judges in GenAI evaluate
Melvin Vivas · X post · 2026-10-12 · Open on X
Topics: Evaluation (Evals) & Testing, LLMOps, Deployment & Monitoring · Level: intermediate
Summary
MLflow 3.17.0 adds 'Jev Decisions'. You can now use TypeSafe's Jev models as judges in mlflow.genai.evaluate() for yes/no and fixed-category checks. These judge calls work with MLflow Tracing and AI Gateway, so you can inspect and govern them in one place.
Key points
- MLflow 3.17.0 adds 'Jev Decisions' for evaluating GenAI apps.
- TypeSafe's Jev models can be used as judges inside mlflow.genai.evaluate().
- Built for yes/no checks and checks with a fixed set of categories.
- Judge calls work with MLflow Tracing, so you can inspect them.
- AI Gateway support lets you govern judge calls in one place.
Resources mentioned
- MLflow · tool · mlflow.org · free · recommended by both Bashiri Smith & Melvin Vivas
Experiment tracking and a model registry.
Also in: Five Core AI Engineering Topic Areas: A Study Roadmap Checklist (Bashiri Smith on Facebook · notes) - mlflow.genai.evaluate() · docs · mlflow.org · free
MLflow's GenAI evaluation API for scoring app outputs with judges and scorers. - Jev · tool · typesafe.ai · paid
A decision model that makes turn-level forecasts on AI agent conversations using only structural signals (turns, tool calls, workflow stages, timing).
Also in: JevDev v0.1.3: Free UI Environment for Experimenting with Jev Models (Melvin Vivas on X · notes), Automating Insurance Receipt Claims with OCR and Jev (Melvin Vivas on X · notes), Generating a Repo Promo Video with a Claude Skill on Sonnet 5.5 vs Opus 5.5 (Melvin Vivas on X · notes), GLiDE by Fastino Labs: A Post-Trainable Reasoning Decision Model (Melvin Vivas on X · notes) and 15 more
Try this
- Upgrade to MLflow 3.17.0 and try Jev judges in mlflow.genai.evaluate() for yes/no checks