Agent Harness Efficiency: Scaffolding Beats MCP vs. CLI
Melvin Vivas · X post · 2026-08-21 · Open on X
Topics: AI Agents, Tool Use & MCP, LLMOps, Deployment & Monitoring · Level: intermediate
Summary
Melvin comments on an academic benchmark that ran one software task across 7 agents and 5 models. Where a mature CLI ecosystem exists, agents without built-in MCP were just as reliable and 5-28x cheaper. The biggest savings came from the scaffolding around the model, and the lightweight Pi harness used the fewest tokens.
Key points
- Benchmark setup: one software task across 7 agents and 5 models.
- In domains with a mature CLI ecosystem, agents without MCP were just as reliable and 5-28x cheaper.
- The biggest efficiency gains came from the harness or scaffolding, not from MCP vs. CLI.
- Pi was the most efficient harness, at 14,660 median input tokens per completed run.
- Lightweight agents like Pi or Tau can cost as little as $0.0141 per run with hosted models.
- Local 27B models can make runs even cheaper.
Resources mentioned
- Pi · tool · x.com · free
A customizable coding agent that can be extended through its Extensions API and connected to several model providers.
Also in: 100-Iteration Autonomous Coding Loop with Local Qwen on an RTX 3090 (Melvin Vivas on X · notes), Pi Agent: A Highly Customizable Agent with Durable Sessions (Melvin Vivas on X · notes), Prompting an Autonomous Agent Loop: Be Specific, Show Visible Progress (Melvin Vivas on X · notes), Running an Overnight 100-Iteration Agent Loop with a Local Qwen Model (Melvin Vivas on X · notes) and 54 more - Tau · tool · github.com · free
Minimalist coding agent written in Python, a port of Pi's design, with native local-model support and extensions that can change its interface.
Also in: Herdr Crash Course: A Terminal Multiplexer for Running Many Coding Agents (Melvin Vivas on X · notes) - arXiv paper: MCP vs. CLI agents benchmark (7 agents, 5 models) · paper · arxiv.org · free
Academic study comparing agents with and without MCP on a software task, measuring reliability and cost. - Model Context Protocol (MCP) · docs · modelcontextprotocol.io · free · recommended by both Bashiri Smith & Melvin Vivas
Open standard for connecting models to tools and data sources through one interface. It turns an N×M integration problem into N+M.
Also in: Extend Grok Bot with Your Own MCP Servers and Connectors (Melvin Vivas on X · notes), AI Engineer Vocab Quiz: 11 Acronyms to Know for Interviews (Bashiri Smith on Facebook · notes), The 3 AI Engineering Skills: Generative AI, RAG, and Agentic AI (Bashiri Smith on Facebook · notes), xAI's Experimental TypeScript SDK: Grok Text, Voice, Image and Video in One SDK (Melvin Vivas on X · notes) and 11 more
Try this
- Use lightweight agent harnesses like Pi to cut token costs.
- Prefer CLI tools over MCP in domains with a mature CLI ecosystem.
- Benchmark several agent harnesses on the same task, comparing median input tokens and cost per completed run.
More in AI Agents, Tool Use & MCP
- Let agents run on their own and build an approval layer
- Hermes Agent: each bot is its own profile
- An X research bot built with Hermes
- Run Your Hermes Agent on Free LFM2.5-2.6B via OpenRouter
- 5 Agentic Design Patterns: ReAct, Planning, Reflection, Routing, Multi-Agent
- Running a Local Ornith Model as the Backend for a Hermes Agent