Agent Harness Efficiency: Scaffolding Beats MCP vs. CLI

Melvin Vivas · X post · 2026-08-21 · Open on X

Topics: AI Agents, Tool Use & MCP, LLMOps, Deployment & Monitoring · Level: intermediate

Summary

Melvin comments on an academic benchmark that ran one software task across 7 agents and 5 models. Where a mature CLI ecosystem exists, agents without built-in MCP were just as reliable and 5-28x cheaper. The biggest savings came from the scaffolding around the model, and the lightweight Pi harness used the fewest tokens.

Key points

• Benchmark setup: one software task across 7 agents and 5 models.
• In domains with a mature CLI ecosystem, agents without MCP were just as reliable and 5-28x cheaper.
• The biggest efficiency gains came from the harness or scaffolding, not from MCP vs. CLI.
• Pi was the most efficient harness, at 14,660 median input tokens per completed run.
• Lightweight agents like Pi or Tau can cost as little as $0.0141 per run with hosted models.
• Local 27B models can make runs even cheaper.

Resources mentioned

• Pi · tool · x.com · free
  A customizable coding agent that can be extended through its Extensions API and connected to several model providers.
  Also in: Sign in with ChatGPT: Setting Usage Limits for Each App (Melvin Vivas on X · notes), Pi reaches v1.0 (Melvin Vivas on X · notes), Claude Code mods: customize behavior and UI with plugins (Melvin Vivas on X · notes), Customizing Your Coding Setup with Pi Coding Agent Extensions (Melvin Vivas on X · notes) and 39 more
• Tau · tool · github.com · free
  Minimalist coding agent written in Python, a port of Pi's design, with native local-model support and extensions that can change its interface.
  Also in: Herdr Crash Course: A Terminal Multiplexer for Running Many Coding Agents (Melvin Vivas on X · notes)
• arXiv paper: MCP vs. CLI agents benchmark (7 agents, 5 models) · paper · arxiv.org · free
  Academic study comparing agents with and without MCP on a software task, measuring reliability and cost.
• Model Context Protocol (MCP) · docs · modelcontextprotocol.io · free · recommended by both Bashiri Smith & Melvin Vivas
  An open standard for connecting LLM apps and agents to tools and data sources. The official docs explain how it works and how to build servers and clients.
  Also in: Grok Bot Templates: Sharing, Publishing and Safely Installing Bots (Melvin Vivas on X · notes), 8-Week Roadmap to a $200K+ AI Engineering Role (Bashiri Smith on Facebook · notes), Paste Documentation URLs into Codex Instead of Using MCP (Melvin Vivas on X · notes), OpenAI Agents API: Hosted Codex Harness for Long-Running Cloud Agents (Melvin Vivas on X · notes) and 7 more

Try this

• Use lightweight agent harnesses like Pi to cut token costs.
• Prefer CLI tools over MCP in domains with a mature CLI ecosystem.
• Benchmark several agent harnesses on the same task, comparing median input tokens and cost per completed run.