Be Skeptical of Model Leaderboards: Muse Spark vs Astra
Melvin Vivas · X post · 2026-09-04 · Open on X
Topics: Evaluation (Evals) & Testing, Industry Trends & Job Market · Level: beginner
Summary
The creator tried Muse Spark 1.3 in OpenCode and found it 'no way close to a GPT model'. He doubts an Artificial Analysis chart that ranks Muse Spark 1.3 Max above Astra, though he notes he tested the non-Max version. The lesson: check leaderboard rankings by testing models on your own tasks.
Key points
- Hands-on, Muse Spark 1.3 felt much weaker than GPT models on coding.
- The Artificial Analysis intelligence index reportedly ranked Muse Spark 1.3 Max above GPT-6 Astra, which the creator and the quoted poster doubt.
- Model versions matter: the base model and the 'Max' version can score very differently, so compare the same versions.
- Don't rely on one aggregate index. Test models yourself on real tasks.
Resources mentioned
- Artificial Analysis · website · artificialanalysis.ai · free
Independent benchmarking site that compares LLMs on intelligence index, price, cost per task and the cost-vs-intelligence Pareto frontier.
Also in: Liquid AI releases d1-ultra decision model (Melvin Vivas on X · notes), Is AI Getting Cheaper? How to Measure LLM Cost Per Task and Per Intelligence Level (Melvin Vivas on X · notes), ElevenLabs Eleven v4 & v4 Turbo: Emotive, Multilingual Text-to-Speech Models (Melvin Vivas on X · notes), Pipette: Open-Source Benchmarking for On-Device Models (Melvin Vivas on X · notes) and 3 more - OpenCode · tool · x.com · free
Open-source terminal coding agent that can be configured with different model providers, including open models.
Also in: OpenCode Console Security Breach: Check Your Provider Keys (Melvin Vivas on X · notes), DeepSeek V4.1 Flash Now Free in OpenCode (Melvin Vivas on X · notes), OpenAI Makes GPT-6 Astra and GPT-6.1 Sol About 50% Faster (Melvin Vivas on X · notes), T3Code: a better interface for terminal coding agents (Melvin Vivas on X · notes) and 13 more - Muse Spark 1.3 · tool · research.meta.ai · free
Meta's model release, available for free in OpenCode at the time of the post.
Also in: Try Muse Spark 1.3 for Free in OpenCode (Melvin Vivas on X · notes), Muse Spark 1.3 Is Free to Try in OpenCode (Melvin Vivas on X · notes), Installing Meta's Muse Code CLI (Needs a Paid Plan) (Melvin Vivas on X · notes)
Try this
- Check leaderboard rankings by testing models yourself on your own tasks.
More in Evaluation (Evals) & Testing
- GPT-6 Astra Tops Vending-Bench, Beating Claude Fable 5.1
- Local Evals: Open-Source App for Evaluating Local or OpenAI-Compatible Models
- Building an OCR Evals App with Codex
- Personal Smoke Test for New Models: "Make a CRM App"
- Compare Model Outputs with Artificial Analysis MicroEvals
- Testing Gemini 3.8 Flash in Cursor with a CRM Smoke Test