Run Muse Glimmer 30B Locally with llama.cpp and Connect It to Hermes Agent
Melvin Vivas · X post · 2026-08-11 · Open on X
Topics: LLMOps, Deployment & Monitoring, AI Agents, Tool Use & MCP, LLM Fundamentals · Level: intermediate
Summary
This post is a step-by-step guide to serving Unsloth's quantized Muse Glimmer 30B GGUF model on one RTX 3090 (24 GB) using llama.cpp built with CUDA under WSL Ubuntu on Windows 11. It covers building llama-server, the launch flags and sampling settings, a health check, and pointing an agent (Hermes Agent) at the local OpenAI-compatible endpoint. The author says OpenClaw and Pi should also work but hasn't tested them.
Key points
- Setup: Windows 11 Pro + WSL Ubuntu 24.04.1, RTX 3090. Uses about 23 GB VRAM at full 65,536-token context with the unsloth/Muse-Glimmer-30B-GGUF:UD-Q4_K_XL quantization.
- Build llama.cpp with CUDA: git clone the ggml-org/llama.cpp repo, then run
cmake -S . -B build-cuda -DGGML_CUDA=ON -DLLAMA_OPENSSL=ONandcmake --build build-cuda --config Release --target llama-server -j 8. - Launch with
./build-cuda/bin/llama-server -hf unsloth/Muse-Glimmer-30B-GGUF:UD-Q4_K_XL. The -hf flag downloads the model straight from Hugging Face. - Server flags: --ctx-size 65536, --parallel 1, --alias unsloth-Muse-Glimmer-30B-GGUF, --host 127.0.0.1, --port 8001. Raise ctx-size if you have spare VRAM; any port works.
- Sampling settings: --temp 1.0, --top-p 0.95, --top-k 64.
- Check the server with
curl -fsS http://localhost:8001/health. The OpenAI-compatible base URL is http://localhost:8001/v1. - Hermes Agent setup: choose Custom endpoint, Base URL http://localhost:8001/v1, leave the API key empty, Model = the alias (unsloth-Muse-Glimmer-30B-GGUF), Context length 65536, API mode chat_completions.
- Any agent that can use an OpenAI-compatible endpoint (the author names OpenClaw and Pi, untested) should work with the same settings.
Resources mentioned
- llama.cpp · repo · github.com · free
An open-source C/C++ engine for running GGUF models locally. Its llama-server command provides an OpenAI-compatible HTTP server.
Also in: Run Pi with local models on an RTX Spark laptop via llama.cpp (Melvin Vivas on X · notes), llama.app: Run Local Models on a Mac Without the llama.cpp CLI (Melvin Vivas on X · notes), Deploy Cheap Quantized GGUF Models on HF Inference Endpoints with llama.cpp (Melvin Vivas on X · notes), Run Local LLMs on the Computer You Already Have: A Beginner's Path (Melvin Vivas on X · notes) and 33 more - unsloth/Muse-Glimmer-30B-GGUF · tool · huggingface.co · free
Unsloth's GGUF quantizations of Meta's Muse Glimmer 30B open-weights model, used here with the UD-Q4_K_XL quant.
Also in: Three Local GGUF Models That Fit on an RTX 3090 (24GB) (Melvin Vivas on X · notes), Run Muse Glimmer 30B Locally on an RTX 3090 with llama.cpp (Melvin Vivas on X · notes), Running Muse Glimmer 30B Locally with llama.cpp and the Hermes Agent (Melvin Vivas on X · notes) - Unsloth · tool · unsloth.ai · free
Open-source library for fast, memory-efficient fine-tuning of open LLMs (LoRA/QLoRA) on a single GPU or in Colab.
Also in: 100-Iteration Autonomous Coding Loop with Local Qwen on an RTX 3090 (Melvin Vivas on X · notes), Prompting an Autonomous Agent Loop: Be Specific, Show Visible Progress (Melvin Vivas on X · notes), Running an Overnight 100-Iteration Agent Loop with a Local Qwen Model (Melvin Vivas on X · notes), Agent tasks: Unsloth fine-tuning, ComfyUI face swap, script writing (Melvin Vivas on X · notes) and 36 more - Hermes Agent · tool · github.com · free
Nous Research's open-source AI agent with CLI, TUI and desktop interfaces. It now supports hands-free activation with a wake word.
Also in: jsonl-viewer: Viewing Live Agent Traces from Codex, Claude Code, Pi & Hermes (Melvin Vivas on X · notes), Inspecting Coding-Agent Traces Live with JSONL Viewer (Codex, Claude Code) (Melvin Vivas on X · notes), Hermes Agent: each bot is its own profile (Melvin Vivas on X · notes), An X research bot built with Hermes (Melvin Vivas on X · notes) and 56 more - OpenClaw · tool · github.com · free · recommended by both Bashiri Smith & Melvin Vivas
Popular open-source personal AI agent project that showed strong consumer demand for agents.
Also in: Grok Bot vs OpenAI in the consumer AI agent market (Melvin Vivas on X · notes), Why Simpler Agent Bots Appeal to Non-Technical Users (Melvin Vivas on X · notes), Coworker: A Desktop AI Assistant App, Not a Coding Agent (Melvin Vivas on X · notes), Grok Bot's Real Competitors: OpenClaw and Hermes (Melvin Vivas on X · notes) and 9 more - Pi · tool · x.com · free
A customizable coding agent that can be extended through its Extensions API and connected to several model providers.
Also in: 100-Iteration Autonomous Coding Loop with Local Qwen on an RTX 3090 (Melvin Vivas on X · notes), Pi Agent: A Highly Customizable Agent with Durable Sessions (Melvin Vivas on X · notes), Prompting an Autonomous Agent Loop: Be Specific, Show Visible Progress (Melvin Vivas on X · notes), Running an Overnight 100-Iteration Agent Loop with a Local Qwen Model (Melvin Vivas on X · notes) and 54 more - WSL (Windows Subsystem for Linux) · tool · learn.microsoft.com · free
Runs a Linux environment such as Ubuntu on Windows, here used to build llama.cpp with CUDA.
Also in: Using WSL Inside Cursor on Windows (Melvin Vivas on X · notes), Remote-control Codex and Claude Code in WSL from your phone (Melvin Vivas on X · notes), Any OS works for AI dev: Mac, Windows + WSL (Melvin Vivas on X · notes), Running Gemma 4 E2B Video Understanding Locally in WSL (Melvin Vivas on X · notes) and 1 more - Melvin Vivas · person · x.com · free
A creator who shares experiments with AI agents and developer tools on X.
Also in: Coworker v0.7.0: Open-Source Multi-Bot Workspace with Primary Bot and Web Search (Melvin Vivas on X · notes), How LLMs Work: A Motion-Graphics Explainer Made in One Shot with Claude Opus 5.5 (Melvin Vivas on X · notes), Concept Demo: An Agent Monitor Built on OpenAI Codex Traces (Melvin Vivas on X · notes), Running MiniMax H3 Locally on a Single RTX 3090 (Demo) (Melvin Vivas on X · notes)
Try this
- Build llama.cpp's llama-server with CUDA (-DGGML_CUDA=ON) under WSL Ubuntu.
- Start llama-server with unsloth/Muse-Glimmer-30B-GGUF:UD-Q4_K_XL, 65536 context, temp 1.0, top-p 0.95, top-k 64 on port 8001.
- Raise --ctx-size if your GPU has spare VRAM.
- Check the server with curl http://localhost:8001/health.
- In Hermes Agent, add a Custom endpoint: base URL http://localhost:8001/v1, empty API key, model alias, context 65536, API mode chat_completions.
- Follow @melvindvivas for more guides.
- Run a fully local coding/agent setup: llama.cpp serving a 30B GGUF model on a 24 GB GPU, used as the backend for Hermes Agent, OpenClaw or Pi.
More in LLMOps, Deployment & Monitoring
- Unsloth releases Qwen3.8-27B GGUF quantizations
- Run Muse Glimmer 30B Locally on an RTX 3090 with llama.cpp
- Running Muse Glimmer 30B Locally with llama.cpp and the Hermes Agent
- Running Liquid AI LFM2.5-2.6B Locally with llama-server
- Adding LFM2.5-2.6B support to AIBackends with Cursor
- Zero-Shot Prompt Routing by Task Complexity with LFM2.5-Encoder