Running Qwen3.8 27B Locally on an RTX 3090 with llama.cpp and the Pi Harness
Melvin Vivas · X video post · 2026-08-31 · 0:23 · 6,224 views · Open on X
Topics: LLMOps, Deployment & Monitoring, AI Dev Tools & Productivity, LLM Fundamentals · Level: intermediate
Summary
A 23-second demo with no speech. It shows Unsloth's Qwen3.8 27B model, quantized to Q4_K_M in GGUF format, running locally through llama.cpp on one RTX 3090. The creator says Pi (@pidotdev) is the best harness he has used with this model so far. His point is that the whole setup is local and free.
Key points
- Model: Qwen3.8 27B in Unsloth's GGUF release, using the Q4_K_M quantization.
- Inference engine: llama.cpp, which runs GGUF models on local hardware.
- Hardware: one NVIDIA RTX 3090 (24 GB VRAM) runs the 27B model at 4-bit Q4_K_M quantization.
- Harness: the creator found Pi (@pidotdev) the best harness to pair with this local model so far.
- Q4_K_M is a 4-bit k-quant. It cuts memory use enough to fit a 27B model on a single consumer GPU, and is usually a good trade-off between quality and size.
- The setup costs nothing to run: it is local, with no API fees.
Resources mentioned
- Qwen3.8-27B-Q4_K_M (Unsloth) · tool · huggingface.co · free
A 4-bit (Q4_K_M) quantized version of the 27B Qwen model from Unsloth, small enough to run on one consumer GPU.
Also in: 100-Iteration Autonomous Coding Loop with Local Qwen on an RTX 3090 (Melvin Vivas on X · notes), Prompting an Autonomous Agent Loop: Be Specific, Show Visible Progress (Melvin Vivas on X · notes), Running an Overnight 100-Iteration Agent Loop with a Local Qwen Model (Melvin Vivas on X · notes), Use a Hugging Face Inference Endpoint as a Custom Model Provider in Pi Agent (Melvin Vivas on X · notes) and 5 more - Unsloth · tool · unsloth.ai · free
Open-source library for fast, memory-efficient fine-tuning of open LLMs (LoRA/QLoRA) on a single GPU or in Colab.
Also in: 100-Iteration Autonomous Coding Loop with Local Qwen on an RTX 3090 (Melvin Vivas on X · notes), Prompting an Autonomous Agent Loop: Be Specific, Show Visible Progress (Melvin Vivas on X · notes), Running an Overnight 100-Iteration Agent Loop with a Local Qwen Model (Melvin Vivas on X · notes), Agent tasks: Unsloth fine-tuning, ComfyUI face swap, script writing (Melvin Vivas on X · notes) and 36 more - Pi · tool · x.com · free
A customizable coding agent that can be extended through its Extensions API and connected to several model providers.
Also in: 100-Iteration Autonomous Coding Loop with Local Qwen on an RTX 3090 (Melvin Vivas on X · notes), Pi Agent: A Highly Customizable Agent with Durable Sessions (Melvin Vivas on X · notes), Prompting an Autonomous Agent Loop: Be Specific, Show Visible Progress (Melvin Vivas on X · notes), Running an Overnight 100-Iteration Agent Loop with a Local Qwen Model (Melvin Vivas on X · notes) and 54 more - llama.cpp · repo · github.com · free
An open-source C/C++ engine for running GGUF models locally. Its llama-server command provides an OpenAI-compatible HTTP server.
Also in: Run Pi with local models on an RTX Spark laptop via llama.cpp (Melvin Vivas on X · notes), llama.app: Run Local Models on a Mac Without the llama.cpp CLI (Melvin Vivas on X · notes), Deploy Cheap Quantized GGUF Models on HF Inference Endpoints with llama.cpp (Melvin Vivas on X · notes), Run Local LLMs on the Computer You Already Have: A Beginner's Path (Melvin Vivas on X · notes) and 33 more
Try this
- Download the Q4_K_M file from unsloth/Qwen3.8-27B-GGUF on Hugging Face.
- Run it with llama.cpp on a GPU with about 24 GB of VRAM, such as an RTX 3090.
- Connect the local model to the Pi harness to use it as a free local coding assistant.
- Build a fully local, free AI coding assistant: Qwen3.8 27B (Unsloth Q4_K_M GGUF) served by llama.cpp on a single RTX 3090 and driven by the Pi harness.
More in LLMOps, Deployment & Monitoring
- Running Qwen3.8-27B EXL3 Locally on an RTX 3090 with 220K+ Context
- Serving Qwen3.8-27B (EXL3) with 262K Context on a 24GB RTX 3090
- Ollama vs vLLM: From Local AI Demo to Production Inference Serving
- Low-Cost Agent Run: DeepSeek V4 Flash via OpenRouter in ohmypi
- Serving Qwen3.8 27B FP8 on an H100 with Baseten dedicated inference
- Running GLM 5.3 Flash on Baseten with the Pi Coding Agent