Local AI from a Hardware Stash

• tok/s: tokens per second, the speed of the output (a token is about three-quarters of a word).
• Prompt reading vs writing (pp / tg): pp = reading the prompt, tg = writing the answer; pp512 / tg128 = a 512-token prompt and 128 written tokens.
• MoE (mixture of experts): a model split into many "experts", only a few of which run per token, so a big model is cheap to run.
• Expert offload (-ncmoe N): keeps the experts of N layers in system RAM instead of GPU memory, so models bigger than the GPU still run.
• Speculative decoding (MTP, DFlash, EAGLE3): a small helper guesses the next tokens and the main model checks them: faster output, same result.
• Quantization (Q4_K_XL, IQ2_XS, IQ3_XXS…): storing weights in fewer bits to shrink a model; the number is roughly bits per weight.
• GSQ-RCO: a compression method from IST Austria used for several builds here; the "Coder" build also drops half the experts, chosen for code.
• Thinking budget / effort: a cap on how many reasoning tokens a model may spend before answering; "low" and "xhigh" are effort presets.
• Seeded bugs (24- and 80-item sets): bugs deliberately planted in the app's real code; the score is the share found.
• Mutation killing: unit tests must pass on the real code and fail on deliberately broken copies.
• eGPU / OCuLink / USB4: external GPU connections; OCuLink here is a PCIe 4.0 x4 link, a quarter of a desktop x16 slot.
• Tiers: fast (NVIDIA card), best (iGPU), third opinion (RX 7600), deep (the strongest reviewer, swapped onto the NVIDIA card on demand).

Memory available to the tests. 64 GB installed doesn't mean 64 GB usable:

Notes:

• Docker isn't the cause: idle containers barely register. The big competitor is the AMD-tier model, because an iGPU's "VRAM" is ordinary system RAM. It was stopped to free about 22 GB (leaving roughly 55–57 GiB) for the GLM-5.3-Flash -ncmoe sweeps and evals, and for the 44 GB GLM-5.2 run. The other GLM-5.2 runs (CPU, and CUDA with 28 GB) had it loaded.
• Lowering the carve-out: setting the UMA frame buffer to its BIOS minimum would recover up to about 2 GiB. The iGPU allocates through GTT anyway.

Memory speed.

• Older and slower than the platform's limit: the modules are DDR5-4800. The amdgpu driver's top memory clock is 2400 MHz (DDR: 4800 MT/s), and the kernel detects DDR5 SPD hubs on both SO-DIMMs. The HX 470 supports DDR5-5600 on SO-DIMMs.
• Theoretical peak: 2 channels × 8 bytes × 4800 MT/s = 76.8 GB/s.
• Measured (STREAM-style triad, 24 threads): about 48-49 GB/s idle (copy ~43 GB/s; same at 8, 12 and 24 threads), about 63% of the 76.8 GB/s theoretical; 47 GB/s with a RAM-offloaded model running.
• Why it matters: RAM bandwidth caps every run that keeps model weights or experts in system RAM: the AMD iGPU tier, the CPU parts of MoE offload, and GLM. For example, GLM-5.3-Flash at IQ2 reads several GB of experts per token. DDR5-5600 would add about 17% bandwidth; LPDDR5X-8000 machines (Ryzen AI Max) have about 3× more.

Goal: offload bulk, low-stakes coding work on a real production web app: extraction, first-pass review, drafts, screenshot triage from Claude to local models, measured for both speed and quality of output.

Exploration window: 2026-10-15 to 2026-10-19 (new hardware from the evening of 2026-10-18).

How models are compared:

• An eval script runs project-grounded tasks, each auto-scored from 0 to 1, with no Claude tokens spent judging.
• Inputs come from the app's source, pinned at one fixed commit so scores stay comparable across days.
• A matrix script runs a list of configs unattended, one after another.
• The tables below are regenerated by a report script after every batch.

Effort levels: bounded thinking helps; unbounded or tiny budgets hurt. On the 80-item set:

• A request can set the budget itself with "thinking_budget_tokens": N (llama.cpp; the server-wide equivalent is --reasoning-budget N).
• The harness exposes this as local_ask effort="high", which means thinking capped at 1,024 tokens.

Unbounded thinking hurts, and more bits don't help. On the 80-item set:

• Thinking takes the iGPU model from 0.887 to 0.800, and from 7 min to 2.5 h.
• Gemma's thinking over-deliberates until it hits the token budget, scoring 0.05 at 4K tokens; that's not a parser bug.
• On the iGPU, Q5_K_S and MXFP4_MOE both score 0.875 against 0.887 for Q4_K_XL: quality saturates at 4-bit.
• Gemma's MTP draft length is best at the default 3 (171 / 161 / 189 tok/s on code, summary and JSON). Longer drafts only pay on JSON.

MTP speculative decoding is the biggest win, and it's in production. The Unsloth Qwen3.6-35B-A3B-MTP-GGUF running with --spec-type draft-mtp decodes at about 196 tok/s against 104 without it. Draft acceptance runs 70–92%, at the same or better quality (0.879 → 0.919 on the easy suite).

• The MTP head costs VRAM, so llm-nv-main now runs 2 slots × 32K, and the vision encoder runs on the CPU (--no-mmproj-offload). An image still takes about 9 s.
• On the GPU, the vision encoder hit OOM at every context size tried, down to 40K.
• 99% of real prompts are under 7K tokens (from the router's usage log). The router now skips a backend when a prompt exceeds its max_prompt, so prompts over 30K go to the AMD 64K slots.

Newer llama.cpp (b11706, 64 commits later, including a Vulkan KV-cache change for AMD) gave no real gain. On the same prompts and production configs:

• iGPU: 23.4 tok/s decode on both builds.
• NVIDIA: 146 tok/s decode on both; prompt reading 2,060 → 2,142 tok/s.
• Production stays on the proven build.
• MTP decode speed depends on content (146 tok/s on one prompt, about 196 on the eval tasks), so only compare configs on identical prompts.

ROCm on the 890M is no faster than Vulkan. This used llama.cpp's prebuilt ROCm build with Ubuntu 26.04's packages (sudo apt install libamdhip64-7 libhipblas3 librocblas5).

• Ubuntu's rocBLAS has no kernels for the 890M's chip (gfx1150). HSA_OVERRIDE_GFX_VERSION=11.0.0 makes it run the gfx1100 kernels.
• Production shape: 25.0 tok/s decode and 471 tok/s prompt, against Vulkan's 23.4 and 478.
• Vulkan stays, since it needs no override.

N-gram speculative decoding (ngram-mod) gave no gain on these tasks: the outputs are short and don't copy the input. It may still help whole-file rewrites; not tested yet.

The fast tier is now Gemma 4 26B-A4B QAT (UD-Q4_K_XL) with its MTP draft, replacing Qwen3.6-35B-A3B IQ3_XXS.

• In the production shape (2 × 32K, vision on CPU) it scores higher: 0.894 against about 0.863 over 8 categories.
• It finds more bugs: bugauto 0.792 against 0.708, and bugspot 1.0 against 0.6.
• It's faster: on the same 5.5K prompt, 2,138 tok/s prompt against 1,892, at the same decode speed.
• Vision is as accurate and 2.5× faster: 84 s for 20 pages against 206 s.
• It loses only on codegen (0.8 against 1.0).
• 2 × 32K at the default micro-batch loads, then runs out of VRAM on the first real request. -ub 256 fixes it, so test fit with a real long request, not /health.

CUDA beats Vulkan on the RTX 5060 Ti. Same llama.cpp build (b11706), Gemma 4 26B-A4B, -fa 1:

An 80-item seeded-bug set (bugbig, frontend and backend Java, separate seed) confirms the switch:

• Ternary-Bonsai-2-27B (Qwen3.8-27B in ternary weights, 7.2 GB) needs PrismML's llama.cpp fork. It uses only 9.4 GB of VRAM at 2 × 32K and scores 0.877 on the full suite, so it's a strong pick for 8–12 GB cards. Here, Gemma is faster and better.
• NVIDIA's official Nemotron-3.5-Bolt "24GB" GGUF prunes experts per layer. Builds before about b11706 can't load it (check_tensor_dims ... ffn_gate_inp).

gpt-oss-20b on the GPU (12.1 GB MXFP4, low effort): close to the fast tier, and as good at unit tests (7 of 9), with very fast prompt reading (~5,000 tok/s); 65 of 80 bugs and ~128 tok/s keep Gemma 4 (70 of 80, ~200 tok/s, plus vision) in the fast tier. Its EAGLE3 drafter added a modest 7% (138 tok/s).

New model: Meta's Muse Glimmer 30B (dense 29.6B with a vision encoder, August 2026, Apache-2.0), as Unsloth's UD-IQ3_XXS (13.1 GB) fully on the 16 GB card. At low reasoning strength it found all 80 of 80 seeded bugs — the first perfect score on this rig — plus 23 of 24 on the smaller set, 21 of 23 scripts and 6 of 9 unit tests (tied with Ornith-1.5-35B for the best of any big model here; Gemma 4 has 7). It always thinks, so it writes at ~30 tok/s, but its DFlash drafter nearly doubles that to ~57 tok/s (70% of drafted tokens accepted, same output). Screen reading 0.931, a step below the current tiers. The strongest bug hunter yet that runs on the GPU alone.

Qwen3.8-27B and its Swift 1.5 tune on the card alone:

• Swift 1.5 Qwen3.8-27B IQ3_S + MTP (12.1 GB, fully on the 16 GB card, ~52-64 tok/s) on 10 seeded bugs: low effort found 9 of 10 in 7 minutes; the template's default xhigh found 8 in 16 minutes with twice the thinking (44K tokens); thinking off found 8 in 26 seconds. On this kind of bug hunt, xhigh bought nothing — low effort is the sweet spot, and thinking off is the fast path. The template's default really is xhigh.
• Base Qwen3.8-27B vs Swift 1.5 on the same 10 bugs: at xhigh both found 8, and Swift needed 35% fewer thinking tokens (44K vs 68K; 16 vs 24 minutes) — close to the 42% UkisAI reports. At low effort the base model was the best option of all: 9 of 10 in 4 minutes with only 12K thinking tokens (Swift: 9 of 10, 22K tokens, 7 minutes). Thinking off: 8 of 10 in 26 seconds for both. Takeaway for a 16 GB card: use low effort for bug hunts, xhigh only when you've measured it helps, and Swift if you do run xhigh.
• Swift 1.5 Qwen3.8-27B IQ3_XXS + MTP (10.4 GB) fits entirely on the 16 GB card with a 64K context: it found 12 of 12 functions in a 56K-token prompt in 83 seconds (reading at ~715 tok/s, writing at ~50 tok/s), and matched the larger IQ3_S on the bug sets (8 of 10 at xhigh and with thinking off). A strong long-context option that needs no RAM offload.
• Full test suite, Swift 1.5 27B IQ3_XXS (10.4 GB, entirely on the 16 GB card, with vision): at low thinking effort it found 24 of 24 and 78 of 80 seeded bugs and wrote 22 of 23 scripts — matching the best scores on this rig (the 68 GB Flash-Next deep tier models need system RAM for the same scores), at ~64 tok/s instead of ~24. Screen reading 0.965, level with the current tiers. Thinking off it is quick and solid (0.840 suite, 72 of 80 bugs). Its one weak spot is writing unit tests (2.5 of 9), where the fast tier stays ahead (7). That run of 104 tasks took 38 minutes, so it suits careful review work rather than quick jobs.
• Same test, base Qwen3.8-27B IQ3_XXS at low effort: 23 of 24 and 78 of 80 bugs, 23 of 23 scripts, in 39 minutes with about the same thinking (129K vs 127K tokens). So the jump comes from switching low-effort thinking on, not from the Swift fine-tune: any Qwen3.8-27B at IQ3_XXS gives a 16 GB card deep-tier accuracy at ~63 tok/s with no system RAM. Swift's edge shows at xhigh, where it thinks 35% less.
• Community build of the same quant — kunthawat's Swift 1.5 27B IQ3_XXS (10.2 GB, Unsloth-style imatrix): as good as UkisAI's own file and a little better without thinking: 0.858 vs 0.840 on the suite, 74 vs 72 of 80 bugs, 22 vs 19 scripts. At low effort it found 23 of 24 and 78 of 80 with 23% less thinking (98K vs 127K tokens, 31 vs 38 minutes). Its author's 16 GB profile (one slot, 100K context, 4-bit KV cache) really fits: it found 12 of 12 functions in an 82K-token prompt in 2 minutes, all on the GPU.

Qwen3.8-27B now fits the fast tier's shape. IST Austria's GSQ-RCO IQ3_S with MTP built in (12.1 GB) serves 2 × 32K with the vision encoder on the CPU (-ub 256), and a real 26K-token prompt ran (read at 860 tok/s). Against Gemma 4: more bugs (0.925 against 0.875 on the 80-item set, 21 against 19 of 24), one more script (20), the same screenshot score (0.965, but 204 s against 84 s), far fewer test tasks (2 against 7 of 9), and about 60 tok/s against 200. It's a candidate for a switchable review mode, not the default.

Qwen3-Coder-30B-A3B (UD-Q3_K_XL) loses to Gemma 4 on every category we use (bugs 0.738 vs 0.875, scripts 17 vs 21, tests 3 vs 7) at half the decode speed (~106 tok/s).

LFM2.5-24B-A2B: rejected. It always reasons; neither enable_thinking=false nor reasoning_effort=low stops it, so at a normal token budget it never answers.

• With a 3000-token budget it scores 0.591, finds almost no bugs, and wrote 134K tokens for the suite.

Liquid's small models: rejected. Both were run on the RTX 5060 Ti with llama.cpp on the same suite and screenshot set.

• LFM2.5-1.2B-Instruct scores 0.198 at Q4_K_M and 0.134 at Q8_0. It wrote 0 of 23 scripts and 0 of 9 unit tests, scored 0.062 on the 80 seeded bugs, and only JSON output (0.84 at Q4_K_M) and the simple code tasks (0.45) are usable. It is fast, at ~448 tok/s on the 5060 Ti and ~218 on the RX 7600 (Vulkan), but speed does not make up for it.
• LFM2.5-VL-1.6B reads screenshots at 0.436 in 32 s, against 0.965 for Gemma 4 and Qwen3.8-27B. It got the main heading right on only 25% of pages and found 10% of the buttons, and 3 of 20 answers did not parse.

Unit-test writing, scored by mutation killing. The model writes bun:test tests for 9 pure functions from the app. The tests must pass on the real code and fail on deliberately broken copies (15 mutants, after dropping one equivalent mutant a frontier model's tests also couldn't kill).

• Gemma 4 (fast tier): 7 of 9 functions in 67 s, killing every mutant wherever its tests passed. Qwen3.6 Q4 on the iGPU: 3 of 9, in 680 s.
• Every miss was a test with a wrong expectation, so it fails on the correct code. A real run can detect that without a reference answer, so execute-and-repair should recover most of these.

Long prompts cost speed, not accuracy, on the iGPU tier. A probe packs real source files into one prompt and asks which file defines each of 12 functions spread through it. Qwen3.6 Q4 on the iGPU (64K slots) found all 12 at 18K tokens (prompt read at 357 tok/s, 84 s in all) and at 55K tokens (187 tok/s, 3.6 min).

Ornith-1.5-35B-A3B (a Qwen3.5-MoE fine-tune) doesn't beat the current tiers. On the 5060 Ti (IQ3_XXS) it scores 0.723 without thinking; with a 2,048-token thinking budget it finds 0.887 of the 80 bugs and passes 19 scripts and 6 test tasks at 113 tok/s, close to Gemma 4 but not better (0.963 bugs, 21 scripts, 7 tests at max effort). On the iGPU (Q4_K_M) it scores 0.776 at 12 tok/s, against Qwen3.6's 0.878 at 22. Llama 3.1 8B, Mistral Nemo 12B and Ling 3.0 Tiny score 0.52–0.60; both DeepSeek-R1 distills are worse still (0.37–0.46) and slow.

Vision (local Qwen3.6 multimodal) for UI audits#

• Triage only: about 30% of its UI findings were confirmed by deterministic probes (DOM measurement, axe).
• No exact numbers: pixel and contrast claims from the model are filtered out in code, because scripts measure those.
• Page context didn't help: adding the page's purpose to the prompt didn't improve precision.
• Speed: one 1440p screenshot takes about 8–9 s. Moving the vision encoder to the CPU (to make room for MTP) didn't change that.
• Reading the screen is reliable, judging it isn't. A second eval scores the model against facts read from the live DOM rather than by eye. Playwright saves each screenshot together with its h1, button and tab labels, table headers and all visible text. Across 20 admin pages, the 4-bit Qwen3.6 on the iGPU scored:

• Comparing against <button> tags alone made it look like it invented labels. Those "extras" were real dropdowns, links, form labels and calendar day names, so grounding is checked against all visible text.
• Card-style buttons are scored by their first line (the title, not the subtitle).
• Use the vision model to report what a page shows. Verify any claim about what's wrong with a script.
• The 3-bit NVIDIA model reads screens just as well (same scores, no invented labels), so the compression that hurts bug-finding doesn't hurt reading.
• It's 2× slower per screenshot (about 10 s vs 5 s), because its vision encoder had to move to the CPU to make room for MTP.
• Vision requests now go to the iGPU first, with NVIDIA as overflow.
• MiniCPM-V 4.7 doesn't beat them.
  • 35B-A3B (IQ3_XS) on the iGPU: scores 0.939 (buttons 0.77) at 6.3 s per page, against Qwen's 0.968 at 5 s.
  • 1B on the CPU alone: uses 2.2 GB of RAM and no GPU, and scores 0.838. It gets every heading right and invents nothing, finds 91% of table headers but only 45% of buttons, at about 6 s per page. It's a usable zero-GPU option for identifying pages and columns, not for full UI inventories.

RX 7600 (8 GB, USB4): Vulkan only. Ubuntu's rocBLAS has no gfx1102 kernels, so the ROCm build fails. On Vulkan, Llama 3.1 8B Q4 runs at 874 tok/s prefill and 49 tok/s decode; Qwen3.5 9B at 834 and 39.

Small models are no substitute for the fast tier; Ornith-1.5-9B with a thinking budget comes closest. With thinking capped at 1,024 tokens it scores 0.810, finds 18 of 24 seeded bugs, and passes 16 of 23 script tasks and 6 of 9 test tasks at 32 tok/s. Qwen3.5 9B with the same budget scores 0.773 (14 bugs, 17 scripts, 5 tests). Both need thinking: without it Ornith drops to 0.698. The full suite takes 55 minutes, about five times Gemma 4's wall time, so it's a slow third worker for background jobs on the otherwise idle RX 7600.

• On the 80-item bug set Ornith-1.5-9B finds 0.825 at 35 tok/s, close to Gemma 4 (0.875) on an 8 GB card.

DFlash drafter: about 2.5× faster, same answers (2026-10-21). A community DFlash draft model distilled for Ornith-1.5-9B (audreyt/Ornith-1.5-9B-DFlash-GGUF, Q4_K_M, 0.77 GB) fits beside it in 8 GB (7.3 GB used at 16K context). On a 700-token code answer:

The suite score is unchanged (0.701 with the drafter, 0.698 without), as expected for speculative decoding. Shorter drafts win here: the small card's time goes to checking drafts, so fewer, likelier guesses pay off. The service now runs with --spec-type draft-dflash --spec-draft-n-max 4.

The NPU runs Qwen3.6-35B-A3B well enough to take script jobs. With FastFlowLM 1.0.9 it decodes at about 18 tok/s (the iGPU's best tier: ~22) and passed 20 of 23 script tasks, about as many as Gemma 4 (19 by default, 21 at max effort). It found fewer seeded bugs than the same model on the iGPU (15 vs 22 of 24) and prefills slowly (87 tok/s, so a 5K-token prompt takes about a minute). Moving script generation there would leave both GPUs free.

• The other NPU models are slower and weaker: Llama 3.1 8B 0.535, gpt-oss 20B 0.407 (20 tok/s). Gemma 4 12B and Qwen3.5 9B first scored 0 on scripts and tests. With a 4,096-token limit Gemma 4 12B passed 16 of 23 scripts and 5 of 9 test tasks, but at about 7 tok/s. Qwen3.5 9B still failed: its long answers turn into runs of spaces. A probe showed neither model thinks on short prompts, so the thinking theory was wrong. Of the smaller NPU models, only Gemma 4 E4B is interesting: weak at finding bugs (4 of 24) but 6 of 9 test tasks and 15 of 23 scripts, reading prompts at 421 tok/s (Qwen3.6 on the NPU: 87). Qwen3 4B scored 0.594 and Phi-4-mini 0.352. Qwen3 8B thinks on every answer (80–450 s per bug item) and was stopped; other thinking models were skipped.

The NPU needs no PPA or DKMS on Ubuntu 26.04. FastFlowLM's Ubuntu 26.04 package needs only libxrt2 and libxrt-npu2 from Ubuntu's archive (kernel 7.0 ships amdxdna and the 1.1 firmware), plus an unlimited memlock limit.

NPU (AMD XDNA2, amdxdna driver, fw 1.1.2.64)#

• Colibri: has an --xdna path, but it's experimental and Windows-only.
• FastFlowLM 1.0.9 (working since 2026-10-18):
• Install (root): apt install libxrt2 libxrt-npu2 libxrt-utils-npu ./fastflowlm_1.0.9_ubuntu26.04_amd64.deb. Ubuntu 26.04's own XRT packages are enough; no PPA, no DKMS on kernel 7.0.
• Memlock: the NPU maps large buffers, so set memlock unlimited (pam limits) and LimitMEMLOCK=infinity for user@.service if it runs as a user service. flm validate reports the limit.
• FLM_MODEL_PATH puts models on the data disk; flm serve <tag> -p <port> exposes an OpenAI-style /v1/chat/completions, and reports speeds in usage (prefill_speed_tps, decoding_speed_tps) rather than llama.cpp's timings.
• The NPU kernels are closed source but free to use and redistribute unmodified (FastFlowLM's binary licence).
• Measured decode: Qwen3.6-35B-A3B ~18 tok/s, gpt-oss 20B ~20, Llama 3.1 8B ~12, Qwen3.5 9B ~9, Gemma 4 12B ~6. Prefill 87–375 tok/s, far below the GPUs.

Latest model test — Swift 1.5 (2026-10-20, all builds measured): UkisAI's Swift 1.5 builds of the same models, tuned to think less at the same quality.

• Credits: the Swift 1.5 reasoning fine-tunes of Qwen3.8 (27B and Flash-Next) and the GSQ-RCO GGUF files tested here are UkisAI's, downloaded from their Hugging Face repos ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF and ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF. Community quantizations of the same models (for example kunthawat's IQ3_XXS build of the 27B) were not part of these tests.

• Swift 1.5 Qwen3.8-Flash-Next GSQ-RCO IQ2_XS (68 GB, experts in RAM, -ncmoe 36 on the 16 GB card): 24 of 24 seeded bugs with thinking off, in 5.8 minutes — the best score measured on this rig at the time (Muse Glimmer later found 80 of 80 on the bigger set; the current Coder deep tier: 22 of 24; the old GLM tier needed 26 minutes for 24). On the 80-item set it scored 0.975 (78 of 80), tying the best result on this rig (Coder deep tier: 0.963). A clean speed run (nothing else writing to the SSD) put it level with the current deep tier on output speed: 23.6 vs 26.1 tok/s (it reads long prompts slower: 161 vs 270 tok/s) — the earlier slow timings came from model downloads sharing the disk. More accurate at nearly the same speed: it's now an opt-in deep tier (llm deep start swift).

• Swift 1.5 Flash-Next IQ3_XXS (76 GB, -ncmoe 40): also 24 of 24 and 78 of 80, but slower (9.7 and 34 minutes, ~4.5-4.8 tok/s) — the smaller IQ2_XS gets the same score faster, so it stays the pick among the Swift builds.
• Swift 1.5 Flash-Next Q2_0 (66.5 GB, -ncmoe 32): 23 of 24 and 72 of 80 seeded bugs — clearly below IQ2_XS (24 and 78) at a similar size, so IQ2_XS remains the Swift pick.
• Swift 1.5 Flash-Next IQ3_S (84 GB, the largest, -ncmoe 40): 24 of 24 and 78 of 80 — no better than IQ2_XS and slower (~6 tok/s). Verdict across all four Swift quants: IQ2_XS is the one to run (best score, smallest of the top scorers, as fast as the current deep tier).

Optimization tested: llama.cpp's new GPU cache for MoE experts kept in RAM (--moe-cache-mib, merged 2026-10-07; up to 2.2× on Flash-Next on desktop cards). On this rig the existing fixed layout stays fastest (Coder 28.7 tok/s, Swift 32.8); every cache setting was slower (15–28 tok/s). The likely reason is the eGPU link: the card hangs off OCuLink (PCIe x4), so each cache miss costs about 4× what it does in a desktop x16 slot. Worth re-testing with a card in an x16 slot.

New hardware (2026-10-18): a third GPU, the NPU and a second SSD.

Qwen3.8-Flash-Next (177B, about 6B active per token, Unsloth UD-Q2_K_XL, 79 GB) is the best big model so far, and the first that's quick enough for daytime use. It runs on the 5060 Ti with 40 expert layers in system RAM and the rest streaming from the fast SSD:

• Its MTP draft (a separate 4 GB Q8_0 file) adds only about 10%: 19.1 tok/s against 17.4, with 77% of drafted tokens accepted.
• 64K context fits at the same 40 expert layers in RAM, with no quality loss: still 24 of 24 seeded bugs, and the long-prompt probe found all 12 functions in a 28K-token prompt (read at 150 tok/s, 3.4 min) and a 55K one.

Better builds of the same model: IST Austria's GSQ-RCO quants (the lab behind GPTQ). One keeps every expert at about 2.3 bits (IQ2_XS, 68 GB); the "Coder" build removes half the experts, chosen for code, tool use and vision, and stores the rest at about 3.5 bits (58 GB). Both load with -lm mmap -lzm on so a 29 GB n-gram shard stays on disk.

• The Coder build is the pick: smallest, fastest, and on the 80-item set within one bug of the best. IQ2_XS ties GLM-5.3-Flash at high effort (0.975) without thinking.
• A 2,048-token thinking budget doesn't help the Coder build: 21 of 24 bugs in 25 min, against 22 in 3 min without thinking.

Nemotron-3.5-Super-VL 120B-A12B (UD-IQ2_XXS, 51 GB) loses everywhere. With 80 expert layers in RAM (fewer run out of VRAM) it decodes at 13–18 tok/s and scores 0.906 on the 7-category suite, 17 of 24 and 0.800 on the bug sets, 19 scripts and 5.5 tests. Its screenshot score (0.961) matches Gemma 4's, but takes 8 minutes against 84 s.

• Fewer than 40 expert layers in RAM run out of VRAM.

DeepSeek-V4-Flash (284B, 13B active, Unsloth UD-IQ2_XXS, 91 GB) decodes about twice as fast as the GLM-5.3-Flash deep tier. On the 5060 Ti with experts in RAM and the rest streaming from the fast SSD: 4.98 tok/s at -ncmoe 40 -ub 2048 (prefill 27 tok/s), 5.75 tok/s with the default batch; GLM-5.3-Flash managed 2.91 in the same conditions. -ncmoe 36 and below run out of VRAM.

• Its DSpark speculative decoder doesn't help here. The 11 GB draft model doesn't fit on the card next to the main model. On the CPU it's accepted about 65% of the time but slows decoding to 3.9–4.3 tok/s.
• Quality: close to GLM-5.3-Flash, at about twice the speed, but Qwen3.8-Flash-Next (above) now beats it. 0.942 on the 7-category suite (GLM-5.3-Flash 0.971), 21 of 24 seeded bugs (GLM 24), 0.912 on the 80-item set (GLM 0.975 at high effort, Gemma 4 0.875), and 17 of 23 script tasks. It decoded at 6–7 tok/s during the suite, prefill about 65 tok/s. The 24 bugs took 12 minutes, against GLM-5.3-Flash's 26.

GLM-5.3-Flash (320B / 18B active), Unsloth UD-IQ2_XXS 102 GB, on llama.cpp CUDA with experts in RAM:

• Best setting: -ncmoe 42 -ub 2048, giving 3.6 tok/s decode and 52 tok/s prompt (about 12× GLM-5.2).
• Expert layers 36–40 on the GPU ran out of memory.
• It always thinks; the template defaults to reasoning_effort=max. low is the cheapest option.
• Best quality measured here: 0.971 on the full suite (effort=low, 3000-token budget) and 24/24 on bugauto, in about 26 min for those 24 items.
• Now the deep tier: llm deep start swaps it onto the NVIDIA GPU in place of the fast model (loads in about 50 s), and requests run as background jobs whose answers land in a file.
• First real job, open-ended "find bugs" on a 51-line file:
• Template kwargs: a request's chat_template_kwargs replace the server's --chat-template-kwargs, so reasoning_effort=low has to be sent with every request.
• Low temperature loops: at temperature 0.3 the 2-bit model looped, repeating the same reasoning lines for 6000 tokens (34 min, no answer).
• Fix: the vendor sampling (temperature 1.0, top_p 0.95) plus llama.cpp's DRY anti-repetition sampler (dry_multiplier 0.8). It then finished on its own in 3.8K tokens (22 min).
• Answer quality: 2 of 3 findings were real and subtle; 1 was wrong; line numbers were off. The 4-bit Qwen3.6 on the same prompt rambled without a conclusion.
• Verdict: usable for an overnight second opinion, never interactively.
• 102 GB of experts against 59 GB of RAM means a share of each token streams from the NVMe (about 1.9 GB/s of reads while decoding).
• Decode drops to about 3 tok/s when anything else uses memory bandwidth.
• Colibri's int4 Flash is 195 GB and would be disk-bound, so it wasn't tried.

Qwen3-Coder-Next 80B-A3B (UD-Q4_K_XL, -ncmoe 36 -ub 2048, GPU + RAM): 0.861 on the full suite. That's no better than Qwen3.6-35B, so it isn't worth the RAM.

The bigger 24-item seeded-bug set (bugauto) separates what the 5-item sets couldn't. Share of seeded bugs found:

• The 3-bit quant that makes the fast tier fit in 16 GB costs bug-finding. Reviews and bug hunts now go to the best tier (the 4-bit model on the iGPU), which moved to the MTP GGUF: about 22–24 tok/s in service, up from 19, at the same quality.
• Qwen3.8-27B matches the iGPU model on bugs at about 2.5× its speed, but it can't be the main NVIDIA model.
• Weights plus MTP draft come to 15.6 GB, leaving room for one 16K slot; 2 × 32K fails to load. The fast tier serves 2 × 32K, and real prompts reach 53K.
• Without the draft it would decode at about 30 tok/s.
• It stays an on-demand option for high-quality batches.
• Better: the same 4-bit Qwen3.6 on the NVIDIA card with 20 expert layers in system RAM (-ncmoe 20 -ub 2048 -b 2048, 2 × 32K, MTP).
• It keeps the iGPU tier's bug-finding (bugauto 0.917) at about 3× its speed.
• Same 5.5K-token prompt, prompt / decode tok/s: 967 / 73 here, 1,892 / 142 on the fast tier, about 490 / 22 on the iGPU.
• Every other category scores the same as the fast tier, so it's a switch rather than the default: llm profile review swaps the NVIDIA service to it (a systemd drop-in) and routes best there; llm profile fast reverts.
• Smaller -ncmoe values (12, 16) run out of VRAM; 17 fits but isn't faster.

Qwen3.8-27B (dense, IQ4_XS + MTP draft) scored 0.942 on the 7-category suite, the best of the models that fit entirely on the GPU (only GLM-5.3-Flash, offloaded to RAM, scored higher). It's the only model to find all of the easy injected bugs (bugspot 1.00, against 0.60 for Qwen3.6). It runs at 62 tok/s and takes about 2.5× the wall time, which makes it a candidate for the best tier.

Script generation, so analysis runs on the CPU instead of being read by an LLM.

• The benchmark: 23 tasks, 12 core and 11 hard. The hard ones cover CSV quoting and UTF-8, paths with spaces, date math, joins, multi-line Java annotations, CLI arguments, nearest-rank percentiles, stable ties and a 400k-line stream.
• How it runs: the model writes a Python or bash script, which runs in a bwrap sandbox (read-only root, no network, prlimit CPU and memory limits) against seeded fixtures. Its output must equal a reference answer.
• Validation: a frontier model solving the tasks blind from the prompts scores 23/23, so the specs and references are consistent.

• Effort levels on the 12 core tasks. A reasoning-token budget per request (thinking_budget_tokens; unbounded thinking was worse everywhere):

Hard set with a 2,048 budget: Gemma 9/11 (161 s), against 8/11 without thinking. - Which tasks fail:

• Repair loop results (Qwen3.6 Q4, up to 2 rounds): core 9 → 9/12, hard 8 → 9/11.
• Repair loop: feeding crashes, empty output or invalid JSON back to the model helps only with crashes. Wrong-but-running scripts need another signal.
• That signal is agreement. When two different models independently wrote scripts with identical output, the answer was correct in 21 of 21 cases, and every disagreement marked a wrong script.
• Failures cluster: every model failed comment-aware multi-line Java parsing, and bash scripts often broke on paths with spaces. So default to Python, and use a code map or code graph for code structure.
• Prompt checklist: a "script-writing checklist" added to the prompt didn't help overall.
• The local_script tool puts this together:
• it writes the script, runs it on the real project tree mounted read-only, repairs crashes, and returns only the output;
• optionally it has the second model write its own script and reports VERIFIED or DISAGREE;
• it keeps useful scripts for reuse.

Embeddings (Qwen3-Embedding-0.6B Q8_0, prompt processing)#

The embedding model lives on the AMD iGPU, which keeps the NVIDIA card free for chat.

Code search: what works (known-answer queries on the real app)#

• Hybrid search is the default.
• The 30-query run found that the tool fused only the top 5 semantic hits and counted each file once. It now fuses the top 10 of each list and counts every hit, which lifted MRR from 0.822 to 0.856.
• EmbeddingGemma 2 tied on hits but ranked answers lower, so Qwen3-Embedding stays.

Code-graph MCP servers (same 3 questions on the real repo)#

codebase-memory-mcp was kept. Gortex's daemon was stopped.

• Per-agent cost: a harness-guided UI audit agent costs about 50–65K Claude tokens, regardless of scope.
• Cost per finding: about 10–12K tokens per verified finding at best.
• Model choice: Haiku used more tokens than Sonnet (69.6K over 17 calls against 52.6K over 10), but at a fraction of the price. That makes Haiku the cheaper choice when the harness does the measuring.
• What helped most: going map-first (code map and graph before reading files). It cut screenshots to 0 and source reads to about 200 lines.

First real QA round with this setup (2026-10-19): two Claude teammates ran discovery on a real production web app (its data pipeline and generated reports checked against the data) with the local tools available, and found 14 verified problems (6 P2).

• None of the 14 came from a local model. Local first-pass hunts over long service files produced plausible but wrong claims (an authorization hole the resource layer already blocks, a double count a query already prevents); the teammates refuted each one.
• The one useful local result was a cross-check: a generated script with a second, independent script recomputed a form's totals and confirmed the app's arithmetic (the two scripts disagreed, and the one that matched the API was right).
• Caveat: the teammates' tool server predated the three-model mode, so they ran the two-model mode. The next round measures the three-model mode and the deep tier's commit pre-review against the Claude reviewer.

• Data: 221 Claude Code transcripts (3 main sessions, 218 subagents) and the local server's call log, 2026-10-18 to 22. Read-only replays; tokens ≈ characters / 4. Power was sampled with nvidia-smi on 2026-10-22, and the CPU bench was a CPU-only llama.cpp run taken while the other tiers were serving.
• Anthropic pricing (Claude Opus 5.5: $4 input, $5 5-minute cache write, $0.20 cache read, $20 output per MTok), read 2026-10-22.
• Electricity: Meralco October 2026 residential rate ₱15.0615/kWh (Daily Tribune, 2026-10-13, Philstar, 2026-10-14).
• RTX 5060 Ti 16 GB price: US street median $804.99, Aug–Sep 2026 (TweakTown); lowest $766 on 2026-10-22 (gpuprix); launch MSRP $429 (Gadget Pilipinas, 2025-04).

• From an outside recommendation list (2026-10-18), status:
• Done: Vulkan vs CUDA on the RTX 5060 Ti (CUDA wins), Qwen3-Coder-30B, Llama 3.1 8B, Mistral Nemo 12B, Ling 3.0 Tiny, Qwen3.5 9B without thinking (see finding 3).
• Also done: the DeepSeek-R1 distills, Qwen3.5 9B with a thinking budget, the NPU suite, the deep tier from each SSD, DeepSeek-V4-Flash speed and DSpark, Colibri GLM-5.2 cold runs.
• Also done: DeepSeek-V4-Flash quality suite, the NPU re-check, Ornith-1.5 9B and 35B.
• Also done: Qwen3.8-Flash-Next in three builds (speed, MTP, full suite, 64K context), Qwen3.8-27B GSQ-RCO in the fast tier's shape, Nemotron-3.5-Super-VL, Colibri GLM-5.2 on Vulkan and warm, the smaller NPU models, long-prompt probe, test writing at 4,096 tokens.
• Also done: the Coder build with thinking (same 80-item score, ~10x slower), at 64K context (fits, 12/12 at 55K), and as the deep tier on real jobs: commit reviews in under 70 s with nothing invented, but open-ended single-file bug hunts claimed false DOM facts in 5 of 6 runs.
• Done 2026-10-19: the Coder build is the deep tier (GLM-5.3-Flash deleted); the RX 7600 serves Ornith-1.5-9B as a third opinion; first QA round run with the local tiers (see Today's setup).
• Next: measure the three-model mode and the deep commit pre-review against the Claude reviewer in the second QA round; try Qwen3.8-27B as a switchable review mode.
• Original list:
• Small dense models for speed:
  • Qwen3.5 9B (Q4_K_M and Q8_0)
  • DeepSeek-R1-Distill-Qwen 7B/8B
  • Llama 3.1 8B (Q8_0)
  • Mistral Nemo 12B
• Ling 3.0 Tiny (7.9B MoE, claimed 120+ tok/s).
• Qwen3-Coder-30B (Unsloth UD-Q3_K_XL, GPU only).
• Each one runs through the same suites (bug-finding, script generation, test writing, vision where relevant) against the current fast tier, Gemma 4 26B.

• Image generation (FLUX.1 / SDXL) is out of scope for this harness.

• Deep tier: solved by switching models: the pruned Qwen3.8-Flash-Next Coder build (58 GB) fits with room for 64K context, where no GLM-5.3-Flash quant fit in RAM.

• Whole-file rewrite task: decide whether n-gram speculation is worth it (low priority: delegated work is mostly summaries and extraction).
• Nemotron 3.5 Super VL (120B / 12B active): tested; loses to Qwen3.8-Flash-Next on code and to Gemma 4 on screenshots (finding 3).
• Done 2026-10-21: Ornith-1.5-122B, Qwen3.8-27B IQ4_XS partial offload, the Strata engine (all five models it supports), and the Ornith-1.5-9B DFlash drafter (adopted on the RX 7600). DeepSeek-V4.1-Flash-1017 (763B; smallest GGUF 233 GB) is out of reach for 59 GB of RAM.
• Hardware levers measured: 128 GB RAM would hold GLM-5.3-Flash's experts entirely. A second NVMe was added on 2026-10-18: it speeds up expert streaming by about 20% (finding 3).

The evals use the app's real code, pinned at one commit, and are scored automatically (see "How models are compared" below). "—" means not run.

^* 7-category suite (before the 24-item bug set joined it). ^† 11-category suite without the 80-item bug set. ^‡ Re-run with a 4,096-token limit (the first run gave both 0 on scripts and tests). Gemma 4 12B recovered; Qwen3.5 9B's long answers degenerate into runs of spaces on the NPU, so its low score is real. ^¶ Test writing with a 4,096-token answer limit instead of 1,024: these models write long test files that the default limit cuts off mid-line (Qwen3.8-Flash-Next 2 → 5 of 9; DeepSeek-V4-Flash 0 → 1, its files run to 300–400 lines). Gemma 4 stays at 7 of 9 with the larger limit (49 s). Every Qwen3.8 variant writes 300–470-line test files and some wrong expectations, so test writing stays with Gemma 4. ^§ Its 11-category score includes 0 on tests at the 1,024-token limit. Times, settings and the per-category scores are in the findings and the auto-generated tables.

Full suite, best first. Quality: score = mean of category scores (0–1). Speed: wall seconds for the whole suite, median decode (gen) and prompt (pp) tok/s.

Partial runs (subset of categories, older suite versions):

The second SSD speeds up models that stream from disk. The Samsung 9100 PRO reads at 7.2 GB/s (a PCIe 5 drive held to PCIe 4 x4 by the slot) against the WD's 4.7 GB/s. GLM-5.3-Flash deep tier, llama-bench, cold page cache:

Splitting the shards across both drives is slower than the Samsung alone: llama.cpp waits on the slower drive. So only the models that stream (bigger than RAM) live on the Samsung; everything that loads fully into GPU or NPU memory went back to the WD.

GLM-5.2 on Colibri, rerun on the fast SSD: 0.31 → 0.50 tok/s, still unusable. Best cold setup: CUDA, model on the Samsung, a 248 GB partial mirror of the most-used experts on the WD (COLI_MODEL_MIRROR), direct reads and pipelined loading (DIRECT=1 PIPE=1); 28% of expert bytes came from the mirror. Part of each step's gain is Colibri learning which experts to pin (hit rate 18% → 35%). Details in the GLM log below.

• Vulkan beats CUDA here: 0.60 tok/s on the 5060 Ti with Vulkan, against 0.42 with CUDA in the same cold setup. Adding the RX 7600 as a second Vulkan device gave nothing (0.61). Colibri's launcher flags --gpu and --vram switch on CUDA, which the Vulkan engine refuses; use --gpu none and size the Vulkan expert tier with COLI_VK_TIER_GB.
• Warming up doesn't help: 8 requests in one CUDA process ran at 0.44 tok/s for the first and 0.41 on average for the last four.

GLM-5.2 (744B) isn't viable here, on Colibri or on llama.cpp.

• Decode runs at 0.3 tok/s on CPU, CUDA, and with 28 or 44 GB RAM. The single NVMe limits it, because only about 8% of experts can be cached.
• Rerun on 2026-10-18 with a second, faster SSD: 0.50 tok/s at best (finding 3). Still about 20 minutes for a 500-token answer.
• Prefill of a 19K-token diff didn't finish layer 1 of 78 in 13 minutes.
• The smallest GGUF is 217 GB, about 3× RAM plus VRAM.
• Deleted (429 GB) after the first test; downloaded again for the SSD rerun.

GLM-5.2 (744B MoE, about 40B active) on Colibri v2.0.0#

• Engine: JustVugg/colibri v2.0.0, an MoE engine that streams experts from NVMe.
• Trust check: about 42k stars, and every published advisory affects ≤1.4.0 only.
• Builds: colibri-cpu, colibri-cuda (CUDA=1 CUDA_ARCH=sm_120, CUDA 13.0) and colibri-vk.
• CUDA toolkit: NVIDIA's ubuntu2404 apt repo, cuda-toolkit-13-0 (nvcc 13.0.88).
• Weights: mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp.
• 429 GB, 142 Colibri-format safetensors shards: int4 experts plus an int8 MTP head.
• About 2 hours to download with start-here.sh --model glm-5.2 --backend cpu.
• Gotcha: after the download, start-here.sh launches its own coli web server on :8000 (CPU). Kill it before starting a managed service.
• Colibri's own plan for this box:
• Dense part: 11.9 GB in RAM.
• Expert cache: 17.8 GB (10 per layer).
• Projected expert hit rate: 8%.
• Service: coli serve --model <model dir> --host 127.0.0.1 --port 8000 --gpu auto --vram 14 --ram 28 --auto-tier --ctx 32768 --no-think It swaps with the NVIDIA llama-server because both need the 16 GB card.

Decode benchmark (48-token TypeScript prompt, temp 0, identical output in every run):

Rerun with a second SSD (2026-10-18). Same engine build, CUDA, --vram 14 --ram 28, a 26-token code prompt, 48 tokens out, each run a fresh process:

• The mirror served 28% of expert bytes; about 2 s per token (p50 2.06 s).
• The hit rate rises run to run because Colibri pins the experts it has seen used (.coli_usage), so part of each gain is learning, not the disk.
• Colibri's Vulkan engine needs COLI_VULKAN=1 to use a GPU (without it, it runs on the CPU and prints no [VK] ready line), and its server only accepts the model id /v1/models lists.

Observations:

• Decode is disk-bound.
• Each token needs about 40B active parameters, and with ~8% cache hits most experts stream from one WD SN740 NVMe.
• CUDA cut time to first token 13×, but decode didn't move.
• Adding RAM (28 → 44 GB) barely changed the hit rate.
• MTP speculation was left off (COLI_CUDA_MTP=0, logs show draft=0). Colibri's docs measure it as a net loss below about 99% expert residency.
• Prefill is the real blocker.
• First real task: a review of a 19,349-token diff. After 13 minutes it was still on layer 1 of 78.
• The work was CPU-bound: 51% CPU, no IO wait.
• RAM ran out (about 5 GB in swap).
• Projected time was many hours, so it was killed.
• For comparison, Colibri's README reports GLM-5.2 at:
• 0.05–0.1 tok/s cold on a 25 GB laptop
• 1.83 tok/s on a 128 GB Ryzen AI Max+ 395
• 9 tok/s on 6× RTX 5090
• Harness changes made while testing:
• At 0.3 tok/s, a synchronous tool call times out after about 180 tokens, so deep requests became detached jobs that write the answer to a markdown file.
• Jobs refuse prompts over 8,000 characters.
• GLM-5.2 on llama.cpp instead:
• The Colibri files can't be loaded by llama.cpp.
• The smallest GGUF (Unsloth UD-IQ1_S) is 216.7 GB, about 3× RAM plus VRAM, so it would hit the same disk-streaming limit. Not downloaded.
• Decision: GLM-5.2 was deleted to free 429 GB.
• What would make GLM-5.2 viable:
• 128 GB or more of RAM.
• A second NVMe: the README's 1.5 tok/s DeepSeek-V4-Flash rig used two.
• Or the hosted API.

GLM-5.3-Flash (320B / 18B active, hybrid sparse + linear attention, natively multimodal)#

• Vendor claim: better than GLM-5.2 on agentic coding. DeepSWE 63.4 vs 46.2; Toolathlon 78.4 vs 59.9. These are vendor-reported, not verified here.
• Colibri: supports it in int4: a 328 GB download converted to 195 GB. The README reports about 20 s per token warm with 25 GB of RAM. Not tried: it would be disk-bound like GLM-5.2.
• llama.cpp: support merged in PR #27773 on 2026-09-30.
• The Unsloth model card still links the closed PR #27754, but build b11642 already includes the glm5-next architecture.
• Unsloth GGUF sizes:

A third-party IQ1_M at 65 GiB fails the trust check (2 likes). - Download: about 25 minutes at ~70 MB/s, as 4 resumable parallel shards. - llama-bench (CUDA, -ngl 99, experts kept in RAM except the last ncmoe layers, KV q8_0), with prompt-processing (pp) and decode (tg) speeds in tok/s:

• Thinking can't be disabled. The chat template sets reasoning_effort to max unless given low or high. This matches Z.ai's GLM-5.3 notes: "Disabling thinking is no longer supported", with levels low, high and max, and max recommended for coding. At 3.6 tok/s, thinking tokens dominate latency, so evals use reasoning_effort=low.
• Verdict so far: about 12× GLM-5.2's decode speed and a much faster prefill (a 19K-token prompt takes about 6 minutes). Usable only for batch or overnight jobs. Quality eval pending.

GLM-5.3 (full model)#

• Per Z.ai's GLM-5.3 post, the weights were due "in two weeks". It's the same 744B class as GLM-5.2, so the conclusions above apply.
• Hosted access: the GLM Coding Plan, which works with Claude Code and other agents, with off-peak usage at 50% of points.

Measured with llama-bench b11642, -fa 1. Speeds are in tok/s: prompt-processing at 512 and 4,096 tokens (pp512, pp4096) and decode (tg128).

Decisions:

• The NVIDIA card serves the fast tier, with a quant that fits fully in VRAM. Pushing experts into RAM to fit Q4 makes it CPU-bound, and the GPU's 5× advantage disappears.
• The AMD iGPU serves the "best" tier with the same model at Q4. Using the same model family on both chips keeps answers consistent and lets either side take over if the other fails.
• On this chip, ROCm/HIP is slower than Vulkan, as reported in llama.cpp #20099.

System:

AI stack (all in user space, no root):

• llama.cpp b11642 prebuilt release binaries, two builds side by side:
• Vulkan, for the AMD iGPU.
• CUDA 12.8, for NVIDIA.
• Colibri built from source three ways: CPU, CUDA (CUDA 13.0, sm_120) and Vulkan.
• Services are systemd --user units with linger enabled, so they start at boot with no login:
• llm-nv-main (:8093)
• llm-amd-main (:8091)
• llm-amd-embed (:8092)
• llm-deep (:8000; off by default, swaps with llm-nv-main)
• Idle memory: every llama-server uses --sleep-idle-seconds 1800, which frees VRAM and RAM when idle.

Patches and workarounds that were needed:

1. Driver mismatch after installing utils. Installing nvidia-utils-595-server removed the 580 kernel modules. The fix was installing the matching nvidia-driver-595-server-open.
2. OCuLink eGPU isn't hot-pluggable. Power the dock before a cold boot, or the card isn't on the PCIe bus.
3. CUDA runtime for the llama.cpp CUDA build. The prebuilt binary needs libcudart.so.12, libcublas.so.12 and libcublasLt.so.12. These were copied from NVIDIA's cudart and cuBLAS redistributables into the binary directory, and units set LD_LIBRARY_PATH to it. - Copying the libraries while the tarball was still extracting caused a bus error. Copy only after extraction completes.
4. Driver 595 ships CUDA 13.x, but the llama.cpp build is CUDA 12.8. It works through the driver's forward compatibility, and Blackwell (sm_120) is supported.
5. Vulkan picks the wrong GPU when the eGPU is attached. Pin the Radeon by pointing VK_DRIVER_FILES at the RADV ICD file. On 26.04 it's named radeon_icd.json, not radeon_icd.x86_64.json.
6. ROCm/HIP vs Vulkan: Vulkan (RADV) is faster than ROCm on the 890M (gfx1150), as reported in llama.cpp #20099, so ROCm isn't installed.
7. Playwright refuses Ubuntu 26.04. Install the browsers with: PLAYWRIGHT_HOST_PLATFORM_OVERRIDE=ubuntu24.04-x64 npx playwright install chromium-headless-shell chromium Screenshots and UI audits depend on this.
8. Running sudo from an AI agent: sudo can't prompt for a password through the agent's shell. Run privileged steps in a separate SSH session: - the driver - the CUDA repo and toolkit - libvulkan-dev and glslc (needed for Colibri's Vulkan build) - linger
9. The NPU needs one root step. Install Ubuntu's libxrt2/libxrt-npu2 plus FastFlowLM's .deb, and raise memlock to unlimited (see the NPU section). No PPA or DKMS on Ubuntu 26.04 with kernel 7.0.
10. Colibri's start-here.sh auto-starts a server on :8000 after downloading. Stop it before starting the managed llm-deep unit.

Build and ops gotchas#

• Two llama.cpp builds live side by side:
• Vulkan, for AMD.
• CUDA 12.8, for NVIDIA, with the cudart libraries copied in.
• Bus error in the CUDA build: caused by copying the cudart libraries while the tarball was still extracting. Copy only after extraction finishes.
• Vulkan device choice: with the eGPU attached, Vulkan must be pinned to the Radeon with VK_DRIVER_FILES (the RADV ICD file).
• OCuLink eGPU isn't hot-pluggable: cold-boot with the dock already powered.
• MCP server edits: a syntax error in the MCP server once made /mcp reconnect fail silently. A smoke test (handshake plus tools/list) now runs before every harness commit.