UP | HOME

Local models on straylight

Document history

  • [2026-10-02 Fri] llama.cpp gained /v1/systemone (PR 29818, merged 2026-10-02). Decision models now ship as GGUF (Laya, Julia-1, Kev-4B, lev, OpenJev). titan Homebrew 0.5.0 (build 11146, 7fe450e19) does not include it. Live from titan (HTTP): straylight:11434 and dixie:11434 both build_info b10964-b29c606, POST /v1/systemone 404. flake nixpkgs-unstable lock is 2026-09-22 — bump must land a lock on or after 2026-10-02 (merge a4cb4c61), not merely "past 0.5.0" (titan 0.5.0/11146 has no /v1/systemone). Python sidecar remains the only serve path until that lands.
  • [2026-09-20 Sun] Honcho split-plane deployed: rift runs the Honcho stack (podman: api/deriver/pgvector/redis, api on 100.95.252.120:8000 tailnet-only); straylight runs CPU-only llama lanes honcho-gen (:11436, ornith-1.5-9b, 24k ctx, user unit) and honcho-embed (:11437, Qwen3-Embedding-0.6B Q80, 1024-dim) — both bound to the tailscale IP. Hermes (straylight) memory provider honcho, workspace hermes-fleet. rift CPU benched 1.7 t/s (unusable), straylight CPU 10.2 t/s — memory tier inference stays on straylight CPU.
  • [2026-09-21 Mon] Grok cloud default bumped to grok-4.7 on straylight, talos, and titan (released 2026-09-21; same $2/$6 pricing as 4.6, new larger base model).
  • [2026-09-16 Wed] Ornith 1.5 9B does vision once loaded (mmproj-Q80)
  • [2026-09-16 Wed] no note-RAG: run the agent in ~/org instead of embeddings
  • [2026-09-16 Wed] whisper large-v3-turbo ASR; Qwen3.8-Flash-Next uncensored
  • [2026-09-16 Wed] reconciled with the live router and OpenClaw: Ornith 1.5 35B and Qwen3.8 27B take images; Tailscale HTTPS on :10000
  • [2026-09-19 Sat] OpenClaw retired; sections rewritten for Hermes (straylight + talos). Added Hermes subagent delegation and auxiliary routing; added decision-model (Jev/Laya) support notes.

What this is

straylight runs two local inference servers, both generated from hw/straylight/default.nix:

llama.cpp llama-server
router mode on 0.0.0.0:11434. OpenAI-compatible chat. Models load on demand; at most two resident (--models-max 2). Vulkan backend, gfx1151 (Strix Halo). Weights under /var/lib/llama-server/.
whisper.cpp whisper-server
ASR on 0.0.0.0:11435, POST /v1/audio/transcriptions. CPU only (--no-gpu) so it does not take GTT from llama-server. Model: /var/lib/whisper-server/models/ggml-large-v3-turbo.bin (~1.6 GB, fetched on first start).

Ports 11434 and 11435 are open only on tailscale0.

Tailscale Serve also fronts the chat router with TLS at https://straylight.scylla-hammerhead.ts.net:10000/v1 (→ 127.0.0.1:11434). Port 443 on that name is Mattermost (→ :8065), not llama. Neither mapping is managed by Nix; both live in Tailscale's own serve state (see Operational notes).

Clients:

Grok Build TUI
[model.*] blocks in ~/.grok/config.toml, base_url = "http://127.0.0.1:11434/v1" on straylight (all nine ids). On talos the blocks are straylight-* aliases against http://straylight:11434/v1.
OpenCode
managed config in /etc/opencode/opencode.json (NixOS configs/opencode.nix), provider llama.cpp, endpoint http://straylight:11434/v1
Open WebUI
http://127.0.0.1:8080; chat against llama.cpp, STT against whisper (AUDIO_STT_OPENAI_API_BASE_URL)
Hermes
gateway-driven, from both straylight (loopback) and talos (tailnet). See Hermes agents below.
Other tailnet harnesses
http://straylight:11434/v1 (chat), http://straylight:11435/v1 (transcriptions)

Dummy API key if the client insists: none.

Helper commands: llama-models (what is loaded), llama-unload (free GTT before games or before Flash-Next), whisper-transcribe file.wav (one-shot CLI).

How to pick a model

Default coding agent is Laguna S 2.1. Use a smaller model when you want faster turnaround or to leave room for a second resident model. Use a VLM when the task actually needs pixels. Flash-Next is exclusive with Laguna: unload one before loading the other.

Id Kind Size class Use when
laguna-s-2.1 MoE coding ~64 GiB Q4 primary local coding / agent work
qwen3.8-27b-uncensored dense VLM, abliterated ~20 GiB Q4 + 889 MiB mmproj uncensored coding without the Laguna footprint; images
ornith-1.5-35b-a3b MoE VLM coding ~21 GiB Q4 + 862 MiB mmproj official Ornith 1.5 agentic coding (~3B active); Hermes subagent default
ornith-1.0-35b MoE coding, text ~20 GiB Q4 comparison against 1.5; previous default
qwen3.6-35b-a3b-abliterated MoE, abliterated, text ~20 GiB Q4 uncensored 35B-A3B benchmark (HauhauCS Aggressive)
gemma4-26b-a4b-abliterated MoE, abliterated + MTP, text ~16 GiB Q4 + 240 MiB MTP Western-side MoE benchmark; speculative decode
ornith-1.5-9b-uncensored dense VLM, abliterated ~5.3 GiB Q4 + 596 MiB mmproj small/fast local work and images
qwen3-vl-8b-abliterated VLM, abliterated ~4.7 GiB Q4 + 1.1 GiB mmproj dedicated vision specialist (screenshots, OCR)
qwen3.8-flash-next-uncensored MoE VLM, abliterated ~98 GB IQ4XS (not fetched yet) newer Qwen4-preview (~6B active); unload Laguna first

Grok-hosted grok-4.7 is the default in this TUI when cloud is acceptable (bumped from 4.6 on 2026-09-21): larger context, search, image generation.

The unloaded catalog (/v1/models) is not a reliable vision signal. It shows [text, image] for the 9B and Qwen3.8 27B, but text for Qwen3-VL and Flash-Next, whose projectors come from mmproj-url, and for Ornith 1.5 35B, which does see images. Use /props?model=<id> on a loaded model (modalities.vision).

Laguna S 2.1 (laguna-s-2.1)

Poolside Laguna S 2.1, 118B total / 8B active, UD-Q4KS shards on disk. Parallel 1, 128K ctx. This is the local coding default: tool-using agent work, refactors, tests. It is the heaviest resident; loading it next to another large model will fight the two-model cap and GTT. Unload it before games (llama-unload laguna-s-2.1).

Qwen 3.8 27B uncensored (qwen3.8-27b-uncensored)

huihui-ai GGUF Q4KL of Qwen3.8-27B. Dense, refusal-ablated; MTP and the vision tower were left intact. --hf-repo pulled mmproj-model-bf16.gguf (889 MiB) into the HF cache next to the weights, and the router catalog reports [text, image], so as served it takes images (same mechanism as the Ornith 9B). Not yet load-tested with an image.

Prefer this when you want an uncensored local coder smaller than Laguna and denser than the 35B MoEs. Weights are already cached.

Ornith 1.5 35B-A3B (ornith-1.5-35b-a3b)

Official ornith-ai GGUF Q4KM. ~3B active, 262K native context in the architecture (we split 262K / 2 slots → 131K each). Agentic coding (SWE-bench / Terminal-Bench class).

It does take images: --hf-repo fetched mmproj-Ornith-1.5-35B-BF16.gguf (862 MiB), and once loaded /props reports vision: true, video: true. Checked 2026-09-16 with a test PNG (read the colour and text correctly). The unloaded catalog still says text. For dedicated screenshot/OCR work qwen3-vl-8b-abliterated remains the delegated vision model.

Hermes subagent primary (delegation.model). Hermes will send it images only if input is declared; treat as text-first until a vision send is verified through this harness.

Ornith 1.0 35B (ornith-1.0-35b)

deepreinforce-ai GGUF Q4KM (~20 GiB, 34.7B params). Previous default (Qwen 3.5 derivative), kept for comparison runs. Text only: no projector in the repo cache, /props reports vision: false. Use only when you need the old baseline.

Qwen3.6 35B-A3B abliterated (qwen3.6-35b-a3b-abliterated)

HauhauCS "Aggressive" Q4KM on disk. Benchmark candidate A against Gemma 4. Uncensored 35B MoE for general / coding chat. An mmproj was downloaded next to the weights but is not in the preset.

Gemma 4 26B-A4B abliterated (gemma4-26b-a4b-abliterated)

HauhauCS "Balanced" QAT plus MTP draft head (spec-type = draft-mtp). Benchmark candidate B: Western MoE of the same class as Qwen3.6-35B-A3B. Use when you want Gemma's flavour or the MTP speedup. mmproj is on disk, not wired.

Ornith 1.5 9B uncensored (ornith-1.5-9b-uncensored)

junafinity / mradermacher Q4KM plus mmproj-Q8_0. Dense 9B that does take images: --hf-repo pulls the projector into the HF cache (~5.3 GiB weights, ~596 MiB mmproj) and the router reports input_modalities = [text, image].

Use it as the small local model — OpenCode's small_model, cheap chat, and light vision (screenshots, photos) when you do not want to spin up the dedicated 8B VL. For heavier visual analysis, still prefer qwen3-vl-8b-abliterated.

Qwen3.8-Flash-Next uncensored (qwen3.8-flash-next-uncensored)

orcarouter abliteration of Qwen3.8-Flash-Next, cygnal IQ4XS-NGQ4 GGUF (~98 GB) measured on this class of box (gfx1151 Vulkan, ~21 tok/s). ~177B stored / ~6B active, native vision (mmproj pinned). Architecture is qwen4exp (in llama.cpp 0.4.0). The HF cache entry exists but holds no GGUF yet: it has never been loaded. Weights plus KV fill most of the 104 GiB TTM window: llama-unload Laguna (and anything else) before loading, 32k context / one slot.

This is the "newer Qwen3.8" — not a bigger 27B. Use it when you want the Flash-Next generalist; keep qwen3.8-27b-uncensored when you want a dense uncensored coder that can sit beside a VLM.

Qwen3-VL 8B abliterated (qwen3-vl-8b-abliterated)

prithivMLmods v2 / mradermacher Q4KM plus an explicit mmproj-url (the Nix comment says --hf-repo alone did not attach this repo's projector). ~4.7 GiB weights + 1.1 GiB mmproj-f16, both cached. This is the local eyes: screenshots, photos of hardware, diagrams, OCR. OpenCode subagent vision and Grok agent vision both target it.

How agents should delegate vision

Text models must not invent pixels. If the task is "what is in this screenshot", spawn vision.

OpenCode
primary agents invoke the vision subagent, or @vision in the prompt. Model id llama.cpp/qwen3-vl-8b-abliterated.
Grok Build
agent ~/.grok/agents/vision.md, skill delegate-vision. Spawn with subagent_type = "vision" and absolute image paths. Cloud Grok can see images itself; still delegate when the session is on a local text model or the user asked for the local VLM.

Pass filesystem paths, not remote URLs the VLM cannot fetch. The weights and projector are already in the HF cache, so the first call only pays load time.

Hermes agents

Hermes (Nous Research) runs gateway-driven on both hosts and talks to the router as an OpenAI-compatible custom endpoint.

straylight
the primary instance. Gateway as a systemd user service (hermes-gateway, Mattermost transport). Model routing in ~/.hermes/config.yaml.
talos
same gateway setup; reaches the router over the tailnet at http://straylight:11434/v1. Config mirrors straylight's except for the base URL. (Tailscale Serve on :10000 is the TLS alternative; plain :11434 over tailscale0 is what is actually configured.)

Routing layers in config.yaml, all pointing at the router:

delegation
model used for spawned subagents. base_url http://straylight:11434/v1 on talos (loopback on straylight), model ornith-1.5-35b-a3b. Subagents inherit Hermes toolsets (terminal/file/patch/web/…) so the 35B gets exercised as a tool-using agent, not just chat.
auxiliary.vision
image analysis tasks. qwen3-vl-8b-abliterated — matches the OpenCode vision delegation rule.
auxiliary.web_extract
page-extraction summarization. ornith-1.5-9b-uncensored — cheap, fast, good enough for cleanup summarization.

Operational cautions:

  • Subagent spawns that trigger a cold model load pay minutes, not milliseconds. Fine for background work; wrong for interactive loops.
  • The two-resident cap applies. A subagent load can evict the model another session is using.
  • Ornith 1.5 emits <think> blocks; Hermes does not strip them into reasoning_content for custom endpoints. Expect them in transcripts until a reasoning parser or extrabody shim is set up.
  • Tool-calling quality of each router model inside the Hermes harness is verified only for models we have actually run subagents on. The 35B was chosen on OpenCode history and paper class; if schema fumbling shows up, try ornith-1.0-35b or qwen3.8-27b-uncensored as delegation.model.

Decision-model support (Jev / Laya)

Separate from the chat models above: non-autoregressive decision models (typed question → calibrated probability, one forward pass). Context and experiments live in the Obsidian vault (Decision Models, Laya). Relevant to this machine because they are small encoders (300–450M) that run on CPU or the iGPU and are cheap to fine-tune.

Current state:

  • Laya (Apache-2.0, ModernBERT-large 421M) benchmarked on straylight CPU, 2026-09-19: 96 ms/question single, ~36 ms marginal batched. Reinstalled 2026-09-29 at laya 0.3.22 in ~/src/laya/.venv (py3.12, CPU torch): warm load 2.4 s, 5-question email decision 412 ms. Install CPU torch first (uv pip install torch --index-url https://download.pytorch.org/whl/cpu), then uv pip install "laya[serve]" — a naive pip install laya pulls a multi-GB CUDA wheel the iGPU can't use. 0.3.22 ships an invalid temperature entry (choice:11+ → clamped 0.5, uncalibrated).
  • GPU-only alternative (not on this box): OpenJev (Qwen3.5-27B sequence-classifier, CC-BY-NC-4.0) — vLLM originally; GGUF now at ggml-org/OpenJev-GGUF (vision yes) but license + 27B still skip. Revisit on the CUDA box only if NC is acceptable.
  • Planned harness use: a local decision model as Hermes' context-routing gate (the SoL-Pi ObservationPack pattern — pack/no-pack on oversized tool outputs). Nothing wired yet; plan documents at ~/observation-pack-plan.md and ~/observation-pack-opencode.md.
  • Serve path (2026-10-02): llama.cpp POST /v1/systemone (Jev- compatible) once the binary is newer than 0.5.0. First GGUF candidate: Laya-GGUF (ModernBERT 421M, Apache-2.0, replaces the Python sidecar). Quality pick: Kev-4B Q4KM ~2.8 GiB (Apache-2.0; vendor 12 ms is RTX PRO 6000, not this box). Julia-1 is 144M multilingual. titan brew 0.5.0 has no systemone string — do not add these to a models.ini until the binary is upgraded. Always-on serving stays on straylight; titan sleeps. Hermes community plugin jev-approvals already accepts a Jev-compatible HTTPS base URL (privacy vs TypeSafe). Nothing wired.

Speech (ASR)

Transcription is a separate process from the chat router.

Endpoint
http://127.0.0.1:11435/v1/audio/transcriptions (same path on straylight:11435 over the tailnet)
Model
whisper large-v3-turbo (multilingual, ~1.6 GB, CPU)
CLI
whisper-transcribe recording.wav (ffmpeg conversion is enabled on the server; mp3/m4a/etc. are fine)
curl
curl -sS -F file=@recording.wav -F model=whisper-1 \
  http://127.0.0.1:11435/v1/audio/transcriptions

Open WebUI uses this as its STT backend. No TTS and no live voice loop yet.

Notes vs coding sessions

Do not index ~/org into a vector store for agents running elsewhere. Retrieved notes get weak, unreliable weight next to the live tree, go stale, and mix essays with runbooks.

When the work is the notes, start Grok or OpenCode with cwd ~/org. Grep/read on org files is the retrieval path. Repo + Nix stay authoritative for what the machine is doing; org is for what I meant or already wrote.

Operational notes

  • Idle unload after six hours (sleep-idle-seconds = 21600).
  • KV cache is q80; flash-attn on; n-gpu-layers = 999; no-mmap.
  • Context: global 262144 split across parallel = 2 → 131072 per slot. Laguna overrides to 131072 / parallel 1. Flash-Next overrides to 32768 / parallel 1 so weights + KV fit in the 104 GiB TTM window.
  • BIOS UMA frame buffer should be ≥32 GB; TTM pageslimit is raised so Laguna (and Flash-Next) can sit in GTT (needs a reboot after that knob changes).
  • Flash-Next first load fetches ~98 GB + mmproj into the llama-server HF cache. llama-unload everything else first.
  • Tailscale Serve is not in Nix. The old tailscale-serve-llama unit re-applied --https=443 → 127.0.0.1:11434 every 5 s and fought Mattermost for :443; it was removed from hw/straylight/default.nix on 2026-09-16. Until the next rebuild-nixos the running system still has it (stopped, but enabled), so a reboot before then would take :443 from Mattermost. A tailscale serve reset drops the :10000 mapping; re-add it by hand (tailscale serve --bg --https=10000 http://127.0.0.1:11434).
  • Rebuilds restart llama-server (and leave whisper running unless its unit changed). Deploy with rebuild-nixos from committed master.

What is missing

Gaps I actually hit, not a shopping list.

  • Working vision on the benchmark MoEs. Gemma 4 and Qwen3.6 have projectors on disk next to their weights that the preset does not attach. Ornith 1.5 35B, Qwen3.8 27B, Ornith 1.5 9B, the 8B VL, and (once fetched) Flash-Next take images.
  • Video. Loaded Ornith 1.5 35B reports video: true in /props, but no client here sends video and it has not been tried. Treat the router as image+text.
  • TTS / live voice. ASR is done (whisper on :11435). No local text-to-speech and no always-on mic loop.
  • Local image generation. Grok Imagine is cloud-only. No local SD/FLUX endpoint.
  • Embeddings / rerank. Not serving them. Note retrieval is "run the agent in ~/org", not a side-channel RAG API. A dedicated embed/rerank process is still possible later if some other client wants /v1/embeddings; it is not justified for Grok or OpenCode.
  • Verified tool-calling quality per model. Laguna is the intended coder; the others are available, not equally proven in this harness.
  • More than two residents. models-max = 2. A coding model plus the VLM is the intended pairing; a third request waits or evicts.
  • Reasoning parsers. Ornith 1.5 emits <think> by default; llama.cpp / the harnesses do not all split that into reasoning_content.
  • Uncensored Ornith 1.5 35B. The 35B MoE on the router is the official ( censored ) GGUF. The 9B is the uncensored Ornith.

Pointers

Preset
src/nixos/hw/straylight/default.nix
OpenCode
src/nixos/configs/opencode.nix → /etc/opencode/opencode.json
Grok models
~/.grok/config.toml
Grok vision agent
~/.grok/agents/vision.md
Grok skill
~/.grok/skills/delegate-vision/SKILL.md
Hermes config
~/.hermes/config.yaml (delegation, auxiliary)
Whisper unit
whisper-server.service (model under /var/lib/whisper-server/models/)