.nornir/local-llm-setup.md · nordisk/nornir

Local-LLM setup — the idiot-proof guide

Run nornir from a local, on-prem LLM on this box — an RTX 4090 (24 GB) — with Claude Code / Kilo / Cline as the harness and nornir's MCP server as the ground-truth tool provider. Every step is copy-paste, with the output you should see.

This is the setup guide. For what to ask the model once it's running, see airgapped-agent.md; for the MCP wiring details / relay mode, see mcp-from-claude-code.md. For what nornir does and does not support today for on-prem use, jump to §7 What nornir supports today.

0. What you're building

┌─────────────┐     OpenAI-compatible      ┌──────────────────┐
│ Claude Code │ ───────  HTTP  ──────────▶ │ local LLM server │  (Ollama / vLLM /
│ Kilo / Cline│   (base URL :11434 etc.)   │  on the 4090     │   LM Studio / llama.cpp)
└──────┬──────┘                            └──────────────────┘
       │ stdio (MCP)
       ▼
┌──────────────┐   in-process or gRPC relay   ┌────────────────┐
│  nornir-mcp  │ ───────────────────────────▶ │ nornir warehouse│  (deps, calls,
│ (56 tools)   │                              │   (Iceberg)     │   symbols, vectors)
└──────────────┘                              └────────────────┘

The LLM does the reasoning; nornir's 56 MCP tools hand it exact facts (dependency edges, call graphs, symbols, semantic search) so it never has to hallucinate them from raw source.

1. Pick a model

Ranked for agentic Rust + tool-calling on a single 24 GB GPU (mid-2026). The default is the safe choice; pick another only if you have a reason.

Rank	Model	Size / active	Fits 24 GB?	SWE-bench Verified	Why
1 (default)	Qwen3-Coder-30B-A3B	30B MoE / 3B active	Yes (Q4 ≈ 18 GB)	~67–69.6% (OpenHands)	MoE → only 3B active, so 73–196 tok/s on a 4090; strong, reliable tool-calling. Best all-round local coder.
2	Devstral Small 2	24B dense	Yes (Q4 ≈ 14–15 GB)	~68%	All-Hands agent-scaffold model, Apache-2.0, built for agent loops. Slower than the MoE but rock-solid tool use.
3	Gemma 4 31B	31B dense	Tight (Q4 ≈ 18–19 GB)	— (86.4% τ²-bench tool-use)	Best raw tool-calling accuracy if your workload is tool-heavy over coding-heavy.
—	GLM-5 / DeepSeek V4 / Kimi K2.x	100B+	No (needs ≫24 GB)	higher	Enterprise / multi-GPU only. Out of scope for one 4090.

Rule of thumb: start with Qwen3-Coder-30B-A3B at Q4. If tool calls misfire, try Devstral Small 2. If you need maximum tool-use precision, try Gemma 4 31B.

VRAM note: "Q4" = 4-bit quantized GGUF. A 4090 has 24 GB; leave ~3–4 GB headroom for the KV cache (context). All three defaults above fit at Q4 with a useful context window. Don't try the FP16 weights — they won't fit.

Sources: toolhalla.ai — best local LLMs RTX 4090 2026, kilo.ai — open-source models, artificialanalysis.ai — Devstral vs Qwen3-Coder, mindstudio.ai — best open-source LLMs for agentic coding 2026, morphllm.com — best open-source coding model 2026.

2. Download + run the model

Pick one runner. Ollama is the easiest; vLLM is the fastest for serving.

Option A — Ollama (easiest, recommended to start)

# Install (Linux):
curl -fsSL https://ollama.com/install.sh | sh

# Pull + run the default model (downloads ~18 GB once):
ollama run qwen3-coder:30b

Expected: a >>> prompt after the download. Type /bye to exit — the model stays loaded and Ollama keeps serving on http://localhost:11434.

Check it's serving:

curl -s http://localhost:11434/v1/models | head
# → {"object":"list","data":[{"id":"qwen3-coder:30b", ...}]}

Ollama exposes an OpenAI-compatible API at http://localhost:11434/v1. That's the base URL your client will use in §3.

Option B — LM Studio (GUI, good for trying models)

Install LM Studio, open the Search tab.
Search Qwen3-Coder-30B → pick a Q4_K_M GGUF → Download.
Go to Local Server → Load the model → Start Server.
It serves an OpenAI-compatible API at http://localhost:1234/v1.

curl -s http://localhost:1234/v1/models | head   # confirm it answers

Option C — llama.cpp (max control, raw GGUF)

# Build (one-time), then download a Q4 GGUF (e.g. from Hugging Face) and serve:
llama-server \
  -m ~/models/qwen3-coder-30b-a3b-Q4_K_M.gguf \
  -ngl 99 \                # offload all layers to the 4090
  -c 32768 \               # 32K context (raise/lower to fit VRAM)
  --host 0.0.0.0 --port 8080

Serves OpenAI-compatible at http://localhost:8080/v1. -ngl 99 puts every layer on the GPU; drop it if you run out of VRAM.

Option D — vLLM (fastest serving; needs CUDA toolchain)

pip install vllm
vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct \
  --quantization awq \           # or a Q4 GGUF path
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --port 8000

Serves OpenAI-compatible at http://localhost:8000/v1. Best tokens/sec and concurrency, but the heaviest to install.

Whichever you picked, note the base URL — you need it in §3:

Runner	Base URL
Ollama	`http://localhost:11434/v1`
LM Studio	`http://localhost:1234/v1`
llama.cpp	`http://localhost:8080/v1`
vLLM	`http://localhost:8000/v1`

3. Point your client at the local endpoint

The client talks to the local server over its OpenAI-compatible base URL.

Cline / Kilo / Roo (VS Code extensions)

In the extension settings, choose provider "OpenAI Compatible" and set:

Base URL: the URL from §2 (e.g. http://localhost:11434/v1)
API Key: any non-empty string (local servers ignore it — use local)
Model ID: the exact id from /v1/models (e.g. qwen3-coder:30b)

Claude Code

Claude Code talks to an Anthropic-style endpoint. Point it at the local server via env vars (works against Ollama / LM Studio / llama.cpp / vLLM through their OpenAI-compatible surface or an Anthropic-compat shim):

export ANTHROPIC_BASE_URL=http://localhost:11434     # your runner's host:port
export ANTHROPIC_API_KEY=local                       # any non-empty value
export ANTHROPIC_MODEL=qwen3-coder:30b               # the local model id
claude

If your runner only speaks the OpenAI dialect and your Claude Code build won't accept it directly, put a tiny OpenAI→Anthropic proxy in front (e.g. litellm --model ollama/qwen3-coder:30b) and point ANTHROPIC_BASE_URL at the proxy. The MCP wiring in §4 is identical either way — MCP is independent of which model is behind the chat endpoint.

Sanity check: open the client, ask "say hi in one word". You should get a local, instant reply (no network egress).

4. Wire nornir's MCP server into the client

nornir-mcp is a stdio MCP server exposing 56 tools. Two modes:

Embedded (default): run it in a directory with a nornir.toml; it opens the local warehouse in-process. Use this on the box that owns the warehouse.
Relay: set NORNIR_SERVER + NORNIR_SERVER_TOKEN and it forwards every tool to a running nornir-server over gRPC (bearer auth). Use this when the warehouse lives on another host. See mcp-from-claude-code.md.

4a. Build the binary

cargo install nornir --features cli        # gives `nornir`, `nornir-server`, `nornir-mcp`
which nornir-mcp                            # → ~/.cargo/bin/nornir-mcp  (must be on PATH)

4b. Register it (embedded mode, local warehouse)

Claude Code:

claude mcp add nornir -- nornir-mcp
# run `claude` from a directory containing nornir.toml so it finds the warehouse

Cline / Kilo (.mcp.json or the extension's MCP settings):

{
  "mcpServers": {
    "nornir": {
      "command": "nornir-mcp",
      "env": {
        "NORNIR_CONFIG": "/abs/path/to/nornir.toml"
      }
    }
  }
}

NORNIR_CONFIG is optional — without it, nornir-mcp discovers nornir.toml upward from its working directory.

4c. Register it (relay mode, remote warehouse)

{
  "mcpServers": {
    "nornir": {
      "command": "nornir-mcp",
      "env": {
        "NORNIR_SERVER": "http://oden:7878",
        "NORNIR_SERVER_TOKEN": "<shared bearer>",
        "NORNIR_WORKSPACE": "njord"
      }
    }
  }
}

Get the token on the host: sudo grep NORNIR_SERVER_TOKEN /etc/nornir/nornir.env.

5. Verify the tools are visible

Server starts. nornir-mcp logs to stderr on launch:

starting nornir-mcp; config=/abs/path/to/nornir.toml

(Logging level via RUST_LOG, default nornir_mcp=info.)

Tools listed in the client. Tools appear namespaced as mcp__nornir__*. In Claude Code, run /mcp — you should see nornir connected with 56 tools.
A tool returns real data. Ask the model:

"use the nornir MCP: repo_overview for <repo>"

You should get back internal depends_on/dependents, build-order position, and a symbol/call digest — not a guess. Other quick checks: deps_of, build_order, knowledge_symbol_lookup, search, vector_search (semantic; needs an embed build + a vector index — see §6).

Automated check (when H1 lands). A test asserts every advertised MCP tool is registered and answers:

cargo test --features mcp --test mcp_tools

Until H1 merges, the manual /mcp + repo_overview checks above are the gate.

6. (Optional) Enable semantic search / embeddings on the 4090

vector_search needs embeddings built. nornir embeds with jina-embeddings-v2-base-code (768-dim) via one of two backends — pick at build time:

Feature	Runtime	Hardware	Notes
`embed-tract`	tract-onnx	CPU, pure Rust	Slowest; zero GPU/C-runtime deps beyond the tokenizer.
`embed-ort`	ONNX Runtime (`ort`)	NVIDIA CUDA → CPU fallback	~110× CPU on a 4090. Links onnxruntime C++; CUDA libs auto-`dlopen`ed.

On the 4090, build with embed-ort for the GPU path:

cargo install nornir --features cli,embed-ort

The model weights (ONNX + tokenizer) are fetched once by build.rs from Hugging Face into ~/.cache/nornir/models/... (set NORNIR_SKIP_MODEL_FETCH=1 to reuse existing files; NORNIR_MODEL_DIR / /opt/nornir/models to relocate).

Check the GPU path is healthy, then index a repo:

nornir vector doctor          # CUDA preflight: driver / cudart / cuDNN verdict
nornir vector index <repo>    # embed + materialize into the warehouse
nornir vector search <repo> "the booking flow" --mode semantic

If vector doctor says CPU fallback, see Troubleshooting (CUDA) below.

7. What nornir supports today — vs what's missing

Audited against the real source (src/bin/nornir-mcp.rs, src/vector/, Cargo.toml), 2026-06-12.

Supported today

MCP server, 56 tools over stdio (nornir-mcp, feature mcp, built on rmcp). Embedded against a local warehouse, or gRPC-relay to a remote nornir-server when NORNIR_SERVER is set.
The ground-truth tool surface a local model needs: dependency Mímir (deps_of / dependents_of / build_order / dep_path / external_dep_users / affected_by_change / dep_graph_svg), call graphs from syn (knowledge_callers / callees / call_path / symbol_lookup / defined_in) and from DWARF (dwarf_*, SHA-time-travel), full-text BM25 search, semantic vector_search (hybrid = RRF of both), repo_overview for one-call orientation, plus release/bench/docs/funnel tools.
Two embedding backends, one interface (vector::Embedder): embed-tract (CPU, pure Rust) and embed-ort (GPU/CPU). The cuda.rs module auto-dlopens the CUDA runtime libs so the GPU EP "just works" without a manual LD_LIBRARY_PATH; nornir vector doctor preflights it.
Per-call telemetry to the warehouse (mcp_requests), best-effort and non-blocking.
Auth in relay mode: bearer token (NORNIR_SERVER_TOKEN) + per-workspace scoping (NORNIR_WORKSPACE / the workspace_use tool).

Missing / limited (the on-prem gaps)

GPU embeddings are NVIDIA-CUDA only. embed-ort requests the CUDA execution provider; there is no AMD ROCm / Vulkan / Metal path — non-NVIDIA GPUs silently fall back to CPU. (Plan item G1: add a ROCm/portable GPU embed backend.)
One fixed embedding model. jina-embeddings-v2-base-code (768-dim) is hard-wired in embed_support.rs (MODEL_NAME, DIM); there is no flag to choose a different embedder. (Plan item G3: configurable embedding model.)
MCP transport is stdio-only. No HTTP / SSE / streamable-HTTP MCP transport; remote use goes through the gRPC relay (NORNIR_SERVER), not a network MCP listener. (Plan items G2 / server-transport: native HTTP MCP.)
The chat LLM is bring-your-own. nornir does not run or bundle a generative model — it's purely the tool provider. You stand up Ollama / vLLM / etc. yourself (this guide). nornir only ships the embedding model fetch.
No auth in embedded mode. Embedded nornir-mcp trusts its local caller; the bearer/workspace auth only applies to the gRPC relay path. (Cross-ref H5: harden local MCP auth.)
Embedding weights need one online fetch. build.rs pulls the ONNX + tokenizer from Hugging Face on first build. Fully airgapped builds must pre-seed the cache and set NORNIR_SKIP_MODEL_FETCH=1.

8. Troubleshooting

Model won't load / out of VRAM (CUDA out of memory, OOM). Use a smaller quant or a smaller model: drop FP16→Q4 (qwen3-coder:30b is already Q4 in Ollama), lower the context (-c 16384 in llama.cpp, --max-model-len 16384 in vLLM), or step down to Devstral Small 2 (24B, ~14–15 GB at Q4). On a 4090 keep ~3–4 GB free for the KV cache.

Tool calls never fire / the model ignores the MCP tools. This is almost always a model-choice problem, not a nornir problem. Use a model with first-class tool-calling: Qwen3-Coder-30B-A3B, Devstral Small 2, or Gemma 4 31B (best tool-use score). Avoid base/instruct models not tuned for function calling. Also confirm /mcp shows nornir connected (§5).

nornir MCP shows 0 tools / fails to start. Check the stderr line starting nornir-mcp; config=.... In embedded mode it must find a nornir.toml (set NORNIR_CONFIG to an absolute path). In relay mode, NORNIR_SERVER and NORNIR_SERVER_TOKEN are both required (the latter errors loudly if missing). Bump logging with RUST_LOG=nornir_mcp=debug.

vector_search returns nothing. The repo has no embeddings yet — run nornir vector index <repo> (§6), and make sure the binary was built with embed-tract or embed-ort (a plain build has no embedder).

CUDA not found / vector doctor says CPU fallback. nornir dlopens the CUDA runtime libs itself; if it can't find them:

Point it explicitly: export NORNIR_CUDA_LIBS=/path/to/dir/with/libcudart.so.12 (the dir must hold a matched CUDA-12 + cuDNN-9 .so set).
Or pin a complete set into the built-in search dir: nornir vector setup-cuda copies them into /opt/nornir/cuda (the first dir discovery checks).
The NVIDIA kernel driver (libcuda.so.1) can't be bundled — install it for your distro (sudo apt install nvidia-driver-XXX, sudo dnf install akmod-nvidia, or NVIDIA's .run), which needs root + a matching kernel module + reboot.
Re-check with nornir vector doctor — it names exactly what's missing. See vector.md for the full CUDA discovery order.

Everything works but it's slow. The MoE default (Qwen3-Coder-30B-A3B) is the fastest quality option on a 4090 (3B active params → 73–196 tok/s). vLLM > llama.cpp > Ollama ≈ LM Studio for raw throughput. And remember: nornir tools return a few hundred bytes per call, so a tool-driven session needs far less generation than dumping files into context.