From selecting components to running qwen3.6 35b locally on an RTX 5090 — exposed as an API via llama.cpp and wired into OpenCode.Your data stays yours.
Large language models have become one of the most powerful tools available to software engineers, but the standard way of using them — sending your prompts through a cloud API — comes with tradeoffs that matter more every day. Privacy, cost, and availability are the three pillars that push people toward running models locally.
This project documents the complete process of building a machine purpose-designed for local AI inference. Starting from hardware selection, through assembly and OS setup, all the way to exposing a running model as an API endpoint that OpenCode can consume directly over your LAN.
The target setup runs qwen3.6 35b a3b — a 35-billion parameter mixture-of-experts model — quantized to Q4_K_M and fully offloaded onto a singleNVIDIA RTX 5090 with 32GB VRAM. The llama.cpp inference engine exposes it via an OpenAI-compatible API on port 8080, and OpenCode connects to it using a single config change. No cloud provider. No API keys. No rate limits.
Your code, your data, your prompts — never leave your network. Everything runs on hardware you own.
No subscription, no pay-per-use. After the initial hardware investment, inference is free forever.
Use the model as much as you want. No API caps, no throttling, no vendor lock-in.
Running a 35-billion parameter model locally demands serious hardware. Here's what you need — and why each component matters.
| Model | AMD Ryzen 9 9950X |
| Cores / Threads | 16 / 32 |
| Why it matters | Data preprocessing & OS tasks. 16 cores handle everything without breaking a sweat. |
| Price | ~€470 |
| Model | NVIDIA RTX 5090 |
| VRAM | 32GB GDDR7 |
| Why it matters | The single most important component. 32GB VRAM lets us load the full 35B model in quantized form with room for context. |
| Price | ~€4,075 |
| Capacity | 64GB DDR5-5600 |
| Configuration | 2 × 32GB DDR5 |
| Why it matters | OS, services, and fallback loading. DDR5-5600 ensures fast data transfer to the CPU. |
| Price | ~€943 |
| Model | Samsung 990 Pro 2TB |
| Type | NVMe PCIe 4.0 SSD |
| Why it matters | Model files are large. 2TB gives you room for multiple models, datasets, and the OS. |
| Price | ~€345 |
| Wattage | 1000W 80+ Gold |
| Standard | ATX 3.0 |
| Why it matters | The 5090 has transient power spikes. 1000W ATX 3.0 with native 12V-2x6 cable is non-negotiable. |
| Price | ~€131 |
| Type | Mid-tower, mesh front |
| Key feature | Supports ultra-wide GPUs (360mm+) |
| Why it matters | The 5090 is massive. You need a case with GPU support brackets and excellent airflow for sustained inference loads. |
| Price | ~€90 |
Prices sourced from Amazon.de (July 2025). GPU is the dominant cost at ~67% of total. Prices vary by brand, region, and availability.
Standard PC build process with a few notes specific to the 5090 and high-end builds.
Install the CPU, RAM, and M.2 SSD on the motherboard outside the case. This is significantly easier and reduces the risk of dropping anything inside the case. Update the BIOS to the latest version before installation — AMD's 9950X benefits from recent AGESA updates.
Mount the motherboard, route cables, and install the PSU. Leave extra room behind the motherboard tray — the 5090's power cable is thick and needs bending radius.
Use the PCIE1 slot (closest to CPU). The 5090 is approximately 350mm long — verify your case supports it. Connect the two or three 12V-2x6 connectors from the PSU. Do not use adapters — ATX 3.0 native cables are designed for the 5090's power delivery.
Power on and verify all components are detected in BIOS. Run a quick memory test (MemTest86) and confirm the SSD is recognized. Then proceed to install Ubuntu.
Ubuntu 24.04 LTS is the sweet spot — excellent NVIDIA driver support, mature tooling, and the default environment for most AI frameworks.
| OS | Ubuntu 24.04 LTS (Noble Numbat) |
| Kernel | 6.8+ (HWE kernel for latest hardware) |
| NVIDIA Driver | 570.x (proprietary, required for CUDA) |
| CUDA | 12.6 |
| Python | 3.12 (managed via uv) |
| Network | Static LAN IP recommended (e.g., 192.168.1.100) |
llama.cpp provides a drop-in OpenAI-compatible API server. One command loads the model and starts serving inference requests.
Point OpenCode at your local llama.cpp server. Same LAN, zero latency to the cloud, full privacy.
Every token stays on your machine. Nothing leaves your LAN.
No API rate limits, no network hops. Just your hardware, full speed.
No subscriptions, no vendor lock-in. Swap models whenever you want.