Running Local LLMs: Hardware, Models and Honest Expectations
A practitioner’s guide to running a local LLM in 2026: VRAM and bandwidth maths, MoE vs dense models, a real Ollama guide, and honest performance
A 24GB RTX 3090, bought used for roughly the price of a decent laptop, will run a 70B model at about 15 tokens per second. That sentence is true, and it is also the source of most of the disappointment people feel in week two. Fifteen tokens per second is slower than you read. The model is quantised to roughly 4.5 bits per weight, so it is measurably dumber than the thing you tested on an API. And the moment you paste in a 30,000 token file, you wait 40 seconds before the first token appears.
None of that means running a local LLM is a bad idea. We run several on our own infrastructure for classification, redaction and draft generation, and the economics are excellent once the workload is the right shape. But the gap between “it works” and “it is good enough to replace the API call you are currently paying for” is wider than the YouTube demos suggest, and it is almost always decided by two hardware numbers and one model architecture choice.
Here is the version we wish someone had written for us.
- Decode speed is bounded by memory bandwidth divided by the bytes you read per token. A 40GB model on a 936GB/s card tops out near 23 tok/s in theory and lands at 15 to 18 in practice. No software fixes this.
- Mixture of experts models changed the maths. A 30B model with 3B active parameters reads about a tenth of the weights per token, which is why a 30B MoE on a mid-range machine feels faster than an 8B dense model on the same hardware.
- VRAM budget is weights plus KV cache. Llama-class 70B at 8k context needs roughly 2.5GB of KV cache on top of ~42GB of weights, and quantising the cache to q8_0 halves that at near-zero quality cost.
- Prefill, not decode, is what kills RAG and long-context workflows. Measure prompt processing tokens per second separately, because it is the number that determines whether a 20k token prompt is usable.
- Ollama is the right default for a single developer. vLLM or llama.cpp’s server with continuous batching is the right answer the moment you have more than two concurrent users.
The only two hardware numbers that matter
Everything else is noise. Capacity decides whether the model loads at all; bandwidth decides how fast it generates.
Capacity. At Q4KM, the quantisation most people actually use, budget about 0.55GB per billion parameters once you account for embeddings and the higher-precision tensors that k-quants keep. So 8B lands near 4.7GB, 24B near 14GB, 32B near 19GB, 70B near 42GB. Then add the KV cache, which people consistently forget. For a 70B model with grouped query attention (80 layers, 8 KV heads, 128 head dimension), one token of context costs 2 × 80 × 8 × 128 × 2 bytes, or about 320KB. At 8k context that is 2.5GB. At 32k it is 10GB, and your 48GB of VRAM just stopped being enough.
Bandwidth. During decode, the GPU reads essentially the whole set of active weights for every single token. So your ceiling is bandwidth divided by model size. A 3090 does 936GB/s. A 4090 does about 1,008GB/s. A 5090 does roughly 1,792GB/s. An M4 Max sits near 546GB/s, an M3 Ultra near 819GB/s, and the 128GB unified-memory mini PCs built on AMD’s Ryzen AI Max platform land around 256GB/s. Dual-channel DDR5-5600 on a normal desktop? Roughly 90GB/s, which is why CPU-only inference on a 32B model gives you two tokens per second and a warm room.
Do the division before you buy anything. A 24GB model on 256GB/s is a theoretical 10 tok/s, and real-world efficiency is usually 60 to 75 percent of theoretical. That is the whole prediction.

What each hardware class actually buys you in 2026
Used 3090, 24GB. Still the best value per gigabyte of fast VRAM, and it supports the same CUDA paths as everything newer. Comfortable home for 24B to 32B dense models at Q4 with a 16k to 32k context, or a 30B-class MoE at genuinely pleasant speeds. Two of them on a board with enough PCIe lanes gets you 70B at usable quality. The caveat: power draw, heat, and the fact that you now own a space heater with a fan curve.
Apple Silicon with 48GB or more unified memory. The sleeper pick for developers, because the memory is unified and you can allocate 70 to 75 percent of it to the GPU. An M-series machine with 64GB will hold a 70B at Q4 that no single consumer NVIDIA card can. Decode is respectable. Prefill is not: Apple’s GPUs have far less raw compute than a 4090, so prompt processing on long inputs is where Macs lose badly. If your workflow is short prompts and long outputs, Macs are great. If it is long documents, they are frustrating.
Unified-memory x86 boxes (128GB class). These exist now and they do something useful: they hold very large MoE models that nothing else in the price bracket can. Bandwidth around 256GB/s means dense 70B is sluggish, but a 100B+ MoE with 5B active parameters runs fine. Match the box to the architecture.
Rented GPUs. Worth saying out loud: if your use case is bursty, an hourly A100 or H100 is cheaper than hardware you own and leave idle. Local wins on privacy, offline capability, fixed costs at high volume, and latency with no network hop. It does not automatically win on money.
Dense versus MoE, and which models earn disk space
The single biggest shift in the last two years is that mixture of experts models became good and became small enough to run at home. A MoE with 30B total parameters and 3B active reads only the shared weights plus two or three expert blocks per token. You pay the VRAM cost of 30B. You pay the bandwidth cost of roughly 3B. On a bandwidth-starved machine that is a 5x to 10x speedup over a dense model of the same size.
What we actually keep installed:
- A 30B-class MoE as the daily driver. Qwen3-30B-A3B and its coder variant are the obvious examples. Fast enough to feel interactive on a 3090 or a 36GB Mac, smart enough for refactors, commit messages, and structured extraction.
- A 24B to 32B dense model for anything needing care. Mistral Small and Gemma 3 27B class models still beat similarly sized MoEs on tasks that need tight instruction following. Slower. Worth it for the final pass.
- An 8B for throughput jobs. Classification, tagging, PII redaction, embedding-adjacent work. When you are processing 50,000 rows, an 8B at 90 tok/s with batching beats a 32B at 20 tok/s every time, and the accuracy difference on a well-specified classification prompt is often under two points.
- OpenAI’s open-weight gpt-oss models if you want strong tool calling without a frontier API. The 20B variant fits comfortably in 16GB.
On quantisation: Q4KM is the sweet spot and has been for years. Q5KM if you have the room. Q8 is almost never worth double the bandwidth. Below Q4, quality degrades in ways that are hard to see in a chat demo and very easy to see in a structured output benchmark, especially JSON schema adherence. We have watched a Q3 quant of a good model start emitting trailing commas. Do not ship that.
An Ollama guide that goes past ollama run
Ollama is the correct default. It handles model downloads, GPU offload, memory management and an OpenAI-compatible endpoint with no configuration. The defaults are tuned for “it works on a laptop”, not for your workstation, and nobody tells you which knobs matter.
The big one: the default context window is 4096 tokens. If you pipe in a 10,000 token file, the front gets silently truncated and the model confidently answers about the half it saw. Set it globally:
sudo systemctl edit ollama.service
[Service]
Serve on the LAN, not just loopback
Environment="OLLAMA_HOST=0.0.0.0:11434"
Default context for every model; override per-request if needed
Environment="OLLAMACONTEXTLENGTH=16384"
Required before KV cache quantisation will take effect
Environment="OLLAMAFLASHATTENTION=1"
Halves KV cache VRAM; quality impact is negligible at q8_0
Environment="OLLAMAKVCACHETYPE=q80"
Keep the model resident instead of unloading after 5 minutes
Environment="OLLAMAKEEPALIVE=-1"
Concurrent requests per model. Each one gets its own KV cache slice.
Environment="OLLAMANUMPARALLEL=2"
That OLLAMAKEEPALIVE=-1 line is the difference between a 400ms first token and a 12 second cold load every time you come back from lunch. Be careful with OLLAMANUMPARALLEL too: the context length is divided across parallel slots in some versions, so setting it to 4 with a 16k context can leave each request with 4k. Check with ollama ps, which shows actual VRAM use and whether the model is on GPU or has spilled to CPU.
Pin per-model settings in a Modelfile rather than repeating parameters in every call:
cat > Modelfile <<'EOF'
FROM qwen3:30b-a3b-q4KM
PARAMETER num_ctx 32768
PARAMETER temperature 0.2
PARAMETER repeat_penalty 1.05
SYSTEM """You are a code review assistant. Respond only with findings as JSON matching the supplied schema. No prose."""
EOF
ollama create reviewer -f Modelfile
ollama run reviewer
Then hit it from anything that speaks the OpenAI API. No SDK rewrite required:
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "reviewer",
"messages": [{"role":"user","content":"Review this diff: ..."}],
"responseformat": {"type":"jsonobject"},
"stream": false
}'
One honest limitation: Ollama hides llama.cpp’s flags from you. When you need exact control over tensor split across two GPUs, speculative decoding, or offloading specific MoE expert layers to CPU, drop to llama-server directly:
./llama-server -m ./qwen3-30b-a3b-q4km.gguf \
-c 32768 \
-ngl 99 \ # offload all layers to GPU
--flash-attn on \ # older builds use a bare -fa
-ctk q80 -ctv q80 \ # quantise K and V caches
-np 4 \ # 4 parallel slots, continuous batching
--host 0.0.0.0 --port 8080
Prefill is the number nobody benchmarks
Every benchmark you have read quotes decode speed. Decode is the easy part. Prompt processing (prefill) is compute-bound rather than bandwidth-bound, and it is what decides whether your retrieval pipeline is usable.
Concrete example from a client project: a 32B dense model on a single 4090 decoded at 28 tok/s, which felt fine in chat. The actual job was summarising support threads averaging 18,000 tokens. Prefill ran around 900 tok/s, so every request sat for 20 seconds before producing a word. The fix was not a faster GPU. It was cutting retrieved context from twelve chunks to four, which dropped time-to-first-token to under 6 seconds and, to our mild embarrassment, slightly improved answer quality.
Two things actually help here. Prefix caching, so a stable system prompt plus document preamble is processed once and reused across requests, is available in llama.cpp slots and in vLLM’s automatic prefix caching. An application-level response cache for repeated or near-repeated queries is the other lever, and we have written about it in detail in caching LLM responses without cutting quality. Both apply identically whether the model is local or remote.
When Ollama stops being the right answer
The moment you have real concurrency, Ollama’s limits show. Its parallelism works, but it is not a scheduler built for throughput. vLLM with paged attention and continuous batching will serve 20 simultaneous requests on one GPU at aggregate throughput several times higher, because it packs batches dynamically instead of reserving fixed KV slots.
The trade-off is operational weight. vLLM wants a specific CUDA and PyTorch combination, prefers unquantised or AWQ/GPTQ weights over GGUF, takes minutes to load a large model, and will OOM at startup if --gpu-memory-utilization is wrong. For one developer on one machine, that is pain for nothing. For an internal service behind a queue, it is the correct choice.
Whichever you pick, put a real queue in front of it. GPU inference serialises, so a burst of 50 requests against a naive HTTP handler produces 50 pending connections and timeouts everywhere. Queue the work, return a job ID, deliver results by webhook. The same durability rules apply as any other async pipeline: our notes on building a webhook receiver that does not lose events cover the retry and idempotency side.
Honest expectations
What a well-chosen 24B to 32B local model does genuinely well: structured extraction from documents, classification and tagging, rewriting and tone adjustment, summarisation, SQL generation against a known schema, single-file code edits, and anything where the data must not leave your network. On those tasks the gap to a frontier API is small enough that most users would not pick the winner blind.
What it does not do well, regardless of what you read: multi-step agentic work with ten tool calls where one malformed argument derails the run; reasoning over 100k tokens, where retrieval accuracy in the middle of the context collapses long before the advertised window ends; obscure library APIs it never memorised; and anything needing current knowledge. Local models also degrade less gracefully. A frontier model that is unsure often hedges. A 14B quant just asserts something wrong with perfect formatting.
The practical pattern that works is routing. The local model handles the 80 percent of calls that are cheap, private and well-specified; a frontier API handles the hard 20 percent. That split is the single highest-leverage architectural decision here, and it is the same shape we described in the developer AI toolchain. Build the router before you build the prompts.
Frequently Asked Questions
How much VRAM do I need to run a useful local LLM?
16GB is the realistic floor for work you will not immediately abandon, which gets you a 14B dense model or a 20B MoE at Q4 with a 16k context. 24GB is the sweet spot: 32B dense or 30B MoE with room for a 32k context. Below 12GB you are limited to 7B to 8B models, which are genuinely useful for classification and tagging but weak at code.
Is Ollama or LM Studio better for getting started?
LM Studio if you want a GUI, model browser and instant chat with zero terminal use. Ollama if you intend to call the model from code, because its server, Modelfiles and OpenAI-compatible endpoint are easier to script and deploy. Both wrap llama.cpp, so inference speed is near identical; the choice is purely about workflow.
Can I run a local LLM on a CPU only?
Yes, and it will be slow. Dual-channel DDR5 gives you roughly 90GB/s of bandwidth, so a 7B model at Q4 manages about 10 to 15 tokens per second and a 32B model drops to 2 or 3. Small MoE models are the exception worth trying, since they read far fewer weights per token, but plan for background batch jobs rather than interactive chat.
Does quantisation actually hurt quality?
At Q4KM and above, barely, and mostly on tasks needing precise formatting or long multi-step reasoning. Below Q4 the damage becomes obvious: JSON schema violations, repetition loops and degraded instruction following. The useful rule is that a larger model at Q4 almost always beats a smaller model at Q8 for the same memory budget.
Will running models locally save money versus an API?
Only at sustained volume or when privacy requirements make the API a non-starter. A used 3090 plus power costs real money, and at current API pricing for small models you would need millions of tokens per month to break even on the hardware alone. Run the actual arithmetic on your token volume before buying anything.
Do this: calculate your bandwidth-to-model-size ratio first, pick the quantised model that fits with its KV cache second, and benchmark prefill on a prompt the length of your real workload before you write any application code. If the numbers come out acceptable, Ollama with a tuned systemd override will have you serving in under an hour. If they do not, you have saved yourself a graphics card and a weekend.


