Want to run a local opensource LLM on your own PC? This guide (updated November 2026) lists the best open-weight models you can download today, the tools that run them, and exactly how much RAM, SSD and GPU VRAM each one needs. No cloud, no subscription, full privacy.
Why run a local opensource LLM?
- Privacy: your prompts, code and documents never leave your PC.
- No monthly fees: after the hardware, every token is free. Heavy users often recover the cost of a GPU within a year compared with paid API usage.
- Works offline: useful while travelling, in secure offices or on unreliable internet.
- Full control: pick the model, change system prompts, fine-tune on your own data and keep a version that never changes under you.
- No rate limits: run long batch jobs, agents and coding tools as hard as your hardware allows.
The trade-off is speed and peak quality. Cloud frontier models are still ahead on the hardest tasks, but the gap has narrowed sharply, and for everyday writing, coding, summarising and document Q&A a good local model is now more than enough.
Quick answer: what hardware do you need?
For any local opensource LLM, model size and quantization decide everything. Most people run models in 4-bit (Q4) format, which cuts memory use by roughly 4× versus full precision with little quality loss.
| Your hardware | What runs well (Q4) | System RAM | Free SSD |
|---|---|---|---|
| CPU only / 8 GB VRAM | 2B–9B models | 16 GB | 20–50 GB |
| 12–16 GB VRAM | 9B–24B models, gpt-oss-20b | 32 GB | 50–100 GB |
| 24–32 GB VRAM | 27B–35B dense, 30B-class MoE | 32–64 GB | 100–200 GB |
| 64–128 GB (Mac unified memory or multi-GPU) | 70B–125B models | 64–128 GB | 200–500 GB |
| Multi-GPU server | 280B–1T+ MoE flagships | 256 GB+ | 1 TB+ |
Tip: add 2–6 GB on top of the model file size for the context window (KV cache). Longer chats need more.
What changed for local opensource LLM users in 2026
- Mixture-of-Experts (MoE) went mainstream. Models like gpt-oss-20b, Gemma 4 26B A4B and Nemotron 3.5 only activate a few billion parameters per token, so they run fast on consumer GPUs.
- The 27B class is the new sweet spot. Qwen3.6-27B and Qwen3.8-27B fit on a single 24 GB card and rival last year’s 70B models.
- Licences got friendlier. Gemma 4, gpt-oss, Qwen3.8-27B and IBM Granite ship under Apache 2.0; DeepSeek V4 and GLM-5.1 use MIT.
- Long context is standard. 128K–256K tokens is now normal, and some models claim 1M.
Best local opensource LLM models by hardware tier
Pick the local opensource LLM that matches your memory. Sizes below are approximate Q4 download sizes, which is roughly the minimum VRAM (or unified memory) you need.
Entry level: laptops, CPU-only and 8 GB GPUs
| Model | Size | Approx. memory | Best for |
|---|---|---|---|
| Gemma 4 E2B / E4B (Google) | ~2B / ~4B effective | 2–5 GB | Phones, laptops, quick chat |
| Granite 4.2 3B / 8B (IBM) | 3B / 8B | 2–5.5 GB | Business tasks, Apache 2.0 |
| Phi-4-mini (Microsoft) | 3.8B | ~3 GB | CPU-only reasoning, MIT licence |
| Qwen3.5 9B (Alibaba) | 9B | ~6.6 GB | Best all-rounder under 8 GB, text + images |
Mid range: 12–16 GB GPUs (RTX 4070/5070, 16 GB Macs)
| Model | Size | Approx. memory | Best for |
|---|---|---|---|
| gpt-oss-20b (OpenAI) | 21B total / 3.6B active (MoE) | ~14–16 GB | Fast reasoning, tool use |
| Devstral 24B (Mistral) | 24B | ~14 GB | Coding agents |
| Mistral Small 3.2 | 24B | ~15 GB | Vision, function calling |
Enthusiast: 24–32 GB GPUs (RTX 3090/4090/5090, 32 GB Macs)
| Model | Size | Approx. memory | Best for |
|---|---|---|---|
| Qwen3.8-27B (Alibaba, Aug 2026) | 27B dense | ~18 GB | Top general pick, 256K context, text + image |
| Qwen3.6-27B / 35B | 27B / 35B | 17–24 GB | Coding and agent work |
| Gemma 4 31B / 26B A4B (Google) | 31B dense / 26B MoE (~4B active) | 18–20 GB | Multilingual, vision, Apache 2.0 |
| GLM-4.7-Flash (Z.ai) | ~30B class | ~19 GB | Coding, long context |
| Qwen3-Coder 30B | 30B / 3.3B active (MoE) | ~19 GB | Fast local coding assistant |
| Nemotron 3.5 Bolt 74B-A9B (NVIDIA, Oct 2026) | 74B / 9B active (MoE) | ~25 GB | Agents, 256K context |
Workstation: 64–128 GB unified memory or 2+ GPUs
| Model | Size | Approx. memory | Best for |
|---|---|---|---|
| Llama 3.3 70B (Meta) | 70B dense | ~43 GB | RAG and document Q&A |
| gpt-oss-120b (OpenAI) | 120B MoE | ~65–80 GB | Near-frontier reasoning at home |
| Llama 4 Scout (Meta) | 109B / 17B active | ~55–67 GB | Very long documents |
| Qwen3.5 122B | 122B | ~81 GB | Strong general model, vision |
| Nemotron 3.5 Super (NVIDIA) | 120B / 12B active | ~80–90 GB | Agents, 1M context |
Server class: the open flagships
These need multi-GPU servers or a rented cloud box, but the weights are free to download: DeepSeek V4 Flash (284B / 13B active, MIT), DeepSeek V4 Pro (1.6T / 49B active, MIT), GLM-5.1 (744B / 40B active, MIT) and Kimi K2.6 (1T / 32B active, modified MIT). For the bigger picture on these labs, see our take on the AI war between Google, DeepSeek and ChatGPT.
Best tools to run a local opensource LLM
| Tool | Type | Why use it |
|---|---|---|
| Ollama | CLI + local API | One-command install and model pulls; easiest start |
| LM Studio | Desktop app | Point-and-click GUI, model search, OpenAI-compatible server |
| llama.cpp | C/C++ engine | Runs GGUF models on CPU, NVIDIA, AMD and Apple; powers most other tools |
| Jan / GPT4All | Desktop apps | Offline ChatGPT-style chat, chat with your documents |
| Open WebUI | Web interface | Polished browser UI on top of Ollama |
| text-generation-webui | Web UI | Many backends and loaders for power users |
| KoboldCpp | App | Story writing and role-play |
| LocalAI | Server | Drop-in OpenAI API replacement, text + image + audio |
| vLLM / SGLang | Inference servers | High throughput for teams and multi-user setups |
| MLX / MLX-LM | Apple framework | Fastest option on M-series Macs |
| ExLlamaV2 / V3 | CUDA engine | Very fast quantized inference on NVIDIA GPUs |
| Hugging Face Transformers | Python library | Fine-tuning, research and custom pipelines |
Quantization formats to know: GGUF (llama.cpp, Ollama, LM Studio), MLX (Apple), AWQ and GPTQ (GPU servers), MXFP4/NVFP4 (newer 4-bit formats used by gpt-oss and NVIDIA). For more projects worth exploring, check our list of the best open source projects for AI, Python, Java, Web and DevOps.
How to choose PC hardware for a local opensource LLM
- GPU VRAM matters most. If the whole model fits in VRAM, it is fast. If it spills into system RAM, speed can drop 5–10×. A used RTX 3090 (24 GB) is still the best value card.
- System RAM: 32 GB is the practical minimum in 2026; 64 GB lets you offload MoE layers and run bigger models.
- SSD: use NVMe. Models are 5–80 GB each, and you will collect several. 1 TB is a comfortable start.
- Apple Silicon: unified memory acts like VRAM, so a 64–128 GB Mac can run models no single consumer GPU can.
- CPU-only: fine for 2B–9B models and MoE models with small active sizes; expect slower replies.
Get started in 5 steps
- Check your VRAM (or unified memory) and pick a model from the matching tier above.
- Install Ollama or LM Studio.
- Download a Q4 version of the model (for example
ollama run gpt-oss:20b). - Test with your real tasks: chat, coding, documents.
- Move up a size, or try a higher-quality Q5/Q8 quant, if you have spare memory.
Common mistakes to avoid
- Picking a model that barely fits. Leave 10–20% of VRAM free for the context window, or replies slow down sharply.
- Judging MoE models by total size. A 30B MoE with 3B active runs much faster than a 30B dense model, but it still needs memory for all 30B weights.
- Using very low quants. Q2 and Q3 files fit in less memory but lose noticeable quality. Q4_K_M or Q5 is the usual sweet spot.
- Ignoring the licence. Most models above allow commercial use, but Llama and some Gemma or Qwen variants carry extra terms. Read them before shipping a product.
- Running old models. A 2026 27B model beats most 2024 70B models, so update your library every few months.
Frequently Asked Questions
Which local opensource LLM is best in 2026?
For a single 24 GB GPU, Qwen3.8-27B and Gemma 4 31B are the top all-round picks. On 16 GB, gpt-oss-20b offers the best balance of speed and reasoning.
Can I run a 70B model on a normal desktop GPU?
Not fully on one 24 GB card. A 70B model at Q4 needs about 43 GB, so you need two GPUs, a 64 GB+ Mac, or heavy RAM offloading (slow). Modern MoE models like Nemotron 3.5 Bolt give similar quality on 24–32 GB.
Is a GPU required for local LLM inference?
No. Small and MoE models run on CPU with 16–32 GB RAM, but a GPU is several times faster.
What is the best model for a 12 GB VRAM GPU?
Qwen3.5 9B or Gemma 4 E4B fit comfortably. gpt-oss-20b also works with partial offload to system RAM.
How do I future-proof my PC for local AI?
Prioritise VRAM (24–32 GB), 64 GB system RAM and a 2 TB NVMe SSD. Or choose a Mac with 64 GB+ unified memory.
Conclusion
Running a local opensource LLM in 2026 is easier and cheaper than ever, and the best local opensource LLM for you is simply the strongest one that fits your memory. Start with Ollama or LM Studio and use a Q4 quant. A 16 GB GPU now runs capable assistants, and a 24 GB card puts near-frontier models on your desk with full privacy.
2 Comments