Determining VRAM Requirements for Local LLM Execution
The fundamental consideration when deploying a local LLM is straightforward: does the model fit within your GPU's capacity? This outcome hinges on the model's size, its quantization level, and the context length. This guide provides a practical framework for selecting an appropriate VRAM allocation.
The Impact of Quantization on VRAM
Quantization lowers the precision used for storing model weights. Reducing the bit count shrinks the model and decreases VRAM consumption, though this typically comes at the cost of some quality.
| Quantization Type | Bits per Weight | Typical Application |
|---|---|---|
| Q8_0 | 8 | Premium quality |
| Q6_K | ~6.6 | High quality |
| Q5_K_M | ~5.5 | Optimal balance of quality and size |
| Q4_K_M | ~4.5 | Strong equilibrium between size and performance |
| Q3_K_M | ~3.5 | Reduced VRAM usage with noticeable quality trade-offs |
Q4_K_M is frequently selected when VRAM is constrained. With greater VRAM availability, opting for Q5 or Q6 allows running the same model with less aggressive quantization.
Estimated VRAM Usage by Model Size
The figures below are approximate estimates for model weights alone. Total VRAM demand is higher, as the runtime, KV cache, and context window also consume memory.
| Model Size | Q8_0 | Q6_K | Q4_K_M | Q3_K_M |
|---|---|---|---|---|
| 4B | ~5 GB | ~4 GB | ~3 GB | ~2.5 GB |
| 8B | ~9 GB | ~7 GB | ~5.5 GB | ~4.5 GB |
| 12B | ~13 GB | ~10 GB | ~8 GB | ~6.5 GB |
| 14B | ~16 GB | ~12 GB | ~9 GB | ~7.5 GB |
| 27B | ~30 GB | ~22 GB | ~17 GB | ~13 GB |
| 32B | ~36 GB | ~27 GB | ~20 GB | ~16 GB |
| 70B | ~80 GB | ~60 GB | ~42 GB | ~34 GB |
Please note these are estimates rather than strict limits. Variations in model architecture and quantization formats can influence the actual size.
Model Capabilities Based on VRAM Tiers
| VRAM Capacity | Feasible Range | Current Examples |
|---|---|---|
| 8 GB | Compact models approx. 4B to 9B | Gemma 4 E4B, Qwen3.5 9B |
| 12 GB | Small to mid-range models approx. 9B to 14B | Gemma 4 12B, Qwen3.5 9B |
| 16 GB | 12B to 27B with lower precision quantization | Gemma 4 26B-A4B, Qwen3.6 27B at Q4 |
| 24 GB | 27B to 35B at Q4 to Q6 precision | Qwen3.8 27B, Gemma 4 31B |
| 32 GB | 27B to 35B at higher precision quantization | Qwen3.8 27B, Gemma 4 31B |
| 48 GB | Large dense models with lower precision quantization | 70B-class models at Q3 to Q4 |
| 80 GB | Large dense models with higher precision quantization | 70B-class models at Q4 to Q6 |
These ranges assume the model weights can reside on the GPU. Mixture-of-Experts (MoE) models operate differently: although only a subset of parameters is active per token, the system must still store the entire weight set. Consequently, a model with 100B or more total parameters will not fit within a 100B-sized VRAM budget simply because it has a smaller active parameter count.
Mixture-of-Experts (MoE) Models
MoE architectures consist of multiple parameter groups known as experts. Since only certain experts are activated for each token, inference can be more efficient than with a dense model of equivalent total parameter count.
Nevertheless, inactive experts remain part of the model's memory footprint. Thus, large MoE models may demand significantly more memory than their active parameter count implies. Very large models may necessitate multiple GPUs or offloading to system RAM.
Context Length Consumes Additional VRAM
Model weights constitute only a portion of the total memory requirement. The KV cache expands as context length increases; therefore, operating the same model with a 64K context may require substantially more VRAM than a 4K context.
- Extended context windows demand greater VRAM.
- The precision of the KV cache influences memory consumption.
- Batch size and concurrent user counts also increase memory usage.
- Reserve a portion of VRAM for the runtime rather than allocating all capacity to model weights.
Practical Recommendations
- Verify the exact size of the quantized model you intend to use.
- Avoid treating the model file size as the precise VRAM requirement; account for KV cache and runtime overhead.
- If a model exceeds VRAM capacity, part of it can be offloaded to system RAM, although this typically slows down inference.
- For long-context or agentic workloads, allocate more VRAM than is required for the weights alone.
- Utilize multiple GPUs to distribute a model if a single GPU lacks sufficient VRAM.
Execute on DaDesktop
You are not required to purchase a GPU to run local LLMs. DaDesktop provides a cloud desktop with the necessary VRAM, allowing you to run models directly without owning the hardware.
Select a VRAM tier suitable for your model, load it, and begin usage immediately. There is no setup, no hardware procurement, and no driver complications. View available GPUs for specific options.