Does the speed of running local AI (LLM) models on a GPU depend more on the type of GPU used, or on how much VRAM the GPU has?

GPU Speed vs VRAM for Local LLM Inference

The Short Answer

Both GPU architecture and VRAM matter, but for different reasons. VRAM is the gatekeeper — it determines which models you can run at all. GPU architecture determines speed — once the model fits, memory bandwidth, Tensor Cores, and cache hierarchy drive tokens/second.

If a model doesn't fit in VRAM, inference crashes or falls back to CPU/RAM (100-1000x slower). If it fits, a faster GPU with the same VRAM produces 2-3x more tokens/second.

VRAM: What Models Can You Run?

pythonCopy
# Estimate VRAM needed for local LLM inference
def estimate_vram(model_params_B, quant_bits, context_len, batch_size=1):
    weight_gb = model_params_B * (quant_bits / 8)
    kv_cache_gb = context_len * 0.001  # rough: ~1 MB per token
    overhead_gb = 2.0  # activations, CUDA context, buffers
    return weight_gb + kv_cache_gb + overhead_gb

# Llama 3.1 8B at Q4 with 8K context
print(f"8B Q4 8K: {estimate_vram(8, 4, 8192):.1f} GB")  # ~12 GB
print(f"8B FP16 8K: {estimate_vram(8, 16, 8192):.1f} GB")  # ~26 GB
print(f"70B Q4 4K: {estimate_vram(70, 4, 4096):.1f} GB")  # ~41 GB

GPU Architecture: What Determines Speed

For a model that fits in VRAM, these factors determine tokens/second:

The Memory Bandwidth Formula

Memory bandwidth is the primary bottleneck for single-user local inference:

$\text{Tokens/second} \approx \frac{\text{Memory Bandwidth (GB/s)}}{\text{Model Size in VRAM (GB)} \times 1.2}$

The 1.2x overhead accounts for KV cache reads/writes during autoregressive generation.

pythonCopy
# Token speed estimation
def estimate_tps(bandwidth_gbps, model_gb):
    return bandwidth_gbps / (model_gb * 1.2)

# RTX 4090 (1008 GB/s) with Llama 8B Q4 (~5 GB)
print(f"4090: {estimate_tps(1008, 5):.0f} tok/s")   # ~168 tok/s

# RTX 3070 (448 GB/s) with Llama 8B Q4 (~5 GB)
print(f"3070: {estimate_tps(448, 5):.0f} tok/s")    #  ~75 tok/s

# RTX 4070 (504 GB/s) with Llama 8B Q4 (~5 GB)
print(f"4070: {estimate_tps(504, 5):.0f} tok/s")    #  ~84 tok/s

Real-World Performance Comparison

Key insight: The RTX 4070 has more VRAM (12 GB) than the RTX 3080 (10 GB), but the RTX 3080 is faster (110 vs 85 tok/s) because of its higher memory bandwidth (760 vs 504 GB/s). VRAM doesn't make you fast — it makes you capable.

When VRAM Becomes the Speed Bottleneck

VRAM indirectly affects speed in these scenarios:

Decision Framework

Practical Recommendations

For 7-8B models (Mistral, Llama 3.1, Qwen 2.5):

• Minimum: 8 GB VRAM (Q4 quantization, short context)
• Comfortable: 12-16 GB VRAM (Q4/Q5 with 8K+ context)
• Ideal: 16-24 GB VRAM (Q6/Q8, long context, prompt caching)

For 13-34B models:

• Minimum: 16 GB VRAM (Q4, short context)
• Comfortable: 24 GB VRAM (Q4, 4K+ context)
• Ideal: 48+ GB VRAM (Q6, full context)

For 70B+ models:

• Minimum: 48 GB VRAM (Q4, ~2K context) or dual 24 GB GPUs
• Comfortable: 80 GB VRAM (Q4, 4K+ context)
• Ideal: A100/H100 (80 GB) or Mac Studio (192 GB unified)

Summary

The rule of thumb: Buy the most VRAM your budget allows. Among GPUs with similar VRAM, pick the one with higher memory bandwidth. Architecture generation is the tiebreaker when VRAM and bandwidth are similar.

Learn more at r/LocalLLaMA GPU Guide and llama.cpp hardware discussion.