What is quantization in AI/LLM?

What is Quantization in AI/LLM?

Quantization is the process of reducing the precision of a model's numerical values (weights and activations) from higher-bit formats (float32) to lower-bit formats (int8, int4) — dramatically reducing model size and memory requirements with minimal quality loss.

Why Quantization Matters

A 70B parameter model at full precision (FP32) needs 280 GB of GPU memory — impossible on most hardware. Quantization makes it practical:

How Quantization Works

textCopy
Original weight: 0.45892 (FP32 — 32 bits, high precision)
                    ↓
INT8 quantization:
  scale = max_value / 127
  quantized = round(0.45892 / scale) = 58 (INT8 — 8 bits)

  To use: dequantize = 58 × scale ≈ 0.457 (small error, acceptable)

Types at a Glance

Using Quantization in Practice

pythonCopy
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

# 4-bit quantization (QLoRA-style, runs 7B on 6GB VRAM)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4"  # Normal Float 4 — better than INT4
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B",
    quantization_config=bnb_config,
    device_map="auto"
)
# Model uses ~4 GB VRAM instead of ~16 GB

Ollama (Easiest Quantized Models)

bashCopy
# Download pre-quantized models
ollama pull llama3.1:8b           # 4-bit quantized ~4.7GB
ollama pull llama3.1:8b-instruct-q8_0  # 8-bit ~8.5GB

# Quantization level in model name
# q4_0, q4_K_M, q5_K_M, q8_0 — higher number = better quality

Quality vs Efficiency Trade-off

textCopy
Quality
  ↑
  │  FP32 ●────────────────────── (highest quality, highest cost)
  │  BF16 ●──────────────────── (nearly same quality, 2x smaller)
  │  INT8 ●──────────────── (small quality drop, 4x smaller)
  │  INT4 ●─────────── (noticeable drop on edge cases, 8x smaller)
  │  INT2 ●──── (significant quality loss, experimental)
  └──────────────────────────────→ Memory Efficiency

When to Quantize