What are all the different model formats in LLM?

LLM Model Formats

When you download an LLM, the model weights come in a specific file format. Each format serves different purposes — some optimize for inference speed, others for storage efficiency, and some for cross-framework compatibility.

Overview of Model Formats

Raw/Uncompressed Formats

pythonCopy
# PyTorch .bin — traditional format
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    torch_dtype="auto",
)
# Loads model-00001-of-00002.bin, model-00002-of-00002.bin

SafeTensors is now the preferred raw format — it avoids pickle-based deserialization (security risk) and supports zero-copy loading:

pythonCopy
# SafeTensors — safer, faster
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.2",
    use_safetensors=True,  # Default for most modern models
)
# Loads model-00001-of-00002.safetensors, etc.

Quantized Formats — The Practical Choice

GGUF — The CPU/Edge King

bashCopy
# Download a GGUF model and run with llama.cpp
wget https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF/resolve/main/mistral-7b-instruct-v0.2.Q4_K_M.gguf

# Run with llama.cpp
./main -m mistral-7b-instruct-v0.2.Q4_K_M.gguf -p "What is RAG?" -n 256

# Or with Ollama (which uses GGUF internally)
ollama pull mistral

GGUF quantization levels:

GPTQ vs AWQ — GPU Quantized Formats

Both achieve 4-bit inference on GPU, but they differ in approach:

auto-gptq

autoawq

pythonCopy
# Load GPTQ model
model = AutoModelForCausalLM.from_pretrained(
    "TheBloke/Mistral-7B-Instruct-v0.2-GPTQ",
    device_map="auto",
)

# Load AWQ model
model = AutoModelForCausalLM.from_pretrained(
    "TheBloke/Mistral-7B-Instruct-v0.2-AWQ",
    device_map="auto",
)

ONNX — Cross-Platform Standard

ONNX exports the model to an intermediate representation that runs on any ONNX Runtime — useful for Triton Inference Server, Azure ML, and Windows deployment:

bashCopy
# Export to ONNX
optimum-cli export onnx   --model mistralai/Mistral-7B-v0.1   --task text-generation   ./mistral-onnx

# Load with ONNX Runtime
from optimum.onnxruntime import ORTModelForCausalLM

model = ORTModelForCausalLM.from_pretrained("./mistral-onnx")

Choosing the Right Format

Rule of thumb: Start with SafeTensors for flexibility. For CPU/local use, go GGUF. For GPU serving at scale, pick AWQ. The format should match your deployment target — not your ego.

Learn more at llama.cpp GitHub, AutoAWQ, and HuggingFace Optimum.