What are billions of parameter trained models (e.g., 7B, 120B)?

Billions of Parameters in AI Models (7B, 70B, 120B, etc.)

Parameters are the learned numerical values in a neural network — specifically the weights and biases that determine how the model transforms inputs to outputs. "7B" means 7 billion such numbers.

What Are Parameters?

In a simple linear layer:

pythonCopy
import torch.nn as nn

# A linear layer with 1000 input features → 500 output features
layer = nn.Linear(1000, 500)

# Parameters = weights (1000×500) + biases (500)
total = 1000 * 500 + 500  # = 500,500 parameters
print(f"Parameters: {sum(p.numel() for p in layer.parameters()):,}")

A model with 7 billion parameters has 7,000,000,000 such numbers, arranged across thousands of layers.

Why Parameter Count Matters

Common Model Sizes

Parameter Count vs Actual Storage

Each parameter is typically stored as:

• FP32: 4 bytes → 7B model = ~28 GB
• FP16/BF16: 2 bytes → 7B model = ~14 GB
• INT8 quantized: 1 byte → 7B model = ~7 GB
• INT4 quantized: 0.5 bytes → 7B model = ~3.5 GB

pythonCopy
# Calculate model size
def estimate_model_size_gb(params_billions: float, bytes_per_param: float = 2) -> float:
    return params_billions * 1e9 * bytes_per_param / (1024**3)

print(f"7B at FP16:  {estimate_model_size_gb(7):.1f} GB")    # 13.0 GB
print(f"70B at FP16: {estimate_model_size_gb(70):.1f} GB")   # 130.2 GB
print(f"7B at INT4:  {estimate_model_size_gb(7, 0.5):.1f} GB")  # 3.3 GB

Scaling Laws

Research shows predictable relationships:

• More parameters → better performance (up to a point)
• More training data → better performance
• Optimal: balance parameters and tokens (Chinchilla law: ~20 tokens per parameter)

Quality vs Size Tradeoff

textCopy
Size         │ Quality  │ Speed   │ Cost
─────────────┼──────────┼─────────┼──────
1-3B         │ Basic    │ Fast    │ Low
7-8B         │ Good     │ Fast    │ Low
13-14B       │ Better   │ Medium  │ Medium
70B          │ Excellent│ Slow    │ High
400B+        │ Best     │ Very slow│ Very high

Running Models Locally

bashCopy
# Ollama - easily run models by size
ollama pull llama3.1:8b    # 8B — runs on 8GB RAM
ollama pull llama3.1:70b   # 70B — needs 64GB+ RAM
ollama pull phi3:mini       # 3.8B — great for laptops

The 7B-8B size sweet spot is popular for local inference — good quality with practical hardware requirements.