What is LLM Tokenization?

LLM Tokenization

Tokenization is the process of converting text into tokens — the discrete numerical units that LLMs actually process. It's the bridge between human text and model computation.

Why Tokenization?

LLMs are mathematical functions that only process numbers. Tokenization creates a discrete vocabulary that maps text fragments to integers.

textCopy
"Hello world" → tokenize → [9906, 1917] → model → [logits] → detokenize → response text

Tokenization Algorithms

Key insight: BPE dominates the LLM market. Among generative LLMs (not encoder models like BERT), BPE accounts for over 95% of usage. WordPiece is essentially a legacy algorithm confined to BERT-era encoder models.

BPE in Action

textCopy
Initial: "low", "lower", "newest", "widest"

Frequency count:
  "lo" appears 2x → merge to "lo"
  "low" appears 2x → merge to "low"
  "er" appears 2x → merge to "er"
  ...

Result vocab: ["low", "er", "est", "new", "wid"]
"lower" → ["low", "er"]      (2 tokens)
"newest" → ["new", "est"]    (2 tokens)

Algorithm Deep Dive

BPE (Byte Pair Encoding):

• Starts with individual characters/bytes, merges the most frequent adjacent pairs iteratively
• Vocabulary size is a hyperparameter (typically 32K–200K)
• Used by the vast majority of modern LLMs

WordPiece:

• Similar to BPE but merges are chosen to maximize the language model likelihood, not raw frequency
• Prefixes subwords with
  textCopy

  ##

  (e.g.,
  textCopy

  "playing"

  →
  textCopy

  ["play", "##ing"]

  )
• Limited to BERT-family encoder models

Unigram:

• Opposite of BPE — starts with a large vocabulary and prunes tokens that least affect overall likelihood
• Can produce multiple valid tokenizations and picks the most probable one
• Often used via SentencePiece framework

SentencePiece:

• Not an algorithm itself, but a framework that supports both BPE and Unigram
• Key advantage: language-agnostic — operates on raw byte streams with no pre-tokenization rules
• Excellent for multilingual models

ByteLatent Transformer (BLT) — Meta, 2024:

• A tokenizer-free architecture — eliminates fixed vocabulary entirely
• Uses an entropy-based model to dynamically group bytes into "patches"
• Complex/surprising text gets shorter patches (more compute), predictable text gets longer patches (less compute)
• Advantages: robust to typos, novel words, and character-level tasks
• Status: research only — no production models deployed yet

Python Tokenizer Libraries

Comparison Table

tiktoken

pip install tiktoken

transformers.AutoTokenizer

pip install transformers

sentencepiece

pip install sentencepiece

tokenizers

pip install tokenizers

mistral-common

pip install mistral-common

Speed Comparison

tiktoken

tokenizers

sentencepiece

transformers

tokenizers

transformers

Using

textCopy

tiktoken

(OpenAI Models)

pythonCopy
import tiktoken

# For GPT-4o / GPT-4.1 family
enc = tiktoken.encoding_for_model("gpt-4o")  # Uses o200k_base

text = "Tokenization is key to understanding LLMs!"
token_ids = enc.encode(text)
print(f"Token IDs: {token_ids}")
print(f"Token count: {len(token_ids)}")

# Inspect individual tokens
for tid in token_ids:
    print(f"  {tid}: '{enc.decode([tid])}'")

# Available encodings:
# cl100k_base → GPT-4, GPT-3.5-turbo
# o200k_base  → GPT-4o, GPT-4.1 family

Using

textCopy

AutoTokenizer

(HuggingFace — Any Open-Source Model)

pythonCopy
from transformers import AutoTokenizer

# Works with ANY model on HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")

text = "The quick brown fox"
encoded = tokenizer(text, return_tensors="pt")
print(f"Input IDs: {encoded['input_ids']}")
print(f"Tokens: {tokenizer.convert_ids_to_tokens(encoded['input_ids'][0])}")
# → ['▁The', '▁quick', '▁brown', '▁fox']

# Chat template support (critical for instruction-tuned models)
messages = [
    {"role": "system", "content": "You are helpful."},
    {"role": "user", "content": "What is AI?"}
]
formatted = tokenizer.apply_chat_template(messages, tokenize=False)

Using

textCopy

sentencepiece

(Direct)

pythonCopy
import sentencepiece as spm

# Load a pre-trained model file
sp = spm.SentencePieceProcessor(model_file='tokenizer.model')

text = "Hello, world!"
token_ids = sp.encode(text, out_type=int)
token_strs = sp.encode(text, out_type=str)
print(f"IDs: {token_ids}")
print(f"Tokens: {token_strs}")

# Train a custom tokenizer from scratch
spm.SentencePieceTrainer.train(
    input='corpus.txt',
    model_prefix='my_tokenizer',
    vocab_size=32000,
    model_type='bpe'  # or 'unigram'
)

Using

textCopy

tokenizers

(HuggingFace Rust — Custom Training)

pythonCopy
from tokenizers import Tokenizer, models, trainers, pre_tokenizers

# Train a BPE tokenizer from scratch
tokenizer = Tokenizer(models.BPE())
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel()
trainer = trainers.BpeTrainer(
    vocab_size=32000,
    special_tokens=["<pad>", "<eos>", "<bos>"]
)
tokenizer.train(files=["corpus.txt"], trainer=trainer)

# Use it
encoded = tokenizer.encode("Hello, world!")
print(encoded.ids)
print(encoded.tokens)

Key Differences Between Libraries

tiktoken

AutoTokenizer

sentencepiece

tokenizers

tokenizers

transformers

Max Token Limits for Top LLMs (2025–2026)

OpenAI

o200k_base

o200k_base

o200k_base

o200k_base

o200k_base

Anthropic

Google

Meta (Open Source)

Others

Vocabulary Size Trend

Larger vocabularies improve multilingual coverage, code tokenization, and compression efficiency (fewer tokens per text).

Tokenization Edge Cases

Hello

hello

{

}

[

]

Special Tokens

Models use reserved tokens for structural purposes:

<bos>

<s>

<eos>

</s>

<pad>

<unk>

[INST]

Practical Token Counting & Cost Estimation

pythonCopy
import tiktoken

def count_tokens_and_cost(
    text: str,
    model: str = "gpt-4o",
    price_per_million: float = 2.5
) -> dict:
    enc = tiktoken.encoding_for_model(model)
    n = len(enc.encode(text))
    cost = n / 1_000_000 * price_per_million
    return {"tokens": n, "cost": f"${cost:.5f}"}

result = count_tokens_and_cost("Your document text here...")
print(result)  # {'tokens': 5, 'cost': '$0.00001'}

Understanding tokenization is fundamental to: cost estimation, context window management, debugging prompt behavior, choosing the right tokenizer library, and fine-tuning models.

Tokenization Limitations

Yes — tokenization has several inherent limitations that directly impact how LLMs work in practice.

Core Limitations

"strawberry"

r

Non-English Token Inefficiency

Is the Token Limit Per-Prompt or Per-Session?

This is one of the most misunderstood aspects of LLM usage.

API Usage: Per-Request (Stateless)

Every API call is completely independent. The model has zero memory between requests. You must include the entire conversation history in each call:

pythonCopy
from openai import OpenAI
client = OpenAI()

# EVERY call must include ALL prior messages
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is RAG?"},
    {"role": "assistant", "content": "RAG stands for..."},
    {"role": "user", "content": "How is it different from fine-tuning?"},
]
# ALL of the above counts against the context window for THIS request
response = client.chat.completions.create(model="gpt-4o", messages=messages)

Chat Interfaces: Per-Conversation (Managed by Provider)

The Key Distinction

Critical point: Via API, the context window applies to each individual request — not the entire session. Each request must fit all messages (system + history + new prompt + response) within the limit.

Context Window Limits — All Major LLMs (2025–2026)

OpenAI

o200k_base

o200k_base

o200k_base

o200k_base

o200k_base

o200k_base

o200k_base

o200k_base

o200k_base

Anthropic

Google

Meta (Open Source)

Others

Effective vs. Advertised Context Length

A critical caveat — advertised context ≠ effective context:

"Lost in the Middle" Problem: Research shows LLMs recall information best from the beginning and end of the context, with 30%+ degradation for content in the middle. Always place critical information at the start or end of your prompt.

Managing Tokenization Limits in Practice

pythonCopy
import tiktoken

def validate_and_manage_context(
    messages: list[dict],
    model: str = "gpt-4o",
    max_context: int = 128_000,
    reserve_output: int = 4_096
) -> list[dict]:
    """Ensure messages fit within context window."""
    enc = tiktoken.encoding_for_model(model)
    system = [m for m in messages if m["role"] == "system"]
    others = [m for m in messages if m["role"] != "system"]

    budget = max_context - reserve_output
    budget -= sum(len(enc.encode(m["content"])) for m in system)

    # Keep most recent messages that fit
    kept = []
    for msg in reversed(others):
        tokens = len(enc.encode(msg["content"]))
        if budget - tokens < 0:
            break
        kept.insert(0, msg)
        budget -= tokens

    print(f"Kept {len(kept)}/{len(others)} messages, dropped {len(others) - len(kept)}")
    return system + kept