Explain these LLM concepts: Tokens, Context window, Temperature & Top-p sampling, Beam search.

Core LLM Concepts

Tokens

A token is the basic unit an LLM processes — not a word, but a subword chunk. Tokenisation splits text using algorithms like Byte-Pair Encoding (BPE).

pythonCopy
import tiktoken  # OpenAI tokenizer

enc = tiktoken.encoding_for_model("gpt-4")
tokens = enc.encode("Transformer architecture")
print(tokens)        # [38841, 9756, 23103]
print(len(tokens))   # 3 tokens for 2 words

Rule of thumb: 1 token ≈ 0.75 English words (100 tokens ≈ 75 words)

Context Window

The context window is the maximum number of tokens the model can "see" at once — both input and output combined.

Tokens outside the context window are simply not available to the model — it cannot reference them.

Temperature

Temperature controls randomness in token sampling. At each step, the model produces a probability distribution over all tokens. Temperature rescales this distribution:

$P_i \propto \exp(z_i / T)$

0.0

0.7

1.0

>1.0

pythonCopy
from openai import OpenAI
client = OpenAI()

# Factual answer — low temperature
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What is 2+2?"}],
    temperature=0.0
)

# Creative story — high temperature
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Write a poem about AI"}],
    temperature=0.9
)

Beam Search

Beam search keeps the top-$k$ most probable sequences at each step, expanding each, then pruning back to $k$. Unlike greedy decoding (always picks best single token), it finds globally better sequences.

Trade-off: Better output quality but $k\times$ more compute. Mostly used in translation and summarisation — LLMs typically use sampling instead.