What is a context window and how to manage the context window in LLM?

What Is a Context Window?

A context window is the maximum number of tokens an LLM can process in a single request — including both input tokens (your prompt, system instructions, conversation history) and output tokens (the model's response). Everything outside this window is invisible to the model.

$\text{Context Window} = \text{Input Tokens} + \text{Output Tokens}$

textCopy
You type: "Explain RAG in detail" (5 tokens)
   +
System prompt (500 tokens)
   +
Conversation history (2,000 tokens)
   +
Model response (1,000 tokens)
   =
Total: 3,505 tokens used from context window

Rule of thumb: 1 token ≈ 0.75 English words. So 128K tokens ≈ 96,000 words ≈ a 192-page book.

Per-Prompt or Per-Session?

This is one of the most commonly misunderstood aspects of context windows.

API Usage: Per-Request (Stateless)

Every API call is completely independent. The model has zero memory between requests. You must include the entire conversation history in each request:

pythonCopy
from openai import OpenAI
client = OpenAI()

# Each API call must include ALL prior messages
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is RAG?"},
    {"role": "assistant", "content": "RAG stands for..."},     # previous response
    {"role": "user", "content": "How does it differ from fine-tuning?"},  # new question
]
# ALL of the above counts against the context window
response = client.chat.completions.create(model="gpt-4o", messages=messages)

Chat Interfaces: Per-Conversation (Managed by Provider)

Services like ChatGPT, Claude.ai, and Gemini manage history automatically:

Key Distinction

Context Window Limits — All Major LLMs (2025–2026)

OpenAI

o200k_base

o200k_base

o200k_base

o200k_base

o200k_base

o200k_base

o200k_base

o200k_base

o200k_base

Anthropic

Google

Meta (Open Source)

Mistral

DeepSeek

Others

Why Context Windows Are Limited

1. Quadratic Attention Complexity — O(n²)

Self-attention computes an

n × n

2. KV Cache Memory

During generation, Key/Value vectors for every token are cached:

3. Positional Encoding Limits

Models trained with RoPE (Rotary Position Embeddings) degrade beyond their training length. Extending requires techniques like YaRN or LongRoPE.

Solutions Used by Modern Models

Context Window Management Strategies

Strategy 1: Token Counting Before Sending

Always count tokens before making API calls to avoid errors.

pythonCopy
import tiktoken

def check_fits_context(
    messages: list[dict],
    model: str = "gpt-4o",
    max_context: int = 128_000,
    reserve_for_output: int = 4_096
) -> bool:
    enc = tiktoken.encoding_for_model(model)
    total = sum(len(enc.encode(m["content"])) for m in messages)
    available = max_context - reserve_for_output
    print(f"Using {total}/{available} tokens ({total/available*100:.1f}%)")
    return total <= available

Strategy 2: Sliding Window Truncation

Keep only the most recent messages that fit.

pythonCopy
def sliding_window_truncate(messages, max_tokens=120000):
    """Keep system prompt + most recent messages within budget."""
    enc = tiktoken.encoding_for_model("gpt-4o")
    system = [m for m in messages if m["role"] == "system"]
    others = [m for m in messages if m["role"] != "system"]

    budget = max_tokens - sum(len(enc.encode(m["content"])) for m in system)
    kept = []

    for msg in reversed(others):
        msg_tokens = len(enc.encode(msg["content"]))
        if budget - msg_tokens < 0:
            break
        kept.insert(0, msg)
        budget -= msg_tokens

    return system + kept

Strategy 3: Summarize Older Messages

Compress old history using a smaller/cheaper model.

pythonCopy
async def summarize_and_compact(messages, client):
    """Summarize old messages, keep recent ones verbatim."""
    old_messages = messages[:-10]   # older history
    recent = messages[-10:]          # keep last 10 turns

    summary_prompt = "Summarize this conversation concisely:\n"
    for m in old_messages:
        summary_prompt += f"{m['role']}: {m['content']}\n"

    summary = await client.chat.completions.create(
        model="gpt-4o-mini",  # use cheap model for summarization
        messages=[{"role": "user", "content": summary_prompt}],
        max_tokens=500
    )

    return [
        {"role": "system", "content": f"Previous conversation summary: {summary.choices[0].message.content}"},
        *recent
    ]

Strategy 4: RAG (Retrieval Augmented Generation)

Store information externally, retrieve only what's relevant.

pythonCopy
from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings

# Store documents in vector DB (once)
vectorstore = Chroma.from_documents(documents, OpenAIEmbeddings())

# At query time, retrieve only relevant chunks
relevant_docs = vectorstore.similarity_search(user_query, k=5)
context = "\n".join([doc.page_content for doc in relevant_docs])

# Inject only relevant context into prompt
messages = [
    {"role": "system", "content": f"Answer using this context:\n{context}"},
    {"role": "user", "content": user_query}
]

Strategy 5: Hierarchical Context Management

Layer different types of context at different granularities:

textCopy
┌─────────────────────────────────────────────┐
│ Layer 1: System Prompt (always present)      │  ~500 tokens
│   Role, rules, formatting instructions       │
├─────────────────────────────────────────────┤
│ Layer 2: Long-term Memory (compact)          │  ~500 tokens
│   User preferences, key facts                │
├─────────────────────────────────────────────┤
│ Layer 3: Session Summary (compressed)        │  ~1,000 tokens
│   Condensed history of current conversation  │
├─────────────────────────────────────────────┤
│ Layer 4: Recent Messages (full detail)       │  ~5,000 tokens
│   Last 5-10 turns verbatim                   │
├─────────────────────────────────────────────┤
│ Layer 5: Retrieved Context (dynamic)         │  ~3,000 tokens
│   RAG results relevant to current query      │
├─────────────────────────────────────────────┤
│ Layer 6: Reserved for Output                 │  ~4,096 tokens
│   Space for model's response                 │
└─────────────────────────────────────────────┘

Strategy 6: Prompt Caching

All major providers now offer caching to reduce cost and latency for repeated context:

cache_control

Best practice: Place static content (system prompts, reference documents) at the start of the prompt and dynamic content at the end to maximize cache hits.

The "Lost in the Middle" Problem

Research shows LLMs have a U-shaped performance curve — they recall information best from the beginning and end of the context, with 30%+ degradation for content in the middle.

textCopy
Recall Performance by Position:

  High  ██████████░░░░░░░░░░░░██████████
  Low               ▲
        Beginning  Middle        End
                    ↑
           Worst recall here

Mitigation: Place the most critical information at the start or end of your prompt — never buried in the middle.

Effective vs. Advertised Context Length

Key insight: Advertised context ≠ effective context. Plan for 60–80% of the advertised maximum for reliable performance.

Context Management Decision Tree