Context Windows Explained: What the Token Number Means

A context window is simply how much text an AI model can “hold on its desk” at one time — your question, the conversation so far, any pasted documents, and the answer it’s writing, all counted together in tokens (a token is about three-quarters of an English word, per OpenAI’s own rule of thumb). Three things about that desk matter more than its advertised size: the model doesn’t use the whole desk equally well, a bigger desk costs real money to use, and companies quietly change desk sizes for business reasons.

Tokens, translated

Today’s advertised windows sit at the big end: GPT-5.5 lists just over a million input tokens (OpenAI API docs), Claude’s current models list a million (Anthropic docs), and Meta’s Llama 4 Scout advertises ten million — while Meta’s own announcement notes it was trained at 256,000 and stretched from there (Meta AI). Keep that trained-versus-marketed gap in mind; it’s the whole story in miniature. Across 123 models, the advertised maximum has grown about 30-fold per year since 2023 (Epoch AI).

One caveat for Indian readers: the word-count rules of thumb above are for English. Tokenizers — the software that chops text into tokens — are trained mostly on English, so Hindi, Tamil, Bengali and other non-Latin scripts typically get split into more pieces per word. The same letter or document costs more tokens, which means it fills the window faster and costs more to process. No vendor publishes a reliable per-language multiplier, so we won’t invent one — just budget extra headroom when working in Indian languages.

The number on the box vs the number that works

Here’s the part the spec sheets skip: models get worse as the desk fills up. The evidence, by name:

• RULER (NVIDIA): of 17 models claiming 32k+ contexts, only about half performed acceptably at 32k on realistic tasks (RULER).
• NoLiMa (Adobe): when the test removes word-matching shortcuts, 10 of 12 models lost half their accuracy by 32k tokens — GPT-4o fell from 99.3% to 69.7% (NoLiMa).
• Context Rot (Chroma): performance degrades as input grows even on absurdly simple tasks, across 18 models (Chroma Research).
• Lost in the Middle: models remember the start and end of a prompt best, and the middle worst (Liu et al.).

Models are improving fast — the length at which top models stay above 80% accuracy grew over 250-fold in nine months (Epoch AI) — but at that snapshot the best still only cleared the bar at 8,000 tokens. Rule of thumb: trust the advertised number as a hard limit, and expect reliable performance at some fraction of it.

Why the limit keeps changing (follow the money)

Serving a long conversation forces the provider to keep a growing “memory scratchpad” for you on a GPU — for a big model, tens of gigabytes per long session. That cost shows up in two ways:

1. Pricing steps. GPT-5.5 charges extra beyond 272,000 input tokens; Google’s Pro pricing steps up past 200,000; Anthropic currently prices its full million flat (OpenAI API docs; Anthropic docs). The surcharge is a choice, not physics.
2. Quiet cuts. Limits can shrink without an announcement — as we reported when Codex’s context was halved with nothing in the changelog. When serving costs bite, the desk gets smaller. The electricity and hardware behind all this is its own story: the real cost of running AI.

And filling a big window costs you too: the same question over an 8,000-token excerpt versus a 200,000-token dump runs about $0.055 versus $1.015 on current GPT-5.5 rates (OpenAI API docs) — roughly 18 times more, before any surcharge.

What it means for your prompts and your bill

Five habits that follow directly from the evidence:

1. Put the important stuff first or last. Never bury the key instruction in the middle of a long prompt.
2. Paste less, not more. A focused excerpt usually beats the full dump on both accuracy and cost.
3. Use the 500-page rule. Under ~200,000 tokens of material, pasting it all is defensible; beyond that, use retrieval — Anthropic’s measured setup cut retrieval failures by 49% (Anthropic).
4. Expect forgetting in long chats. Products silently trim old turns when full. Re-state what matters.
5. Compare usable windows, not label windows — especially on free tiers, where the gap between 32k and 200k is the difference between chunking a PDF and swallowing it whole.

The context window is the rare AI spec that’s real, oversold and unstable all at once. Read the number, then ask: how much of it works, what does filling it cost, and how quietly can it shrink? More evidence-first guides on our how-to and explainers hub.