Claude Context Windows & Context Engineering, Explained

Tokens and statelessness: the window is per-call

The context window is measured in tokens, not words or characters. A token is a chunk of text; a rough English rule of thumb is about 3 to 4 characters per token, so a page of prose is several hundred tokens and a large source file can be thousands. Both what you send in and what the model writes out are counted against the same window.

The property that trips up newcomers is statelessness. Claude does not remember your previous turn. There is no server-side session holding your conversation. Each API call is self-contained: whatever the model should 'know' this turn — the system prompt, prior messages, tool definitions, retrieved documents — must be present in the request payload you send. The illusion of a continuous chat is created by your application re-sending the accumulated history every single call.

What actually counts toward the budget

Architects underestimate the window because they only count the user's latest message. In reality a single call is charged for the sum of several components, and the model's own output competes for the same ceiling.

Context rot and lost-in-the-middle

The seductive assumption is that if the right fact is somewhere in the window, the model will use it. Empirically, that is not reliable. As the window fills with more tokens, the model's ability to attend to any specific fact degrades — a phenomenon often called context rot. A closely related effect, lost-in-the-middle, describes how models attend most strongly to the beginning and end of a long context and are weakest on material buried in the middle.

• Signal is diluted: one crucial sentence in 150K tokens of noise is easy to miss.
• Middle placement hurts: the same fact recalled reliably at the top can be missed mid-context.
• Distractors mislead: irrelevant but similar-looking passages actively pull the answer off course.
• Long histories drift: stale earlier turns can contradict or dilute the current instruction.

The practical takeaway is that placement and density matter. Put the most load-bearing instructions and the freshest, most relevant evidence where the model attends best, and remove material that is merely 'nice to have.' Fewer, better tokens beat more tokens.

Bigger window does not mean better accuracy

A larger context window raises the ceiling on how much you can include; it does not raise the floor on how well the model reasons over scattered facts. Moving from a 200K to a 1M window lets you fit an entire codebase, but if the answer depends on correlating three lines from three different files, a bigger window does nothing to make that correlation reliable — and often makes it worse by adding distractors.

Cost and latency scale with the tokens you actually send. A bloated context is slower and more expensive on every call, and those costs recur for the life of the conversation because of statelessness. The window size is a constraint to design within, not a problem you solve by buying more of it.

Prompt caching: pay once for the stable prefix

Because the same system prompt, tool definitions, and reference material are re-sent on every call, you pay to process identical tokens repeatedly. Prompt caching fixes the economics: the provider caches the processed prefix so that on subsequent calls those tokens are read from cache at a large discount and lower latency, instead of being reprocessed.

Caching works on a prefix basis, so ordering is everything. Put the stable, reusable content first — system prompt, then tool schemas, then long static references — and place the volatile, per-request content (the user's new message, freshly retrieved chunks) last. Anything that changes invalidates the cache from that point onward, so a single variable token near the top defeats the whole cache.

{
  "system": [
    { "type": "text", "text": "You are a support triage assistant..." },
    {
      "type": "text",
      "text": "<long stable policy handbook>",
      "cache_control": { "type": "ephemeral" }
    }
  ],
  "messages": [
    { "role": "user", "content": "<the volatile, per-request question goes last>" }
  ]
}

Retrieval vs stuffing

When the knowledge you need is larger than the window — or larger than the slice of it you want to spend — the answer is retrieval, not stuffing. Instead of pasting the entire knowledge base into every prompt, you index it externally and, per request, retrieve only the handful of passages relevant to the current question. This keeps the window dense with signal and directly counters context rot.

Retrieval is not free either: bad chunking, over-fetching, or poor ranking reintroduces noise and distractors. The goal is the same as everywhere else in context engineering — get the right tokens in front of the model, and keep the wrong ones out.

A budgeting approach you can reuse

Treat the window like a project budget. Decide up front how many tokens each category may consume, then design to stay within it. A concrete, repeatable process:

• Reserve output first: decide how many tokens the answer needs and subtract that from the ceiling before allocating anything else.
• Fix the stable prefix: system prompt plus tool schemas plus static references — measure it, and mark it for caching.
• Cap the history: set a turn limit or summarize older turns into a compact rolling summary instead of carrying raw transcript.
• Budget retrieval: allot a fixed slice for retrieved chunks (for example top-k with a token cap) rather than letting it grow open-ended.
• Measure, do not guess: count tokens for each category on real traffic and watch the p95, not just the average.
• Place by importance: freshest, most load-bearing content near the end where attention is strongest; stable reference earlier.