Is a Longer Context Always Better? From Tokens and Context Windows to Context Engineering
If the previous article, “LLMs Are Not Knowledge Bases,” established the first mental model — an LLM generates output from the conditions it can currently see — the next question is:
What can the model actually see right now?
Many AI application failures are not simply “the model is too weak.” The model may never have received the information it needed at inference time.
You can switch to a stronger model, write a longer prompt, or attach more documents. But if the current context omits the governing rule, mixes old and new versions, or buries the important signal under irrelevant material, the result can still be poor.
This article separates three concepts that are often mixed together:
- Token: a unit used by the model to process text.
- Context window: the token budget available to one inference.
- Context engineering: deciding what information should occupy that budget.
The goal is not “put in more data.”
The goal is to make the limited context contain the most useful information for the task.
1. A Token Is Not the Same as a Character or a Word
An LLM does not directly process text as the same characters or words that a person sees.
A tokenizer first converts text into token IDs, which are then processed by the model.
A simplified view is:
Text
↓
Tokenizer
↓
Token IDs
↓
Model
A token may correspond to a whole word, part of a word, punctuation, whitespace, or another text fragment. Different tokenizers can split the same text differently.
For example, an uncommon English word may be decomposed into several subwords. The same sentence can also produce a different token count when processed by another model or tokenizer.
So:
Token count ≠ character count
Token count ≠ English word count
The current Hugging Face Transformers documentation describes BPE, Unigram, and WordPiece as common subword tokenization approaches. Their shared purpose is to represent a large space of text with a manageable vocabulary rather than requiring every possible word to be its own token.
Why should an application engineer care about tokens?
Because tokens are not only a model-internals concept. They directly affect engineering constraints such as:
- how much material can be sent in one request
- how much output can be generated
- API usage and cost
- how much conversation history can be retained
- whether documents need to be chunked or filtered
- whether tool results should be compressed
- when a long-running task needs summaries or external memory
A useful first approximation is:
Tokens are one of the capacity units of an LLM system.
2. The Context Window Is the Working Space of One Inference
Suppose an AI system needs to answer:
“Is this customer eligible under the latest refund policy?”
The user’s final question is not the only information that can affect the answer.
One inference may include:
System / developer instructions
Conversation history
User question
Retrieved documents
Tool results
Examples
Output constraints
Together, these form the context visible to the model at that moment.
The exact accounting of a context window differs across model providers and APIs. OpenAI’s current model documentation, for example, publishes model-specific context-window and output-token limits, and reasoning-capable models may also use reasoning tokens. These implementation details should not be generalized across every provider.
The provider-neutral mental model is simpler:
The context window is the finite information space available to the model for this inference.
That is not the same thing as permanent memory.
3. Do Not Confuse Context, State, and Memory
This distinction matters in AI application design.
A useful simplified boundary is:
State / Memory
= information the application retains outside the model across time
Context
= information actually supplied to the model for this inference
A customer-service application may store:
- three years of order history
- twenty prior support conversations
- all company policies
- user preferences
- the current ticket state
All of that information may exist in the system, but the model does not need to see all of it on every turn.
The application might select only:
Latest refund policy
+
Most recent order
+
Current support ticket
+
Current user question
and assemble those items into the current context.
So:
Having data in the system does not mean the data is in the model’s current context.
Likewise, when a model appears to “remember” the previous turn, that does not necessarily imply persistent model memory. Many chat systems simply include earlier messages again in the next request.
This is why context engineering and memory engineering solve different problems.
Memory asks:
What information should survive across time?
Context engineering asks:
Which part of that available information should the model see now?
4. A Larger Context Window Does Not Guarantee a Better Answer
A common intuition is:
If the model supports more tokens, putting everything into the context must be safer.
The problem is that input capacity and effective information use are not the same thing.
The 2023 Lost in the Middle study evaluated multi-document question answering and key-value retrieval. In the models tested, performance was sensitive to where relevant information appeared in a long context: results were often stronger when the relevant information appeared near the beginning or end and weaker when it appeared in the middle.
That result should not be turned into a claim that:
“Every current model is unable to use information in the middle of its context.”
Models, training methods, and long-context capabilities continue to change.
The more durable lesson is:
A model accepting a certain context length does not guarantee equally reliable use of every item placed inside it.
Anthropic’s 2025 context-engineering guidance makes a related engineering point: context is a finite resource, and the target should be a compact set of high-signal information rather than unlimited accumulation.
Long context therefore has at least four practical costs.
1. More irrelevant information
The model has more material from which it must separate the useful signal.
2. More conflicts
Old policies, new policies, and competing sources can all appear together.
3. More processing, cost, and latency
The actual effect depends on the model and API, but more input generally means more material for the system to process.
4. Harder debugging
When the output is wrong, you need to determine whether:
- the necessary information never entered context
- correct information was diluted by irrelevant or conflicting material
- instructions conflicted
- retrieval returned the wrong evidence
- the model failed to use the right evidence
A context window answers:
Can this information fit?
An AI application still has to answer:
What should be included?
5. A Concrete Example: Why a Refund Policy Should Not Be “More Is Better”
Imagine a customer-service AI receives this context:
A. 2026 current refund policy
B. 20 pages of product catalog
C. 2024 old refund policy
D. Customer's current order
E. User question
Everything technically “fits.”
The system can still fail:
- the old and new policies conflict
- the product catalog is irrelevant
- the current policy has no explicit version marker
- the model may rely on the older rule
- the final answer may not reveal which policy supported the decision
The solution may not be a larger context window.
A better pipeline might be:
Define the task
↓
Identify required information
↓
Select the current valid policy
↓
Retrieve the current order
↓
Attach source and version information
↓
Assemble the context
↓
Ask the model to analyze / explain
That is no longer only prompt writing.
That is context engineering.
6. Context Quality Matters More Than Context Quantity
A quick context review can start with five dimensions.
Relevance
Does this information actually help with the current task?
Authority
Where do the rules, numbers, and facts come from? Is the source appropriate to trust?
Freshness
Is the information still current? Is there a newer version?
Consistency
Do different items conflict? If they do, which one has priority?
Structure
Can the model distinguish among:
- system instructions
- trusted business data
- user-provided content
- untrusted external content
- tool results
A useful engineering checklist is:
Good Context
=
Relevant
+ Trusted
+ Current
+ Consistent
+ Well-structured
This is not a formal equation. It is a practical review framework.
7. Prompt Engineering vs. Context Engineering
Prompt engineering still matters.
But a prompt is only one part of context.
| Prompt Engineering | Context Engineering | |
|---|---|---|
| Core question | How should the instruction be written? | What should the model see for this inference? |
| Typical material | task, role, constraints, format, examples | prompt, history, retrieval, memory, tool results, permission information |
| Common failure | ambiguous instruction | stale, conflicting, polluted, excessive, or missing information |
| Engineering focus | express the task | manage the inference environment |
Anthropic describes context engineering in a similar way: not merely writing prompts, but curating and maintaining the optimal set of tokens used during inference, including system instructions, tools, external data, and message history.
A useful relationship is:
Prompt Engineering
⊂
Context Engineering
Prompt engineering is part of the larger context problem, not the whole problem.
8. A Simple Context-Engineering Workflow
You do not need a complex framework to begin.
A useful five-step process is:
Step 1: Define what the model must do at this step
Classification, summarization, answering, extraction, planning, and decision support require different information.
Step 2: Identify necessary information
What facts, rules, state, and evidence are required for the model to perform the task reasonably?
Step 3: Filter
Remove:
- irrelevant material
- duplicated information
- obsolete versions
- unnecessarily large raw tool outputs
Step 4: Mark source and trust level
At minimum, distinguish among:
Trusted instruction
Trusted business data
User-provided content
External / untrusted content
Step 5: Evaluate
Do not judge the design from one or two outputs.
Test questions such as:
- Did the model receive every required fact?
- Did it use the correct version?
- How sensitive is the answer to retrieval changes?
- Does shorter context actually reduce quality?
- Are tokens being spent on information that does not improve the result?
Context engineering eventually comes back to evaluation, not intuition alone.
9. Long-Horizon Work Cannot Depend on One Ever-Growing Context Window
The limitation becomes clearer when a task grows from one question into:
- long-form research
- a large codebase change
- a multi-step agent run
- work that spans multiple sessions
Information keeps accumulating, but keeping every raw event forever in one context is neither necessary nor always useful.
Common strategies include:
- Retrieval: load information when it becomes relevant.
- Compaction: compress older history into a shorter representation.
- Structured notes: preserve important state outside the model.
- Memory: retain information for later interactions.
- Tools: let the model retrieve files or data when needed instead of preloading everything.
- Subagents: isolate focused work in cleaner contexts and return distilled results.
The common principle remains:
Do not require the model to carry everything forever. Design the system so each step can retrieve what it actually needs.
10. Conclusion: Context Window Is a Capacity Problem; Context Engineering Is a Selection Problem
If the only thing you remember about context windows is that a particular model supports a certain number of tokens, that knowledge will age quickly.
Model specifications will continue to change.
The more durable mental model is:
System knows a lot
↓
Context Engineering selects
↓
Model sees a subset
↓
Model makes this inference
An AI application engineer is not only controlling the model.
The engineer is also controlling:
What information enters the model, at what time, and in what form.
The context window defines the capacity boundary.
Context engineering determines what occupies that capacity.
That leads naturally to the next question:
Once the model receives those tokens, how does it build relationships among them? What roles do attention and embeddings play?
The next article goes one layer deeper into the model.
Sources and Verification Scope
This article extends the Hello-Agents Chapter 3 study material. The following sources were rechecked on September 21, 2026. The article does not generalize one provider's context-window specification across all models, and it does not claim that long context causes a fixed amount of degradation in every current model.
- Datawhale / Hello-Agents — Chapter 3, Large Language Model Fundamentals: instructional context for tokenization, tokens, and context windows. https://github.com/datawhalechina/hello-agents/blob/main/docs/chapter3/%E7%AC%AC%E4%B8%89%E7%AB%A0%20%E5%A4%A7%E8%AF%AD%E8%A8%80%E6%A8%A1%E5%9E%8B%E5%9F%BA%E7%A1%80.md
- Hugging Face Transformers — Tokenization algorithms: BPE, Unigram, WordPiece, and subword tokenization. https://huggingface.co/docs/transformers/tokenizer_summary
- OpenAI API model documentation: model-specific context-window, output-token, and reasoning-token support. https://developers.openai.com/api/docs/models
- Anthropic Engineering — Effective context engineering for AI agents: finite context, token curation, just-in-time retrieval, compaction, structured note-taking, and subagents. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
- Liu et al. — Lost in the Middle: How Language Models Use Long Contexts. https://arxiv.org/abs/2307.03172