Context Engineering: The Discipline Replacing Prompt Tricks
Prompt wording no longer decides system quality; the contents of the context window do. This article defines context engineering as of mid-2025: what enters the window, how compaction, retrieval, tool results and memory files behave, why claimed context lengths mislead, and how measurable token budgets turn context into an engineered resource.
The prompt was never the whole problem
For two years the craft of working with language models was mostly wording: role instructions, delimiter tricks, chain-of-thought phrases. That was adequate as long as a model saw one prompt and returned one answer. Production systems in mid-2025 look different. At any moment an agent's context window holds a system prompt, tool definitions, retrieved documents, tool outputs, conversation history and memory files. The sentence a user typed is a small fraction of the input.
Context engineering is the discipline of deciding what enters that window — in what form, at which position and at what cost. It is not a new model capability and it does not make a model smarter. It changes only the input distribution. In our project work, that is where most of the quality variance now sits.
What actually enters the window
A request to a modern agent assembles several distinct categories: a system prompt with instructions and policies; tool and function definitions whose schemas cost tokens on every call; retrieved document chunks; conversation history; tool results; and persistent memory files. Each category has a different owner, a different update frequency and a different failure mode. Together they routinely exceed the user's actual message by an order of magnitude.
Most teams cannot state how many tokens each category consumes in a typical request. That is the first deliverable of context engineering: an accounting. What is not measured cannot be budgeted, and what is not attributed cannot be cut.
Long windows are not free
Windows have grown fast. OpenAI's GPT-4.1 (14 April 2025) accepts 1,047,576 tokens of input. Google's Gemini 2.5 Pro offers one million. Meta's Llama 4 Scout advertises ten million. Anthropic's Claude Opus 4 (22 May 2025) stays at 200,000 and ranks among the strongest coding models regardless. The table shows the state at publication.
Advertised capacity is not usable capacity. Liu et al. showed in 2023 that accuracy drops when relevant information sits in the middle of a long context. NVIDIA's RULER benchmark (April 2024) found that of 17 models claiming 32K tokens or more, only about half held their performance at 32K — and almost all fell below their claimed length. Latency adds up too: OpenAI reports roughly a minute to first token for GPT-4.1 at a full million tokens. A window is capacity, not a target.
| Model | Released | Advertised window | Note |
|---|---|---|---|
| GPT-4.1 | 14 Apr 2025 | 1,047,576 tokens | API only |
| Gemini 2.5 Pro | 25 Mar 2025 | 1,048,576 tokens | experimental at launch |
| Claude Opus 4 | 22 May 2025 | 200,000 tokens | entire Claude 4 family |
| Llama 4 Scout | 5 Apr 2025 | 10,000,000 tokens | pre-trained at 256K |
Compaction keeps sessions alive
Agent sessions accumulate history faster than any window grows. Compaction replaces old turns with a structured summary and continues the session on top of it. MemGPT (October 2023) framed this as virtual memory: page information between the window and external storage the way an operating system pages RAM. Claude Code, generally available since 22 May 2025, compacts automatically near the limit and exposes /compact and /clear as manual controls.
Compaction is lossy. Exact identifiers, error strings and diffs can vanish from a summary, and the agent then pays twice — once to lose the detail, once to re-read it. Our working rule: compact the narrative, keep artifacts — file paths, commands, decisions — verbatim.
Retrieval and tool results as supply lines
Retrieval does not disappear in this picture; it becomes one supply line among several, competing for budget with agentic file search and direct tool calls. The Model Context Protocol, published by Anthropic on 25 November 2024 and adopted by OpenAI in March 2025, standardizes how external systems deliver context and tools. Standardization has a price: every connected server injects its tool definitions into every request, before the model has done anything.
Tool results are the least disciplined category. A verbose API response or an unfiltered log dump enters the window whole. The correction belongs at the tool boundary: truncate, filter and paginate in the integration, rather than hoping the model ignores noise. It does not reliably ignore noise.
Memory files as persistent context
Persistent memory files are the cheapest context mechanism in production today: instruction files checked into the repository and loaded at session start. Claude Code reads CLAUDE.md, Cursor reads .cursor/rules, GitHub Copilot reads .github/copilot-instructions.md. They are versioned, reviewed and shared like code. Anthropic's best-practices guide of 18 April 2025 states the operative constraint: keep the file short and iterate, because it is prepended to every session.
Memory files do not learn. They are static context, they cost tokens in every request, and a bloated file buries its own rules. Treat them as prompts under version control — with the same review discipline and an owner who prunes them regularly.
Context budgets you can measure
A context budget assigns each category a share of the window and makes that assignment testable. The metrics are unglamorous: tokens per category per request, window fill level at task completion, compaction frequency per session, cache hit rate. We track these values per agent the way we track latency percentiles per service.
Caching rewards stable budgets directly. Anthropic's prompt caching prices cache reads at roughly 10 percent of the base input rate; OpenAI lists cached GPT-4.1 input at $0.50 instead of $2.00 per million tokens. Both require byte-stable prefixes — which enforces the useful habit of ordering context from stable to volatile. The unbudgeted alternative has a visible price: a full million-token GPT-4.1 request costs about two dollars in input tokens alone on every cache miss.
The outlook from June 2025
From where we stand in June 2025, four developments look likely. First, the term itself will stick: context engineering names work that teams already do without a name, and vocabulary tends to follow practice within months. Second, context observability — per-category token accounting in traces — will become a standard feature of LLM tooling rather than a homegrown spreadsheet.
Third, memory will move from static files toward managed stores that agents read and write across sessions; MemGPT sketched the mechanics, and products are close. Fourth, benchmarks will shift from claimed to effective context length as the honest number. Windows will keep growing. What enters them will remain an engineering decision — more capacity changes what fits, not whether curation matters.
Sources
- Lost in the Middle: How Language Models Use Long Contexts (Liu et al., arXiv, 6 Jul 2023)
- MemGPT: Towards LLMs as Operating Systems (Packer et al., arXiv, 12 Oct 2023)
- RULER: What's the Real Context Size of Your Long-Context Language Models? (Hsieh et al., NVIDIA, arXiv, 9 Apr 2024)
- Introducing the Model Context Protocol (Anthropic, 25 Nov 2024)
- Introducing GPT-4.1 in the API (OpenAI, 14 Apr 2025)
- Claude Code: Best practices for agentic coding (Anthropic, 18 Apr 2025)