Context Engineering in Practice: Building Agents That Remember Just Enough

Most agents that fail don't fail because the model is weak. They fail because whoever built them assumed the fix for a confused agent was a bigger context window - just feed it everything, and it will always know what it needs. That assumption is backwards for almost every real workflow, and it's worth walking through why, using a concrete case: a writing agent that drafts, edits, and schedules blog posts across multiple turns.

The naive version falls apart fast

The first version of an agent like this is almost always a single stateless call: take the user's message, send it to the model, return the response. That's fine until the user says "expand section two" - now the model is supposed to know what "the draft" is, and it has no memory of anything.

The obvious fix is to dump the entire conversation history into every prompt. It works for a while. Then the user starts writing real content - five-hundred-word drafts, several rounds of revision, long feedback threads - and the context balloons, costs climb, latency spikes, and the model starts behaving less predictably the longer the conversation runs.

Tools fix the memory problem and create a new one

Adding tools - create_post, get_post, update_post, list_posts - solves the state problem. The agent can now manipulate real stored data instead of hallucinating what "the draft" contains.

But tool results get inserted directly into the model's next prompt, and that's easy to underestimate. If the agent calls get_post on a 2,500-word draft to edit it, that entire draft - title, body, everything - becomes part of the model's next input. That's not a bug; the model can't edit text it can't see. But it means every fetch adds the full document back into context, even for a request as small as "change the tone of the intro." Ask for three small edits in a row and you've re-added the same 2,500 words three times.

Trimming conversation history helps a little, but it doesn't touch this problem, because the thing bloating the prompt isn't old messages - it's the current tool result, and the model genuinely needs to see the content it's editing.

Give the model only what it needs, only when it needs it

The actual fix is structural: the model should see exactly what's relevant to the current step, not the full accumulated state of the conversation. In practice that means splitting the prompt into layers instead of one growing blob - a short rolling summary of what's happened so far, a small structured state object like {"current_draft": "9e3d", "mode": "editing"}, the user's current request, and only then, if actually needed, a tool result scoped to what's relevant.

If the user is editing section two, the tool should return section two - not the whole document. That single change, more than any amount of history-trimming, is what keeps a multi-turn editing session from ballooning into a 40,000-token prompt by the fifth exchange.

Shrinking what tools return is the real lever

Since tool results go straight into context, the size of what a tool returns matters as much as anything in the prompt template. A few patterns do most of the work: return metadata instead of full content for anything list-shaped, return a slice instead of a whole document when the user is only touching one part of it, return a diff instead of the full new version for a versioned document, let the model make a follow-up call for the rest instead of front-loading everything it might need, and return a cached summary instead of re-serializing raw text where a summary is good enough.

The mental model that actually helps here: treat the LLM like a senior editor and only put the relevant page on their desk, not the whole binder.

The payoff, and what happens without it

Done well, this shows up as lower token usage, lower latency, agents that don't quietly forget what they were doing mid-task, and - not a small thing - a system that's actually debuggable, because you can inspect exactly what the model saw at each step instead of guessing from a 30,000-token transcript.

Done poorly, the failure modes are specific and recognizable: the agent edits the wrong section, loses track of what it was doing a few turns back, hallucinates steps that didn't happen, or gets stuck looping on tool calls. Almost all of these read like model failures. Most of them are context failures wearing a model failure's mask.

The instinct to give an agent "more memory" is understandable, but memory was never the bottleneck. What a multi-turn agent actually needs is focus - the minimum information required to reconstruct the right state at the right moment, deliberately curated rather than accumulated by default.