Prompt caching lets a model provider reuse eligible, unchanged prompt content across calls. We enabled it for agents—model-driven programs that choose tools and continue over multiple turns—and expected repeated context to cost less.

The cache existed. The agent kept missing it.

The problem was not the provider. It was the shape of the conversation. Tool definitions moved around. Large observations were copied into later turns. Small utility prompts inherited an entire investigation. Context grew until emergency compaction rewrote the prefix.

Every request looked new because, structurally, it was new.

The provider setting alone was insufficient: the runtime also had to construct reusable prompt prefixes.

Why Chat Caching Advice Breaks for Agents

For a simple chat application, the reusable prefix is easy to imagine:

  • system instructions,
  • a stable conversation,
  • one new user turn.

Agents are different. A single task can add tool schemas, tool results, delegated work, notes, retries, summaries, and governance messages. The context is not only longer; it changes shape more often.

Provider caches generally reward a stable prefix. Agent runtimes often produce a turbulent prefix.

That is why a healthy cache strategy starts before the API call. You have to control how the runtime builds context.

Stability Beats Cleverness

Stable ordered instructions and schemas feed a reusable prompt prefix while evidence references and utility calls stay outside it

Keep the reusable prefix stable and large observations out of repeated prompts.

Start with deterministic ordering.

If the same tools appear in a different order on each turn, the shared prefix changes even when the capabilities are identical. If instructions are assembled from maps or registries without a stable order, logically equivalent prompts become different byte sequences.

Stable ordering does not make the agent smarter. It makes repeated work recognizable.

The same applies to instruction blocks. Put stable policy and persona material before volatile session content. Avoid rebuilding a semantically identical prefix with cosmetic differences.

Prompt caching rewards predictability. Agent architecture often celebrates dynamism. Production systems need to know where each one belongs.

Pass References, Not Copies

Agent context balloons when every layer repeats the same evidence.

A worker stores an observation. The coordinator copies it into a note. A later worker receives both the observation and the note. The final synthesizer receives all of it again.

Nothing new was learned, but the prompt pays for the same fact repeatedly.

A better pattern is to let durable working memory hold the full value and pass a compact reference through later turns. The model can retrieve the detail when needed instead of carrying every payload forever.

Illustrative shape — carry a pointer, not the payload:

{"evidence_ref": "obs-42"}          // later turns pass this
{"text": "…5,000 words of logs…"}  // instead of re-sending this

This has three benefits:

  • less context duplication,
  • a more stable prompt prefix,
  • and clearer provenance for where a fact came from.

References are not free. If retrieval is unreliable, the model may lose access to necessary evidence. Important facts still need a protected path into synthesis. The goal is not “replace everything with pointers.” It is “stop copying large values by default.”

This is the same discipline behind memory compaction for long-running agents: preserve meaning, not transcript volume.

Compact Before the Context Limit

Late compaction is expensive.

By the time the prompt approaches its limit, the runtime has already paid to carry oversized history across several turns. Emergency summarization also changes a large portion of the prefix at once, which can wipe out cache reuse.

Earlier, gradual compaction is less dramatic:

  • shrink old tool payloads after their facts are recorded,
  • keep recent decisions and unresolved questions visible,
  • preserve evidence needed for final claims,
  • and remove conversational scaffolding that no longer changes the outcome.

The right moment is not one universal percentage. Different tasks have different evidence density. A short code review and a multi-system incident investigation should not compact on the same schedule.

What matters is making compaction an operating policy, not a last-second rescue.

Isolate One-Shot Utility Calls

One easily missed source of waste was utility calls.

Agent runtimes make many small model calls that are not part of the user conversation:

  • score whether a memory matters,
  • summarize a payload,
  • check whether an answer is grounded,
  • reflect on a failed attempt,
  • classify a short piece of text.

If those calls inherit the full conversation, a tiny classification request can replay an entire investigation. It may also inherit unfinished tool-call state that has nothing to do with the utility task.

The fix is a boundary: one-shot work gets the minimum context it needs in an isolated request. It should not silently join the main session just because both operations use the same model client.

Isolation can reduce unnecessary input tokens and make the helper easier to audit. Cache reuse may improve when the helper itself has a stable prefix; isolation alone does not guarantee a cache hit.

It also forces an important design question: what inputs does this helper actually require? “The whole session” is often an accidental answer.

Budgets Must Survive Delegation

A coordinator can have a careful context budget and still lose control when delegated workers ignore it.

Explicit limits need to remain explicit across the task tree. A child agent should not silently expand a small investigative budget into a much larger default. Output summaries should be the normal return path, with full detail available through references when the coordinator needs it.

This is not only about cost. Smaller delegated outputs reduce the amount of volatile text inserted into the parent’s next prompt, which keeps more of the prefix reusable.

Tokenomics for production agents is therefore inseparable from orchestration. The budget is a property of the whole run, not just the top-level model call.

What to Measure

A cache dashboard without context-shape signals tells only half the story.

Watch for:

  • repeated calls with unexpectedly low cache reuse,
  • prompt growth between turns that learned nothing new,
  • large payloads copied across multiple roles,
  • utility calls with investigation-sized inputs,
  • major prefix churn after compaction,
  • and delegated work that returns far more text than the parent needs.

Cost is the lagging signal. Context shape is the cause.

Pair provider usage data with agent-level observability so you can attribute misses to a stage, helper, or tool boundary instead of blaming the model vendor.

Trade-Offs and Blind Spots

Cache-friendly does not mean frozen.

  • Fresh policy must override cache reuse.
  • Revoked access cannot remain embedded in a convenient prefix.
  • New evidence must be visible even when it changes the prompt.
  • Summaries can omit a detail that later becomes important.
  • Stable ordering must not become a hidden priority rule for the model.

Correctness and governance come first. The aim is to make unchanged context stable, not to stop context from changing.

Lessons Learned

  1. Prompt caching starts in context construction. Provider settings cannot repair a turbulent prefix.
  2. Deterministic ordering matters. Equivalent toolsets should look equivalent across turns.
  3. References beat repeated payloads. Keep full evidence available without carrying every copy.
  4. Compact before crisis. Gradual context hygiene costs less than emergency rewriting.
  5. Isolate utility work. A small helper call should not replay the user’s whole investigation.
  6. Carry budgets through delegation. Child work can destroy parent-level savings.

The cheapest prompt is not merely the shortest one. It is the one whose unchanged parts the system can recognize and reuse.

Are your prompt-cache misses a provider problem, or a context-shape problem? Find me on GitHub or LinkedIn.

StackGen develops AI tools for site reliability engineering (SRE), including incident triage and diagnostic workflows. Product details are at ai.stackgen.com.