Agent Framework Prompt Caching: A Cost Control Guide
Agent Framework prompt caching now gives Python teams an explicit control point for deciding which GPT-5.6 prompt prefix should be cached. Microsoft’s implementation verified roughly 3100 cached tokens on the second call for a repeated prefix of 1600 tokens, while the same request without a breakpoint reported zero. That makes caching observable, but not automatically economical: cache writes are billed, unstable prefixes churn, and an old tool schema can preserve the wrong behavior. The Microsoft Agent Framework 1.12.1 release turns prompt caching into a production configuration decision rather than a hidden provider optimization.
Key Takeaways
- Agent Framework 1.12.1 exposes explicit cache breakpoints for GPT-5.6 OpenAI clients.
- Breakpoints belong after stable instructions, policies, examples, and tool schemas.
- Cache writes can cost money, so a hit-rate dashboard matters more than theoretical savings.
- Repeat tests should verify cached tokens, output quality, latency, and model compatibility.
- Version cached prefixes and invalidate them when instructions, tools, permissions, or data boundaries change.
What changed in Agent Framework prompt caching?
The Python 1.12.1 release added two related controls to the framework’s OpenAI clients. First, typed prompt_cache_options can travel with chat options. Second, a prompt_cache_breakpoint stored on a content part can survive conversion into an OpenAI Responses or Chat Completions request. Before the change, the clients could silently drop the per-part breakpoint, which meant the application appeared configured while the provider never received the control.
The change covers text, image, audio, and file parts supported by the corresponding API. It also preserves the existing request shape when caching is not enabled, making the feature opt-in. That last detail matters for rollout safety: teams can add one measured caching path without changing every agent request. The implementation pull request records 7 changed files, 439 additions, 26 deletions, unit coverage, and a usage sample that displays cached-token counts.
Why explicit breakpoints matter now
The original feature request states the economic reason directly: newer GPT-5.6 models support explicit breakpoints, and cache writes are billed. An implicit cache is easy to treat as “free acceleration.” A paid cache write forces a different question: is this prefix stable and frequently reused enough to recover the write cost?
An explicit breakpoint divides the request into two conceptual regions. The prefix before the breakpoint is the candidate reusable asset. The suffix after it is the changing task: a customer question, retrieved documents, a current date, or a tool result. If volatile material lands before the breakpoint, the application produces a stream of distinct cache entries instead of repeat hits.
The framework does not choose the boundary
Agent Framework forwards the option; it cannot know which instructions are stable for your product. The application owner still decides whether system policy, a long style guide, few-shot examples, tool definitions, connector descriptions, or project context belong in the prefix. That is why this release is useful but not self-optimizing.
A cache breakpoint is a version boundary for agent behavior, not merely a token-saving switch.
Where should an agent place the cache breakpoint?
Put the breakpoint immediately after the longest prefix that is both stable and shared across enough requests to justify reuse. For a support agent, that might include the system contract, escalation policy, response format, approved examples, and stable tool schemas. It should usually exclude the customer message, retrieved account records, current ticket history, live search results, and timestamps.
| Prompt block | Cache before breakpoint? | Reason | Invalidate when |
|---|---|---|---|
| System role and safety policy | Usually | Reused across many calls | Policy or permission changes |
| Tool schemas and connector rules | Usually | Large and stable within a release | Tool signature or approval behavior changes |
| Few-shot examples | Sometimes | Valuable when reused at high volume | Evaluation shows drift or examples change |
| Retrieved documents | Rarely | Content varies by user and task | Source version or access scope changes |
| User request and live tool output | No | Unique to the current run | Every request |
Start with one breakpoint. Multiple boundaries can improve reuse for layered prompts, but they also multiply the states you must explain, measure, and invalidate. A single stable-prefix experiment gives a clean baseline: first call writes, second equivalent call should report cached tokens, and a deliberately changed prefix should miss.
Keep sensitive and scoped data out of shared prefixes
Caching cannot weaken authorization rules. Do not place user-specific records, private project instructions, retrieved files, or connector tokens into a prefix that could be reused across tenants or permission scopes. Even when the provider isolates caches correctly, mixing data scopes makes application reasoning and incident review harder. The safest design keeps the cached prefix product-owned and places user-owned context after the boundary.
In practice, I treat the prefix like a build artifact. It gets a content hash, a human-readable version, an owner, and a list of inputs. If a tool schema changes from read-only to write-capable, the prefix version changes even if the prose does not. That makes cache invalidation follow the same release decision as the behavior change.
How do you measure whether prompt caching saves money?
Measure four things together: cache writes, cache reads or cached input tokens, uncached input tokens, and completed-task quality. A cache hit that saves latency but increases total cost is not a cost optimization. A high hit rate that preserves an obsolete instruction is not a product success. The production decision needs both economics and behavior.
Microsoft’s live verification offers a useful test pattern. The pull request describes a repeated prefix of about 1,600 tokens and a second request reporting approximately 3,100 cached tokens across GPT-5.6 Luna, Sol, and Terra on the Responses API, plus Luna through Chat Completions. Without the breakpoint, the same setup reported zero cached tokens. Treat those figures as implementation evidence, not a universal ratio; provider accounting and prompt structure determine your own result.
Track these metrics by prefix version and model:
- Cache-write requests and written-token volume.
- Cache-hit requests and cached-token volume.
- Hit rate after the first request in each deployment cohort.
- Input cost per completed task, including paid writes.
- p50 and p95 response latency.
- Evaluation score and tool-call success rate.
The OpenAI prompt caching guide is the source of truth for current model behavior, supported controls, minimums, retention, and usage fields. Its automatic-caching guidance uses a threshold of 1024 tokens, another reason short-agent prompts need measurement before extra machinery. Provider details can change, so keep price and retention assumptions in configuration or a dated cost model instead of hard-coding them into an evergreen architecture diagram.
Run a controlled break-even test
Choose one high-volume agent route with a large stable prefix. Replay at least 20 requests against caching off and caching on. Keep model, temperature, tools, and inputs fixed. The first cached run establishes write behavior; subsequent calls establish reuse. Then calculate cost per successful task, not cost per API call, because a cheaper response that causes more retries is false economy.
Use ZeroTwo to compare the same evaluation brief across models and preserve the decision record beside sources and outputs. The useful workflow is not “ask which model is cheapest.” It is “hold the task and rubric constant, then compare total cost, latency, tool success, and answer quality.”
Which failure modes should teams test?
The first failure is a silent miss. A framework option may be present in application code but absent in the provider request, as happened before 1.12.1 forwarded the breakpoint. Verify the outgoing request shape in a safe development trace and confirm the usage response reports cached tokens. Do not infer success from lower latency alone.
The second failure is unsupported-model behavior. Microsoft’s implementation testing reports that opting in on an older model surfaced the API’s own HTTP 400 response stating that prompt_cache_breakpoint was unsupported. This is the right kind of failure because it is visible. Keep model capability metadata explicit and fail configuration early rather than retrying without the breakpoint and hiding the mismatch.
The third failure is prefix churn. Dates, request IDs, user names, dynamic tool descriptions, or retrieved context before the breakpoint can make nearly every write unique. A dashboard may show cache writes without meaningful repeat reads. Normalize or move volatile fields after the boundary; do not add a fallback cache layer that masks the wrong prompt structure.
The fourth failure is stale behavior. Tool schemas, permission rules, system instructions, and examples evolve. If a deploy changes one of those inputs without changing the prefix version, an old cache entry may survive longer than the team expects. The Microsoft running-agents documentation shows how provider-specific run options enter the framework; your release process must connect those options to prompt and tool versions.
Test replay and tool-call pairing
Python 1.12.1 also fixed stateless replay of reasoning-paired tool calls and preserved Gemini thought signatures across function-call replays. Those are separate release items, but they highlight the same operational truth: agent requests are structured histories, not plain text blobs. Add regression cases for tool calls, files, images, audio parts, system messages, and replayed turns when you enable caching.
Microsoft positions Agent Framework as a unified agent foundation across provider and orchestration paths in its release-candidate announcement. A unified surface is useful only if provider-specific behaviors remain visible enough to test. Cache telemetry, request-part shape, and model errors should stay observable through that abstraction.
Make the rollback equally explicit. Keep a configuration switch that disables the breakpoint without changing the prompt itself, then compare the same evaluation cohort before and after the switch. If cost, latency, or quality moves unexpectedly, preserve the trace identifiers and prefix version needed to explain why. The rollback should remove the caching control, not substitute a second hidden cache that makes the primary provider behavior impossible to diagnose.
Frequently Asked Questions
What is Agent Framework prompt caching?
Agent Framework prompt caching is the Microsoft Agent Framework support for forwarding provider cache options and explicit breakpoints with OpenAI requests. In Python 1.12.1, applications can mark the end of a reusable GPT-5.6 prompt prefix and inspect usage telemetry on repeated calls. The application still owns boundary placement, versioning, measurement, and invalidation.
Where should I put a prompt cache breakpoint?
Place the breakpoint after the longest product-owned prefix that stays stable across many requests, such as system instructions, policies, examples, and tool schemas. Keep user messages, retrieved documents, timestamps, live tool results, and permission-scoped records after the breakpoint. Verify the choice with cache-hit telemetry and a controlled prefix-change test.
Do GPT-5.6 prompt cache writes cost money?
Microsoft’s feature request and implementation state that cache writes are billed for the newer GPT-5.6 behavior, which is why explicit placement matters. Consult current OpenAI pricing and prompt-caching documentation before calculating savings. Measure write cost, cached reads, uncached input, latency, retries, and task success together because a high hit rate alone does not prove lower cost.
How can I tell whether the breakpoint worked?
Repeat a request with the same stable prefix and inspect the provider usage fields for cached tokens. Also capture the outgoing request structure in a safe development trace to confirm the breakpoint reached the API. Microsoft’s test reported cached tokens on the second request with a breakpoint and zero without one, which is a practical regression pattern.
When should I invalidate a cached agent prefix?
Invalidate or version the prefix whenever system instructions, safety policy, examples, tool schemas, connector permissions, model compatibility, or data-scope rules change. Also invalidate when evaluation reveals that a cached example or instruction causes drift. Store a prefix hash and release version so cost and quality metrics can be attributed to the exact behavior users received.
What Comes Next
Agent Framework 1.12.1 closes an important plumbing gap: the cache control written in application code can now reach supported OpenAI requests. The next work belongs to operators. Define a stable prefix, keep scoped data outside it, measure paid writes against repeat reads, and make invalidation part of the agent release.
The teams that benefit most will not have the most breakpoints. They will have the clearest evidence connecting one breakpoint to lower cost per successful task without stale instructions or hidden model fallbacks.
Agent Framework prompt caching is now controllable; disciplined prefix ownership is what makes it economical.