proffesor-for-testing__agentic-qe/docs/implementation/adrs/ADR-088-prompt-cache-latch-fields.md

ADR-088: Prompt Cache Latch Fields for API Cost Optimization

Field	Value
Decision ID	ADR-088
Status	Implemented
Date	2026-04-01
Author	Architecture Team
Review Cadence	6 months
Source	Claude Code Internals Research (Six Thinking Hats Analysis)

---

WH(Y) Decision Statement

In the context of AQE agents making repeated LLM API calls within sessions where the system prompt and API headers are functionally identical between invocations,

facing no mechanism to stabilize API request parameters across calls, causing Anthropic's prompt cache to invalidate on every invocation even when the system prompt, model selection, and configuration are unchanged -- wasting 50-70K tokens of cacheable prefix per session,

we decided for a prompt cache latch system that freezes API headers and system prompt boundaries at session start, only recomputing when an explicit mode change occurs (model switch, permission change, or session boundary reset),

and neglected per-request header recomputation (current behavior, cache-hostile), external caching proxy (adds infrastructure), and client-side response caching (stale results risk),

to achieve 30-50% improvement in Anthropic API prompt cache hit rates, reduced token costs per session, lower latency on subsequent API calls within a session, and measurable cache performance via x-cache-read-tokens response header tracking,

accepting that latched values may become stale if underlying config changes silently (mitigated by explicit reset on mode changes), this optimization is Anthropic-API-specific and may not benefit other providers, and incorrect latching could cause subtle behavior differences.

---

Context

Problem

AQE agents make 10-100+ LLM API calls per QE session. Each call currently recomputes all request parameters from scratch: system prompt compilation, model selection, max_tokens, and auxiliary headers. The Anthropic API supports prompt caching -- if consecutive requests share an identical prefix (system prompt + initial messages), the cached portion is processed at ~10% of normal cost and with lower latency.

However, any change in the system prompt text, header values, or parameter ordering between requests invalidates the cache. AQE's current architecture rebuilds these from multiple dynamic sources (context compiler, routing decisions, domain state), producing subtly different outputs even when the semantic content is unchanged. This cache-hostile behavior means AQE sessions miss 70-90% of available prompt cache savings.

Prior Art -- Claude Code Internal Pattern

Research into Claude Code's architecture (reverse-engineered from source) reveals two specific latch fields:

• afkModeHeaderLatched -- Locks the API header for AFK/idle mode, preventing cache bust when the user goes idle
• fastModeHeaderLatched -- Locks the fast-mode header, preventing unnecessary cache invalidation

These fields are set once at the start of a session mode and only reset on explicit mode transitions. The system prompt is split at a __SYSTEM_PROMPT_DYNAMIC_BOUNDARY__ marker, where everything above the boundary is stable/cacheable and everything below is volatile.

Relationship to Existing ADRs

ADR-042 (Token Tracking) provides visibility into token consumption but does not address cache optimization. ADR-082 (Neural Model Routing) optimizes model selection but does not stabilize the selection across a session. ADR-068 (Mincut Routing) makes routing decisions per-request without caching awareness. This ADR addresses the gap between these decisions -- how to maximize cache reuse once model and context decisions are made.

---

Options Considered

Option 1: Session-Scoped Prompt Cache Latch (Selected)

Introduce a PromptCacheLatch class that captures API headers and system prompt hash at session start. Subsequent requests use the latched values. Explicit reset() on mode changes (model switch, permission change, session boundary). System prompt compiler produces deterministic output with stable ordering and a cache boundary marker.

Pros:

• 30-50% prompt cache hit improvement (measured via API response headers)
• Zero additional API calls or infrastructure
• Simple implementation (Map-based latch with freeze semantics)
• Works with existing routing and compilation pipeline

Cons:

• Stale latch if config changes without explicit reset
• Anthropic-API-specific optimization
• Requires ContextCompiler changes for deterministic output

Option 2: Per-Request Header Recomputation (Rejected -- Current Behavior)

Continue recomputing all parameters on every API call.

Why rejected: Wastes 50-70K tokens of cacheable prefix per session. At scale (100+ daily sessions), this represents significant cost and latency overhead.

Option 3: External Caching Proxy (Rejected)

Deploy a proxy between AQE and Anthropic API that caches responses.

Why rejected: Caches full responses, not prompt prefixes. Would serve stale results for different queries with same context. Adds infrastructure complexity. Does not leverage Anthropic's native prompt caching.

Option 4: Client-Side Response Cache (Rejected)

Cache full API responses locally keyed by request hash.

Why rejected: QE queries are rarely identical (different test files, coverage data). Cache hit rate would be <5%. Stale results could mask real quality changes.

Option 5: Anthropic Explicit cache_control Breakpoints (Complementary)

Use Anthropic's native cache_control: { type: "ephemeral" } breakpoint annotations in message arrays to mark cacheable boundaries explicitly in each API request.

Why complementary, not alternative: This is the actual API mechanism for prompt caching. The latch system (Option 1) ensures the content at those breakpoints stays stable across requests, maximizing cache hits. Using cache_control breakpoints without stable content (current behavior) still produces cache misses when the system prompt recompiles differently between calls. The two approaches are complementary: latch fields stabilize content; cache_control breakpoints tell the API where to cache. Implementation should use both: latch fields in PromptCacheLatch, and cache_control annotations injected by the Claude provider when constructing API messages.

---

Execution Model Applicability

This improvement applies ONLY to code paths where AQE acts as an LLM API consumer (calling Anthropic/OpenAI/etc.), NOT when AQE is being called as an MCP server:

Context	Applies?	Why
Consensus providers (src/coordination/consensus/providers/*.ts)	Yes	AQE calls Anthropic API for LLM consensus
Sub-agent calls (internal analysis)	Yes	AQE spawns LLM sub-agents for compaction, test generation
QE Quality Daemon LLM calls	Yes	Daemon calls LLM for test suggestions, pattern consolidation
MCP tool responses to external clients	No	AQE is the server, not the API caller; client manages its own cache
Model routing decisions	No	Routing selects models but does not call them directly

---

Architecture

Components

PromptCacheLatch (src/shared/prompt-cache-latch.ts)
├── latch(key, value)     -- Set once, immutable until reset
├── get(key)              -- Read latched value
├── reset(key)            -- Explicit unlatch (mode change)
├── resetAll()            -- Session boundary reset
└── getSnapshot()         -- Diagnostic: all latched key-value pairs

ContextCompiler (src/context/compiler.ts) -- MODIFIED
├── Deterministic source ordering (by priority, then alphabetically)
├── Cache boundary marker (<cache_control> tag)
└── Deduplication of repeated context fragments

Claude Provider (src/coordination/consensus/providers/claude-provider.ts) -- MODIFIED
├── Latch model, max_tokens, system prompt hash at session start
├── Use latched headers for subsequent requests
├── Track x-cache-read-tokens from API responses
└── Reset on explicit mode change signal

Latch Lifecycle

Session Start
  └── ContextCompiler produces system prompt
       └── PromptCacheLatch.latch('system_prompt_hash', hash)
       └── PromptCacheLatch.latch('model', 'claude-sonnet-4-6')
       └── PromptCacheLatch.latch('max_tokens', 8192)

API Call 1..N (within session)
  └── Use latched values (no recomputation)
  └── Track cache_read_tokens from response

Mode Change (model switch, permission change)
  └── PromptCacheLatch.reset('model')
  └── Recompute and re-latch

Session End
  └── PromptCacheLatch.resetAll()
  └── Report cache hit statistics

Telemetry

Metric	Source	Purpose
cache_read_tokens	API response header x-cache-read-tokens	Measure cache hits
cache_creation_tokens	API response header x-cache-creation-tokens	Measure cache misses
latch_resets	PromptCacheLatch events	Track mode change frequency
cache_hit_ratio	Computed: read/(read+creation)	Overall effectiveness

---

Dependencies

Relationship	ADR ID	Title	Notes
Relates To	ADR-042	Token Tracking	Token metrics capture cache performance data
Relates To	ADR-082	Neural Model Routing	Model selection feeds latch; latch stabilizes selection
Relates To	ADR-068	Mincut Routing	Routing decisions latched for session duration
Relates To	ADR-039	MCP Optimization	MCP tool calls benefit from lower API latency

---

References

Ref ID	Title	Type	Location
RESEARCH-001	Claude Code Internals Analysis	Research	docs/research/claude-code-internals-analysis.md
PLAN-001	CC-Internals Improvements Plan (IMP-05)	Implementation Plan	docs/plans/cc-internals-improvements-plan.md

---

Governance

Review Board	Date	Outcome	Next Review
Architecture Team	2026-04-01	Proposed	2026-10-01

---

Status History

Status	Date	Notes
Proposed	2026-04-01	Initial creation from Claude Code internals research
Implemented	2026-04-02	IMP-05 complete: PromptCacheLatch wired in claude-provider.ts, 13 tests passing

---

Definition of Done Checklist

Core (ECADR)

• E - Evidence: Pattern validated in Claude Code production architecture (reverse-engineered)
• C - Criteria: 4 options compared (latch, per-request, proxy, client cache)
• A - Agreement: Relevant stakeholders consulted
• D - Documentation: WH(Y) statement complete, ADR published
• R - Review: Review cadence set, owner assigned

Extended

• Dp - Dependencies: All relationships documented
• Rf - References: Research and plan documents linked
• M - Master: N/A (standalone improvement)