## Summary
- Productizes the Claude SDK Python and TypeScript showcase demos with
LangGraph-parity frontends.
- Wires the Claude demo backends through the official Claude Agent
SDK/AG-UI adapter paths using `claude-sonnet-4.6`.
- Keeps Claude integration docs hidden for this PR and excludes
generated/authored docs artifacts from scope.
## Why
The goal is to bring the productized LangGraph demo surface to Claude
Agents SDKs without publishing integration docs in this pass. This keeps
the PR focused on local showcase demos, runtime behavior, fixtures, and
validation support.
## How
- Ported the demo frontend surfaces and local shell-dojo support for
Claude SDK Python/TypeScript.
- Added official Claude SDK adapter/backend wiring plus real-Claude
local compose support.
- Updated Claude aimock fixtures and validation ratchets for the
expanded demo set.
- Set both Claude manifests to `docs_mode: hidden` and removed docs
setup/snippet artifacts from the PR scope.
## Problem
\`@ag-ui/a2ui-middleware\` v0.0.10's \`getOperationSurfaceId()\` reads
only the A2UI v0.9 NESTED op format:
\`\`\`json
{"version": "v0.9", "createSurface": {"surfaceId": "...", "catalogId":
"..."}}
\`\`\`
The integrations were emitting the legacy FLAT format:
\`\`\`json
{"type": "create_surface", "surfaceId": "...", "catalogId": "..."}
\`\`\`
Result: all ops fell back to the \"default\" surface key → frontend
never mounted the named surface → \`surface-missing\` failure on
\`declarative-gen-ui\` across ~11 integrations.
## Fix
Convert all a2ui op builders and inline ops to the nested v0.9 format
in:
- \`tools/generate_a2ui.py\` — 9 integrations (agno, claude-sdk-python,
crewai-crews, langgraph-fastapi, langgraph-python, langroid, llamaindex,
pydantic-ai, strands)
- \`tools/search_flights.py\` — 11 integrations (ag2, agno,
claude-sdk-python, crewai-crews, langgraph-fastapi, langgraph-python,
langroid, llamaindex, ms-agent-python, pydantic-ai, strands)
- \`src/agents/a2ui_fixed_agent.py\` / \`a2ui_fixed.py\` /
\`beautiful_chat.py\` — agno, crewai-crews, langroid, pydantic-ai
Already-correct integrations skipped: google-adk,
ms-agent-python/generate\_a2ui.py, ag2/generate\_a2ui.py.
**Total: 25 files changed.**
## Verification
Zero flat-format ops remain in non-comment/non-test code. 96 occurrences
of \`"version": "v0.9"\` present in changed integrations (excluding
google-adk which was already correct).
## Red→Green
\`bin/showcase test\` runs against Docker containers — requires
infrastructure startup. The structural change is a mechanical
search-and-replace: \`getOperationSurfaceId()\` in \`a2ui-middleware\`
v0.0.10 reads \`op.createSurface?.surfaceId\` (nested), which is exactly
what these changes now emit. The old flat \`op.surfaceId\` path is not
read at all by the middleware, explaining the surface-missing fallback.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
- Commit `7c3edca2b7` (May 11) changed the `autoPrompt` strings in all
`sample-attachment-buttons.tsx` from `"describe the sample image"` /
`"summarize the sample document"` to `"can you tell me what is in this
demo image I just attached"` / `"can you tell me what is in this demo
pdf I just attached"` — but never updated the 41 aimock fixture files
- Every integration that uses the auto-send pattern was sending a
message matching no fixture → strict MISS → 404 → agent error banner →
`dom-missing` / `done-signal-missing` D6 timeouts
- Fixed 19 `multimodal.json` D6 fixtures, 20 `agentic-chat.json` D6
fallback fixtures, 1 shared D5 `multimodal.json`, and the
`split-fixtures.ts` router
## Red-Green Proof
**RED (before fix) — `langgraph-typescript:multimodal --d6 --direct`:**
```
turn 1: TIMEOUT dom-missing (60s) — aimock 404, no assistant text rendered
turn 2: TIMEOUT dom-missing (60s) — same
```
**GREEN (after fix) — `langgraph-typescript:multimodal --d6 --direct`:**
```
turn 1: PASS — assistant text "The attached image is the CopilotKit logo..." settled
turn 2: PASS — assistant text with document summary settled
```
## Remaining failures (out of scope, separate issues)
- `ms-agent-python`, `crewai-crews`: Python backend
`ChatClientException` when receiving binary (image/PDF) AG-UI content
parts — same class as active `wt-pydantic-multimodal` worktree
- `built-in-agent`, `claude-sdk-python`: DOM-inject only (no
`agent.addMessage` / `copilotkit.runAgent` auto-send) — probe design
mismatch, not a fixture issue
## Test plan
- [x] `langgraph-typescript:multimodal --d6 --direct` RED before / GREEN
after
- [ ] D6 repro sweep after merge to confirm cluster clears for auto-send
integrations
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Commit 7c3edca changed sample-attachment-buttons.tsx across all integrations
to auto-send via agent.addMessage with autoPrompt strings:
- "can you tell me what is in this demo image I just attached"
- "can you tell me what is in this demo pdf I just attached"
But the d5 harness fixture and all 19 d6 per-integration multimodal.json
fixtures still matched on the old strings:
- "describe the sample image"
- "summarize the sample document"
Aimock received requests with the new prompts, found no match, returned
a STRICT 404, and the agent emitted a streaming error back to the UI
(exact symptom: "An internal error has occurred while streaming events").
Also update agentic-chat.json across all 20 integrations (those files had
duplicate fallback entries for the old prompts) and fix split-fixtures.ts
to route the new strings to the "multimodal" feature bucket.
Local RED: ms-agent-python and crewai-crews both fail with fixture-miss
status=miss before this change.
Local GREEN: langgraph-typescript passes after this change (both turns
settle with "image" / "document" keywords confirmed in transcript).
Remaining failures after this fix are pre-existing Python backend issues
(ChatClientException on binary content parts in ms-agent-python; CrewAI
flow failure on binary content in crewai-crews) — unrelated to fixture
keys and tracked separately in the pydantic-ai multimodal work.
Semantically-identical reformat of `renovate.json` (compact single-line,
same content) to nudge Mend/Renovate to re-evaluate against the NEW
github-actions-only central preset.
Background: Renovate hasn't re-scanned since 2026-06-16 due to the
cached-old-preset gotcha, so no Dependency Dashboard has appeared yet
after the onboarding cutover (#5031). A no-op touch of the consumer
config forces re-evaluation.
Content is unchanged:
```json
{"$schema":"https://docs.renovatebot.com/renovate-schema.json","extends":["local>CopilotKit/renovate"]}
```
No behavior change; npm remains human-controlled.
Migrates this repo's Renovate configuration to extend the org-wide
central config at https://github.com/CopilotKit/renovate.
Phase 1 of the migration ([Notion
plan](https://www.notion.so/3613aa38185281a38863fcff2907021c)) scopes
Renovate to the github-actions ecosystem only; npm/pip remain on
Dependabot.
Replaces the previous renovate.json (which had `extends:
config:recommended` plus a global "ignore all packages initially" rule
that effectively disabled the prior Renovate install). The new config
inherits from the org-wide central preset, which is scoped to
github-actions for Phase 1.
Pre-existing open Renovate PRs (#4751, #3295) from the prior install can
be closed separately once this lands.
## Root Cause
`agno 2.6.20` removed `agno.os.interfaces.agui.utils`. The floating
`agno>=2.5.17` pin in `requirements.txt` caused staging to pull the
breaking version on the next build, causing a startup failure.
## Red-Green Proof
**RED** — with `agno>=2.6.20` installed:
```
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'agno.os.interfaces.agui.utils'
```
**GREEN** — with `agno==2.6.19` installed:
```
GREEN: all 3 symbols OK
```
(Symbols confirmed: `async_stream_agno_response_as_agui_events`,
`extract_agui_user_input`, `validate_agui_state`)
## Changes
- `showcase/integrations/agno/requirements.txt`: pinned `agno>=2.5.17` →
`agno==2.6.19` (exact pin, last version with `agui.utils`)
- `showcase/integrations/agno/src/agent_server.py`: added TODO comment
at line 73 import site noting migration to agno 2.6.20+ API is a
follow-up; no structural changes to imports
## Follow-up
Migration of `agent_server.py` imports to the agno 2.6.20+ API (once the
replacement for `agui.utils` is identified) is tracked in the TODO
comment at line 73.
## Note on CI
The `validate-pins` CI check will likely flag pre-existing non-exact
pins across ~15 other integrations (`openai ^5.9.0`, `crewai` ranges,
etc.). This is pre-existing debt not introduced by this PR.
## What & why
Resolves [OSS-132](https://linear.app/copilotkit/issue/OSS-132).
Investigated with systematic-debugging; every conclusion verified
against the **real** OpenAI Responses API.
**Net change: a TanStack version bump only.** No showcase schema change.
- `@tanstack/ai` `0.18.0` → `0.35.0`
- `@tanstack/ai-openai` `0.9.1` → `0.15.6`
- `package-lock.json` regenerated (Dockerfile uses `npm ci
--legacy-peer-deps`)
## The bug
The built-in-agent showcase 400s on every prompt against real OpenAI.
The state tools (`AGUISendStateSnapshot` / `AGUISendStateDelta` /
`set_steps`) declare arbitrary payloads as `z.any()`, which serializes
to a **typeless** JSON-Schema property (`{ "description": ... }`, no
`"type"`).
The old `@tanstack/openai-base`'s `isStrictModeCompatible()` only
screened for `oneOf/allOf/not/$ref/$defs`, so it missed the missing
`type`, sent the tool with `strict: true`, and OpenAI rejected it:
```
400 Invalid schema for function 'AGUISendStateSnapshot':
In context=('properties','snapshot'), schema must have a 'type' key.
```
This was **masked in production** because the deployed showcase runs
against aimock, which replays fixtures without validating the request
schema — a raw `curl` to prod returns a clean `RUN_FINISHED`, green for
the wrong reason.
The ticket's original framing (zod3/zod4 drift → typeless *root*, `got
"None"`) was already fixed by the zod-4 migration; this is the same
symptom one layer down (typeless *property*).
## The fix is upstream
`@tanstack/ai-openai@0.15.6` (via `@tanstack/openai-base@0.9.2`) fixes
`isStrictModeCompatible`: it now detects typeless / `z.any()` properties
and sends `strict: false`. OpenAI accepts typeless properties under
`strict: false` — so `z.any()` works again with no schema change on our
side.
(`@tanstack/ai-openai@0.15.5` also dropped `@tanstack/ai-client` from
its peerDependencies, so no `ai-client` dep is added.)
## Verification (real OpenAI, gpt-4o)
| Probe | Result |
|---|---|
| Typeless property, `strict: true` (raw OpenAI) | **400** — `schema
must have a 'type' key` |
| Typeless property, `strict: false` (raw OpenAI) | **ACCEPTED** —
confirms it was the strict flag, not the schema |
| `z.any()` tool on old adapter (0.9.1/0.15.4) | adapter sends `strict:
true` → **400** |
| `z.any()` tool on new adapter (0.15.6) | adapter sends **`strict:
false`** → **ACCEPTED**, model calls the tool |
| All 3 `z.any()` state tools attached, new adapter | **ACCEPTED**, no
400 |
## Not covered here
The showcase's aimock + Playwright e2e suite was **not** run locally
(this worktree has no installed toolchain). CI runs it on this PR;
please confirm the gen-ui / shared-state demos still pass before merge.
---
_Branch history shows an interim `z.string()` workaround that was
reverted once the upstream fix shipped; the net diff is the version bump
only. Squash-merge recommended._
One-line fix: `bin/railway`'s `run_staging_probe` invoked `npx --yes tsx
verify-deploy.ts` from the repo root with no `chdir`, so under Node 22
tsx failed to resolve (MODULE_NOT_FOUND in the ESM preload) → the
promoter misread it as 'staging not green' → hard REFUSE. This
tier-gated all prod promotes (incl. the langgraph fix in #5825). Fix
adds `chdir: File.expand_path("../scripts", __dir__)` so tsx resolves
from `showcase/scripts/node_modules`.
Red-green: from repo root `npm ls tsx` is empty and `npx tsx` crashes;
from showcase/scripts it resolves (tsx declared in
showcase/scripts/package.json).
NOTE: the agno `<2.6.20` pin (originally bundled here) was split out —
it edits a requirements file which trips the fleet-wide validate-pins
ratchet (pre-existing non-exact-pin debt across ~15 integrations).
Tracking separately.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
agno 2.6.20 removed agno.os.interfaces.agui.utils; the floating
agno>=2.5.17 pin in requirements.txt caused staging to pull the
breaking version. Pinned to 2.6.19 (last version with the module).
Added TODO comment at the import site for future migration.
Without chdir, npx resolves tsx from the repo root where it is not installed.
tsx is a dev dependency of showcase/scripts; chdir ensures npx resolves it correctly.
## What & why
Showcase services were being killed on Railway. Root causes, all fixed
here:
1. **langgraph-python / langgraph-fastapi — watchfiles log flood →
Railway 500-logs/sec replica kill.** `langgraph dev` ran with
hot-reload, emitting "1 change detected" per request; under D6 probe
fan-out this blew past Railway's 500 logs/sec cap and killed the
replica. Fix: `--no-reload` + `export
LANGGRAPH_DISABLE_FILE_PERSISTENCE=true` (also stops unbounded
pickle-state OOM).
2. **langgraph-typescript — `FileSystemPersistence` RangeError crash
loop.** `@langchain/langgraph-api` serialized unbounded thread state via
`JSON.stringify`; past V8's ~512MB string ceiling it threw `RangeError`
in a timer, hung the event loop, and the watchdog kill-looped (state
persisted on disk, so restarts re-crashed). Fix: boot-purge stale state
+ a **size-gated** restart (checks dir size, only restarts near the
ceiling — no in-flight-wiping timer, no unpinned `/internal/truncate`).
3 & 4. **Per-request proxy log flood across all integrations.**
`[copilotkit/route] POST` + `Response status` logged on every
sub-request, unconditionally, in 19 `route.ts`. Fix: gate them behind
`SHOWCASE_ROUTE_DEBUG` (off in prod) — **but keep non-2xx responses
logged unconditionally** so production errors stay visible, and gate the
health-probe GET too.
## Verification
- Every fix carries local red-green. langgraph-typescript entrypoint:
**18 mutation-sensitive subprocess tests** (reversed comparison / broken
du|awk / wrong-kill-target all caught; orphan-cleanup reaped). route.ts
gating verified on the real Next.js surface across ≥3 integrations
(non-2xx logged, 2xx+health gated, `SHOWCASE_ROUTE_DEBUG=1` restores
verbose).
- Code review: Round 1 (7 agents) → fixes → Round 2 (7-agent
confirmation) → fix → Round 3 (3-lens targeted) → fix → converged to
zero mandatory findings.
## ⚠ Before merge
The two entrypoint changes (`--no-reload` +
`LANGGRAPH_DISABLE_FILE_PERSISTENCE` on pinned `langgraph-cli 0.4.21`)
are **source-verified but could not be run locally** (the langgraph
packages are on a private index; `0.4.21`'s `--no-reload` was confirmed
only in public `0.4.3`). **Requires live-Railway validation** (branch
deploy: boots, serves 200, no watchfiles spam, no pickle files) before
merge. Kept as a **draft** until validated and the maintainer approves.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Gates per-request POST + 2xx Response-status + GET health-probe logs behind SHOWCASE_ROUTE_DEBUG across 19 integrations to stay under Railway's 500-logs/sec cap, while logging non-2xx responses unconditionally so production errors stay visible.
Boot-purge of stale .langgraph_api state plus a size-gated restart (du > threshold -> kill agent -> container restart -> purge), replacing an in-flight-wiping periodic truncate loop. Adds mutation-sensitive subprocess tests for the watchdog.
## What this does
Re-platforms the banking showcase's self-learning off the **abandoned**
offline-distill path (which targeted the now-closed Intelligence #192
`record → /annotate → sl-worker → /knowledge` pipeline) onto the
**shipped** memory substrate (`libs/memory`, Intelligence #294/#321).
The agent now saves a demonstrated over-limit procedure as a
`project`-scoped, `procedural` memory via `save_memory`, and
`recall_memory`s it at the start of later over-limit requests — so a
**fresh thread, or a different user on the same team, completes the
approval unaided**. That's the FOR-149 durable cross-thread + cross-user
proof.
## Verified live (local stack)
- Vendored memory-enabled Intelligence stack comes up healthy; `POST
/api/memories` → `201`, `/recall` → `200`, and
`save_memory`/`recall_memory`/`forget_memory` MCP tools attach
(`SL_ENABLED` + embedder).
- Cross-user: a project memory saved by one user recalls for a different
user.
- App boots in Intelligence mode; OSS fallback (`InMemoryAgentRunner`)
untouched and still the default.
## Changes
- **`docker-compose.yml`** — vendored stack cloned from the proven
`memory-chat` recipe (postgres/pgvector, redis, minio, TEI, composite
app-api + gateway). Hardened during a real bring-up: `minio-init`
DNS-race retry, **pluggable embedder** (`MEMORY_EMBEDDINGS_URL` + `tei`
dependency `required:false`, so RAM-constrained / Apple-Silicon machines
can point at a host TEI), and non-colliding `715x` host ports.
- **Runtime** (`route.ts`) — Intelligence branch gains `licenseToken` +
lock config + `generateThreadNames`; recall-first / save-on-teach
prompt; `recall_memory`/`save_memory` added to the tool list.
- **`saveLearnedWorkflow`** resolves a `status: saved` result that
drives the agent's `save_memory` call (Option A — agent-initiated),
keeping the already-approved guard.
- **Removed** the dead `record-user-action` `/annotate` seam (kept the
visual `useRecording` UX).
- **README** rewritten: one-command stack, host-TEI override, ports,
`.env`, cross-thread + cross-persona walkthrough, testing notes. Adds
`.env.example`.
## Tests
- **Deterministic E2E (CI gate):** `e2e/memory-learning.spec.ts` +
aimock fixtures — agent LLM served by `@copilotkit/aimock` (fixtured
`recall_memory` → exception → approve tool calls) against the **real**
local memory backend; asserts a fresh thread unlocks from recalled
memory with no recording offer.
- **Real-LLM drift smoke (manual, non-gating):**
`scripts/memory-drift-smoke.mjs`.
## ⚠️ Why draft — needs a green E2E run
The Task 7 E2E is **authored + statically validated** (`playwright test
--list` compiles spec + config; fixtures/JSON/launcher all valid) but
**has not had a green run yet** — it needs `@copilotkit/aimock`
installed, the docker stack up, and the dev server in Intelligence mode
(a 4-process orchestration). Each E2E file carries a `VERIFY ON FIRST
GREEN RUN` checklist (aimock fixture schema/launch API, chat + HITL
selectors, the `sequenceIndex` ordering key). Marking draft until that
passes.
## Out of scope (deferred)
- Per-run demo reset for a repeatable public embed (user-scope memory /
periodic DB reset / dashboard control).
- Managed-Intelligence target: PRD/handoff prefer
`api.intelligence.copilotkit.ai`; this PR ships the local-vendored stack
per direction. Reconciling for the V1 website/Railway deploy is a
follow-up.
- Pre-existing demo `tsc` looseness (`page.tsx`, `copilot-context.tsx`)
— untouched.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---
## Update — booth-bundle pass (2026-06-30)
Follow-up to make the demo booth-ready and reproducible from the
CopilotKit repo by teammates. Four commits on top of the above:
- **`/api/v1/dev/reset` now clears durable memory**, not just the
transaction store — so the full *fail → teach → succeed* arc replays for
each booth visitor. New scope-complete `forgetAllMemories` helper
enumerates via a bare `GET /api/memories` (the backend `400`s on
`?scope=` filters, so a single bare GET is inherently scope-complete)
and `DELETE`s each id; the route returns
`{ok,reset:["store","memory"],forgot:N}`, or a `502` on partial failure
so a half-reset state is never silently used. Live-validated
(`forgot:2`). **This resolves the "per-run demo reset" item listed as
deferred above.**
- **Memory `kind` migrated `operational` → `procedural`** to match the
Intelligence demo branch's current schema (`semantic | episodic |
procedural`). *(Supersedes the "operational" wording earlier in this
description.)*
- **Fixed the aimock E2E launcher** — `new LLMock({ fixtures })` ignores
`options.fixtures`, so the mock was serving 0 fixtures; now registers
via `addFixtures()`.
- **E2E status:** `test:unit` green; `test:self-learning` — the Glass
Engine inspector test passes; the autonomous-recall test has a known
aimock fixture-sequencing flake (harness-only, not a demo/backend bug).
The booth relies on the manual real-LLM arc.
**Build the Intelligence backend from `david/for-162-splat-demo`, not
`main`.** Verified by building both: the demo branch boots healthy and
runs the arc; `main` crash-loops with this compose — it requires the new
`INTELLIGENCE_DEPLOYMENT_MODE=self_hosted` auth contract (rejecting the
`DEPLOYMENT_MODE` + `DEFAULT_ORGANIZATION_ID` env this compose sets) and
its memory `kind` vocabulary is `topical/episodic/operational`.
Targeting `main` is a separate migration (compose auth env + org-seed
model + `kind` taxonomy). A full local-setup runbook exists for
teammates (internal Notion).
---
## Update — CR pass (2026-07-03)
A 7-agent review-and-fix loop converged (2 rounds + a bucket-(c)
promotion audit; 0 mandatory findings remaining). Four fixes landed,
each its own commit; `tsc`, unit tests (41/41), eslint, and `next build`
all green:
- **Recorder feed** — `handleApprove` in `transactions-list.tsx` and
`pending-approvals-chat.tsx` called `logStep()` *before*
`beginRecording()`, so the "Approved the charge" line was silently
dropped (`logStep` no-ops when inactive; `beginRecording` then resets
the feed). Reordered to `beginRecording → logStep → endRecording`; added
`recording-context.test.tsx` with red-green coverage.
- **Docs** — corrected the memory-learning spec path `tests/e2e/` →
`e2e/` (README, `.env.example`, smoke script), and the
top-of-description memory `kind` `operational` → `procedural`.
- **docker-compose header** — infra host-port comments corrected
`705x/706x` → the actual `715x/716x` mappings.
Deferred (pre-existing, out of this PR's subject; candidates for a
follow-up): the dual/divergent "current page" agent readable
(`copilot-context.tsx:96` vs `layout.tsx:147`), and the `PUT
/api/v1/transactions/[id]` error-swallow returning `undefined`.
---
## Update — migration to Intelligence `main` + presenter reset +
Apple-Silicon fresh-setup (2026-07-06)
This branch now targets Intelligence **`main`** (the earlier sections
assumed the `david/for-162-splat-demo` branch). Changes on top of the
above:
**Migration to `main`'s contract**
- **Compose auth:** `INTELLIGENCE_DEPLOYMENT_MODE=self_hosted` (legacy
`DEPLOYMENT_MODE` / `DEFAULT_ORGANIZATION_ID` removed — `main`'s
`loadAuthEnv` rejects them).
- **Memory `kind` renamed `semantic|procedural` →
`topical|operational`** to match `main`'s closed enum (`topical |
episodic | operational`). ⚠️ *This supersedes the earlier "migrated
operational → procedural" note (that was for the old branch): the
over-limit procedure is now **`operational`**, general facts
**`topical`**.*
- **Self-hosted memory is license-gated on `main`.** New
`scripts/mint-dev-license.mjs` (`pnpm mint-dev-license --write`) signs
an enterprise dev license (`features.memory=true`) with a throwaway key
and bakes the public half via `BAKED_LICENSE_KEYS_JSON`, which the local
(unbaked) app-api trusts. Drives the signer from the private
Intelligence source via `INTELLIGENCE_REPO` — no signing code vendored
into this public repo. Managed-Intelligence users instead supply a
CopilotKit-issued token and omit the baked key.
**Presenter reset button** (finishes the deferred "per-run demo reset",
now UI-driven)
- New `PRESENTER_RESET_ENABLED` flag gates **both** a sidebar reset
button **and** the `/api/v1/dev/reset` endpoint (403/hidden by default —
safe-off for public hosts).
- Full clean slate: re-seeds transactions + forgets memory for **both**
seeded personas (`SEEDED_USER_IDS`), with partial-progress reporting on
a mid-clear failure. TDD; spec + code-quality reviewed.
**Apple-Silicon fresh-setup fix**
- The bundled amd64 `tei` crash-loops under arm64 emulation (Candle
backend unavailable → ONNX/ORT backend → 404 on ONNX files
`Qwen3-Embedding-0.6B` doesn't publish). Gated it behind the
`cpu-fallback` profile (a bare `up` skips it), and added `run-demo.sh`
that runs a native Metal TEI on Apple Silicon (same 1.9.3 + model →
byte-identical embeddings) and the docker `tei` on amd64/CI. README
diagnosis corrected (was mis-attributed to OOM).
**Verification:** 55 unit tests green, `tsc` clean, `eslint` clean. A
from-scratch run (Intelligence `main` rebuild + clean `pnpm install` +
native TEI) was validated end-to-end — memory save/recall through the
native embedder, teach→recall arc, and reset all working. The
deterministic aimock e2e still has the known fixture-sequencing flake
(see follow-up comment below).
## Problem — the leak
The v2 runtime's `shouldForwardHeader` forwarded `authorization` **and
any header whose name starts with `x-`** onto the outgoing agent call.
In a real deployment the inbound request has already traversed a
browser, CDN/edge, load balancer, and hosting platform — each stamping
its own `x-*` headers — so the wide `x-*` wildcard silently forwarded:
- **Hop-by-hop / topology:** `x-forwarded-for`, `x-real-ip`,
`x-forwarded-proto/host/port`
- **Cloud / CDN tracing:** `x-amzn-trace-id`, `x-amz-cf-id`,
`x-cloud-trace-context`, `x-azure-*`, `x-fastly-*`, `x-request-id`
- **Platform-injected:** `x-vercel-*`, `x-middleware-*`
- **CopilotKit Cloud platform credential:**
`x-copilotcloud-public-api-key`
The last item is a real credential-exfiltration concern: a platform key
scoped to Copilot Cloud reaching a third-party agent URL. This is the
**breadth** half of #5712 (option 3); the **precedence** half was fixed
in #5782.
## Design — denylist default + config knob, both paths
- **Default denylist (safe default).** Keep the `authorization` + `x-*`
base eligibility, but strip a curated, greppable set of known
infra/proxy/platform headers (exact names + prefix families) before
forwarding. Legitimate custom `x-*` application headers (`x-tenant-id`,
`x-api-key`, …) keep flowing untouched. The authoritative list is a
single exported constant in `header-utils.ts`.
- **Configurable policy (`forwardHeaders` runtime option).**
- `useDefaultDenylist?: boolean` (default **true**) — `false` restores
the previous wide-open behavior.
- `deny?` / `denyPrefixes?` — extend the default denylist.
- `allow?` — opt into strict allowlist mode (only listed headers
forward).
- **Resolve once.** The constructor resolves `forwardHeaders` into a
`forwardHeadersPolicy: ResolvedForwardHeadersPolicy` field (mirroring
the existing `debug` → `ResolvedDebugConfig` resolve-once), exposed on
`CopilotRuntimeLike` / `BaseCopilotRuntime` with a passthrough getter on
the `CopilotRuntime` shim.
- **Both paths.** The resolved policy is read at **/run**
(`configureAgentForRequest`) and **/connect** (`handleSseConnect`) via
`mergeForwardableHeaders`, so the two can never diverge. Server-wins
precedence and server-self case-dedup from #5782 are untouched.
## Semver
**Minor with an opt-out.** Removing a leak is a fix, not a contract
change, and we ship a documented escape hatch: `new CopilotRuntime({
agents, forwardHeaders: { useDefaultDenylist: false } })` restores the
prior behavior. Custom-header forwarders (the common case) are
unaffected.
## Red-green proof (real surface, both paths)
RED — with the predicate reverted to the old wide-open `authorization ||
x-*` (policy ignored), the new behavior assertions fail; the leak
reproduces (`x-forwarded-for: 203.0.113.7` forwards on both /run and
/connect):
```
❯ header-utils.test.ts (19 tests | 8 failed)
× strips known infra/proxy/platform headers by exact name → expected true to be false
× strips known infra/platform header families by prefix → expected true to be false
× strips denylisted headers case-insensitively → expected true to be false
× deny extends the default set → expected true to be false
× denyPrefixes extends the default set → expected true to be false
× allow switches to allowlist mode → expected true to be false
× extractForwardableHeaders drops denylisted x-* infra → expected {…4} to deeply equal {…1}
❯ agent-utils-header-forwarding.test.ts (/run) (10 tests | 1 failed)
× strips denylisted infra/platform headers (#5712 breadth) → expected '203.0.113.7' to be undefined
❯ sse-connect-agent-id.test.ts (/connect) (5 tests | 1 failed)
× strips denylisted infra/platform headers → expected '203.0.113.7' to be undefined
```
GREEN — with the real policy in place:
```
✓ header-utils.test.ts (19 tests)
✓ agent-utils-header-forwarding.test.ts (10 tests) # /run path
✓ sse-connect-agent-id.test.ts (5 tests) # /connect path
✓ agent-header-precedence.test.ts (2 tests)
Test Files 4 passed (4)
Tests 36 passed (36)
```
Full `@copilotkit/runtime` suite: **113 files / 1593 tests passed.**
Typecheck, oxlint (0 errors), oxfmt, and build all green.
## Builds on #5782
This branches off #5782's head (`636bcad05`) and reuses that PR's
`mergeForwardableHeaders` (server-wins precedence + server-self
case-dedup). It should land **after #5782**. It addresses the
**forwarding-breadth half of #5712** — #5712's precedence core is fixed
by #5782; this is the breadth follow-up (not `Fixes #5712`).
## Problem
When a self-hosted v2 `CopilotRuntime` is configured with a server-side
agent (an `@ag-ui/client` `HttpAgent` with static `headers` for
service-to-service auth), the runtime forwards inbound
`authorization`/`x-*` request headers onto the agent's outgoing call
**and lets them override the headers the server configured** — silently
breaking service-to-service auth to a secured backend (e.g. a private
Cloud Run agent behind IAM).
`Fixes #5712`
## Root cause
`packages/runtime/src/v2/runtime/handlers/shared/agent-utils.ts:125-128`
merged forwarded inbound headers **last**, so they won on collision:
```ts
agent.headers = {
...agent.headers, // server-configured
...extractForwardableHeaders(request), // inbound — overrode the above
};
```
There are actually **two** failure modes:
1. **Same-case collision** — inbound `authorization` overwrites a server
`authorization` (last-write-wins).
2. **Case-mismatch collision** — `extractForwardableHeaders` lowercases
inbound keys (`authorization`), while the server typically configures
canonical casing (`Authorization`). A plain spread treats those as
*distinct* keys and emits **both** — which undici downstream comma-joins
into a single invalid `"Bearer A, Bearer B"` ("multiple JWTs") value.
Flipping the spread order alone does **not** fix this case.
## Fix
In `agent-utils.ts`, make server-configured `agent.headers`
authoritative on collision, matched **case-insensitively**: drop any
forwarded inbound header whose name (case-insensitively) is already set
on the agent, and let non-colliding inbound headers pass through
unchanged. This preserves the existing forward-for-auth behavior for
headers the server does *not* set, while guaranteeing a server-set token
is never overridden or duplicated.
The merge logic lives in a shared
`mergeForwardableHeaders(serverHeaders, request)` helper in
`packages/runtime/src/v2/runtime/handlers/header-utils.ts` so the
precedence semantics are defined in exactly one place.
### Scope note
This is the conservative precedence + case-insensitive-dedup fix (the
issue's suggested fix#1). I did **not** tighten the default allowlist
to drop hop-by-hop/platform `x-*` headers (`x-serverless-*`,
`x-forwarded-*`, …) or add an opt-out — those alter existing forwarding
behavior and are worth a separate, deliberate change. The precedence fix
alone resolves the reported breakage (the server-set token now wins
regardless of what the platform injects on a colliding header name).
A documented workaround already exists for users on released versions:
pass a custom `fetch` to the `HttpAgent` that builds outgoing headers
from scratch (it runs after `configureAgentForRequest` and survives the
per-request `agent.clone()`).
## Red-green proof (the real fix — `/run` path)
The load-bearing assertion: there must be exactly **one** authorization
header carrying the **server** value.
### RED (fix stashed, against unmodified `agent-utils.ts`)
```
❯ src/v2/runtime/__tests__/agent-header-precedence.test.ts (2 tests | 1 failed)
× configureAgentForRequest — header precedence (#5712) > server-configured agent headers win over a colliding inbound header
AssertionError: expected [ 'Authorization', 'authorization' ] to have a length of 1 but got 2
81| expect(authKeys).toHaveLength(1);
Test Files 1 failed (1)
Tests 1 failed | 1 passed (2)
```
The pre-existing `agent-utils-header-forwarding.test.ts` also failed,
because it explicitly encoded the buggy behavior
(`expect(...["x-aimock-context"]).toBe("new-context")` — inbound
winning):
```
FAIL src/v2/runtime/__tests__/agent-utils-header-forwarding.test.ts > ... > request forwardable headers override matching pre-existing agent headers
AssertionError: expected 'old-context' to be 'new-context'
```
### GREEN (fix applied)
```
✓ src/v2/runtime/__tests__/agent-header-precedence.test.ts (2 tests) 2ms
✓ src/v2/runtime/__tests__/agent-utils-header-forwarding.test.ts (8 tests) 3ms
Test Files 2 passed (2)
Tests 10 passed (10)
```
The colliding test (`agent-utils-header-forwarding.test.ts`) was updated
from asserting the old bug to asserting corrected precedence + a new
case-insensitive-dedup guard. The non-colliding-forward test is retained
unchanged as a regression guard.
## Quality gates
```
NX Successfully ran target check-types for project @copilotkit/runtime
NX Successfully ran target test for project @copilotkit/runtime — Test Files 113 passed (113), Tests 1576 passed (1576)
```
---
## `/connect`-path change — forward-looking plumbing, inert today
The original issue and a prior eval flagged the same forwarding pattern
at `handlers/sse/connect.ts`. To keep the two paths' merge semantics
consistent, the `/connect` path now builds the same server-wins merged
headers (via the shared `mergeForwardableHeaders` helper) and passes
them into `runner.connect()`.
**This is not an active auth fix, and it is not red-green-proven as one
— because there is no live bug to fix on the connect path today.** No
shipped runner consumes the `headers` field of
`AgentRunnerConnectRequest`: the in-memory, intelligence, telemetry, and
sqlite runners all destructure only `threadId` from the connect request
and ignore `headers` entirely. Connect is a thread replay/reconnect, not
a fresh outgoing agent call. So whatever headers we pass into
`runner.connect()` are dropped on the floor by every runner that ships.
What this change actually does:
- Threads the per-request agent clone through `handle-connect.ts →
handleSseConnect` so the connect path *has access to* the
server-configured `agent.headers` (it previously did not).
- Passes `mergeForwardableHeaders(agent?.headers, request)` into
`runner.connect()` — the correct, server-wins argument **shape** for a
future outbound-connecting runner that *would* consume connect-path
headers.
- Rewrites the comments/JSDoc on this path to say this plainly, rather
than implying an active auth fix. It also documents that the
connect-site `cloneAgentForRequest` call is the sole `agentId`-existence
guard (the intelligence branch never re-validates the id), and documents
`cloneAgentForRequest`'s `AbstractAgent | Response` (404) dual-return
contract that both callers depend on.
The real outbound header forwarding — the thing that fixes#5712 — is
the `/run` path's `agent.headers` mutation described above. The connect
change is staged plumbing so that if/when a runner starts honoring
connect-path headers, it inherits the same server-wins precedence
without a second fix.
### Tests on the `/connect` path
The connect tests assert the *merge shape* that reaches
`runner.connect()` (server value wins on collision, exactly one
`authorization` key, non-colliding `x-*` still forwards) and that the
agent-undefined case (no server `agent.headers`) degrades to forwarding
allowlisted inbound headers only and does not crash. These verify the
argument we construct is correctly shaped — not that any shipped runner
consumes it.
## Files
- `packages/runtime/src/v2/runtime/handlers/header-utils.ts` — shared
`mergeForwardableHeaders` helper (case-insensitive, server-wins).
- `packages/runtime/src/v2/runtime/handlers/shared/agent-utils.ts` —
`/run` path uses the helper so server headers win on collision (**the
real fix**).
- `packages/runtime/src/v2/runtime/handlers/sse/connect.ts` — `/connect`
path uses the helper; forward-looking plumbing, inert until a runner
consumes connect-path headers.
- `packages/runtime/src/v2/runtime/handlers/handle-connect.ts` — threads
the per-request agent clone into `handleSseConnect`.
-
`packages/runtime/src/v2/runtime/__tests__/agent-header-precedence.test.ts`
— `/run` regression test exercising the real `configureAgentForRequest`
surface with a real `HttpAgent`.
-
`packages/runtime/src/v2/runtime/__tests__/agent-utils-header-forwarding.test.ts`
— updated the test that encoded the old (buggy) precedence; added a
case-mismatch dedup guard.
-
`packages/runtime/src/v2/runtime/handlers/sse/__tests__/sse-connect-agent-id.test.ts`
— connect-path merge-shape + agent-undefined coverage.
### Notes
- A documented `@ag-ui/client` `HttpAgent` `fetch` workaround already
exists for attaching service-to-service auth the runtime can't override
(see the issue). This change makes the workaround unnecessary for the
`/run` precedence case.
- Conservative scope: this is the **precedence flip on `/run`** plus
forward-looking connect plumbing. Tightening the default allowlist
(dropping hop-by-hop / platform `x-serverless-*`, `x-forwarded-*`,
`x-cloud-trace-context`, …) and an opt-out switch — issue suggestions
#2/#3 — are intentionally left as a follow-up to keep the
security-policy change minimal.
Document the v2 runtime's inbound-header forwarding behavior on the
Copilot Runtime page: the default denylist (authorization + x-* minus
known infra/proxy/platform headers), the x-request-id upgrade note,
server-configured header precedence (#5782), and the forwardHeaders
config option (deny/denyPrefixes/allow/useDefaultDenylist) with the
allowlist-mode denylist-bypass footgun.
Refs #5712, #5783
The bundled tei embedder image is amd64-only; under arm64 emulation the Candle
backend is unavailable and TEI falls back to the ONNX/ORT backend, which needs
onnx/model.onnx files Qwen3-Embedding-0.6B doesn't publish (404) -> crash-loop.
A fresh clone on Apple Silicon therefore couldn't stand up the embedder, so
memory save/recall were dead. Ports the proven pattern from the Intelligence
repo's docker-compose.deps.yml + demos/splat-demo/run-demo.sh into this demo:
- docker-compose.yml: gate the bundled `tei` behind the `cpu-fallback` profile,
so a bare `docker compose up` skips the crash-looping emulated image. amd64/CI
opt back in with `--profile cpu-fallback`. (intelligence's tei dep is
required:false, so it starts fine without it, using MEMORY_EMBEDDINGS_URL.)
- run-demo.sh: one-command cold start. On Apple Silicon it runs a native Metal
TEI on :7067 (same 1.9.3 + Qwen3-Embedding-0.6B => byte-identical embeddings,
~20x faster) and points app-api at it; on amd64/CI it uses the docker tei via
the profile. Mints a dev license if .env lacks one, then starts `pnpm dev`.
- README: correct the failure description (emulation->ONNX crash-loop, not OOM),
document run-demo.sh as the recommended start, and the profile-gated manual path.
All CopilotKit-repo-only (banking's compose is standalone); no Intelligence
changes. Validated: shellcheck clean, compose valid, bare `up` skips tei and
keeps intelligence healthy, memory save/recall verified through the native TEI.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 3 of the banking->Intelligence-main migration. Self-hosted Intelligence
gates the paid `memory` feature behind a signed offline license; a locally-built
(unbaked) app-api trusts a runtime BAKED_LICENSE_KEYS_JSON, so a throwaway
keypair can sign an enterprise license with features.memory=true.
- scripts/mint-dev-license.mjs: prints (or --write upserts into .env)
COPILOTKIT_LICENSE_TOKEN + BAKED_LICENSE_KEYS_JSON + INTELLIGENCE_DEPLOYMENT_MODE.
Drives the signer from the PRIVATE Intelligence source via INTELLIGENCE_REPO
(same coupling the docker-compose image build already has) rather than
vendoring any signing code into this public repo. No secret is embedded; the
script is dev-only and never imported by the app runtime.
- .env.example: documents BOTH the managed path (CopilotKit-issued token, no
baked key — the eventual hosting target) and the self-hosted dev path, so the
demo is not locked to the local stack.
- package.json: add `mint-dev-license` script.
Replaces the ephemeral Intelligence/tmp/mint-banking-license.ts. .env stays
gitignored; nothing sensitive is committed. Local-only until verified.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 2 of the banking->Intelligence-main migration. Main's memory lib
(libs/memory/src/types.ts) closes MemoryKind to topical|episodic|operational;
the demo was authored against the legacy semantic|procedural names, which the
backend now rejects/misfiles. Rename across the whole surface:
- agent prompt (route.ts CLASSIFY + SAVE-THE-PROCEDURE): semantic->topical,
procedural->operational
- recorder instruction (copilot-context.tsx), learning-tab dual-read dropped,
memory-tab KIND_COLORS, memory unit-test fixture
- smokes (facts + drift) and the e2e spec seed + fixtures comment
Only true kind: values renamed; "semantic recall"/"top-k semantic search"
mechanism descriptions left intact (recall is vector search regardless of enum).
aimock fixture re-record was a no-op: the one fixture pins the recall-and-apply
arc (no kind: values); the seed is REST-side in the spec.
Verified: pnpm test:unit 47/47, tsc --noEmit clean, eslint clean on touched files.
Local-only until the full migration is verified against the main stack.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 1 of the banking->Intelligence-main migration (branch:
feat/banking-intelligence-main-migration). PROVEN GREEN against main:
- INTELLIGENCE_DEPLOYMENT_MODE=self_hosted (renamed from legacy DEPLOYMENT_MODE)
- dropped legacy DEFAULT_ORGANIZATION_ID (main's loadAuthEnv rejects it)
- BAKED_LICENSE_KEYS_JSON wired: main gates memory behind a signed license
carrying the "memory" feature (MEMORY_NOT_ENTITLED otherwise). A locally
minted dev enterprise license + baked public key unlocks it (recipe mirrors
Intelligence apps/app-api-e2e global-setup). Verified: /mcp attaches
recall/save/forget_memory and save_memory(kind=topical) round-trips via the
cpk key.
REMAINING (next session): (1) reproducible dev-license mint helper + .env wiring
(mint script currently at Intelligence/tmp/mint-banking-license.ts, ephemeral);
(2) kind rename semantic->topical, procedural->operational across prompt, memory
lib, smokes, e2e spec; (3) aimock fixture re-record for new kinds; (4) re-verify
e2e/smokes/manual arc. Working demo (PR #5763, demo branch) is untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Adds the **user-scoped long-term memory** feature end to end:
- **`@copilotkit/core`** — a single, core-owned, user-scoped memory
store (`getMemoryStore()`): REST snapshot (`created_at DESC`) + realtime
`memory_metadata` over a dedicated `user_meta:memories:<joinCode>`
Phoenix socket, session-guarded reducers, mutation tracking, and silent
degrade on unconfigured routes. Public memory types are exported
unprefixed (`Memory`, `NewMemory`, `MemoryChanges`, `MemoryKind`,
`MemoryScope`).
- **`@copilotkit/react-core`** — `useMemories()` (no-arg) via
`useSyncExternalStore`, SSR-safe.
- **`@copilotkit/angular`** — `injectMemories()` (no-arg) via
`toSignal`, mirroring the React binding.
- **`@copilotkit/web-inspector`** — a read-only **Memories** tab (card
list + kind filter + text search) that consumes the core store
cross-framework.
- **`@copilotkit/runtime`** — memory REST endpoints (list / create /
supersede / retire / subscribe-credentials) with boundary validation.
Memory is **user-scoped** (no agentId); the store and its realtime are
owned by core, so every framework binding and the inspector get it for
free.
## Hardening (code review)
This branch went through a thorough multi-pass review. Notable fixes:
- **Correctness:** session-guard mutation outcomes (no stale
cross-session error leak); handle credentials events; validate supersede
`retiredId` (no duplicate rows); `available` is list-route-scoped (no
order-dependent race); `isMutating` → in-flight counter
(concurrent-mutation safe); `useMemories` SSR `getServerSnapshot`;
`refresh()` settles on `listUnavailable` / context switch (no hang);
treat runtime `422` (not-configured) as graceful degrade.
- **Validation:** mutation-response and `sourceThreadIds`/`kind`/`scope`
boundary validation; list-shape assertion.
- **Observability:** `realtimeStatus` (`connecting` / `connected` /
`unavailable`) surfaced through both bindings and the inspector's live
indicator, so a permanent socket give-up is no longer silent; a memory
error registry with stable codes.
- **Privacy:** the inspector's `memories_tab_clicked` telemetry now
honors `telemetryDisabled`; the inspector creates the memory store
**lazily on tab activation** (attaching the inspector no longer starts a
store / opens realtime in apps that don't use memory).
- **UX:** inspector renders mutation failures inline instead of blanking
the list; distinct "upgrade SDK" teaser for older `@copilotkit/core`.
## Testing
`lint`, `check-types`, `test`, and `build` are green across
`@copilotkit/core`, `@copilotkit/react-core`, `@copilotkit/angular`,
`@copilotkit/web-inspector`, and `@copilotkit/runtime`. New coverage
spans session guards, realtime deltas + idempotency,
unavailable/timeout/SSR paths, telemetry gating, lazy inspector
activation, and angular mutation parity.
Tyler Slaton