mirror of
https://github.com/CopilotKit/CopilotKit.git
synced 2026-09-14 16:26:20 +08:00
v1.64.0
4665 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ben Taylor
|
8a8e52d9e9 |
fix(runtime): non-optional listener.channels and honest lifecycle docs (OSS-646) (#6207)
Closes OSS-646. Split out of OSS-641 as the unambiguous half. This PR does **not** change when activation happens — whether the long-running wrappers should auto-connect stays open on OSS-641. ## Why `createCopilotRuntimeHandler` builds the `ChannelManager` but opens no connection; activation is lazy, triggered by the first `channels.ready()`. That is deliberate (`fbf35ac59`, OSS-473) — Cloudflare/Next isolates freeze and recycle per request, so cold starts would mint conflicting listeners. Two things were left inconsistent with it: 1. `endpoints/node.ts` still documented the pre-`fbf35ac59` world — "the same `ChannelsControl` surface the underlying fetch handler **activates at creation time**" — and labelled the one required call as `// Optional:`. That's the TSDoc developers and coding agents see in-editor, and it contradicted every channel-package README. Same failure class as OSS-634. 2. `68349bc1f` gave the fetch handler a branded overload so `handler.channels.ready()` type-checks without `?.`, but the node wrapper never got it — so every call site, including our own example and all nine showcase docs pages, was written defensively. ### A live consequence, found en route `examples/slack/app/managed.ts` never called `ready()`. It built the runtime, mounted the listener, logged `[channel] started managed Channel "…"`, and only ever called `stop()` — so since activation went lazy it has connected nothing while reporting success. It was written against exactly the creation-time model the TSDoc described. Fixed here, with a regression assertion. ## What changed - **Types** — `createCopilotNodeListener` gets the branded overload pair mirroring `createCopilotRuntimeHandler`: a runtime with at least one declared Channel yields non-optional `.channels`; `activateChannels: false` and channel-less runtimes keep the optional shape. Adds `NodeCopilotListenerWithChannels`; both listener types are now exported from `@copilotkit/runtime/v2/node`. - **Docs** — node/express/hono TSDoc corrected: creation opens no connection, `ready()` is what activates, and it is required on a long-running host. Same stale claim fixed in the three example comments and `examples/slack/README.md` that repeated it. - **Call sites** — `?.` dropped from `examples/slack`, `examples/teams`, both READMEs, and the nine `showcase/shell-docs` channel pages. ## Deliberate scope choices, called out - **Express/Hono keep an optional `.channels`.** Only their TSDoc is corrected here. Their own type docs name Node as the lifecycle-owning surface and attach `.channels` best-effort, so the branded overload is Node-only for now; `endpoints-channels.test.ts` still uses `!` for those two. Say the word if the overload should extend to them. - **The non-optional shape requires a literal `channels` tuple** (`readonly [Channel, ...Channel[]]`). A runtime built from a dynamically-assembled `Channel[]` is unbranded and still needs `?.`. Now stated in the node TSDoc. - **`examples/slack/app/managed.ts` now exits nonzero if activation fails**, where before it stayed up serving HTTP with nothing connected. Intentional — fail loud, and it matches `index.ts`. Note that `ready()` resolves for `setup_required`, so a declared-but-unprovisioned channel still logs as started. - **Signal handlers are registered before awaiting activation** in `managed.ts`, so a Ctrl-C inside the 30s activation window still tears the Channel down instead of hitting Node's default handler. ## Verification - **Type contract, red → green:** the new `KeyIsRequired<typeof listener, "channels">` assertion in `handler-channels-types.test.ts` failed to compile before the overload (`error TS2344: Type 'false' does not satisfy the constraint 'true'`) and passes after. - **Example bug, red → green:** stashing only `managed.ts` fails the new guard with `expected "vi.fn()" to be called once, but got 0 times`. - **Strict-null proof:** `slack-example` and `teams-example` both `tsc --noEmit` clean under `strict: true` with the `?.` removed. This matters because the runtime package compiles with `strict: false`, so its own type test can only probe the optionality modifier structurally. - Runtime channel suites 54/54; slack example 63/63. - **Coverage limit:** the `managed.ts` guard is mocked — it proves the example *calls* `ready()` with a bound, not that a Channel connects. Nothing in CI exercises a real gateway connect for these examples. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Benjamin Taylor
|
3f0bbe4a7b |
feat(runtime): default the Intelligence platform URLs to the managed service
`CopilotKitIntelligence` required `apiUrl` and `wsUrl` on every construction,
so the two correct hosts had to be found and copied by hand — which is how an
agent came to invent them. Both now default to CopilotKit's managed platform,
making `new CopilotKitIntelligence({ apiKey })` the whole managed-service setup.
Overrides are unchanged for self-hosted and non-production deployments, with two
guards that the previous required-field signature made unnecessary:
- A blank value counts as unset. These URLs are usually wired from env vars, and
a declared-but-empty variable arrives as `""`, which would otherwise produce
host-relative requests instead of falling back to the managed platform.
- Setting only one of the pair warns. The API and realtime planes are separate
hosts, so a lone override silently splits the client across two deployments —
and that failure surfaces as a hang, not an error.
Sweeps the doc, skill, README, and example surfaces to the short form so the
copy-paste path no longer hands anyone URLs to get wrong, and reattaches the
`CopilotKitIntelligence` class JSDoc, which was orphaned above an interface and
so never appeared on hover.
Linear: OSS-638
|
||
|
Benjamin Taylor
|
956c6448a7 |
fix(runtime): non-optional listener.channels and honest lifecycle docs (OSS-646)
`createCopilotNodeListener` now mirrors `createCopilotRuntimeHandler`'s branded overload pair, so a runtime with at least one declared Channel yields a listener whose `.channels` is non-optional and the documented `listener.channels.ready()` call type-checks with no `!` and no `?.`. `activateChannels: false` and channel-less runtimes keep the optional shape. Both listener types are exported from `@copilotkit/runtime/v2/node`. Corrects TSDoc on the node, express, and hono wrappers that still claimed activation happens "at creation time" and labelled `ready()` as optional — stale since activation was deferred to make the Fetch handler serverless-safe. On a long-running host that call is required, not optional. Fixes a live consequence of that stale model: `examples/slack/app/managed.ts` never called `ready()`, so it mounted a listener, logged "started managed Channel", and connected nothing. Covered by a regression assertion. Drops the now-unnecessary `?.` from the examples, READMEs, and channel docs, and adds compile-time contracts for the listener shape alongside the existing handler ones. The examples compile with `strict: true`, so they prove the `?.`-free call under strict null checks, which the runtime package (strict: false) cannot. |
||
|
Tyler Slaton
|
1d1ca9ea8b |
fix(docs): correct the documented Intelligence endpoints for Channels (#6189)
Fixes the Channels docs/examples pointing at an Intelligence host that does not serve the API, and the websocket URL guidance that can never produce a working prod value. Linear: [OSS-621](https://linear.app/copilotkit/issue/OSS-621). ## The two bugs **1. The documented host does not serve the API.** Probed every plausible path, not just `/`: | URL | Result | | --- | --- | | `api.copilotkit.ai` — `/`, `/api`, `/api/health`, `/health`, `/api/threads`, `/api/v1/threads` | **404 on all**, `server: awselb/2.0`, `content-length: 0` — an ALB with no target-group rule behind it | | `realtime.copilotkit.ai` | **no DNS record at all** | | `api.intelligence.copilotkit.ai/` and `/api/health` | 200, `x-powered-by: Express` | | `api.intelligence.copilotkit.ai/api/threads` | **401** — a real, auth-gated endpoint | | `realtime.intelligence.copilotkit.ai/runner/websocket` | **403** — mounted and auth-gated (a 404 would mean unmounted) | So the documented host is not merely returning 404 at the root — nothing is routed there on any path, and it is not the app (no `x-powered-by`). The working pair, matching the CLI's baked-in prod defaults and `gitops/environments/prod/values.yaml`, is `https://api.intelligence.copilotkit.ai` + `wss://realtime.intelligence.copilotkit.ai`. **2. `wsUrl` was documented as derivable from `apiUrl`.** Prod splits the API and realtime planes across *different hosts*, so a scheme-only swap yields `wss://api.intelligence.copilotkit.ai` — wrong host. That failure is silent: a wrong `apiUrl` returns a clean HTTP error, but a wrong `wsUrl` sits in `connecting` until the settle timeout and reports only "did not settle in time". The derive is not even correct locally, where the API and gateway are on different ports (4201 vs 4401) and the swap preserves the port. It is essentially never right, so it is deleted rather than relabelled. Demonstrated end to end rather than asserted — running `examples/teams` with the WS URL unset: ``` # on main: derive(https://api.intelligence.copilotkit.ai) -> wss://api.intelligence.copilotkit.ai (wrong host, 30s hang) # on this branch: exit code: 1 Missing COPILOTKIT_INTELLIGENCE_WS_URL. export COPILOTKIT_INTELLIGENCE_URL=https://api.intelligence.copilotkit.ai export COPILOTKIT_INTELLIGENCE_WS_URL=wss://realtime.intelligence.copilotkit.ai The API and websocket URLs are DIFFERENT hosts (api.… vs realtime.…), so the websocket URL cannot be derived from the API URL — set both. ``` ## Scope note The ticket listed 8 sites; the actual blast radius was 30 across 22 files. Beyond the ticket's list: - **Five more channels package READMEs** — `channels`, `channels-core`, `channels-discord`, `channels-slack`, `channels-teams` (the ticket named only telegram + whatsapp). - **The live product docs** — `showcase/shell-docs/src/content/docs/channels/` taught the broken derive in 8 files. These escaped the ticket's grep because their host was already a `your-intelligence-url` placeholder; only bug 2 was present. This is the surface developers actually read. - **A generated skills mirror** — `skills/runtime/` is produced from `packages/runtime/skills/runtime/` by `pnpm sync:plugin-skills`; fixing one without the other leaves the bug live and fails the `check-plugin-skills` gate. - `skills/copilotkit-debug/references/runtime-debugging.md` and a third occurrence in `client.ts`. ## Acceptance item 4 — resolved, chain verified The ticket flagged a contradiction: `realtime-gateway.ts` documented `wss://gateway.example/socket` while the runtime skill listed `/socket` as a mistake. **The skill is right**, confirmed by tracing the real runtime path rather than inferring it: 1. `channel-activation-config.ts:145` — `const wsUrl = intelligence.ɵgetRunnerWsUrl()`, i.e. base + `/runner`. 2. → `channel-manager.ts:237` `wsUrl: config.wsUrl` → `startChannelsOverRealtimeGateway` → `connectRealtimeGateway`. 3. `realtime-gateway.ts:253` hands that to Phoenix's `Socket`, which appends `/websocket`. 4. The gateway mounts exactly `/runner` and `/client` (`realtime_gateway/endpoint.ex:10,17`). There is no `/socket`. The two docs survived contradicting each other because they describe **different layers**: the public `wsUrl` is a bare base, while `connectRealtimeGateway` receives the already-derived runner URL. Its doc comment and the test fixtures move to `/runner` and now name the layer. Independently, `get-runtime-info.ts:93` uses `ɵgetClientWsUrl()`, which confirms the `/info` sample in the debug skill correctly keeps its `/client` suffix — only the host there was wrong. ## Acceptance item 3 — split out Failing loudly instead of hanging is a behavior change in the launcher's connect path that overlaps OSS-622's error classification, so it is [OSS-623](https://linear.app/copilotkit/issue/OSS-623) rather than smuggled into a docs fix. ## Verification - 44/44 gateway tests in `packages/channels-intelligence`; `examples/slack/app/managed.test.ts` passes; `examples/teams` typechecks clean. - `nx run-many -t test,publint,attw` across the 15 affected projects passed via the pre-commit hook; full CI green (38 checks, including `build-check (shell-docs)`). - Behavior proven by running the example, not by mocks (above). - 45 commits behind `main` at time of review with **zero overlap** on changed files, and the diff covers every dead-host and derive site present on *current* `main`. - No logic changes in either published package — `client.ts` and `realtime-gateway.ts` are JSDoc/field-comment only. Behavior changes are confined to the three example apps. ## Self-review pass An adversarial pass over this PR tried to falsify its central claim (that `api.copilotkit.ai` does not serve the API) by probing non-root paths — the evidence got stronger, not weaker. It also cleared a suspected regression: three `channels-intelligence` files read `COPILOTKIT_INTELLIGENCE_URL` without the WS var, but they are the HTTP-only transport path and need no socket URL. Two genuine gaps it *did* find are fixed in the last commit: the five platform pages had lost deployment neutrality (managed hosts now carry a self-hosted note), and `index.mdx`/`mcp.mdx` referenced the new variable without telling the reader where it comes from. **One product decision for the reviewer:** `api.copilotkit.ai` is a live ALB that routes nothing. If it is meant to become the public API alias, this PR is documenting the wrong long-term string and the ALB wants a listener rule instead. I used the host the product actually hands users today (CLI prod defaults + gitops). 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Alem Tuzlak
|
9a501d5e0d |
fix(showcase/built-in-agent): repair six real-LLM defects D6 was masking (#6200)
Six defects reported on staging. All six reproduce; five share one cause. ## The shared cause `built-in-agent/src/app/api/copilotkit/route.ts` maps **~20 demos to the same prompt-less `createBuiltInAgent()`**. The reference wires each to its own graph **and its own system prompt** (28 graphs in `langgraph-python/langgraph.json`). aimock replays a scripted tool-call sequence keyed on `userMessage` + `context`, so D6 is green whether or not the model could have reasoned its way there. This is `showcase/GOTCHAS.md` #8 verbatim — including its stated "structural tell": *grep the integration's `route.ts` for demos left in the generic fallthrough that LGP wires to a dedicated graph.* ## Per-defect | Demo | Symptom | Root cause | |---|---|---| | `gen-ui-tool-based` | charts plot zeros | No prompt. The assistant says so itself: *"I used placeholder values since no sales figures were provided."* Nondeterministic — one run gave a real 0..220 axis, the next `0 1 2 3 4` with no bars, which is how it passed review. | | `gen-ui-agent` | wall of text, steps frozen at "step 1 of 4" | `set_steps` + its `/steps` STATE_DELTA already worked; nothing told the model to *walk* pending→in_progress→completed. | | `subagents` | left panel always empty | `grep -rn delegations src/ \| grep -v demos/subagents` → **no output**. Frontend reads `agent.state.delegations`; the converter handles `/steps` and `/notes` and has no `delegations` case. Chat works because the tools do run. | | `declarative-json-render` | raw JSON for pills 2 & 3 | Intercepted the SSE: **the wire is already one closing brace short** — nothing is dropped in transport, and the client parser correctly requires a balanced object. Pill 1 only works because the model copies the prompt's worked example verbatim (`\$1.24M`, `+18% vs Q2`). Model-level JSON enforcement had been removed with the note *"the system prompt already enforces JSON-only output"* — it does not. | | `a2ui-recovery` | 5 identical cards | Reuses declarative-gen-ui's prompt with no "call once" constraint, while the pills literally ask it to *"self-correct a malformed first attempt"*. The retry loop is **inside** the tool and returns on first valid pass, so every supervisor retry is pure duplication. | | `declarative-gen-ui` | D4 though it works live | **The inverse case.** Badges: `UI ✓ BE ✓ 1P ✗ D6 —` (*"gated — blocked by a lower rung"*). Not a live bug — a fixture bug. See below. | ## The D4 cap (`declarative-gen-ui`, `a2ui-recovery`) Both demos' secondary design-call fixtures gated on `match.responseFormat: \"json_object\"`. That matcher **can never match this backend**: - The agent talks to the OpenAI *Responses* API, where JSON mode is `text.format`. - aimock has **no `text.format` handling at all** — `responsesToCompletionRequest` forwards only a top-level `response_format` (absent in 1.19.1, forwarded in 1.37.4), a key the Responses API doesn't accept and this client doesn't send. - So `effective.response_format?.type` is always `undefined` and `router.ts` skips the fixture → the design call never matched → surface never painted → D5 red, D6 never ran. Re-keyed onto `match.toolName`, which aimock *does* normalize out of a Responses request (`responsesToolsToCompletionsTools`): the outer call declares `generate_a2ui`, the in-tool design call declares nothing. Tool-less secondary fixtures moved last, because several pills' brief is a substring of the pill text (`\"Build my Q2 revenue summary …\"` contains `\"Q2 revenue summary\"`) and would otherwise win the outer request. ## Verification **Reproduced live** (staging, real LLM): the zero-axis chart plus its self-incriminating message, the unbalanced JSON on the wire, and the reference rendering the *same pill* correctly for contrast. **Two mutation-verified test suites** — both fail with the fix reverted: - `aimock-a2ui-routing.test.ts` drives aimock's **real `matchFixture`**. On the old fixtures 14/16 cases fail with `no fixture matched the secondary design call for brief \"…\"` — the D5 red, reproduced as a unit test. - `tanstack-factory.test.ts` covers the `/delegations` and `/steps` deltas (3 delegation tests fail with the branch disabled; the 2 controls still pass). **No regressions:** `tsc --noEmit` is 61 errors before *and* after (all pre-existing — `gpt-5.4` missing from the adapter's model union, zod v3/v4 skew); oxlint clean on changed files; the repo-wide fixture validator passes (858 assertions). `text.format` was verified by reading the request mapping in the **pinned** `@tanstack/ai-openai@0.15.6` → `@tanstack/openai-base@0.9.2`: `modelOptions` is spread straight into `responses.create()`, `validateTextProviderOptions` only inspects `metadata`/`conversation`/`previous_response_id`, and the adapter sets `text.format` itself only when an `outputSchema` is passed (none here, so nothing is clobbered). ## What is NOT verified **The four prompt/`text.format` changes have not had a real-LLM click-through** — that needs an OpenAI key this environment doesn't have, and D6/aimock cannot verify them by construction (that's the whole point of gotcha #8). Reasoning is documented inline at each site. This area has burned the repo before: a previous `response_format` attempt made the call return an empty string "verified against real OpenAI", which is why `text.format` — the Responses API's own param — is used instead. **Worth a live click-through on the staging deploy before this is considered closed.** ## Deliberate non-goals - **`a2ui-recovery` only demonstrates recovery under aimock.** The heal/exhaust branches need a designer LLM that emits invalid surfaces on demand; a real one succeeds on attempt 1. The single-call constraint fixes the 5-card bug and makes it honest; making recovery visible live needs deliberate fault injection — a product decision, recorded in `PARITY_NOTES.md`. - **The reference pins `openai:gpt-4o-mini`** (`gen_ui_agent.py:92`). Flagged, not touched. ## Docs `GOTCHAS.md` gains built-in-agent as a second worked instance of #8 (including that the masking runs *both* ways, and that `1P ✗ D6 —` means gated, not failing), plus the `text.format` and dead-`responseFormat` traps. `PARITY_NOTES.md` records that the named-agent registry is **not** prompt-neutral, and that any new `state.<slot>` needs a converter branch. |
||
|
Alem Tuzlak
|
1d5cdb35cf |
fix(showcase/built-in-agent): repair six real-LLM defects D6 was masking
Every affected cell was D6-green (or D4-gated) while the deployed demo was visibly broken. The common cause is GOTCHAS #8: aimock replays a scripted tool-call sequence keyed on userMessage + context, so the fixture answers a question the model was never asked. Prompt gaps — ~20 demos shared one prompt-less `createBuiltInAgent()` where the reference wires each to its own graph AND its own system prompt. Adds `createBuiltInAgent({ systemPrompt })` + `demo-prompts.ts`, ported from the reference graphs: - gen-ui-tool-based plotted zeros; the assistant said "I used placeholder values since no sales figures were provided". Nondeterministic — some runs invented real values, which is how it passed review. - gen-ui-agent published its plan once then narrated, freezing the progress card on step 1 of 4. - a2ui-recovery painted five identical cards: nothing constrained the supervisor to one `generate_a2ui` call, and the pills literally ask it to "self-correct". The retry loop lives inside the tool, so supervisor retries are duplication. subagents' delegation panel was permanently empty — the frontend reads `agent.state.delegations` and no code emitted that slot. The converter now emits a `/delegations` delta per sub-agent result (whole-array `add`, since initial state is `{}` and strict fast-json-patch rejects unresolvable paths while @ag-ui/client swallows the throw), and ports the reference's `_MAX_CRITIQUE_ITERATIONS = 1` cap. declarative-json-render dumped raw JSON for any prompt the model couldn't crib from the worked example. Captured the SSE: the wire is already one closing brace short, so nothing is dropped in transport — model-level enforcement had been removed on the grounds that "the system prompt already enforces JSON-only output". It does not. Restores it via the Responses API's `text.format`, the param the removed `response_format` maps to, verified against the pinned @tanstack/ai-openai@0.15.6 -> @tanstack/openai-base@0.9.2 request mapping. declarative-gen-ui and a2ui-recovery were the inverse: capped at D4 (UI ✓ BE ✓ 1P ✗ D6 gated) while working live. Their secondary design-call fixtures gated on `match.responseFormat`, which aimock can never satisfy here — it has no `text.format` handling and only forwards a top-level `response_format` the Responses API doesn't accept. Re-keyed onto `match.toolName` (the outer call declares `generate_a2ui`, the in-tool design call declares nothing) with the tool-less fixtures moved last, since several pills' brief is a substring of the pill text. Tests (both mutation-verified — they fail with the fix reverted): - aimock-a2ui-routing.test.ts drives aimock's real `matchFixture`; 14/16 cases fail on the old fixtures with "no fixture matched the secondary design call". - tanstack-factory.test.ts covers the `/delegations` and `/steps` deltas. Typecheck unchanged at 61 pre-existing errors; oxlint clean on changed files. NOT verified live: the four prompt/`text.format` changes need a real-LLM click-through, which needs a key this environment doesn't have. Reasoning and the exact request mapping are documented inline. |
||
|
Alem Tuzlak
|
734294155a |
fix(showcase/ms-agent-dotnet): copy manifest.yaml into the runtime image
PR #6130 added `src/middleware.ts`, which sets the `x-pathname` header that `src/app/demos/layout.tsx`'s `generateMetadata()` reads. That header is what makes the layout actually call `loadDemoIndex()` — a request-time `readFileSync(process.cwd()/manifest.yaml)`. The Dockerfile's runner stage never copied `manifest.yaml`, so every `/demos/*` route now 500s with "An error occurred in the Server Components render" (ENOENT), while the statically-prerendered home page keeps returning 200. On the dashboard that reads as "service is up, every cell pinned at D3": D4 fails on `page.type` waiting for `textarea`, D5 times out, D6 fails on `waitForSelector('[role="textbox"]')` — the chat input never mounts because the page is an error boundary. langgraph-python hit this exact bug and fixed it with the same one-line COPY; ms-agent-dotnet is the only integration shipping `middleware.ts` without it. Adds a ratchet test over that invariant (mutation-verified: fails with the COPY removed). |
||
|
Benjamin Taylor
|
ad67e9f4d6 |
docs(channels): keep the Channels pages deployment-neutral and self-sourcing
Two gaps from the previous commit, found reviewing it. The five platform pages previously carried a `your-intelligence-url` placeholder, which was neutral about where Intelligence runs. Replacing it with the managed hosts read as "this is the endpoint" to a self-hosted reader, and only the quickstart said otherwise; every env block now notes that self-hosted deployments substitute their own two hosts. index.mdx and mcp.mdx are code-only pages with no env block, so after requiring a second variable they referenced COPILOTKIT_INTELLIGENCE_WS_URL without ever telling the reader where it comes from. Both now point at the dashboard and the quickstart's env block. Refs OSS-621 |
||
|
Alem Tuzlak
|
809bb72e72 |
feat(showcase): bring ms-agent-dotnet to D6 (frontend parity + shared-state-streaming, a2ui-recovery, threadid) (#6130)
## What
Brings the **ms-agent-dotnet** (Microsoft Agent Framework .NET) showcase
integration from D5 to **D6**, using **langgraph-python** as the
north-star reference.
### 1. Frontend parity with langgraph-python
Restores near-identical frontends where ms-agent-dotnet had drifted,
while **preserving the load-bearing .NET adaptations** (per the showcase
iron rules — differences belong in fixtures/minimal backend, not the
shared frontend):
- Root shell: `globals.css` (Tailwind `@theme` block + brand green),
manifest-driven index `page.tsx`, `layout.tsx`, new `middleware.ts`
(`x-pathname`), `tsconfig` include.
- `declarative-gen-ui` subtree restored (fixes divergent pill testids
the shared probe asserts).
- Doc-snippet `@region` markers, import-style normalization, `subagents`
revert, stale-file cleanup, `auth` inspector flag.
- **Kept** (load-bearing, not reverted): `parse-json-result` 3-layer
unwrap, multimodal legacy-shim, tool-based `hitl` (MAF has no
`interrupt()`), `agent-config` `properties=`.
### 2. shared-state-streaming → per-token (removed from
`not_supported_features`)
`write_document`'s `document` arg now streams into `state.document`
per-token via a `createSharedStateStreamingAgent` route shim (mirrors
the proven `createGenUiAgent` bridge, with a partial-JSON string
decoder), since the .NET AG-UI host has no `predict_state_config`.
### 3. a2ui-recovery cell (new)
First MS-Agent-Framework implementation of the A2UI
validate→retry→`a2ui_recovery_exhausted` recovery loop. Because the MAF
AG-UI adapter can't emit the custom `ACTIVITY_SNAPSHOT{status:"failed"}`
the exhausted card needs, it's a **raw-SSE `MapPost` endpoint**
(`RecoveryAgent.cs`) — the same adapter-bypass pattern already shipped
for `/multimodal`. Adds the demo frontend, API route, deterministic
aimock fixture (heal seq0-invalid→seq1-valid; exhaust always-invalid),
and a unique per-slug `PROMPTS` entry in the shared probe.
### 4. threadid-frontend-tool-roundtrip demo (parity)
Added for demo-set parity (reuses the `frontend_tools` passthrough; not
a D6-scored feature, mirroring the reference).
`gen-ui-interrupt` / `interrupt-headless` remain honestly quarantined
(upstream `@copilotkit/react-core` `useInterrupt` resume-path bug — not
a backend gap).
## Verification
- Code was authored in parallel worktree-isolated slots, each
cross-verified against the reference + the shared probe contracts; the
a2ui-recovery fixture was cross-checked against
`RecoveryAgent.ValidateComponents`.
- Local D6 harness: the image builds and the stack + probes run, but
**full local green was blocked by Windows-only harness friction**
(`core.symlinks=false` breaks `stage_shared`'s `[ -L ]` materialization;
`--direct` doesn't context-scope the `x-aimock-context` header so
context-keyed a2ui fixtures miss). These are environmental, not code
issues. **Relying on CI's Linux harness (real symlinks + fleet worker)
for authoritative D6.**
## Follow-up (not in this PR)
- `stage_shared()` should also materialize Windows symlink-as-file
entries (detect a regular file whose content is a relative path), so
forced local rebuilds work on `core.symlinks=false` checkouts.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
lukasmoschitz
|
eca1d741c8 |
fix(showcase): align langgraph-fastapi to north-star langgraph-python (OSS-582) (#6184)
## Summary
Aligns the **langgraph-fastapi** showcase integration to the north-star
**langgraph-python** (OSS-582 — "Align LangGraph (FastAPI) Showcase
demos and code"). Moves the integration onto the product-centric demo
set: backend agent graphs, runtime wiring, frontend chrome, the
demo-browser overview (`manifest.yaml`), and aimock fixtures.
Linear: OSS-582.
## What changed
**Backend agents (`src/agents/src/`)**
- Ported the missing dedicated graphs: `agentic_chat`, `gen_ui_agent`,
`gen_ui_tool_based`, `shared_state_streaming`.
- v2 `create_agent` ports + config alignment for `agent_config_agent`,
`headless_complete`, `reasoning_agent`, `tool_rendering_agent`,
`tool_rendering_reasoning_chain_agent`, `a2ui_dynamic`.
**Runtime wiring**
- `route.ts`: wired the canonical demos to their dedicated graphs and
removed them from the generic `sample_agent` fallthrough loop (this is
what made several demos behave correctly against a real LLM — see
below); added `recursion_limit`.
- `langgraph.json`: registered the 3 new graphs.
- Renamed the dedicated API routes
`copilotkit-byoc-{hashbrown,json-render}` →
`copilotkit-declarative-{hashbrown,json-render}`.
**Overview / content**
- `manifest.yaml`: demos + features aligned to LGP (names, descriptions,
tags, order, canonical ids; deprecated aliases migrated; phantom
`hitl-in-chat-booking` removed; `shared-state-streaming` un-quarantined
now that it works). Integration identity (name/slug/logo) preserved. The
demo-browser overview is now card-for-card identical to
langgraph-python.
- Restored two missing `@region` markers (factory-automation snippet
extraction).
**Frontend chrome**
- `globals.css`, `layout.tsx`, `middleware.ts` (was missing),
`declarative-gen-ui/*` + `sales-context.ts`, `beautiful-chat`;
`Dockerfile` now copies `manifest.yaml` (fixed an RSC crash on every
`/demos/*`).
**aimock fixtures (`aimock/d4|d6/langgraph-fastapi/`)**
- D6 fixture fixes for the previously-red cells (stale `turnIndex`
gates, cross-file substring shadowing, missing `chunkSize`, prompt
narrowing).
**Showcase tooling / docs**
- Fixed harness services racing on a shared image tag
(`docker-compose.local.yml`).
- `GOTCHAS.md` #8: documents that aimock D6 can be green while a demo is
broken against a real LLM (fixtures replay scripted tool calls
regardless of which graph ran), plus how to catch it.
- `PARITY_NOTES.md`: sanctioned divergences (a2ui-recovery per-slug
prompt isolation; declarative-json-render scoped-test divergence).
## Verification
- **D6 sweep: 38 green / 1 red.** The single red is **a2ui-recovery**,
which reproduces **identically on the north star** (shared Python
`ag_ui_langgraph` recovery loop; mastra's TS impl is green). It is not a
fastapi defect and is tracked as a separate PR against the north star.
Documented in `PARITY_NOTES.md`.
- `validate-manifests` (manifest → registry, all 20 integrations) —
green.
- `validate-routes --all` (runtime-route wiring) — green.
- Build + TypeScript typecheck (`next build` via the Docker image build)
— green.
- `@region` audit — all region pairs balanced and LGP-consistent (0
orphans/typos/mismatches).
- Live manual QA against real OpenAI (`:3102` vs `:3100`) confirmed the
key demos (reasoning-custom, shared-state-streaming, gen-ui-tool-based,
agentic-chat).
## Notable finding
Several demos were D6-green but broke against a real model because they
fell back to the generic `sample_agent` instead of their dedicated graph
— the aimock fixture masked the wiring bug by replaying scripted tool
calls. This PR fixes the wiring and documents the gap (GOTCHAS #8).
D6-green is necessary but not sufficient for graph/tool-dependent demos;
a real-LLM click-through is required.
## Deferred / follow-ups
- **a2ui-recovery** — shared north-star defect in the Python
`ag_ui_langgraph` recovery loop; separate PR against the north star.
- **d5-byoc probe** — always sends the hashbrown pill even on the
json-render page (fleet-wide harness limitation); tracked as a
follow-up. The gating `byoc` grid cell is green.
## Scope note
Only `langgraph-fastapi` integration code, its aimock fixtures, and
showcase tooling/docs changed. **langgraph-python (the north star) was
not touched.**
|
||
|
Ran Shemtov
|
0f52c5260e | Merge branch 'main' into claude/youthful-williams-bf7746 | ||
|
David McKay
|
d892b7975c | Merge branch 'main' into docs/react-native-production-guide | ||
|
Benjamin Taylor
|
8fb739fc53 |
docs(channels): stop teaching a websocket URL derive that cannot work
The published Channels docs configured the Intelligence client with `wsUrl: apiUrl.replace(/^http/, "ws")` in all eight pages. Because the API and realtime planes are separate hosts, that yields a URL serving no socket, and the failure is a silent 30s hang rather than an error — so this was the most harmful copy of the bug: it is the surface developers actually read. Every page now reads COPILOTKIT_INTELLIGENCE_WS_URL from the environment alongside the API URL, and the six env blocks that previously showed only a `your-intelligence-url` placeholder document both hosts with the values that work against the managed service. The quickstart gains a short paragraph on why the second host is required and what going wrong looks like. Refs OSS-621 |
||
|
Mike Ryan
|
dacc667355 | fix(docs): resolve Angular overview guide links | ||
|
Mike Ryan
|
852990ccd1 | fix(docs): select Angular quickstart branch before MDX | ||
|
Mike Ryan
|
f43682b3db | test(showcase): align Angular staging assertion | ||
|
Mike Ryan
|
b7e83d7c1f | docs: make Angular journeys capability-aware | ||
|
Mike Ryan
|
8a2844b571 | docs: enforce angular-only documentation routes | ||
|
Mike Ryan
|
9988c23890 | docs: render shared concepts for Angular | ||
|
Mike Ryan
|
cc8689e21f | fix(showcase): bundle Angular docs source in image | ||
|
Mike Ryan
|
a87f1c9a30 | docs: close Angular information architecture gaps | ||
|
Mike Ryan
|
db732d3966 | docs: complete Angular threads API journey | ||
|
Mike Ryan
|
38333d3625 | docs: enforce Angular parity across backend guides | ||
|
Mike Ryan
|
92e5c31332 | docs: align Angular metadata with rendered variants | ||
|
Mike Ryan
|
fcf2357c25 | docs: add Angular-native content and source-backed snippets | ||
|
Mike Ryan
|
c569aa595d | docs: make Angular share the full information architecture | ||
|
David McKay
|
6eefe2b551 |
docs(react-native): correct polyfill auto-install claim for /components
Independent re-verification against packages/react-native source found the page overstated polyfill auto-install: only /headless (and the root barrel, via its /headless re-export) side-effect-imports the polyfill barrel. src/components/index.ts imports no polyfills, so a consumer importing only from @copilotkit/react-native/components does not get them auto-installed; it relies on the provider imported alongside it. Corrects the 'all three surfaces auto-install' and 'every entry point installs these automatically' statements accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Lukas Moschitz
|
5d3dac1298 |
fix(showcase): restore langgraph-fastapi gen-ui-open.json fixtures (CI: aimock-fixtures)
CI 'Showcase: Validate' failed: aimock-fixtures.test.ts asserts every d6/*.json yields >=1 fixture, but gen-ui-open.json had been emptied to a _meta stub (0 fixtures) by an earlier open-gen-ui shadowing fix that consolidated the pills into gen-ui-tool-based.json. Restore gen-ui-open.json as a byte-for-byte context-swap of the canonical langgraph-python file (context -> langgraph-fastapi): 4 fixtures for the '3D axis' and 'Inline expression evaluator' pills, LGP's clean shape (hasToolResult:false leg-1, distinct toolCallId terminals). No shadowing loop: each file's leg-1/terminal pair is anchored by its own distinct toolCallId, so the done-signal always resolves. Verified: aimock-fixtures.test.ts 839/839 green locally; D6 gen-ui-open, gen-ui-open-advanced, gen-ui-custom all green (two real runs each). |
||
|
github-actions[bot]
|
b4cff204d1 | style: auto-fix formatting | ||
|
Ran Shem Tov
|
1225bd7999 |
fix(showcase): sweep PNI-100 relaxed-turnIndex re-emit — frontend-tools-async
Same class as the hitl-in-app fix: turnIndex:0 emit legs get re-served on the post-tool-result turn under aimock's default (relaxed) turn-index selection, re-firing the frontend tool instead of narrating. frontend-tools-async (query_notes): add hasToolResult:false to the 4 emit legs (3 pills + the 'project planning' catch-all) so they match the emit turn and are excluded once the tool result returns. The toolCallId narration legs round-trip fine here (verified on the wire) and are unchanged. Verified under relaxed aimock: pill -> single result card + narration, no re-emit, no error (was: double card + 'toLowerCase' error). subagents.json left as-is: its turnIndex 1/2 legs are intentional server-side step sequencing, not the client-re-POST pattern. open-gen-ui / open-gen-ui-advanced NOT touched here: their real fixtures live in gen-ui-open.json / gen-ui-open-advanced.json (already hasToolResult-gated, no turnIndex). The stale duplicate generateSandboxedUi entries in gen-ui-tool-based.json are a separate cleanup, tracked outside this PR. |
||
|
Lukas Moschitz
|
c9d8a9ac14 |
fix(showcase): restore missing @region markers on 2 langgraph-fastapi agents
The factory automation extracts curated snippets via @region markers. readonly_state_agent_context.py and shared_state_read_write.py were byte-aligned to langgraph-python except their @region markers had been stripped, so the factory would extract nothing for those two regions. Restore them to match LGP: - agent-context-setup around create_agent in readonly_state_agent_context.py - shared-state-setup around create_agent in shared_state_read_write.py Comment-only; no behavior change. Audit confirms all 49 fastapi region pairs are now balanced and LGP-consistent (0 orphans/typos/mismatches). |
||
|
Lukas Moschitz
|
efe51e7deb |
chore(showcase): remove fe-parity tool
fe-parity.ts was a scratch parity checker for this alignment work; it is no longer needed in the worktree. Drop it and de-reference it from langgraph-fastapi/PARITY_NOTES.md (reword the intro, convert the machine-readable fe-parity-allow allowlist into a plain human-readable 'sanctioned file-level divergences' list). No other references existed. D6 behavior remains the parity judge. |
||
|
Lukas Moschitz
|
b8f37c0c85 |
docs(showcase): document declarative-json-render scoped-D6 divergence as sanctioned
The byoc D6 featureType covers both declarative demos; the grid cell is green (hashbrown route+pill). A scoped declarative-json-render --d6 run reds for two non-fastapi reasons: (1) the shared d5-byoc probe always sends the hashbrown pill even on the json-render page (harness limitation — build context lacks demos[]; tracked as a separate follow-up), and (2) fastapi deliberately keeps hashbrown (@hashbrownai/react) and json-render (@json-render/react) as separate library integrations with their own pills/fixtures, vs LGP's unified render_dashboard contract. Sanctioned; do not collapse fastapi's json-render to the unified contract (content downgrade) — the real fix is probe-side. |
||
|
Lukas Moschitz
|
4aee3c6529 |
docs(showcase): GOTCHAS #8 — aimock D6 green can mask a demo broken against a real LLM
aimock replays scripted tool calls keyed on userMessage+context regardless of which graph ran or its tool set, so a demo mis-wired to the sample_agent fallback passes D6 green while breaking against a real model (OSS-582: gen-ui-tool-based looped on query_data; shared-state-streaming wrote only to chat; agentic-chat ran with the wrong tool set). Document that D6-green is necessary but not sufficient, the real-LLM live click-through procedure, and the structural tell (route.ts fallthrough loop vs LGP's dedicated wiring). |
||
|
Lukas Moschitz
|
bf2de51705 |
fix(showcase): wire langgraph-fastapi demos to dedicated graphs (drop sample_agent fallbacks)
Live QA surfaced demos that behaved wrong on fastapi because they fell back to the generic sample_agent (or were unregistered) instead of the dedicated graph LGP uses. aimock D6 masked these (fixtures script the tool calls), so they were green in the grid but broken against a real model. Audit of fastapi's agentNames fallthrough vs LGP's neutralAssistantCells found exactly these: - gen-ui-tool-based: port gen_ui_tool_based graph (tools=[], frontend supplies render_bar/pie_chart via useComponent). Was sample_agent, whose query_data tool + prompt made the model loop on data queries instead of rendering. - shared-state-streaming: port shared_state_streaming graph (StateStreaming middleware + write_document tool + document state). Was sample_agent, which never emits state.document, so it only wrote to chat. Remove from not_supported_features (now works, D6 green). - agentic-chat: port agentic_chat graph (tools=[]). Was sample_agent (7+ tools). - threadid-frontend-tool-roundtrip: wire to frontend_tools (was unregistered). - reasoning-custom: align reasoning_agent config to LGP (gpt-5.4 / effort medium / summary detailed; were gpt-5-mini / low / auto). Register the 3 new graphs in langgraph.json; remove the 3 names from the sample_agent fallthrough loop in route.ts. D6 green x2 for reasoning-display, gen-ui-custom, shared-state-streaming, agentic-chat; agent-config + tool-rendering-reasoning-chain re-verified (no regression). |
||
|
Lukas Moschitz
|
7bd1ca6e70 |
docs(showcase): record a2ui-recovery D6 red as a shared north-star defect
Soften the stale 'D6 green'/'under investigation' wording. A live trace confirmed the a2ui-recovery surface-missing D6 red reproduces identically on the langgraph-python north star (shared Python ag_ui_langgraph recovery loop; mastra's TS impl is green), so fastapi is already behaviorally aligned and there is nothing to fix under integrations/langgraph-fastapi/. Fixing the shared loop is a separate PR against the north star / the package. |
||
|
Lukas Moschitz
|
6af08217b2 |
fix(showcase): align langgraph-fastapi demo overview + canonical cell wiring to parity
The demo-browser overview (page.tsx, identical to LGP) groups cards by
demo.tags[0] and orders within a group by manifest.features[] index, so the
card arrangement is fully driven by manifest.yaml. fastapi's manifest was an
older curation: a different naming scheme (~26 demos), deprecated feature/demo
ids (agentic-chat-reasoning, reasoning-default-render, byoc-hashbrown,
byoc-json-render, hitl, phantom hitl-in-chat-booking), divergent tags/order,
and a missing shared-state-streaming/shared-state-read card.
Align manifest.yaml demos+features to LGP (names, descriptions, tags, order,
canonical ids), preserving fastapi identity (name/slug/logo/description) and
the sanctioned a2ui-recovery prompt divergence. Overview is now card-for-card
identical to langgraph-python.
Canonicalizing the ids pulled two cells into the D6 tested set that were
mis-wired to the old names; wire them to LGP's shape:
- reasoning-custom/reasoning-default -> reasoning_agent in copilotkit/route.ts
(were wired under agentic-chat-reasoning/reasoning-default-render).
- rename api routes copilotkit-byoc-{hashbrown,json-render} ->
copilotkit-declarative-{hashbrown,json-render}; align endpoints + the
declarative-hashbrown-demo agent id to what the demo pages request.
Full D6 sweep: 37 green / 1 red; reasoning-display and byoc now green (two
real runs each); the lone red is a2ui-recovery (known shared north-star
defect, identical on LGP).
|
||
|
Ran Shem Tov
|
96116dab1a |
fix(showcase): hitl-in-app mastra fixtures re-trigger approval loop under relaxed aimock (PNI-100)
The three turn-0 pill fixtures (refund/downgrade/escalate) were keyed on turnIndex:0. aimock's default (relaxed) turn-index selection re-serves a turnIndex-only fixture on the post-approval turn, so approving a useFrontendTool(request_user_approval) call re-emitted the tool call instead of narrating the outcome — the turn never settled (stuck text-unstable). Re-key each emit fixture on hasToolResult:false instead of turnIndex:0. It matches the emit turn (no tool result yet) and is excluded on the post-approval turn (hasToolResult:true), where the toolCallId narration fixture wins. No turnIndex dependency, so it is correct under both strict and relaxed aimock, and multi-pill-safe. Not a bridge/runtime bug: @ag-ui/mastra loops back correctly on real OpenAI and under strict aimock. Fault was fixture selection only. |
||
|
Lukas Moschitz
|
a19ed403cf |
fix(showcase): port langgraph-fastapi tool-rendering-reasoning-chain to parity
Two independent fastapi divergences from the green north-star kept this red:
1. Missing chunkSize:9999 on the 8 tool-call fixtures. Under aimock's global
8-byte chunking the large tool-call args failed to JSON-parse in one piece,
so the AAPL->MSFT reasoning chain stopped at leg 1 (turns 1 & 2). Byte-align
to LGP (and the green mastra/langgraph-typescript siblings carry the same
pattern).
2. Fixture-pool shadowing broke turn 3. aimock pools all d4+d6 fixtures for a
context and matches by userMessage substring, first-match-wins in load order
(d4 before d6). fastapi had broad keys LGP doesn't:
- d4/chat.json: 'weather' -> 'weather in San Francisco', 'flights from SFO
to JFK' -> 'Find flights from SFO to JFK.' (period-terminated so it stops
being a substring of turn 3's 'Find flights from SFO to JFK and show me
the weather there.').
- tool-rendering-{custom,default}-catchall.json: bare 'Find flights' ->
'Find flights from SFO to JFK.' (now matches LGP's exact key).
Also byte-align the backend agent to LGP: scriptable get_stock_price signature,
the detailed chaining system prompt, model gpt-5.4, reasoning summary detailed.
D6 tool-rendering-reasoning-chain green (two real ~21s runs). Shared-fixture
regression all green: agentic-chat, tool-rendering, tool-rendering-custom-catchall,
tool-rendering-default-catchall, headless-complete (Tokyo-weather consumer).
|
||
|
Mark
|
a3ad8960c9 | Merge branch 'main' into mark/ms-agent-dotnet-apikeyresolver | ||
|
Jordan Ritter
|
4947c4a785 |
fix(showcase): update validate-pins hash for the ag2 bound
The `<1.0.0` upper bound changes the text of an existing `[FAIL]` line
from `ag2 is not an exact pin (>=0.9.0)` to `(>=0.9.0,<1.0.0)`, which
shifts the ratchet's SHA-256 over the sorted FAIL set.
The FAIL *set* is otherwise identical -- count stays 31, nothing healed
and nothing regressed. Verified by reproducing the committed baseline
hash b47ca987 on a pristine origin/main tree, then diffing the FAIL
lines against the fixed tree; exactly one line differs:
-[FAIL] ag2: ag2 is not an exact pin (>=0.9.0)
+[FAIL] ag2: ag2 is not an exact pin (>=0.9.0,<1.0.0)
`ag2` matches FRAMEWORK_PATTERNS, which demands an exact `==` pin, so
the dep was already in the baseline's drift set before this change and
remains in it after. Hash-only update; the count is untouched, so the
"never raise the count without sign-off" invariant is not engaged.
|
||
|
Jordan Ritter
|
5a09e0e787 |
fix(showcase): pin ag2<1.0.0 to unblock CI after upstream module rename
ag2 1.0.0 (published 2026-07-27T00:58:37Z) renames its top-level module
from `autogen` to `ag2` and ships no `autogen/` package at all. The ag2
integration floats on `ag2[openai,ag-ui]>=0.9.0`, so every fresh CI
resolution now picks up 1.0.0 and collection fails at:
src/agents/_multimodal_normalize.py:75
from autogen.ag_ui import AGUIStream, RunAgentInput
E ModuleNotFoundError: No module named 'autogen'
This breaks `Python unit tests (3.10)` and `(3.12)` in showcase_validate
on main and on every open PR. It is time-based rather than commit-based:
main only looks green because it has not re-run since the release.
Pin rather than migrate. 1.0.0 is not a rename -- it deletes the stable
`autogen/ag_ui/adapter.py` and promotes the former `autogen.beta.ag_ui`
stream into its place, dropping `AGUIStream(event_interceptors=...)` and
replacing `dispatch(context=...)` with `dispatch(variables=...)`. Since
`_multimodal_normalize.py` overrides `dispatch` and forwards `context=`,
rewriting the import path alone would swap an import error for a runtime
TypeError. A real migration needs to port ~20 `autogen` import sites and
re-verify every ag2 demo cell, which is not an urgent-unblock change.
|
||
|
Mark
|
a9ce37b75c |
fix(showcase/ms-agent-dotnet): resolve OPENAI_API_KEY-first via ApiKeyResolver
Port ms-agent-harness-dotnet's ApiKeyResolver into ms-agent-dotnet so the 15 agent factories resolve the OpenAI credential as OPENAI_API_KEY (env) -> config[OPENAI_API_KEY] -> GitHubToken, and the endpoint via OPENAI_BASE_URL -> default, instead of hardcoding configuration["GitHubToken"] per agent. Previously ms-agent-dotnet's main chat clients authenticated only with the GitHub token. On a fixture-miss fall-through, aimock proxies to real api.openai.com, which rejects the GitHub token (invalid_api_key, surfaced as 502). ms-agent-harness-dotnet already works because it resolves OPENAI_API_KEY first; this brings ms-agent-dotnet to parity so its fall-through returns 200. Interim showcase-side mitigation while the aimock cross-provider guard (PNI-108, CopilotKit/aimock#340) is deferred. - Copy agent/ApiKeyResolver.cs verbatim from ms-agent-harness-dotnet (code copy, no new dependency) - Rewire 15 factories + A2uiSecondaryToolCaller to ResolveApiKey/ResolveEndpoint; drop the dead per-file DefaultOpenAiEndpoint const - Add tests/ApiKeyResolverTests.cs (precedence, mock-endpoint fallback, non-mock fail-fast) Verified: dotnet build 0/0, dotnet test 76/76, whitespace format clean, and a live OpenAI call through the resolver returned 200. |
||
|
Jordan Ritter
|
bde2d99a5b |
fix(showcase/ms-agent-python): keep the user's prompt on the multimodal PDF turn (#6159)
`d6:ms-agent-python/multimodal` has been red in staging and prod since
2026-05-30. Turn 1 (image) passes; turn 2 (PDF) fails. This fixes it —
**without touching the fixture**, because the fixture was never the
problem.
## The verbatim turn-2 error
Backend (`showcase-ms-agent-python`), and reproduced locally:
```
[/multimodal] Streaming failed
openai.InternalServerError: Error code: 503 - {'error': {'message': 'Strict mode: no fixture matched',
'type': 'invalid_request_error', 'param': None, 'code': 'no_fixture_match'}}
The above exception was the direct cause of the following exception:
agent_framework.exceptions.ChatClientException: ("<class
'agent_framework_openai._chat_completion_client.OpenAIChatCompletionClient'> service failed to
complete the prompt: Error code: 503 - {'error': {'message': 'Strict mode: no fixture matched', …
```
Surfaced in the browser as `An internal error has occurred while
streaming events.`, with the probe reporting `failure_turn: 2`,
`turns_completed: 1`.
## Request-shape diagnosis
This reads like a fixture gap and is not one. I pulled the **actual
outbound request** off the local aimock's `GET /__aimock/journal` during
a failing run. Turn 2, verbatim (bodies elided):
```
[0] role=system "You are a helpful assistant. The user may attach images or documents…"
[1] role=user "can you tell me what is in this demo image I just attached"
[2] role=user [image_url <data:image/png;base64,iVBORw0K…>]
[3] role=user [image_url <data:image/png;base64,iVBORw0K…>]
[4] role=assistant "The attached image is the CopilotKit logo — a clean, geometric mark…"
[5] role=user "can you tell me what is in this demo pdf I just attached"
[6] role=user "[Attached document]\nCopilotKit Quickstart\nAdd AI copilots to your React…"
[7] role=user "[Attached document]\nCopilotKit Quickstart\nAdd AI copilots to your React…"
```
One logical user turn arrived as **three separate user messages**, and
the *last* one carries only the flattened document — the question is
nowhere in it. That is why aimock's strict mode refused it:
`userMessage` is a substring match against the last user turn, and the
last user turn was a PDF dump.
**Root cause:** `agent_framework_openai` emits **one OpenAI message per
`Content`**. `_chat_completion_client._prepare_message_for_openai`
builds a fresh `args` dict on every iteration of its content loop, so a
user `Message` carrying `[prompt_text, flattened_doc_text]` serialises
to two consecutive user messages — prompt-only, then document-only.
`_PdfFlattenChatMiddleware` was appending the flattened `[Attached
document]` text as a *second* text `Content` beside the prompt, which is
exactly the shape that gets split.
Two corroborating details that make the mechanism airtight:
- **Why turn 1 (image) passes.** aimock already skips *text-less*
trailing user messages (`getLastUserText` in `router.ts`, whose comment
documents this exact MS Agent Framework behavior). The image turn's
split-off trailing message has no text at all, so aimock falls back to
the prompt message and matches. The PDF turn's trailing message *does*
have text — the document — so there is nothing to skip past.
- **Why `langgraph-python` is green** doing the identical `[Attached
document]` flattening: LangChain keeps multiple text parts *inside one
message* rather than splitting them into separate messages.
This is a product bug, not a mock artefact. Against a real LLM it would
not 503 — the model would just answer the wrong thing, because the
question is buried behind a document dump instead of being the current
turn.
## The fix
`showcase/integrations/ms-agent-python/src/agents/multimodal_agent.py`
1. **Merge** the flattened document *into* the message's existing prompt
text content instead of appending it as a second content. The turn stays
a single text content and serialises to a single user message:
`"<prompt>\n[Attached document]\n<body>"`.
2. The merge **copies** the prompt `Content` rather than mutating it.
This is load-bearing: the middleware restores the original `contents`
list after `call_next`, and that restore only undoes the *list* swap —
an in-place mutation would leak the raw PDF body into the AG-UI
`MESSAGES_SNAPSHOT` and render a wall of PDF text in the user's chat
bubble. There is a test for this.
3. **Attachment-only turns** (a PDF with no question) still work: with
no text content to merge into, the flattened document stands alone as
the message body.
4. **Dedupe identical flattened blocks.** The page's
`LegacyConverterShim` appends a legacy `binary` mirror alongside every
modern attachment part, so the same PDF reached the middleware twice and
its body was being sent to the model twice (visible as the duplicated
`[6]`/`[7]` above). Now emitted once.
Post-fix outbound turn 2, same journal endpoint:
```
[5] role=user "can you tell me what is in this demo pdf I just attached\n[Attached document]\nCopilotKit Quickstart\nAdd AI copilots to your React application with CopilotKit…"
matched fixture userMessage: "can you tell me what is in this demo pdf I just attached"
```
One user message, prompt intact, document intact, emitted once.
## The fixture is untouched
```
$ git diff --stat origin/main -- showcase/aimock/
(empty)
```
The existing `userMessage` match key was always correct; the corrected
request shape is what satisfies it. Relaxing or re-recording the fixture
to match the broken request was an explicit non-goal — it would have
made the cell actively certify a model that never sees the user's
question.
## Same-pattern audit
- `_PdfFlattenChatMiddleware` is the **only** `ChatMiddleware` in
`ms-agent-python`, and the only place in the integration that constructs
`Content` or reassigns `message.contents` (`grep` for `ChatMiddleware` /
`Content.from_text` / `.contents =` across `src/` returns hits in this
one file only). No second instance of the pattern to fix.
- `ms-agent-python` is the only MS-Agent-Framework Python integration
doing PDF flattening — `ms-agent-dotnet` has a multimodal e2e spec but
no Python agent. The other `[Attached document]` implementations
(`langgraph-python`, `langgraph-fastapi`, `agno`, `claude-sdk-python`,
`langroid`, `pydantic-ai`, `langgraph-typescript`, `built-in-agent`) run
on frameworks that do not split a message's contents into separate wire
messages, so they are not exposed to this. The upstream
one-message-per-`Content` behavior is pinned by a dedicated test, so if
it ever changes we find out by that test failing rather than by a silent
regression.
- The file is a regular per-integration file, not a `shared/` symlink
(`git ls-files -s` → `100644`). No shared code touched;
`validate-shared-symlinks.ts` confirms no new erosion.
## Red / green / control
All three on the real probe surface, from a clean worktree at
`origin/main` `38613623f4`.
### RED — before the change
```
$ bin/showcase test ms-agent-python:multimodal --d6 --direct --verbose --cycle --isolate
[conversation-runner] turn 1/2 — assistant settled { bubbleIndex: 0, textLength: 100, hasAssertions: true }
[conversation-runner] turn 1/2 — assertions passed
[conversation-runner] turn 2/2 — sending message { inputLength: 29, timeoutMs: 60000 }
[conversation-runner] turn 2/2 — FAILED {
errorCategory: 'assertion-failed',
turnsCompleted: 1,
elapsedMs: 1577,
bodyTextLength: 421,
hasTextarea: true,
hasErrorBoundary: false
}
[warn] CVDIAG component=harness-d6 boundary=fixture-match … status=miss … error=chat errored: copilot-error-banner visible — An internal error has occurred while streaming events.
[info] probe.e2e-full.service-complete {"slug":"ms-agent-python","passed":0,"failed":1,"skipped":0,"incapable":0,"total":1,"state":"red","durationMs":9384}
✗ d6:ms-agent-python red (9.5s)
multimodal: chat errored: copilot-error-banner visible — An internal error has occurred while streaming events.
0 passed, 1 failed (9.5s)
⚠ Tests failed for ms-agent-python:multimodal (exit 1)
```
Evidence the outbound request lacked the prompt — aimock journal from
that run, 8 entries, `200,503,503,503,200,503,503,503` (2 attempts × 3
retries on turn 2):
```
[5] role=user STRING "can you tell me what is in this demo pdf I just attached"
[6] role=user STRING "[Attached document]\nCopilotKit Quickstart\nAdd AI copilots to…"
[7] role=user STRING "[Attached document]\nCopilotKit Quickstart\nAdd AI copilots to…"
status: 503
```
### GREEN — after the change, fixture unchanged
```
$ bin/showcase test ms-agent-python:multimodal --d6 --direct --verbose --rebuild --keep --isolate
[conversation-runner] turn 1/2 — assistant settled { bubbleIndex: 0, textLength: 100, hasAssertions: true }
[conversation-runner] turn 1/2 — assertions passed
[conversation-runner] turn 2/2 — assistant settled { bubbleIndex: 1, textLength: 233, hasAssertions: true }
[conversation-runner] turn 2/2 — assertions passed
[conversation-runner] conversation completed successfully { turnsCompleted: 2, totalDurationMs: 8279 }
[info] probe.e2e-full.feature-complete {"slug":"ms-agent-python","featureType":"multimodal","pass":true,"durationMs":8788}
[info] probe.e2e-full.service-complete {"slug":"ms-agent-python","passed":1,"failed":0,"skipped":0,"incapable":0,"total":1,"state":"green","durationMs":10187}
✓ d6:ms-agent-python green (10.5s)
1 passed (10.5s)
✓ Tests passed for ms-agent-python:multimodal
```
Both turns pass. aimock journal for that run: **2 entries, statuses
`200,200`** (down from 8 entries with six 503s — no retries needed).
**The fixture was not modified**; `git diff origin/main --
showcase/aimock/` is empty and the diff is two files, both under
`showcase/integrations/ms-agent-python/`.
### CONTROL — an already-green integration, same command, same stack
```
$ bin/showcase test langgraph-python:multimodal --d6 --direct --isolate
[conversation-runner] turn 2/2 — assistant settled { bubbleIndex: 1, textLength: 233, hasAssertions: true }
[conversation-runner] turn 2/2 — assertions passed
[conversation-runner] conversation completed successfully { turnsCompleted: 2, totalDurationMs: 8395 }
✓ d6:langgraph-python green (9.1s)
1 passed (9.1s)
✓ Tests passed for langgraph-python:multimodal
```
Local harness, shared probe, shared frontend and fixtures are all sound
— the red was specific to this integration.
## Covering test
`showcase/integrations/ms-agent-python/tests/python/test_multimodal_pdf_prompt.py`
— 7 tests. Not fakes: each one drives the real
`_PdfFlattenChatMiddleware` and then the real
`OpenAIChatCompletionClient._prepare_message_for_openai`, and asserts
against the actual OpenAI wire payload. The PDF is the bundled
`public/demo-files/sample.pdf` through real `pypdf`, and the prompt
asserted on is **read out of the real aimock fixture** rather than
hardcoded, so the test fails if either side drifts.
Test-level red→green (stash the source change, keep the tests):
```
# pre-fix
FAILED test_multimodal_pdf_prompt.py::test_pdf_turn_last_user_message_contains_the_prompt
FAILED test_multimodal_pdf_prompt.py::test_pdf_turn_serialises_to_a_single_user_message
FAILED test_multimodal_pdf_prompt.py::test_duplicate_pdf_parts_are_flattened_once
3 failed, 4 passed in 2.37s
```
with the primary failure reading:
```
AssertionError: expected the PDF turn to serialise to 1 user message, got 2:
['can you tell me what is in this demo pdf I just attached',
'[Attached document]\nCopilotKit Quickstart\nAdd AI copilots to']
```
```
# post-fix — full integration suite (6 pre-existing CVDIAG + 7 new), CI's exact invocation
$ PYTHONPATH=".:src" python -m pytest tests/python/ -q
13 passed in 2.40s
```
Coverage: prompt survives to the final user turn; the turn stays one
user message; the upstream one-message-per-`Content` split is pinned;
original `contents` restored and the prompt `Content` not mutated;
duplicate mirror parts flattened once; attachment-only turn still
flattens; image turn left byte-identical.
## Pre-push
`validate-parity.ts` 20/20 pass · `validate-shared-symlinks.ts` no new
erosion · `aimock-fixtures.test.ts` 842 pass · full `tests/python/`
suite 13 pass · lefthook `lint-fix` + `commitlint` clean · Python lines
≤88 cols matching the file's existing style · no lockfile churn, two
files in the diff.
## Scope
One cell, one middleware, one integration. The other five red
`multimodal` cells from the same sweep have five different root causes
and are not addressed here.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01PYdjeveT8Xof9TyHWMLoJr
|
||
|
Jordan Ritter
|
edc59c476c |
fix(showcase): cold-load gray chip was masking genuinely-red cells (#6156)
> **Scope, stated up front.** This PR fixes **one** of the ways a
genuinely-red
> showcase cell can render as something other than red: the cold-load
`signal`
> polarity. It does **not** eliminate the symptom. **Nine further
masking
> mechanisms remain** and are enumerated, with measured counts, under
> [What this PR does NOT fix](#what-this-pr-does-not-fix). Please read
that
> section before treating a green dashboard as evidence of a green
fleet.
## The bug
Absence of evidence about *why* a cell failed was erasing the fact that
it failed.
`classifyRung` opened its red branch with:
```ts
if (!raw.signalKnown) {
return { ...base, contribution: "NO_DATA", rawStatus: null };
}
```
`signalKnown` is `false` when the dashboard's bulk fetch projected the
`signal`
blob away. So a **genuinely RED rung whose infra-vs-product attribution
was merely
PENDING got classified `NO_DATA` and painted gray** — the muted "no live
probe
data" treatment — while the depth strip beside it, built from the raw
fold,
correctly rendered `1P ✗`. One model object simultaneously said
`d5.status ===
"red"` and `chipColor === "gray"`.
Two things make that polarity wrong:
1. **`signalKnown === false` never meant "known to have no signal."**
PocketBase
omits the `signal` key *only* under a `fields=` projection; a row that
genuinely has no signal arrives as `null`, and `null !== undefined`. The
flag
is pure provenance: "this row came from a projected fetch." It was never
evidence the failure was infra.
2. **The guess was wrong on the overwhelming majority.** Of the
non-green rows in
production, **94.2% are product-class and 5.8% infra-class** (measured
today
against prod PocketBase using the classifier's own imported
`signalHasInfraErrorClass` predicate: 3082 rows, 359 red, 21 infra-class
—
`driver-error` 11, `abort` 10 — 338 product-class). Graying every
unattributed
red lost 338 real failures to suppress 21 false alarms. A masked red is
never
investigated; a false alarm is looked at once and closed.
This was **not** a first-paint race. The gray state is a fully-rendered,
confident-looking wrong answer that persists until the probe's next
sweep rewrites
that specific row and the SSE delta redelivers it with `signal`, and it
recurs on
**every** reload, because the projection strips `signal`
unconditionally.
**Worst case for the routine per-cell dimensions is ~60 min**, from the
hourly
`e2e-demos` / `starter_smoke` / `d6-all-pills-e2e` crons in
`harness/config/probes/*.yml`; drift dimensions are effectively
unbounded. *(An
earlier revision of this body claimed "~29 min observed, ~14 min mean".
Both were
withdrawn during review: "~29 min" was derived from the `*/30` tier and
missed the
hourly tier entirely, and "~14 min mean" had no derivable basis. Do not
cite them.)*
## Measured effect
PR #6169's exhaustive sweep — **279,936 rung combinations** (7 status
rows × 6 row
variants), the same map fed to the pre-fix and post-fix engines and
diffed
field-by-field — puts the direction of this change beyond argument:
- **74,256 cells change**, and **every one moves toward *more* red.**
- **Zero** flips toward less red: no `red→gray`, no `red→green`, no
`amber→green`
anywhere in the diff.
- INV2–INV6 hold across the full sweep on the branch engine (0
violations).
Red-green on **live production data** (local `next dev` wired to prod
PocketBase,
driven with Playwright, screenshots visually inspected — DOM
measurements alone are
not trusted for visual verification; both captures back-to-back on the
same
underlying rows, only the code changing):
| Band | before | after |
|---|---|---|
| green | 632 | **632 — unchanged, no green regression** |
| amber | 0 | 0 |
| **red** | **0** | **51** |
| no data | 76 | **25** |
`0 red → 51 red`, `76 → 25 no data` — 51 recovered, arithmetic closes.
Red on
**first paint**, no gray plateau.
## The fix — two parts
### (a) Flip the fail-safe polarity — `cell-model.contribution.ts`
A red rung with unknown infra-ness now classifies `FAIL_FRESH`. Graying
a red
requires **positive infra evidence**: `hasNonInfraRed` is false only
when every
contributing red row's blob actually carries an `INFRA_ERROR_CLASSES`
attribution,
and signal-provenance is retained as an explicit second precondition so
a missing
blob can never be read as an infra attribution even if
`signalHasInfraErrorClass`
is loosened later.
```ts
if (fold.redSignalKnown && !fold.hasNonInfraRed) {
return { ...base, contribution: "INFRA_RED_FRESH", rawStatus };
}
```
`redSignalKnown` is derived inside `foldFamily`'s **red-row**
bookkeeping, not
family-wide, so a projected-away GREEN sibling cannot suppress a
legitimate infra
gray. (The original family-wide flag made `INFRA_RED_FRESH` effectively
**unreachable in the browser for every multi-row family** — D4, any
multi-pill
D5/D6; only single-key D3 was unaffected, which is why every
pre-existing fixture
missed it. Fixed by round-1 unit G1.)
This is the durable fail-safe: any future projection or fetch regression
now
degrades toward **over-reporting** failure rather than silently hiding
it.
### (b) Make first paint correct — `useLiveStatus.ts`
The supplemental `signal` fetch is widened from the comm-error
aggregates to a
**union**: the existing clause OR `state != "green"`.
```
before: filter=(dimension = "d6" || … || dimension = "d5-single-pill-e2e") && key !~ "%/%"
after: filter=((dimension = "d6" || … || dimension = "d5-single-pill-e2e") && key !~ "%/%") || (state != "green")
```
The existing fetch was not a viable vehicle as-is: it excluded these
rows twice
over — dimension `d5` is absent from `FLEET_COMM_AGGREGATE_DIMENSIONS`
entirely,
*and* its `key !~ "%/%"` predicate excludes every per-cell ladder-rung
key.
Relaxing only the `%/%` predicate would still have missed them.
**Why widen the existing fetch rather than add a second one:** it reuses
the
freshness guard (`supplementalRowIsOlder`) and the chimera-avoidance
merge
verbatim — that merge already reasons carefully about a newer bulk row
vs an older
signal-bearing twin, and duplicating it would be the risky part. It also
keeps
first paint at one extra request instead of two.
**Why scope by STATE, not by key:** signal-provenance is consulted
**only** in
`classifyRung`'s red branch — green and degraded paths never touch it —
so only
non-green rows can change a chip verdict. `!= "green"` rather than an
explicit
red/degraded list is deliberate: it also catches an out-of-vocabulary
state, which
`rankOfState` ranks **worst** — so the rows that matter most can never
fall outside
the fetch. The comm-error clause is kept as an independent disjunct, not
replaced:
a **green** aggregate row can still carry an active comm-error blob.
The fetch is page-capped at `MAX_INITIAL_FETCH_PAGES = 20`, projected to
`STATUS_LIST_FIELDS,signal`, and non-fatal at the fetch site —
truncation is handled
**asymmetrically**: supplemental truncation degrades (warn, use what
arrived), bulk
truncation throws. (Round-1 unit FETCH; the fetch was originally
unbounded, and its
rejection propagated to `setRows([])`, so an *enrichment* failure
blanked the whole
matrix.)
**Rejected: suppressing the chip while pending.** A blank or spinning
chip for up
to an hour on a large slice of the matrix hides reds exactly as
effectively as a
gray one, while additionally looking broken.
### What each part buys
Real engine over all 3082 production rows, three wires built from the
identical
row set *(green / amber / red / gray over the 708 catalog cells the band
counts)*:
| Wire | pre-fix engine | post-fix engine |
|---|---|---|
| FULL (server / `/api/matrix`) | 596 / 37 / 51 / 24 | 596 / 37 / 51 /
24 |
| COLD, pre-fix projection | **596 / 0 / 0 / 112** | 596 / 37 / 51 / 24
|
| COLD, post-fix projection | 596 / 0 / 49 / 63 | 596 / 37 / 51 / 24 |
- The pre-fix browser wire collapsed **88 cells** (51 red + 37 amber)
into gray.
The entire red *and* amber population was zeroed, because the
`!signalKnown`
early return preempted `FAIL_FRESH`, `FIRST_STRIKE_FRESH` **and**
`INFRA_RED_FRESH` alike.
- **(b) alone is not sufficient**: it recovers 49 of 51 reds and
**zero** of the
37 ambers, leaving 39 cells still mis-painted.
- **(a) is what delivers correctness** — with it, all three wires equal
server
truth exactly.
- **(b) is what keeps (a) from having to guess.** Today (a)-alone
happens to cost 0
false-positive cells, because no cell's verdict-determining rung is
currently a
pure infra red. That is luck, not a guarantee: prod holds 21
infra-classed red
rows. (b) supplies the actual evidence so those stay correctly gray, and
it also
feeds `decodeCellCommError` for non-green per-cell rows.
## Value test — per cell, all 6 (≥3 required)
Every affected cell verified individually, not inferred from a shared
symptom.
Live DOM on the real surface, plus the real engine driven offline over
all 3082
production rows through faithful replicas of the pre- and post-fix
browser wire:
| Cell (`d5:<slug>/multimodal`) | PB row | Server truth | Cold PRE |
Cold POST | Live chip PRE | Live chip POST | Flipped |
|---|---|---|---|---|---|---|---|
| `agno` | red, fc 1817 | red | **gray** | red | gray `D4` + `1P ✗` |
**red `D4`** | ✅ |
| `crewai-crews` | red, fc 1716 | red | **gray** | red | gray `D4` + `1P
✗` | **red `D4`** | ✅ |
| `llamaindex` | red, fc 2005 | red | **gray** | red | gray `D4` + `1P
✗` | **red `D4`** | ✅ |
| `mastra` | red, fc 1806 | red | **gray** | red | gray `D4` + `1P ✗` |
**red `D4`** | ✅ |
| `ms-agent-python` | red, fc 1710 | red | **gray** | red | gray `D4` +
`1P ✗` | **red `D4`** | ✅ |
| `built-in-agent` | red, fc 1803 | red | **gray** | red | gray `D4` +
`1P ✗` | **red `D4`** | ✅ |
6/6 flip. All six are `errorClass: conversation-error` — a real probe
conversation
failure, not in `INFRA_ERROR_CLASSES`, so `FAIL_FRESH` is the correct
classification. (Prod figures; `built-in-agent` is green on staging, so
staging
shows 5.)
**Depth distribution is byte-identical across the pair** (`D6:632 D5:0
D4:71 D3:0
D0:5` both times). The fix changes only the *colour*, never the depth —
the `D4`
label on these chips was always correct, so this must not be
mis-validated by
looking for a depth change.
## Tally-site audit
Every tally is downstream of `chipColor`, so all self-correct — none
needed
refactoring. All verified on the live surface post-fix:
| Site | Mechanism | Pre-fix | Post-fix | Verified |
|---|---|---|---|---|
| `feature-grid.tsx` `computeColumnTally` | gray-skip | `✗ 0` | `✗ 8`
(Agno) | ✅ header DOM |
| `feature-grid.tsx` `computeColumnTallyDetail` | gray-skip | empty list
| 8 features incl. **Attachments** | ✅ panel opened via
`tally-trigger-red` |
| `page-stats.ts` `computeHealthStats` | `case "gray": noData++` | `0
red / 76 no data` | `51 red / 25 no data` | ✅ band DOM + screenshot |
| `page-stats.ts` `computeDepthDistribution` | — (no gray gating at all)
| `D4:71` | `D4:71` | ✅ unaffected |
| `page-stats.ts` `computeD6Stats` | `d6Effective` | `null` → gray |
`"red"` → red | ✅ golden fixture |
**No other `chipColor` consumer changes behaviour.**
`d0-gone-predicate.ts` gates
the outage alerter on `chipColor === "red"`, but `d0-gone-monitor` reads
*full rows
with `signal`*, so provenance is always known there — as it is for
`/api/matrix`.
The polarity flip is reachable **only** where rows are projected, i.e.
exclusively
the browser cold-load path.
## A test asserted the buggy behaviour
`api-matrix-equivalence.test.ts` asserted:
```ts
expect(serverChip).toBe("red");
expect(coldLoadChip).toBe("gray");
expect(serverChip).not.toBe(coldLoadChip);
```
commented as "the state the browser converges to after its supplemental
`signal`
fetch … the more accurate answer." That comment is factually wrong — the
supplemental fetch resolved in ~1 s and structurally excluded those rows
twice over
— and the assertion pinned the defect as intended behaviour. **This is
why CI never
caught it.** Read in full, it is asserting the defect, not an adjacent
legitimate
case: the fixture is a genuine product red (`errorDesc: "assertion
failed: wrong
answer"`, no infra class) with `signal: undefined` on the red rung.
Inverted to assert **red on both paths**, with the fail-safe direction
explained
inline. Same for the harness's `§7 I5` case (`cell-model-v2.test.ts`)
and
`classifyRung`'s unit case. Four golden-master fixtures
(`pos-d{3,4,5,6}-red-signal-unknown`) re-frozen.
Negative controls added so the flip can't over-reach: an infra-classed
red **still**
grays (positive evidence present), and `§7 I4` (infra-only red → gray,
`isRegression` false) still passes untouched.
## The structural lever: coherence invariant INV7
`cell-model.coherence.test.ts` asserted INV1–INV6, all of which relate
`chipColor`
only to the other **chip-side** outputs. **None inspected the
`d3`/`d4`/`d5`/`d6`
depth pills.** So the engine could return one object saying both
`d5.status ===
"red"` and `chipColor === "gray"` with every invariant intact — which is
exactly
what happened, on every cold load.
INV7: **a gray chip above a red depth pill requires positive infra
evidence on
every contributing red row.** Red-greened against the old polarity — it
fails on 3
fixtures with a legible message:
```
pos-d5-red-signal-unknown: INV7 gray chip over a red pill requires POSITIVE infra
evidence on every red row — d5:acme/agentic-chat has none (signal STRIPPED (pending)).
A red whose cause is merely unknown must render red, not gray.
```
It closes the gap in the **only** safe direction: by fixing the chip
upward, never
by muting the strip. Deriving the pills from the classified
contributions — the
superficially cleaner unification — would render `1P ✗` as `1P ?` on
cold load and
destroy the only on-page element that was telling the truth, converting
a visible
inconsistency into an invisible failure.
INV7's own limitations are disclosed under
[Known limitations of the guards
themselves](#known-limitations-of-the-guards-themselves).
---
# Review-round fixes folded in
Three full 12-reviewer review rounds were run against this branch. Round
1
contributed five units plus an integration fix; round 2 contributed four
more. The
loop was **deliberately stopped** after round 3 by explicit decision
rather than
running a fourth; round 3's findings are catalogued (132 items in
`DEFERRED-r3.md`) and routed to the follow-up epic, not silently
dropped.
## Round 1 — five units
| # | Commit | Concern | Surface |
|---|---|---|---|
| G1 | `2a68282d` | Signal-provenance was derived family-wide but gated
a red-row-scoped predicate. Precondition moved into `foldFamily` as
`redSignalKnown`; `RawRung.signalKnown` **removed entirely** with all
its plumbing. | production + tests |
| A5 | `ebd7ac68` | The autocancel fake servers ignored
`filter`/`fields`, so the supplemental fetch's leftover-append silently
repaired a dropped bulk page — three pagination guards passed under
every mutation. Servers rebuilt to evaluate the query. | test-only |
| FETCH | `f5178765` | The supplemental fetch was unbounded (`for(;;)`),
unprojected, and its rejection propagated to `setRows([])`. Now capped,
projected, and non-fatal at the fetch site. | production + tests |
| INV7 | `3dea9f64` | INV7 was quantified over the whole live map rather
than the cell's contributing keys, and had no `isStaleCell` carve-out —
it failed two classes of legitimate rendering. Scoped, exempted, plus an
anti-vacuity guard. | test-only |
| DOCS | `2797618c` | Stale/unanchored numeric claims (row counts, byte
estimates, re-delivery window) corrected against production and given
their source queries. | comments only |
| — | `938be2ac` | **Integration fix.** A5 and FETCH disagreed about how
a supplemental request is *identified*; all three fake servers
misclassified it as a bulk page and the whole autocancel suite failed
3/3. Both dashboard suites now key on `signal` being IN the projection.
| test-only |
## Round 2 — the four integrated units
Integrated in order Q2 → Q1 → Q3 → Q4. Every guard was re-mutated
**after**
integration and reproduced its pre-integration result; no guard was
disarmed by
the merge.
| # | Commit(s) | Concern | What it proved | Surface |
|---|---|---|---|---|
| **Q2** | `b8b75b7d` | The dashboard's `useLiveStatus` test doubles
carried **three divergent implementations** of the same PocketBase query
evaluator, and the nominally-shared one was the weakest. Collapsed to
exactly one, every semantic a measured observation from a real
PocketBase v0.22.21 server. Deleted autocancel's 380-line private copy.
`evaluatePbList` **cannot throw**, so a parse error can no longer send
NO response and hang the hook to a 20 s `waitFor` death. | Six mutations
(short page dropped in wave merge, boundary logic removed, merge order
reversed, supplemental filter broken, supplemental filter widened to
`""`, bulk filter selects nothing) each reproduced their pre-integration
RED count exactly — 3/1/2/6/5/6 failures. The consolidated evaluator is
strictly stronger, not merely shorter. | test-only (−488 lines net in
the doubles) |
| **Q1** | `75fdfb3a` + `ed10b61f` | `truncated` meant "cap exhausted",
so a **complete** read that happened to end exactly on the cap page was
reported as truncated — the bulk caller threw, burned its reconnects and
landed on a blank dashboard behind an offline banner, on every retry and
every reload. `truncated` now means exactly "there are rows we did not
read", proven by ONE lookahead read of page cap+1 whose rows are
discarded and only whose emptiness is load-bearing. Algorithm hoisted to
an exported `paginateStatusPages(readPage)`. | Lowering the cap 20→10
and 20→5 each produces **10 failed / 6 passed**, where the pre-fix code
was **6/6 GREEN** — i.e. the guard was previously disarmed. A complete
10,000-row dataset renders `live`, `truncated === false`, seeded red
cell still `chipColor: "red"`, page set exactly `[1..21]`; 10,001 rows
still throws with `status: "error"` and "exceeded" in the message. |
production + tests |
| **Q3** | `49dea099` | INV7 false-failed on the ordinary I1 ladder-gap
shape ("no `e2e:` row + red `chat:<slug>`"). Added a carve-out keyed on
`ladderGapDepth`, **one-directional** (a gap *above* the red excuses
nothing) and never firing without a witness rung — plus a two-sided
canary that fails loudly the moment the underlying gap-break is fixed. |
Deleting the `isStaleCell` exemption fails exactly the U8 force-gray
case (the exemption is load-bearing, not decorative). Changing the
engine's gap-break `break` → `continue` fires the **two-sided canary**
with its intended message (`expected 'red' to be 'gray'`) — so the
allowance cannot go on silently excusing a shape the engine no longer
produces. Gapless and gap-above-red incoherence still FAIL; the
legitimate gap shape still PASSES. | test-only |
| **Q4** | `b04330ab` | Three explicitly load-bearing family folds were
invisible at golden-master level — `maxNonInfraRedFailCount`'s MAX,
`allRedSoftClass`'s EVERY quantifier, and `redSignalKnown`'s RED-ROW
scope — because no fixture anywhere carried two *differing* red rows. 5
new fixtures, baseline 56→61, 140 insertions / 0 deletions. | Each of
three mutations now fails via **exactly one** fixture:
`Math.max`→`Math.min` ⇒ `hetero-d4-red-failcount-max`; `every`→`any` ⇒
`hetero-starter-mixed-soft-hard`; `redSignalKnown` red-row→family scope
⇒ `hetero-d4-green-stripped-infra-red`. Precise attribution, not a
shotgun. | test-only |
One merge conflict, in `useLiveStatus.supplemental-bounds.test.tsx`: Q2
deleted the
file-local `classifyRequest` helper on the same lines where Q1 added its
new
`PAGES_THROUGH_LOOKAHEAD` constant. Resolved by keeping **both** intents
rather than
taking a side.
Baseline golden-master key count is **61**, and the completeness check
(`has exactly
one baseline entry per fixture (no stale/missing)`) runs, so a fixture
added without
a baseline entry fails rather than being skipped.
---
# What this PR does NOT fix
**This PR does not fully eliminate the original symptom.** It fixes the
`signal`-polarity masking mechanism and makes the cold load correct. A
genuinely-red
cell can still render **gray, green, or amber** by nine other routes.
All nine were
found by review of this branch and are catalogued, not speculative.
### Quantified by the #6169 sweep (279,936 combinations)
| Mechanism | Rung combinations |
|---|---|
| **M1** — ladder gap-break: `scanWorst` folds the FIRST `ABSENT`/`STUB`
rung as `ABSENT` and **`break`s**, so a fresh non-infra RED on a rung
*above* the gap is never folded and paints gray | **17,088** |
| **M2** — D6 outside the `scanWorst` walk: a red D6 never reaches the
chip | **2,048** |
| **M3** — gap not below the red: the remaining gray-over-red shapes |
**192** |
| | **19,328 total, 0 unexplained** |
M1 is highly reachable in practice: `chat:<slug>` / `tools:<slug>` are
integration-scoped, so one red `chat:` row plus an unemitted
`e2e:<slug>/<featureId>`
row (new feature, rotation slot, D3 driver never ran) is a gray chip
over a red D4
pill. `combine.ts` is deliberately untouched here — fixing any of M1–M3
re-verdicts
cells across the matrix and therefore needs its own PR with a
golden-master
**baseline re-freeze** and a **live value test**. Q3's INV7 carve-out is
silent about
M1 **on purpose and only about that**, and its two-sided canary is the
tripwire that
forces the carve-out to be deleted the moment M1 is actually fixed.
### Found in round 3, not quantified by the sweep
4. **Prototype-chain property lookups invert the anti-masking guard.**
`STATE_RANK` / `rankOfState`, `WORST_STATE_RANK` / `worstStateRank`,
`firstStrikeConfig[kind]` and `CATALOG_TO_D5_KEY` are plain object
literals read
without an own-property guard, so a runtime value colliding with an
`Object.prototype` member (`constructor`, `toString`, `valueOf`, …)
resolves to an
inherited function that the `??` fallback then never replaces. For
`rankOfState`
the out-of-vocabulary guard **inverts** — an unknown state, which the
code
deliberately ranks worst, instead classifies as an infra red and grays;
`worstStateRank` silently swallows the row entirely. `CATALOG_TO_D5_KEY`
is
unguarded at four further sites in `cell-model.ts`, where the
consequence is a
**whole-matrix render crash**. Fix is
`Object.prototype.hasOwnProperty.call` (or
a prototype-free table) at each site.
5. **Wave truncation bypasses the fail-loud throw entirely.** In the
fan-out merge,
a short page inside a wave causes every page issued *after* it in the
same wave
to be discarded — sound for a stable collection, but the pages resolve
concurrently against a table the harness writes continuously. The
discard
produces no log, no warning, no counter, no `degraded` flip, and
**`truncated`
stays `false`** because it is only ever set by the cap lookahead. So the
bulk
caller's deliberate "a truncated BULK read cannot degrade gracefully …
throw"
posture fires only for cap exhaustion, never for this class of loss. The
missing
rows render as *absent*, which `buildCellModel` resolves to `ABSENT` /
`NO_DATA`
and paints gray — the exact polarity this PR exists to remove.
`INITIAL_FANOUT_BATCH`
is documented as "purely a WIRE-EFFICIENCY knob" but is load-bearing for
correctness: at a larger batch the loss is `INITIAL_FANOUT_BATCH - 1`
pages.
6. **`INITIAL_PAGE_SIZE` above the server clamp grays ~6/7 of the
matrix.**
Termination is the client-side comparison `first.length ===
INITIAL_PAGE_SIZE`,
correct only because PocketBase clamps `perPage` to 500 server-side.
Raise the
constant — the obvious "fewer round-trips" tuning, which the surrounding
comment
invites — and every page returns 500, i.e. *short*, so the loop stops
after page
1 and returns ~500 of ~3100 rows with `truncated: false`. The bulk
caller sees a
complete read, never throws, and ~6/7 of the matrix renders no-data gray
with no
warning and no banner. The `truncated` contract cannot catch it because
the read
genuinely never hit the cap.
7. **An empty bulk page 1 publishes `live` with 0 rows.**
`paginateStatusPages`
returns `{rows: [], truncated: false}` for an empty collection and
`connect()`
then does `setRows([])` + `setStatus("live")` + `setError(null)` with no
empty-collection guard. Every cell renders no-data gray behind a healthy
"live"
indicator. Untested — the smallest fixture anywhere in these files is 3
rows.
The server-side analogue is already a known deferred item.
8. **First-strike de-amplification fires at `fail_count === 0`, painting
a genuine
red AMBER.** The gate is `maxNonInfraRedFailCount < threshold` with
`threshold`
2, so `fail_count` of **0** de-amplifies exactly like 1 →
`FIRST_STRIKE_FRESH` →
amber, i.e. "transient, not yet actionable". The threshold's semantics
are valid
only if a red row's `fail_count` is always ≥ 1, which is asserted as a
*producer*
property and enforced nowhere on the read side; out-of-vocabulary states
(the
harness can persist `"error"` as a no-data marker, which `rankOfState`
ranks above
red) and back-filled or externally-written rows both reach the
classifier at 0. A
`fail_count` of 0 on a red row is not positive evidence of a first
strike. The
golden master cannot see it: every red-bearing fixture uses
`fail_count: opts.failCount ?? (isRed ? 1 : 0)`, so no fixture puts a
red row at 0.
9. **An infra-red or `degraded` D1/D2 over a green ladder paints
chip=green,
achieved=6.** `combine.ts`'s §F gate fires only on `contribution ===
"FAIL_FRESH"`
in both the achieved-depth walk and `computeChip` step 1;
`INFRA_RED_FRESH` and
`DEGRADED` fall through as non-gating, so the cell keeps its green
ladder verdict
with `d6Effective: "green"`. Measured on the branch engine: `D1
infra-red
(driver-error) + green D2 → chip=green ach=6 reg=false d6Eff=green`, and
the same
for `D1 degraded`. **Mutation-proven invisible to the golden master:**
patching
*both* §F gate sites to also fire on `INFRA_RED_FRESH` and `DEGRADED` —
a
materially different verdict for every column whose health probe is
failing —
leaves **all 61 baseline entries byte-identical**. `health:<slug>` /
`agent:<slug>`
are integration-scoped, so this is whole-column. The polarity flip
closed the
`signal: undefined` hole here and left the `driver-error` / `abort` hole
wide open
— and `driver-error` / `abort` on a health probe is precisely what an
unreachable
container or a cancelled sweep produces.
# Known limitations of the guards themselves
Disclosed rather than omitted. These are properties of this PR's own
tests.
- **INV7's Q3 carve-out does not cover the M2 shape.** It is keyed on
`ladderGapDepth` and excuses only a gap strictly *below* the shallowest
red pill,
which is M1. The D6-outside-the-walk shape (M2, 2,048 combos) is a
different
mechanism and is not addressed by the carve-out.
- **The starter-axis INV7 canary short-circuits and proves nothing.**
`assertChipStripCoherent` returns immediately on `m.chipColor !== "gray"
||
redDepth === null`. The hard-red starter cell the test builds has
`chipColor ===
"red"`, so the function returns on the **first** disjunct — not on the
strip condition the comments claim. The call would pass identically if a
starter
cell *did* expose a red depth strip. The canary role is actually carried
by the
four `lvl.exists === false` / `lvl.status === null` assertions beside
it; the
`assertChipStripCoherent` call adds nothing. Compounding it,
`stripReadsRed` — the
identifier the comments reason about — does not exist anywhere in the
repo.
- **The five INV7 "teeth" assert `/INV7/`, which the drift-guard message
also
matches.** `assertChipStripCoherent` raises two structurally different
failures
and both messages contain the literal `INV7`: the keyspace-drift guard,
and the
real coherence failure. Every teeth test asserts only
`.toThrow(/INV7/)`. So if
`contributingKeys` ever drifts from the engine keyspace —
`collectAgentLadder`
gains a rung, `CATALOG_TO_D5_KEY` fan-out changes shape, a `keyFor`
segment changes
— `redRows` goes empty, the drift guard fires, and **all five teeth
still pass**
while INV7 has gone vacuous over the entire fixture matrix. That is
precisely the
state the drift guard was written to make impossible; it is silenced
inside the
only tests that reach it. The fix is to assert the specific message
(`/requires POSITIVE infra evidence/`) and pin the drift message in its
own case.
- **The test double cannot express absent-vs-null projection
provenance.**
`applyPbFields` silently drops projection tokens it does not model, so
the fake
cannot distinguish a key *omitted by a `fields=` projection* from a key
present
with value `null` — which is the single distinction `redSignalKnown`
rests on.
Fixtures therefore cannot validate the provenance semantics the fix is
built on;
that validation lives only in the live red-green capture above.
# Follow-ups
- **Epic: "A red must render red" — a single enforcement point for the
showcase
cell-model read model**, on Notion under CopilotKit *Plans / Proposals*:
https://app.notion.com/p/3a83aa38185281b99f7bfdd1d0f46bb9 — carries
M1–M3 and the
round-3 mechanisms above toward one enforcement point rather than nine
point-fixes.
- **`DEFERRED-r3.md` — 132 catalogued items** from review round 3
(comment accuracy,
test-double conformance, coverage gaps, altitude). Nothing in round 3
was
discarded; it was catalogued and routed.
# Quality gate
Re-run at head `b04330ab` in a clean detached worktree.
- **`oxfmt@0.36.0 --check`**: clean — "All matched files use the correct
format" (16
changed TS/TSX files)
- **`oxlint@1.51.0`** (repo `.oxlintrc.json`): **5 warnings, 0 errors.**
4 of the 5
reproduce verbatim on the `origin/main` versions of the same files
(`unicorn(consistent-function-scoping)` on `c`, `eslint(no-shadow)` on
`rows`,
`unicorn(no-array-reverse)`,
`typescript-eslint(consistent-type-imports)`). The one
net-new against `origin/main` is a second
`unicorn(consistent-function-scoping)` warning, on the test-local helper
`lvl` in
`cell-model.coherence.test.ts`. Zero errors; the repo's `oxlint` CI job
is green.
- **`tsc --noEmit`**: **0 errors** in `shell-dashboard`; `harness` clean
under
`tsconfig.build.json`. `harness`'s `tsc --noEmit` reports one
pre-existing error,
`frontend-matrix.test.ts` resolving
`../../../shell/src/data/frontend-catalog.json`
— a stale path in a file byte-unchanged from `origin/main`.
- **`shell-dashboard` tests** (measured locally at `b04330ab`): **71
files / 1407
passed / 1 skipped / 0 failed.** Scoped to `src/`;
`tests/runtime-env-switch.spike.test.ts` was deliberately not run (it
spawns
`next build` plus two `next start` invocations and is not a unit test).
- **`harness` tests** (measured locally at `b04330ab`): **3659 passed /
18 skipped /
3 failed** — the same three pre-existing D5 fixture-registry-drift
failures, in
files **byte-unchanged from `origin/main`** and driven entirely by
inputs none of
this PR's 17 files touch:
- `frontend showcase matrix > plans every runnable catalog cell without
loss or duplication` — `expected 664 to have a length of 660`
- `d5-mapping-drift > dashboard CATALOG_TO_D5_KEY structurally mirrors
harness REGISTRY_TO_D5` — the drift regex cannot locate
`CATALOG_TO_D5_KEY` in `shell-dashboard/src/lib/live-status.ts` (a
675-byte re-export shim on `main`, not modified here)
- `D5_REPRESENTATIVES > covers every D5FeatureType that has a registered
script` — missing `browser-use-smoke`
Those three are now **explicitly quarantined** by the `test /
unit-showcase`
workflow that landed on `main` in `7282ecddfb`
(`showcase/harness/vitest.quarantine.json`), and that workflow's
**quarantine
ratchet** re-runs each one to confirm it *still* fails — so they neither
hide nor
block. At this PR's head the ratchet passes: `Quarantine ratchet OK: all
3
quarantined file(s) still fail.`
- Diff hygiene: no lockfiles, no generated `src/data/*.json`.
## CI provenance — this is the PR's first honest CI
`main` has been **merged in** at `7282ecddfb` (merge commit `12ea621a`,
textually
clean, zero conflicts; the 13 commits above are unrewritten). This
matters for more
than being up to date: the `harness unit suite` and `shell-dashboard
unit suite`
jobs **did not exist** at `b04330ab`, because the workflow that enables
them landed
on `main` afterwards and GitHub runs `pull_request` workflows from the
PR head. The
earlier "17 pass" therefore ran **none** of the ~5,000 tests covering
this PR's own
subject matter (`cell-model/*` and `useLiveStatus*`).
- CI at this head (`12ea621a`): **21 pass / 0 pending / 0 fail** across
24 registered
rows.
- Both new jobs run and both pass — `shell-dashboard unit suite` **70
files / 1404
passed / 1 skipped**, `harness unit suite` green, quarantine ratchet OK.
- The **Q1–Q4 guard battery (16 re-proofs)** was re-run at the merge
head and
reproduced its pre-merge numbers exactly, so nothing was disarmed by the
merge:
Q1 cap→10 and cap→5 each **10 failed / 6 passed** (unmutated **16/16
green**);
Q2 M1–M6 = **3 / 1 / 2 / 6 / 5 / 6** (M4 flips the supplemental cap
test, M6 the
bulk cap test, M5 red in both worlds by design); Q3 `isStaleCell`
deletion = **1
failure** (the U8 force-gray case) and the engine `break`→`continue`
fires the
two-sided canary with `expected 'red' to be 'gray'`; Q4 each of the
three fold
mutations = **1 failed / 61 passed** via exactly one fixture
(`hetero-d4-red-failcount-max`, `hetero-starter-mixed-soft-hard`,
`hetero-d4-green-stripped-infra-red`), baseline key count **61**,
completeness
check present.
# Review guidance
**Surface: 17 files, +4880 / −382.** Most of that volume is tests,
fixtures, and the
regenerated golden-master baseline. Only three files carry production
behaviour:
`cell-model.contribution.ts`, `cell-model.ts`, and `useLiveStatus.ts`.
**The four round-2 units are separable commits** — `b8b75b7d` (Q2),
`75fdfb3a` +
`ed10b61f` (Q1), `49dea099` (Q3), `b04330ab` (Q4) — and round 1's are
too, so this
can be read unit by unit rather than as one diff.
**Highest-value review targets, in order:**
1. **`cell-model.contribution.ts` — the gate.** `if (fold.redSignalKnown
&&
!fold.hasNonInfraRed)`. This is the whole polarity decision. Worth
checking: that
`redSignalKnown` really is red-row-scoped inside `foldFamily` (a
family-wide
derivation is what made `INFRA_RED_FRESH` unreachable in the browser),
and that
the two conjuncts are genuinely independent evidence rather than one
restated.
2. **`useLiveStatus.ts` — `truncated` semantics.** Q1 redefined
`truncated` from
"cap exhausted" to "there are rows we did not read", proven by one
lookahead read
of page cap+1 whose rows are discarded and only whose emptiness is
load-bearing.
Worth checking: the asymmetry (bulk throws, supplemental warns) is
applied at the
right call sites, and that the lookahead genuinely cannot be confused
with a data
page. Note limitation 5 above — this contract does **not** cover
wave-level loss.
Everything else is fixtures, doubles, and the coherence suite; the
disclosed
weaknesses in those are listed under *Known limitations of the guards
themselves*
rather than left for a reviewer to rediscover.
|
||
|
Jordan Ritter
|
db75a04837 |
chore(showcase): mark multimodal unsupported for llamaindex and crewai-crews (#6158)
## What this is
`multimodal` (Attachments) has never worked on **`llamaindex`** or
**`crewai-crews`**, but both manifests
listed it under `features`, so the fleet probed it and reported **red**.
A red chip says "this regressed".
The truth is "this was never built". This PR marks both cells
**unsupported** instead. It does **not**
implement the feature, and it does **not** suppress the cell — the cell
still exists, the demo stays wired,
and the chip renders the 🚫 unsupported glyph.
## Mechanism used (existing, not invented)
The repo already has exactly one way to declare a feature unsupported
for an integration:
**`not_supported_features` in the integration's `manifest.yaml`**. The
full derivation chain:
| Step | Location |
|---|---|
| Declaration | `showcase/integrations/<slug>/manifest.yaml` ->
`not_supported_features:` |
| Schema | `showcase/shared/manifest.schema.json` — *"feature IDs that
this integration's framework cannot architecturally support … excluded
from parity computation"* |
| Status fold |
`showcase/harness/src/shared/catalog/catalog-flatten.ts:239` —
`determineCellStatus()` checks `not_supported_features` **first**,
returns `status: "unsupported"` |
| Input mapping |
`showcase/harness/src/shared/cell-model/catalog-input.ts:53` —
`isSupported: cell.status !== "unsupported"` |
| Model | `showcase/harness/src/shared/cell-model/cell-model.ts:847` —
`if (!isSupported) return UNSUPPORTED;` (the frozen singleton at `:551`:
`supported: false`, `chipColor: "gray"`, `isRegression: false`) |
| `/api/matrix` | `showcase/harness/src/http/matrix.ts:206` ->
`matrix-compute.ts:53` — projects that same model, so the API value
**is** the rendered chip by construction |
| Render |
`showcase/shell-dashboard/src/components/unified-cell.tsx:308` — `if
(!model.supported)` renders `data-testid="unified-cell-unsupported"`
with 🚫 and `title="Not supported by this framework"` |
The mechanical guard at `catalog-flatten.ts:169` rejects a feature that
appears in **both** `features` and
`not_supported_features`, so each entry was **moved**, not duplicated.
Note this mechanism is strictly stronger than a probe-side skip:
`buildCellModel` returns `UNSUPPORTED`
regardless of what the live PocketBase row says. Verified against the
existing not-supported cells on these
same two integrations, which carry **green** PB rows and still render 🚫:
```
llamaindex/gen-ui-interrupt matrix: chip=gray supported=False | PB rows: d5=green d6=green e2e=green
llamaindex/shared-state-streaming matrix: chip=gray supported=False | PB rows: d5=green d6=green e2e=green
crewai-crews/mcp-apps matrix: chip=gray supported=False | PB rows: d5=green d6=green e2e=green
```
So the cell can never read green *or* red once declared here — which is
the property we want.
## Per-integration reason (recorded inline in each manifest)
**`llamaindex` — upstream gap.** The pinned
`llama-index-protocols-ag-ui==0.2.2`
(`llama_index/protocols/ag_ui/utils.py:82-85`) passes an AG-UI
`UserMessage`'s `content` straight into
`ChatMessage(...)`. A text-only turn passes a plain string (fine — every
other llamaindex cell is green);
an attachment turn passes a **list** of AG-UI content-part models, which
pydantic routes into
`ChatMessage.blocks`, a union discriminated on `block_type` — a field
AG-UI's `TextInputContent` /
`ImageInputContent` / `BinaryInputContent` do not carry. Live backend
error:
```
pydantic_core._pydantic_core.ValidationError: 3 validation errors for ChatMessage
blocks.0
Unable to extract tag using discriminator 'block_type' [type=union_tag_not_found,
input_value=TextInputContent(type='te... image I just attached'), input_type=TextInputContent]
```
Fixing it needs a content-part ->
`TextBlock`/`ImageBlock`/`DocumentBlock` conversion, upstream or on our
side of `get_ag_ui_workflow_router`. This PR also corrects that module's
docstring, which asserted the
router *"normalizes them via the OpenAI `input_file` path"* — it does
not.
**`crewai-crews` — never implemented, ours.** `src/agent_server.py`
registers a dedicated AG-UI endpoint for
every other demo but **no `/multimodal` route** (the block ends at the
catch-all
`add_crewai_crew_fastapi_endpoint(app, LatestAiDevelopment(), "/")`),
and there is no
`src/agents/multimodal_agent.py`. So
`src/app/api/copilotkit-multimodal/route.ts` aliases the generic shared
crew, which has no vision handling and dies on a content-part message:
```
[CopilotKit] Error (agent_run_error_event): Error: thread=… run=…: CrewAI flow failed; see server logs
```
That route file's own header comment already concedes the gap ("A
dedicated per-demo crew with vision-tuned
agent prompts is tracked as follow-up work").
## Proof
Method: the **real** `GET /api/matrix` handler (`registerMatrixRoute`)
driven over the **real** production
PocketBase `status` collection (all 3082 rows, fetched verbatim from
`showcase-pocketbase-production.up.railway.app`) and the **real**
on-disk manifests (default `loadCells` =
`buildCatalogCells`, the single flattening authority). Same rows, same
fixed clock
(`now = max(observed_at) + 60s = 1784932566438`) for both runs — the
only variable is the manifest diff.
### BEFORE — real `GET /api/matrix`, real prod PocketBase rows,
manifests at `origin/main` (
|
||
|
Jordan Ritter
|
59f275eedc |
fix(showcase/mastra): restore shared-tools symlink, shrink erosion ratchet (#6161)
## What
`showcase/integrations/mastra/shared-tools` was a **real committed
directory** where a **symlink into the single source of truth** belongs.
This restores the symlink (`-> ../../shared/typescript/tools`) and
shrinks the erosion ratchet accordingly.
This is the iron-rule violation described in `showcase/AGENTS.md` → "The
single-source symlink mechanism": `*/shared-tools`, `*/tools`, and
`*/_shared` are meant to be symlinks into `showcase/shared/...`, so
edits to the shared source silently do NOT reach an integration whose
symlink has been clobbered by a real directory.
## Root cause
| | |
|---|---|
| Symlink originally added | `93d5815cdb` — *"refactor: add shared tools
symlink to mastra, update path alias"* (target
`../../shared/typescript/tools`) |
| Symlink clobbered | `534cd1efa7` — *"fix(showcase): D5 integration
fixes across 12 frameworks"* — deleted the symlink (`shared-tools \| 1
-`) and committed **14 real files** in its place |
That commit clobbered `shared-tools` for **three** TS integrations at
once: `mastra`, `langgraph-typescript`, and `claude-sdk-typescript` —
all three grandfathered in
`showcase/scripts/validate-shared-symlinks.baseline.json`.
mastra's Dockerfile already documented the symlink as the expected
state, so the tree had drifted from its own documented contract:
```
# shared-tools/ is a symlink to ../../shared/typescript/tools — resolved at
# build time by CI which copies the target into the build context.
```
## Divergence inventory — **NONE**
Diffed the real directory against `showcase/shared/typescript/tools`
exhaustively **before** changing anything. **Zero divergence**:
identical file set, identical git blob hashes, identical sha256s across
all 14 files.
| File | git blob (main) | classification |
|---|---|---|
| `__tests__/generate-a2ui.test.ts` | `b3c8223e` | identical copy |
| `__tests__/get-weather.test.ts` | `57dd5b94` | identical copy |
| `__tests__/query-data.test.ts` | `ef284b0b` | identical copy |
| `__tests__/sales-todos.test.ts` | `8c5c0d6e` | identical copy |
| `__tests__/schedule-meeting.test.ts` | `ab470b3b` | identical copy |
| `__tests__/search-flights.test.ts` | `f64b8782` | identical copy |
| `generate-a2ui.ts` | `7d76c00c` | identical copy |
| `get-weather.ts` | `0b444f7a` | identical copy |
| `index.ts` | `3b0e2a45` | identical copy |
| `query-data.ts` | `48a2cbdc` | identical copy |
| `sales-todos.ts` | `96b1874f` | identical copy |
| `schedule-meeting.ts` | `d5e626e8` | identical copy |
| `search-flights.ts` | `7a943023` | identical copy |
| `types.ts` | `f87f26d1` | identical copy |
Classification totals:
- **stale** (shared moved on, mastra behind): none
- **mastra-specific** (intentional or accidental local edit): **none**
- **additive** (file only in mastra's copy): none
- files present in one and not the other: **none**
```
$ git diff --no-index --stat integrations/mastra/shared-tools shared/typescript/tools
(no output — identical)
```
Because the divergence set is empty, restoring the symlink is a **pure
structural fix with zero content change** — nothing mastra-specific
existed in the copy, so nothing can be silently lost. The restored link
blob is `ddc634b6` — **the same git object** the pre-erosion symlink
had.
## AFTER — structural proof
```
$ ls -la showcase/integrations/mastra/shared-tools
lrwxr-xr-x shared-tools -> ../../shared/typescript/tools
$ git ls-files -s showcase/integrations/mastra/shared-tools
120000
|
||
|
Jordan Ritter
|
c2e9264dde | Merge branch 'main' into fix/ms-agent-python-multimodal-prompt | ||
|
Jordan Ritter
|
4a303f8bed | Merge branch 'main' into chore/multimodal-unsupported-llamaindex-crewai | ||
|
Jordan Ritter
|
2825b10ba1 | Merge branch 'main' into fix/mastra-shared-tools-symlink |