mirror of
https://github.com/CopilotKit/CopilotKit.git
synced 2026-09-14 16:26:20 +08:00
fix/python312-showcase-tests
12629 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jordan Ritter
|
9491b89340 |
fix(showcase): disjoint catchall userMessages + content-asserting probes (supersedes #5465)
Closed PR #5465 introduced cross-fixture leakage by stripping the toolName discriminator on shared {userMessage, context} keys: with aimock's alphabetical first-match-wins ordering, tool-rendering-custom-catchall.json sorts before tool-rendering-default-catchall.json, so default-catchall page requests were served the custom file's content. The probe was structurally blind because it asserted only DOM testids (copilot-tool-render + data-tool-name=get_weather) — both fixtures emit get_weather, so the testid signal passed regardless of which fixture won. Real LGP-gold pattern is disjoint userMessages between default and custom catchall fixtures (NOT shared keys discriminated by toolName). This PR ports the LGP-gold pattern to the other 17 integrations and adds page-text content assertions to both d5 catchall probes so this class of regression can't recur silently: - Probe prompts disjoint between default-catchall and custom-catchall - Fixture userMessages updated to match the new disjoint prompts - Page-text content assertions in both d5 catchall probes (default negatively asserts the custom-catchall leak phrase; custom positively asserts it) Local gold-standard red-green proof captured: - Step A: live staging Playwright baseline (RED-OF-RECORD) - Step B: local control-plane on origin/main reproduces structural fragility (probes pass at testid level, custom fixture wins on default's userMessage path) - Step C: local control-plane on this branch — both catchall probes GREEN with the new content-asserting assertions tool-rendering-default-catchall: pass=true (5443ms) tool-rendering-custom-catchall: pass=true (8956ms) cross-tool signature passed LGP (langgraph-python) was already disjoint; it remains untouched. |
||
|
Maxim
|
f9bcf11295 |
fix(examples): sync langgraph-js agent lockfile to @langchain/core 1.1.49
Keeps package-lock.json in sync with the core bump pulled in from main (#5489) so the agent lockfile is not stale after the rebase. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Maxim
|
a144323d35 |
fix(examples): regenerate integration lockfiles for npm ci
The previous lockfiles were written with `npm install --package-lock-only`, which left out optional transitive deps (e.g. @emnapi/wasi-threads). The Docker smoke-starter build runs strict `npm ci`, which then failed with "package.json and package-lock.json not in sync" (crewai-crews, 9s fail). Regenerated each lockfile with a full `npm install --ignore-scripts` so the tree is complete. Verified `npm ci` now succeeds (crewai-crews, mastra) and all resolve @ag-ui/client 0.0.57. |
||
|
Maxim
|
cef4fab9d7 |
fix(examples): bump remaining integrations to 1.60.1 + @ag-ui 0.0.57
Follow-up to the @copilotkit 1.60.1 bump so the integration-demo parity check passes. The north-star (langgraph-python) is now on 1.60.1, and the _parity manifest requires every examples/integrations/* demo to track the same @copilotkit version — so all integrations must move together. - @copilotkit/* 1.60.0 -> 1.60.1 across the remaining integrations. - Starter-fleet apps force a single @ag-ui tree via `overrides` (@ag-ui/client|core|encoder|proto). Bump that single-tree pin 0.0.53/0.0.55 -> 0.0.57 so they actually carry the HttpAgent fetch fix (Illegal invocation) instead of a 1.60.1<->0.0.55 skew. Adapter packages (@ag-ui/crewai, @ag-ui/mastra, @ag-ui/llamaindex, @ag-ui/a2a, middlewares) are left untouched. - Lockfiles regenerated; all resolve @ag-ui/client 0.0.57. Per-integration smoke-starter CI validates each starter against 0.0.57. |
||
|
Maxim
|
0363017503 |
fix(examples): bump @copilotkit to 1.60.1 for @ag-ui/client fetch fix
The chat never responds and the browser console shows "TypeError: Failed to execute 'fetch' on 'Window': Illegal invocation" on every agent run (onRunFailed, agentId: default). Root cause is in @ag-ui/client@0.0.56's HttpAgent: it stores the global fetch unbound (`this.fetch = config.fetch ?? fetch`) and later invokes it as `this.fetch(...)`, so native fetch runs with the agent instance as its receiver instead of `window`. Native fetch is brand-checked and throws "Illegal invocation". It surfaces in both dev and prod. Fixed upstream in @ag-ui/client@0.0.57 (`config.fetch ?? ((url, init) => fetch(url, init))`), which shipped in @copilotkit/* 1.60.1. Bumping these examples 1.60.0 -> 1.60.1 pulls 0.0.57 transitively; lockfiles regenerated. Scope: only the examples that take @ag-ui transitively from react-core are bumped here. Other integration examples force-pin @ag-ui via `overrides` to 0.0.53/0.0.55, where 1.60.1 would create a version skew; those need a per-integration bump and are intentionally left out of this change. |
||
|
Maxim
|
e9f1f21ddb |
fix(examples): bump langgraph-js agent @langchain/core to 1.1.49 (v6 uuid export) (#5489)
## Summary The `smoke-starter (langgraph-js)` CI job is failing on `main` (and on every open PR, e.g. #5457). The agent container crashes on startup: ``` SyntaxError: The requested module '@langchain/core/utils/uuid' does not provide an export named 'v6' ``` ## Root cause - `examples/integrations/langgraph-js/docker/Dockerfile.agent` copies only `package.json` (not the lockfile) and runs `npm install`, so transitive deps resolve fresh at build time. - `@langchain/langgraph@1.3.0` → `@langchain/langgraph-checkpoint@^1.0.2`. - Checkpoint `1.1.1` (published 2026-06-12) started importing `v6` and raised its peer to `@langchain/core@^1.1.48`. - The agent pinned `@langchain/core@1.1.44`, which predates the `v6` export → import crash. Verified that `@langchain/core@1.1.44` does not export `v6` from `utils/uuid` while `1.1.49` does, and a clean `npm install` with core `1.1.49` dedupes to checkpoint `1.1.1` + core `1.1.49` and imports `v6` successfully. ## Change Bump `@langchain/core` `1.1.44` → `1.1.49` in the langgraph-js agent. The npm `package-lock.json` is intentionally left untouched — no CI job consumes it (the Docker agent build runs `npm install` against `package.json`, per the Dockerfile's own "Mirrors the user experience" comment). ## Test plan - [ ] `smoke-starter (langgraph-js)` passes on this PR |
||
|
Jordan Ritter
|
f316d055fb |
refactor(showcase): rename internal Wired counters to BeAgent
Renames the internal display-counter variables that fed the stats-bar / coverage-bar "Wired" labels to match the new "BE (Agent)" taxonomy: totalWired → totalBeAgent in cells-view.tsx and parity-view.tsx, pctWired → pctBeAgent in coverage-bar.tsx. These are local render-time aggregates, not part of any persisted contract. The status enum literal "wired" (the .filter((c) => c.status === "wired") guard, the coverage-segment-wired test ID, and the wiredByCategory Map's local name) is intentionally preserved — those all key directly off the persisted catalog Status enum and changing them would expand scope into the catalog contract. |
||
|
Jordan Ritter
|
bfebac6565 |
refactor(showcase): relabel catalog status Wired display → BE (Agent)
Unifies the catalog integration status display label with the same
"BE (Agent)" taxonomy as the live-probe agent dimension. The
underlying Status enum literal ("wired"), the catalog.metadata.wired
data field, and the filter chip id ("wired" → cell-matrix.tsx:311
status === "wired" filter key) are all PRESERVED — they are
persisted catalog data + filter state contracts that must not move.
Only the user-facing display label on stats-bar, adaptive-stats-bar,
and the filter chip flips.
This collapses the two distinct dashboard "Wired" surfaces (the L1
live-probe dimension and the per-cell build-state count) under one
unified label, matching the user-confirmed Path B.
|
||
|
Jordan Ritter
|
ef40bd7cfb |
refactor(showcase): relabel live-probe agent dimension Wired → BE (Agent)
Unifies the L1 "agent" live-probe display label with the taxonomy convention established by #5473 (UI (Frontend), E2E, CV, D6 — layer descriptor in parentheses). The dimension name stays "agent" in code (PocketBase row keys agent:<slug>, LiveDimension union, keyFor lookups are all unchanged stable contracts) — only the visible label changes. Updates the level-strip L1 badge label and the packages-section L1-L4 header legend (W → B, "Wired" → "BE (Agent)") so the level-strip's ToneChip first-letter abbreviation matches the legend key. Test assertions covering the rendered letter, the legend text, and the degraded-tone test's local variable follow suit. |
||
|
Varun Nuthalapati
|
5d67341881 | docs(react-ui): clarify all modalities in attachments prop examples | ||
|
Maxim
|
c82e3b40e2 |
fix(examples): bump langgraph-js agent @langchain/core to 1.1.49 for v6 uuid export
The langgraph-js smoke-starter job crashes on agent startup with "'@langchain/core/utils/uuid' does not provide an export named 'v6'". @langchain/langgraph@1.3.0 depends on @langchain/langgraph-checkpoint@^1.0.2. Checkpoint 1.1.1 (published 2026-06-12) imports v6 and raised its peer to @langchain/core@^1.1.48. The agent pinned core 1.1.44, which predates the v6 export. The Docker agent build runs `npm install` against package.json (no lockfile), so it floats to checkpoint 1.1.1 against the too-old core. Bumping core to 1.1.49 restores the v6 export and satisfies the peer range. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Alem Tuzlak
|
d305dc6ef2 |
fix(examples): regenerate standalone slack lockfile for @copilotkit/bot* 0.0.2 (#5481)
## What Regenerates the **standalone** `examples/slack/pnpm-lock.yaml` so its `@copilotkit/bot*` deps match `package.json` (`~0.0.2`). ## Why (the actual Railway failure) `examples/slack` commits its own standalone `pnpm-lock.yaml`, which Railway frozen-installs when building with `rootDirectory=/examples/slack`. PR #5478 fixed the **root** workspace lockfile but not this standalone one, so it still recorded `@copilotkit/bot*` at `~0.0.1` while `package.json` requires `~0.0.2`: ``` ERR_PNPM_OUTDATED_LOCKFILE Cannot install with "frozen-lockfile" because pnpm-lock.yaml is not up to date with <ROOT>/package.json specifiers in the lockfile ({"@copilotkit/bot":"~0.0.1",...}) don't match specs in package.json ({"@copilotkit/bot":"~0.0.2",...}) ``` ## How Regenerated with the repo's pinned pnpm (10.33.4, via corepack) in standalone mode — matching how the file was originally produced and how Railway builds it: ``` pnpm -C examples/slack install --ignore-workspace --lockfile-only ``` Result: importer specifiers are now `~0.0.2` resolving to the published `0.0.2`; the `@ai-sdk/mcp: 1.0.21` override is preserved. > Note: regenerating with pnpm 11 dropped the `@ai-sdk/mcp` override (pnpm 11 ignores `package.json` `pnpm.overrides` in `--ignore-workspace` mode), so this was generated with the pinned 10.33.4 specifically. ## Verified `CI=true pnpm -C examples/slack install --ignore-workspace --frozen-lockfile` → **"Lockfile is up to date"** (exit 0). Full pre-commit suite + commitlint green. Only `examples/slack/pnpm-lock.yaml` changed (+103/−32). |
||
|
Alem Tuzlak
|
7c884b931b |
fix(examples): regenerate standalone slack lockfile for @copilotkit/bot* 0.0.2
examples/slack ships a standalone pnpm-lock.yaml (used by Railway when building with rootDirectory=/examples/slack). PR #5478 fixed the root workspace lockfile but not this one, so it still pinned @copilotkit/bot* at ~0.0.1 while package.json now requires ~0.0.2 — pnpm install --frozen-lockfile failed with ERR_PNPM_OUTDATED_LOCKFILE on deploy. Regenerate it with the repo's pinned pnpm (10.33.4) via pnpm -C examples/slack install --ignore-workspace --lockfile-only: importer specifiers are now ~0.0.2 resolving to the published 0.0.2, the @ai-sdk/mcp override is preserved. Verified: CI=true frozen install reports "Lockfile is up to date" (exit 0). |
||
|
Alem Tuzlak
|
eea1c861c5 |
docs(examples): note monorepo deploy watch-paths for slack example (#5479)
## What A small docs touch to `examples/slack/README.md` that (a) documents the monorepo-deploy watch-path gotcha and (b) **touches a watched path so Railway redeploys the slack services off latest `main`**. ## Why PR #5478 fixed the example's lockfile so a standalone deploy resolves the published `@copilotkit/bot*` `0.0.2` — but that fix only changed **repo-root** files (`pnpm-lock.yaml`, `package.json`, `.npmrc`). The `kite-slack-bot` / `runtime` Railway services watch `examples/slack/**`, so they reported "watched paths not modified" and never picked up the fix. This commit changes a file under `examples/slack/**`, so merging it triggers a rebuild of latest `main` (which carries #5478's fix). The added note also tells future deployers to include the repo-root `pnpm-lock.yaml` / `package.json` in their Railway watch paths so this doesn't recur. Docs-only; no code change. |
||
|
Alem Tuzlak
|
d6d487ba47 |
docs(examples): note monorepo deploy watch-paths for slack example
Document that the slack example consumes the published @copilotkit/bot* packages and that deploys (e.g. Railway) must include the repo-root pnpm-lock.yaml / package.json in their watch paths, so a dependency bump triggers a rebuild and a frozen install doesn't fail on an out-of-date lockfile. Touches examples/slack/** to trigger the Railway redeploy of the current main (which now carries the deployable-lockfile fix). |
||
|
Alem Tuzlak
|
a7688dd35e |
fix(examples): make slack example lockfile deployable (drop workspace override) (#5478)
## What Fixes the Railway deploy of `examples/slack` failing with `ERR_PNPM_OUTDATED_LOCKFILE`. ## Why it broke PR #5476 bumped the example's `@copilotkit/bot*` deps to `~0.0.2` but the lockfile didn't change — because the root `pnpm.overrides` (`@copilotkit/bot* → workspace:*`, added for local dev) pinned those deps to **workspace links** for every importer. So the committed lockfile recorded the example's bot deps as `~0.0.1` / `link:../../packages/bot`, which: - frozen-install validates fine **inside** the monorepo (the override masks it), but - a **standalone Railway deploy** frozen-installs only `examples/slack`, sees `package.json ~0.0.2` vs lockfile `~0.0.1`, and can't resolve `link:` paths → crash. ## The fix Now that `bot`/`bot-slack`/`bot-ui` are published at `0.0.2`: - **Drop the three `@copilotkit/bot*` overrides** from root `package.json`. Workspace packages still link each other via `workspace:~`; only the **example** switches to consuming the published versions — the correct model for a deployable demo. - **Regenerate `pnpm-lock.yaml`** — the example's importer now records `specifier: ~0.0.2` resolving to the **registry** `0.0.2` (not a `link:`), so a standalone frozen install resolves cleanly. - **Add `@copilotkit/bot*` to `minimum-release-age-exclude`** in `.npmrc` (matching the existing `@ag-ui/*` entries) so the freshly-published `0.0.2` resolves past the 24h supply-chain gate. ## Verified - `pnpm install --frozen-lockfile` → "Lockfile is up to date"; the registry `0.0.2` packages download cleanly. - Example importer: `specifier: ~0.0.2`, `version: 0.0.2(...)` (registry, no `link:`). - Full pre-commit suite + commitlint green. ## Trade-off Local monorepo dev of the example now consumes the **published** `0.0.2` rather than live worktree source. To iterate on the bot packages against the example locally, temporarily set the example's bot deps to `workspace:*` (or re-add the overrides) — don't commit that. |
||
|
Alem Tuzlak
|
761ae8caec |
fix(examples): make slack example lockfile deployable (drop workspace override)
The root pnpm.overrides pinned @copilotkit/bot* to workspace:* for every importer, so the committed lockfile resolved the slack example's bot deps to workspace links. A standalone deploy (Railway) frozen-installs only the example and can't resolve those, failing with ERR_PNPM_OUTDATED_LOCKFILE (lockfile specifiers ~0.0.1 vs package.json ~0.0.2, and link: refs that don't exist outside the monorepo). Now that bot/bot-slack/bot-ui are published at 0.0.2, drop the overrides so the example resolves the published ~0.0.2 from the registry, and regenerate the lockfile (importer specifiers now ~0.0.2, versions resolve to registry 0.0.2 — deployable). Add @copilotkit/bot* to minimum-release-age-exclude (matching the @ag-ui/* entries) so the freshly published 0.0.2 resolves past the 24h gate. Workspace packages still link each other via workspace:~; only the example switches to published versions (the correct model for a deployable demo). |
||
|
Alem Tuzlak
|
5f1d771cbe |
chore(examples): bump slack example to @copilotkit/bot* 0.0.2 (#5476)
## What Bumps the `examples/slack` dependency ranges for the bot packages from `~0.0.1` to `~0.0.2`, now that `@copilotkit/bot`, `@copilotkit/bot-slack`, and `@copilotkit/bot-ui` are all published at **0.0.2** (the agent-native assistant pane + native streaming release). ```diff - "@copilotkit/bot": "~0.0.1", - "@copilotkit/bot-slack": "~0.0.1", - "@copilotkit/bot-ui": "~0.0.1", + "@copilotkit/bot": "~0.0.2", + "@copilotkit/bot-slack": "~0.0.2", + "@copilotkit/bot-ui": "~0.0.2", ``` ## Why A standalone/deployed install of the example (outside the monorepo) resolves these from npm. At `~0.0.1` it would pull `bot-slack@0.0.2` but could still resolve `bot`/`bot-ui@0.0.1`, which lack the engine surface the pane code calls at runtime (`sink.onThreadStarted`, `thread.setSuggestedPrompts/setTitle`, the new capability flags) — so the assistant pane would crash. Pinning all three to `~0.0.2` keeps the trio aligned. ## Notes - **Lockfile unchanged**: local monorepo installs already resolve these to the workspace copies via the root `pnpm.overrides` (`@copilotkit/bot* → workspace:*`), so this range bump only affects deployed/standalone installs. - No code changes — `package.json` only. |
||
|
Alem Tuzlak
|
ba159ce8e1 |
chore(examples): bump slack example to @copilotkit/bot* 0.0.2
@copilotkit/bot, @copilotkit/bot-slack, and @copilotkit/bot-ui are published at 0.0.2 (agent-native assistant pane + native streaming). Point the slack example's dependency ranges at ~0.0.2 so a deployed/standalone install pulls the new packages. Local monorepo installs already use the workspace copies via root pnpm.overrides, so the lockfile is unchanged. |
||
|
Alem Tuzlak
|
7ed4a44447 |
chore: release bot v0.0.2 (#5475)
## Release bot v0.0.2 **Scope:** `bot` | **Bump:** `patch` --- ### How this release process works 1. **This PR was created automatically** by the "release / create-pr" workflow. It bumped the `bot` packages to `0.0.2` and generated AI-enhanced release notes. 2. **CI runs on this PR** — the full test suite (unit tests, lint, type checks, build) must pass before merging. This is the review gate. 3. **Review the release notes** in `release-notes.md` in this PR. If a Notion draft was created, you can edit the release notes there before merging. 4. **When this PR is merged**, the `release / publish` workflow automatically: - Builds all packages - Publishes the `bot` packages to npm at version `0.0.2` - Creates git tag `bot/v0.0.2` - Creates a GitHub Release with the final release notes ### Before merging - [ ] CI is green (tests, lint, types, build) - [ ] Version bumps look correct - [ ] Release notes are accurate (edit in Notion if a draft was created) --- > **Do not merge until CI is fully green.** The full test suite runs automatically on this PR.bot/v0.0.2 |
||
|
AlemTuzlak
|
035e7cf9c9 | chore: release bot v0.0.2 | ||
|
Alem Tuzlak
|
72203fbcba |
chore: release bot-slack v0.0.2 (#5466)
## Release bot-slack v0.0.2 **Scope:** `bot-slack` | **Bump:** `patch` --- ### How this release process works 1. **This PR was created automatically** by the "release / create-pr" workflow. It bumped the `bot-slack` packages to `0.0.2` and generated AI-enhanced release notes. 2. **CI runs on this PR** — the full test suite (unit tests, lint, type checks, build) must pass before merging. This is the review gate. 3. **Review the release notes** in `release-notes.md` in this PR. If a Notion draft was created, you can edit the release notes there before merging. 4. **When this PR is merged**, the `release / publish` workflow automatically: - Builds all packages - Publishes the `bot-slack` packages to npm at version `0.0.2` - Creates git tag `bot-slack/v0.0.2` - Creates a GitHub Release with the final release notes ### Before merging - [ ] CI is green (tests, lint, types, build) - [ ] Version bumps look correct - [ ] Release notes are accurate (edit in Notion if a draft was created) --- > **Do not merge until CI is fully green.** The full test suite runs automatically on this PR.bot-slack/v0.0.2 |
||
|
Jordan Ritter
|
e941049d3c |
fix(harness): cascade fallback to whole-bubble-minus-toolbar for tool-only responses (Class B) (#5474)
## Summary Adds a **Class B cascade fallback** in `readCascadeStateLast` so the harness settle gate can resolve text for **single-bubble tool-only responses** (recharts SVG, gen-UI cards, A2UI render-only output). These bubbles have ALL scoped text selectors empty but carry substantial rendered content in non-cascade children. The current cascade-pollution guard correctly returns `null` for arbitrary-index reads (`readCascadeState` / `findAssistantBubbleAt`) — but for the LAST bubble, by `RUN_FINISHED` the content is mounted and stable, so we can safely fall back to `bubble.textContent` MINUS the assistant-toolbar's textContent (the suffix the toolbar contributes). This is the residual fix on top of PR #5472 (Class A: multi-bubble last-bubble-has-prose) and #5473 (probe taxonomy cleanup). The ~200-red plateau on PocketBase after #5472 landed is consistent with Class B affecting the long tail of tool-render-only demos (ms-agent-python:beautiful-chat-*, plus other tool-render-only Class-B-shaped demos). ## Class B failure mode (live-confirmed) Probed against `https://showcase-ms-agent-python-staging.up.railway.app/demos/beautiful-chat` with the prompt _"Show me a bar chart of monthly sales for Q1 2026."_: - `count = 1` (single canonical bubble) - `bubble.querySelector('[data-message-content]')` -> empty string - `bubble.querySelector('.cpk:prose')` -> empty string - `bubble.querySelector('.prose')` -> empty string - `bubble.textContent` -> **351 chars** of real content (chart title, axis labels, animation styles, query metadata) - Current `readCascadeStateLast` returns `text=null` -> settle gate spins on `text-unstable` until timeout (RED). ## Red-green proof (verbatim from /tmp/cr/class-b-red-green.mjs) The probe runs TWO in-browser readers against the same DOM snapshot — the OLD (current production) and the NEW (with fallback). Run live against staging, **before any code changes** in this PR: ``` [probe] navigating to https://showcase-ms-agent-python-staging.up.railway.app/demos/beautiful-chat [probe] locating chat input [probe] sending prompt: Show me a bar chart of monthly sales for Q1 2026. [probe] waiting 20000ms for response to render [probe] result: { "count": 1, "oldText": null, "newText": "query_dataquery:\"monthly sales for Q1 2026\"\n @keyframes barSlideIn {\n from { transform: translateY(40px); opacity: 0; }\n 20% { opacity: 1; }\n to { transform: translateY(0); opacity: 1; }\n }\n Monthly Sales for Q1 2026Breakdown of income generated each month in Q1 2026.JanuaryFebruaryMarch0255075100January", "oldLen": null, "newLen": 351, "newPreview": "query_dataquery:\"monthly sales for Q1 2026\"\n @keyframes barSlideIn {\n from { transform: translateY(40px); opacity: 0; }\n 20% { opacity: 1; }\n to { transform: translat" } [probe] verdict: { "red": true, "green": true, "count": 1 } [probe] RED-GREEN confirmed: old=null, new=non-empty. ``` - **RED**: `oldText = null`, `oldLen = null` — matches current production behavior; settle gate cannot resolve text. - **GREEN**: `newText` is 351 chars of stable rendered content; settle gate can lock in `text-stable`. ## The fix In `showcase/harness/src/probes/helpers/assistant-message-count.ts`, inside `readCascadeStateLast`'s per-tier loop — AFTER the existing scoped-selector cascade exhausts without a non-empty match and BEFORE returning `{count, text: null}`: 1. Read `bubble.textContent` (whole bubble). 2. Read `bubble.querySelector('[data-testid="copilot-assistant-toolbar"]').textContent` if present. 3. If the toolbar's text is a **trailing suffix** of the whole-bubble text (which it canonically is — toolbar is a leaf sibling of the prose div), strip it. 4. If what remains is non-empty after trim -> return `{count, text: <trimmed>}`. 5. Otherwise (no toolbar, no content, or only-toolbar text) -> return `{count, text: null}` to keep polling. The same fallback is NOT applied to `findAssistantBubbleAt` / `readCascadeState` because those address arbitrary indices and intermediate bubbles in multi-step responses can transiently carry empty scoped text while the NEXT bubble is mounting. Reading whole-bubble at intermediate indices would re-introduce the cross-bubble text flap PR #5462 was designed to prevent. The LAST bubble in a multi-step turn is the agent's terminal output, mounted by `RUN_FINISHED`. ## Why the fallback is safe (toolbar-suffix invariant) In the canonical CopilotKit assistant bubble (`CopilotChatAssistantMessage.tsx`), the toolbar is the **last child** of the bubble, mounted as a leaf sibling of the prose/markdown wrapper. Its `textContent` is therefore the trailing slice of `bubble.textContent`. Stripping it by suffix-match yields the message content. When the suffix-match fails (defensive — e.g. the DOM shape changes), we return `bubble.textContent` unchanged rather than mangle it. When the toolbar isn't present (headless / custom-composer), we return whole text. When even whole-text is empty, we return `null` (don't lock the settle gate on a placeholder). ## Tests - `assistant-message-count.test.ts`: 9 new test cases under a new `describe("readCascadeStateLast")` block. Pins: - non-empty scoped text path unchanged (no fallback when scoped works) - Class B fallback: scoped empty + whole-bubble non-empty + toolbar suffix -> strip suffix, return content - Class B fallback works without a toolbar (returns whole text) - defensive: toolbar text NOT a suffix -> return whole text unchanged - everything empty -> return `text:null` (settle gate keeps polling) - only-toolbar text -> return `text:null` after strip - multi-bubble: reads the LAST bubble, not the first - no tier matches -> `{count:0, text:null}` - `evaluate()` throws -> swallowed -> `{count:0, text:null}` - Full harness suite: **2769 / 2769 passing across 129 files** (`pnpm exec vitest run` in `showcase/harness`). - TypeScript: `pnpm exec tsc --noEmit` clean. - Lint: `oxlint` 0 warnings, 0 errors. - Format: `oxfmt --check` clean. ## Out-of-scope (untouched) - `conversation-runner.ts` (the settle-gate logic stays — the cascade does the right thing now). - `sse-interceptor.ts`, `init-scripts.ts`, `d6-all-pills.ts`. - `findAssistantBubbleAt` / `readCascadeState` (intentionally retain the pollution guard — they address arbitrary indices). - `resolveBubbleTextFromSelectors` (pure sibling of `findAssistantBubbleAt`, intentionally retains pollution-guard semantics). ## Verification plan after merge 1. Watch CI green on the PR. 2. Promote `showcase-ms-agent-python` and other tool-render-only services. 3. Observe PocketBase D6 red counts: expect the ~200-red plateau to drop further toward baseline. 4. If any demo regresses, revert `readCascadeStateLast` alone. ## Files changed - `showcase/harness/src/probes/helpers/assistant-message-count.ts` (production fix) - `showcase/harness/src/probes/helpers/assistant-message-count.test.ts` (9 new tests) |
||
|
Jordan Ritter
|
72520de43d | fix(harness): cascade last-bubble whole-bubble-minus-toolbar fallback for tool-only responses | ||
|
Jordan Ritter
|
f7779f90bf |
refactor(showcase): probe taxonomy cleanup — drop Smoke, rename E2E (Demo) → UI (Frontend) (#5473)
## Summary
The smoke probe was the same HTTP contract as `/health` on the same
service (200-OK JSON body), so every tick paid two HTTP calls for the
same liveness signal. This PR:
- Drops the `/smoke` GET probe and the `smoke:<slug>` primary
`ProbeResult`. The driver now emits `health:<slug>` as the primary and
`agent:<slug>` via writer side-emit (half the per-tick cost).
- Renames the dashboard's D3 / e2e row label from `E2E (Demo)` to `UI
(Frontend)` (and the short cell badge from `E2E` to `UI`) — the row was
already about "the demo page renders in a browser", not E2E in the
traditional sense.
- Drops the `Smoke` row from the cell drilldown (CellState.smoke field
is retained for back-compat so historical rows still parse).
- Adds an explicit `Health` row to the legend (was implicit).
The driver's registry `kind` stays `"smoke"` so existing YAML configs +
orchestrator family wiring keep routing to this driver — the emission
key (`health:<slug>`) is the taxonomy contract that matters. The
underlying probe key (`e2e:<slug>/<feature>`) is preserved on PocketBase
so historical rows render correctly during the rename window.
## RED proof (BEFORE)
`smoke:<slug>` primary emission in `liveness.ts` (origin/main):
```
18: * 1. `smoke:<slug>` — the RETURN VALUE of `run()`. The invoker runs
56: * `smoke:<slug>`/`health:<slug>`/`agent:<slug>` triple the dashboard
241: input.mode === "discovery" ? `smoke:${slug}` : input.key;
```
`E2E (Demo)` / `Smoke` strings in dashboard render code (origin/main):
```
cell-drilldown.tsx:38: * Trip)" (chat+tools round-trip), D3/e2e = "E2E (Demo)" (the demo page loads
cell-drilldown.tsx:49: { key: "e2e", label: "E2E (Demo)" },
cell-drilldown.tsx:52: { key: "smoke", label: "Smoke" },
cell-pieces.tsx:382: name="E2E"
unified-cell.tsx:231: <TestBadge name="E2E" level={model.d3} />
adaptive-legend.tsx:59: E2E (Demo): demo page loads and round-trips in a browser
```
## GREEN proof (AFTER)
Primary emission rewritten — driver returns `health:<slug>` and no
`smoke:<slug>` ProbeResult is ever emitted:
```
33: * to `/smoke` and emitted a `smoke:<slug>` primary result, but that probe
230: // `kind: "smoke"` is the registry/family identifier the orchestrator
236: kind: "smoke", // registry-key back-compat; no smoke ProbeResult is emitted
367: if (input.mode === "discovery") return `health:${slug}`;
368: if (input.key.startsWith("smoke:")) {
369: return `health:${input.key.slice("smoke:".length)}`;
```
New labels in dashboard:
```
cell-drilldown.tsx:60: { key: "e2e", label: "UI (Frontend)" },
cell-pieces.tsx:382: name="UI"
unified-cell.tsx:231: <TestBadge name="UI" level={model.d3} />
adaptive-legend.tsx:65: UI (Frontend): demo page renders in browser (Playwright)
```
Confirmation that the old labels are gone from render code:
```
$ grep -rn 'E2E (Demo)\|"Smoke"' cell-drilldown.tsx cell-pieces.tsx unified-cell.tsx adaptive-legend.tsx
(no matches — labels removed)
```
## Test results
- `pnpm exec vitest run` in `showcase/shell-dashboard`: **63 files, 1089
tests passed, 1 skipped**
- `pnpm exec vitest run` in `showcase/harness`: **129 files, 2760 tests
passed**
- `pnpm exec tsc --noEmit` in both: clean
Liveness driver tests include a regression guard (`never emits a
smoke:<slug> key`) so the smoke contract cannot be re-introduced
silently.
## Test plan
- [x] Dashboard vitest green (1089/1089)
- [x] Harness vitest green (2760/2760)
- [x] tsc --noEmit green in both packages
- [x] Visual cross-check: drilldown renders 6 rows (no Smoke), e2e row
labelled `UI (Frontend)`; cell strip shows `UI` badge in place of `E2E`;
legend shows explicit `Health` row + `UI (Frontend)` D3 line
|
||
|
Jordan Ritter
|
53f021afe8 |
refactor(showcase): probe taxonomy cleanup — drop Smoke, E2E (Demo) → UI (Frontend)
The smoke probe was the same HTTP contract as /health on the same service (200-OK JSON body), so every tick paid two HTTP calls for the same liveness signal. Drop the /smoke GET + the smoke:<slug> primary ProbeResult; the driver now emits health:<slug> as the primary and agent:<slug> via writer side-emit (half the per-tick cost). The driver's registry kind stays "smoke" so existing YAML configs and orchestrator family wiring keep routing to this driver — the emission key (health:<slug>) is the taxonomy contract that matters. Dashboard: - Drilldown: D3/e2e row labelled "UI (Frontend)" (was "E2E (Demo)"); Smoke row dropped (CellState.smoke field retained for back-compat). - Cell badges: short label "UI" (was "E2E") in cell-pieces + unified-cell. - Legend: D3 = "UI (Frontend): demo page renders in browser (Playwright)" plus an explicit Health row at the top. Tests updated to assert new labels: - cell-drilldown.test.tsx: 6 dimensions (no Smoke); UI (Frontend) label. - cell-drilldown.lazy-signal.test.tsx: drilldown-badge-ui--frontend- testid. - cell-pieces.test.tsx + .signal-degrade.test.tsx: badge name "UI". - unified-cell.test.tsx: mock-badge-UI. - overlay-selector-integration.test.tsx: "UI" in place of "E2E". - dashboard-color-matrix.test.tsx: badge: "UI" case names. - liveness.test.ts: two-call contract (health + agent), regression guard asserting no smoke:<slug> ever emitted. Underlying probe key (e2e:<slug>/<feature>) preserved on PocketBase so historical rows render correctly during the rename window. |
||
|
Jordan Ritter
|
665dd5aae6 |
fix(harness): waitForTurnComplete reads LAST bubble (multi-step agent regression from #5462) (#5472)
## Summary **URGENT hotfix.** Staging is widely red for D5/D6 multi-step demos. Root cause confirmed via live Playwright DOM inspection on staging (langgraph-typescript : beautiful-chat : pie chart prompt). PR #5462's defect-2 fix uses `bubbleIndex = turnIndex - 1`, which assumes one assistant bubble per turn. Multi-step agents (LangGraph, Mastra, CrewAI, llama-index, ag2, etc.) emit 2-3+ bubbles per turn — tool-call bubble + tool-render bubble + final-text bubble. The strict-index read lands on an intermediate **tool-call** bubble whose scoped-text selectors (`.cpk:prose`, `[data-message-content]`, `.prose`, `p`) are **empty** (tool-call content lives in a sibling card div outside the scoped cascade). The cascade-pollution guard correctly returns `null` → the text-stable conjunct never converges → `reason=text-unstable` timeout across virtually every multi-step demo (beautiful-chat-*, gen-ui-*, shared-state-*, agentic-chat on most backends). ## Live RED-GREEN proof (staging) Standalone Playwright probe against `showcase-langgraph-typescript-staging.up.railway.app/demos/beautiful-chat`, prompt = `"pie chart of revenue distribution by category from the sample sales data"`, captured BOTH the OLD-logic read (`bubbleIndex = turnIndex - 1 = 0`) AND the NEW-logic read (`bubbleIndex = count - 1 = 2`) in ONE atomic page evaluate after the turn settled: ``` TOTAL BUBBLE COUNT FOR TURN 1: 3 (true multi-step — tool-call + tool-render + final-text) FINAL TIER: [data-testid="copilot-assistant-message"] OLD LOGIC (bubbleIndex = turnIndex - 1 = 0): count = 3 text = null reason = all scoped selectors empty ← would time out: reason=text-unstable NEW LOGIC (bubbleIndex = count - 1 = 2): count = 3 text = "Here is the pie chart showing the revenue distribution by category from the sample sales data." textLen = 94 ← gate settles cleanly ``` RED ≠ GREEN; new logic returns 94 chars of stable final-text where old logic returns `null`. The bug and the fix are proven on real staging DOM. ### Live DOM evidence LangGraph TS beautiful-chat, prompt = "pie chart of revenue distribution by category…": ``` count = 3 bubbles in [data-testid="copilot-assistant-message"] bubble[0]: tool-call "query_data" — `.cpk:prose` exists but EMPTY (content in sibling <div class="my-1.5">) bubble[1]: pie-chart card — `.cpk:prose` exists but EMPTY (content in sibling styled card div) bubble[2]: final text "Here is the pie chart..." — `.cpk:prose` has 94 chars + toolbar mounted ``` The old pre-#5462 runner read `list[last]` (always the last bubble — always bubble[2] above). The PR moved to `turnIndex - 1`, which is bubble[0] (the tool-call) for any multi-step turn → broken. ## Fix Read the LAST bubble in the matched cascade tier (`count - 1`) instead of a fixed strict index. Defect-2 protection (don't read a leftover bubble from a prior turn) is preserved via an explicit **pre-submit baselineCount sentinel snapshot** in the runner — the gate now requires `count > baselineCount` rather than `count > turnIndex - 1`. This correctly rejects stale prior-turn bubbles **without assuming 1 bubble = 1 turn**. ### Changes - `assistant-message-count.ts`: add `readCascadeStateLast(page)` — same atomic single-evaluate contract as `readCascadeState`, but resolves the bubble index internally as `count - 1`. - `conversation-runner.ts`: - Snapshot `baselineCount = countAssistantMessages(page)` BEFORE `sendTurnMessage()` each turn. - Pass through to `waitForTurnComplete` via new `baselineCount` option (defaults to `turnIndex - 1` to preserve unit-test fake compatibility). - `waitForTurnComplete` reads via `readCascadeStateLast` and gates on `domOk = count > baselineCount`. - Cold-start retry re-snapshots `baselineCount` after `page.reload()`. - Returned `bubbleIndex` in the success ctx = `count - 1` (matches the bubble whose text settled the gate). - `dom-missing` reason classifier updated to `countFinal <= baselineCount`. ## Why this doesn't regress defect-2 The original defect-2 (un-turn-scoped bubble selection) leaked a later turn's bubble into THIS turn's assertions because `readLastAssistantText` read `list[last]` GLOBALLY with no per-turn baseline. The new fix re-introduces the "read last" semantics but adds the **per-turn pre-submit baseline count snapshot** — the gate only advances once a NEW bubble has appeared (`count > baselineCount`), so we cannot read a leftover from a prior turn, and the "last" we return is the last NEW bubble (which for multi-step turns is the final-text bubble whose scoped text actually settles). ## Test plan - [x] **Live RED-GREEN staging probe** (see above) — confirms OLD logic = null and NEW logic = 94-char final-text on a real 3-bubble multi-step turn. - [x] All 72 `conversation-runner.test.ts` + `assistant-message-count.test.ts` unit tests pass (default `baselineCount = turnIndex - 1` keeps existing count-progression scripts working). - [x] Full harness vitest suite: 2758/2758 green across 129 test files. - [x] `tsc --noEmit` clean on `showcase/harness`. - [x] `oxfmt --check` clean on changed files. - [x] `oxlint` clean on changed files. - [ ] Staging D5/D6 sweep after merge confirms multi-step demos (beautiful-chat, gen-ui, shared-state, langgraph/mastra/crewai agentic-chat) recover from `text-unstable` red. ## Scope Strictly the two files above. No unrelated changes. No refactors. |
||
|
Jordan Ritter
|
88e77f1403 |
fix(showcase): allow harness to boot locally without SHARED_SECRET (unblock local verify) (#5471)
## Problem PR #5458 (`c81b361f1` — *fix(showcase/harness): register /webhooks/deploy on CP path + fail-loud on missing SHARED_SECRET*) added a fail-loud gate to `loadWebhookSecrets` in `showcase/harness/src/orchestrator.ts` that refuses harness boot in any deployable mode (`NODE_ENV !== "test"`) without `SHARED_SECRET` (or `SHARED_SECRET_PREV`) set, because `POST /webhooks/deploy` is only registered when `webhookSecrets.length > 0` (gate at `showcase/harness/src/http/server.ts:119`). The local `showcase/docker-compose.local.yml` harness services inherit `NODE_ENV=production` from the image and do not set `SHARED_SECRET` (and should not — that secret only matters for the Showcase: Verify Deploy webhook flow on Railway). As a result, the harness crash-loops on every local `bin/showcase test --d5/--d6` run: ``` FATAL-CONFIG: SHARED_SECRET (or SHARED_SECRET_PREV) is required — refusing to boot in any deployable mode … Set SHARED_SECRET (or SHARED_SECRET_PREV) in the env, or set NODE_ENV=test / HARNESS_ALLOW_NO_SECRET=1 for local dev. Current NODE_ENV=production. ``` …which blocks **all** local D5/D6 verify. ## Fix Add `HARNESS_ALLOW_NO_SECRET=1` (the documented local-dev escape hatch — explicitly named in the FATAL-CONFIG message itself, and the dedicated env var checked by `loadWebhookSecrets` at `orchestrator.ts:190`) to both harness services in `showcase/docker-compose.local.yml`: - `harness-control-plane` - `harness-pool-worker` Inline comments explain the rationale and pin the relevant source locations (the gate at `src/http/server.ts:119` and `loadWebhookSecrets` in `src/orchestrator.ts`). Diff: 17 lines added (comments + 2 env vars), no other files touched. ## Prod impact: NONE Railway sets `SHARED_SECRET` explicitly via env on every harness service, so `loadWebhookSecrets` sees a real secret, registers `POST /webhooks/deploy`, and **never reads `HARNESS_ALLOW_NO_SECRET`**. The escape hatch only takes effect when both secrets are absent. This change only affects the local docker-compose stack. ## Local verification Brought up the slot-1 isolated stack with the fix applied via `SHOWCASE_ISO_SLOT=3 ./bin/showcase test langgraph-typescript --d6 --verbose --isolate` and observed clean boot: ``` showcase-iso1-harness Up 12 seconds (healthy) showcase-iso1-langgraph-typescript Up 12 seconds (healthy) showcase-iso1-harness-pool-worker-1 Up 12 seconds (healthy) showcase-iso1-dashboard Up 13 seconds (healthy) showcase-iso1-aimock Up 18 seconds (healthy) showcase-iso1-pocketbase Up 18 seconds (healthy) ``` Harness boot logs confirm: - `fleet.role-selected` + `fleet.control-plane.started` + `showcase-harness.fleet.control-plane.boot` (port 8080) — control-plane up - `webhook auth disabled — neither SHARED_SECRET nor SHARED_SECRET_PREV is set` with `escapeHatch:true` at **warn** level — the expected log produced by `loadWebhookSecrets` when the escape hatch fires - `worker.registered` + `fleet.worker.boot` — worker self-registered - `worker.claimed jobId=… probeKey=d6:langgraph-typescript` — real D6 job picked up and run end-to-end NO `FATAL-CONFIG`, NO restart loop. (The D6 cell may still be RED on this branch — that is a pre-existing issue, unrelated to this infra fix. The point of this PR is "harness boots and runs a job at all locally", which it now does.) ## References - PR #5458 (`c81b361f1`): the fail-loud gate this PR re-enables an escape for - `showcase/harness/src/orchestrator.ts` `loadWebhookSecrets` (~line 180-220): the predicate that reads `HARNESS_ALLOW_NO_SECRET` - `showcase/harness/src/http/server.ts:119`: the route-registration gate |
||
|
Jordan Ritter
|
d145bb1708 |
fix(harness): waitForTurnComplete reads LAST bubble in matched tier
PR #5462's defect-2 fix assumed 1 bubble per turn (bubbleIndex = turnIndex - 1). Multi-step agents (LangGraph, Mastra, CrewAI) emit 2-3+ bubbles per turn — tool-call + tool-render + final-text — so the strict-index read lands on an intermediate tool-call bubble whose scoped-text selectors (`.cpk:prose`, `[data-message-content]`, `p`) are EMPTY (its content lives in a sibling card div outside the scoped cascade). The cascade-pollution guard returns null forever and the text-stable conjunct times out with `reason=text-unstable` across virtually every multi-step demo (beautiful-chat-*, gen-ui-*, shared-state-*, agentic-chat on LangGraph/Mastra/CrewAI/etc.). Live DOM evidence from staging (langgraph-typescript : beautiful-chat : pie chart prompt): count = 3 bubbles bubble[0]: tool-call "query_data" — `.cpk:prose` empty bubble[1]: pie chart card — `.cpk:prose` empty bubble[2]: final text "Here is the pie chart..." — 90 chars Fix: read the LAST bubble in the matched cascade tier (count - 1) instead of a fixed strict index. Defect-2 protection (don't read a leftover bubble from a prior turn) is preserved via an explicit pre-submit `baselineCount` sentinel snapshot in the runner; the gate now requires `count > baselineCount` rather than `count > turnIndex - 1`, which correctly rejects stale bubbles without assuming 1 bubble = 1 turn. Changes: - assistant-message-count.ts: add `readCascadeStateLast(page)` — same atomic single-evaluate contract as `readCascadeState` but resolves the bubble index internally as `count - 1`. - conversation-runner.ts: snapshot `baselineCount` before sendTurnMessage; pass through to `waitForTurnComplete` via new `baselineCount` option (defaults to `turnIndex - 1` for unit- test fakes); `waitForTurnComplete` reads via `readCascadeStateLast` and gates on `count > baselineCount`. Cold-start retry re-snapshots `baselineCount` after page.reload. Test plan: - All 72 conversation-runner + assistant-message-count unit tests pass (default baselineCount preserves count-progression scripts). - Full harness suite 2758/2758 green, tsc clean, oxfmt/oxlint clean on changed files. |
||
|
Jordan Ritter
|
48ef506abf |
fix(showcase): allow harness to boot locally without SHARED_SECRET
PR #5458 (
|
||
|
Jordan Ritter
|
7cc579a58d |
chore: drop dead .changeset/ debris and workflow path filters (#5470)
## Summary The repo migrated off `@changesets/*` to conventional-commit-driven releases. This chore PR removes the leftover debris: - **11 stale `.changeset/*.md` files** (the entire `.changeset/` directory) - **4 dead `paths:` filter lines** in two e2e workflows ## Migration context Releases are driven by `scripts/release/prepare-release.ts` / `scripts/release/lib/changes.ts::getChangesSummary`, which reads `git log <lastTag>..HEAD` commit subjects. It never touches `.changeset/`. Verification — none of the changesets tooling is wired up anymore: - No `.changeset/config.json` - No `@changesets/*` in any `package.json` (root or workspace) - No npm scripts reference `changeset` The 11 `.changeset/*.md` files describe changes that have either already shipped (via commit subjects in prior releases) or will ship in the next release (via the current commit subjects in the `v1.60.1..main` window). The files are inert with respect to releases — most are 2+ months old; the newest (`slack-agent-native-apis.md`) was added recently by a contributor who didn't know about the migration. ## What's removed `.changeset/` (11 files): - `angular-21-install-and-types.md` - `angular-disable-license-watermark.md` - `bump-license-verifier-ent-251.md` - `debug-mode.md` - `empty-mails-applaud.md` - `ent-314-thread-connect-ux.md` - `ent-658-generated-thread-tool-roundtrip.md` - `five-avocados-visit.md` - `fix-thread-switch-state-reset.md` - `little-pears-tell.md` - `slack-agent-native-apis.md` Workflow `paths:` filters (4 lines across 2 files): - `.github/workflows/test_e2e-dojo.yml` — `push.paths`, `pull_request.paths`, and the `dorny/paths-filter` `ts:` filter - `.github/workflows/test_e2e-legacy-v1.yml` — `push.paths` Nothing to fire on after the directory is gone. ## Test plan - [ ] CI green on this PR |
||
|
Jordan Ritter
|
5afa55f067 |
chore: drop dead .changeset/ debris and workflow path filters
The repo migrated off @changesets/* to conventional-commit-driven releases. scripts/release/lib/changes.ts::getChangesSummary reads `git log <lastTag>..HEAD` commit subjects and never touches .changeset/. No .changeset/config.json, no @changesets/* in any package.json, no npm scripts reference it. The 11 .changeset/*.md files describe changes that have either already shipped (via commit subjects in prior releases) or will ship in the next release (via the current commit subjects in the v1.60.1..main window) — the .changeset/ files are inert. Also removes the dead workflow `paths:` filters in test_e2e-dojo.yml and test_e2e-legacy-v1.yml that re-fired e2e on .changeset/ changes — nothing to fire on after the directory is gone. |
||
|
Jordan Ritter
|
f8b14f1d6a |
fix(react-core): await runAgent in useInterrupt::resolve and propagate to demo-local hooks (#5461)
## Summary
Convert `useInterrupt::resolve` and the demo-local
`useHeadlessInterrupt::resolve` callbacks from fire-and-forget into
`async` + `return await copilotkit.runAgent(...)`, so callers receive a
Promise that settles when the resume run settles.
## Mechanism
Pre-fix: `resolve(...)` called `copilotkit.runAgent({...})` without
`await` and without `return` — the arrow returned `undefined`. The
showcase harness DOM-settle check on the assistant confirmation bubble
timed out because consumers had no handle to sequence against the resume
run's settle.
Post-fix:
- Framework `useInterrupt::resolve` is `async`, `return await`s
`runAgent`, wraps in try/catch + `setPendingEvent(null)` +
`console.error` + rethrow on rejection.
- `onRunFailed` now also `setPendingEvent(null)` — symmetric with
`onRunStartedEvent`, prevents stuck popups on run failure.
- `InterruptHandlerProps.resolve` / `InterruptRenderProps.resolve` typed
`() => Promise<RunAgentResult>` (was `() => void`).
- The same fix-pattern applied to 13 demo-local `useHeadlessInterrupt`
implementations across integrations: ag2, agno, claude-sdk-typescript,
crewai-crews, langgraph-{fastapi,python,typescript}, langroid,
llamaindex, mastra, pydantic-ai, spring-ai, strands.
## Verification
- **Unit:** new tests `resolve returns a Promise that settles when
runAgent settles (RESUME-PATH)` + `resolve rejects when runAgent
rejects, logs the failure, and clears pending (RESUME-PATH-REJECT)`.
Red-then-green on baseline. 22/22 react-core hook tests passing.
- **Backplane (end-to-end):** showcase control plane on langgraph-python
via `./bin/showcase test langgraph-python:gen-ui-interrupt --d5 --direct
--verbose` (GREEN, 70.1s, was RED-RESUME-PATH baseline) and
`:interrupt-headless --d5 --direct --verbose` (GREEN, 7.7s, was
RED-RESUME-PATH baseline). Built Docker artifact + aimock D6 fixture
replay — staging-equivalent path.
## CR
4 review rounds × 7 unbiased agents = 28 agent-runs. 2 Procedure 3
audits (3 promotions in round 1 → Fix C + D; 0 promotions in round 2).
Convergence achieved per agent-confirmed `bucket (a) empty` + Procedure
3 zero `PROMOTE_TO_A`.
## Out of scope (deferred follow-ups)
- **No manifest unquarantine.** A prior version of this branch flipped
`interrupt-headless` and `gen-ui-interrupt` from
`not_supported_features` to `features` across 12 manifests, but Phase 4
backplane validation across 8 PATCHED integrations × 2 demos surfaced
(a) `langgraph-fastapi:gen-ui-interrupt` still RED-RESUME-PATH because
the framework patch doesn't propagate from workspace to the
integration's Docker image without a published `@copilotkit/react-core`
release, and (b) a separate popup-mount RED-RUNTIME-OTHER cluster across
6 integrations that's pre-existing rot unrelated to this PR. Manifest
unquarantine + `@copilotkit/react-core` version bump are deferred to a
follow-up `copilotkit-release` PR after this fix lands.
- **No version bump.** Release follows in a separate PR.
- **Bucket (c) follow-up backlog** (audit verdicts: 11 STAY_IN_C):
- JSON.parse guarding in showcase demos (pre-existing, all 13)
- Type-design quirks in `InterruptHandlerProps.result` (non-nullable in
type, nullable in runtime)
- `renderInChat` dynamic toggle leak (pre-existing edge case)
- JSDoc clarity ("symmetric with onRunFailed" wording, `@typeParam
TRenderInChat`, `event.value` "any" → "unknown")
- StrictMode test coverage gap
- 13× duplicate `useHeadlessInterrupt` → bucket (d): consolidate into a
shared module in a future PR
## Files changed
16 files, +486/-154:
- `packages/react-core/src/v2/hooks/use-interrupt.tsx`
- `packages/react-core/src/v2/hooks/__tests__/use-interrupt.test.tsx`
- `packages/react-core/src/v2/types/interrupt.ts`
- 13×
`showcase/integrations/<integ>/src/app/demos/interrupt-headless/page.tsx`
## Test plan
- [x] react-core unit tests (22/22 passing including new RESUME-PATH +
RESUME-PATH-REJECT regression tests)
- [x] Backplane D5 probe on langgraph-python (gen-ui-interrupt +
interrupt-headless GREEN end-to-end via the showcase control plane)
- [ ] CI green on this PR
|
||
|
tylerslaton
|
4564e6205d | chore: release bot-slack v0.0.2 | ||
|
Tyler Slaton
|
431d5baae0 |
feat(bot-slack): agent-native Slack APIs — assistant pane + native streaming (default-on) (#5447)
## What
Activates Slack's agent-grade APIs as the **default** experience for
`@copilotkit/bot-slack`, with zero config and safe degradation.
Implements the Notion spec *"Slack Agent APIs in bot-slack — assistant
pane + native streaming by default"* (Rev 2).
### Assistant pane ("Agents & AI Apps")
- Opening the pane greets the user + shows tappable prompt chips; each
pane conversation is its own thread (replies stay in-thread, instead of
leaking to a merged flat DM).
- While the agent runs, native composer status
(`assistant.threads.setStatus`: "is thinking…", "is using `tool`…")
replaces placeholder/`🔧` messages.
- Pane threads are auto-titled from the first message.
- Customize via the new `assistant` option; `assistant: false` disables
pane handling. Apps without the Agents toggle behave exactly as before
(pane machinery dormant).
### Native streaming
- Replies stream via `chat.startStream` / `appendStream` / `stopStream`
(raw markdown — real tables / fenced code render natively) wherever a
thread exists.
- Flat DMs and workspaces without the streaming API fall back to the
legacy `chat.update` transport **automatically** — the first
`startStream` failure marks the workspace legacy and replays via legacy.
Opting in can never break a bot. `streaming: "legacy"` forces the old
transport.
### Portable engine surface (`@copilotkit/bot`)
- `bot.onThreadStarted` lifecycle handler + `IncomingThreadStart` sink
event.
- Capability-gated `thread.setSuggestedPrompts` / `thread.setTitle` (the
shipped `postFile` gating pattern).
- Two `SurfaceCapabilities` flags + optional `PlatformAdapter` methods.
All degrade gracefully on surfaces without support.
## Files
- **New:** `bot-slack/src/assistant.ts` (Bolt `Assistant` middleware ⇄
engine sink), `bot-slack/src/native-stream.ts` (`NativeMessageStream`,
same `append/finish` contract as `MessageStream`, legacy fallback).
- **Engine:** `platform-adapter.ts`, `thread.ts`, `create-bot.ts`,
`index.ts` (+ `bot-ui` `Thread` type).
- **Adapter:** `adapter.ts` (options/capabilities/stream
branch/methods), `slack-listener.ts` (one-line no-double-delivery
guard), `event-renderer.ts` (pane status mode + native text transport).
- **Example/docs:** `examples/slack` manifest (`assistant_view` + scope
+ events) & dev-ex, package READMEs/ARCHITECTURE, `slack.mdx`.
- **Hooks:** `chore(hooks)` makes the `lint-fix` script
double-quote-free (it was breaking `sh -c "…"` on Windows/lefthook
2.1.1).
## Tests / verification (CI-runnable)
- New unit tests: `native-stream` (cadence, 12k continuation, fence
re-open, first- **and** continuation-`startStream` fallback),
`assistant` (defaults-before-onThreadStarted ordering, thread-scoped
turn, auto-title), listener no-double-delivery, renderer pane-status
mode, engine capability gating.
- Verified locally: `@copilotkit/bot` 31 tests, `@copilotkit/bot-slack`
199 tests, `bot-ui` tests; `tsc` clean (bot + bot-slack); build clean;
oxlint/oxfmt clean; publint/attw pass.
## ⚠️ E2E not covered here
The §8 end-to-end flows need a **real Slack workspace** with the Agents
toggle (open pane → chips → streamed reply w/ live status; channel
mention → native markdown; kill `startStream` → legacy fallback;
non-agent app → shipped behavior). `examples/slack` is the vehicle.
Three items remain to confirm against a live workspace (spec §7 spikes):
the `isAssistantThread` runtime detection, the exact `assistant_view`
manifest acceptance, and `stopStream` finalize tolerance.
Spec:
https://app.notion.com/p/copilotkit/Slack-Agent-APIs-in-bot-slack-assistant-pane-native-streaming-by-default-spec-37b3aa38185281f5948dc6a665064d04
|
||
|
Jordan Ritter
|
c88d687456 |
fix(showcase/harness): bubble-race elimination — 4 defects + atomic readCascadeState + cold-start retry + SSE counter (#5462)
## Summary
Fixes 4 bubble-race defects in the CopilotKit showcase e2e harness's
conversation runner that caused flaky turn-completion detection on
slow-streaming and cold-start integrations.
**Defects fixed (RED → GREEN):**
- **Defect 1** (fast-replay): turn settled on count, not text/SSE →
false-positive settle
- **Defect 2** (multi-turn flicker): un-turn-scoped bubble selection via
`list[last]` → cross-turn leak
- **Defect 3** (cascade blindness): diagnostic cascade picked wrong tier
→ wrong bubble text
- **Defect 4** (boot-time baseline staleness): pre-paint/stale bubble at
boot poisoned turn-1 settle
**Core changes:**
- `waitForTurnComplete` 3-conjunct primitive (SSE counter + DOM-at-index
+ text-quiet for settleMs)
- Atomic `readCascadeState` helper — single `page.evaluate` returns
`{count, text}` from one cascade tier (prevents within-poll cross-tier
inconsistency)
- Probe contract: `assertions(page, ctx: {bubbleIndex, text})` —
turn-scoped bubble retrieval (replaces `list[last]`)
- SSE interceptor — `__hk_runsFinished` counter via CDP + page-side
fetch wrap; idempotent attach/detach + handle-cache invalidation on stop
+ `framenavigated` reset (cold-start retry compatible)
- Cold-start retry — single bounded retry on first-attempt banner;
reload + re-resolve chat input + settleMs floor honoring shared turn
deadline
- Banner fast-fail — baseline banner snapshot + differs-from-baseline
2-poll debounce; in-poll `BannerVisibleError` translates to
`AssistantErroredError`
- Pre-paint placeholder env adapter for defect-4 repro infrastructure
**Defect repro tests:** real-browser tests in
`test/integration/bubble-race-repro-defect-{1,2,3,4}.test.ts` plus
mechanism-GREEN tests and `wait-for-turn-complete.test.ts` (3-conjunct
classification matrix), `probe-contract.test.ts` (ctx bridge),
`sse-counter.test.ts` (counter increment + framenavigated reset +
handle-cache invalidation).
## Test plan
- [x] Full harness vitest: 2742/2742 passing
- [x] Defect-1, defect-2, defect-4 RED tests verified GREEN after fix
- [x] Mechanism-GREEN tests pin internal contracts (cascade pollution
guard, SSE counter atomicity, init-script navigation re-entry, etc.)
- [x] cr-loop converged across 10 rounds (25→6→6→6→4→2→0 bucket-(a)
findings) + Procedure 3 audit clean
- [x] `oxfmt --check` clean across all 31 changed TS files
- [x] `tsc -p tsconfig.build.json` clean
- [ ] CI green on this PR
- [ ] Manual smoke: `bin/showcase test --d5
langgraph-python:agentic-chat` post-merge
## Follow-up debt (deferred from cr-loop; not subject-scope)
Detected by cr-loop reviewers but out of this PR's scope:
- `d6-all-pills.ts` deploy-churn NSF aggregation — features in
`notSupportedFeatures` lose `incapable` classification during
deploy-churn skip
- `d6-all-pills.ts` `joinAimockJournal` slug fallback under D6
concurrency — can pick wrong-feature entry when aimock doesn't echo
`x-diag-run-id`
- `d6-all-pills.ts` `BrowserDisconnectedError` sentinel is constructed
but never matched in `runFeature` catch — gets bucketed as
`driver-error`
- `d5-tool-rendering-default-catchall.ts` Path B narration
false-positive (substring matches in assistant prose)
- HITL registry side-effect tests use manual fallback registration that
masks real registration failures
- `installPrePaintFromEnv` and `attachSseInterceptor` are re-registered
on every `page.goto` — accumulate init scripts; harmless today via
in-script idempotency guards
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Jordan Ritter
|
1ef6a0960b |
test(harness): update d5-* + d6-all-pills unit tests to new probe + cascade contract
- d5-* unit tests: update page fakes / assertion-call shapes to pass
the ctx bridge ({bubbleIndex, text}) instead of querying the live
bubble list
- d6-all-pills.test.ts: update page-fake dispatch + cascade-state
expectations to match the atomic readCascadeState return shape
({count, text} or null) — picks up the cross-tier consistency
guarantee in the unit layer
|
||
|
Jordan Ritter
|
cd2af0f9fd |
test(harness): conversation-runner unit coverage — banner debounce, cold-start retry, skipFill, error-banner shapes
Comprehensive unit coverage for: - preFill + fillAndVerifySend retry behavior - baseline banner snapshot + differs-from-baseline debounce matrix (stable banner = no fail; changed banner = fast-fail after 2 polls) - cold-start retry cases (first-attempt banner → reload + retry; retry-then-banner → translatedErr; retry exhausts deadline) - skipFill / skipSend short-circuit paths - ErrorBannerReadResult union shape variants (present/absent/unknown) |
||
|
Jordan Ritter
|
3d49d4c38f |
test(harness): bubble-race integration tests + defect reproductions
- bubble-race-repro.ts: shared driver harness for repro tests
- bubble-race-repro-defect-{1,2,3,4}.test.ts: real-browser RED tests
that fail without the fix and pass with it
- Defect 1: fast-replay false-positive settle on count
- Defect 2: multi-turn flicker via list[last]
- Defect 3: cascade-tier blindness
- Defect 4: boot-time baseline staleness / pre-paint placeholder
- bubble-race-mechanisms.test.ts: GREEN tests pinning internal
contracts (cascade-pollution guard, atomic readCascadeState,
init-script idempotency)
- wait-for-turn-complete.test.ts: 3-conjunct classification matrix
- probe-contract.test.ts: ctx-bridge shape
- sse-counter.test.ts: counter increment + framenavigated reset +
handle-cache invalidation
|
||
|
Jordan Ritter
|
cf7efbda95 |
fix(harness): d6 driver — installPrePaintFromEnv + attachSseInterceptor wiring + helper-based captureDiagnostics
- d6-all-pills.ts: wire installPrePaintFromEnv and attachSseInterceptor PRE-goto in both defaultLauncher and pooled launcher paths, so first-paint state is deterministic and the SSE counter is armed before any page navigation - captureDiagnostics rewired through the Node-side helper (readCascadeState) — single page.evaluate per call, no within-snapshot tier drift - Consume BUBBLE_RACE_MESSAGES override for defect-4 repro flows |
||
|
Jordan Ritter
|
38e155e141 |
refactor(harness): probe-contract ctx bridge — assertions(page, {bubbleIndex, text}) for d5 scripts
- d5-gen-ui-custom, d5-gen-ui-open-advanced, d5-mcp-apps,
d5-subagents, d5-tool-rendering-default-catchall: wrap assertion
bodies in ctx bridge so they receive {bubbleIndex, text} for the
turn under test, replacing turn-leaky list[last] retrieval
- _gen-ui-shared: add readAssistantTextAt(page, bubbleIndex)
adapter; remove readLastAssistantText (no longer turn-scoped)
This is the read-side counterpart to the atomic readCascadeState
on the helper side: probes ask for the bubble at the runner's
chosen index instead of guessing.
|
||
|
Jordan Ritter
|
3f3986b00b |
refactor(harness): waitForTurnComplete 3-conjunct primitive + cold-start retry + banner debounce
- waitForTurnComplete: 3-conjunct settle (SSE counter + DOM-at-index + text-quiet for settleMs) replaces count-only false-positive path - Cold-start retry: single bounded retry on first-attempt banner; reload + re-resolve chat input + settleMs/POLL_INTERVAL_MS floor honoring shared turn deadline; translatedErr throw on exhaustion - Banner fast-fail: baseline banner text snapshot + differs-from-baseline 2-poll debounce; in-poll BannerVisibleError translates to AssistantErroredError - New error classes: BannerVisibleError, AssistantErroredError - readErrorBanner returns 3-state union via ErrorBannerReadResult discriminated union - chatInputSelector cascade for input resolution across integrations |
||
|
Jordan Ritter
|
473a9539a0 |
feat(harness): add bubble-race helpers — assistant-message-count, sse-interceptor, init-scripts
- assistant-message-count: 4-tier cascade + atomic readCascadeState
returning {count, text} from a single page.evaluate (prevents
within-poll cross-tier inconsistency); null fallback for
cascade-pollution
- sse-interceptor: __hk_runsFinished CDP counter + page-side fetch
wrapper; idempotent attach/detach with handle-cache invalidation
on stop and framenavigated reset (cold-start retry compatible)
- init-scripts: pre-paint placeholder env adapter
(installPrePaintFromEnv), strip-selector helper, and
BUBBLE_RACE_MESSAGES override consumer for defect-4 repro
infrastructure
|
||
|
Jordan Ritter
|
6fbd66fa83 |
fix(showcase): mirror useInterrupt RESUME-PATH contract in 13 demo-local hooks
Each integration's interrupt-headless demo defines a local useHeadlessInterrupt
hook around the framework useInterrupt. Slot-2 originally identified 8
quarantined integrations (claude-sdk-typescript, langgraph-{fastapi,python,
typescript}, langroid, pydantic-ai, spring-ai, strands); review-round
follow-ups extended the sweep to llamaindex, mastra, ag2, agno, and
crewai-crews (5 more integrations sharing the same byte-identical hook).
The demo-local resolve() previously fire-and-forgot copilotkit.runAgent(...)
via `void runAgent(...).catch(() => {})`. Mirroring the framework fix:
- Make resolve async, return await copilotkit.runAgent(...).
- Use a pendingRef so resolve has stable identity (drop pending from
useMemo deps).
- Type signature: resolve: (response: unknown) => Promise<unknown>.
- Wrap in try/catch + setPending(null) + console.error + rethrow,
symmetric with the framework hook.
- onRunFailed also setPending(null).
13 integrations patched byte-identically.
|
||
|
Jordan Ritter
|
7b80590f67 |
fix(react-core): await runAgent in useInterrupt::resolve
resolve() previously called copilotkit.runAgent(...) without await and without return, so callers had no handle to sequence against the resume run's settle. The harness DOM-settle check timed out for any consumer awaiting the assistant confirmation bubble. Changes: - Make resolve async, return await copilotkit.runAgent(...) so callers receive a Promise that settles when the resume run settles. - Update InterruptHandlerProps / InterruptRenderProps resolve return type from () => void to () => Promise<RunAgentResult>. - Wrap runAgent in try/catch + setPendingEvent(null) + rethrow, so rejection clears the popup AND propagates to awaiting callers (mirrors onRunFailed handler symmetry; closes the case where runAgent rejects before any run-failed event fires, e.g. network error pre-RUN_STARTED). - onRunFailed now also setPendingEvent(null) symmetric with onRunStartedEvent. - Regression tests: RESUME-PATH asserts resolve() returns a Promise that settles 1:1 with runAgent; RESUME-PATH-REJECT asserts rejection propagates, popup clears, console.error logs. |
||
|
Jordan Ritter
|
c81b361f15 |
fix(showcase/harness): register /webhooks/deploy on CP path + fail-loud on missing SHARED_SECRET (#5458)
## Problem
The \`Showcase: Verify Deploy\` workflow runs \`notify-harness\` after
every merge to \`main\` and POSTs an HMAC-signed payload to
\`\${SHOWCASE_HARNESS_URL}/webhooks/deploy\`. That POST has been
returning **404** on every main deploy since at least 2026-06-12 (5+
consecutive failures), so the harness dashboard has not reflected any
recent deploys.
**Root cause:**
1. \`POST /webhooks/deploy\` is only registered when
\`webhookSecrets.length > 0\` — gate at
\`showcase/harness/src/http/server.ts:119\`.
2. The CP \`buildServer\` call in \`runControlPlane\`
(\`showcase/harness/src/orchestrator.ts\` ~line 2961 pre-fix)
**omitted** both \`webhookSecrets\` and \`metrics\`. The public Railway
host running the CP role never mounted the route — every notify-harness
POST returned 404.
3. The FATAL-CONFIG guard on missing \`SHARED_SECRET\` (pre-fix
\`orchestrator.ts:782-790\`) only fired when \`NODE_ENV ===
"production"\`. Any deploy whose NODE_ENV was unset, set to something
other than the literal \`"production"\`, or set after the check,
silently booted with \`webhookSecrets=[]\` and no fatal error.
## Fix
**Part 1: Register the route on the CP path.**
- Lifted the env→secrets loader into a new exported helper
\`loadWebhookSecrets()\` (orchestrator.ts ~175-209) so both boot paths
consume the same predicate.
- Worker \`boot()\` (~line 836) calls it; CP \`runControlPlane\` (~line
2421) now also calls it.
- The CP \`buildServer\` call (~line 3047) now passes \`webhookSecrets\`
and \`metrics\` alongside the existing \`bus\`/\`probes\`/\`fleetRuns\`
wiring — same shape as the worker call.
**Part 2: Tighten the FATAL-CONFIG predicate.**
- New predicate (inside \`loadWebhookSecrets\`): throw unless EITHER a
secret is set OR \`NODE_ENV === "test"\` OR \`HARNESS_ALLOW_NO_SECRET
=== "1"\` (narrow escape hatch for local dev).
- Thrown error message names the env-vars, the gate location
(\`src/http/server.ts:119\`), the silent-404 symptom, and the escape
hatches — so an operator reading the deploy log can recover without
spelunking.
## Test coverage
Four new red-green tests in \`orchestrator.test.ts\` under \`B2:
/webhooks/deploy registered on CP + fail-loud on missing
SHARED_SECRET\`:
- **Test A** — CP boot registers \`POST /webhooks/deploy\` when
\`SHARED_SECRET\` is set. Probe with no HMAC headers: pre-fix returned
404, post-fix returns 401 (route mounted, HMAC reject path fires).
RED→GREEN.
- **Test B** — Worker boot still registers the route (regression guard).
Pre-fix and post-fix both 401.
- **Test C** — FATAL-CONFIG fires when \`SHARED_SECRET\` unset AND
\`NODE_ENV=development\`. Pre-fix resolved silently, post-fix rejects
with \`/FATAL-CONFIG.*SHARED_SECRET/\`. RED→GREEN.
- **Test D** — Boot succeeds when \`SHARED_SECRET\` unset AND
\`NODE_ENV=test\` (escape hatch). Pre-fix and post-fix both green.
Existing R5-G4 D5 test (orchestrator.test.ts:1592) updated to match the
new error message via \`/FATAL-CONFIG.*SHARED_SECRET/\` — same throw,
broader match.
## Verification
- Full harness suite: **2719/2719 green** across 128 files
- \`tsc --noEmit\`: clean
- RED capture pre-fix:
- Test A: \`AssertionError: expected 404 to be 401\`
- Test C: \`AssertionError: promise resolved "{ port: ..., bus: { ... },
... }" instead of rejecting\`
## Deploy note
Requires a CP-role redeploy on the public Railway host for the new route
registration to take effect — once redeployed, the next
\`notify-harness\` POST will land on the harness dashboard.
## Test plan
- [x] B2 Tests A-D pass post-fix (red→green captured)
- [x] Full harness vitest suite green (2719/2719)
- [x] \`tsc --noEmit\` clean
- [ ] Post-deploy: confirm next \`Showcase: Verify Deploy\` run shows a
2xx notify-harness POST (not 404)
- [ ] Post-deploy: confirm harness dashboard reflects the next deploy
|
||
|
Jordan Ritter
|
3ce3f5394d |
fix(showcase/harness): hoist OPS_TRIGGER_TOKEN fail-loud + emit on subscribeDeployResults sync throw (R3 cleanups)
R3-F1 (OPS_TRIGGER_TOKEN hoist): Extract loadOpsTriggerToken() and hoist its invocation to the top of both boot() and runControlPlane(), alongside loadWebhookSecrets() and loadPocketbaseUrl(). Pre-fix the empty/whitespace check fired AFTER pb / bus / scheduler / writer / S3 uploader allocations, so a typo'd `OPS_TRIGGER_TOKEN=` allocated expensive resources before throwing. Now all three fail-loud config predicates fire at the top, before any allocation needing teardown. Behaviour is unchanged for valid tokens (trimmed via R3-A.5 contract) and for the unset case (router omitted with info log). R3-F2 (sync-throw also emits deploy.writer.failed): subscribeDeployResults() pre-fix only emitted `deploy.writer.failed` on writer.write() promise rejection — a synchronous throw inside deployEventToProbeResult() (malformed event, type drift) bypassed the catch and we lost both the log AND the bus emit. Wrap the sync mapping in try/catch and mirror the rejection path so alert rules / metrics subscribers observe sync throws the same way they observe async write failures. Tests: - 5 unit tests for loadOpsTriggerToken: undefined / empty-string fail-loud / whitespace-only fail-loud / R3-A.5 trim contract / verbatim value. - 1 test for the sync-throw path: vi.doMock deployEventToProbeResult to throw, assert bus emits deploy.writer.failed with the err message AND writer.write was NOT called. Red-green verified locally (test fails without the try/catch). Suite: 2731 passing (+6 new). Typecheck clean. |
||
|
Jordan Ritter
|
d5bdf2c9fd | fix(showcase/harness): track CP deploy.result unsubscribe + emit on write failure (R2 cleanups) | ||
|
Jordan Ritter
|
b0ba8e869d |
fix(showcase/aimock): align catchall fixture shape for 6 remaining D5 reds (#5460)
## Summary Follow-up to #5459. That PR added the right `userMessage` matchers ("forecast for Tokyo" + "current price of AAPL") and flipped most integrations green, but 6 cells stayed red because the FIXTURE SHAPE on the matched fixtures was also broken. **Failing cells (post-#5459 deploy):** - `d5:agno/tool-rendering-custom-catchall` - `d5:llamaindex/tool-rendering-custom-catchall` - `d5:langroid/tool-rendering-custom-catchall` - `d5:claude-sdk-python/tool-rendering-custom-catchall` - `d5:strands/tool-rendering-custom-catchall` - `d5:strands/tool-rendering-default-catchall` ## Root cause The failing tool-emit fixtures all used `hasToolResult: false` (or `toolName: "<tool>"`) as their gate. The custom-catchall probe drives TWO sequential prompts (Tokyo, then AAPL). After Tokyo's tool result lands in the thread, aimock's `hasToolResult` is permanently true (`messages.some(m => m.role === 'tool')`), so `hasToolResult:false` can never match the AAPL turn → no fixture → 30s timeout. PocketBase confirms this exact mode — e.g. agno: ``` errorDesc: "timeout: assistant did not respond within 30000ms" failure_turn: 2 turns_completed: 1 / 2 ``` claude-sdk-python's `toolName` gate fails for an analogous reason on the second turn when the backend doesn't forward the tool definition uniformly. The same trap is documented in `showcase/aimock/d6/langgraph-python/tool-rendering.json`: > Gated on toolName:get_stock_price rather than hasToolResult:false. The D5 tool-rendering-custom-catchall probe runs 'weather in Tokyo' first, which leaves a get_weather tool result in the thread; the aimock router implements hasToolResult as messages.some(m=>m.role==='tool'), so hasToolResult is permanently true on the AAPL turn and a hasToolResult:false gate could never match (→ no_fixture_match → 503 → 30s timeout). ## Fix Align all 6 files to the canonical pattern used by working integrations (mastra, spring-ai, built-in-agent): 1. `toolCallId`-keyed narration fixture FIRST 2. tool-emit fixture SECOND, gated ONLY on `userMessage` + `context` (no `hasToolResult` / `toolName`) aimock's `toolCallId` matcher checks `messages[last].role === 'tool' && tool_call_id === ...`, which correctly fires only on the post-tool-result iteration regardless of older tool results in the thread. The tool-emit fixture below it always matches on turn-init (last message is user). Strands' two files additionally needed REORDERING — they had the tool-emit fixture before the toolCallId fixture, defeating first-match-wins. ## Scope discipline - Pure fixture-content alignment — no harness, probe, frontend, or backend changes. - Did NOT touch `showcase/harness/**` (peer session territory per `/tmp/coordinate/showcase-reds-coord/CHANNEL.md`). - Did NOT touch the two `d5-tool-rendering-*-catchall.ts` probes. ## Caveat (strands) During investigation, `showcase-strands-staging.up.railway.app` was returning 502 / page-load failures (infra, not fixture). Strands' fixture defects are still real and worth correcting, but the cells may stay red until the backend recovers. ## Test plan - [ ] CI green - [ ] Wait for `showcase_build.yml -f service=aimock` rebuild + Railway redeploy - [ ] Re-check PocketBase `state` for the 6 cells flips to `green` (excluding strands if backend stays down) - [ ] Confirm no regression on the other catchall cells already green post-#5459 |
||
|
Jordan Ritter
|
df97ae8d2a |
fix(showcase/aimock): align catchall fixture shape for 6 remaining D5 reds
PR #5459 added userMessage matchers ("forecast for Tokyo" + "current price of AAPL") on the catchall fixtures, which flipped most integrations from red to green. Six cells stayed red on staging because the FIXTURE SHAPE itself was broken on the matched fixtures, not just the userMessage key. Failing cells (all custom-catchall except strands default): d5:agno/tool-rendering-custom-catchall d5:llamaindex/tool-rendering-custom-catchall d5:langroid/tool-rendering-custom-catchall d5:claude-sdk-python/tool-rendering-custom-catchall d5:strands/tool-rendering-custom-catchall d5:strands/tool-rendering-default-catchall PocketBase confirms the failure mode: turn 1 (Tokyo) completes; turn 2 (AAPL) times out at 30s or renders the wrong content. E.g. agno: errorDesc: "timeout: assistant did not respond within 30000ms" failure_turn: 2 turns_completed: 1 / 2 Root cause: the failing tool-emit fixtures used `hasToolResult: false` (or `toolName: "<tool>"`) as their gate. aimock's hasToolResult check is `messages.some(m => m.role === 'tool')` over the WHOLE thread — so once turn 1's Tokyo tool result lands in the conversation, hasToolResult is permanently true and `hasToolResult:false` can never match turn 2 → no fixture → 30s timeout. `toolName:get_stock_price` likewise fails when an integration backend doesn't forward the tool definition on turn 2. The same trap is documented in showcase/aimock/d6/langgraph-python/tool-rendering.json: "_comment": "Gated on toolName:get_stock_price rather than hasToolResult:false. The D5 tool-rendering-custom-catchall probe runs 'weather in Tokyo' first, which leaves a get_weather tool result in the thread; the aimock router implements hasToolResult as messages.some(m=>m.role==='tool'), so hasToolResult is permanently true on the AAPL turn and a hasToolResult:false gate could never match (→ no_fixture_match → 503 → 30s timeout)." Fix: align all 6 files to the canonical pattern used by mastra/spring- ai/built-in-agent on this probe: 1. toolCallId-keyed narration fixture FIRST 2. tool-emit fixture SECOND with ONLY `userMessage` + `context` (no hasToolResult / toolName gate) The toolCallId narration uses aimock's `messages[last].role === 'tool' && tool_call_id === ...` check, so it correctly wins on iteration 2 (post-tool-result) without being affected by older turns' tool results. The tool-emit fixture matches turn 1 (last message is user) and re-emits only when the narration above hasn't matched. Strands' two cells additionally needed REORDERING — they had the tool- emit fixture before the toolCallId fixture, defeating first-match-wins. Strands' staging backend was also returning 502 during testing; once it recovers, the corrected fixtures should let the probe pass. The fixture changes are necessary but may not be sufficient for strands if backend remains down. No probe-side, harness, or backend changes — pure fixture-content alignment. Six fixture files modified; line totals: -87 / +68. |