Commit Graph

4305 Commits

Author SHA1 Message Date
Austin Merrick b97157cadc docs: add Vue quick start guide
Add a minimal getting-started guide for @copilotkit/vue under Platforms.
Connects a Vue app directly to an AG-UI agent via HttpAgent so there is no
runtime to stand up, and imports from @copilotkit/vue/v2 to match the v2
docs convention. Links out to Integrations for agent-side setup.
2026-06-16 11:51:50 -07:00
Jordan Ritter 02275af2ab fix(showcase/aimock/d6/claude-sdk-typescript): delete dead turnIndex:0 fallback entries (consistency with A2-cleaned siblings)
After A2 (commit 6c596d8d6) stripped toolName from the primary emit entries, sibling turnIndex:0 fallback entries became unreachable under first-match-wins. The 7 A2-target siblings (ag2/google-adk/lgf/lgts/mastra/msdotnet/csdkts toolName strip) had equivalent dead fallbacks deleted in that commit; csdkts was inconsistently treated. Removing the 2 dead entries restores cross-fleet consistency. Probe contract unaffected — toolCallId-gated narration entries still emit the required content phrase.

(cherry picked from commit 9720636519f4cd858fcdc08ed84597be05604a2e)
2026-06-16 11:44:17 -07:00
Jordan Ritter 8ad31e7810 fix(showcase/aimock/d6): bring AAPL response payload to canonical $338.37/-2.96% on 3 fixtures
crewai-crews/ms-agent-python/pydantic-ai catchall AAPL entries still carried stale $189.42/up 1.27% narration and toolCall args lacking price_usd/change_pct. Round 1 CR finding #3 was identified but never given a fix agent. Brought all 3 to canonical shape matching built-in-agent (A5) and csdkts narration (A6): price_usd=338.37, change_pct=-2.96, narration 'AAPL is trading at $338.37, down 2.96% on the day — rendered through the custom wildcard catchall.'

(cherry picked from commit df84657310451500278d0b3d0125c9c490042d2b)
2026-06-16 11:44:17 -07:00
Jordan Ritter 601e45f0d8 refactor(showcase): rename cell-badge RT → BE (parallel to E2E → UI in #5473)
PR #5498 was mis-scoped — it renamed the L1 row label and catalog status
display from "Wired" to "BE (Agent)", but the dashboard's per-cell badge
code rendered in the grid was still "RT". This commit lands the parallel
flip that mirrors PR #5473's E2E → UI rename for the D4 chat/tools
round-trip badge.

Visible-label flips (RT → BE):
  - unified-cell.tsx: <TestBadge name="RT" level={model.d4} /> → name="BE"
  - cell-drilldown.tsx: { key: "d4", label: "RT (Round Trip)" } → "BE (Round Trip)"
  - adaptive-legend.tsx: legend entry "Round Trip (RT)" → "Round Trip (BE)"
  - Doc-comment taxonomy notes in cell-pieces.test.tsx, composed-cell.tsx,
    cell-model.ts, page-stats.ts, unified-cell.tsx, cell-drilldown.tsx,
    overlay-selector-integration.test.tsx, dashboard-color-matrix.test.tsx
  - Test assertions in dashboard-color-matrix.test.tsx, cell-drilldown.test.tsx,
    unified-cell.test.tsx (mock-badge-RT → mock-badge-BE, drilldown-badge-rt-
    → drilldown-badge-be-, "RT (Round Trip)" → "BE (Round Trip)")

Stable contract identifiers are PRESERVED — only the visible name flips:
  - level={model.d4} unchanged (D4 is still D4)
  - dimensionKey={keyFor("e2e", ...)} unchanged
  - probe registry-key "d4"/"chat"/"tools" unchanged
  - persisted dimension codes unchanged

Two intentional residual "RT" mentions remain in
cell-pieces.signal-degrade.test.tsx as historical commentary documenting
the rename lineage (RT → UI on the e2e badge in #5473, then RT → BE on the
D4 badge here).

Verification (in showcase/shell-dashboard):
  - npm run typecheck: clean
  - npm test: 1089 tests passed
  - npm run build: clean (next build OK)
  - LC_ALL=C grep -rlP '\x00' showcase/shell-dashboard/src: empty
2026-06-16 11:40:50 -07:00
Jordan Ritter e266b6215c fix(showcase/harness): wire requireContentPhrase=true in custom-catchall turn-2 assertion
validateCustomCatchall's requireContentPhrase branch was unreachable: assertCustomCatchall defaulted it to false and no caller overrode. The whole point of the PR is to catch cross-fixture leakage via content assertion — wiring it on. Turn-2 (Quote AAPL) now asserts the custom content phrase is present in the rendered bubbles, not just the testid.

(cherry picked from commit c9620c96f9addfee18f87fb1f8b2ae7fb040b3b8)
2026-06-16 11:27:56 -07:00
Jordan Ritter d091749477 fix(showcase/harness): correct fixtureFile metadata on catchall probes
Probes registered fixtureFile: 'tool-rendering.json' but actual files are tool-rendering-{default,custom}-catchall.json. Field is signal-only (aimock loads directory-wide content-driven matching) but accurate metadata helps debugging. Test files updated to enforce the correct values.

(cherry picked from commit 3c9de26375546f2269870b563ecd1526ec5a3bc2)
2026-06-16 11:27:46 -07:00
Jordan Ritter 6c596d8d65 fix(showcase/aimock/d6): strip toolName gate + dead turnIndex:0 content fallbacks in 7 catchall fixtures
ag2/claude-sdk-typescript/google-adk/langgraph-fastapi/langgraph-typescript/mastra/ms-agent-dotnet: AAPL is turn-1 so turnIndex:0 fallback unreachable; toolName gate fails on wildcard-renderer integrations that don't register get_stock_price. Aligned to LGP-gold pattern (userMessage+context discriminator, no toolName, no turnIndex:0 fallback). Preserved legitimate multi-pill matchers (SF/flights/d20/chain) on the 4 multi-pill integrations.

(cherry picked from commit c10821b5d904b31bee2ab2a39db3b1565aecba26)
2026-06-16 11:27:37 -07:00
Jordan Ritter 9e2c91d02f fix(showcase/aimock/d6/claude-sdk-typescript): replace duplicated stock_price toolCall in AAPL fallback with narration content
(cherry picked from commit 5081966fbddf5d0090e333005bb63f5dd4b36c58)
2026-06-16 11:27:27 -07:00
Jordan Ritter 6d60cfd310 fix(showcase/aimock/d6/built-in-agent): align AAPL price to $338.37 (fleet parity)
built-in-agent shipped $189.42 vs the rest of the fleet's $338.37. Drift makes any content-asserting test on AAPL price brittle. Aligned.

(cherry picked from commit c0796da2a63abfeaa1d8ea06200df0754d30a2e6)
2026-06-16 11:27:17 -07:00
Jordan Ritter 14667871a8 fix(showcase/aimock/d6/spring-ai): strip hasToolResult from weather emit (LGP-gold parity)
spring-ai retained hasToolResult:false on the Tokyo weather emit fixture — outlier vs LGP-gold and the other 16 catchall fixtures. Aligned: userMessage+context discriminator only.

(cherry picked from commit d268a88c4a3f405dfcc15b2aaccc190e61cd7ac7)
2026-06-16 11:27:06 -07:00
Jordan Ritter 072c18bf19 fix(showcase/aimock/d6): strip permanently-true hasToolResult gate on AAPL in 4 catchall fixtures
built-in-agent, crewai-crews, ms-agent-python, pydantic-ai: Tokyo turn-1 tool result makes hasToolResult permanently true → AAPL fixture never matched → 30s timeout. Aligned with agno/langroid/llamaindex/strands/claude-sdk-python pattern: rely on userMessage+context (and toolCallId where relevant) as the gate.

(cherry picked from commit 8e313cd1b043cf1c61efb82143171a28d7699c49)
2026-06-16 11:26:57 -07:00
Martha Kelly Schumann fae4d30ad8 Merge branch 'main' into codex/fix-docs-model-validator-trigger 2026-06-16 10:10:06 -07:00
Tyler Slaton 3fb24ef403 fix(shell-docs): address docs review edge cases 2026-06-16 10:08:54 -07:00
Jordan Ritter 06112f4c36 refactor(showcase): rename Wired → BE (Agent) across dashboard taxonomies (#5498)
## Summary

Renames the display label \"Wired\" to **\"BE (Agent)\"** across both
taxonomies in the showcase dashboard, unifying the two overloaded
meanings (live-probe runtime health + catalog integration build state)
under one user-visible name. Mirrors the shape of #5473's `E2E (Demo)` →
`UI (Frontend)` rename — same atomic, label-only refactor.

## Why

The word \"Wired\" was overloaded across the dashboard:
- The **live-probe L1 row** in `level-strip` rendered \"Wired\" for
runtime backend health (the agent is up and responding).
- The **catalog integration-status** column in
`stats-bar`/`adaptive-stats-bar`/`coverage-bar`/`filter-chips` rendered
\"Wired\" for build state (the integration is built and shipped).

Both refer to the same underlying thing — the backend agent integration
— at different lifecycle stages. Calling both **\"BE (Agent)\"** makes
the dashboard internally consistent and aligns with the parallel \"UI
(Frontend)\" label introduced in #5473.

## Scope

- Visible label flip across `level-strip.tsx`, `packages-section.tsx`
legend (incl. `W=Wired` → `B=BE (Agent)` mnemonic), `stats-bar.tsx`,
`adaptive-stats-bar.tsx`, `coverage-bar.tsx`, `filter-chips.tsx`.
- Internal counter renames: `pctWired`/`totalWired` →
`pctBeAgent`/`totalBeAgent` (`coverage-bar.tsx`, `cells-view.tsx`,
`parity-view.tsx`).
- Test assertion + comment updates in `level-strip.test.tsx`,
`packages-section.test.tsx`.
- Slugifier fix in `stats-bar.tsx` testid generator —
`label.toLowerCase()` on `\"BE (Agent)\"` previously produced `stat-be
(agent)` (spaces + parens). Now slugifies to `stat-be-agent`.
Incidentally also fixes the pre-existing `stat-max depth` →
`stat-max-depth`. Surfaced and fixed during cr-loop Round 1.

## Stable contracts preserved (no behavior change)

- PocketBase row keys `agent:<slug>`
- `LiveDimension` union value `\"agent\"`
- `Status` enum string values `\"wired\" | \"stub\" | \"unshipped\" |
\"unsupported\"` (persisted catalog data; only display labels flip)
- Filter chip `id: \"wired\"` (filter state key joining to
`cell-matrix.tsx`)
- `catalog.metadata.wired` data field reads
- Harness driver `kind`/emission contracts
- `data-testid=\"coverage-segment-wired\"`

## Commits (4)

- \`ef40bd7c\` live-probe agent dimension Wired → BE (Agent)
- \`bfebac65\` catalog status Wired display → BE (Agent)
- \`f316d055\` rename internal Wired counters to BeAgent
- \`80b526d6\` slugify stats-bar testids (CR fix)

## Verification

- typecheck (\`tsc --noEmit\`): clean
- lint (next build inline): clean
- tests (\`vitest run\`): 1089 pass / 1 skip / 0 fail across 63 files
- build (\`next build\`): clean
- oxfmt: clean
- NUL-byte scan: empty
- cr-loop converged at Round 2 with Procedure 3 audit returning 0
PROMOTE_TO_A items

## Follow-up backlog

Round 1 + Round 2 surfaced ~30 pre-existing structural findings in
adjacent code (not subject-scope for this rename). Grouped into 8
follow-up PR subjects: fail-loud sweep, catalog view derivation
extraction, derivePackageDepth gate consistency, parity tier counting
hardening, coverage-bar tooltip/rounding, cells-view stats useMemo, test
hardening, Tailwind opacity-on-CSS-var. None block this rename.
2026-06-16 09:58:23 -07:00
wuyangfan a76b2eff69 docs: apply LangGraph link fixes to shell-docs (canonical source)
The top-level docs/ folder is retired; the live docs are sourced from
showcase/shell-docs. Move the same three link fixes there and revert the
no-op edits to docs/:

- configurable.mdx: graph-api -> use-graph-api (#add-runtime-configuration)
- configurable.mdx: drop malformed %23 double-anchor on the schema link
- step-2-langgraph-agent.mdx: studio.langchain.com -> docs.langchain.com/.../studio

Closes #3190.
2026-06-16 09:56:53 -07:00
Martha Schumann 38e987f7e4 fix(docs): validate showcase docs model names 2026-06-16 09:55:14 -07:00
Jordan Ritter 80b526d68b refactor(showcase): slugify stats-bar testids to handle multi-word labels
The testid generator in stats-bar.tsx used `label.toLowerCase()` directly,
producing fragile selectors with spaces and parens for compound labels.
After this PR's rename of "Wired" to "BE (Agent)", the testid became
`stat-be (agent)` — a CSS-hostile selector. A pre-existing
`stat-max depth` (from "Max Depth") had the same problem but predated
this PR.

Fix: replace with a proper slugifier:
  label.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/(^-|-$)/g, "")

Resulting testids:
  "BE (Agent)"   -> stat-be-agent   (was stat-be (agent), broken)
  "Max Depth"    -> stat-max-depth  (was stat-max depth, pre-existing fix)
  "Stub"         -> stat-stub       (unchanged)
  "Unshipped"    -> stat-unshipped  (unchanged)
  "Unsupported"  -> stat-unsupported (unchanged)
  "Regressions"  -> stat-regressions (unchanged)
  "Failures"     -> stat-failures   (unchanged)

Call-site enumeration receipt (mandatory):
  rg 'stat-wired|stat-be|stat-max|stat-stub|stat-unshipped|stat-unsupported|stat-regressions|stat-failures' showcase/shell-dashboard/
    -> no matches (zero hardcoded references anywhere)
  rg 'data-testid=.stat-|"stat-|`stat-' showcase/shell-dashboard/
    -> only one hit: stats-bar.tsx:29 (the generator itself)
  rg 'stat-' showcase/shell-dashboard/tests/
    -> no matches (no Playwright/visual tests depend on these testids)

No test updates required.

Verification:
  npm run typecheck   -> clean
  npm test            -> 1088 pass / 1 pre-existing unrelated failure
                         (useLiveStatus.test.tsx R5 F5.2 — does not
                         reference stats-bar, exists on PR HEAD)
  npm run build       -> clean (Next.js lint inline; no lint script)
  NUL byte sweep      -> empty
2026-06-16 09:40:49 -07:00
Austin Merrick d88245199f docs: testing that previews are up (#5431) 2026-06-16 09:38:39 -07:00
Jordan Ritter 9491b89340 fix(showcase): disjoint catchall userMessages + content-asserting probes (supersedes #5465)
Closed PR #5465 introduced cross-fixture leakage by stripping the
toolName discriminator on shared {userMessage, context} keys: with
aimock's alphabetical first-match-wins ordering,
tool-rendering-custom-catchall.json sorts before
tool-rendering-default-catchall.json, so default-catchall page requests
were served the custom file's content. The probe was structurally blind
because it asserted only DOM testids (copilot-tool-render +
data-tool-name=get_weather) — both fixtures emit get_weather, so the
testid signal passed regardless of which fixture won.

Real LGP-gold pattern is disjoint userMessages between default and
custom catchall fixtures (NOT shared keys discriminated by toolName).
This PR ports the LGP-gold pattern to the other 17 integrations and
adds page-text content assertions to both d5 catchall probes so this
class of regression can't recur silently:

- Probe prompts disjoint between default-catchall and custom-catchall
- Fixture userMessages updated to match the new disjoint prompts
- Page-text content assertions in both d5 catchall probes
  (default negatively asserts the custom-catchall leak phrase;
   custom positively asserts it)

Local gold-standard red-green proof captured:
- Step A: live staging Playwright baseline (RED-OF-RECORD)
- Step B: local control-plane on origin/main reproduces structural
  fragility (probes pass at testid level, custom fixture wins on
  default's userMessage path)
- Step C: local control-plane on this branch — both catchall probes
  GREEN with the new content-asserting assertions
    tool-rendering-default-catchall: pass=true (5443ms)
    tool-rendering-custom-catchall:  pass=true (8956ms)
                                     cross-tool signature passed

LGP (langgraph-python) was already disjoint; it remains untouched.
2026-06-16 09:37:51 -07:00
Jordan Ritter f316d055fb refactor(showcase): rename internal Wired counters to BeAgent
Renames the internal display-counter variables that fed the
stats-bar / coverage-bar "Wired" labels to match the new "BE (Agent)"
taxonomy: totalWired → totalBeAgent in cells-view.tsx and
parity-view.tsx, pctWired → pctBeAgent in coverage-bar.tsx. These
are local render-time aggregates, not part of any persisted
contract.

The status enum literal "wired" (the .filter((c) => c.status ===
"wired") guard, the coverage-segment-wired test ID, and the
wiredByCategory Map's local name) is intentionally preserved — those
all key directly off the persisted catalog Status enum and changing
them would expand scope into the catalog contract.
2026-06-16 09:23:00 -07:00
Jordan Ritter bfebac6565 refactor(showcase): relabel catalog status Wired display → BE (Agent)
Unifies the catalog integration status display label with the same
"BE (Agent)" taxonomy as the live-probe agent dimension. The
underlying Status enum literal ("wired"), the catalog.metadata.wired
data field, and the filter chip id ("wired" → cell-matrix.tsx:311
status === "wired" filter key) are all PRESERVED — they are
persisted catalog data + filter state contracts that must not move.
Only the user-facing display label on stats-bar, adaptive-stats-bar,
and the filter chip flips.

This collapses the two distinct dashboard "Wired" surfaces (the L1
live-probe dimension and the per-cell build-state count) under one
unified label, matching the user-confirmed Path B.
2026-06-16 09:22:41 -07:00
Jordan Ritter ef40bd7cfb refactor(showcase): relabel live-probe agent dimension Wired → BE (Agent)
Unifies the L1 "agent" live-probe display label with the taxonomy
convention established by #5473 (UI (Frontend), E2E, CV, D6 — layer
descriptor in parentheses). The dimension name stays "agent" in code
(PocketBase row keys agent:<slug>, LiveDimension union, keyFor
lookups are all unchanged stable contracts) — only the visible label
changes.

Updates the level-strip L1 badge label and the packages-section
L1-L4 header legend (W → B, "Wired" → "BE (Agent)") so the
level-strip's ToneChip first-letter abbreviation matches the legend
key. Test assertions covering the rendered letter, the legend text,
and the degraded-tone test's local variable follow suit.
2026-06-16 09:22:31 -07:00
Varun Nuthalapati 5d67341881 docs(react-ui): clarify all modalities in attachments prop examples 2026-06-16 09:17:55 -07:00
Jordan Ritter 72520de43d fix(harness): cascade last-bubble whole-bubble-minus-toolbar fallback for tool-only responses 2026-06-16 04:05:49 -07:00
Jordan Ritter f7779f90bf refactor(showcase): probe taxonomy cleanup — drop Smoke, rename E2E (Demo) → UI (Frontend) (#5473)
## Summary

The smoke probe was the same HTTP contract as `/health` on the same
service (200-OK JSON body), so every tick paid two HTTP calls for the
same liveness signal. This PR:

- Drops the `/smoke` GET probe and the `smoke:<slug>` primary
`ProbeResult`. The driver now emits `health:<slug>` as the primary and
`agent:<slug>` via writer side-emit (half the per-tick cost).
- Renames the dashboard's D3 / e2e row label from `E2E (Demo)` to `UI
(Frontend)` (and the short cell badge from `E2E` to `UI`) — the row was
already about "the demo page renders in a browser", not E2E in the
traditional sense.
- Drops the `Smoke` row from the cell drilldown (CellState.smoke field
is retained for back-compat so historical rows still parse).
- Adds an explicit `Health` row to the legend (was implicit).

The driver's registry `kind` stays `"smoke"` so existing YAML configs +
orchestrator family wiring keep routing to this driver — the emission
key (`health:<slug>`) is the taxonomy contract that matters. The
underlying probe key (`e2e:<slug>/<feature>`) is preserved on PocketBase
so historical rows render correctly during the rename window.

## RED proof (BEFORE)

`smoke:<slug>` primary emission in `liveness.ts` (origin/main):

```
18: *   1. `smoke:<slug>`  — the RETURN VALUE of `run()`. The invoker runs
56: * `smoke:<slug>`/`health:<slug>`/`agent:<slug>` triple the dashboard
241:        input.mode === "discovery" ? `smoke:${slug}` : input.key;
```

`E2E (Demo)` / `Smoke` strings in dashboard render code (origin/main):

```
cell-drilldown.tsx:38: * Trip)" (chat+tools round-trip), D3/e2e = "E2E (Demo)" (the demo page loads
cell-drilldown.tsx:49:  { key: "e2e", label: "E2E (Demo)" },
cell-drilldown.tsx:52:  { key: "smoke", label: "Smoke" },
cell-pieces.tsx:382:        name="E2E"
unified-cell.tsx:231:      <TestBadge name="E2E" level={model.d3} />
adaptive-legend.tsx:59:        E2E (Demo): demo page loads and round-trips in a browser
```

## GREEN proof (AFTER)

Primary emission rewritten — driver returns `health:<slug>` and no
`smoke:<slug>` ProbeResult is ever emitted:

```
33: * to `/smoke` and emitted a `smoke:<slug>` primary result, but that probe
230:  // `kind: "smoke"` is the registry/family identifier the orchestrator
236:  kind: "smoke",        // registry-key back-compat; no smoke ProbeResult is emitted
367:  if (input.mode === "discovery") return `health:${slug}`;
368:  if (input.key.startsWith("smoke:")) {
369:    return `health:${input.key.slice("smoke:".length)}`;
```

New labels in dashboard:

```
cell-drilldown.tsx:60:  { key: "e2e", label: "UI (Frontend)" },
cell-pieces.tsx:382:        name="UI"
unified-cell.tsx:231:      <TestBadge name="UI" level={model.d3} />
adaptive-legend.tsx:65:        UI (Frontend): demo page renders in browser (Playwright)
```

Confirmation that the old labels are gone from render code:

```
$ grep -rn 'E2E (Demo)\|"Smoke"' cell-drilldown.tsx cell-pieces.tsx unified-cell.tsx adaptive-legend.tsx
(no matches — labels removed)
```

## Test results

- `pnpm exec vitest run` in `showcase/shell-dashboard`: **63 files, 1089
tests passed, 1 skipped**
- `pnpm exec vitest run` in `showcase/harness`: **129 files, 2760 tests
passed**
- `pnpm exec tsc --noEmit` in both: clean

Liveness driver tests include a regression guard (`never emits a
smoke:<slug> key`) so the smoke contract cannot be re-introduced
silently.

## Test plan

- [x] Dashboard vitest green (1089/1089)
- [x] Harness vitest green (2760/2760)
- [x] tsc --noEmit green in both packages
- [x] Visual cross-check: drilldown renders 6 rows (no Smoke), e2e row
labelled `UI (Frontend)`; cell strip shows `UI` badge in place of `E2E`;
legend shows explicit `Health` row + `UI (Frontend)` D3 line
2026-06-16 01:57:15 -07:00
Jordan Ritter 53f021afe8 refactor(showcase): probe taxonomy cleanup — drop Smoke, E2E (Demo) → UI (Frontend)
The smoke probe was the same HTTP contract as /health on the same
service (200-OK JSON body), so every tick paid two HTTP calls for the
same liveness signal. Drop the /smoke GET + the smoke:<slug> primary
ProbeResult; the driver now emits health:<slug> as the primary and
agent:<slug> via writer side-emit (half the per-tick cost).

The driver's registry kind stays "smoke" so existing YAML configs and
orchestrator family wiring keep routing to this driver — the
emission key (health:<slug>) is the taxonomy contract that matters.

Dashboard:
- Drilldown: D3/e2e row labelled "UI (Frontend)" (was "E2E (Demo)");
  Smoke row dropped (CellState.smoke field retained for back-compat).
- Cell badges: short label "UI" (was "E2E") in cell-pieces + unified-cell.
- Legend: D3 = "UI (Frontend): demo page renders in browser (Playwright)"
  plus an explicit Health row at the top.

Tests updated to assert new labels:
- cell-drilldown.test.tsx: 6 dimensions (no Smoke); UI (Frontend) label.
- cell-drilldown.lazy-signal.test.tsx: drilldown-badge-ui--frontend- testid.
- cell-pieces.test.tsx + .signal-degrade.test.tsx: badge name "UI".
- unified-cell.test.tsx: mock-badge-UI.
- overlay-selector-integration.test.tsx: "UI" in place of "E2E".
- dashboard-color-matrix.test.tsx: badge: "UI" case names.
- liveness.test.ts: two-call contract (health + agent), regression guard
  asserting no smoke:<slug> ever emitted.

Underlying probe key (e2e:<slug>/<feature>) preserved on PocketBase so
historical rows render correctly during the rename window.
2026-06-16 01:48:41 -07:00
Jordan Ritter 665dd5aae6 fix(harness): waitForTurnComplete reads LAST bubble (multi-step agent regression from #5462) (#5472)
## Summary

**URGENT hotfix.** Staging is widely red for D5/D6 multi-step demos.
Root cause confirmed via live Playwright DOM inspection on staging
(langgraph-typescript : beautiful-chat : pie chart prompt).

PR #5462's defect-2 fix uses `bubbleIndex = turnIndex - 1`, which
assumes one assistant bubble per turn. Multi-step agents (LangGraph,
Mastra, CrewAI, llama-index, ag2, etc.) emit 2-3+ bubbles per turn —
tool-call bubble + tool-render bubble + final-text bubble. The
strict-index read lands on an intermediate **tool-call** bubble whose
scoped-text selectors (`.cpk:prose`, `[data-message-content]`, `.prose`,
`p`) are **empty** (tool-call content lives in a sibling card div
outside the scoped cascade). The cascade-pollution guard correctly
returns `null` → the text-stable conjunct never converges →
`reason=text-unstable` timeout across virtually every multi-step demo
(beautiful-chat-*, gen-ui-*, shared-state-*, agentic-chat on most
backends).

## Live RED-GREEN proof (staging)

Standalone Playwright probe against
`showcase-langgraph-typescript-staging.up.railway.app/demos/beautiful-chat`,
prompt = `"pie chart of revenue distribution by category from the sample
sales data"`, captured BOTH the OLD-logic read (`bubbleIndex = turnIndex
- 1 = 0`) AND the NEW-logic read (`bubbleIndex = count - 1 = 2`) in ONE
atomic page evaluate after the turn settled:

```
TOTAL BUBBLE COUNT FOR TURN 1: 3  (true multi-step — tool-call + tool-render + final-text)
FINAL TIER: [data-testid="copilot-assistant-message"]

OLD LOGIC (bubbleIndex = turnIndex - 1 = 0):
  count = 3
  text  = null
  reason = all scoped selectors empty           ← would time out: reason=text-unstable

NEW LOGIC (bubbleIndex = count - 1 = 2):
  count = 3
  text  = "Here is the pie chart showing the revenue distribution by category from the sample sales data."
  textLen = 94                                  ← gate settles cleanly
```

RED ≠ GREEN; new logic returns 94 chars of stable final-text where old
logic returns `null`. The bug and the fix are proven on real staging
DOM.

### Live DOM evidence
LangGraph TS beautiful-chat, prompt = "pie chart of revenue distribution
by category…":
```
count = 3 bubbles in [data-testid="copilot-assistant-message"]
bubble[0]: tool-call "query_data"   — `.cpk:prose` exists but EMPTY (content in sibling <div class="my-1.5">)
bubble[1]: pie-chart card           — `.cpk:prose` exists but EMPTY (content in sibling styled card div)
bubble[2]: final text "Here is the pie chart..." — `.cpk:prose` has 94 chars + toolbar mounted
```

The old pre-#5462 runner read `list[last]` (always the last bubble —
always bubble[2] above). The PR moved to `turnIndex - 1`, which is
bubble[0] (the tool-call) for any multi-step turn → broken.

## Fix

Read the LAST bubble in the matched cascade tier (`count - 1`) instead
of a fixed strict index. Defect-2 protection (don't read a leftover
bubble from a prior turn) is preserved via an explicit **pre-submit
baselineCount sentinel snapshot** in the runner — the gate now requires
`count > baselineCount` rather than `count > turnIndex - 1`. This
correctly rejects stale prior-turn bubbles **without assuming 1 bubble =
1 turn**.

### Changes
- `assistant-message-count.ts`: add `readCascadeStateLast(page)` — same
atomic single-evaluate contract as `readCascadeState`, but resolves the
bubble index internally as `count - 1`.
- `conversation-runner.ts`:
- Snapshot `baselineCount = countAssistantMessages(page)` BEFORE
`sendTurnMessage()` each turn.
- Pass through to `waitForTurnComplete` via new `baselineCount` option
(defaults to `turnIndex - 1` to preserve unit-test fake compatibility).
- `waitForTurnComplete` reads via `readCascadeStateLast` and gates on
`domOk = count > baselineCount`.
  - Cold-start retry re-snapshots `baselineCount` after `page.reload()`.
- Returned `bubbleIndex` in the success ctx = `count - 1` (matches the
bubble whose text settled the gate).
- `dom-missing` reason classifier updated to `countFinal <=
baselineCount`.

## Why this doesn't regress defect-2

The original defect-2 (un-turn-scoped bubble selection) leaked a later
turn's bubble into THIS turn's assertions because
`readLastAssistantText` read `list[last]` GLOBALLY with no per-turn
baseline. The new fix re-introduces the "read last" semantics but adds
the **per-turn pre-submit baseline count snapshot** — the gate only
advances once a NEW bubble has appeared (`count > baselineCount`), so we
cannot read a leftover from a prior turn, and the "last" we return is
the last NEW bubble (which for multi-step turns is the final-text bubble
whose scoped text actually settles).

## Test plan

- [x] **Live RED-GREEN staging probe** (see above) — confirms OLD logic
= null and NEW logic = 94-char final-text on a real 3-bubble multi-step
turn.
- [x] All 72 `conversation-runner.test.ts` +
`assistant-message-count.test.ts` unit tests pass (default
`baselineCount = turnIndex - 1` keeps existing count-progression scripts
working).
- [x] Full harness vitest suite: 2758/2758 green across 129 test files.
- [x] `tsc --noEmit` clean on `showcase/harness`.
- [x] `oxfmt --check` clean on changed files.
- [x] `oxlint` clean on changed files.
- [ ] Staging D5/D6 sweep after merge confirms multi-step demos
(beautiful-chat, gen-ui, shared-state, langgraph/mastra/crewai
agentic-chat) recover from `text-unstable` red.

## Scope

Strictly the two files above. No unrelated changes. No refactors.
2026-06-16 01:12:10 -07:00
Jordan Ritter d145bb1708 fix(harness): waitForTurnComplete reads LAST bubble in matched tier
PR #5462's defect-2 fix assumed 1 bubble per turn (bubbleIndex =
turnIndex - 1). Multi-step agents (LangGraph, Mastra, CrewAI) emit
2-3+ bubbles per turn — tool-call + tool-render + final-text — so
the strict-index read lands on an intermediate tool-call bubble
whose scoped-text selectors (`.cpk:prose`, `[data-message-content]`,
`p`) are EMPTY (its content lives in a sibling card div outside the
scoped cascade). The cascade-pollution guard returns null forever
and the text-stable conjunct times out with `reason=text-unstable`
across virtually every multi-step demo (beautiful-chat-*, gen-ui-*,
shared-state-*, agentic-chat on LangGraph/Mastra/CrewAI/etc.).

Live DOM evidence from staging (langgraph-typescript :
beautiful-chat : pie chart prompt):
  count = 3 bubbles
  bubble[0]: tool-call "query_data" — `.cpk:prose` empty
  bubble[1]: pie chart card — `.cpk:prose` empty
  bubble[2]: final text "Here is the pie chart..." — 90 chars

Fix: read the LAST bubble in the matched cascade tier (count - 1)
instead of a fixed strict index. Defect-2 protection (don't read a
leftover bubble from a prior turn) is preserved via an explicit
pre-submit `baselineCount` sentinel snapshot in the runner; the
gate now requires `count > baselineCount` rather than
`count > turnIndex - 1`, which correctly rejects stale bubbles
without assuming 1 bubble = 1 turn.

Changes:
  - assistant-message-count.ts: add `readCascadeStateLast(page)` —
    same atomic single-evaluate contract as `readCascadeState` but
    resolves the bubble index internally as `count - 1`.
  - conversation-runner.ts: snapshot `baselineCount` before
    sendTurnMessage; pass through to `waitForTurnComplete` via new
    `baselineCount` option (defaults to `turnIndex - 1` for unit-
    test fakes); `waitForTurnComplete` reads via
    `readCascadeStateLast` and gates on `count > baselineCount`.
    Cold-start retry re-snapshots `baselineCount` after page.reload.

Test plan:
  - All 72 conversation-runner + assistant-message-count unit tests
    pass (default baselineCount preserves count-progression scripts).
  - Full harness suite 2758/2758 green, tsc clean, oxfmt/oxlint
    clean on changed files.
2026-06-16 01:02:52 -07:00
Jordan Ritter 48ef506abf fix(showcase): allow harness to boot locally without SHARED_SECRET
PR #5458 (c81b361f1) added a fail-loud gate that refuses harness boot
in any deployable mode (NODE_ENV != "test") without SHARED_SECRET or
SHARED_SECRET_PREV: POST /webhooks/deploy is only registered when
webhookSecrets.length > 0 (src/http/server.ts:119 +
loadWebhookSecrets in src/orchestrator.ts). The local docker-compose
stack inherits NODE_ENV=production from the harness image and does
not (and should not) set SHARED_SECRET, so every local D5/D6 verify
run via bin/showcase test --d5/--d6 was crashing the harness in a
restart loop with FATAL-CONFIG.

Fix: add HARNESS_ALLOW_NO_SECRET=1 (the documented local-dev escape
hatch — explicitly referenced in the FATAL-CONFIG message itself) to
both harness services in showcase/docker-compose.local.yml. Inline
comments explain the rationale and pin the relevant source locations.

Prod impact: NONE. Railway sets SHARED_SECRET explicitly via env on
every harness service, so loadWebhookSecrets sees a real secret,
registers POST /webhooks/deploy, and never reads
HARNESS_ALLOW_NO_SECRET. This change only affects the local
docker-compose stack.

Verified locally: showcase-iso1-harness boots cleanly (Up healthy),
the expected warn-level webhook-auth-bypass log fires
(escapeHatch:true), worker registers, scheduler starts, and a real
d6:langgraph-typescript job claims successfully — confirming the
gate fires only in deployable contexts.
2026-06-16 01:00:22 -07:00
Jordan Ritter f8b14f1d6a fix(react-core): await runAgent in useInterrupt::resolve and propagate to demo-local hooks (#5461)
## Summary

Convert `useInterrupt::resolve` and the demo-local
`useHeadlessInterrupt::resolve` callbacks from fire-and-forget into
`async` + `return await copilotkit.runAgent(...)`, so callers receive a
Promise that settles when the resume run settles.

## Mechanism

Pre-fix: `resolve(...)` called `copilotkit.runAgent({...})` without
`await` and without `return` — the arrow returned `undefined`. The
showcase harness DOM-settle check on the assistant confirmation bubble
timed out because consumers had no handle to sequence against the resume
run's settle.

Post-fix:
- Framework `useInterrupt::resolve` is `async`, `return await`s
`runAgent`, wraps in try/catch + `setPendingEvent(null)` +
`console.error` + rethrow on rejection.
- `onRunFailed` now also `setPendingEvent(null)` — symmetric with
`onRunStartedEvent`, prevents stuck popups on run failure.
- `InterruptHandlerProps.resolve` / `InterruptRenderProps.resolve` typed
`() => Promise<RunAgentResult>` (was `() => void`).
- The same fix-pattern applied to 13 demo-local `useHeadlessInterrupt`
implementations across integrations: ag2, agno, claude-sdk-typescript,
crewai-crews, langgraph-{fastapi,python,typescript}, langroid,
llamaindex, mastra, pydantic-ai, spring-ai, strands.

## Verification

- **Unit:** new tests `resolve returns a Promise that settles when
runAgent settles (RESUME-PATH)` + `resolve rejects when runAgent
rejects, logs the failure, and clears pending (RESUME-PATH-REJECT)`.
Red-then-green on baseline. 22/22 react-core hook tests passing.
- **Backplane (end-to-end):** showcase control plane on langgraph-python
via `./bin/showcase test langgraph-python:gen-ui-interrupt --d5 --direct
--verbose` (GREEN, 70.1s, was RED-RESUME-PATH baseline) and
`:interrupt-headless --d5 --direct --verbose` (GREEN, 7.7s, was
RED-RESUME-PATH baseline). Built Docker artifact + aimock D6 fixture
replay — staging-equivalent path.

## CR

4 review rounds × 7 unbiased agents = 28 agent-runs. 2 Procedure 3
audits (3 promotions in round 1 → Fix C + D; 0 promotions in round 2).
Convergence achieved per agent-confirmed `bucket (a) empty` + Procedure
3 zero `PROMOTE_TO_A`.

## Out of scope (deferred follow-ups)

- **No manifest unquarantine.** A prior version of this branch flipped
`interrupt-headless` and `gen-ui-interrupt` from
`not_supported_features` to `features` across 12 manifests, but Phase 4
backplane validation across 8 PATCHED integrations × 2 demos surfaced
(a) `langgraph-fastapi:gen-ui-interrupt` still RED-RESUME-PATH because
the framework patch doesn't propagate from workspace to the
integration's Docker image without a published `@copilotkit/react-core`
release, and (b) a separate popup-mount RED-RUNTIME-OTHER cluster across
6 integrations that's pre-existing rot unrelated to this PR. Manifest
unquarantine + `@copilotkit/react-core` version bump are deferred to a
follow-up `copilotkit-release` PR after this fix lands.
- **No version bump.** Release follows in a separate PR.
- **Bucket (c) follow-up backlog** (audit verdicts: 11 STAY_IN_C):
  - JSON.parse guarding in showcase demos (pre-existing, all 13)
- Type-design quirks in `InterruptHandlerProps.result` (non-nullable in
type, nullable in runtime)
  - `renderInChat` dynamic toggle leak (pre-existing edge case)
- JSDoc clarity ("symmetric with onRunFailed" wording, `@typeParam
TRenderInChat`, `event.value` "any" → "unknown")
  - StrictMode test coverage gap
- 13× duplicate `useHeadlessInterrupt` → bucket (d): consolidate into a
shared module in a future PR

## Files changed

16 files, +486/-154:
- `packages/react-core/src/v2/hooks/use-interrupt.tsx`
- `packages/react-core/src/v2/hooks/__tests__/use-interrupt.test.tsx`
- `packages/react-core/src/v2/types/interrupt.ts`
- 13×
`showcase/integrations/<integ>/src/app/demos/interrupt-headless/page.tsx`

## Test plan

- [x] react-core unit tests (22/22 passing including new RESUME-PATH +
RESUME-PATH-REJECT regression tests)
- [x] Backplane D5 probe on langgraph-python (gen-ui-interrupt +
interrupt-headless GREEN end-to-end via the showcase control plane)
- [ ] CI green on this PR
2026-06-16 00:08:56 -07:00
Tyler Slaton 431d5baae0 feat(bot-slack): agent-native Slack APIs — assistant pane + native streaming (default-on) (#5447)
## What

Activates Slack's agent-grade APIs as the **default** experience for
`@copilotkit/bot-slack`, with zero config and safe degradation.
Implements the Notion spec *"Slack Agent APIs in bot-slack — assistant
pane + native streaming by default"* (Rev 2).

### Assistant pane ("Agents & AI Apps")
- Opening the pane greets the user + shows tappable prompt chips; each
pane conversation is its own thread (replies stay in-thread, instead of
leaking to a merged flat DM).
- While the agent runs, native composer status
(`assistant.threads.setStatus`: "is thinking…", "is using `tool`…")
replaces placeholder/`🔧` messages.
- Pane threads are auto-titled from the first message.
- Customize via the new `assistant` option; `assistant: false` disables
pane handling. Apps without the Agents toggle behave exactly as before
(pane machinery dormant).

### Native streaming
- Replies stream via `chat.startStream` / `appendStream` / `stopStream`
(raw markdown — real tables / fenced code render natively) wherever a
thread exists.
- Flat DMs and workspaces without the streaming API fall back to the
legacy `chat.update` transport **automatically** — the first
`startStream` failure marks the workspace legacy and replays via legacy.
Opting in can never break a bot. `streaming: "legacy"` forces the old
transport.

### Portable engine surface (`@copilotkit/bot`)
- `bot.onThreadStarted` lifecycle handler + `IncomingThreadStart` sink
event.
- Capability-gated `thread.setSuggestedPrompts` / `thread.setTitle` (the
shipped `postFile` gating pattern).
- Two `SurfaceCapabilities` flags + optional `PlatformAdapter` methods.
All degrade gracefully on surfaces without support.

## Files
- **New:** `bot-slack/src/assistant.ts` (Bolt `Assistant` middleware ⇄
engine sink), `bot-slack/src/native-stream.ts` (`NativeMessageStream`,
same `append/finish` contract as `MessageStream`, legacy fallback).
- **Engine:** `platform-adapter.ts`, `thread.ts`, `create-bot.ts`,
`index.ts` (+ `bot-ui` `Thread` type).
- **Adapter:** `adapter.ts` (options/capabilities/stream
branch/methods), `slack-listener.ts` (one-line no-double-delivery
guard), `event-renderer.ts` (pane status mode + native text transport).
- **Example/docs:** `examples/slack` manifest (`assistant_view` + scope
+ events) & dev-ex, package READMEs/ARCHITECTURE, `slack.mdx`.
- **Hooks:** `chore(hooks)` makes the `lint-fix` script
double-quote-free (it was breaking `sh -c "…"` on Windows/lefthook
2.1.1).

## Tests / verification (CI-runnable)
- New unit tests: `native-stream` (cadence, 12k continuation, fence
re-open, first- **and** continuation-`startStream` fallback),
`assistant` (defaults-before-onThreadStarted ordering, thread-scoped
turn, auto-title), listener no-double-delivery, renderer pane-status
mode, engine capability gating.
- Verified locally: `@copilotkit/bot` 31 tests, `@copilotkit/bot-slack`
199 tests, `bot-ui` tests; `tsc` clean (bot + bot-slack); build clean;
oxlint/oxfmt clean; publint/attw pass.

## ⚠️ E2E not covered here
The §8 end-to-end flows need a **real Slack workspace** with the Agents
toggle (open pane → chips → streamed reply w/ live status; channel
mention → native markdown; kill `startStream` → legacy fallback;
non-agent app → shipped behavior). `examples/slack` is the vehicle.
Three items remain to confirm against a live workspace (spec §7 spikes):
the `isAssistantThread` runtime detection, the exact `assistant_view`
manifest acceptance, and `stopStream` finalize tolerance.

Spec:
https://app.notion.com/p/copilotkit/Slack-Agent-APIs-in-bot-slack-assistant-pane-native-streaming-by-default-spec-37b3aa38185281f5948dc6a665064d04
2026-06-15 22:10:17 -07:00
Jordan Ritter 1ef6a0960b test(harness): update d5-* + d6-all-pills unit tests to new probe + cascade contract
- d5-* unit tests: update page fakes / assertion-call shapes to pass
  the ctx bridge ({bubbleIndex, text}) instead of querying the live
  bubble list
- d6-all-pills.test.ts: update page-fake dispatch + cascade-state
  expectations to match the atomic readCascadeState return shape
  ({count, text} or null) — picks up the cross-tier consistency
  guarantee in the unit layer
2026-06-15 18:37:26 -07:00
Jordan Ritter cd2af0f9fd test(harness): conversation-runner unit coverage — banner debounce, cold-start retry, skipFill, error-banner shapes
Comprehensive unit coverage for:
- preFill + fillAndVerifySend retry behavior
- baseline banner snapshot + differs-from-baseline debounce matrix
  (stable banner = no fail; changed banner = fast-fail after 2 polls)
- cold-start retry cases (first-attempt banner → reload + retry;
  retry-then-banner → translatedErr; retry exhausts deadline)
- skipFill / skipSend short-circuit paths
- ErrorBannerReadResult union shape variants (present/absent/unknown)
2026-06-15 18:37:26 -07:00
Jordan Ritter 3d49d4c38f test(harness): bubble-race integration tests + defect reproductions
- bubble-race-repro.ts: shared driver harness for repro tests
- bubble-race-repro-defect-{1,2,3,4}.test.ts: real-browser RED tests
  that fail without the fix and pass with it
    - Defect 1: fast-replay false-positive settle on count
    - Defect 2: multi-turn flicker via list[last]
    - Defect 3: cascade-tier blindness
    - Defect 4: boot-time baseline staleness / pre-paint placeholder
- bubble-race-mechanisms.test.ts: GREEN tests pinning internal
  contracts (cascade-pollution guard, atomic readCascadeState,
  init-script idempotency)
- wait-for-turn-complete.test.ts: 3-conjunct classification matrix
- probe-contract.test.ts: ctx-bridge shape
- sse-counter.test.ts: counter increment + framenavigated reset +
  handle-cache invalidation
2026-06-15 18:37:26 -07:00
Jordan Ritter cf7efbda95 fix(harness): d6 driver — installPrePaintFromEnv + attachSseInterceptor wiring + helper-based captureDiagnostics
- d6-all-pills.ts: wire installPrePaintFromEnv and
  attachSseInterceptor PRE-goto in both defaultLauncher and pooled
  launcher paths, so first-paint state is deterministic and the SSE
  counter is armed before any page navigation
- captureDiagnostics rewired through the Node-side helper
  (readCascadeState) — single page.evaluate per call, no
  within-snapshot tier drift
- Consume BUBBLE_RACE_MESSAGES override for defect-4 repro flows
2026-06-15 18:37:25 -07:00
Jordan Ritter 38e155e141 refactor(harness): probe-contract ctx bridge — assertions(page, {bubbleIndex, text}) for d5 scripts
- d5-gen-ui-custom, d5-gen-ui-open-advanced, d5-mcp-apps,
  d5-subagents, d5-tool-rendering-default-catchall: wrap assertion
  bodies in ctx bridge so they receive {bubbleIndex, text} for the
  turn under test, replacing turn-leaky list[last] retrieval
- _gen-ui-shared: add readAssistantTextAt(page, bubbleIndex)
  adapter; remove readLastAssistantText (no longer turn-scoped)

This is the read-side counterpart to the atomic readCascadeState
on the helper side: probes ask for the bubble at the runner's
chosen index instead of guessing.
2026-06-15 18:37:25 -07:00
Jordan Ritter 3f3986b00b refactor(harness): waitForTurnComplete 3-conjunct primitive + cold-start retry + banner debounce
- waitForTurnComplete: 3-conjunct settle (SSE counter + DOM-at-index
  + text-quiet for settleMs) replaces count-only false-positive path
- Cold-start retry: single bounded retry on first-attempt banner;
  reload + re-resolve chat input + settleMs/POLL_INTERVAL_MS floor
  honoring shared turn deadline; translatedErr throw on exhaustion
- Banner fast-fail: baseline banner text snapshot +
  differs-from-baseline 2-poll debounce; in-poll BannerVisibleError
  translates to AssistantErroredError
- New error classes: BannerVisibleError, AssistantErroredError
- readErrorBanner returns 3-state union via
  ErrorBannerReadResult discriminated union
- chatInputSelector cascade for input resolution across integrations
2026-06-15 18:36:59 -07:00
Jordan Ritter 473a9539a0 feat(harness): add bubble-race helpers — assistant-message-count, sse-interceptor, init-scripts
- assistant-message-count: 4-tier cascade + atomic readCascadeState
  returning {count, text} from a single page.evaluate (prevents
  within-poll cross-tier inconsistency); null fallback for
  cascade-pollution
- sse-interceptor: __hk_runsFinished CDP counter + page-side fetch
  wrapper; idempotent attach/detach with handle-cache invalidation
  on stop and framenavigated reset (cold-start retry compatible)
- init-scripts: pre-paint placeholder env adapter
  (installPrePaintFromEnv), strip-selector helper, and
  BUBBLE_RACE_MESSAGES override consumer for defect-4 repro
  infrastructure
2026-06-15 18:36:59 -07:00
Jordan Ritter 6fbd66fa83 fix(showcase): mirror useInterrupt RESUME-PATH contract in 13 demo-local hooks
Each integration's interrupt-headless demo defines a local useHeadlessInterrupt
hook around the framework useInterrupt. Slot-2 originally identified 8
quarantined integrations (claude-sdk-typescript, langgraph-{fastapi,python,
typescript}, langroid, pydantic-ai, spring-ai, strands); review-round
follow-ups extended the sweep to llamaindex, mastra, ag2, agno, and
crewai-crews (5 more integrations sharing the same byte-identical hook).

The demo-local resolve() previously fire-and-forgot copilotkit.runAgent(...)
via `void runAgent(...).catch(() => {})`. Mirroring the framework fix:

- Make resolve async, return await copilotkit.runAgent(...).
- Use a pendingRef so resolve has stable identity (drop pending from
  useMemo deps).
- Type signature: resolve: (response: unknown) => Promise<unknown>.
- Wrap in try/catch + setPending(null) + console.error + rethrow,
  symmetric with the framework hook.
- onRunFailed also setPending(null).

13 integrations patched byte-identically.
2026-06-15 17:11:40 -07:00
Jordan Ritter c81b361f15 fix(showcase/harness): register /webhooks/deploy on CP path + fail-loud on missing SHARED_SECRET (#5458)
## Problem

The \`Showcase: Verify Deploy\` workflow runs \`notify-harness\` after
every merge to \`main\` and POSTs an HMAC-signed payload to
\`\${SHOWCASE_HARNESS_URL}/webhooks/deploy\`. That POST has been
returning **404** on every main deploy since at least 2026-06-12 (5+
consecutive failures), so the harness dashboard has not reflected any
recent deploys.

**Root cause:**

1. \`POST /webhooks/deploy\` is only registered when
\`webhookSecrets.length > 0\` — gate at
\`showcase/harness/src/http/server.ts:119\`.
2. The CP \`buildServer\` call in \`runControlPlane\`
(\`showcase/harness/src/orchestrator.ts\` ~line 2961 pre-fix)
**omitted** both \`webhookSecrets\` and \`metrics\`. The public Railway
host running the CP role never mounted the route — every notify-harness
POST returned 404.
3. The FATAL-CONFIG guard on missing \`SHARED_SECRET\` (pre-fix
\`orchestrator.ts:782-790\`) only fired when \`NODE_ENV ===
"production"\`. Any deploy whose NODE_ENV was unset, set to something
other than the literal \`"production"\`, or set after the check,
silently booted with \`webhookSecrets=[]\` and no fatal error.

## Fix

**Part 1: Register the route on the CP path.**
- Lifted the env→secrets loader into a new exported helper
\`loadWebhookSecrets()\` (orchestrator.ts ~175-209) so both boot paths
consume the same predicate.
- Worker \`boot()\` (~line 836) calls it; CP \`runControlPlane\` (~line
2421) now also calls it.
- The CP \`buildServer\` call (~line 3047) now passes \`webhookSecrets\`
and \`metrics\` alongside the existing \`bus\`/\`probes\`/\`fleetRuns\`
wiring — same shape as the worker call.

**Part 2: Tighten the FATAL-CONFIG predicate.**
- New predicate (inside \`loadWebhookSecrets\`): throw unless EITHER a
secret is set OR \`NODE_ENV === "test"\` OR \`HARNESS_ALLOW_NO_SECRET
=== "1"\` (narrow escape hatch for local dev).
- Thrown error message names the env-vars, the gate location
(\`src/http/server.ts:119\`), the silent-404 symptom, and the escape
hatches — so an operator reading the deploy log can recover without
spelunking.

## Test coverage

Four new red-green tests in \`orchestrator.test.ts\` under \`B2:
/webhooks/deploy registered on CP + fail-loud on missing
SHARED_SECRET\`:

- **Test A** — CP boot registers \`POST /webhooks/deploy\` when
\`SHARED_SECRET\` is set. Probe with no HMAC headers: pre-fix returned
404, post-fix returns 401 (route mounted, HMAC reject path fires).
RED→GREEN.
- **Test B** — Worker boot still registers the route (regression guard).
Pre-fix and post-fix both 401.
- **Test C** — FATAL-CONFIG fires when \`SHARED_SECRET\` unset AND
\`NODE_ENV=development\`. Pre-fix resolved silently, post-fix rejects
with \`/FATAL-CONFIG.*SHARED_SECRET/\`. RED→GREEN.
- **Test D** — Boot succeeds when \`SHARED_SECRET\` unset AND
\`NODE_ENV=test\` (escape hatch). Pre-fix and post-fix both green.

Existing R5-G4 D5 test (orchestrator.test.ts:1592) updated to match the
new error message via \`/FATAL-CONFIG.*SHARED_SECRET/\` — same throw,
broader match.

## Verification

- Full harness suite: **2719/2719 green** across 128 files
- \`tsc --noEmit\`: clean
- RED capture pre-fix:
  - Test A: \`AssertionError: expected 404 to be 401\`
- Test C: \`AssertionError: promise resolved "{ port: ..., bus: { ... },
... }" instead of rejecting\`

## Deploy note

Requires a CP-role redeploy on the public Railway host for the new route
registration to take effect — once redeployed, the next
\`notify-harness\` POST will land on the harness dashboard.

## Test plan

- [x] B2 Tests A-D pass post-fix (red→green captured)
- [x] Full harness vitest suite green (2719/2719)
- [x] \`tsc --noEmit\` clean
- [ ] Post-deploy: confirm next \`Showcase: Verify Deploy\` run shows a
2xx notify-harness POST (not 404)
- [ ] Post-deploy: confirm harness dashboard reflects the next deploy
2026-06-15 16:44:07 -07:00
Jordan Ritter 3ce3f5394d fix(showcase/harness): hoist OPS_TRIGGER_TOKEN fail-loud + emit on subscribeDeployResults sync throw (R3 cleanups)
R3-F1 (OPS_TRIGGER_TOKEN hoist):
Extract loadOpsTriggerToken() and hoist its invocation to the top of both
boot() and runControlPlane(), alongside loadWebhookSecrets() and
loadPocketbaseUrl(). Pre-fix the empty/whitespace check fired AFTER pb /
bus / scheduler / writer / S3 uploader allocations, so a typo'd
`OPS_TRIGGER_TOKEN=` allocated expensive resources before throwing. Now
all three fail-loud config predicates fire at the top, before any
allocation needing teardown. Behaviour is unchanged for valid tokens
(trimmed via R3-A.5 contract) and for the unset case (router omitted with
info log).

R3-F2 (sync-throw also emits deploy.writer.failed):
subscribeDeployResults() pre-fix only emitted `deploy.writer.failed` on
writer.write() promise rejection — a synchronous throw inside
deployEventToProbeResult() (malformed event, type drift) bypassed the
catch and we lost both the log AND the bus emit. Wrap the sync mapping
in try/catch and mirror the rejection path so alert rules / metrics
subscribers observe sync throws the same way they observe async write
failures.

Tests:
- 5 unit tests for loadOpsTriggerToken: undefined / empty-string fail-loud
  / whitespace-only fail-loud / R3-A.5 trim contract / verbatim value.
- 1 test for the sync-throw path: vi.doMock deployEventToProbeResult to
  throw, assert bus emits deploy.writer.failed with the err message AND
  writer.write was NOT called. Red-green verified locally (test fails
  without the try/catch).

Suite: 2731 passing (+6 new). Typecheck clean.
2026-06-15 16:37:05 -07:00
Jordan Ritter d5bdf2c9fd fix(showcase/harness): track CP deploy.result unsubscribe + emit on write failure (R2 cleanups) 2026-06-15 16:23:29 -07:00
Jordan Ritter df97ae8d2a fix(showcase/aimock): align catchall fixture shape for 6 remaining D5 reds
PR #5459 added userMessage matchers ("forecast for Tokyo" + "current price
of AAPL") on the catchall fixtures, which flipped most integrations from
red to green. Six cells stayed red on staging because the FIXTURE SHAPE
itself was broken on the matched fixtures, not just the userMessage key.

Failing cells (all custom-catchall except strands default):
  d5:agno/tool-rendering-custom-catchall
  d5:llamaindex/tool-rendering-custom-catchall
  d5:langroid/tool-rendering-custom-catchall
  d5:claude-sdk-python/tool-rendering-custom-catchall
  d5:strands/tool-rendering-custom-catchall
  d5:strands/tool-rendering-default-catchall

PocketBase confirms the failure mode: turn 1 (Tokyo) completes; turn 2
(AAPL) times out at 30s or renders the wrong content. E.g. agno:

  errorDesc: "timeout: assistant did not respond within 30000ms"
  failure_turn: 2
  turns_completed: 1 / 2

Root cause: the failing tool-emit fixtures used `hasToolResult: false`
(or `toolName: "<tool>"`) as their gate. aimock's hasToolResult check is
`messages.some(m => m.role === 'tool')` over the WHOLE thread — so once
turn 1's Tokyo tool result lands in the conversation, hasToolResult is
permanently true and `hasToolResult:false` can never match turn 2 → no
fixture → 30s timeout. `toolName:get_stock_price` likewise fails when an
integration backend doesn't forward the tool definition on turn 2.

The same trap is documented in
showcase/aimock/d6/langgraph-python/tool-rendering.json:

  "_comment": "Gated on toolName:get_stock_price rather than
   hasToolResult:false. The D5 tool-rendering-custom-catchall probe runs
   'weather in Tokyo' first, which leaves a get_weather tool result in
   the thread; the aimock router implements hasToolResult as
   messages.some(m=>m.role==='tool'), so hasToolResult is permanently
   true on the AAPL turn and a hasToolResult:false gate could never
   match (→ no_fixture_match → 503 → 30s timeout)."

Fix: align all 6 files to the canonical pattern used by mastra/spring-
ai/built-in-agent on this probe:

  1. toolCallId-keyed narration fixture FIRST
  2. tool-emit fixture SECOND with ONLY `userMessage` + `context`
     (no hasToolResult / toolName gate)

The toolCallId narration uses aimock's
`messages[last].role === 'tool' && tool_call_id === ...` check, so it
correctly wins on iteration 2 (post-tool-result) without being affected
by older turns' tool results. The tool-emit fixture matches turn 1 (last
message is user) and re-emits only when the narration above hasn't
matched.

Strands' two cells additionally needed REORDERING — they had the tool-
emit fixture before the toolCallId fixture, defeating first-match-wins.

Strands' staging backend was also returning 502 during testing; once it
recovers, the corrected fixtures should let the probe pass. The fixture
changes are necessary but may not be sufficient for strands if backend
remains down.

No probe-side, harness, or backend changes — pure fixture-content
alignment. Six fixture files modified; line totals: -87 / +68.
2026-06-15 16:09:04 -07:00
Jordan Ritter 16a59a45c7 chore(showcase/harness): warn on webhook escape hatch + vi.stubEnv in B2 tests
CB-1 (Slot 2 #22): convert B2 deploy-webhook tests to vi.stubEnv with
`vi.unstubAllEnvs()` in afterEach instead of manual process.env mutation
+ try/finally restoration. Test runner now guarantees restoration and the
test bodies are shorter / less error-prone (CR Slot 2 #22).

CB-2 (Slot 2 #28): when `loadWebhookSecrets`' escape hatch fires with a
real-looking NODE_ENV (anything except "test"), log at `warn` instead of
`info` so a production typo (NODE_ENV=staging + HARNESS_ALLOW_NO_SECRET=1)
is visible in dashboards / log alerting. Pure local-dev (NODE_ENV=test)
stays at info level so a normal unit-test boot doesn't spam warnings.

CB-3 (Slot 4 #17): clarify `loadWebhookSecrets`' docstring + bypass-log
message — "set" is ambiguous (empty string would qualify pre-fix); use
"non-empty" to match the actual predicate.
2026-06-15 15:25:42 -07:00
Jordan Ritter f42180f7bf fix(showcase/harness): hoist fail-loud config checks + symmetric POCKETBASE_URL predicate
R1-F2 (bucket b, defensive ordering): hoist fail-loud config validation
to the TOP of boot() and runControlPlane() — BEFORE any pb client, queue,
bus, scheduler, writer, S3 uploader, fleet-health, or aggregator
allocations. Pre-fix `loadWebhookSecrets` lived AFTER the entire
scheduler+writer (and CP queue+aggregator) setup, so a misconfigured boot
allocated expensive resources and mounted file watchers before throwing.

R1-F3 (bucket b, predicate symmetry): broaden the POCKETBASE_URL fail-loud
predicate to match SHARED_SECRET semantics. Pre-fix the POCKETBASE_URL
guard only fired on `NODE_ENV === "production"`, while `loadWebhookSecrets`
fired unless NODE_ENV=test or an explicit escape hatch — so staging /
unset / "development" deploys silently bound to http://localhost:8090.

Extracted `loadPocketbaseUrl(logger)` with the same test-or-escape-hatch
predicate as `loadWebhookSecrets`. A new `HARNESS_ALLOW_NO_PB_URL=1` env
flag mirrors `HARNESS_ALLOW_NO_SECRET=1` for local dev. Worker
`boot()` and the CP's `resolveFleetPbConfig` both call the helper.

Tests:
  - HF13-A2 production-only assertion broadened (SHARED_SECRET set so the
    test specifically exercises the POCKETBASE_URL guard).
  - New R1-F3 tests cover NODE_ENV=development without POCKETBASE_URL
    (throws) and the HARNESS_ALLOW_NO_PB_URL=1 escape hatch (succeeds).
2026-06-15 15:25:11 -07:00
Jordan Ritter fc828bb08c fix(showcase/harness): subscribe deploy.result in CP boot to deliver dashboard event
R1-F1 (bucket a, load-bearing): the CP boot path now subscribes to
`deploy.result` events through its status writer. Pre-fix, B2 mounted POST
/webhooks/deploy on the CP role but only the worker boot path had a
`bus.on("deploy.result", ...)` listener — signed POSTs to the CP host
returned 202, the bus event fired with no subscriber, and the deploy-overall
dashboard row never landed.

Extracted the handler into a shared `subscribeDeployResults(bus, writer)`
helper exported from orchestrator.ts so both boot paths share the IDENTICAL
logic. Worker boot keeps its inline pattern (busUnsubs.push) so teardown is
unchanged; CP keeps the subscription alive for the lifetime of the bus.

Red-green tests in src/orchestrator.test.ts under "R1-F1" assert that
emitting deploy.result on the returned handle's bus drives one write keyed
"deploy:overall" through a stubbed status writer — once for runControlPlane
(the new path) and once for boot() (regression guard for the helper
extraction).
2026-06-15 15:23:24 -07:00
Jordan Ritter 07e4998798 fix(showcase/aimock): add forecast-Tokyo + AAPL catchall fixtures to flip remaining D5 reds
The catchall userMessage rename (#5453) only renamed STALE strings to 'forecast for Tokyo'. Integrations whose catchall fixtures used PILL-ALIGNED userMessages ('What's the weather in San Francisco?', 'Find flights from SFO to JFK.', 'Chain a few tools in this single turn', etc.) had no 'forecast for Tokyo' matcher to begin with — so the D5 catchall probe still fell through to live LLM and the cells stayed red on staging.

This PR ADDS the canonical 'forecast for Tokyo' (default+custom catchall) and 'What's the current price of AAPL?' (custom catchall only) emit+narrate fixture pairs to each integration that was missing them. Existing pill-aligned fixtures are preserved (pure prepend at the start of the fixtures array; first-match-wins means new fixtures match the D5 probe inputs without colliding with existing pill prompts).

Integrations touched:
- langgraph-typescript: both default + custom
- langgraph-fastapi: both default + custom
- google-adk: custom only (default already green)
- ms-agent-dotnet: custom only (default already green)

After merge, fleet-cp's e2e-deep probe will re-run within ≤6h staleness window and flip cells GREEN.
2026-06-15 15:00:53 -07:00
github-actions[bot] bfd1f12b00 style: auto-fix formatting 2026-06-15 21:56:46 +00:00
Jordan Ritter 2939696415 fix(showcase/harness): register /webhooks/deploy on CP path + fail-loud on missing SHARED_SECRET
The 'Showcase: Verify Deploy' workflow's notify-harness step has been
POSTing to /webhooks/deploy and getting 404 on every main deploy since
at least 2026-06-12 (5+ consecutive failures). Root cause:

  1. POST /webhooks/deploy is only registered when
     webhookSecrets.length > 0 (gate at src/http/server.ts:119).
  2. The CP buildServer call in runControlPlane omitted BOTH
     webhookSecrets and metrics — so the public Railway host running
     the CP role never mounted the route. The worker boot path had
     them, but it isn't the host receiving notify-harness POSTs.
  3. The FATAL-CONFIG guard on missing SHARED_SECRET was gated on
     NODE_ENV === 'production' — any deploy with NODE_ENV unset or set
     to something other than the literal 'production' silently shipped
     with webhookSecrets=[] and no fatal error.

Part 1 — wire webhookSecrets + metrics into the CP buildServer call.
Lifted the env→secrets loader into a new exported helper
'loadWebhookSecrets()' (~orchestrator.ts:175-209) so BOTH boot paths
consume the same predicate. The worker 'boot()' call (~line 836) and
the CP 'runControlPlane' call (~line 2421) both invoke it, and the CP
buildServer call (~line 3047) now passes webhookSecrets+metrics
alongside the existing bus/probes/fleetRuns wiring.

Part 2 — tighten the FATAL-CONFIG predicate.
The new predicate throws unless EITHER a secret is set OR NODE_ENV ===
'test' OR HARNESS_ALLOW_NO_SECRET === '1' (narrow escape hatch for
local dev). The thrown error message names the env-vars, the gate
location ('src/http/server.ts:119'), the silent-404 symptom, and the
escape hatches, so an operator reading the deploy log can recover
without spelunking.

Updated the existing R5-G4 D5 regex (orchestrator.test.ts:1592) since
the error message changed; covered by the new B2 Tests A/C/D below.

Tests: 4 new red-green tests in 'B2: /webhooks/deploy registered on CP
+ fail-loud on missing SHARED_SECRET':
  - Test A (CP /webhooks/deploy registration): RED 404 → GREEN 401.
  - Test B (worker /webhooks/deploy regression): GREEN 401.
  - Test C (FATAL-CONFIG when NODE_ENV=development): RED resolves →
    GREEN rejects with /FATAL-CONFIG.*SHARED_SECRET/.
  - Test D (NODE_ENV=test escape hatch): GREEN, no throw.

Full harness suite: 2719/2719 green. typecheck: clean.

Requires a CP-role redeploy on the public Railway host for the new
route registration to take effect.
2026-06-15 14:55:21 -07:00
Jordan Ritter fb3d5b3547 test(showcase/harness): cover CP webhook registration + fail-loud on missing SHARED_SECRET
Adds four red-green tests for the notify-harness 404 follow-up:

  A. CP boot registers POST /webhooks/deploy when SHARED_SECRET is set
     — RED pre-fix (404, route not mounted on CP); GREEN post-fix (401,
     the route's own HMAC reject path fires on an unsigned POST).
  B. Worker boot still registers POST /webhooks/deploy (regression
     guard) — already green; pins the unchanged behavior.
  C. boot throws FATAL-CONFIG when SHARED_SECRET unset AND NODE_ENV is
     non-test — RED pre-fix (current guard fires only on
     NODE_ENV='production'); GREEN post-fix once the predicate is
     tightened.
  D. boot succeeds when SHARED_SECRET unset AND NODE_ENV='test'
     (test-mode escape hatch) — already green; pins the escape hatch.

Red capture (pre-fix run):
  - Test A: AssertionError: expected 404 to be 401
  - Test C: AssertionError: promise resolved instead of rejecting
2026-06-15 14:52:31 -07:00