mirror of
https://github.com/CopilotKit/CopilotKit.git
synced 2026-09-14 16:26:20 +08:00
v1.60.2
11448 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Mike Ryan
|
237a176fbf |
chore: release monorepo v1.60.2 (#5517)
## Release monorepo v1.60.2 **Scope:** `monorepo` | **Bump:** `patch` --- ### How this release process works 1. **This PR was created automatically** by the "release / create-pr" workflow. It bumped the `monorepo` packages to `1.60.2` and generated AI-enhanced release notes. 2. **CI runs on this PR** — the full test suite (unit tests, lint, type checks, build) must pass before merging. This is the review gate. 3. **Review the release notes** in `release-notes.md` in this PR. If a Notion draft was created, you can edit the release notes there before merging. 4. **When this PR is merged**, the `release / publish` workflow automatically: - Builds all packages - Publishes the `monorepo` packages to npm at version `1.60.2` - Creates git tag `monorepo/v1.60.2` - Creates a GitHub Release with the final release notes ### Before merging - [ ] CI is green (tests, lint, types, build) - [ ] Version bumps look correct - [ ] Release notes are accurate (edit in Notion if a draft was created) --- > **Do not merge until CI is fully green.** The full test suite runs automatically on this PR.v1.60.2 |
||
|
Sam Julien
|
5345f8f8fd |
docs: clarify shell-docs hybrid authoring (#5519)
## Summary - document the shell-docs hybrid docs architecture and `docs_mode` meanings - add a shell-docs README with npm local dev, validation commands, and authoring recipes - keep retired top-level `docs/` guidance intact while making shell-docs the clear source of truth ## Verification - `git diff --check -- .claude/docs/documentation.md showcase/shell-docs/README.md` - pre-commit: `nx run-many -t test --projects=packages/**` passed after rerun; Nx flagged `@copilotkit/react-core:test` as flaky from an earlier timeout - pre-commit: `nx run-many -t publint,attw --projects=packages/**` passed ## Notes - Archive refs were created before this branch: `archive/docs-save-do-not-prune` and `archive/docs-retired-2026-06-17`. - This PR intentionally does not delete the retired top-level `docs/` tree; that should be a follow-up cleanup PR. |
||
|
Sam Julien
|
2d3bafd61c | docs: clarify shell-docs hybrid authoring | ||
|
Tyler Slaton
|
506997f4f9 |
showcase(docs): update premium features to be enterprise (#5511)
Updating the premium features section to instead be "enterprise" with some reworked documentation pages.archive/docs-retired-2026-06-17 |
||
|
davidmckayv
|
692e52244c | chore: release monorepo v1.60.2 | ||
|
Tyler Slaton
|
8ec7d0d4e5 | docs(shell-docs): restore enterprise intelligence product name | ||
|
Jordan Ritter
|
2842e39f8e |
fix(showcase/harness): retry+cached-catalog producer enumerate + 3-tick family-silence threshold (#5515)
## Summary
Harden the showcase harness producer against transient Railway-GQL 429 /
Cloudflare-WAF flaps. Today's incident: a ~25-min Cloudflare WAF
burst-block on `backboard.railway.com/graphql/v2` caused the producer's
catalog-enumerate to hard-fail every cron tick, zeroing out D4/D5/D6
writes and turning the entire staging dashboard red within one tick
window.
Three discrete behavior changes, one PR:
1. **Retry with exponential backoff in `source.enumerate`** — three
retries at 1s/4s/16s on HTTP 429, 5xx, Cloudflare 1015/1020/1022
markers, or transport-level errors. Does NOT retry on
`DiscoverySourceAuthError`, non-429 4xx, or schema errors
(operator-actionable, fail loud). Lives in
`showcase/harness/src/fleet/control-plane/catalog-enumerator.ts` (the
seam every family enumerator passes through).
2. **Cached-catalog fallback** — per-enumerator in-memory cache of the
last successful `services[]`. On persistent failure (all retries
exhausted) AND a cache present, the wrapper logs
`fleet.producer.enumerate-failed-using-cache` (warn, with `services`
count, `ageMs`, and `reason`) and returns the cached catalog. With NO
cache (fresh-boot first enumerate fails), the wrapper re-throws so the
producer's `enumerate-failed` short-circuit still runs — without a
catalog there's nothing to enqueue.
3. **3-tick family-silence threshold** —
`SILENCE_CONSECUTIVE_TICK_THRESHOLD = 3` layered ON TOP of the existing
`3 × period` elapsed-time gate. The silence alert now requires BOTH:
`now - lastSuccessAt > 3 × period` AND three consecutive evaluation
cycles observed silent. A single bad tick on a stale `lastSuccessAt` can
no longer page every family at once.
## Red-Green proof
**RED on main (`5a62acbf`)** — observed BEFORE the fix. The RED file
asserts the BUG (`calls === 1` after a 429 throw; `posts.length === 1`
after a single silent tick):
```
> vitest run src/fleet/control-plane/red-baseline-railway-gql-resilience.test.ts
RUN v3.2.4 .../showcase/harness
✓ src/fleet/control-plane/red-baseline-railway-gql-resilience.test.ts (2 tests) 3ms
✓ [BUG] one 429 + Cloudflare 1015 from the source aborts the whole enumerate (no retry)
✓ [BUG] silence alert posts on the FIRST silent evaluation tick (no consecutive-tick threshold)
Test Files 1 passed (1)
Tests 2 passed (2)
```
**GREEN on fix branch** — observed AFTER the fix (assertions inverted to
the fixed behavior; `calls === 4` after retries; `posts === []` until
the third silent tick):
```
> vitest run src/fleet/control-plane/red-baseline-railway-gql-resilience.test.ts src/fleet/control-plane/catalog-enumerator.test.ts src/fleet/control-plane/family-silence-monitor.test.ts
RUN v3.2.4 .../showcase/harness
✓ src/fleet/control-plane/red-baseline-railway-gql-resilience.test.ts (2 tests) 4ms
✓ src/fleet/control-plane/family-silence-monitor.test.ts (19 tests) 8ms
✓ src/fleet/control-plane/catalog-enumerator.test.ts (29 tests) 6ms
Test Files 3 passed (3)
Tests 50 passed (50)
```
Full harness suite — **131 files / 2812 tests pass** (`pnpm -F
@copilotkit/showcase-harness test`).
## Test plan
- [x] RED proof captured on `main` (single-attempt enumerate;
single-tick silence alert)
- [x] GREEN proof on this branch (retry to 4 calls; cached fallback;
3-tick threshold)
- [x] `pnpm -F @copilotkit/showcase-harness test` — 2812 passed
- [x] `pnpm -F @copilotkit/showcase-harness typecheck` — clean
- [x] `pnpm -F @copilotkit/showcase-harness build` — clean
- [x] `oxfmt --write` applied; `oxlint` clean
- [ ] CI green
- [ ] Staging redeploy verified via Railway CLI + Playwright dashboard
snapshot
## Files touched
- `showcase/harness/src/fleet/control-plane/catalog-enumerator.ts`
(+296, -3): retry+cache wrapper, exports `ENUMERATE_RETRY_BACKOFF_MS` +
`isRetryableEnumerateError` + `SleepFn`
- `showcase/harness/src/fleet/control-plane/catalog-enumerator.test.ts`
(+295): retry/cache/auth-not-retried/backoff-SSOT tests
- `showcase/harness/src/fleet/control-plane/family-silence-monitor.ts`
(+61): `SILENCE_CONSECUTIVE_TICK_THRESHOLD = 3`, per-family counter,
counter reset on healthy
-
`showcase/harness/src/fleet/control-plane/family-silence-monitor.test.ts`
(+218, -39): 3-tick threshold + counter-reset gate tests; updated
existing tests to advance through 3 silent ticks
-
`showcase/harness/src/fleet/control-plane/red-baseline-railway-gql-resilience.test.ts`
(new, +212): the literal RED→GREEN gate
## Operational notes
- No new env vars, feature flags, or backward-compat shims (per scope
directive).
- The cached-catalog warn surfaces in observability via
`fleet.producer.enumerate-failed-using-cache` (services count, ageMs,
reason).
- `SLACK_WEBHOOK_OSS_ALERTS` is intentionally unset (user config); not
touched.
Pre-existing repo-wide lefthook failures (`@copilotkit/core`,
`@copilotkit/runtime`, `@copilotkit/shared` etc.) reproduce on `main`
without my changes and are unrelated to harness code; harness-scoped
quality gates all passed before commit.
|
||
|
Jordan Ritter
|
5f00cb9771 |
ci(release): one-click canary publish orchestrator + release-pipeline lint guards (#5370)
## Summary Ports [ag-ui-protocol/ag-ui#1914](https://github.com/ag-ui-protocol/ag-ui/pull/1914) to CopilotKit — plus the two supporting guard files ag-ui already had: - **`.github/workflows/canary.yml`** — discoverable **`canary / publish`** `workflow_dispatch` orchestrator. Any maintainer can publish a prerelease of the branch they're on straight from the Actions tab. It is a thin orchestrator — it does **not** publish to npm itself: 1. Guards against `main` and non-branch refs. 2. Mints the devops-bot App token (app-id `1108748`, `DEVOPS_BOT_PRIVATE_KEY`) with scoped `contents:write` + `actions:write`. 3. Mirrors the dispatched ref to a unique `canary/<slug>-<run_id>-<attempt>` branch via the GitHub API (no checkout). 4. Dispatches **`publish-release.yml --ref canary/<slug> -f mode=prerelease …`**, locates the run, and waits (`gh run watch --exit-status` + explicit conclusion check). 5. Deletes the canary ref — status-gated (never yanks the ref under a still-running delegated run) with a fresh cleanup token (90-min job ceiling exceeds the 1h App-token TTL). - **`scripts/release/verify-release-scope-dropdowns.sh`** — drift guard: the hand-maintained `scope` dropdowns in `publish-release.yml` / `stable-release.yml` / `canary.yml` must exactly match `release.config.json`'s `.scopes` keys. Parsers fail loud and distinct on structural changes instead of silently passing. - **`.github/workflows/lint-release-workflows.yml`** — actionlint + shellcheck + the dropdown-sync job over the release pipelines. ### Why a separate orchestrator (and not a flag in publish-release.yml) - A GitHub Environment's deployment-branch policy is evaluated against the ref a run is **triggered on** — not branches created mid-run. The orchestrator exists to get the publish run *onto* a `canary/*` ref. - `publish-release.yml` holds the **single npm OIDC trusted-publisher binding**; a second publishing entry point would break OIDC for every `@copilotkit/*` package. The orchestrator never touches npm. - The cross-workflow dispatch uses the **App token, not `GITHUB_TOKEN`** — `GITHUB_TOKEN`-authenticated events never start new workflow runs. **Note:** the `npm` environment currently has *no* deployment-branch policy, so the orchestrator is a convenience wrapper today. Tightening the policy to `main` + `canary/*` + `release/publish/*` (matching ag-ui's security posture) is being applied as repo configuration alongside this PR — requires admin. This PR includes the prerequisite: `publish-commit.yml` (pkg-pr-new) is removed from the `npm` environment, since it runs on every PR and would be blocked by the policy (it publishes to pkg.pr.new, not npm, and uses no environment secrets). ## Testing done - Drift guard: positive run against all three real workflows; negative tests (scope removed → drift FAIL with diff; bogus scope → FAIL; `case "${SCOPE}"` quoting refactor → loud parser-degradation FAIL; whole case block deleted → loud zero-block FAIL; quoted arm `"angular")` → accepted; blank/comment lines inside `options:` → still parsed; prose comments mentioning case/SCOPE/in → no false positive). - `shellcheck` clean at all severities; `bash -n` on every workflow `run:` block; YAML parses. - 3 rounds of 7-agent code review converged to zero load-bearing findings. ## ⚠️ Still to verify before first real use - [ ] devops-bot App (id 1108748) has **Actions: write** — required for the in-workflow `gh workflow run`. Safe first test: dispatch once with `dry_run=true`. - [ ] First `dry_run=false` run clears the `npm` environment end-to-end via the App token once the deployment-branch policy is tightened. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Post-merge follow-ups (maintainer action required) These need repo **admin** rights and must happen **in this order**: 1. **Merge this PR first.** `main`'s current `publish-commit.yml` (pkg-pr-new) still sits in the `npm` environment and runs on every PR touching `packages/**` — applying the branch policy before this PR lands would block every snapshot publish. This PR removes that environment association. 2. **Tighten the `npm` environment's deployment-branch policy** to `main` + `canary/*` + `release/publish/*` (matching ag-ui). With an admin-scoped token: ```bash gh api --method PUT repos/CopilotKit/CopilotKit/environments/npm \ -F "deployment_branch_policy[protected_branches]=false" \ -F "deployment_branch_policy[custom_branch_policies]=true" gh api --method POST repos/CopilotKit/CopilotKit/environments/npm/deployment-branch-policies -f name="main" -f type=branch gh api --method POST repos/CopilotKit/CopilotKit/environments/npm/deployment-branch-policies -f name="canary/*" -f type=branch gh api --method POST repos/CopilotKit/CopilotKit/environments/npm/deployment-branch-policies -f name="release/publish/*" -f type=branch ``` Or via UI: Settings → Environments → npm → Deployment branches and tags → "Selected branches and tags" → add the three patterns above. Why these three: `main` covers stable `workflow_dispatch` retries and `stable-release.yml`; `release/publish/*` covers the merged-release-PR runs (the run's head branch is the release PR branch); `canary/*` covers the orchestrator's delegated prerelease runs. After this, direct `mode=prerelease` dispatches from arbitrary feature branches stop working — the `canary / publish` orchestrator becomes the one-click path (by design). 3. **Verify the devops-bot App (id `1108748`) has `Actions: write`** (org/App settings). The orchestrator's `gh workflow run` dispatch 403s without it. Safe end-to-end test, after step 2: Actions tab → **canary / publish** → pick any feature branch, any scope, **`dry_run=true`** → confirm the delegated `release / publish` run is created, watched, and the `canary/*` ref is deleted afterward. 4. **First real canary** (`dry_run=false`) confirms the npm OIDC publish clears the environment gate end-to-end on a `canary/*` ref. |
||
|
Jordan Ritter
|
c8a9053bee |
test(showcase/harness): red-green gate for Railway-GQL resilience + 3-tick silence threshold
Adds an integration-style test file that pins the BOTH layers of the
2026-06-17 Cloudflare-WAF-burst incident fix together:
- the enumerator retries 3× on a 429+Cloudflare-1015 burst before
bubbling (the original bug let one 429 abort the whole enumerate);
- the silence monitor requires THREE consecutive silent evaluation
cycles before posting an alert (the original bug fired on tick #1).
These assertions were the LITERAL red proof on `main`:
- on main both `it` blocks PASSED while asserting the buggy behavior
(calls === 1, posts.length === 1 after a single silent tick),
- on this branch the same gates re-pin the fixed behavior (calls === 4
after retries; posts === [] until the third silent tick).
Run-time output captured for the PR body confirms the inversion.
|
||
|
Jordan Ritter
|
94f6d87f54 |
fix(showcase/harness): require 3 consecutive silent ticks before family-silence alert
Layer a per-family consecutive-silent-tick counter ON TOP of the existing
3×period elapsed-time gate (`SILENCE_PERIOD_MULTIPLIER`). The silence
alert now requires BOTH:
- `now - lastSuccessAt > 3 × period` (existing elapsed-time gate), AND
- `SILENCE_CONSECUTIVE_TICK_THRESHOLD = 3` consecutive evaluation cycles
observed silent (new — the counter resets on any successful evaluation).
Without the new gate a single bad cron tick on a family whose
`lastSuccessAt` was already stale (e.g. after a long quiet window or a
deploy gap) tripped the alert immediately — the failure mode the
2026-06-17 Cloudflare-WAF-burst incident exposed where one ~25 min flap
on backboard.railway.com/graphql/v2 paged every family at once.
The counter is NOT incremented during boot grace, so a cold-start cycle
can't alone push it to threshold. The meta-alert (`family-silence-eval`)
path keeps its own clock and is unaffected. Existing tests advance
through three consecutive silent ticks before asserting the post.
|
||
|
Jordan Ritter
|
7f118c5955 |
fix(showcase/harness): retry+cached-catalog producer enumerate (Railway-GQL resilience)
Three-retry exponential backoff (1s/4s/16s) on `source.enumerate` against Railway-GQL when the underlying error is transient (HTTP 429, 5xx, or a Cloudflare 1015/1020/1022 WAF marker, or a transport-level reject). On persistent failure, fall back to the last successful catalog from a per-enumerator in-memory cache — LOUDLY logged via `fleet.producer.enumerate-failed-using-cache` so the cache-use shows up in observability. A fresh-boot process with no cached entry preserves the current hard-fail behavior (the producer's `enumerate-failed` short-circuit) — without a catalog there is nothing to enqueue. Real config errors (`DiscoverySourceAuthError`, non-429 4xx, schema rot) are NOT retried so operator-actionable failures surface immediately. Context: 2026-06-17 Cloudflare WAF burst-blocked backboard.railway.com/graphql/v2 for ~25 min, hard-failing the producer enumerate on every cron tick and zeroing out D4/D5/D6 writes — the entire staging dashboard went red within one tick. Retries + cache ride out the burst on the same tick and preserve job production across longer outages. Pre-existing repo-wide lefthook test failures (@copilotkit/core, @copilotkit/runtime, @copilotkit/shared, etc.) are unrelated to the harness; verified by stashing my changes and running the same hook on main with identical failures. Harness suite (131 files / 2812 tests), typecheck, and build pass on this branch. |
||
|
Jordan Ritter
|
5a62acbf72 |
docs(showcase): cell red→green SOP + agent-tiered fanout from README.md (#5512)
## Summary Two-commit docs PR sequenced AFTER #5495 — it references CLI semantics introduced there (control-plane `:demo` scoping, `--isolate` rebuild scope). 1. **SOP + CLI reference + prune stale.** `showcase/TESTING.md` gains: - The cell red→green SOP (10-step procedural workflow for agents) - `bin/showcase test` CLI invocation table (control-plane vs `--direct` semantics, post-A18 / post-A21+A21b) - Operational gotchas added to `showcase/GOTCHAS.md` (aimock fixture caching, `--isolate` slot collisions) - Stale invocation guidance pruned across RUNBOOK/README/DEBUGGING/TESTING (9 items) 2. **Consolidation + agent-tiered fanout from README.md.** - DELETE `showcase/QA-COVERAGE.md` → folded into `TESTING.md` as Per-Demo Coverage Matrix - DELETE `showcase/RUNBOOK.md` → unique ops content merged into `DEBUGGING.md`; duplicated `--isolate` mechanics/CLI rules already covered in `TESTING.md` - README.md re-tiered as agent entry point: top-of-file fanout table ("when X, see Y.md") routing to procedural docs - Each remaining doc gains a one-line tagline answering "what does this answer" - Cross-refs use relative `./<file>.md` paths 3. **style: auto-fix formatting** — oxfmt applied locally during pre-push to prevent CI auto-format-bot from firing. ## Test plan - [x] All cross-refs resolved (no dangling links after deletions) - [x] Pre-push quality on docs branch (oxfmt clean, commit hygiene clean) - [ ] CI gates pass (CI is the only gate for doc-only PRs per `feedback_cr_rigor_scales`) Note: depends on #5495 for accurate CLI semantics references. |
||
|
Jordan Ritter
|
cf615df21a |
fix(showcase): disjoint catchall userMessages + content-asserting probes (supersedes #5465) (#5495)
## Summary Restores **all 4 D5 custom-catchall cells** (LGP, crewai-crews, built-in-agent, claude-sdk-typescript) to green via the production-equivalent control-plane pipeline, plus 4 harness honesty fixes that turn `--isolate` and `:demo` invocations into apples-to-apples staging mirrors. **Cells GREEN locally (verified post-CR via `bin/showcase test <slug>:tool-rendering-custom-catchall --d5 --isolate`):** - `langgraph-python` ✓ - `crewai-crews` ✓ - `built-in-agent` ✓ - `claude-sdk-typescript` ✓ ## Harness honesty fixes (the load-bearing ones) 1. **A11 — probe-scan inline-needle.** `page.evaluate(fn, arg)` was passing `undefined` to the browser closure → `customContentPhrasePresent` was permanently false fleet-wide, masking every other failure mode. Fix: inline the canonical phrase literal in the closure. A25a propagated the same fix to `d5-tool-rendering-default-catchall.ts` (still had the broken pattern). 2. **A18 — control-plane honors `:demo`.** `bin/showcase test <slug>:<demo> --d5/--d6 --isolate` previously ignored the demo qualifier (d5 hardcoded to `agentic-chat`; d6 aggregate-only). Now per-demo scoping flows through `buildLocalServicesJson` + `expectedKeys`. Eliminates a class of silent false-positive PASS. 3. **A21 + A21b — `--isolate` rebuild scope.** A21 scoped `--build` to target slug (BuildKit contention unblock); A21b corrected an A21 regression where positional-after-`up` restricted which services started. Result: two-call compose split — `compose --profile infra up -d` (no build, uses cached images), then `compose --profile infra --profile <slug> up -d --build <slug>` (rebuild target only). Cold-build ~30s–2 min instead of 10+ min full-stack rebuild. ## Cell fixes - **A14 (crewai-crews)**: backend defect — `tool-rendering-custom-catchall` agentId routed to shared `LatestAiDevelopment` flow at `/` with no `get_weather`/`get_stock_price` handlers; tool-loop never closed. Added `get_stock_price_impl`, re-routed agentId to `/tool-rendering`. - **A19b (built-in-agent)** + **A20 (claude-sdk-typescript)**: backend ID-rewrite (TanStack `fc_*`, Anthropic `toolu_*`) broke `toolCallId`-gated narration fixtures. Swapped to `turnIndex` discriminator (backend-id-invariant). `response.content` (canonical phrase) preserved verbatim. - **A10 (langgraph-python)**: parent commit \`9491b8934\` disjoined the d5 probe userMessages but didn't add matching LGP-gold fixture entries. Added 4 entries (turn-1 emit + turn-2 narration for Tokyo + AAPL). - **A1-A9 (R1+R2 fixture hygiene)**: cleanups to dead `turnIndex:0` fallbacks, `hasToolResult:false` gates, AAPL value drift, csdkts copy-paste bug — pre-A11 era. ## Test coverage (A25 round, addresses post-A21b CR findings) - A11 inline-needle invariant (regression test — fails if the fix is reverted) - A7 `requireContentPhrase=true` branch end-to-end - A18 `buildLocalServicesJson` + `expectedKeys` + `dedupeScopes` + `runViaControlPlane` error surfacing (17 new tests in \`control-plane-run.test.ts\`) - A21+A21b two-call compose argv contract (lifecycle.test.ts, 7 tests) Full harness vitest suite: **2790/2790 pass**. Typecheck clean. oxfmt clean. ## Test plan - [x] Local control-plane (`--d5 --isolate`) on all 4 cells — GREEN - [x] LGP regression check across every fix-round — GREEN throughout - [x] vitest harness suite (2790 tests) — GREEN - [x] tsc --noEmit — clean - [x] oxfmt --check — clean - [x] Pre-push quality + 7-agent cr-loop + 3-slot post-A25 confirmation round — converged ZERO blockers - [ ] CI gates (gh pr checks 5495) on push — to be observed ## Notes for reviewers Docs PR (\`docs/showcase-sop-tiering\`, off main) sequences AFTER this one — it documents the new CLI semantics (\`:demo\` scoping, \`--isolate\` rebuild scope) introduced here. |
||
|
github-actions[bot]
|
ac85ac4f96 | style: auto-fix formatting | ||
|
Jordan Ritter
|
38aa931c71 |
fix(showcase/harness): add A18 test coverage + tighten control-plane error surfacing
- Add control-plane-run.test.ts (17 tests) covering buildLocalServicesJson, expectedKeys, dedupeScopes, and runViaControlPlane error surfacing - Export SlugScope, buildLocalServicesJson, expectedKeys, dedupeScopes for unit-test coverage (factored inline dedup loop into dedupeScopes helper at the same time) - runViaControlPlane: surface scopeLabel (demo-aware) in the 0-enqueue error instead of the bare-slug join, with an empty-targets guard so the error never renders with a double-space gap - runViaControlPlane: treat tick.enqueueFailures > 0 as fatal — partial enqueue used to silently proceed and either mask missing cells or hang the poll loop to timeout - Eliminate a stray literal NUL byte in the source by switching the dedup key separator to a \x00 escape - lifecycle.up(): name the compose call (infra-up vs target rebuild) in the health-fail error so an operator can tell which call left a service unhealthy (cherry picked from commit 9f35c64adfdf7f5ff2bf0a5ae57ed03818cea607) |
||
|
Jordan Ritter
|
61edef02a6 |
fix(showcase/harness): apply A11-style inline-needle to default-catchall + add regression tests for inline-needle + requireContentPhrase
CR Finding 1 (BLOCKER): d5-tool-rendering-default-catchall.ts used the broken page.evaluate(fn, arg) second-arg form to pass the leak-phrase needle into the browser-side closure. A11 proved empirically that the arg arrives as undefined inside the closure, making 'if (needle)' guard the entire leak-detection cascade as dead code — customLeakPhrasePresent stayed false forever, rendering validateDefaultCatchall's leak branch dead code as well. Mirrors the A11 fix on the sibling custom-catchall probe by inlining the needle as a JS string literal inside the closure; no page.evaluate(fn, arg) dependency at all. Both probes now share the same inline-needle pattern and keep the canonical literal in lock-step with their exported phrase constant. CR Finding 2 (MAJOR): the A11 inline-needle fix on the sibling custom-catchall probe had no regression test — fake Page.evaluate in makePageReturning never executes the probe closure, so reverting the fix would not be caught. Added regression tests that capture the probe's function source via toString() and assert (a) the canonical phrase appears as a literal inside the page.evaluate(...) closure and (b) the closure takes no parameter / the evaluate call has no second arg. Added the same coverage to default-catchall to protect the new A25a fix. CR Finding 3 (MAJOR): A7's requireContentPhrase=true branch in validateCustomCatchall / assertCustomCatchall had zero coverage — tests omitted the third arg and exercised only the default false branch. Added coverage for the true branch (pass on phrase present, fail on phrase absent, fail on phrase undefined, default-branch preserved) plus assertCustomCatchall plumbing through the options form. Also added coverage for default-catchall's customLeakPhrasePresent branch in validateDefaultCatchall for symmetry. Local proof: - RED (fix reverted via git stash): 2 inline-needle regression tests fail on d5-tool-rendering-default-catchall.test.ts - GREEN (fix restored): 34/34 tests pass across both files Out of scope (NOT touched this commit): showcase/harness/src/cli/ control-plane-run.ts and lifecycle.ts (A25b's scope). (cherry picked from commit c409e3a8d99e16ad0bb05ee3c2e2051e5792049f) |
||
|
Jordan Ritter
|
15f828915a | style(showcase/harness): apply oxfmt to lifecycle.ts compose argv call | ||
|
Jordan Ritter
|
19b65882d1 | style: auto-fix formatting | ||
|
Jordan Ritter
|
6cc1803f37 |
docs(showcase): consolidate + re-tier for agent navigation (README fanout entry)
Re-tier the showcase docs tree to be an agent entry point: README.md opens with a 'when X, see Y' fanout table that routes to the right procedural doc; each procedural doc gets a one-line tagline answering 'what does this answer'. Consolidation: - DELETE showcase/RUNBOOK.md — operational content merged into DEBUGGING.md (Integration Patterns, Docker Compose Environment, Production Debugging, Anti-Patterns, Aimock Fixture Deployment, Dev Iteration Speed). The --isolate mechanics + CLI rules were already duplicated in DEBUGGING.md. - DELETE showcase/QA-COVERAGE.md — per-demo coverage matrix + starter hero matrix + probe depth + infra locations + gaps folded into TESTING.md as the 'Per-Demo Coverage Matrix' section. Taglines added (no behavioral change to content): TESTING.md, DEBUGGING.md, GOTCHAS.md, INTEGRATION-CHECKLIST.md, STYLING-GUIDE.md, FRONTEND-STRATEGY.md, RAILWAY.md, bin/README.md, aimock/README.md, aimock/RAILWAY.md, harness/README.md, harness/docs/rotation-drill.md. Cross-link fixups: FRONTEND-STRATEGY.md (was QA-COVERAGE.md → TESTING.md#per-demo-coverage-matrix), TESTING.md (removed dangling RUNBOOK companion reference), README.md (rewritten as fanout entry + retained from-scratch setup + dashboard SOPs below the fanout). PARITY_NOTES.md × 12 left alone (per-slug context, not redundant). (cherry picked from commit 75c9d9755c9118c8abc1fa52deda2012b768cab1) (cherry picked from commit b64189bae0fe2c9e3a5e3ca440013deb4121f23b) |
||
|
Jordan Ritter
|
423167d12e |
docs(showcase): SOP for cell red→green + control-plane vs --direct CLI reference; prune stale invocation guidance
New content: - TESTING.md: add 10-step cell red→green SOP + bin/showcase test invocation table (control-plane vs --direct, per-demo scoping matrix); retain existing CI gating matrix below. - GOTCHAS.md: add operational gotchas — aimock caches fixtures at container startup (warm-slot reuse needs docker restart) + --isolate slot collisions with foreign Docker projects. - README.md: cross-link to TESTING.md SOP from CLI section; flesh out --isolate / --direct in test options table; update use cases. - RUNBOOK.md: update Verifying a Slug's D6 State to use auto-named --isolate; note A21+A21b per-slug rebuild scoping; rewrite Fixture Matching to teach picking the backend-id-invariant discriminator (turnIndex post-A12/A13/A20); modernize Debugging Sequence to --isolate flow. - DEBUGGING.md: lead with TESTING.md SOP cross-link; update Phase 1 to --isolate canonical; soften turnIndex-only log-line description; note aimock startup caching in Phase 5; switch Strategy 5 gold-standard check to --isolate. Pruned/updated stale claims (post-A11/A12/A13/A18/A20/A21/A21b): - RUNBOOK.md "Do not use turnIndex in new fixtures" — turnIndex is now the canonical backend-id-invariant alternative when toolCallId is fragile (Anthropic / TanStack Responses API ID rewrites). Replaced with discriminator selection guidance. - RUNBOOK.md anti-pattern "NEVER use turnIndex" — replaced with NEVER anchor on toolCallId strict equality against ID-rewriting backends, and NEVER use --direct for value-tests. - RUNBOOK.md bin/showcase test <slug> --d5 (no --isolate) as canonical SOP — replaced with --isolate canonical, no manual name required. - README.md --d5 option description claiming "subagents/tool-rendering/agentic-chat" fixed slate — replaced with "defaults to agentic-chat representative; :demo qualifier honored post-A18". - DEBUGGING.md Phase 1 "showcase up aimock <slug> && showcase test <slug> --d5" as primary — kept as legacy alternative; --isolate is now lead. - DEBUGGING.md Phase 5 "fixtures baked into Docker image" — clarified that aimock additionally caches fixtures in memory at startup (volume-mounted isolated stack still requires docker restart for warm-slot edits). - DEBUGGING.md Strategy 5 "showcase test langgraph-python --d5" — replaced with :demo + --isolate so the gold-standard check exercises the same cell. (cherry picked from commit 0e548455043396972f7fb5b96f8c0ea8abdf1d98) (cherry picked from commit 592c02d392350d02cc5e17544e663a6605b8da65) |
||
|
Tyler Slaton
|
449237af0c | fix: rename premium docs to intelligence platform | ||
|
Jordan Ritter
|
80fad7bbfb |
fix(showcase/aimock/d6/claude-sdk-typescript): tune custom-catchall match for Responses API turn-index discriminator
Mirrors A19b's BIA fix pattern for the Anthropic-family csdkts integration.
Root cause: csdkts uses Anthropic SDK which generates its own toolCallIds (toolu_*) rather than echoing aimock's prescribed call_d6_cc_*. The fixture's toolCallId-gated narration entries never matched on turn-2, causing fall-through to less-specific entries (or 503/no-match).
Fix: replace toolCallId discriminator with turnIndex (count of role:assistant messages). turnIndex is backend-id-invariant — it works regardless of how the backend rewrites tool_call_id values. Same shape as A19b BIA fix.
- Tokyo narration: toolCallId → turnIndex: 1
- AAPL narration: toolCallId → turnIndex: 3
- AAPL emit: added turnIndex: 2
- Tokyo emit: turnIndex: 0
response.content + canonical phrase ("rendered through the custom wildcard catchall") and response.toolCalls UNTOUCHED.
Verified locally on cr5495/fix-a20-csdkts-green at HEAD
|
||
|
Jordan Ritter
|
d178e67300 |
fix(showcase/harness): split --isolate compose into infra-up + target-build (A21b)
A21 (
|
||
|
Jordan Ritter
|
a9114a8317 |
fix(showcase/harness): scope --isolate rebuild to target slug only (A21)
Root cause ---------- Both the shell `cmd_up` (showcase/bin/showcase line 98) and the harness auto-start path (showcase/harness/src/cli/lifecycle.ts up() line 268) emit `docker compose up -d --build` with NO positional service filter after `up`. Under `docker compose` semantics, an unfiltered `--build` rebuilds EVERY service in every active profile — for `--isolate` runs that means the target slug PLUS aimock + pocketbase + dashboard + harness-pool-worker all rebuild on every isolated test invocation. Concurrent `--isolate` runs (one per `SHOWCASE_ISO_SLOT=<N>`) contend on the shared BuildKit instance, serializing what should be parallel work. This is the BuildKit-contention stall that has been blocking csdkts (A20) when it runs alongside other concurrent isolated cells: a small slug rebuild waits on every other slot's full-stack rebuild. Fix --- Scope `--build` to the targeted slug(s) by passing them as positional service args AFTER `up`. Compose then rebuilds ONLY those services and falls through to cached images for infra services. First-time bootstrap still works because `--build` only FORCES a rebuild of services with an existing image — missing images are built automatically by compose. Two call sites: - showcase/bin/showcase (user-facing `showcase up <slug>...`) - showcase/harness/src/cli/lifecycle.ts (npx tsx test path, the one actually triggered under `--isolate`) When no slugs are provided (infra-only bring-up), the blanket `--build` is preserved so first-time infra bootstrap rebuilds whatever is missing. `--rebuild` regression: the `--rebuild` flag goes through `rebuild()`, not `up()`, and that path is unchanged — it still force-rebuilds and force-recreates every targeted slug as before. Red→Green proof --------------- 3 new vitest cases in showcase/harness/src/cli/lifecycle.test.ts assert the compose argv shape: - up(["langgraph-python"]): slug positional after `up` (was: missing) - up([]): no slug positional (infra-only blanket) - up(["a","b"]): both slugs as positionals after `up` Pre-fix: 2 failed | 4 passed (6) Post-fix: 0 failed | 6 passed (6) The compose-argv assertion is the deterministic proof of behavior change. Wall-clock timing improvement on concurrent --isolate runs follows mechanically from the compose semantics: with the fix, slot N's BuildKit work no longer blocks slot M's because they only rebuild their respective slug images, not the shared infra stack. (cherry picked from commit 8a47dbb0d2a2fe72e21c09a824485e1bb5afc463) |
||
|
Jordan Ritter
|
97e9ce324a |
fix(showcase/aimock/d6/built-in-agent): tune custom-catchall match for /v1/responses turn-index discriminator
Replace per-leg toolCallId pin with turnIndex (assistant-count) +
userMessage on the two narration fixtures, and add explicit turnIndex
to the AAPL-emit fixture, so first-match-wins partitions the four
request shapes BIA produces against the OpenAI Responses API.
Root cause:
- BIA uses @tanstack/ai-openai openaiText('gpt-4o') which calls
/v1/responses. aimock converts each /v1/responses request to a
chat-completions-shaped completionReq via responsesInputToMessages()
and matches with the same router. The matcher's toolCallId check is
strict equality against the last message's tool_call_id.
- BIA's TanStack runtime auto-generates tool_call_id at request time
(e.g. 'fc-fCgLtvquOtRpCJTM'), so the fixture-side literal
'call_d6_cc_weather_001' / 'call_d6_cc_stock_001' never matched.
Result: 503 STRICT no-fixture-match on the narration turns, BIA
agent looped on AAPL emit indefinitely.
Fix shape:
- Tokyo narration: toolCallId -> turnIndex: 1
- AAPL narration: toolCallId -> turnIndex: 3 (was off-by-one until I
accounted for the Tokyo-narration assistant message itself adding
to the assistant-count tally seen at AAPL emit time)
- AAPL emit: add turnIndex: 2 so first-match-wins partitions emit vs
narration on the second prompt's two turns
Verification (worktree wt-5495-a19-bia-record, slot iso6):
- RED: bin/showcase test built-in-agent:tool-rendering-custom-catchall
--d5 --isolate -> state=red, 0 passed/1 failed, INNER_EXIT=1
(aimock journal: 3x 503 'No fixture matched' on turn-2 narration
request; AAPL emit fixture matched repeatedly = infinite loop)
- GREEN: same command, post-fix and aimock-restart so the container
reloads the fixture -> state=green, 1 passed, INNER_EXIT=0; aimock
journal: 4 requests, all 200, clean progression
Tokyo-emit (asstCount=0) -> Tokyo-narrate (asstCount=1) ->
AAPL-emit (asstCount=2) -> AAPL-narrate (asstCount=3)
- LGP regression: bin/showcase test
langgraph-python:tool-rendering-custom-catchall --d5 --isolate ->
state=green, 1 passed, INNER_EXIT=0 (uses its own fixture under
aimock/d6/langgraph-python/ — untouched by this change)
Constraints honored: response.content and response.toolCalls
preserved verbatim; canonical narration phrases unchanged; only the
match keys (and their explanatory _comment fields) were modified;
other integrations' fixtures and the harness probe were not touched.
(cherry picked from commit 60d027a1376ba72309b5b3fcf94cc26c64757307)
|
||
|
Jordan Ritter
|
c27bd6bebe |
fix(showcase/harness): honor target.demo in control-plane dispatch (per-demo scoping)
Before A18, `bin/showcase test <slug>:<demo> --d5/--d6` routed through the
control-plane runner but `runViaControlPlane()` collapsed every TestTarget
to its bare slug:
const slugs = [...new Set(targets.map((t) => t.slug))];
`target.demo` was silently dropped, and:
- `buildLocalServicesJson` hardcoded `demos: ["agentic-chat"]` for d5 and
the full demo set for d6 — regardless of what the operator typed.
- `expectedKeys` always emitted the level's DEFAULT-scope key
(`d5:<slug>/agentic-chat` for d5, `d6:<slug>` aggregate for d6).
The net effect on d5: typing `built-in-agent:tool-rendering-custom-catchall
--d5` enqueued the agentic-chat representative, ran agentic-chat in the
worker fleet, and wrote the side row `d5:built-in-agent/agentic-chat` —
green in ~0.0s — while the dashboard reported PASS for a cell the run
never exercised. Validation was dishonest by construction; the CLI claim
of per-demo coverage was substituted with the default representative.
Fix (additive):
- Introduce a per-call SlugScope = { slug, demo? } and thread it through
`buildLocalServicesJson` + `expectedKeys` + `runViaControlPlane`.
- When `target.demo` is set:
* `LOCAL_SERVICES_JSON` synthesizes `demos: [<demo>]` (the worker's
d6 driver reads `input.demos` → `demosToFeatureTypes` → the closed
featureType set, so the matrix narrows to exactly that demo's
featureType(s)).
* `expectedKeys` translates the demo ID into its featureType(s) via
the same `REGISTRY_TO_D5` mapping the driver uses, and waits on
`<level>:<slug>/<featureType>` — NOT the default-scope key. An
unmappable demo throws (would otherwise hang to timeout).
- When `target.demo` is absent: zero behavioral change. d5 still
enqueues the agentic-chat representative and waits on
`d5-single-pill-e2e:<slug>` + `d5:<slug>/agentic-chat`; d6 still
enqueues the full demo set and waits on the `d6:<slug>` aggregate.
- CLI banner now prints the per-scope label (`<slug>:<demo>` qualifier
when present), mirroring the legacy direct path's labeling.
Repro (built-in-agent:tool-rendering-custom-catchall, --d5 --isolate):
BEFORE
Waiting for worker fleet to produce cells:
d5-single-pill-e2e:built-in-agent, d5:built-in-agent/agentic-chat
✓ 2 passed (0.0s) ← agentic-chat representative ran;
catchall never touched. False-positive.
AFTER
Waiting for worker fleet to produce cells:
d5:built-in-agent/tool-rendering-custom-catchall
✗ 1 failed: state=red ← catchall actually ran. Cell is RED at
this HEAD (BIA narration→DOM is a
separate task A12). Validation is now
HONEST.
Regression: bare-slug d5 default UNCHANGED:
built-in-agent --d5 →
d5-single-pill-e2e:built-in-agent, d5:built-in-agent/agentic-chat
✓ 2 passed (0.0s)
Files:
showcase/harness/src/cli/control-plane-run.ts (+96 -17)
- import demosToFeatureTypes
- SlugScope interface
- buildLocalServicesJson: per-slug demo override
- expectedKeys: per-demo featureType-derived keys; refuse-loud
on unmappable demo
- runViaControlPlane: (slug, demo) dedup; scope-aware banner/log
(cherry picked from commit 82ae79279415b7eaf91759df4d8bc1625816a8c3)
|
||
|
Jordan Ritter
|
8ff6dfa306 |
fix(showcase/integrations/crewai-crews): wire tool-rendering catchall to /tool-rendering flow + add get_stock_price handler
Pre-existing crewai backend defect: tool-rendering-custom-catchall agentId was routed to the shared LatestAiDevelopment ChatWithCrewFlow at /, with no get_weather/get_stock_price handlers. tool_result never returns, second LLM call never fires, toolCallId-gated narration fixture entries unreachable, A7 requireContentPhrase=true probe fails.
Fix: add get_stock_price_impl in src/agents/tool_rendering.py mirroring LGP-python tool shape; re-route 'tool-rendering-custom-catchall' agentId in src/app/api/copilotkit/route.ts to createAgent('/tool-rendering').
Local verification on cr5495/fix-a14-crewai-green:
- /tmp/cr/a14-green-crewai.log: 1 passed (6.8s), INNER_EXIT=0
- /tmp/cr/a14-green-lgp.log (LGP regression): 1 passed (9.0s), INNER_EXIT=0
(cherry picked from commit 208d90018a3fc41c263fd4514b104c78815f8303)
|
||
|
Tyler Slaton
|
17c36421bb | Rename premium docs sections to Enterprise | ||
|
Jordan Ritter
|
23b9215c8f |
fix(showcase/harness): inline phrase needle in custom-catchall DOM scan
Root cause (A11 investigation, PR #5495): the probe passed the canonical phrase into the browser-side `page.evaluate` closure via the second-arg form (`evaluate((expectedPhrase?: string) => …, phrase)`). Empirically the arg arrives as `undefined` inside the closure — verified via in-closure return diagnostics showing `needleLen === 0` while `bubble.textContent` and `body.textContent` BOTH contained the canonical phrase at the SAME poll moment. With `needle === """, the `if (needle)` guard skipped the entire cascade, leaving `customContentPhrasePresent` false forever and failing every integration's `tool-rendering-custom-catchall` cell even when the narration was reaching the DOM correctly (LGP gold). Fix: inline the canonical phrase as a JS literal inside the closure — no `page.evaluate(fn, arg)` second-arg dependency. Keep the existing per-tier scoped cascade and `body.textContent` fallback semantics unchanged. The shared constant `CUSTOM_CATCHALL_CONTENT_PHRASE` and the inlined literal MUST stay in lock-step (existing `exports the testid contract for cross-test reuse` unit test verifies the constant; manual sync required on edit of the literal). RED-GREEN proof: RED (pre-fix, /tmp/cr/a11-red-lgp.log INNER_EXIT=1): LGP `d5-single-pill-e2e:langgraph-python red` settled text turn 2 = 'AAPL is trading at $338.37, down 2.96% — rendered through the custom wildcard catchall renderer.' (phrase IS in bubble.textContent) but snap.customContentPhrasePresent: false ← THE BUG GREEN (post-fix, /tmp/cr/a11-final-lgp.log INNER_EXIT=0): LGP `d5-single-pill-e2e:langgraph-python green (8.9s) → 1 passed` BIA remains red (separate fixture-routing concern — BIA's narration is not reaching the DOM at all per `settled text` = pure tool-card text, `assistantMsgCount: 2`, no canonical-phrase bubble; outside this fix's scope per the worktree's modify-harness-only constraint). (cherry picked from commit a980be48c514a2ab4972395e80e70e5b19247131) |
||
|
Jordan Ritter
|
87f350f240 |
refactor: revert label-coupled vars + rename D2/D3 drivers to dN- pattern (#5506)
## Summary Two cleanup commits following the just-merged taxonomy PRs (#5498/#5503/#5505): 1. **Revert label-derived var names back to enum-mirroring** — vars should mirror the persisted `Status` enum contract, not the display label that changes. 2. **Rename D2/D3 drivers to follow `dN-` pattern** — `liveness.ts` → `d2-liveness.ts`, `e2e-readiness.ts` → `d3-readiness.ts`. Consistent with `d4-chat-roundtrip.ts`, `d5-single-pill.ts`, `d6-all-pills.ts`. ## Commit 1 — `pctBeAgent`/`totalBeAgent` → `pctWired`/`totalWired` PR #5498 introduced `pctBeAgent`/`totalBeAgent` when the L1 display label was being renamed `"Wired"` → `"BE (Agent)"`. The label has since changed again (to `"API (HTTP)"` in #5505), making the var names doubly stale. The underlying `Status` enum string `"wired"` is a hard-constraint persisted contract that will never change. Revert vars to mirror the enum. **Strategy: vars mirror persisted contracts, labels are the translation layer.** Future label renames don't ripple into var names. 3 files, 9 occurrences renamed: `cells-view.tsx`, `coverage-bar.tsx`, `parity-view.tsx`. ## Commit 2 — D2/D3 driver rename to `dN-` pattern Driver naming was inconsistent: `d4-`/`d5-`/`d6-` prefixed but D2 (`liveness.ts`) and D3 (`e2e-readiness.ts`) weren't. Now consistent. | Old | New | |---|---| | `showcase/harness/src/probes/drivers/liveness.ts` | `d2-liveness.ts` | | `showcase/harness/src/probes/drivers/liveness.test.ts` | `d2-liveness.test.ts` | | `showcase/harness/src/probes/drivers/e2e-readiness.ts` | `d3-readiness.ts` | | `showcase/harness/src/probes/drivers/e2e-readiness.test.ts` | `d3-readiness.test.ts` | (Drop the redundant "e2e" prefix for D3 — the file's job is "is the demo frontend ready", which the dimension code `d3-` already conveys.) 16 files touched: 4 file renames via `git mv` (100% / 99% similarity preserved), plus import + doc updates across `orchestrator.{ts,test.ts}`, `cli/runner.ts`, `cli/targets.ts`, `probes/liveness.ts` (helper file, not the driver), `probes/loader/probe-invoker.ts`, `probes/discovery/railway-services.ts`, `probes/drivers/starter-smoke.{ts,test.ts}`, `types/dimensions.test.ts`, `config/probes/e2e-demos.yml`, `harness/README.md`. ## Stable contracts preserved - Exported function names (`runLiveness`, `runE2EReadiness`) — file rename only, NOT symbol rename - `Status` enum string values - `LiveDimension` union values - Filter chip `id: "wired"` (filter state key) - `keyFor(...)` dimension keys - All visible labels (already correct per #5505) ## Audit of lowercase `"wired"` comments Audited every `\bwired\b` hit in `showcase/shell-dashboard/src/` and `showcase/harness/src/`. All hits fall into three accurate categories: 1. `Status` enum string literal `"wired"` 2. Enum-noun phrasing (`wired-cell count`, `wired chip`, `Wired + supported` section header in `cell-model.ts:813`) 3. Unrelated code-plumbing usage (`probe-wired`, `wired up`, `wired through`, `wired by buildServer`) No stale display-label leftovers found — PR #5505 already cleaned those up. No comment edits needed. ## Verification - shell-dashboard: typecheck clean, **1089 pass / 1 skip / 0 fail** across 63 files, build clean - showcase/harness: typecheck clean, **2769/2769 pass** across 129 files, build clean - oxfmt `--check`: clean across all 17 changed TS/TSX files - `rg pctBeAgent|totalBeAgent`: empty - `rg drivers/liveness|drivers/e2e-readiness`: empty - NUL-byte scan: empty ## Related - #5498 — original "Wired" → "BE (Agent)" L1 + catalog rename (introduced the var names this PR reverts) - #5503 — RT → BE cell badge - #5505 — CV → 1P cell badge + API (HTTP) / BE (Agent) long-form normalization |
||
|
Jordan Ritter
|
5842249c2e |
refactor(harness): rename D2/D3 drivers to follow dN- pattern
Other dimension drivers under showcase/harness/src/probes/drivers/ follow the dN-<purpose>.ts naming pattern (d4-chat-roundtrip.ts, d6-all-pills.ts). D2 and D3 didn't — bring them in line. drivers/liveness.ts -> drivers/d2-liveness.ts drivers/liveness.test.ts -> drivers/d2-liveness.test.ts drivers/e2e-readiness.ts -> drivers/d3-readiness.ts drivers/e2e-readiness.test.ts -> drivers/d3-readiness.test.ts The d3 driver loses the redundant 'e2e' prefix — the dimension is D3 and the corresponding dashboard badge is 'UI', so the file's job is best described as 'is the demo frontend ready?'. Imports updated in: - showcase/harness/src/orchestrator.ts - showcase/harness/src/orchestrator.test.ts - showcase/harness/src/cli/runner.ts - showcase/harness/src/probes/drivers/d2-liveness.test.ts (self-ref) - showcase/harness/src/probes/drivers/d3-readiness.test.ts (self-ref) Doc-comment references to the old file paths updated in: - showcase/harness/src/cli/targets.ts - showcase/harness/src/probes/liveness.ts (the helper, not the driver) - showcase/harness/src/probes/loader/probe-invoker.ts - showcase/harness/src/probes/discovery/railway-services.ts - showcase/harness/src/probes/drivers/starter-smoke.ts - showcase/harness/src/probes/drivers/starter-smoke.test.ts - showcase/harness/src/types/dimensions.test.ts - showcase/harness/config/probes/e2e-demos.yml - showcase/harness/README.md (driver table) Function/symbol exports unchanged (livenessDriver, e2eReadinessDriver, createE2eDemosDriver, etc.) — file rename only. The parent helper module showcase/harness/src/probes/liveness.ts is also untouched: it's a separate, non-driver helper that exports deriveHealthUrl, livenessProbe, and LIVENESS_SLACK_SAFE_FIELDS. Not renamed: drivers/e2e-parity.ts, drivers/aimock-wiring.ts, drivers/pin-drift.ts, drivers/image-drift.ts (non-dimensional probes). |
||
|
Jordan Ritter
|
4ef08bb112 |
refactor(showcase): revert label-derived var names to mirror Status enum
The internal counter vars pctBeAgent and totalBeAgent were introduced in PR #5498 when the display label happened to be "BE (Agent)". The label has since changed (#5505 made it "API (HTTP)"). Coupling internal variable identifiers to display labels is fragile — labels move; the underlying Status enum string "wired" is a persisted contract that does not. Revert the var identifiers to mirror the enum: pctBeAgent -> pctWired (coverage-bar.tsx) totalBeAgent -> totalWired (cells-view.tsx, parity-view.tsx) Display labels ("API (HTTP)", etc.) are unchanged — this is internal identifiers only. |
||
|
Jordan Ritter
|
91b7c5efb4 |
fix(showcase/aimock/d6/langgraph-python): add d5-probe catchall fixture entries for Tokyo/AAPL
Parent commit
|
||
|
Jordan Ritter
|
71f05281ab |
refactor(showcase): rename CV→1P + normalize API (HTTP) / BE (Agent) labels (#5505)
## Summary Two related taxonomy-hygiene commits cleaning up the dashboard cell/legend labels. Follows on from #5473 (E2E→UI), #5498 (Wired→BE (Agent) L1 + catalog), #5503 (RT→BE cell badge). ### Commit 1 — `CV → 1P` cell-badge rename The per-cell badge code `CV` (Conversation) becomes `1P` (Single Pill). Clearer scope framing: - D5 driver is literally `d5-single-pill.ts` (one scripted conversation) - D6 driver is `d6-all-pills.ts` (full pill suite + parity vs reference) `CV` ("Conversation") was opaque — `1P` reads as "1 pill out of N" and pairs naturally with `D6` (all pills). Scope ladder now: `API → UI → BE → 1P → D6`. ### Commit 2 — `API (HTTP)` / `BE (Agent)` long-form normalization Resolves a label collision introduced by #5498. Previously: - L1 row D2 (agent-liveness) was labeled `BE (Agent)` - Per-cell D2 badge was labeled `API (Agent)` - Per-cell D4 badge (chat round-trip) was labeled `BE (Round Trip)` So `BE (Agent)` (D2 L1) and `API (Agent)` (D2 per-cell) described the SAME concept under two different names, while `BE (Round Trip)` (D4) shared the "BE" prefix with `BE (Agent)` (D2) for DIFFERENT concepts. Now normalized by **layer**, not subject: | Dimension | Old labels | New label | |---|---|---| | **D2** (transport / Railway liveness) | "BE (Agent)" L1 + "API (Agent)" per-cell | **`API (HTTP)`** everywhere | | **D4** (agent chat round-trip) | "BE (Round Trip)" per-cell | **`BE (Agent)`** | Reads cleanly: API = is the transport up; BE = can the agent process a chat message. Both probes test "the agent" — the parens distinguish what's actually being checked. ## Scope 28 files total (17 + 11), 114 insertions, 112 deletions. Pure label/comment rename. Touches: - Source: `unified-cell`, `cell-pieces`, `cell-drilldown`, `adaptive-legend`, `adaptive-stats-bar`, `stats-bar`, `filter-chips`, `packages-section`, `level-strip`, `composed-cell`, `depth-utils`, `cell-model`, `live-status`, `page-stats` - Tests: matching test files for assertion + comment updates - Mnemonic legend: `(U=Up, B=BE (Agent), C=Chats, T=Tools)` → `(U=Up, A=API (HTTP), C=Chats, T=Tools)` ## Stable contracts preserved (no behavior change) - All `keyFor(...)` dimension keys (`"agent"`, `"d2"`, `"d4"`, `"d5"`, `"e2e"`) - Filter chip `id: "wired"` (filter state key) - `LiveDimension` union string values - `Status` enum string values (`"wired" | "stub" | …`) - Internal variable names (`wired`, `pctBeAgent`, `totalBeAgent`, etc.) — intentionally not renamed to avoid scope creep - Driver names, probe registry keys, PocketBase keys, harness API contracts ## Verification - typecheck: clean - tests: 1089 pass / 1 skip / 0 fail across 63 files - build: clean - oxfmt `--check`: clean across all 25 changed files - NUL-byte scan: empty - No remaining stale labels — `rg '"API \(Agent\)"|"BE \(Round Trip\)"'` empty; `"BE (Agent)"` only appears at D4 sites ## Related - #5473 — `E2E → UI` (original taxonomy refactor; cell-badge template) - #5498 — `Wired → BE (Agent)` L1 + catalog (introduced the label collision this PR resolves) - #5503 — `RT → BE` cell badge (set the cell-badge naming pattern this PR mirrors) |
||
|
Jordan Ritter
|
14451d5c3d |
refactor(showcase): normalize API/BE long-form labels — API (HTTP) for transport, BE (Agent) for chat round-trip
PR #5498 renamed the L1 row "Wired" → "BE (Agent)" for the D2 agent-liveness dimension (transport/Railway up). PR #5503 renamed the per-cell D4 badge "RT" → "BE", whose long form was already "BE (Round Trip)". The result was a same-label-different-concept collision: "BE (Agent)" referred to D2 in some places while the D4 per-cell badge used "BE (Round Trip)", and the per-cell D2 badge separately used "API (Agent)" — three names for two concepts. Normalize the long-form labels so each user-facing layer has exactly one name: D2 (transport / Railway up, HTTP-reachable) = "API (HTTP)" D4 (agent chat round-trip, end-to-end) = "BE (Agent)" That cleanly distinguishes by layer: HTTP transport vs Agent message handling. Sites touched (visible labels + matching legend prose / test assertions only): - stats-bar.tsx, adaptive-stats-bar.tsx — wired count label - filter-chips.tsx — chip label (id "wired" preserved) - packages-section.tsx + .test.tsx — UWCT legend mnemonic B → A - level-strip.tsx + .test.tsx — agent-dimension badge label (and derived first letter B → A) - cell-drilldown.tsx + .test.tsx + cell-drilldown.lazy-signal.test.tsx — D4 label and D2 label, plus testid-derivation drift - adaptive-legend.tsx — D2 / D4 prose Stable contracts preserved (NOT changed): - keyFor("agent" | "d2" | "d4", …) and the "agent" LiveDimension value - Filter-chip id "wired" - Variable names (`wired`, etc.) — internal; pure label rename is in scope - Status enum values "wired" | "stub" | "unshipped" | "unsupported" - Driver names, probe registry keys, harness API contracts Verified: typecheck clean, 1089 pass / 1 skip / 0 fail, build clean. |
||
|
Jordan Ritter
|
1d6d063c76 |
refactor(showcase): rename cell-badge CV → 1P (clearer scope framing vs D6 all-pills)
The per-cell health badge previously labelled "CV" (for "Conversation") is renamed to "1P" — Single Pill. The new label tells operators what the badge covers in scope terms (one pill out of N), which is the actual contrast the D5/D6 ladder draws: D5 driver is `d5-single-pill.ts` (one canonical scripted conversation), D6 driver is `d6-all-pills.ts` (the full suite). "CV" was opaque — operators had to remember what "Conversation" meant and how it differed from D6's full run. "1P vs D6 all-pills" is self-describing. Mirrors PR #5503's RT → BE pass: only the badge LABEL changes; every stable contract is preserved. Stable contracts preserved: - Dimension level identifier `model.d5` / `cell.d5` / `level={model.d5}` - Drilldown dimension key `keyFor("d5", ...)` / `key: "d5"` - Probe registry key `d5:<slug>/<featureId>` - PocketBase row keys unchanged - LiveDimension union / Status enum strings unchanged - Driver file names `d5-single-pill.ts`, `d6-all-pills.ts` - `e2e-deep` producer name unchanged (separate from CV → 1P label) Updates: - Source: badge label in `unified-cell.tsx`, `cell-pieces.tsx`, drilldown label in `cell-drilldown.tsx` ("CV (Conversation)" → "1P (Single Pill)"), legend text in `adaptive-legend.tsx`, comment refs in `composed-cell.tsx`, `cell-model.ts`, `page-stats.ts`, `depth-utils.ts`, `live-status.ts`. - Tests: testid strings (`mock-badge-CV` → `mock-badge-1P`), label-text assertions, type unions, and comment refs in the affected component + drilldown + integration + lib tests. Verification: typecheck clean, `npm test` = 1089 pass / 1 skip / 0 fail (same shape as #5503), `npm run build` clean, no remaining `\bCV\b` in `showcase/shell-dashboard/src/`. |
||
|
Tyler Slaton
|
c6a14e7969 |
feat(shell-docs): serve Built-in Agent docs at the docs root (#5404)
## Summary Built-in Agent docs now render at bare root URLs, such as `/quickstart`, instead of under `/built-in-agent`. Legacy `/built-in-agent/*` and `/integrations/built-in-agent/*` paths continue to redirect to canonical root or backend URLs. ## Why The Built-in Agent is the default docs surface, so public docs URLs should not expose it as a sub-slug. Existing links still need to keep working, and root-page navigation must stay stable regardless of a visitor's stored framework selection. ## How - Root docs resolve Built-in Agent authored pages first while preserving reserved routes like `/ag-ui` and framework-prefixed docs. - Redirect rules in `next.config.ts` and `seo-redirects.ts` point retired Built-in Agent and unselected paths directly at canonical destinations, with regression coverage that prevents redirect destinations from targeting `/built-in-agent`. - The sidebar/framework provider treats bare URLs as the default Built-in Agent surface instead of letting a stored framework value rewrite root-page chrome. - MDX link and search-result href rewriting strip retired Built-in Agent prefixes on root-rendered pages while preserving explicit cross-framework links. - Stale docs links, sitemap, `llms.txt`, markdown exports, and OG resolution now align with the root-served Built-in Agent surface. - Showcase generated-data tests now serialize shared fixture restoration to avoid concurrent drift in CI. ## Verification - GitHub checks are green on `f655013dd2576a46dc17b5901eb4dc501cb21028`. - `npm --prefix showcase/shell-docs run test -- src/lib/__tests__/search-hrefs.test.ts src/lib/__tests__/docs-link-rewrite.test.ts src/lib/__tests__/seo-redirects.test.ts src/lib/__tests__/next-config-redirects.test.ts src/components/__tests__/docs-landing-next.test.tsx 'src/app/[framework]/[[...slug]]/__tests__/framework-root-shell-layout.test.ts'` - `npm --prefix showcase/shell-docs run test -- --exclude src/app/__tests__/public-assets.test.ts` - `npm --prefix showcase/shell-docs run typecheck` - `npm --prefix showcase/shell-docs run lint` - `npm --prefix showcase/shell-docs run build` - `npm --prefix showcase/scripts run test` - `pnpm exec nx run @copilotkit/bot-slack:build --skip-nx-cache --verbose` |
||
|
Austin Merrick
|
b2f1eb79b1 |
docs: add Vue quick start guide (#5492)
## Vue quick start guide Adds a minimal getting-started guide for `@copilotkit/vue` under **Platforms**, matching the existing React Native guide. It connects a Vue app directly to an AG-UI agent with `HttpAgent`, so there is no runtime to stand up and nothing framework-specific to configure. The same setup works with any AG-UI agent, and agent-side setup links out to Integrations. ### New page   ### Changes - `showcase/shell-docs/src/content/docs/vue.mdx` (new): five steps from `npm create vue` to a working chat connected to an agent, plus a troubleshooting accordion and next-steps links. - `meta.json`: adds `vue` to the Platforms nav after `react-native`. ### Verification - `pnpm build` of shell-docs passes: TypeScript clean, all static pages generated, no errors. - Built a throwaway Vue app from this guide verbatim. `vite build` and `vue-tsc` resolve every import from `@copilotkit/vue/v2` (installed via the published `@copilotkit/vue`), the chat renders, and a message round-trips to a local AG-UI agent and streams back. |
||
|
Tyler Slaton
|
b93fd77a4d | fix(shell-docs): protect redirect aliases in docs links | ||
|
Ben Taylor
|
ba7e83d0fc |
fix(inspector): persist announcement popout dismissal via X control (#5448)
## Problem
Users reported that the announcement banner popping out of the
inspector's floating icon **does not stay closed after dismissing it**.
Confirmed in a real browser: the popout reappears on every reload. Root
cause — the popout preview bubble (`renderAnnouncementPreview`) had **no
dismiss control of its own**. Its only hide paths
(`handleAnnouncementPreviewClick`, `openInspector`) clear an in-memory
flag without persisting. On every mount `fetchAnnouncement()` recomputes
`showAnnouncementPreview` from the stored timestamp — which only the
*in-window* banner X (`markAnnouncementSeen`) ever wrote. So unless the
user opened the inspector and dismissed the inner banner, the popout
came back on every load.
## Fix
Add an X / dismiss control directly to the popout bubble:
- Click (or Enter/Space) → `markAnnouncementSeen()`, which **persists**
the announcement timestamp to `localStorage`.
- `event.stopPropagation()` so the X does not bubble up to the preview
body / floating button (i.e. it dismisses without opening the
inspector).
- Implemented as a `role="button"` span (not a `<button>`): the popout
renders *inside* the floating `<button>`, so a nested `<button>` would
be invalid HTML. Includes keyboard support and `:focus-visible` styling.
Body-click behavior is unchanged: clicking the bubble itself still opens
the inspector (engagement), and intentionally does **not** persist, so
the in-window banner still surfaces the announcement.
## Verification
- **Live browser:** popout shows X → click → `localStorage` persists
`{"timestamp":...}`, inspector stays closed → **reload keeps it gone**.
- **Unit tests (3 new, full suite 32/32 green):** X persists timestamp &
hides bubble; X does not open the inspector; body-click still opens
*without* persisting.
- Format (`oxfmt`), lint (`oxlint`), and build all clean; pre-commit
full-suite passed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Jordan Ritter
|
9d5f214047 |
refactor(showcase): rename cell-badge RT → BE (parallel to E2E → UI in #5473) (#5503)
## Summary Renames the per-cell health badge from **`RT`** to **`BE`** in the showcase dashboard cell grid. Mirrors PR #5473's `E2E → UI` shape exactly — a follow-up to PR #5498 which renamed the L1 row label / catalog stats display ("Wired" → "BE (Agent)") but missed the per-cell badge code, which is what the user actually asked for. ## Why PR #5498 unified the long-form display label "Wired" → "BE (Agent)" across the L1 row in `level-strip`, the legend in `packages-section`, and the catalog status display in `stats-bar`/`adaptive-stats-bar`/`coverage-bar`/`filter-chips`. But the per-cell badge grid (D3=`UI`, D4=`RT`, D5=`CV`, D6=`D6`) still showed `RT ✓` because that label lives in `unified-cell.tsx`'s `HealthLayer` component — outside #5498's diff. PR #5473 set the exact precedent: when relabeling `E2E (Demo)` → `UI (Frontend)`, both the long form AND the 2-char cell badge code (`name="E2E"` → `name="UI"`) flipped. This PR applies the same shape: the long form `BE (Agent)` already exists (from #5498); now the 2-char cell badge code flips `RT` → `BE`. ## Scope 13 files, +51/-51 lines: **Visible label flips (sources, 6 files):** - `unified-cell.tsx` — `<TestBadge name="RT" level={model.d4} />` → `name="BE"`, legend comment `"D4 = RT"` → `"D4 = BE"`, doc-comment refs `CV/API/RT` → `CV/API/BE` - `cell-drilldown.tsx` — `{ key: "d4", label: "RT (Round Trip)" }` → `"BE (Round Trip)"`, doc-comments updated - `adaptive-legend.tsx` — `"Round Trip (RT)"` → `"Round Trip (BE)"` - `composed-cell.tsx`, `lib/cell-model.ts`, `lib/page-stats.ts` — comment references **Test updates (7 files):** - `cell-pieces.test.tsx`, `cell-pieces.signal-degrade.test.tsx` - `__tests__/cell-drilldown.lazy-signal.test.tsx`, `__tests__/cell-drilldown.test.tsx` - `__tests__/dashboard-color-matrix.test.tsx`, `__tests__/overlay-selector-integration.test.tsx`, `__tests__/unified-cell.test.tsx` - Updates: `mock-badge-RT` → `mock-badge-BE`, `drilldown-badge-rt-…` → `drilldown-badge-be-…`, `"RT (Round Trip)"` → `"BE (Round Trip)"`, `"RT"` literal assertions → `"BE"` ## Stable contracts preserved (no behavior change) - `level={model.d4}` — D4 dimension level identifier - `dimensionKey={keyFor("e2e", …)}` (where applicable) and other persisted dimension codes - Probe registry keys `d4` / `chat` / `tools` - All PocketBase row keys, LiveDimension union values, Status enum string values Only the visible `name` string flips. ## Remaining "RT" references (intentional) 2 historical-commentary references in `cell-pieces.signal-degrade.test.tsx` lines 123–124 documenting the rename lineage (`RT → UI → BE`). Not visible UI strings or test assertions. ## Verification - typecheck (`tsc --noEmit`): clean - tests (`vitest run`): 1089 passed / 1 skipped / 0 failed across 63 files - build (`next build`): clean - oxfmt `--check`: all 13 files correctly formatted - NUL-byte scan: empty ## Related - PR #5473 — original `E2E → UI` taxonomy rename (template) - PR #5498 — `"Wired" → "BE (Agent)"` L1 row + catalog stats (which missed this cell-badge rename) |
||
|
Austin Merrick
|
b97157cadc |
docs: add Vue quick start guide
Add a minimal getting-started guide for @copilotkit/vue under Platforms. Connects a Vue app directly to an AG-UI agent via HttpAgent so there is no runtime to stand up, and imports from @copilotkit/vue/v2 to match the v2 docs convention. Links out to Integrations for agent-side setup. |
||
|
Jordan Ritter
|
02275af2ab |
fix(showcase/aimock/d6/claude-sdk-typescript): delete dead turnIndex:0 fallback entries (consistency with A2-cleaned siblings)
After A2 (commit
|
||
|
Jordan Ritter
|
8ad31e7810 |
fix(showcase/aimock/d6): bring AAPL response payload to canonical $338.37/-2.96% on 3 fixtures
crewai-crews/ms-agent-python/pydantic-ai catchall AAPL entries still carried stale $189.42/up 1.27% narration and toolCall args lacking price_usd/change_pct. Round 1 CR finding #3 was identified but never given a fix agent. Brought all 3 to canonical shape matching built-in-agent (A5) and csdkts narration (A6): price_usd=338.37, change_pct=-2.96, narration 'AAPL is trading at $338.37, down 2.96% on the day — rendered through the custom wildcard catchall.' (cherry picked from commit df84657310451500278d0b3d0125c9c490042d2b) |
||
|
Jordan Ritter
|
601e45f0d8 |
refactor(showcase): rename cell-badge RT → BE (parallel to E2E → UI in #5473)
PR #5498 was mis-scoped — it renamed the L1 row label and catalog status display from "Wired" to "BE (Agent)", but the dashboard's per-cell badge code rendered in the grid was still "RT". This commit lands the parallel flip that mirrors PR #5473's E2E → UI rename for the D4 chat/tools round-trip badge. Visible-label flips (RT → BE): - unified-cell.tsx: <TestBadge name="RT" level={model.d4} /> → name="BE" - cell-drilldown.tsx: { key: "d4", label: "RT (Round Trip)" } → "BE (Round Trip)" - adaptive-legend.tsx: legend entry "Round Trip (RT)" → "Round Trip (BE)" - Doc-comment taxonomy notes in cell-pieces.test.tsx, composed-cell.tsx, cell-model.ts, page-stats.ts, unified-cell.tsx, cell-drilldown.tsx, overlay-selector-integration.test.tsx, dashboard-color-matrix.test.tsx - Test assertions in dashboard-color-matrix.test.tsx, cell-drilldown.test.tsx, unified-cell.test.tsx (mock-badge-RT → mock-badge-BE, drilldown-badge-rt- → drilldown-badge-be-, "RT (Round Trip)" → "BE (Round Trip)") Stable contract identifiers are PRESERVED — only the visible name flips: - level={model.d4} unchanged (D4 is still D4) - dimensionKey={keyFor("e2e", ...)} unchanged - probe registry-key "d4"/"chat"/"tools" unchanged - persisted dimension codes unchanged Two intentional residual "RT" mentions remain in cell-pieces.signal-degrade.test.tsx as historical commentary documenting the rename lineage (RT → UI on the e2e badge in #5473, then RT → BE on the D4 badge here). Verification (in showcase/shell-dashboard): - npm run typecheck: clean - npm test: 1089 tests passed - npm run build: clean (next build OK) - LC_ALL=C grep -rlP '\x00' showcase/shell-dashboard/src: empty |
||
|
Jordan Ritter
|
e266b6215c |
fix(showcase/harness): wire requireContentPhrase=true in custom-catchall turn-2 assertion
validateCustomCatchall's requireContentPhrase branch was unreachable: assertCustomCatchall defaulted it to false and no caller overrode. The whole point of the PR is to catch cross-fixture leakage via content assertion — wiring it on. Turn-2 (Quote AAPL) now asserts the custom content phrase is present in the rendered bubbles, not just the testid. (cherry picked from commit c9620c96f9addfee18f87fb1f8b2ae7fb040b3b8) |
||
|
Jordan Ritter
|
d091749477 |
fix(showcase/harness): correct fixtureFile metadata on catchall probes
Probes registered fixtureFile: 'tool-rendering.json' but actual files are tool-rendering-{default,custom}-catchall.json. Field is signal-only (aimock loads directory-wide content-driven matching) but accurate metadata helps debugging. Test files updated to enforce the correct values.
(cherry picked from commit 3c9de26375546f2269870b563ecd1526ec5a3bc2)
|
||
|
Jordan Ritter
|
6c596d8d65 |
fix(showcase/aimock/d6): strip toolName gate + dead turnIndex:0 content fallbacks in 7 catchall fixtures
ag2/claude-sdk-typescript/google-adk/langgraph-fastapi/langgraph-typescript/mastra/ms-agent-dotnet: AAPL is turn-1 so turnIndex:0 fallback unreachable; toolName gate fails on wildcard-renderer integrations that don't register get_stock_price. Aligned to LGP-gold pattern (userMessage+context discriminator, no toolName, no turnIndex:0 fallback). Preserved legitimate multi-pill matchers (SF/flights/d20/chain) on the 4 multi-pill integrations. (cherry picked from commit c10821b5d904b31bee2ab2a39db3b1565aecba26) |
||
|
Jordan Ritter
|
9e2c91d02f |
fix(showcase/aimock/d6/claude-sdk-typescript): replace duplicated stock_price toolCall in AAPL fallback with narration content
(cherry picked from commit 5081966fbddf5d0090e333005bb63f5dd4b36c58) |
||
|
Jordan Ritter
|
6d60cfd310 |
fix(showcase/aimock/d6/built-in-agent): align AAPL price to $338.37 (fleet parity)
built-in-agent shipped $189.42 vs the rest of the fleet's $338.37. Drift makes any content-asserting test on AAPL price brittle. Aligned. (cherry picked from commit c0796da2a63abfeaa1d8ea06200df0754d30a2e6) |
