Split generative-ui category into 4 (controlled/declarative/open/
operational), add chat-customization-css + tool-rendering-frontend-tools,
replace old tool-rendering-status/-result IDs with default-catchall /
custom-catchall. Port langgraph-python manifest features + highlight:
paths (column-relative to preserve pre-existing Docker structure).
Update tests for new category/feature counts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Every peer validator under showcase/scripts/ already has a matching
__tests__ suite — this one shipped without. Model the new suite on
validate-parity.test.ts: per-test tmpdir fixtures + CLI subprocess
runs gated by VALIDATE_WORKFLOW_STARTERS_REPO_ROOT.
Extend the existing env-var override to re-home STARTERS_DIR as well
(previously it only redirected .github/). One env var, one root,
matches the pattern used by the other validators.
Coverage:
- happy path (all slugs registered, exit 0)
- slug missing from workflow_dispatch options only (exit 1)
- slug missing from ALL_SERVICES matrix only (exit 1)
- slug missing from both (exit 1, both sources named)
- empty showcase/starters/ (exit 3, refuses trivial pass)
- template/ excluded (not flagged as missing)
- substring-spoof: starter-ag2 missing vs starter-ag2-extended present
— regression guard for the word-boundary anchor
- showcase_deploy.yml absent (exit 3)
All 12 specs pass against the existing regex-based checks. The
substring-spoof case in particular pins the word-boundary contract so
a later refactor can't silently lose it.
## Summary
Three showcase test suites leak working-tree drift (workflow YAMLs +
shell data JSONs) on every run. This fixes the leaks and adds a shared
`FileSnapshotRestorer` + `restoreFromGitHead` harness so the suites are
idempotent.
Also moves `showcase/scripts/vitest.config.ts` from the thread pool to
the fork pool, which is required under Node 20 for the test subprocess
churn in `validate-pins` + the three generator-invoking suites.
## What this PR does NOT fix — vitest 3.2.4 RPC timeout on Node 20
(upstream)
`unit (20.x)` still reports `Timeout calling "onTaskUpdate"` ->
ELIFECYCLE **after** all 14/14 test files and 1011/1011 tests pass.
Known upstream bug: https://github.com/vitest-dev/vitest/issues/6129.
The birpc timeout is hardcoded at 60 s (`DEFAULT_TIMEOUT = 6e4` in
vitest's bundled `index.B521nVV-.js`) and is NOT exposed to
`vitest.config.ts`. No `poolOptions.forks.*` / `teardownTimeout` /
`hookTimeout` knob influences it:
- `singleFork: true` made the run STRICTLY WORSE — only 1/14 files
completed (validate-pins consumes the whole 60 s budget on its own, run
24602985507).
- `fork-per-file` (default) gives each file a fresh RPC channel, and
every file passes — but the final pool-teardown RPC still races on Node
20 and surfaces as a process-level exit 1.
Under vitest 3.2.4 the fix requires either (a) upgrading to vitest 4.x
(out of scope — monorepo-wide upgrade), or (b) a pnpm patch against the
bundled `DEFAULT_TIMEOUT` constant (cross-cutting change; declined
here). The observable reality: this PR makes the showcase-scripts suites
idempotent and green; the remaining `unit (20.x)` redness is a known
vitest flake orthogonal to what this PR is trying to fix.
## CI-killer error (verbatim)
```
Error: Package directory already exists: /home/runner/work/CopilotKit/CopilotKit/showcase/packages/test-integration-tmp
⎯⎯⎯⎯⎯⎯ Unhandled Errors ⎯⎯⎯⎯⎯⎯
Error: [vitest-worker]: Timeout calling "onTaskUpdate"
ELIFECYCLE Test failed.
Failed tasks:
- @copilotkit/showcase-scripts:test
```
## Root cause (fixed here)
Three test suites invoke real generator scripts that write to tracked
files OUTSIDE any tmp dir, leaking drift on every `nx run-many -t test`:
- `create-integration.test.ts` scaffolds
`showcase/packages/test-integration-tmp/` AND mutates three CI workflow
YAMLs (`showcase_deploy.yml`, `showcase_drift-detection.yml`,
`starter-smoke.yml`).
- `generate-registry.test.ts` rewrites
`showcase/shell/src/data/registry.json` + `constraints.json`.
- `bundle-demo-content.test.ts` rewrites
`showcase/shell/src/data/demo-content.json`.
## Fix (9 commits, by area of concern)
1. **`test(showcase/scripts): add shared test-cleanup snapshot/restore
helper`** — new `__tests__/test-cleanup.ts` + `__tests__/paths.ts` +
direct unit coverage in `__tests__/test-cleanup.test.ts`:
- `FileSnapshotRestorer` — snapshots file content as `Buffer`
(byte-exact, preserves non-utf8), restores only files that drifted,
writes atomically via temp-file + rename, recreates parent dirs on write
ENOENT. Temp filenames use `crypto.randomBytes(8).toString("hex")` so
concurrent writes can't collide and the `snapshot()` sweep can
unambiguously identify stragglers. On `snapshot()`, sweeps
`.<basename>.<16-hex>.tmp` stragglers **scoped to the snapshotted
basenames only**.
- `restoreFromGitHead(repoRoot, paths)` — partitions the input via `git
ls-files --error-unmatch` BEFORE any destructive op. **Narrow catch**:
only genuine exit-1 pathspec errors are treated as untracked; ENOENT /
EACCES / non-exit-1 failures re-raise with captured stderr. Uses
`execFileSync` (no shell), forces `LC_ALL=C` / `LANG=C`, scrubs all
`GIT_*` environment overrides, frozen exec options.
- `test-cleanup.test.ts` itself strips `GIT_*` from child env when it
creates tmp repos — pre-commit hooks (lefthook) run with `GIT_DIR` /
`GIT_INDEX_FILE` set, which would otherwise cause tmp-repo `git commit`
calls to write to the HOST working-tree HEAD.
2. **`fix(showcase/test-integration): clean up test-integration-tmp
between runs`** — `create-integration.test.ts` wires
`FileSnapshotRestorer` + `restoreFromGitHead` into the suite, wraps
`rmSync` in `try/finally` so workflow restoration always runs, and
migrates `execSync(string)` -> `execFileSync("npx", [...args])` via a
shared `runGenerator()` helper.
3. **`fix(showcase/test-integration): stop generate-registry +
bundle-content leaks`** — same pattern applied to
`generate-registry.test.ts` and `bundle-demo-content.test.ts`; drops a
redundant bundler pre-run in the latter.
4. **`fix(showcase/scripts): switch vitest to forks pool for Node 20
stability`** — `vitest.config.ts`: thread -> fork pool.
5. **`docs(showcase/scripts): tidy test-cleanup comments and JSDoc`** —
documentation cleanup.
6. **`fix(showcase/scripts): pin vitest to a single fork + bump teardown
timeouts`** — SUPERSEDED by commit 9 below (left in history for
auditability).
7. **`fix(showcase/scripts): add post-heal drifted-baseline guard`** —
the PR had claimed a drifted-baseline guard on CI for
`restoreFromGitHead`, but no post-heal `git diff --quiet` was actually
running. Adds the missing check: on CI, any tracked path still drifted
post-heal throws `drifted-baseline guard: post-heal diff failed`; off-CI
warns. Red-green unit coverage via a counter-based git shim that
selectively fails the N-th `diff --quiet` (so the post-heal diff is
targeted independently of the off-CI pre-checkout diff).
8. **`fix(showcase/scripts): decouple generate-registry test 2 from test
1 output`** — `sorts integrations by sort_order` was reading
`registry.json` without invoking the generator, so `afterEach(restore)`
between tests meant it was exercising the committed baseline rather than
live output. Adds a `runGenerator()` call at the top.
9. **`fix(showcase/scripts): revert singleFork — fork-per-file is
strictly better`** — empirical data from run 24602985507 proved
`singleFork: true` was worse than fork-per-file (1/14 vs 14/14 files
completing before RPC timeout). Reverts the
`poolOptions.forks.singleFork` change; keeps the 30 s `teardownTimeout`
/ `hookTimeout` bumps. Comments now accurately reflect that the RPC
timeout is upstream-hardcoded in birpc and NOT tunable via vitest
config.
## Proof of idempotence
```
pnpm nx run @copilotkit/showcase-scripts:test --skip-nx-cache
Run 1: Test Files 14 passed (14), Tests 1011 passed (1011)
Run 2: Test Files 14 passed (14), Tests 1011 passed (1011)
git status after each: only the intentional test file edits.
```
Red-green verified for the HIGH CR-findings:
- narrow `partitionTrackedPaths` catch: unit test with empty PATH
reproduces ENOENT; pre-fix hid it as "untracked", post-fix throws.
- basename-scoped tmp sweep: unit test places both target-basename and
unrelated `.something-else.<hex>.tmp`; pre-fix swept both, post-fix
sweeps only target.
- post-heal drifted-baseline guard: counter-based git shim fails the 2nd
`diff --quiet`; pre-fix tests pass (guard absent), post-fix tests throw
on CI / warn off-CI with the advertised message.
## Test plan
- [x] `pnpm nx run @copilotkit/showcase-scripts:test` passes 1011/1011
twice in a row (locally, Node 25)
- [x] Red-green: disabling `restore()` fails the regression + safety-net
tests
- [x] Red-green: disabling the post-heal drift guard fails the new guard
tests
- [x] Working tree clean after full run
- [x] `prettier --check` + `oxlint` clean on touched files
- [x] `GIT_*` scrub in test harness prevents pre-commit-hook-induced
pollution of real HEAD
## CI status
- **`unit (22.x)`**: pass
- **`unit (24.x)`**: pass
- **`unit (20.x)`**: all 14/14 files + 1011/1011 tests pass; post-suite
`onTaskUpdate` RPC timeout emits exit 1. Upstream bug
https://github.com/vitest-dev/vitest/issues/6129; not fixable at
`vitest.config.ts` level on vitest 3.2.4.
## Caveats
- Local verification ran on Node v25.8.0. No Node 20 binary on this dev
host; Node 20 CI was the final gate.
- The residual `unit (20.x)` failure is orthogonal to this PR. To
resolve it we would need to upgrade vitest to 4.x (monorepo-wide change)
or apply a pnpm patch to bump `DEFAULT_TIMEOUT` in vitest's bundled
`birpc`. Both are tracked separately.
parseManifest (lib/manifest.ts):
- Shape errors for demos[i].route: number / null / object / empty string
- Shape error for route not starting with /demos/
- Happy path persists frozen demo.route on the parsed entry
- Backward-compat: demos[i] without route is accepted with undefined route
validate-parity:
- Negative-case regression: missing-demo-dir's expectedDir is derived from
demo.route (not demo.id) when route is present; mismatched id + route
was silently hiding drift.
- Fallback-case: demo with no route resolves expectedDir from demo.id
- routeToDirName unit tests: undefined / bare /demos/ / normal tail segment
TDD verified: mutating the /demos/ prefix guard failed the relevant test
(RED), restoring the guard passed it (GREEN). Mutating expectedDir to
demo.id-only failed the negative-case test (RED), restoring passed
(GREEN).
demo.id is the CATALOG identifier (matched to qa/spec filenames and
shell registry entries). demo.route is the URL + filesystem path
(/demos/<dir> → src/app/demos/<dir>/). They are deliberately separate
— a manifest with id: hitl-in-chat and route: /demos/hitl lives at
src/app/demos/hitl/.
validate-parity.ts previously resolved the demo directory from
demo.id, producing a spurious missing-demo-dir MUST for every such
split. Fix:
- lib/manifest.ts: add optional route field to ManifestDemo; if present,
parser requires it to be a non-empty string beginning with "/demos/".
- validate-parity.ts: introduce routeToDirName helper (matches
bundle-demo-content.ts idiom); loop over demos resolving expected
dir from route and falling back to id. missing-demo-dir PackageIssue
now carries both demoId and expectedDir so deriveMessage can flag
route-resolved paths distinctly.
- __tests__/validate-parity.test.ts: red-green regression test — a
package with id: hitl-in-chat, route: /demos/hitl, and dir
src/app/demos/hitl/ must PASS (no missing-demo-dir error).
Adds rows for the demos being built in this branch:
- Split `chat-prebuilt` into `prebuilt-chat`, `prebuilt-sidebar`,
`prebuilt-popup` (CopilotChat / CopilotSidebar / CopilotPopup).
- Reorder `generative-ui` rows and add: `hitl-in-chat` (In-Chat HITL),
`gen-ui-interrupt`, `declarative-gen-ui` (Dynamic Schema),
`a2ui-fixed-schema` (new), `mcp-apps`, `open-gen-ui`, `gen-ui-agent`,
`tool-rendering`.
- `constraints.yaml`: expand `constrained-explicit` allowlist so the
langgraph-python manifest validates against the new feature ids.
- Bump expected langgraph-python feature/demo count in
`generate-registry.test.ts` from 10 -> 22 to match the manifest.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`sorts integrations by sort_order` read registry.json without invoking
the generator, making it dependent on whichever run last left disk
state in place. afterEach(dataRestorer.restore()) restores the data
files to HEAD between tests, so test 2 was reading the committed
baseline registry — not the live generator output — and would have
silently agreed with whatever state happened to be on main.
Add a `runGenerator()` call at the top of test 2 mirroring the
`runBundlerAndRead()` pattern that tests 2-5 of bundle-demo-content
already use. Test now exercises the actual generator's sort
behavior under live conditions.
The PR description advertised a drifted-baseline guard on CI for
restoreFromGitHead, but the implementation never actually ran a
post-heal `git diff --quiet HEAD -- <tracked>` check — the only
diff was the off-CI pre-checkout guard against clobbering developer
edits. A racing external mutator (or a parallel test suite we haven't
accounted for) could rewrite a tracked file between our `git checkout
HEAD --` and our subsequent snapshot, and we'd silently bake the drift
into the baseline and the afterEach restore loop would maintain it
forever.
This commit adds the missing post-heal check immediately after the
`git checkout HEAD --`. On CI it throws with `drifted-baseline guard:
post-heal diff failed` citing every offending path; off-CI it warns so
developers iterating on a dirty tree aren't blocked.
Also:
- Rephrases the `isBenignPathspec` catch comment to describe the
realistic case (belt-and-braces against a rm race, not the normal
flow) now that partitionTrackedPaths pre-filters untracked paths.
The branch is intentionally kept — cheap to tolerate, and it
guards against a race that's hard to rule out in a shared worktree.
- Red-green unit coverage: new tests use a counter-based git shim
that fails the N-th `diff --quiet` invocation, so we can target
the post-heal diff independently of the off-CI pre-checkout diff.
- CI throws on drift with the advertised message
- off-CI warns (does not throw) with the same message
- no false-positive on a clean tracked path
Verified red without the guard, green with it.
- Rewrite restoreFromGitHead JSDoc to match actual behavior for tracked
vs untracked paths (including the on-CI throw / off-CI warn for an
entirely-untracked input list).
- Rephrase internal review-round markers ("CR4/CR5 HIGH/MEDIUM") in
test-cleanup.ts and test-cleanup.test.ts to describe the behavior or
invariant being guarded instead of the review history.
- Point the WINDOWS comment at the sibling test files that actually
invoke npx (this module itself doesn't).
- Drop the "belt-and-braces" afterAll disclaimer from
bundle-demo-content.test.ts and generate-registry.test.ts — the
afterAll(restore) pattern is self-explanatory.
unit (20.x) CI was failing with 'Timeout calling onTaskUpdate' during
the showcase-scripts test run — an unhandled vitest error that causes
ELIFECYCLE after otherwise-green tests. Reproducible on Node 20 only;
22.x and 24.x are green on the same code.
Root cause: vitest's default thread-based worker pool times out on the
parent-worker RPC channel when a test file spawns many subprocesses
(validate-pins.test.ts runs 134 tests each invoking a subprocess;
create-integration / generate-registry / bundle-demo-content each
spawn npx tsx via execFileSync). Under Node 20 the stdio / signal
traffic from these children contends with the worker-thread RPC
channel and surfaces as an unhandled timeout mid-suite.
Switch to pool: 'forks' — the fork pool uses node IPC for the RPC
rather than worker-thread messageports and is robust under the same
load. fileParallelism: false keeps files sequential so shared-env /
tmp-dir mutations don't race, but each file now gets its own fresh
fork so one file's subprocess churn can't stall the RPC for
subsequent files. Node 22/24 unaffected either way.
Also pipes stdio explicitly on every git subprocess in
test-cleanup.test.ts — inherited stdio on a fork vitest worker
interleaves with the worker's own stdout/stderr and is another input
to the RPC contention under Node 20.
generate-registry.ts writes to shell/src/data/registry.json and
shell/src/data/constraints.json. bundle-demo-content.ts writes to
shell/src/data/demo-content.json. All three files are tracked, and
without restoration every run leaks regenerated JSON into the working
tree.
This commit:
- replaces execSync-with-path-interpolation invocations with
execFileSync argv form via runGenerator() / runBundler() helpers;
eliminates any shell-parser involvement (clean hygiene even when
the interpolated constant happens to be safe today).
- snapshots the three data files in beforeAll, restores them in
afterEach + afterAll, adds a regression-guard test that proves the
hooks actually heal drift (sentinel append + bit-exact in-memory
comparison), and a terminal safety-net bit-for-bit check.
- drops a redundant bundler pre-run in beforeAll (test 1 exercises
the bundler itself), and replaces byte-length sentinel checks with
content-level Buffer.concat assertions so the regression guard
survives any hypothetical fs shim that updates stat but not bytes.
create-integration generates an integration package in
showcase/test-integration-tmp and mutates existing workflow YAMLs in
.github/workflows/ (showcase_deploy.yml, showcase_drift-detection.yml,
starter-smoke.yml) to register the new slug. Without restoration, both
the generated tmp package and the workflow-YAML drift leak into the
working tree on every run, and the workflow-YAML drift in particular
breaks pnpm run check on subsequent test invocations.
This commit wires FileSnapshotRestorer + restoreFromGitHead into the
test so:
- tracked workflow files are restored to HEAD before each test run
- the tmp output dir is deleted eagerly in afterEach (independent of
the generator — no reliance on its internal cleanup)
- a regression-guard test proves the snapshot/restore hooks actually
heal drift via a sentinel append + bit-exact assertion against
the in-memory snapshot (not a re-read of disk, which would
silently agree with a buggy restore()).
- a terminal safety net re-asserts every snapshotted file is
byte-identical to its baseline at the end of the suite.
Introduces FileSnapshotRestorer + restoreFromGitHead helpers used by the
showcase test suites to snapshot tracked files in beforeAll and restore
them in afterEach / afterAll. Several of our test scripts invoke real
generators (create-integration, generate-registry, bundle-demo-content)
that write to tracked files outside any tmp dir: .github/workflows/ and
showcase/shell/src/data/*.json. Without explicit restoration these writes
leak into the working tree on every nx run-many -t test and, under Node
20 + vitest worker pools, the accumulated drift races the worker-RPC
channel surfacing as 'Timeout calling onTaskUpdate' -> ELIFECYCLE on CI.
Highlights:
- FileSnapshotRestorer captures bytes at snapshot time, rewrites only
drifted files via atomic temp+rename, and sweeps leftover
.<basename>.<hex>.tmp stragglers scoped to the snapshotted basenames
(no more whole-directory unlink).
- restoreFromGitHead uses execFileSync with a frozen env (GIT_*
scrubbed, PATH/HOME preserved) to heal a working tree left dirty by
a crashed prior run before we snapshot.
- On CI, a baseline that drifts after the pre-snapshot heal is a hard
error (git binary missing, sandbox, etc.); off-CI it warns instead
of blocking local iteration.
- Narrow catch in the git path partitioner: only genuine 'not in
index' pathspec errors are treated as untracked; ENOENT / EACCES /
non-exit-1 failures re-raise so sandbox and missing-binary cases
don't get silently swallowed and lock in a drifted baseline.
- test-cleanup.test.ts itself strips GIT_* from child env when it
creates tmp repos — pre-commit hooks (lefthook) run with GIT_DIR /
GIT_INDEX_FILE set, which would cause tmp-repo 'git commit' calls
to ignore cwd and write to the HOST working-tree HEAD. Without the
scrub, running 'git commit' itself silently accumulates 'initial' /
'init' commits on the real repo.
Also pulls scripts-dir / repo-root / data-dir constants into a shared
paths.ts so future layout changes flip in one place.
M2: The imported `AGENT_URL_LOCALHOST_8000_RE` carries the /g flag, so
`.test()` / `.exec()` advance lastIndex — sharing the exact instance
across the describe-loop iterations coupled any two iterations that
happened to touch it. Clone the regex per-call via a `re8000()` factory
that returns `new RegExp(source, flags)` each time. Same treatment for
the companion `re8123()`.
LOW: Replace the conditional `it()` registration (only created when the
package .env.example contained `:8000`) with an UNCONDITIONAL `it()`
that internally early-returns when there's nothing to rewrite. Vitest's
reporter registers the test either way so a future regression where every
package suddenly stopped matching the pattern would surface as "test
skipped" rather than silently vanishing from CI output.
Extends generate-starters.ts so every starter whose package ships .env.example
gets the file copied through (not just non-langgraph Python), and ships the
aimock_toggle.py alongside agent_server.py for any Python package that has one.
The .env.example copy rewrites AGENT_URL=http://(localhost|127.0.0.1):8000 to
:8123 during propagation because starter dev scripts bind the agent on 8123
while package dev scripts bind on 8000.
The port-rewrite regex is now exported as AGENT_URL_LOCALHOST_8000_RE and
imported by the starter-consistency test, so the two sides cannot drift.
Previously the test used a broader [^:\/]+ host pattern while the generator
correctly narrowed to localhost/127.0.0.1 — a future package documenting
AGENT_URL=https://api.corp.example:8000 would have tripped the test while the
generator correctly preserved the non-localhost hostname.
## Showcase validation tooling (Bundle 3)
Ships three CLI validators, a shared parsing lib, and two CI workflows
that enforce consistency across the 17 showcase packages and detect
drift before it lands on main. Consolidates four earlier tooling PRs
(#3985, #3987, #3995, #3996).
## What's in the box
### `showcase/scripts/` — three validators
| Tool | Purpose | Exit codes |
|------|---------|-----------|
| `audit.ts` | Cross-checks manifest-declared demos against
`tests/e2e/*.spec.ts` and `qa/*.md`, plus `examples/integrations/`
provenance via `SLUG_TO_EXAMPLES` / `FALLBACK_MAP` | 0 ok, 1 anomalies,
2 invalid-input, 3 unreadable, 4 internal, 5 strict-warnings |
| `validate-pins.ts` | Framework-dep pin-drift between
`showcase/packages/*/` and their dojo `examples/integrations/*/`
counterparts. Parses package.json, requirements.txt, pyproject.toml
(Poetry + PEP 621) | 0 ok, 1 drift, 2 internal, 3 unreadable |
| `validate-parity.ts` | Enforces demo ↔ spec ↔ qa coverage per package
with a monotonic demo-count floor | 0 ok, 1 warnings, 2 invalid-input, 3
unreadable, 4 internal, 5 must-failure |
### `showcase/scripts/lib/` — shared primitives
- **`slug-map.ts`** — single-source-of-truth `ENTRIES` for the showcase
slug taxonomy;
`BORN_IN_SHOWCASE`/`SLUG_MAP`/`SLUG_TO_EXAMPLES`/`FALLBACK_MAP` derived
and frozen at module load. `SlugEntry` is a discriminated union that
makes illegal states (born-in-showcase with non-empty examples)
unrepresentable. `freezeSet`/`freezeMap` helpers install throwing
replacements via `Object.defineProperty({writable:false,
configurable:false})` so `Set.add` / `Map.set` truly fail at runtime.
- **`manifest.ts`** — `parseManifest` returns a tagged `ParsedManifest`
union (`ok` | `missing` | `malformed{subkind: "syntax"|"shape"}` |
`unreadable`) with a never-throws content contract. Uses `statSync` +
errno inspection (not `existsSync`, which conflates ENOENT with EACCES).
`DemoId` is a branded string minted only via `createDemoId`.
### `.github/workflows/` — CI enforcement
- **`showcase_validate.yml`** — runs on PR and push-to-main. Enforces
the e2e-spec floor, runs the validators, and drives the pin-drift
ratchet.
- **`showcase_drift-report.yml`** — weekly Monday 10:00 UTC +
workflow_dispatch. Computes `set_status` (OK / SET DRIFTED / COUNT
DRIFTED) and posts to Slack.
Both workflows run on `depot-ubuntu-24.04-4` (Startup plan, unlimited)
for persistent pnpm/npm cache across runs.
## The pin-drift ratchet
`validate-pins.ts` currently finds **111 existing pin-drift failures**
across 12 showcase packages. Rather than block the PR on those, we
baseline them in `showcase/scripts/fail-baseline.json` and ratchet:
- `validatePinsFailCount` must not increase; CI tells you to ratchet
down when it decreases.
- `validatePinsFailHash` is SHA-256 of the sorted-uniqued `[FAIL]` set.
When the count is equal but the hash differs, a fail healed AND a new
one regressed — CI prints the diff and fails.
- `baselineDemoCount` (9) is the single source of truth for the e2e-spec
floor; consumed by both the workflow and `validate-parity.ts` with sync
enforced by a dedicated regression test.
Tracked in #4047. The 111 failures are mostly showcase packages pinning
`@copilotkit/*` to the `next` dist-tag while dojo pins concrete
versions; direction of fix (align showcase → dojo vs. bump dojo →
showcase) is a separate versioning decision outside this PR.
## Correctness posture
- **977 tests**, 13 files, covering every `Anomaly` / `PackageIssue` /
`ParsedManifest` variant in-process and via subprocess CLI for every
exit code. EACCES/ENOTDIR/TOCTOU paths are exercised via chmod probes
(with `it.skipIf` fallback when CI runs as root) and path-filtered
`vi.spyOn` fall-throughs.
- **`fs.statSync` + errno everywhere** — `fs.existsSync` silently
collapses ENOENT with EACCES and is a known anti-pattern in validation
tooling; the codebase uses structured errno discrimination throughout.
- **Tagged discriminated unions with exhaustive `switch` + `never`
guards** — `bucketFor` in `audit.ts`, `deriveMessage` in
`validate-parity.ts`. Adding a new variant without wiring every site is
a compile error.
- **Partial-report preservation** — when an infra error hits
mid-slug-loop, `UnreadableInputError.partialReport` carries
already-collected drift findings so the top-level catch prints them
before exiting 3. One bad package never orphans signal for the rest.
- **Per-slug isolation** — in `validate-parity.ts runParityImpl`, each
slug's audit is wrapped; a crash surfaces as a `crashed` `PackageIssue`
and forces `EXIT_INTERNAL` without aborting siblings.
- **Pipefail + scoped `|| true`** — every workflow step uses `set -euo
pipefail` with grep's no-match tolerance wrapped in `{ grep || true; }`
so producer failures (sort, shasum, cut) still surface.
## Diff
+16,455 / −18 across 34 files (26 source + 5 fixture trees + 2 workflows
+ 1 baseline).
Commits grouped by purpose:
1. `chore(showcase/scripts)`: vitest config + test deps
2. `feat(showcase/scripts)`: shared slug-map and manifest parsing lib
3. `feat(showcase/scripts)`: audit.ts coverage auditor
4. `feat(showcase/scripts)`: validate-pins.ts pin-drift validator
5. `feat(showcase/scripts)`: validate-parity.ts demo/spec/qa parity
validator
6. `ci(showcase)`: validation + weekly drift-report workflows (Depot
runners)
## Test plan
- [x] `pnpm vitest run` in `showcase/scripts/` — 977/977 green
- [x] Exit-code taxonomy verified end-to-end via subprocess tests for
every documented code
- [x] EACCES/ENOENT/ENOTDIR routing verified in all three validators
- [x] Partial-report preservation verified in both in-process and
subprocess paths
- [x] Per-slug crash isolation verified (one broken slug does not orphan
siblings)
- [x] Baseline sync contract (`BASELINE_DEMO_COUNT` ↔
`fail-baseline.json.baselineDemoCount`) pinned by test
- [ ] First CI run on Depot to confirm cold-cache timing (expected 5–8m
vs. 18–20m on ubuntu-latest)
Refs: [Full Action
Inventory](https://www.notion.so/3443aa38185281b5a1dfc6e0890264e1),
#4047
Audits each showcase package's manifest-declared demos for matching
spec (tests/e2e/*.spec.ts) and qa/*.md coverage, and enforces a
monotonic demo-count baseline via fail-baseline.json.
Key design:
- PackageIssue tagged union (13 variants) cleanly separates MUST
errors from warnings; deriveMessage is the single renderer so new
variants cannot emit mismatched prose.
- ProbeResult tagged union (missing | ok | unreadable) driven by
statSync + errno inspection, distinguishing ENOENT from EACCES and
surfacing ENOTDIR as a misconfiguration rather than a silent miss.
- runParityImpl isolates each slug's audit in try/catch; a crash in
one slug surfaces as a crashed PackageIssue variant and forces
EXIT_INTERNAL without aborting siblings.
- runParity never throws for content errors. InvalidBaselineError
covers coerceBaseline failures; unknown errors route to
EXIT_INTERNAL via formatErrorChain (walks .cause with cycle
guard + depth cap).
- parseMainArgs rejects unrecognised flags and duplicate --baseline
with EXIT_INVALID_INPUT (2), mirroring audit.ts parseArgs
discipline.
- BASELINE_DEMO_COUNT default must match
fail-baseline.json.baselineDemoCount; enforced by
__tests__/baseline-sync.test.ts.
- Exit codes: 0 ok, 1 should-warnings-only, 2 invalid-input, 3
unreadable, 4 internal, 5 must-failure.
Tests cover every PackageIssue variant, every exit code
(in-process + subprocess), per-slug crash isolation, EACCES routing,
ENOTDIR classification, cascade suppression when tests/e2e or qa
dirs are unreadable, and baseline coercion edge cases (leading
zeros, negative, float, hex, non-numeric).
Compares framework dependency pins across showcase/packages/*/ and
the corresponding dojo examples/integrations/* trees, flagging drift
between the two and rejecting non-exact specs on the showcase side.
Key design:
- Parses package.json, requirements.txt, and pyproject.toml
(including Poetry's [tool.poetry.dependencies] and PEP 621
[project.dependencies] / optional-dependencies). Separate jsDeps
and pythonDeps maps prevent cross-ecosystem name collisions.
- isExactSpec enforces exact-version pins per ecosystem (npm: no
operators, workspace refs, or ranges; Python: ==X / ===X / ~=X
with PEP 440 body). Symmetric rejection of bare MAJOR-only forms.
- parseRequirementsTxt and parsePyprojectToml thin wrappers throw
when the detailed form produced skipped[] or dropped[] entries,
preventing silent data loss in simpler callers.
- canonicalizeDepMap canonicalises names per PEP 503 and surfaces
same-file collisions with differing specs as warnings.
- First-writer-wins at both file and package levels.
- UnreadableInputError carries an optional partialReport so an
infra failure mid-slug-loop preserves already-collected drift
findings for other slugs.
- Exit codes: 0 ok, 1 drift, 2 internal, 3 unreadable.
fail-baseline.json is the single source of truth for the CI ratchet
(validatePinsFailCount + validatePinsFailHash) and the demo-count
floor (baselineDemoCount, cross-checked against validate-parity.ts
in a dedicated sync test).
Test coverage spans every parser variant, EACCES routing via chmod
probe + fs spies, exit-code taxonomy subprocess tests, partial-report
preservation on mid-loop infra throws, and Poetry/PEP 503 edge cases
via committed fixture files under __tests__/fixtures/pins/.
Cross-checks each showcase package's manifest-declared demos against
the spec (tests/e2e/) and qa/ directory contents, plus
examples/integrations provenance via SLUG_TO_EXAMPLES / FALLBACK_MAP.
Key design:
- Discriminated Anomaly union with nine variants
(count-mismatch, not-deployed, missing-examples, missing-manifest,
malformed-manifest, unreadable-dir, unreadable-manifest,
unreadable-examples, mapped-candidate-not-directory). bucketFor uses
an exhaustive switch with a never guard so a new variant cannot
silently escape routing.
- CountState tagged union separates known-count, legitimate-missing,
and unreadable cases so an EACCES on tests/e2e/ cannot be
misclassified as a real zero count.
- ExamplesSourceResult carries structured unreadableForSlug /
nonDirectoryForSlug flags; classification never substring-matches
the human-readable warning text.
- SHOWCASE_AUDIT_ROOT env var is validated with statSync + distinct
error messages for ENOENT vs ENOTDIR vs EACCES.
- Text and --json output modes. Exit-code taxonomy: 0 ok, 1 anomalies,
2 invalid-input, 3 unreadable, 4 internal, 5 strict-warnings.
- Deep-freezes AuditReport.packages and anomalies before return.
Tests cover every Anomaly variant, every exit code (in-process + CLI
subprocess), buildReport bucket exhaustiveness, EACCES routing via
path-filtered fs spies, TOCTOU ENOENT races, and the --columns filter
surface.
Rework the Controlled Generative UI demo to be minimal, self-contained,
and render charts via `useComponent`.
Frontend (`src/app/demos/gen-ui-tool-based/`):
- `page.tsx` -- `CopilotKit` + full-screen `CopilotChat` (no sidebar),
two `useComponent` registrations (`render_bar_chart`,
`render_pie_chart`), three chart-oriented suggestions.
- `bar-chart.tsx`, `pie-chart.tsx` -- ported from
`examples/integrations/langgraph-python`. Each file reads top-to-
bottom as imports -> schema -> props type -> component; colors and
animation helpers are inlined inside the component. No shared
chart-config module.
- Haiku card + schema removed.
Agent (`src/agents/main.py`):
- Rewritten using `create_agent` + `CopilotKitMiddleware()` so the
LangGraph middleware injects the frontend `render_*_chart` tools into
the model request at runtime.
- Deleted all backend-tool modules (`tools.py`, `todos.py`,
`a2ui_dynamic_schema.py`, `a2ui_fixed_schema.py`). System prompt
trimmed to a data-viz assistant.
Peripheral cleanups:
- `demos/agentic-chat/page.tsx` -- `change_background` tool + its
suggestion removed; it had nothing to do with chat.
- New `demos/frontend-tools/` (In-App Actions) hosts
`change_background` as its own demo with a `frontend_tools` agent
name registered in `api/copilotkit/route.ts`.
- Stub `gen-ui-tool-based/agent.py` removed.
- `generate-starters.test.ts` no longer requires every python package
to have backend tool imports -- a package with only frontend tools
(like this controlled-gen-ui demo) is now valid.
- Whitelist generated shell data (`demo-content.json`,
`search-index.json`, `starter-content.json`) in check-binaries hook
-- they're generated artifacts like `package-lock.json`.
- Regenerated `demo-content.json` + `registry.json`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add --validate-on-load to all aimock invocations (4 workflows/scripts
+ 13 integration docker-compose files)
- Replace hardcoded 2-file fixture list with dynamic discovery across
showcase/, examples/integrations/*/, scripts/doc-tests/ (16 fixtures)
- Add sanity check to prevent silent zero-test pass when discovery fails
- Extend showcase_validate.yml path filter to trigger on
examples/integrations/**/fixtures/** and scripts/doc-tests/fixtures/**
- Import and use ValidationResult type for callback parameters
- Fix scripts/doc-tests/fixtures/default.json to use { fixtures: [...] }
envelope shape
Follow-up to #3971. aimock supports fixture schema validation at startup via
--validate-on-load, but it's opt-in. The showcase Dockerfile did not pass
the flag, so fixtures with unrecognized response keys (e.g. "text" instead
of "content") loaded silently and only failed at request time with HTTP 500.
That's what crashed crewai-crews on startup.
Changes:
- showcase/aimock/Dockerfile: pass --validate-on-load so broken fixtures
fail the container boot, not individual requests.
- showcase/scripts/__tests__/aimock-fixtures.test.ts: new vitest spec that
loads feature-parity.json and smoke.json via @copilotkit/aimock's
loadFixtureFile + validateFixtures and asserts zero errors. Runs as part
of the existing showcase-validate CI workflow.
- showcase/scripts/package.json: add @copilotkit/aimock dependency for the
validator import.
Verified red-green: with the pre-#3971 broken "text" fixtures, validateFixtures
flags 5 errors; post-#3971 it returns zero. Docker red-green: container with
an intentionally broken fixture fails to start with "Validation failed: 1
error(s)" and non-zero exit.
## Summary
Addresses all findings from 7-agent MSAL code review of the starter
crash fix PRs (#3933, #3941, #3943).
**Agent server (claude-sdk-typescript):**
- Remove 6 unnecessary `as any` casts — use TypeScript discriminated
union narrowing on `msg.role`
- Add `ANTHROPIC_API_KEY` validation on startup (exit if missing)
- Add `console.warn` logging to empty catch blocks (tool args + schema
parse failures)
- Sanitize error messages sent to client — no raw `err.message` in SSE
stream
- Fix misleading comment about system/developer role handling
- Set explicit 2MB JSON body limit on Express
- Remove unused `assistantMsgId` variable
**Generation script:**
- Remove dead `_agentDir` parameter from `rewritePythonImports`
- Add warning when `PIN_OVERRIDES` dep not found in framework
dependencies
**Tests (3 new):**
- Context-aware import rewriting for files inside `tools/` directory
- `from agents.X import` relative import rewriting
- `PIN_OVERRIDES` integration test for mastra version pinning
## Test plan
- [x] 254 generate-starters tests pass (3 new)
- [x] Drift check passes
- [x] Pre-commit hooks pass
Agent server (claude-sdk-typescript):
- Remove 6 unnecessary as-any casts, use TS discriminated union narrowing
- Add ANTHROPIC_API_KEY validation on startup (exit if missing)
- Add logging to empty catch blocks (tool args + schema parse)
- Sanitize error messages sent to client (no raw err.message in SSE)
- Fix misleading comment about system/developer role handling
- Set explicit 2MB JSON body limit
- Remove unused assistantMsgId variable
Generation script:
- Remove dead _agentDir parameter from rewritePythonImports
- Add warning when PIN_OVERRIDES dep not found in dependencies
Tests:
- Add context-aware import rewriting test (files inside tools/)
- Add from-agents.X relative import rewriting test
- Add PIN_OVERRIDES integration test for mastra version pinning
The generate-starters script was flattening src/main/{java,resources} into
agent/{java,resources}, breaking Maven builds. Add a post-copy step that
moves them to agent/src/main/{java,resources} as Maven expects.
The langgraph-python starter crashes on Railway because of a directory
structure mismatch: agents were in agent/ with langgraph.json inside
agent/, causing langgraph_cli to fail resolving module paths.
Match the demo package's working layout:
- Move agents from agent/ to src/agents/ (same as demo)
- Put langgraph.json at project root (same as demo)
- Point entrypoint at root langgraph.json instead of agent/langgraph.json
- Templatize Dockerfile.python with AGENT_DIR and DOCKER_EXTRA_COPY vars
- Skip langgraph.json path rewriting when agentDir matches agentSourceDir
- Update tests to use framework agentDir instead of hardcoding "agent"
Regenerate all 17 starters.
Atai Barkai