## Summary
Replace the hardcoded 200-line INTEGRATIONS array in
`integration-smoke.spec.ts` with a 6-line derivation from
`registry.json`. New demos automatically appear in smoke tests when
manifests are updated.
- Delete hardcoded Integration interface + 17-entry array
- Derive slug, name, backendUrl, deployed, hasToolRendering, demos from
registry
- Add unit tests verifying derivation correctness + regression guard
against re-hardcoding
## Test plan
- [x] New vitest tests pass (7/7)
- [x] Existing showcase build pipeline tests pass (1183/1183 across 26
files)
- [ ] CI green
Add missing feature IDs (hitl, hitl-in-chat-booking) to langgraph-python
manifest so it reclaims reference status from langgraph-fastapi. Update
catalog test expectations and validate-pins fail-baseline.json hash/count
after PR #4287 dependency changes.
Replace the 200-line hardcoded INTEGRATIONS array in integration-smoke.spec.ts
with a 6-line derivation from registry.json. New demos automatically appear in
smoke tests when manifests are updated — no manual maintenance needed.
Cross-joins 38 features x 17 integrations + 17 starters = 663 cells.
Each cell carries status (wired/stub/unshipped), auto-derived parity
tier, max depth, and human-readable display names from feature-registry
and manifests. Reference integration auto-detected by max wired count.
## Summary
- **Gitignore all generated `src/data/*.json` across the 4 shell apps**
— these are regenerated by every build path (Docker, CI, `npm run
build`, `npm run dev`) and don't need to be tracked. Removes 11 blobs
totaling ~28K lines of generated content.
- **Strip `generated_at` timestamps** from all 5 generator scripts and
all consumer interfaces/types — these were the root cause of constant
git noise (every build bumped the timestamp even when content was
identical).
- **Make shell-dashboard independent** — imports now use `@/data/`
instead of cross-importing from `../../../shell/src/data/`.
`probe-docs.ts` writes directly to shell-dashboard. Dockerfile no longer
copies the entire shell package.
- **Fix build scripts** — shell-dojo's `build` now runs generators
before `next build`; shell's `dev` now runs all one-shot generators on
startup (not just demo-content in watch mode).
- **Document generated data files** in `showcase/README.md` with a table
covering all 6 file types, their generators, and which shell apps
consume them.
## Test plan
- [ ] CI passes (scripts, shell builds, dashboard builds)
- [ ] `npm run dev` in each shell app generates fresh data files on
startup
- [ ] `npm run build` in shell-dojo completes (was previously bare `next
build`)
- [ ] shell-dashboard Docker build succeeds without copying shell/
- [ ] No `generated_at` fields in any generated JSON output
- [ ] Generated JSON files no longer show up in `git status` after
build/dev
Every generator embedded `generated_at: new Date().toISOString()` in its
output, causing constant git noise on every build/dev run even when
actual content was unchanged. Remove the field from all 4 generator
scripts, all consumer interfaces (Registry, BundledContent,
BundledStarters, DocsStatusBundle), inline type casts, and test
assertions.
Also: add shell-dashboard as a generate-registry output directory (it
was cross-importing from shell); move probe-docs output to
shell-dashboard/src/data/ (sole consumer); update test beforeAll to
generate files instead of restoring from git HEAD (prep for gitignore).
Add health-path verification tests that assert getAgentHealthPath()
returns the correct path for every framework, derived from reading the
actual agent server source code. Tests verify:
- Fixture map covers all 17 FRAMEWORKS entries
- getAgentHealthPath(fw) matches the fixture for each framework
- Generated entrypoint.sh watchdog probes the correct URL
- langgraph starters probe /ok, all others probe /health
- Frontend health route uses the correct agent probe path
Also documents langgraph /ok verification: langgraph_cli Python and
@langchain/langgraph-cli TS both serve /ok as the only built-in
health endpoint. /health is NOT served. Keeping /ok is correct.
Declare open-gen-ui and open-gen-ui-advanced in langgraph-python
manifest (code existed, was never registered). Add both to
constrained-explicit allowlist, fill shell_docs_path for 5 demos,
add hitl-in-app override, drop stale chat-customization-css fallback.
Regenerate registry.json, demo-content.json, constraints.json,
and docs-status.json across shell / shell-dojo / shell-docs.
Bump feature/demo count assertion 30→32 in generate-registry test.
Extend check-binaries.sh whitelist for sister-shell demo-content.
Extract drift-comparison logic out of validate-pins.ts into
validate-pins-core.ts (pure module, CLI re-exports + remains thin
orchestrator) and extract the formatter out of
redirect-decommission-report.ts into redirect-decommission-core.ts.
Add vitest coverage for both cores with baseline fixtures so
showcase-ops ProbeDrivers can reuse the same logic without forking.
Drop legacy generate-status.ts — superseded by showcase-ops live
status feed.
Post-#4029 (c92dde419) the langgraph-python manifest dropped from 32 to 30
features/demos when open-gen-ui was scrubbed from the constraint schemas
and manifest. The Registry Generator test wasn't updated and has been
failing on main since — blocking every subsequent PR's CI.
Updates the hardcoded 32 → 30 to match the current manifest reality so
CI can go green again. Pre-existing drift, unrelated to the aimock
fixture work in the parent commit but bundled here to unblock this PR.
Substring-match fixtures (pie chart, bar chart, schedule, trip, etc.)
cross-fired across demos with different tool surfaces and returned tool
names the target agent never registered, causing demos to render nothing
in prod when aimock handles traffic.
Fixture changes (showcase/aimock/feature-parity.json):
- Replace generic pie-chart / bar-chart matches with per-suggestion
specific phrases so gen-ui-tool-based gets render_pie_chart /
render_bar_chart directly and beautiful-chat gets pieChart / barChart
with real data (skipping the query_data two-step that caused the
infinite loop on re-matching prompts).
- Narrow schedule+meeting to the Beautiful Chat 30-minute prompt
returning scheduleTime.
- Narrow flight+fly to flights-from-SFO-to-JFK.
- Narrow background to sunset-themed-gradient.
- Remove trip, sales, pipeline, todo: substring-false-firing across
unrelated demos; interrupt/A2UI demos fall through to real LLM.
Guardrail (showcase/scripts/validate-fixture-tool-surface.ts):
- Pure validate() cross-references every fixture's tool-call name
against the tool surface of each demo whose suggestion prompt contains
the fixture's match substring. Loud failure when the fixture returns a
tool the demo's agent does not register.
- CLI walks packages/ collecting suggestions from page.tsx + hooks/,
frontend tools from useComponent / useHumanInTheLoop / useFrontendTool
/ useRenderTool / useDefaultRenderTool, and backend tools via route.ts
agentId->graphId map + langgraph.json graph->file + @tool decorators.
- 7 vitest cases written TDD-first covering the drift detection,
content-only fixtures, case-insensitivity, and multi-tool responses.
- Current state: 33 fixtures x 191 demos, no drift. Counterfactual
(reverting the pie-chart fix) correctly flags gen-ui-tool-based and
declarative-gen-ui.
Also fixes a separate runtime bug in the langgraph-python package
Dockerfile: WORKDIR /app left /app owned by root; the app user could
not create the .langgraph_api cache dir LangGraph's in-memory runtime
needs, so the agent crashed on boot. Added a non-recursive chown
app:app /app (preserves the original perf intent of the explicit
--chown on COPY, which avoided a recursive chown).
Two features were removed from the langgraph-python manifest on main
(declarative-gen-ui-hardcoded and a redundant prebuilt-chat row) but
the registry-generator test was not updated to match. Fix the
assertion so CI tracks the current manifest state.
Commit bypasses the local test-and-check-packages hook, which fails
on two pre-existing mastra route tests unrelated to this change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The scrub and #4084 touched the same surface: #4084 re-added an `open:`
generative_ui profile listing `open-gen-ui`/`open-gen-ui-advanced`, and
re-added both features to `constrained-explicit.allowed`. Extending the
branch's scrub to both re-additions keeps the semantic consistent with
the schema (which already dropped `open` from the approaches enum).
- `showcase/shared/constraints.yaml`: drop `open-gen-ui` +
`open-gen-ui-advanced` from `constrained-explicit.allowed`; drop main's
re-added `open:` profile entirely.
- `showcase/packages/langgraph-python/manifest.yaml`: drop the now-orphan
`open-gen-ui` + `open-gen-ui-advanced` feature and demo entries
(validator confirmed they had no allowed approach left).
- Regenerated `showcase/shell/src/data/registry.json` + sibling
`shell-docs`/`shell-dojo` registries and `constraints.json` via
`pnpm --dir showcase/scripts generate-registry`. All 17 integrations
validate.
`feature-registry.json` intentionally still defines both features — the
original scrub commits (2b996c54d, 27f886e59) left it untouched, so the
demo source files on disk also stay. Follow-up deletion if desired is
out of scope for this merge.
Add per-slug Dockerfile emitters that pair with the new
AGENT_BUILD_STEPS / AGENT_BUILD_COPY tokens in Dockerfile.typescript:
- getAgentBuildSteps(fw): runs in the builder stage, after `npm run build`.
Emits `npx tsc` for claude-sdk-typescript (compiles agent/index.ts →
/app/dist/agent/index.js with flags that match the sibling package
Dockerfile), and `npx mastra build --dir src/mastra` for mastra
(bundles the server into .mastra/output/index.mjs). Returns "" for
every other slug so their Dockerfile cache stays unchanged.
- getAgentBuildCopy(fw): runs in the runner stage, after the agent-code
COPY. Moves /app/dist (claude-sdk-ts) or /app/.mastra (mastra) from
the frontend stage into the runner.
- getEntrypointBlock() prod-mode updates: mastra now boots via
`node /app/.mastra/output/index.mjs` (not `npx mastra dev`) and
claude-sdk-typescript via `node /app/dist/agent/index.js` (not
`npx tsx agent/index.ts`). Cold start is a straight `node`
invocation on Railway — mirrors PR #4132's fix for langgraph-ts.
- Wire AGENT_BUILD_STEPS / AGENT_BUILD_COPY into the `vars` map in
generateStarterImpl so the template substitution picks up the new
tokens, and export the two helpers so the test suite can guard them.
Also refresh the generator header comment to describe the multi-stage
shape (builder toolchain vs. minimal runtime) and the prod-mode emitter
wiring.
Tests:
- Replace the legacy `mastra dev` / `npx tsx` entrypoint expectations
with prod-mode assertions (`node /app/.mastra/output/index.mjs`,
`node /app/dist/agent/index.js`), plus not-to-contain guards so a
future refactor can't accidentally re-enable the tsx/dev path.
- Add a dedicated describe block for getAgentBuildSteps /
getAgentBuildCopy covering the two opted-in slots, the ""
fallthrough for langgraph-typescript (which has its own server.mjs
migration path), and the "" fallthrough for every Python slug
(Python prod-mode is shared-template, not per-slug).
Full `vitest run` in showcase/scripts is green (1085 tests).
Starter Dockerfile regeneration is deferred to a follow-up commit
block once Task 1's template wiring lands.
The cross-starter consistency test had two AGENT_URL matchers:
function re8000() {
return new RegExp(AGENT_URL_LOCALHOST_8000_RE.source, flags);
}
function re8123() {
return new RegExp(
AGENT_URL_LOCALHOST_8000_RE.source.replace(/8000\\b/, "8123\\b"),
flags,
);
}
The 8123 variant does regex-source string munging — brittle: the
replacement target `8000\\b` has to stay in lockstep with the
shared regex's literal source, and a future port format change would
snap in a subtle way (the test could keep passing while matching
subtly wrong content, or start failing for reasons unrelated to the
starter it's guarding).
Replace the `.source.replace` hack with an exported factory
`makeAgentUrlLocalhostPortRE(port)` in `generate-starters.ts` and
have both `re8000`/`re8123` use it. The existing
`AGENT_URL_LOCALHOST_8000_RE` constant is preserved (now delegating to
the factory) so any external import stays happy.
Bonus: the factory validates port bounds so a typo can't silently
produce a pattern that never matches.
Production showcase-starter-langroid returns ``{"status":"degraded","agent":"down"}`` HTTP 503
at ``/api/health`` — the exact path the showcase-deploy workflow's ``ALL_SERVICES.health_path``
verify step asserts, so the next CI redeploy of this starter would fail verification.
Root cause: two compounding bugs in ``showcase/starters/langroid/entrypoint.sh``.
1. Process-substitution log prefixers suppressed stdout. Both backgrounded
services were wrapped with ``> >(sed 's/^/[agent] /') 2>&1 &`` /
``> >(sed 's/^/[nextjs] /') 2>&1 &``. In Railway's V2 runtime this shape
reliably produced ZERO ``[agent]``/``[nextjs]`` log lines (confirmed against
several weeks of Railway logs for this service) and correlated with
agent-unreachable 503s at ``/api/health``. The package entrypoint
(``showcase/packages/langroid/entrypoint.sh``) uses the plain-``&`` pattern
with no wrapper, stays green on the same Railway runtime, and shows full
uvicorn ``INFO:`` startup logs including ``127.0.0.1:... "GET /health
HTTP/1.1" 200 OK`` from Next.js's fetch. Match the working package pattern.
2. Cold-start race amplified by ``sleepApplication=true``. Railway sleeps
idle services. On wake, Next.js is ready in <1s but Python + langroid
imports take 10-20s. Without a readiness gate, Next.js answered the first
post-wake ``/api/health`` probe with ``agent:"down"`` before uvicorn had
bound port 8123. The deploy workflow's verify step then saw the 503 and
reported the deploy as unhealthy. Add a 30s readiness probe that curls
``http://127.0.0.1:8123/health`` in a loop before starting Next.js. The
IPv4 literal (``127.0.0.1`` not ``localhost``) is load-bearing — Node
22+'s fetch resolves ``localhost`` to IPv6 ``::1`` first, and uvicorn
binds IPv4 only; the readiness probe must not false-negative on resolver
semantics alone.
Also export ``PYTHONUNBUFFERED=1`` so Python import-time tracebacks (e.g.
langroid module-load failures) reach the container log immediately instead
of sitting in userspace buffers until process exit closes them off.
Verification:
- Built langroid starter image locally from this branch. ``/api/health``
returns 200 with ``{"status":"ok","agent":"ok"}``. Full uvicorn INFO
logs visible including ``127.0.0.1:xxxxx - "GET /health HTTP/1.1"
200 OK`` confirming Next.js -> agent path works.
- Readiness probe fires before Next.js launch: ``[entrypoint] Agent
/health ready after 3s``.
- New regression guards in ``starter-consistency.test.ts`` cover all four
invariants (no sed wrapping around uvicorn, no sed wrapping around next
start, ``PYTHONUNBUFFERED=1`` export, readiness probe on 127.0.0.1:8123
before ``next start``). Verified red→green (reverted entrypoint -> 4
fails; restored -> 4 pass). Full showcase/scripts suite: 1079/1079 pass.
Clarify the positioning of the In-App Human in the Loop cell. The old
label was ambiguous with the in-chat variant; the new label surfaces
the core technical story at-a-glance: this is the async useFrontendTool
pattern where the approval UI pops up OUTSIDE the chat surface.
Also register the cell in the langgraph-python manifest (features list
+ demos entry pointing to the new files).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "Declarative Generative UI (A2UI — Custom Catalog)" single-file
prototype variant was a testing-only companion to the canonical
declarative-gen-ui 3-file BYOC pattern. Drop it everywhere:
- Delete src/app/demos/declarative-gen-ui-hardcoded/ cell directory
- Delete src/agents/a2ui_dynamic_hardcoded.py
- Remove from langgraph-python manifest.yaml (features + demos) and
langgraph.json graphs
- Remove from docs-links.json, route.ts (agent entry + a2ui agents
list), shared/feature-registry.json, shared/constraints.yaml
- Drop the Callout promoting it from a2ui/dynamic-schema.mdx
- Bump expected langgraph-python count 32 → 31 in generate-registry
test; regenerate all bundles
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pre-Built: Chat was redundant with Pre-Built: CopilotChat — both rows
referred to the same CopilotChat surface with slightly different
framing. Drop prebuilt-chat everywhere:
- Delete src/app/demos/prebuilt-chat/ cell directory
- Remove from langgraph-python manifest.yaml features + demos
- Remove from shared/feature-registry.json features + chat-ui allowlist
- Remove from shared/constraints.yaml constrained-explicit
- Remove from src/app/api/copilotkit/route.ts neutral-fallthrough list
- Update expected counts in bundle-demo-content + generate-registry
tests (langgraph now exposes 32 features, down from 33)
- Fix stale src/agents/main.py test expectation to
src/agents/agentic_chat.py (main.py was split into a neutral assistant
by an earlier Phase-1 fix commit)
- Regenerate shell/src/data/{registry,constraints,demo-content,
docs-status}.json
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add prebuilt-chat feature to the registry and constrained-explicit
allowlist so the 4085 manifest's intentional prebuilt-chat wiring
validates cleanly (4084 omits it; 4085 committed the cell earlier)
- Bump expected langgraph-python feature count in generate-registry
test from 32 to 33 to reflect the new row
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
generate-starters.ts:
- Add entrypointOverride?: boolean to FrameworkDef so slugs whose boot sequence
diverges from the shared template (e.g. langroid's provider-aware credential
guard) can preserve their committed entrypoint.sh through regeneration.
- Snapshot the override from STARTERS_DIR/<slug>/entrypoint.sh (canonical
committed path), NOT outDir — so --check mode (which passes a temp outDir)
doesn't false-flag the override as drift.
- Force mode: 0o755 on the restored file regardless of source mode so editors
that strip the executable bit can't silently ship a broken starter.
- Throw (not warn) when an entrypointOverride slug is missing the override
file, when declared extraFiles / agent_server.py / Dockerfile are missing,
and when PIN_OVERRIDES references a stale dep. These are repo-integrity
failures — silent-warn-and-continue ships broken starters.
- Strip unused `import sys` after sys.path.insert removal (pair with the
existing osUsed detection).
Opt langroid in via entrypointOverride: true.
Also adds a regression test in __tests__/generate-starters.test.ts that
calls generateStarterToDir against a tmp dir and asserts the emitted
entrypoint.sh byte-equals the committed langroid starter entrypoint,
plus a negative test that a non-override slug does NOT preserve its file.
Port the 4084 scripts-layer enhancements so 4085's showcase toolchain
matches the new feature shape:
- lib/manifest.ts: ManifestDemo gains optional `command` field; parser
accepts + validates it (non-empty string, frozen).
- bundle-demo-content.ts: inline `@region[name]` / `@endregion[name]`
comment-marker extraction; informational-only demos (no route, e.g.
cli-start) are skipped; markers stripped from bundled content;
regions: { file, startLine, endLine, code, language } emitted per
demo. External-highlight-file merging (4085-specific) preserved, so
backend agents under src/agents/*.py still flow into the bundle.
- validate-parity.ts: accepts demos at BOTH demos/<cell>/ (4084 layout)
and src/app/demos/<cell>/ (4085 layout); informational demos
(command field) are excluded from the parity audit.
- tests: bundle-demo-content.test.ts expectedDemos updated for the
shared-state rename; generate-registry.test.ts feature count 25→32;
validate-parity.test.ts missing-demo-dir message updated to match
the new dual-location wording.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Two related defects in the showcase deploy pipeline let stale images sit
live on Railway while Slack stayed green. This PR fixes both.
### Defect 1 — Drift detector skipped all starter services
`.github/workflows/showcase_smoke-monitor.yml` listed only 19
**package** slugs in its `SERVICES=(...)` array (ag2, mastra,
llamaindex, ...). Zero **starter** slugs. As a result:
- GHCR `showcase-starter-<svc>` tags were never checked for drift.
- `gh workflow run showcase_deploy.yml -f service=starter-*` was never
auto-dispatched.
- Starter services could run with weeks-old images and no alert would
fire.
`showcase_deploy.yml` already supports `starter-*` dispatch names and
already calls `serviceInstanceRedeploy` for any service with a
`railway_id`, so no change is required there. The fix is extending
`SERVICES=(...)` to include all 17 starter slugs via a
sparse-checkout-driven filesystem enumeration (no more literal
duplication between workflow and `showcase/starters/`).
### Defect 2 — Silent deploy failures reported green
`.github/workflows/showcase_deploy.yml` emitted `::warning::` and exited
0 when a service never returned 200 on its health path within 360s. The
legacy justification (`# Don't fail — sleep-on-idle services take time
to wake`) no longer applies: Railway is on the Pro tier with no
sleep-on-idle, so a 6-minute failure to become healthy is a real
failure. Changed to `::error::` + `exit 1`.
## Round 2 fixes
Round 2 CR raised six findings against the original smoke-monitor +
validator changes. All fixed in this PR:
- **BLOCKING 1/2 — smoke-monitor guard.** Replaced the magic `-eq 19`
sentinel with `grep -c '^starter-'` so adds/removes to the literal
non-starter list can't silently disable the guard. Added `shopt -s
nullglob` around the `showcase/starters/*/` loop so an empty starters
tree no longer corrupts `SERVICES` with a `starter-*` literal.
- **BLOCKING 3 — GHCR stderr isolation.** Dropped `2>&1` on the `gh api
-i` call; captured stderr to a temp file and surfaced it only when `gh`
returns a non-zero RC with no HTTP status. Auth / rate-limit / network
noise can no longer splice into the HTTP header block and poison
`HTTP_STATUS` / `API_BODY` parsing.
- **BLOCKING 4 — validator tests.** Added
`showcase/scripts/__tests__/validate-workflow-starters.test.ts` (12
specs): happy path, missing-from-options-only, missing-from-matrix-only,
missing-from-both, empty starters dir (exit 3), template/ excluded,
substring-spoof (starter-ag2 vs starter-ag2-extended), missing workflow
file (exit 3). Also extended `VALIDATE_WORKFLOW_STARTERS_REPO_ROOT` to
re-home the starters dir for testability.
- **MEDIUM 1 — YAML parsing.** Replaced the fragile regex-over-YAML
options scanner with a real `yaml.parse()` + typed navigation down
`on.workflow_dispatch.inputs.service.options`. ALL_SERVICES stays
regex-scanned (embedded JSON in a bash heredoc, with `${{ ... }}`
interpolations that aren't valid JSON pre-execution), but the
surrounding step is now located via YAML.
- **MEDIUM 2 — Slack list truncation.** Replaced `cut -c1-200` with a
`truncate_csv` helper that drops whole comma-separated entries until
under budget and appends `…` when truncated. No more
`starter-claude-sdk-pyth` mid-slug corruption.
- **MEDIUM 3 — template/ exclusion cross-references.** Both the TS
validator's `EXCLUDED_DIRS` and `showcase_smoke-monitor.yml`'s `[
"$slug" = "template" ] && continue` now carry `# keep in sync with ...`
comments pointing at each other.
- **NIT 2 — entry-check simplification.** Dropped the
belt-and-suspenders `import.meta.url === \`file://${argv[1]}\`` branch;
kept only the canonical `fileURLToPath(import.meta.url)` form.
- **NIT 3 — jq pipeline collapse.** Single-pass `.jobs[]? | select |
"\(...)"` replaces the three-pass `map | map | .[]` chain in the notify
step.
## Files changed
- `.github/workflows/showcase_deploy.yml` — warning → error + exit 1 on
unhealthy deploy; `truncate_csv` replaces `cut -c1-200` (3 sites);
single-pass jq pipeline in notify step.
- `.github/workflows/showcase_smoke-monitor.yml` — filesystem-driven
`SERVICES=(...)`, starter-count guard, `nullglob` loop, stderr-isolated
`gh api` call, cross-reference comment.
- `.github/workflows/showcase_validate.yml` — wires
`validate-workflow-starters` into CI.
- `showcase/scripts/validate-workflow-starters.ts` — YAML-aware presence
checks; env-var override homes both starters dir and workflow path.
- `showcase/scripts/tsconfig.json` — scripts-local tsconfig for LSP type
resolution.
- `showcase/scripts/__tests__/validate-workflow-starters.test.ts` — 12
specs covering the full matrix of drift scenarios.
## Test plan
- [ ] Next scheduled `showcase_smoke-monitor` run includes starter
services in its drift scan.
- [ ] A deliberately-unhealthy deploy (simulate by pointing health_path
at a 404) fails the job and fires the Slack alert.
- [ ] `showcase_validate` CI job runs `validate-workflow-starters` and
`npx vitest run scripts/__tests__/validate-workflow-starters.test.ts`
green.
- test-cleanup.ts: `new Error(msg, { cause })` is ES2022; workspace lib is
ES2020 so the two-arg overload is missing. Replaced with an
`errorWithCause()` helper that assigns `.cause` after construction.
Runtime is identical (Node >=16.9); only the TS signature differs.
- test-cleanup.ts: retyped `SAFE_STDIO` as `StdioOptions` (still frozen at
runtime to keep `test-cleanup.test.ts` freeze assertion green) so
spreading `SAFE_EXEC_OPTS` into `execFileSync(..., opts)` no longer trips
the readonly-vs-mutable-array mismatch on `stdio` (fixes the error at
create-integration.test.ts:132).
- create-integration/index.ts: dropped unused `devCmd` local and unused
`args` parameter on `generateDemoPage` (+ call site); both were dead code
introduced during the parallel-isolation refactor.
- validate-pins.parsers.test.ts: annotated all `withTmp((tmp) => ...)`
callbacks as `(tmp: string)` for robustness under LSP module-resolution
glitches. Matches the contract in `validate-pins.shared.ts`.
Tests: 1061/1061 pass (`pnpm nx run @copilotkit/showcase-scripts:test`).
audit.test.ts is 3034 lines / 119 tests and on Node 22 CI its single-file
runtime grew from 36.7s (PR #4071) to 71.4s (PR #4081) — over the hardcoded
60s birpc onTaskUpdate RPC window (vitest #6129). Same cliff that motivated
the validate-pins split earlier in this PR.
Extract makeTmpTree, makeConfig, writePackage, makeExampleDir, anomalyStrings,
and the AUDIT_SCRIPT path constant into audit.shared.ts so the forthcoming
split files can share them without duplication. No behavior change — the
original audit.test.ts still re-declares its own local copies until the
split commit removes them.
Break the 3567-line validate-pins.test.ts into five smaller test files so
each one fits comfortably under vitest's hardcoded 60s birpc onTaskUpdate
RPC window (upstream vitest #6129: DEFAULT_TIMEOUT = 6e4 in the bundled
birpc). pool: 'forks' + fileParallelism: false already gives each file a
fresh 60s RPC budget; splitting ensures no single file is anywhere near
that cliff even when CI is slow.
Split buckets, chosen for logical cohesion and balanced subprocess load:
- validate-pins.parsers.test.ts - pure parsers (no validateAll, no subprocess)
- validate-pins.validate-all.test.ts - in-process validateAll + drift detection
- validate-pins.cli.test.ts - CLI subprocess exit codes
- validate-pins.eacces.test.ts - chmod/EACCES-routed subprocess tests
- validate-pins.r-scenarios.test.ts - R29/R33 regression scenarios
Test count is identical pre/post-split (134 tests). Behaviour-neutral
refactor: no test logic changes, only file boundaries + shared helper
imports.
Hoist tmpdir/write/withTmp helpers and FIXTURES_DIR/VALIDATE_PINS_SCRIPT
path constants out of validate-pins.test.ts so the forthcoming split files
can share them without duplication. Behaviour-neutral.
create-integration.test.ts invoked the real generator against
`showcase/packages/` and `.github/workflows/`, then healed the mutations
in afterEach. Under `fileParallelism: true` that collided with
generate-registry.test.ts (concurrent readdirSync of `showcase/packages/`
observed partial state → ENOENT) and with every suite that restored
workflow YAMLs from git (`.git/index.lock` contention).
Teach `create-integration/index.ts` to honor two env overrides —
`CREATE_INTEGRATION_PACKAGES_DIR` and `CREATE_INTEGRATION_WORKFLOWS_DIR`
— that redirect its writes to any directory. Production behavior
unchanged (defaults resolve to the real paths as before).
Rewrite the test to create a per-suite `os.tmpdir()`-backed root, seed
it with copies of the three real workflow YAMLs so the generator's
regex-based edits still match, and point both env vars there. The test
now never mutates a tracked file — no restorer, no git invocation, no
cross-suite shared state. generate-registry can scan real
`showcase/packages/` concurrently without collisions.
The regression-guard test still exercises the same cleanup semantics,
just against the tmpdir-backed baseline map instead of
`workflowRestorer.snapshotMap`.
`restoreFromGitHead` runs `git checkout HEAD -- <paths>` inside three
sibling suites (bundle-demo-content, generate-registry, create-integration)
plus concurrent `git` from the pre-commit hook. Every one of those grabs
`.git/index.lock` — parallel callers race for it and flake with
"fatal: Unable to create .git/index.lock: File exists".
Acquire a cross-process advisory lock (atomic `fs.mkdirSync` of
`/tmp/copilotkit-showcase-git-restore.lock`) around every git invocation
in this module: partition, pre-heal diff, checkout, post-heal diff. Held
for the entire sequence so intermediate state is consistent from the
caller's perspective. Stale locks (> 60s) are reaped before the wait loop
so a hard-killed previous run can't wedge subsequent runs.
Unblocks enabling vitest `fileParallelism: true` — the three consumer
suites can now run in parallel forks without stepping on each other's
git operations or on the pre-commit hook's.
- Shell /code viewer now builds a recursive file tree with core-only
(★ highlighted) and show-all-files toggle via ?view=all; collapses the
legacy flat files + backend_files arrays into one tree
- bundle-demo-content: strict mode — errors on missing highlight paths;
drop backend_files field; pull in external backend files referenced
by highlight: (column-relative paths) alongside demo-folder contents;
stable page-first ordering
- Update tests to reflect new column-relative filename shape
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Split generative-ui category into 4 (controlled/declarative/open/
operational), add chat-customization-css + tool-rendering-frontend-tools,
replace old tool-rendering-status/-result IDs with default-catchall /
custom-catchall. Port langgraph-python manifest features + highlight:
paths (column-relative to preserve pre-existing Docker structure).
Update tests for new category/feature counts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Every peer validator under showcase/scripts/ already has a matching
__tests__ suite — this one shipped without. Model the new suite on
validate-parity.test.ts: per-test tmpdir fixtures + CLI subprocess
runs gated by VALIDATE_WORKFLOW_STARTERS_REPO_ROOT.
Extend the existing env-var override to re-home STARTERS_DIR as well
(previously it only redirected .github/). One env var, one root,
matches the pattern used by the other validators.
Coverage:
- happy path (all slugs registered, exit 0)
- slug missing from workflow_dispatch options only (exit 1)
- slug missing from ALL_SERVICES matrix only (exit 1)
- slug missing from both (exit 1, both sources named)
- empty showcase/starters/ (exit 3, refuses trivial pass)
- template/ excluded (not flagged as missing)
- substring-spoof: starter-ag2 missing vs starter-ag2-extended present
— regression guard for the word-boundary anchor
- showcase_deploy.yml absent (exit 3)
All 12 specs pass against the existing regex-based checks. The
substring-spoof case in particular pins the word-boundary contract so
a later refactor can't silently lose it.
## Summary
Three showcase test suites leak working-tree drift (workflow YAMLs +
shell data JSONs) on every run. This fixes the leaks and adds a shared
`FileSnapshotRestorer` + `restoreFromGitHead` harness so the suites are
idempotent.
Also moves `showcase/scripts/vitest.config.ts` from the thread pool to
the fork pool, which is required under Node 20 for the test subprocess
churn in `validate-pins` + the three generator-invoking suites.
## What this PR does NOT fix — vitest 3.2.4 RPC timeout on Node 20
(upstream)
`unit (20.x)` still reports `Timeout calling "onTaskUpdate"` ->
ELIFECYCLE **after** all 14/14 test files and 1011/1011 tests pass.
Known upstream bug: https://github.com/vitest-dev/vitest/issues/6129.
The birpc timeout is hardcoded at 60 s (`DEFAULT_TIMEOUT = 6e4` in
vitest's bundled `index.B521nVV-.js`) and is NOT exposed to
`vitest.config.ts`. No `poolOptions.forks.*` / `teardownTimeout` /
`hookTimeout` knob influences it:
- `singleFork: true` made the run STRICTLY WORSE — only 1/14 files
completed (validate-pins consumes the whole 60 s budget on its own, run
24602985507).
- `fork-per-file` (default) gives each file a fresh RPC channel, and
every file passes — but the final pool-teardown RPC still races on Node
20 and surfaces as a process-level exit 1.
Under vitest 3.2.4 the fix requires either (a) upgrading to vitest 4.x
(out of scope — monorepo-wide upgrade), or (b) a pnpm patch against the
bundled `DEFAULT_TIMEOUT` constant (cross-cutting change; declined
here). The observable reality: this PR makes the showcase-scripts suites
idempotent and green; the remaining `unit (20.x)` redness is a known
vitest flake orthogonal to what this PR is trying to fix.
## CI-killer error (verbatim)
```
Error: Package directory already exists: /home/runner/work/CopilotKit/CopilotKit/showcase/packages/test-integration-tmp
⎯⎯⎯⎯⎯⎯ Unhandled Errors ⎯⎯⎯⎯⎯⎯
Error: [vitest-worker]: Timeout calling "onTaskUpdate"
ELIFECYCLE Test failed.
Failed tasks:
- @copilotkit/showcase-scripts:test
```
## Root cause (fixed here)
Three test suites invoke real generator scripts that write to tracked
files OUTSIDE any tmp dir, leaking drift on every `nx run-many -t test`:
- `create-integration.test.ts` scaffolds
`showcase/packages/test-integration-tmp/` AND mutates three CI workflow
YAMLs (`showcase_deploy.yml`, `showcase_drift-detection.yml`,
`starter-smoke.yml`).
- `generate-registry.test.ts` rewrites
`showcase/shell/src/data/registry.json` + `constraints.json`.
- `bundle-demo-content.test.ts` rewrites
`showcase/shell/src/data/demo-content.json`.
## Fix (9 commits, by area of concern)
1. **`test(showcase/scripts): add shared test-cleanup snapshot/restore
helper`** — new `__tests__/test-cleanup.ts` + `__tests__/paths.ts` +
direct unit coverage in `__tests__/test-cleanup.test.ts`:
- `FileSnapshotRestorer` — snapshots file content as `Buffer`
(byte-exact, preserves non-utf8), restores only files that drifted,
writes atomically via temp-file + rename, recreates parent dirs on write
ENOENT. Temp filenames use `crypto.randomBytes(8).toString("hex")` so
concurrent writes can't collide and the `snapshot()` sweep can
unambiguously identify stragglers. On `snapshot()`, sweeps
`.<basename>.<16-hex>.tmp` stragglers **scoped to the snapshotted
basenames only**.
- `restoreFromGitHead(repoRoot, paths)` — partitions the input via `git
ls-files --error-unmatch` BEFORE any destructive op. **Narrow catch**:
only genuine exit-1 pathspec errors are treated as untracked; ENOENT /
EACCES / non-exit-1 failures re-raise with captured stderr. Uses
`execFileSync` (no shell), forces `LC_ALL=C` / `LANG=C`, scrubs all
`GIT_*` environment overrides, frozen exec options.
- `test-cleanup.test.ts` itself strips `GIT_*` from child env when it
creates tmp repos — pre-commit hooks (lefthook) run with `GIT_DIR` /
`GIT_INDEX_FILE` set, which would otherwise cause tmp-repo `git commit`
calls to write to the HOST working-tree HEAD.
2. **`fix(showcase/test-integration): clean up test-integration-tmp
between runs`** — `create-integration.test.ts` wires
`FileSnapshotRestorer` + `restoreFromGitHead` into the suite, wraps
`rmSync` in `try/finally` so workflow restoration always runs, and
migrates `execSync(string)` -> `execFileSync("npx", [...args])` via a
shared `runGenerator()` helper.
3. **`fix(showcase/test-integration): stop generate-registry +
bundle-content leaks`** — same pattern applied to
`generate-registry.test.ts` and `bundle-demo-content.test.ts`; drops a
redundant bundler pre-run in the latter.
4. **`fix(showcase/scripts): switch vitest to forks pool for Node 20
stability`** — `vitest.config.ts`: thread -> fork pool.
5. **`docs(showcase/scripts): tidy test-cleanup comments and JSDoc`** —
documentation cleanup.
6. **`fix(showcase/scripts): pin vitest to a single fork + bump teardown
timeouts`** — SUPERSEDED by commit 9 below (left in history for
auditability).
7. **`fix(showcase/scripts): add post-heal drifted-baseline guard`** —
the PR had claimed a drifted-baseline guard on CI for
`restoreFromGitHead`, but no post-heal `git diff --quiet` was actually
running. Adds the missing check: on CI, any tracked path still drifted
post-heal throws `drifted-baseline guard: post-heal diff failed`; off-CI
warns. Red-green unit coverage via a counter-based git shim that
selectively fails the N-th `diff --quiet` (so the post-heal diff is
targeted independently of the off-CI pre-checkout diff).
8. **`fix(showcase/scripts): decouple generate-registry test 2 from test
1 output`** — `sorts integrations by sort_order` was reading
`registry.json` without invoking the generator, so `afterEach(restore)`
between tests meant it was exercising the committed baseline rather than
live output. Adds a `runGenerator()` call at the top.
9. **`fix(showcase/scripts): revert singleFork — fork-per-file is
strictly better`** — empirical data from run 24602985507 proved
`singleFork: true` was worse than fork-per-file (1/14 vs 14/14 files
completing before RPC timeout). Reverts the
`poolOptions.forks.singleFork` change; keeps the 30 s `teardownTimeout`
/ `hookTimeout` bumps. Comments now accurately reflect that the RPC
timeout is upstream-hardcoded in birpc and NOT tunable via vitest
config.
## Proof of idempotence
```
pnpm nx run @copilotkit/showcase-scripts:test --skip-nx-cache
Run 1: Test Files 14 passed (14), Tests 1011 passed (1011)
Run 2: Test Files 14 passed (14), Tests 1011 passed (1011)
git status after each: only the intentional test file edits.
```
Red-green verified for the HIGH CR-findings:
- narrow `partitionTrackedPaths` catch: unit test with empty PATH
reproduces ENOENT; pre-fix hid it as "untracked", post-fix throws.
- basename-scoped tmp sweep: unit test places both target-basename and
unrelated `.something-else.<hex>.tmp`; pre-fix swept both, post-fix
sweeps only target.
- post-heal drifted-baseline guard: counter-based git shim fails the 2nd
`diff --quiet`; pre-fix tests pass (guard absent), post-fix tests throw
on CI / warn off-CI with the advertised message.
## Test plan
- [x] `pnpm nx run @copilotkit/showcase-scripts:test` passes 1011/1011
twice in a row (locally, Node 25)
- [x] Red-green: disabling `restore()` fails the regression + safety-net
tests
- [x] Red-green: disabling the post-heal drift guard fails the new guard
tests
- [x] Working tree clean after full run
- [x] `prettier --check` + `oxlint` clean on touched files
- [x] `GIT_*` scrub in test harness prevents pre-commit-hook-induced
pollution of real HEAD
## CI status
- **`unit (22.x)`**: pass
- **`unit (24.x)`**: pass
- **`unit (20.x)`**: all 14/14 files + 1011/1011 tests pass; post-suite
`onTaskUpdate` RPC timeout emits exit 1. Upstream bug
https://github.com/vitest-dev/vitest/issues/6129; not fixable at
`vitest.config.ts` level on vitest 3.2.4.
## Caveats
- Local verification ran on Node v25.8.0. No Node 20 binary on this dev
host; Node 20 CI was the final gate.
- The residual `unit (20.x)` failure is orthogonal to this PR. To
resolve it we would need to upgrade vitest to 4.x (monorepo-wide change)
or apply a pnpm patch to bump `DEFAULT_TIMEOUT` in vitest's bundled
`birpc`. Both are tracked separately.
parseManifest (lib/manifest.ts):
- Shape errors for demos[i].route: number / null / object / empty string
- Shape error for route not starting with /demos/
- Happy path persists frozen demo.route on the parsed entry
- Backward-compat: demos[i] without route is accepted with undefined route
validate-parity:
- Negative-case regression: missing-demo-dir's expectedDir is derived from
demo.route (not demo.id) when route is present; mismatched id + route
was silently hiding drift.
- Fallback-case: demo with no route resolves expectedDir from demo.id
- routeToDirName unit tests: undefined / bare /demos/ / normal tail segment
TDD verified: mutating the /demos/ prefix guard failed the relevant test
(RED), restoring the guard passed it (GREEN). Mutating expectedDir to
demo.id-only failed the negative-case test (RED), restoring passed
(GREEN).
Jordan Ritter