Regression from the 2026-04-21 incident: 18 production Railway services
were found with malformed image refs of the form
`ghcr.io/copilotkit/showcase-<slug>atest` (missing the `:` before
`latest`, so Docker treats `...atest` as the tag). Root cause was an
out-of-band MCP/manual mutation — no committed code touched those refs,
so the data has been fixed but no source-controlled guardrail exists.
Add a standalone script that queries Railway's GraphQL API for every
service in the CopilotKit Showcase project and asserts each image ref
matches the canonical shape `ghcr.io/copilotkit/<service-name>:latest`.
Wire it into showcase_deploy.yml as a pre-build job so any drift aborts
the workflow before the build matrix fans out.
On violation the script prints the service name, the current image, the
expected shape, and the reason, so the fix is obvious in the run log.
Slack classification in the notify job distinguishes a drift failure
from other pre-build failures.
Verified locally: 41 services pass against current Railway state; the
exported `validateImage` function rejects the exact `...atest`
corruption, mismatched service/image names, missing tags, wrong
registries, wrong tag values, and null sources (9/9 simulated cases).
Generalize the watchdog shape proven in showcase/packages/crewai-crews/entrypoint.sh
(PRs #4114 + #4115) so that every Bucket-B starter gets:
- PYTHONUNBUFFERED=1 export (harmless for non-Python frameworks)
- A backgrounded watchdog subshell that polls the agent health endpoint
every 30s and kills the agent after 3 consecutive failures, letting
wait -n + container runtime handle the restart through the normal path.
- python -u on uvicorn / langgraph_cli invocations (Python frameworks).
- awk ... fflush() log prefixing (replaces the previous sed pipe; keeps $!
pointing at the real agent process).
Per-framework agent health paths:
* FastAPI / uvicorn agents -> /health
* langgraph-python, langgraph-fastapi, langgraph-typescript -> /ok
* claude-sdk-typescript, ms-agent-dotnet, spring-ai -> /health
* mastra -> /api
Spring Boot starter also gets a 60s /health startup probe (replacing the
blind sleep 5) because JVM warmup + context refresh can exceed 30s.
Root-cause fix for the 04-21 silent-hang incident on the crewai-crews
Railway deploy. Three tightly-coupled changes:
1. Bump ag-ui-crewai pin from `>=0.1.4,<0.1.6` to `>=0.2.0,<0.3.0`.
0.1.5 had three defects that wedged the agent: unguarded
`source.state.messages` access, an orphan `asyncio.create_task` with
no cancel ref, and a sync `completion()` call that pinned the event
loop. All three are fixed in ag-ui PR #1550, released as 0.2.0 on
2026-04-18. Our upper ceiling was blocking the fix.
2. Remove the pre-bind LLM crash hardening shim in `agent_server.py`.
The shim monkey-patched `crewai.cli.crew_chat.generate_*_description_with_ai`
to static strings so that `ChatWithCrewFlow.__init__` — which
ag-ui-crewai <= 0.1.5 invoked at endpoint-registration time, BEFORE
uvicorn bound its port — could not crash the process before the HTTP
server was listening. 0.2.0 defers `ChatWithCrewFlow` construction to
first request via a module-scoped `_cached_flow` + `asyncio.Lock`
inside `add_crewai_crew_fastapi_endpoint`. Any LLM hiccup now
surfaces as a 5xx on the first request instead of a startup crash,
which is what the shim was reaching for. The shim is dead code on
0.2.0 and has been removed (with `logging` import dropped as it was
only used by the shim).
3. Add `python -u` to the uvicorn invocation in `entrypoint.sh` as a
belt-and-suspenders complement to the existing `PYTHONUNBUFFERED=1`
export. The env var can in principle be un-exported by a child;
`-u` forces unbuffered stdout/stderr at the interpreter level and
is not overridable by user code. Combined with `awk '{...; fflush()}'`
in the pipe (already in place), this guarantees uvicorn request
lines reach Railway's log stream line-at-a-time. During the 04-21
incident Railway saw only ~15 log lines over 9h of uptime because
of buffering through a previous `sed` formulation.
Also updates `showcase/scripts/fail-baseline.json`'s `validatePinsFailHash`
to match the new `ag-ui-crewai` spec string. Pin-drift FAIL count is
unchanged (110); the hash changed only because the `ag-ui-crewai` line
in the FAIL set went from `>=0.1.4,<0.1.6` to `>=0.2.0,<0.3.0`.
Verified locally:
- `pip install -r requirements.txt` resolves `ag-ui-crewai-0.2.0` cleanly.
- `python -u -m uvicorn agent_server:app` starts; `/health` returns
200 `{"status":"ok"}`; request lines appear in real-time logs.
- `pytest tests/python/` — 94/94 pass.
- `pnpm -C showcase/scripts test` (vitest) — 1079/1079 pass.
- `validate-pins.ts` — count=110 matches baseline; hash updated.
Upstream refs:
- crewAI issue: https://github.com/crewAIInc/crewAI/issues/5510
- ag-ui PR #1550: https://github.com/ag-ui-protocol/ag-ui/pull/1550
Intentionally NOT in this PR:
- `showcase/starters/crewai-crews/` parity backport (the starter still
carries the 0.1.5 pin and the shim).
- The 14-starter watchdog generalisation.
Both belong to the silent-hang vulnerability-class work tracked
separately.
Correctness and portability fixes in the demo-content bundler:
- Track contributor snippets per-file so edits to multiple files in
one commit all get attributed, not just the last one walked.
- Extend endLine when the same file is seen again rather than
dropping the earlier slice — previously a later, smaller region
overwrote a larger one.
- Warn when the watch flag is set on Linux without the recursive
fs.watch support matrix, so the user sees why nothing is firing
instead of assuming silent success.
- Normalize path separators for Windows so bundle manifests use
POSIX paths regardless of the host OS.
- bundle-demo-content: reject highlight: paths that resolve outside
the package root. The bundle output is committed to the repo and
consumed by both shells at build time, so a malicious or
mistake-riddled manifest could otherwise smuggle arbitrary
filesystem contents (../../secrets, absolute paths) into
demo-content.json. Resolve relative to pkgRoot and throw on
escape (finding #19).
- generate-registry: add a runtime guard that manifest.slug is a
non-empty string before path.join(PACKAGES_DIR, manifest.slug).
Schema validation upstream already enforces this, but a
silently-undefined slug fed to path.join yields
"<packages-dir>/undefined" and would produce an empty docs_links
without surfacing any error. Fail loudly instead (finding #20).
If shell-docs/src/content/{reference,ag-ui,docs} didn't exist the
script silently produced a tiny index with no warning — operators
only found out by noticing Cmd-K search returning nothing.
Warn per missing directory and exit non-zero if ALL scan roots are
missing (that means we're running outside a prepared tree, e.g.
shell-docs didn't emit into the expected layout).
The watch loop logged '[watch] bundle failed' once and then fell
silent — repeat failures looked like success, and recoveries were
invisible (no news = assumed fine). Track the last error in module
scope so we distinguish first-failure from repeat-failure, and log an
explicit 'bundle recovered' note when the next green run clears the
state. Makes dev-mode transitions visible instead of silent.
A malformed docs-links.json was only console.warn'd and treated as
empty — build ran green while the override file silently rotted.
Accept an errors accumulator and push parse failures into it so
main()'s non-zero exit path fires. Missing file and stale-shape
tolerance are unchanged (both are legitimate states).
Without a timeout on the inner fetch, a hung upstream would stall the
entire docs-probe run indefinitely (Node's fetch has no default
timeout). Add a 10s AbortController. The bare catch also left 'error'
states opaque — log URL + error kind so operators can tell an
abort from DNS from TLS.
The bare catch returned 'unknown' with no trace, making it impossible
to distinguish DNS failures from aborts from TLS errors in the unknown
bucket. Log the URL plus a kind:code pair so spikes of 'unknown' are
actually diagnosable.
mockE2E/mockSmoke/mockQA were called unconditionally from main(), so
production dashboards shipped seeded test data in columns that aren't
wired to real CI yet. Gate each behind GENERATE_STATUS_MOCK_*
env vars (default off → emit null / 'unknown'). Print a prominent
banner to stderr naming which columns are mock whenever any gate is
on, so operators can't discover stub data by surprise.
The cross-starter consistency test had two AGENT_URL matchers:
function re8000() {
return new RegExp(AGENT_URL_LOCALHOST_8000_RE.source, flags);
}
function re8123() {
return new RegExp(
AGENT_URL_LOCALHOST_8000_RE.source.replace(/8000\\b/, "8123\\b"),
flags,
);
}
The 8123 variant does regex-source string munging — brittle: the
replacement target `8000\\b` has to stay in lockstep with the
shared regex's literal source, and a future port format change would
snap in a subtle way (the test could keep passing while matching
subtly wrong content, or start failing for reasons unrelated to the
starter it's guarding).
Replace the `.source.replace` hack with an exported factory
`makeAgentUrlLocalhostPortRE(port)` in `generate-starters.ts` and
have both `re8000`/`re8123` use it. The existing
`AGENT_URL_LOCALHOST_8000_RE` constant is preserved (now delegating to
the factory) so any external import stays happy.
Bonus: the factory validates port bounds so a typo can't silently
produce a pattern that never matches.
Two orthogonal bugs in the demo-content bundler:
1) README regions were silently dropped. `collectDemoFiles` pulls the
demo-dir root README out into the `readme` field rather than
appending to `files`, but it STILL wrote any `@region[...]`
slices it found inside the README into `perFileRegions` under the
README's bundled path. The region-collation loop at write time
iterated `fileOrder = files.map(f => f.filename)`, which by
construction never contains the README path — so those regions
went nowhere. Walk any `perFileRegions` keys not in fileOrder
after the main pass, so README regions still land in the public
map. Preserves the stable file-order rule (README contributions
come after source files, alphabetical tiebreak).
2) Binary-like files were being passed through
`fs.readFileSync(abs, "utf-8")` and then stringified into the
bundle. A PNG or font file typically produces a mess of
replacement characters in `demo-content.json` — waste at best,
visible corruption at worst when the Code tab ever tried to
render them. Extend `SKIP_EXTENSIONS` with the usual binary
suspects (images, fonts, archives, media, PDFs) so the walker
drops them at scan time.
The canonical schema in `scripts/generate-registry.ts` + per-package
`docs-links.json` calls the shell path `shell_docs_path`. probe-docs
was still reading ONLY the legacy `shell_docs_url` key, so any
future cleanup of `shared/feature-registry.json` to the canonical
key would silently turn every shell-docs status into 'missing'.
Prefer `shell_docs_path`, fall back to `shell_docs_url` so older
registry snapshots still contribute. Emit a one-shot dev-mode warning
when we see only the legacy key, so a lingering registry stays visible
without spamming CI logs.
showcase/scripts/ now ships a package-lock.json (committed here for the
first time), so the Docker builder can use 'npm ci' for deterministic
installs instead of 'npm install'. Update the comment to match actual
state and copy the lockfile into the scripts stage.
- Extend .oxfmtrc.json ignore to cover shell-docs/src/content
(mirrors existing shell/src/content ignore, which was not updated
when shell-ops-v2 moved MDX content into the new shell-docs package)
- Format showcase/scripts/generate-registry.ts and probe-docs.ts
MDX docs moved from shell to shell-docs, but several generated artifacts
are still consumed by both shells:
- registry.json: shell uses it (home grid, integrations, matrix,
middleware, layout); shell-docs uses it (docs renderer framework lookup)
- demo-content.json: shell uses it (integrations/[slug]/[demo]); shell-docs
uses it (<Snippet> in docs renderer)
- search-index.json: shell-docs consumes it for the docs search modal;
shell also keeps a copy so its header search still works — links 301
across to docs.showcase.copilotkit.ai.
Updated scripts:
- generate-registry.ts: dual-emits registry.json to both shells
(constraints.json stays shell-only — integration-explorer is shell)
- bundle-demo-content.ts: dual-emits demo-content.json
- generate-search-index.ts: scans from shell-docs/src/content (where MDX
now lives), writes to both shells' data dirs
- probe-docs.ts: scans shell-docs/src/content/docs (content source moved),
still writes docs-status.json under shell/ for the dashboard
- sync-docs-from-main.ts: target path updated to shell-docs/src/content
Tests in __tests__/ reference shell/src/data paths; dual-emit keeps
those stable so existing afterEach-restore hooks continue to work.
- Renamed showcase/shell-internal/ → showcase/shell-dashboard/ (git mv for history).
- Updated package name to @copilotkit/showcase-shell-dashboard (+ lockfile).
- Updated external references: showcase/scripts/{probe-docs,generate-status}.ts comments and showcase/README.md.
- Added dispatch + filter + ALL_SERVICES entry for shell-dashboard in .github/workflows/showcase_deploy.yml.
- Provisioned Railway service showcase-shell-dashboard (id 4d5dfd74-be61-40b2-8564-b53b7dd4c15b):
- image source ghcr.io/copilotkit/showcase-shell-dashboard:latest
- env vars OPENAI_API_KEY, ANTHROPIC_API_KEY, PORT copied from showcase-shell
- custom domain dashboard.showcase.copilotkit.ai attached (CNAME → 6u9icvje.up.railway.app)
- image will be built and pushed via showcase_deploy.yml on merge
Production showcase-starter-langroid returns ``{"status":"degraded","agent":"down"}`` HTTP 503
at ``/api/health`` — the exact path the showcase-deploy workflow's ``ALL_SERVICES.health_path``
verify step asserts, so the next CI redeploy of this starter would fail verification.
Root cause: two compounding bugs in ``showcase/starters/langroid/entrypoint.sh``.
1. Process-substitution log prefixers suppressed stdout. Both backgrounded
services were wrapped with ``> >(sed 's/^/[agent] /') 2>&1 &`` /
``> >(sed 's/^/[nextjs] /') 2>&1 &``. In Railway's V2 runtime this shape
reliably produced ZERO ``[agent]``/``[nextjs]`` log lines (confirmed against
several weeks of Railway logs for this service) and correlated with
agent-unreachable 503s at ``/api/health``. The package entrypoint
(``showcase/packages/langroid/entrypoint.sh``) uses the plain-``&`` pattern
with no wrapper, stays green on the same Railway runtime, and shows full
uvicorn ``INFO:`` startup logs including ``127.0.0.1:... "GET /health
HTTP/1.1" 200 OK`` from Next.js's fetch. Match the working package pattern.
2. Cold-start race amplified by ``sleepApplication=true``. Railway sleeps
idle services. On wake, Next.js is ready in <1s but Python + langroid
imports take 10-20s. Without a readiness gate, Next.js answered the first
post-wake ``/api/health`` probe with ``agent:"down"`` before uvicorn had
bound port 8123. The deploy workflow's verify step then saw the 503 and
reported the deploy as unhealthy. Add a 30s readiness probe that curls
``http://127.0.0.1:8123/health`` in a loop before starting Next.js. The
IPv4 literal (``127.0.0.1`` not ``localhost``) is load-bearing — Node
22+'s fetch resolves ``localhost`` to IPv6 ``::1`` first, and uvicorn
binds IPv4 only; the readiness probe must not false-negative on resolver
semantics alone.
Also export ``PYTHONUNBUFFERED=1`` so Python import-time tracebacks (e.g.
langroid module-load failures) reach the container log immediately instead
of sitting in userspace buffers until process exit closes them off.
Verification:
- Built langroid starter image locally from this branch. ``/api/health``
returns 200 with ``{"status":"ok","agent":"ok"}``. Full uvicorn INFO
logs visible including ``127.0.0.1:xxxxx - "GET /health HTTP/1.1"
200 OK`` confirming Next.js -> agent path works.
- Readiness probe fires before Next.js launch: ``[entrypoint] Agent
/health ready after 3s``.
- New regression guards in ``starter-consistency.test.ts`` cover all four
invariants (no sed wrapping around uvicorn, no sed wrapping around next
start, ``PYTHONUNBUFFERED=1`` export, readiness probe on 127.0.0.1:8123
before ``next start``). Verified red→green (reverted entrypoint -> 4
fails; restored -> 4 pass). Full showcase/scripts suite: 1079/1079 pass.
The auditableDemos filter (introduced for cli-start's informational
`command:` entry) was applied to the spec/qa SHOULD checks but not
the missing-demo-dir MUST check. Result: cli-start in langgraph-python's
manifest triggered a false missing-dir failure even though it has no
on-disk folder by design.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Clarify the positioning of the In-App Human in the Loop cell. The old
label was ambiguous with the in-chat variant; the new label surfaces
the core technical story at-a-glance: this is the async useFrontendTool
pattern where the approval UI pops up OUTSIDE the chat surface.
Also register the cell in the langgraph-python manifest (features list
+ demos entry pointing to the new files).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "Declarative Generative UI (A2UI — Custom Catalog)" single-file
prototype variant was a testing-only companion to the canonical
declarative-gen-ui 3-file BYOC pattern. Drop it everywhere:
- Delete src/app/demos/declarative-gen-ui-hardcoded/ cell directory
- Delete src/agents/a2ui_dynamic_hardcoded.py
- Remove from langgraph-python manifest.yaml (features + demos) and
langgraph.json graphs
- Remove from docs-links.json, route.ts (agent entry + a2ui agents
list), shared/feature-registry.json, shared/constraints.yaml
- Drop the Callout promoting it from a2ui/dynamic-schema.mdx
- Bump expected langgraph-python count 32 → 31 in generate-registry
test; regenerate all bundles
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pre-Built: Chat was redundant with Pre-Built: CopilotChat — both rows
referred to the same CopilotChat surface with slightly different
framing. Drop prebuilt-chat everywhere:
- Delete src/app/demos/prebuilt-chat/ cell directory
- Remove from langgraph-python manifest.yaml features + demos
- Remove from shared/feature-registry.json features + chat-ui allowlist
- Remove from shared/constraints.yaml constrained-explicit
- Remove from src/app/api/copilotkit/route.ts neutral-fallthrough list
- Update expected counts in bundle-demo-content + generate-registry
tests (langgraph now exposes 32 features, down from 33)
- Fix stale src/agents/main.py test expectation to
src/agents/agentic_chat.py (main.py was split into a neutral assistant
by an earlier Phase-1 fix commit)
- Regenerate shell/src/data/{registry,constraints,demo-content,
docs-status}.json
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ports probe-docs.ts from 4084. For each entry in shared/feature-registry.json
it HEADs the feature's og_docs_url and checks whether a matching .mdx exists
under shell/src/content/docs/<shell_docs_url>, then writes the results to
shell/src/data/docs-status.json. shell-internal's DocsRow falls back to this
probed state for (integration, feature) cells that don't have a per-column
docs-links.json override. Exposes the script as `probe-docs` in
showcase/scripts/package.json.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ports the loadDocsLinks() helper from 4084 into generate-registry.ts. After
schema validation, each integration gets a docs_links field merged in from its
sibling packages/<slug>/docs-links.json (best-effort: missing file or stale
shell_docs_url shape is tolerated). The shell-internal DocsRow reads this via
integration.docs_links.features[<id>] to prefer curated per-column overrides
over the per-feature defaults.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add prebuilt-chat feature to the registry and constrained-explicit
allowlist so the 4085 manifest's intentional prebuilt-chat wiring
validates cleanly (4084 omits it; 4085 committed the cell earlier)
- Bump expected langgraph-python feature count in generate-registry
test from 32 to 33 to reflect the new row
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three interlocking fixes to generate-starters.ts that together restore
langgraph-fastapi and langgraph-typescript /api/health to "ok" after
#4099 shipped an entrypointOverride feature that accidentally reverted
earlier fixes and left gaps for the TS starter.
1. Resolution-aware python import rewrite for langgraph starters.
langgraph_cli loads agent modules standalone (not as packages), so the
generator rewrites `from .X import ...` to absolute form. The previous
flat rewrite produced `from <agentDir>.X import ...`, and the
subsequent subdir-aware variant used the file's OWN dir as the
package. Both assume sibling targets sit alongside the importing file.
langgraph-fastapi violates that: agent.py lives at
`src/agents/src/agent.py` but the `tools/` it imports lives at
`src/agents/tools/` -- one level up, not inside `src/`. Either prior
rewrite produced `from src.agents.src.tools import ...`, a path that
doesn't exist, and langgraph_cli crashed on module import with
`ModuleNotFoundError: No module named 'src.agents.src.tools'` before
ever binding 8123.
Walk UP from the file's own dir toward agentDest and rebase the
absolute import on the shallowest directory that actually contains
`<firstSeg>/` or `<firstSeg>.py`. Correct both for co-located imports
(`tools/get_weather.py` importing `.types`) and for sibling-directory
imports (`src/agent.py` importing `.tools` from `../tools`).
2. Merge langgraph-typescript agent runtime deps into root package.json.
The TS Dockerfile deliberately deletes `agent/package.json` to
collapse the ESM package boundary between the Next.js frontend and
the agent subtree. Without merging the agent's runtime deps up into
the root package.json, the langgraph-cli's runtime import of
graph.ts fails with `Cannot find module '@langchain/openai'` and
the agent never binds 8123.
Add `extraDependencies` to the langgraph-typescript framework def
with @langchain/core, @langchain/langgraph,
@langchain/langgraph-checkpoint, @langchain/langgraph-cli,
@langchain/openai, and @copilotkit/sdk-js.
3. Restore AGENT_LOG_PREFIX process-substitution helper.
#4099 inadvertently replaced `cmd &> >(awk … fflush …)` with
`cmd 2>&1 | sed …` across every getEntrypointBlock branch. After a
pipeline `$!` points at `sed`, not the agent, so `kill -0 $AGENT_PID`
and `wait -n $AGENT_PID` monitor the wrong process and mask real
crashes; `sed` also line-buffers so crash output can be lost.
Restore the helper so the committed starter entrypoint.sh files and
regenerated output stay consistent.
generate-starters.ts:
- Add entrypointOverride?: boolean to FrameworkDef so slugs whose boot sequence
diverges from the shared template (e.g. langroid's provider-aware credential
guard) can preserve their committed entrypoint.sh through regeneration.
- Snapshot the override from STARTERS_DIR/<slug>/entrypoint.sh (canonical
committed path), NOT outDir — so --check mode (which passes a temp outDir)
doesn't false-flag the override as drift.
- Force mode: 0o755 on the restored file regardless of source mode so editors
that strip the executable bit can't silently ship a broken starter.
- Throw (not warn) when an entrypointOverride slug is missing the override
file, when declared extraFiles / agent_server.py / Dockerfile are missing,
and when PIN_OVERRIDES references a stale dep. These are repo-integrity
failures — silent-warn-and-continue ships broken starters.
- Strip unused `import sys` after sys.path.insert removal (pair with the
existing osUsed detection).
Opt langroid in via entrypointOverride: true.
Also adds a regression test in __tests__/generate-starters.test.ts that
calls generateStarterToDir against a tmp dir and asserts the emitted
entrypoint.sh byte-equals the committed langroid starter entrypoint,
plus a negative test that a non-override slug does NOT preserve its file.
Port the 4084 simplification of the status generator: remove the
MOCK_VARIANTS seeding table + mockVariantTest / mockVariantQA /
mockVariantHealth helpers + Variant type. Informational demos (no
route) are skipped from health probing. The status JSON no longer
carries per-demo `variants[]` — the shell-internal dashboard renders
one row per demo without variant expansion.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Port the 4084 scripts-layer enhancements so 4085's showcase toolchain
matches the new feature shape:
- lib/manifest.ts: ManifestDemo gains optional `command` field; parser
accepts + validates it (non-empty string, frozen).
- bundle-demo-content.ts: inline `@region[name]` / `@endregion[name]`
comment-marker extraction; informational-only demos (no route, e.g.
cli-start) are skipped; markers stripped from bundled content;
regions: { file, startLine, endLine, code, language } emitted per
demo. External-highlight-file merging (4085-specific) preserved, so
backend agents under src/agents/*.py still flow into the bundle.
- validate-parity.ts: accepts demos at BOTH demos/<cell>/ (4084 layout)
and src/app/demos/<cell>/ (4085 layout); informational demos
(command field) are excluded from the parity audit.
- tests: bundle-demo-content.test.ts expectedDemos updated for the
shared-state rename; generate-registry.test.ts feature count 25→32;
validate-parity.test.ts missing-demo-dir message updated to match
the new dual-location wording.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Regenerate agno starter to pick up agno>=2.5.17 (from #4095)
- Ratchet validate-pins fail-baseline hash to match new FAIL set
(count unchanged at 110; hash rotates because agno Dojo/showcase
pair now reflects the SDK upgrade)
generate-starters.ts rewrites relative imports to absolute for langgraph
starters because langgraph_cli loads modules standalone rather than as
packages. The rewrite was flat:
from .X import ... -> from <agentDir>.X import ...
For a file at <agentDir>/tools/get_weather.py, `from .types import ...`
resolves to `tools.types` — the CURRENT package — not `<agentDir>.types`.
The flat rewrite dropped the `tools` segment and produced:
ModuleNotFoundError: No module named 'src.agents.types'
at startup. The agent crashed during module import; the entrypoint
pipe swallowed the traceback (see previous commit for the pipe bug);
the 2-3s sleep guard happened to fire while `sed` was still alive; and
Railway's /api/health probe reported `agent: "error"`.
Make the rewrite subdir-aware: compute the file's containing Python
package from its relative path under agentDest, and prepend that to the
relative import target. So `from .types import ...` inside
`src/agents/tools/get_weather.py` becomes
`from src.agents.tools.types import ...`.
langgraph-python has the same broken imports in its tools/__init__.py
but doesn't crash at runtime because main.py doesn't import from tools
(dead path). Regenerating fixes the dead code too.
The entrypoint.sh template used `cmd 2>&1 | sed 's/^/[agent] /' &`
followed by `AGENT_PID=\$!`. After a pipeline, `\$!` points to the LAST
command in the pipe (the `sed` process), not the agent. Every subsequent
`kill -0 \$AGENT_PID` and `wait -n \$AGENT_PID` was therefore monitoring
`sed`, which stays alive until its stdin closes — long after the agent has
crashed. Railway restarts the container mid-loop; the health probe sees
`{status: "degraded", agent: "down"}` for a few seconds during each cycle.
A second, compounding bug: `sed` buffers by default, and Python agents
buffer their own stdout, so a stack trace emitted during module import
could sit in userspace memory until the pipe closed — by which point the
log was discarded and the real cause of the crash was lost.
Fix both by switching to bash process substitution:
cmd &> >(awk '{print "[agent] " \$0; fflush()}') &
AGENT_PID=\$!
Process substitution does not create a pipeline, so `\$!` remains the
agent's PID. `awk` with `fflush()` flushes each prefixed line to the
container log immediately. Also export `PYTHONUNBUFFERED=1` at the
entrypoint level so Python-based agents don't buffer before awk.
Applies to all 17 starters (python, langgraph-python, langgraph-fastapi,
langgraph-typescript, mastra, typescript, java/spring-ai, csharp/
ms-agent-dotnet). Done once in generate-starters.ts + the template +
regenerated entrypoint.sh files.
## Summary
Replaces `RUN chown -R app:app /app` with `COPY --chown=app:app` across
all 17 starter Dockerfiles + 4 shared templates.
Every starter ended with a recursive chown over `/app`, which walks ~50k
files (Next.js `node_modules` dominates) to fix ownership after the
fact. Under the 23-way Depot runner fan-out used by
`.github/workflows/deploy-showcase-services.yml`, that step consistently
ran 5+ minutes under I/O contention — busting the **15-minute GH Actions
job budget** before images could finish pushing.
Failed run this fixes:
https://github.com/CopilotKit/CopilotKit/actions/runs/24621469277 (22/23
showcase services cancelled mid-push).
## What changed
- Create the `app` user right after `WORKDIR /app` in every runner stage
so `--chown=app:app` resolves by name.
- Add `--chown=app:app` to every runner-stage COPY (including
multi-stage `COPY --from=frontend` and `COPY --from=<agent-builder>`).
- Drop the trailing `RUN chown -R app:app /app` (or the tail of the
compound RUN in the TS starters).
- For langgraph starters, fold `chown app:app /app/.langgraph_api` into
the same RUN as the `mkdir`, so `langgraph_cli`'s scratch dir remains
writable by the runtime user.
- Update `showcase/scripts/generate-starters.ts` so the shared templates
(`Dockerfile.python/typescript/dotnet/java`) emit the new pattern.
Starters are regenerated from templates.
## Local timing (Docker Desktop, single starter, no contention)
| Starter | Pre-fix | Post-fix | `chown -R` step |
| --- | --- | --- | --- |
| ag2 (Python) | 3:03 | 1:39 | 50.1s → removed |
| langgraph-fastapi | n/a | 1:41 | replaced with 0.2s targeted chown on
`.langgraph_api` |
| mastra (TS) | n/a | 2:16 | removed |
The real-world win on the 23-way Depot fan-out is substantially larger
than the local 50s baseline — recursive chown degrades super-linearly
with concurrent I/O pressure, which is exactly what the 5+ min Depot
step demonstrated.
## Smoke tests
- ag2 image: container starts clean as `app` user, all files under
`/app` owned by `app:app`, `/api/health` returns 200.
- langgraph-fastapi image: `/app/.langgraph_api` exists and is owned by
`app:app`.
- mastra image: builds cleanly, ownership correct.
## Test plan
- [ ] CI green (existing showcase starter smoke suite covers startup)
- [ ] Watch the next showcase deploy workflow run — expect jobs to
finish well under 15m
## Not touched
No workflow files, no `examples/` Dockerfiles (none matched the problem
pattern), no `chmod -R` offenders (none found).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Two related defects in the showcase deploy pipeline let stale images sit
live on Railway while Slack stayed green. This PR fixes both.
### Defect 1 — Drift detector skipped all starter services
`.github/workflows/showcase_smoke-monitor.yml` listed only 19
**package** slugs in its `SERVICES=(...)` array (ag2, mastra,
llamaindex, ...). Zero **starter** slugs. As a result:
- GHCR `showcase-starter-<svc>` tags were never checked for drift.
- `gh workflow run showcase_deploy.yml -f service=starter-*` was never
auto-dispatched.
- Starter services could run with weeks-old images and no alert would
fire.
`showcase_deploy.yml` already supports `starter-*` dispatch names and
already calls `serviceInstanceRedeploy` for any service with a
`railway_id`, so no change is required there. The fix is extending
`SERVICES=(...)` to include all 17 starter slugs via a
sparse-checkout-driven filesystem enumeration (no more literal
duplication between workflow and `showcase/starters/`).
### Defect 2 — Silent deploy failures reported green
`.github/workflows/showcase_deploy.yml` emitted `::warning::` and exited
0 when a service never returned 200 on its health path within 360s. The
legacy justification (`# Don't fail — sleep-on-idle services take time
to wake`) no longer applies: Railway is on the Pro tier with no
sleep-on-idle, so a 6-minute failure to become healthy is a real
failure. Changed to `::error::` + `exit 1`.
## Round 2 fixes
Round 2 CR raised six findings against the original smoke-monitor +
validator changes. All fixed in this PR:
- **BLOCKING 1/2 — smoke-monitor guard.** Replaced the magic `-eq 19`
sentinel with `grep -c '^starter-'` so adds/removes to the literal
non-starter list can't silently disable the guard. Added `shopt -s
nullglob` around the `showcase/starters/*/` loop so an empty starters
tree no longer corrupts `SERVICES` with a `starter-*` literal.
- **BLOCKING 3 — GHCR stderr isolation.** Dropped `2>&1` on the `gh api
-i` call; captured stderr to a temp file and surfaced it only when `gh`
returns a non-zero RC with no HTTP status. Auth / rate-limit / network
noise can no longer splice into the HTTP header block and poison
`HTTP_STATUS` / `API_BODY` parsing.
- **BLOCKING 4 — validator tests.** Added
`showcase/scripts/__tests__/validate-workflow-starters.test.ts` (12
specs): happy path, missing-from-options-only, missing-from-matrix-only,
missing-from-both, empty starters dir (exit 3), template/ excluded,
substring-spoof (starter-ag2 vs starter-ag2-extended), missing workflow
file (exit 3). Also extended `VALIDATE_WORKFLOW_STARTERS_REPO_ROOT` to
re-home the starters dir for testability.
- **MEDIUM 1 — YAML parsing.** Replaced the fragile regex-over-YAML
options scanner with a real `yaml.parse()` + typed navigation down
`on.workflow_dispatch.inputs.service.options`. ALL_SERVICES stays
regex-scanned (embedded JSON in a bash heredoc, with `${{ ... }}`
interpolations that aren't valid JSON pre-execution), but the
surrounding step is now located via YAML.
- **MEDIUM 2 — Slack list truncation.** Replaced `cut -c1-200` with a
`truncate_csv` helper that drops whole comma-separated entries until
under budget and appends `…` when truncated. No more
`starter-claude-sdk-pyth` mid-slug corruption.
- **MEDIUM 3 — template/ exclusion cross-references.** Both the TS
validator's `EXCLUDED_DIRS` and `showcase_smoke-monitor.yml`'s `[
"$slug" = "template" ] && continue` now carry `# keep in sync with ...`
comments pointing at each other.
- **NIT 2 — entry-check simplification.** Dropped the
belt-and-suspenders `import.meta.url === \`file://${argv[1]}\`` branch;
kept only the canonical `fileURLToPath(import.meta.url)` form.
- **NIT 3 — jq pipeline collapse.** Single-pass `.jobs[]? | select |
"\(...)"` replaces the three-pass `map | map | .[]` chain in the notify
step.
## Files changed
- `.github/workflows/showcase_deploy.yml` — warning → error + exit 1 on
unhealthy deploy; `truncate_csv` replaces `cut -c1-200` (3 sites);
single-pass jq pipeline in notify step.
- `.github/workflows/showcase_smoke-monitor.yml` — filesystem-driven
`SERVICES=(...)`, starter-count guard, `nullglob` loop, stderr-isolated
`gh api` call, cross-reference comment.
- `.github/workflows/showcase_validate.yml` — wires
`validate-workflow-starters` into CI.
- `showcase/scripts/validate-workflow-starters.ts` — YAML-aware presence
checks; env-var override homes both starters dir and workflow path.
- `showcase/scripts/tsconfig.json` — scripts-local tsconfig for LSP type
resolution.
- `showcase/scripts/__tests__/validate-workflow-starters.test.ts` — 12
specs covering the full matrix of drift scenarios.
## Test plan
- [ ] Next scheduled `showcase_smoke-monitor` run includes starter
services in its drift scan.
- [ ] A deliberately-unhealthy deploy (simulate by pointing health_path
at a 404) fails the job and fires the Slack alert.
- [ ] `showcase_validate` CI job runs `validate-workflow-starters` and
`npx vitest run scripts/__tests__/validate-workflow-starters.test.ts`
green.
Every starter Dockerfile ended with `RUN chown -R app:app /app`, which
recursively walks the full `/app` tree (Next.js `node_modules` dominates
— ~50k files) to fix ownership after the fact. On the 23-way Depot
runner fan-out used by .github/workflows/deploy-showcase-services.yml
this step consistently took 5+ minutes under I/O contention, busting
the 15-minute GH Actions job budget before the image could even finish
pushing (run 24621469277 — 22/23 services cancelled).
The standard Docker idiom is `COPY --chown=<user>:<group>` which applies
ownership during the copy step itself — no extra layer, no full-tree
traversal, zero runtime cost.
Changes:
- Create the `app` user right after `WORKDIR /app` in every runner
stage so `--chown=app:app` resolves by name.
- Add `--chown=app:app` to every COPY instruction that lands files
under `/app` in the runner stage (including `COPY --from=frontend`
and `COPY --from=<agent-builder>` multi-stage copies).
- Drop the trailing `RUN chown -R app:app /app` (or the tail of the
compound RUN in the TS starters).
- For langgraph starters, fold `chown app:app /app/.langgraph_api`
into the mkdir RUN so langgraph_cli's scratch dir is still writable
by the runtime user.
- Update showcase/scripts/generate-starters.ts so the shared templates
(Dockerfile.python/typescript/dotnet/java) emit the new pattern and
the framework-specific COPY lines (`COPY ${dest} ./` for extra files
and `COPY agent_server.py ./` for Python starters) include `--chown`.
Local verification (Docker Desktop, single starter, no contention):
ag2 pre-fix: 3:03 total, `RUN chown -R app:app /app` = 50.1s
ag2 post-fix: 1:39 total, no chown step
langgraph-fastapi post-fix: 1:41 (langgraph_api chown is 0.2s)
mastra post-fix: 2:16
Smoke test on ag2 image: container starts clean, all files under /app
owned by app:app, /api/health returns 200.
Under the 23-way Depot fan-out the real-world win is substantially
larger than the local 50s baseline because recursive chown degrades
super-linearly with concurrent I/O pressure.
Failed run that motivated this: https://github.com/CopilotKit/CopilotKit/actions/runs/24621469277
Cold-start on showcase packages with heavy Python agents (agno in particular)
consistently lands just over the 25s budget, producing 502s on first probe
with latency_ms: 25001 and stage: "timeout". Raise the upstream
AbortSignal.timeout on /api/smoke from 25s to 45s across all 16 showcase
packages that exercise the full agent round-trip, and bump Next.js
maxDuration from 30s to 60s so the route can actually run that long.
Also bumps the create-integration template so new packages inherit the
new budget.
Tail-latency beyond 45s will still alert — which is the intent.
Evidence: agno 502 alerts tonight at 18:40 PDT and 19:42 PDT, both with
latency_ms: 25001, stage: "timeout" on /api/smoke. /api/health already
200 on both probes — pure cold-start boundary issue.
## Summary
Adds a one-command local smoke harness so the full 17-integration suite
can be exercised against Docker on the dev machine instead of Railway.
Useful when Railway is degraded (aimock OOM, rate limits, cold-start
drift) or when validating changes that haven't been deployed yet.
## Usage
```bash
# one-time
cp showcase/.env.example showcase/.env # fill in keys
pnpm --filter @showcase/e2e-smoke install
# full L1-L4 smoke
pnpm --filter @showcase/e2e-smoke smoke:local
# single level / keep containers up between runs
pnpm --filter @showcase/e2e-smoke smoke:local:L1
pnpm --filter @showcase/e2e-smoke smoke:local:keep
pnpm --filter @showcase/e2e-smoke smoke:local:nobuild
```
## What's in here
- **`docker-compose.local.yml`**: `aimock` added as 18th service →
integration containers reach `http://aimock:4010` on the compose
network, mirroring Railway's `showcase-aimock`.
- **`integration-smoke.spec.ts`**: `LOCAL_PORTS=1` env gates URL
rewriting from `https://showcase-<slug>-production.up.railway.app` →
`http://localhost:<port>` via `shared/local-ports.json`. Starters are
skipped under the flag because they're not in `local-ports.json`.
- **`scripts/smoke-local.sh`**: thin orchestrator — `build → up → wait
20s → playwright → down`. Flags: `--level=L1|L2|L3|L4`, `--keep`,
`--no-build`.
- **`tests/package.json`**: `pnpm smoke:local[:L1|:keep|:nobuild]`
wrappers.
- **`.env.example`**: documents optional
`OPENAI_BASE_URL`/`ANTHROPIC_BASE_URL` + `GitHubToken` (ms-agent-dotnet)
and `GOOGLE_API_KEY` (google-adk).
## Verification
Run locally against a fresh checkout of this branch:
- `LOCAL_PORTS=1 SMOKE_ALL=true npx playwright test integration-smoke
--grep @health` → **17/17 pass in 478ms**
- Full L1-L4 against the local stack → **42/51 pass** (9 failures in
L3/L4 for mastra, google-adk, ms-agent-dotnet, strands, langroid,
spring-ai — these are test-data / fixture gaps unrelated to this
infrastructure and will be filed separately)
- `docker compose -f showcase/docker-compose.local.yml config` validates
with 18 services
## Scope
Pure dev-ergonomics addition. No runtime behaviour changes in the
shipped containers. `LOCAL_PORTS` is opt-in; unset = existing
Railway-URL behaviour preserved.
## Test plan
- [ ] `Validate Showcase` CI still green (no package-source changes)
- [ ] No unrelated CI regressions
- [ ] Follow-up PR will investigate and fix the 9 L3/L4 failures
surfaced by local smoke
- test-cleanup.ts: `new Error(msg, { cause })` is ES2022; workspace lib is
ES2020 so the two-arg overload is missing. Replaced with an
`errorWithCause()` helper that assigns `.cause` after construction.
Runtime is identical (Node >=16.9); only the TS signature differs.
- test-cleanup.ts: retyped `SAFE_STDIO` as `StdioOptions` (still frozen at
runtime to keep `test-cleanup.test.ts` freeze assertion green) so
spreading `SAFE_EXEC_OPTS` into `execFileSync(..., opts)` no longer trips
the readonly-vs-mutable-array mismatch on `stdio` (fixes the error at
create-integration.test.ts:132).
- create-integration/index.ts: dropped unused `devCmd` local and unused
`args` parameter on `generateDemoPage` (+ call site); both were dead code
introduced during the parallel-isolation refactor.
- validate-pins.parsers.test.ts: annotated all `withTmp((tmp) => ...)`
callbacks as `(tmp: string)` for robustness under LSP module-resolution
glitches. Matches the contract in `validate-pins.shared.ts`.
Tests: 1061/1061 pass (`pnpm nx run @copilotkit/showcase-scripts:test`).
Prior reasons for `fileParallelism: false` are resolved:
- Env-var mutation races (VALIDATE_PINS_REPO_ROOT, SHOWCASE_AUDIT_ROOT,
VALIDATE_PARITY_REPO_ROOT) are moot under `pool: 'forks'` — every file
already gets its own node process with its own `process.env`.
- `.git/index.lock` races between suites that call `restoreFromGitHead`
are fixed by the cross-process lock in test-cleanup.ts.
- The create-integration vs generate-registry collision on
`showcase/packages/` is fixed by the create-integration tmpdir
isolation.
Under fork-per-file + parallel, each test file also gets a fresh 60s
birpc `onTaskUpdate` budget (vitest #6129), eliminating the cumulative
RPC back-pressure that tripped unit(20.x/22.x/24.x) on #4018/#4068/#4079.
Empirical: local full suite 158s → 12s, 1061/1061 passing across three
consecutive `--skip-nx-cache` runs with zero timeouts, zero ENOENT, zero
index.lock contention.
audit.test.ts is 3034 lines / 119 tests and on Node 22 CI its single-file
runtime grew from 36.7s (PR #4071) to 71.4s (PR #4081) — over the hardcoded
60s birpc onTaskUpdate RPC window (vitest #6129). Same cliff that motivated
the validate-pins split earlier in this PR.
Extract makeTmpTree, makeConfig, writePackage, makeExampleDir, anomalyStrings,
and the AUDIT_SCRIPT path constant into audit.shared.ts so the forthcoming
split files can share them without duplication. No behavior change — the
original audit.test.ts still re-declares its own local copies until the
split commit removes them.
Jordan Ritter