## Summary
- Docs Sync workflow has been failing since 2026-04-21 with `fatal:
pathspec 'showcase/shell/src/content/' did not match any files` (exit
128) on the "Create PR for docs sync" step (example: run 24726678848).
- The path was moved by the shell platform restructure (#4109, #4112) —
the docs content now lives under `showcase/shell-docs/src/content/`, and
the sync marker at `showcase/shell-docs/.docs-sync-sha`. The sync script
(`showcase/scripts/sync-docs-from-main.ts`) already writes to those
locations; only the workflow's `git add` was still referencing the old
monolithic `showcase/shell/` path.
- One-line path fix.
## Test plan
- [ ] Verify yaml parses.
- [ ] Confirm next docs-sync run succeeds end-to-end (opens a sync PR
for the 11 pending review items from run 24726678848).
## Incident
On 2026-04-21, 18 production Railway showcase services were found with
malformed image refs of the form
`ghcr.io/copilotkit/showcase-<slug>atest` — the `:` before `latest` was
dropped, so Docker treated `...atest` as the tag. Railway dutifully
tried to pull an image that doesn't exist on GHCR, so every affected
service began failing to redeploy.
The underlying data has been cleaned up, but the corruption did NOT come
from committed code — it was introduced by an out-of-band MCP/manual API
mutation. Nothing in the repo would have caught this, and nothing
prevents it from happening again.
## Current state
- 41 showcase services now have correct image refs (verified live
against Railway in this PR's local run).
- No source-controlled guardrail exists today to catch future drift.
## New assertion
`showcase/scripts/verify-railway-image-refs.ts` — queries Railway
GraphQL for every service in the CopilotKit Showcase project
(`6f8c6bff-...`) and asserts each service's image source matches:
```
ghcr.io/copilotkit/<service-name>:latest
```
The regex `^ghcr\.io\/copilotkit\/showcase-[a-z0-9-]+:latest$` covers
every service class — `showcase-<slug>`, `showcase-starter-<slug>`,
`showcase-aimock`, `showcase-pocketbase`, `showcase-ops`. The script
additionally requires the image's trailing path segment to match the
Railway service name exactly (identity invariant: service name ↔ image
name).
On violation the script prints the service name, current image, expected
shape, and reason — then exits 1. On success: `✓ N services verified`,
exit 0.
## Wire-up
Added a `verify-image-refs` job to
`.github/workflows/showcase_deploy.yml` that runs in parallel with
`check-lockfile` and before the deploy matrix. If drift is detected, the
workflow aborts before any build starts. The `notify` job classifies a
verify failure distinctly so Slack alerts say "Railway image-ref drift
detected" rather than a generic pre-build failure.
Uses the existing `RAILWAY_TOKEN` secret already plumbed into the deploy
step — no new secrets needed.
## Local verification
Normal run (all 41 services correctly configured):
```
✓ 41 services verified
```
Simulated-corruption cases (unit-tested the exported `validateImage`):
- `showcase-mastra` → `ghcr.io/copilotkit/showcase-mastraatest` →
**rejected** (the exact incident shape)
- `showcase-mastra` → `ghcr.io/copilotkit/showcase-crewai-crews:latest`
→ rejected (service↔image mismatch)
- `showcase-mastra` → `ghcr.io/copilotkit/showcase-mastra` → rejected
(no tag)
- `showcase-mastra` → `docker.io/copilotkit/showcase-mastra:latest` →
rejected (wrong registry)
- `showcase-mastra` → `ghcr.io/copilotkit/showcase-mastra:v1` → rejected
(wrong tag)
- `showcase-mastra` → `null` → rejected (no source)
- Plus 3 valid cases (`showcase-mastra`, `showcase-starter-agno`,
`showcase-aimock` all at `:latest`) → accepted.
All 9/9 behave as expected.
## Test plan
- [x] `npx tsx showcase/scripts/verify-railway-image-refs.ts` against
current Railway state → `✓ 41 services verified`
- [x] Simulated-corruption validation → `9/9` rejection/acceptance
behaviour correct
- [x] `tsc --noEmit` clean on `showcase/scripts`
- [x] `prettier --check` passes on both modified files
- [x] `pnpm --filter=@copilotkit/showcase-scripts test` → 21 files /
1079 tests pass (no regressions from pre-existing suite)
- [x] YAML parse of `showcase_deploy.yml` succeeds
- [ ] CI workflow run on this PR includes the new `verify-image-refs`
job and it goes green against current Railway state
Regression from the 2026-04-21 incident: 18 production Railway services
were found with malformed image refs of the form
`ghcr.io/copilotkit/showcase-<slug>atest` (missing the `:` before
`latest`, so Docker treats `...atest` as the tag). Root cause was an
out-of-band MCP/manual mutation — no committed code touched those refs,
so the data has been fixed but no source-controlled guardrail exists.
Add a standalone script that queries Railway's GraphQL API for every
service in the CopilotKit Showcase project and asserts each image ref
matches the canonical shape `ghcr.io/copilotkit/<service-name>:latest`.
Wire it into showcase_deploy.yml as a pre-build job so any drift aborts
the workflow before the build matrix fans out.
On violation the script prints the service name, the current image, the
expected shape, and the reason, so the fix is obvious in the run log.
Slack classification in the notify job distinguishes a drift failure
from other pre-build failures.
Verified locally: 41 services pass against current Railway state; the
exported `validateImage` function rejects the exact `...atest`
corruption, mismatched service/image names, missing tags, wrong
registries, wrong tag values, and null sources (9/9 simulated cases).
The "Showcase: Docs Sync" workflow has been failing with
`fatal: pathspec 'showcase/shell/src/content/' did not match any files`
since the shell platform restructure (#4109, #4112) split the monolithic
`showcase/shell/` package into `showcase/shell-docs/`,
`showcase/shell-dashboard/`, and `showcase/shell-dojo/`.
The sync-docs-from-main.ts script already writes to
`showcase/shell-docs/src/content/` and maintains the sync marker at
`showcase/shell-docs/.docs-sync-sha`. Update the "Create PR for docs
sync" step to stage from the new location so the workflow can commit
synced content again.
## Summary
Generalizes the silent-hang watchdog + unbuffered-stdout shape proven in
`showcase/packages/crewai-crews/entrypoint.sh` (PRs #4114 + #4115) to
every Bucket-B starter, the Bucket-C spring-ai starter, the generator,
and the shared starter template.
**Reference shape:** `showcase/packages/crewai-crews/entrypoint.sh` on
`origin/main` (unchanged in this PR; we generalize from it).
**Notion proposal:**
https://www.notion.so/3493aa3818528191bc08f8b2f3fb1e88
**Prior art:** #4114 (watchdog), #4115 (unbuffered stdout)
## What the watchdog does
A backgrounded subshell polls the agent's health endpoint every 30s;
after 3 consecutive failures (~90s), it `kill -9`s the agent so `wait
-n` returns and the container runtime restarts via the normal path
rather than a forced `exit` that would bypass logging.
## Per-starter coverage
| Slug | Framework | Agent port | Health path | Watchdog applied |
|---|---|---|---|---|
| ag2 | Python / FastAPI | 8123 | `/health` | Yes |
| agno | Python / FastAPI | 8123 | `/health` | Yes |
| claude-sdk-python | Python / FastAPI | 8123 | `/health` | Yes |
| claude-sdk-typescript | TypeScript / Express | 8123 | `/health` | Yes
|
| crewai-crews | Python / FastAPI | 8123 | `/health` | Yes (via regen
from template; matches reference shape) |
| google-adk | Python / FastAPI | 8123 | `/health` | Yes |
| langgraph-fastapi | Python / langgraph_cli | 8123 | `/ok` | Yes |
| langgraph-python | Python / langgraph_cli | 8123 | `/ok` | Yes |
| langgraph-typescript | TS / @langchain/langgraph-cli | 8123 | `/ok` |
Yes |
| langroid | Python / FastAPI | 8123 | `/health` | Yes |
| llamaindex | Python / FastAPI | 8123 | `/health` | Yes |
| mastra | TypeScript / mastra dev | 8123 | `/api` | Yes (starter only;
package side is single `exec next start` — N/A) |
| ms-agent-dotnet | .NET / ASP.NET | 8123 | `/health` | Yes |
| ms-agent-python | Python / FastAPI | 8123 | `/health` | Yes |
| pydantic-ai | Python / FastAPI | 8123 | `/health` | Yes |
| spring-ai | Java / Spring Boot | 8123 (starter) / 8000 (package) |
`/health` | Yes; extends the existing 60s startup probe |
| strands | Python / FastAPI | 8123 | `/health` | Yes |
## Per-package coverage (hand-applied, not regen-able)
Every `showcase/packages/<slug>/entrypoint.sh` that runs a separate
agent process received the watchdog + unbuffered block. Python variants
also received `python -u` on their uvicorn / langgraph_cli invocations
and `awk ... fflush()` log prefixing (replacing the prior `sed` pipe
formulation so `$!` correctly captures the agent PID).
**N/A:** `packages/mastra/entrypoint.sh` is a single `exec npx next
start` — no backgrounded agent to watch.
## Generator + template changes
- `showcase/scripts/generate-starters.ts` — new `getAgentHealthPath()` +
`getWatchdogBlock()` helpers, plus Spring Boot `/health` 60s startup
probe.
- `showcase/starters/template/entrypoint.template.sh` — adds
`$WATCHDOG_PID` to cleanup, injects `{{WATCHDOG_BLOCK}}`, narrows the
final `wait -n` to `$AGENT_PID $NEXTJS_PID` only.
## Out of scope (not touched)
- `showcase/packages/crewai-crews/entrypoint.sh` (reference shape;
already correct on main).
- `showcase/{shell,shell-dashboard,shell-docs}` (separate workstream).
- Any agent source (Python / TS / Java / .NET).
- Dependency version bumps.
## Test plan
- [x] `bash -n` clean on all 32 modified `.sh` files.
- [x] `pnpm -C showcase/scripts test` — 1079/1079 tests pass (includes
starter-output snapshots).
- [x] Watchdog marker present in every modified
`packages/<slug>/entrypoint.sh` except mastra (intentional N/A).
- [x] `PYTHONUNBUFFERED=1` present in every Python-based package
entrypoint.
- [ ] CI green before merge.
- [ ] Post-merge: observe one deploy per starter on Railway to confirm
watchdog doesn't false-positive against a healthy agent.
## Judgment calls (documented)
- **`starters/crewai-crews/agent/{requirements.txt,agent_server.py}`**:
kept as regen output. These match the current `packages/crewai-crews/`
source (shimless, `ag-ui-crewai >=0.2.0`) that landed on main in commit
9379b8855. Reverting would re-desync the starter from the upstream
source the generator reads.
- **`langgraph-typescript` packages side**: classified as Bucket B. The
file on `origin/main` backgrounds `@langchain/langgraph-cli dev` on
`:8123` and then `exec`s Next.js — the langgraph process can hang
independently, so the watchdog applies.
- **`mastra` packages side**: classified as N/A. The file on
`origin/main` is a single `exec npx next start` with no backgrounded
agent.
- **`mastra` starter side**: Bucket B (the starter DOES run `mastra dev`
on `:8123`). Watchdog probes `/api` (mastra's REST surface; no dedicated
health endpoint but returns 200 once the dev server is ready).
Hand-applied watchdog + unbuffered-stdout block for every Bucket-B package
entrypoint where the shape couldn't come from the starter generator:
* ag2, agno, claude-sdk-python, claude-sdk-typescript, google-adk,
langgraph-fastapi, langgraph-python, langgraph-typescript, langroid,
llamaindex, ms-agent-dotnet, ms-agent-python, pydantic-ai, strands
-> watchdog polls the framework-native health endpoint; Python variants
also get `python -u` + awk fflush log prefixing.
* spring-ai (Bucket C) -> watchdog extends the existing startup probe;
polls :8000/health every 30s and kills PID on 3 consecutive failures.
Added WATCHDOG_PID cleanup to the post-wait block.
* mastra is single-`exec next start` on the package side — no agent
process to probe, so no watchdog needed (N/A).
Reference shape: showcase/packages/crewai-crews/entrypoint.sh (unchanged
from origin/main; proven in production via PRs #4114 + #4115).
Regenerates all 16 starter entrypoint.sh files from the new template. Every
starter now ships:
- $WATCHDOG_PID tracked in cleanup + kill block
- PYTHONUNBUFFERED=1 export (harmless for Java/Node/.NET)
- Explicit `wait -n $AGENT_PID $NEXTJS_PID` (excludes the watchdog so its
exit doesn't short-circuit the agent's true status)
- Per-framework health probe (see prior commit for path matrix)
Also reflects the crewai-crews root-cause fix (already on origin/main via
commit 9379b8855): ag-ui-crewai >=0.2.0 bump and drop of the crewai.cli
monkey-patch shim. These files are regenerated from showcase/packages/
crewai-crews source.
Generalize the watchdog shape proven in showcase/packages/crewai-crews/entrypoint.sh
(PRs #4114 + #4115) so that every Bucket-B starter gets:
- PYTHONUNBUFFERED=1 export (harmless for non-Python frameworks)
- A backgrounded watchdog subshell that polls the agent health endpoint
every 30s and kills the agent after 3 consecutive failures, letting
wait -n + container runtime handle the restart through the normal path.
- python -u on uvicorn / langgraph_cli invocations (Python frameworks).
- awk ... fflush() log prefixing (replaces the previous sed pipe; keeps $!
pointing at the real agent process).
Per-framework agent health paths:
* FastAPI / uvicorn agents -> /health
* langgraph-python, langgraph-fastapi, langgraph-typescript -> /ok
* claude-sdk-typescript, ms-agent-dotnet, spring-ai -> /health
* mastra -> /api
Spring Boot starter also gets a 60s /health startup probe (replacing the
blind sleep 5) because JVM warmup + context refresh can exceed 30s.
## Summary
- Bump `ag-ui-crewai` pin to `>=0.2.0,<0.3.0`. 0.1.5 had three defects
(unguarded `source.state.messages`, orphan `asyncio.create_task`, sync
`completion()`) that wedged the agent on 2026-04-21. All fixed in ag-ui
[PR #1550](https://github.com/ag-ui-protocol/ag-ui/pull/1550), released
as 0.2.0 on 2026-04-18 — our pin was blocking it.
- Remove the pre-bind monkey-patch shim in `src/agent_server.py`;
superseded by 0.2.0's native deferred `ChatWithCrewFlow` construction
(module-scoped `_cached_flow` + `asyncio.Lock` inside
`add_crewai_crew_fastapi_endpoint`). Any LLM hiccup now surfaces as a
5xx on first request instead of a startup crash, which is exactly what
the shim was reaching for. `logging` import dropped since it was only
used by the shim.
- Add `python -u` to the uvicorn invocation in `entrypoint.sh` as a
belt-and-suspenders complement to the existing `PYTHONUNBUFFERED=1`.
`awk '{...; fflush()}'` in the pipe was already line-flushing; `-u` is
not overridable by user code. Railway saw only ~15 log lines over 9h
during the 04-21 incident because of pipe-buffering through a previous
`sed` formulation.
- Update `showcase/scripts/fail-baseline.json`'s `validatePinsFailHash`
to match the new `ag-ui-crewai` spec string in the FAIL set. Pin-drift
FAIL count is unchanged (110); only the hashed line text changed.
Root-cause context: Notion [Showcase starters: silent-hang vulnerability
class](https://www.notion.so/3493aa3818528191bc08f8b2f3fb1e88).
## Test plan
- [x] `pip install -r requirements.txt` resolves `ag-ui-crewai==0.2.0`
(verified locally in a fresh 3.12 venv; no dep conflicts with
`crewai==0.130.0` or `crewai-tools`).
- [x] Agent starts cleanly under `python -u -m uvicorn
agent_server:app`; `/health` returns 200 `{"status":"ok"}` (verified
locally).
- [x] Agent import no longer references the `crewai.cli.crew_chat`
hardening shim; `ag_ui_crewai.endpoint.add_crewai_crew_fastapi_endpoint`
registers without pre-bind LLM calls (verified via FastAPI route
inspection after import).
- [x] `pytest tests/python/` — 94/94 pass.
- [x] `pnpm -C showcase/scripts test` — 1079/1079 pass.
- [x] `validate-pins.ts` — FAIL count matches baseline (110); new hash
committed to `fail-baseline.json`.
- [ ] `/api/smoke` completes against live upstream on Railway (requires
deploy).
- [ ] Railway deploy logs show uvicorn request lines (not just
entrypoint banner).
- [ ] Watchdog from #4114 still loops.
## Intentionally out-of-scope (follow-ups)
- `showcase/starters/crewai-crews/` parity backport (starter still
carries the 0.1.5 pin and the shim).
- The 14-starter watchdog generalisation — tracked in the Notion
proposal above.
Root-cause fix for the 04-21 silent-hang incident on the crewai-crews
Railway deploy. Three tightly-coupled changes:
1. Bump ag-ui-crewai pin from `>=0.1.4,<0.1.6` to `>=0.2.0,<0.3.0`.
0.1.5 had three defects that wedged the agent: unguarded
`source.state.messages` access, an orphan `asyncio.create_task` with
no cancel ref, and a sync `completion()` call that pinned the event
loop. All three are fixed in ag-ui PR #1550, released as 0.2.0 on
2026-04-18. Our upper ceiling was blocking the fix.
2. Remove the pre-bind LLM crash hardening shim in `agent_server.py`.
The shim monkey-patched `crewai.cli.crew_chat.generate_*_description_with_ai`
to static strings so that `ChatWithCrewFlow.__init__` — which
ag-ui-crewai <= 0.1.5 invoked at endpoint-registration time, BEFORE
uvicorn bound its port — could not crash the process before the HTTP
server was listening. 0.2.0 defers `ChatWithCrewFlow` construction to
first request via a module-scoped `_cached_flow` + `asyncio.Lock`
inside `add_crewai_crew_fastapi_endpoint`. Any LLM hiccup now
surfaces as a 5xx on the first request instead of a startup crash,
which is what the shim was reaching for. The shim is dead code on
0.2.0 and has been removed (with `logging` import dropped as it was
only used by the shim).
3. Add `python -u` to the uvicorn invocation in `entrypoint.sh` as a
belt-and-suspenders complement to the existing `PYTHONUNBUFFERED=1`
export. The env var can in principle be un-exported by a child;
`-u` forces unbuffered stdout/stderr at the interpreter level and
is not overridable by user code. Combined with `awk '{...; fflush()}'`
in the pipe (already in place), this guarantees uvicorn request
lines reach Railway's log stream line-at-a-time. During the 04-21
incident Railway saw only ~15 log lines over 9h of uptime because
of buffering through a previous `sed` formulation.
Also updates `showcase/scripts/fail-baseline.json`'s `validatePinsFailHash`
to match the new `ag-ui-crewai` spec string. Pin-drift FAIL count is
unchanged (110); the hash changed only because the `ag-ui-crewai` line
in the FAIL set went from `>=0.1.4,<0.1.6` to `>=0.2.0,<0.3.0`.
Verified locally:
- `pip install -r requirements.txt` resolves `ag-ui-crewai-0.2.0` cleanly.
- `python -u -m uvicorn agent_server:app` starts; `/health` returns
200 `{"status":"ok"}`; request lines appear in real-time logs.
- `pytest tests/python/` — 94/94 pass.
- `pnpm -C showcase/scripts test` (vitest) — 1079/1079 pass.
- `validate-pins.ts` — count=110 matches baseline; hash updated.
Upstream refs:
- crewAI issue: https://github.com/crewAIInc/crewAI/issues/5510
- ag-ui PR #1550: https://github.com/ag-ui-protocol/ag-ui/pull/1550
Intentionally NOT in this PR:
- `showcase/starters/crewai-crews/` parity backport (the starter still
carries the 0.1.5 pin and the shim).
- The 14-starter watchdog generalisation.
Both belong to the silent-hang vulnerability-class work tracked
separately.
## Summary
Production `showcase-crewai-crews` was stuck at `agent:"down"` for ~4-5h
on 2026-04-21 (retries #1-8, 503 on `/api/health`, 30s+ hang on
`/api/smoke`). Railway runtime logs show the FastAPI agent on `:8000`
handled startup and hundreds of requests cleanly, then silently
**stopped responding** at ~06:56-07:01 UTC after an `unhandledRejection:
UND_ERR_BODY_TIMEOUT`. The Python process did not exit — it hung — so
bash's `wait -n` never fired, the container never restarted, and every
subsequent runtime-to-agent request waited out undici's default 5-minute
headers timeout (`UND_ERR_HEADERS_TIMEOUT` in the logs).
Two defensive fixes in the package:
1. **`entrypoint.sh` — agent watchdog.** Poll
`http://127.0.0.1:8000/health` every 30s (5s `curl --max-time`). After 3
consecutive failures (~90s unreachable), SIGKILL the agent PID so `wait
-n` returns and Railway restarts the container. Also adds `[entrypoint]`
/ `[agent]` / `[nextjs]` / `[watchdog]` log prefixes, a startup
PID-is-alive check, and `PYTHONUNBUFFERED=1` — bringing this package's
entrypoint closer to the starter's richer template.
2. **`src/app/api/smoke/route.ts` — split inner/outer timeouts.**
`UPSTREAM_TIMEOUT_MS = 25_000` on the inner fetch, keeping `maxDuration
= 60` as the route budget. Previously the single
`AbortSignal.timeout(45000)` on the upstream fetch, when the agent hung,
consumed nearly all the 60s envelope before Next.js cut the request —
producing HTTP 000 at the caller. Now a wedged agent surfaces as a
structured `stage: "timeout"` JSON response in ~25s.
Upstream `showcase-crewai-crews-production.up.railway.app` verified RED
(503, agent:down) via `curl` before this PR. Merge will trigger GHCR
build + Railway redeploy; recovery expected within ~4 min of deploy.
## Test plan
- [ ] Railway deploys latest image after merge; container boots healthy
- [ ] `curl
https://showcase-crewai-crews-production.up.railway.app/api/health`
returns 200 `agent:"ok"`
- [ ] `curl
https://showcase-crewai-crews-production.up.railway.app/api/smoke`
returns 200 within ~25s (or 502 `stage:"timeout"` if aimock glitches)
- [ ] Logs show `[entrypoint]` / `[agent]` / `[watchdog]` prefixes
(evidence the new entrypoint is active)
- [ ] Induce hang by sending a malformed agent request; verify watchdog
kicks in after ~90s (visible in logs as `[watchdog] Agent health probe
failed` + container restart)
- [ ] Starter service (`showcase-starter-crewai-crews`) unaffected —
only the package entrypoint/smoke route changed
Draft until: feature flag strategy is in place.
<!--
Thank you for sending the PR! We appreciate you spending the time to
work on these changes.
Help us understand your motivation by explaining why you decided to make
this change.
**Please PLEASE reach out to us first before starting any significant
work on new or existing features.**
By the time you've gotten here, you're looking at creating a pull
request so hopefully we're not too late.
We love community contributions! That said, we want to make sure we're
all on the same page before you start.
Investing a lot of time and effort just to find out it doesn't align
with the upstream project feels awful, and we don't want that to happen.
It also helps to make sure the work you're planning isn't already in
progress.
As described in our contributing guide, please file an issue first:
https://github.com/ag-ui-protocol/ag-ui/issues
Or, reach out to us on Discord: https://discord.com/invite/6dffbvGU3D
You can learn more about contributing to copilotkit here:
https://github.com/copilotkit/copilotkit/blob/master/CONTRIBUTING.md
Happy contributing!
-->
## What does this PR do?
Adds initial pass at docs for new CopilotKit agent framework-agnostic
Threads feature.
Since part of that included tutorials and arch discussion under the
"Learn" heading, this also adds a dedicated "Learn" landing page
(before, in the absence of one, the user would always be redirected to
the first page under Learn, which was agentic protocols)
## Related PRs and Issues
- ENT-14
## Checklist
- [ ] I have read the [Contribution
Guide](https://github.com/copilotkit/copilotkit/blob/master/CONTRIBUTING.md)
- [ ] If the PR changes or adds functionality, I have updated the
relevant documentation
Production deploy of showcase-crewai-crews was stuck in agent:"down" for
~4-5h on 2026-04-21. Railway deployment logs show the FastAPI agent on
:8000 was reachable at startup and handled hundreds of requests, then
stopped responding at ~06:56-07:01 UTC after an
unhandledRejection: UND_ERR_BODY_TIMEOUT from the Next.js runtime. The
Python process did not exit — it hung — so bash's wait -n never fired,
the container never restarted, and every subsequent request stacked up
on undici's default 5-minute headers timeout (UND_ERR_HEADERS_TIMEOUT).
Two surgical fixes:
1. entrypoint.sh: add a watchdog goroutine that polls
http://127.0.0.1:8000/health every 30s with a 5s curl timeout. After
3 consecutive failures (~90s unreachable) it SIGKILLs the agent PID,
causing wait -n to return and the container to restart. This converts
a silent agent hang into a fast, loggable restart. Also adds structured
[entrypoint] / [agent] / [nextjs] / [watchdog] log prefixes, a startup
PID-is-alive check, and PYTHONUNBUFFERED=1 so crashes surface before
the log pipe closes — mirroring the starter entrypoint pattern.
2. src/app/api/smoke/route.ts: split the upstream fetch timeout
(UPSTREAM_TIMEOUT_MS = 25_000) from the route's overall budget
(maxDuration = 60). Previously the inner fetch had AbortSignal.timeout(45000)
which, on a hung agent, burned nearly all of the 60s before Next.js
killed the request — producing HTTP 000 at the caller instead of a
structured stage: "timeout" JSON response. With the shorter inner
timeout the route can report a clean timeout within ~25s even when
the agent is wedged.
Both fixes are defensive against upstream CrewAI ChatWithCrewFlow
deadlocks (which the logs show happening repeatedly via the pre-existing
"Cannot send 'RUN_FINISHED' while steps are still active" errors). The
watchdog is scoped to this package only; the starter already has a
richer entrypoint with similar intent.
- package.json: remove the view/title binding that put
'Open AG-UI Inspector' on the A2UI Catalog view — the inspector is
its own sidebar tab now, so adding that action to the catalog title
bar was redundant noise. Inspector view keeps its own binding.
- Both sidebar webviews (hook-list + catalog-list) had an in-webview
refresh button that duplicated the native VS Code title-bar refresh
icon already registered under view/title. Drop the React buttons
and rely on the native icons; single source of truth and cleaner
header.
## Summary
Tech-debt follow-up to #4109. This PR fixes a batch of
production-relevant issues across the shell-docs runtime, the shared
showcase deploy workflow, and the content bundler scripts — MDX
rendering correctness, XSS hardening on author-controlled sinks, UI
component invariants under edge cases, route-level content handling, and
CI reliability.
## Why
#4109 restructured shell-docs and landed a lot of new shared surface
area. Reviewing that landing surfaced real bugs that weren't in scope to
fix inline: Slack failure alerts that silently never posted, MDX regex
bugs that mangled nested `<Tabs>`, XSS vectors via `javascript:` hrefs,
iframe sandboxes that negated themselves, undeployed integrations
rendered as clickable dead links, a framework-selector that trapped
users in redirect loops, and a few bundler correctness bugs that
produced captions pointing at the wrong file. All of it is user-visible
or deploy-visible, so none of it can stay deferred.
## What changed
### CI workflow (`showcase_deploy.yml`)
Hygiene on the shared deploy pipeline so failures are reported and
concurrent runs don't race.
- Slack failure notification was gated on a step-local `env:` it
couldn't read from its own `if:`, so alerts never posted. Promoted
`SLACK_WEBHOOK_URL` to job-level env.
- Concurrency group included the service name, so a push event and a
manual `workflow_dispatch service=all` run could race on the same
Railway service. Dropped the suffix.
- Shared-module copy step ran for every service context and polluted
build contexts for shell-family services that already have the full
repo. Scoped to a `showcase/packages/*` allowlist.
- `shell-dashboard` and `shell-docs` path filters didn't watch
`showcase/shared/**` or `showcase/scripts/**`, so shared-code changes
rebuilt `shell` but not its siblings. Extended the filters.
- Dropped a redundant `fromJSON()` on an already-numeric matrix value
and a dead `HTTP_CODE="000"` sentinel.
### MDX renderer correctness
(`showcase/shell-docs/src/lib/docs-render.tsx` + friends)
Regex and rendering bugs that either mangled output or opened XSS holes.
- `convertTablesInJSX` used non-backref alternation on close tags, and
JS alternation is leftmost-first — so `<Tabs>…</Tab>` closed the outer
container with the inner tag. Added a backref and sorted the tag
alternation longest-first.
- `inlineSnippets` matched `components=` props with `[^}]*`, truncating
doubled-brace MDX object literals. Replaced with an explicit `\{\{…\}\}`
match.
- Frontmatter H1 fallback ran against the raw file including the YAML
block, so a `# comment` line inside frontmatter could become the page
title. Moved the match to `matter(raw).content` and surfaced the
previously-silent parse catch as a log.
- XSS: `<A>` and `<Link>` rendered any href scheme verbatim, making
`javascript:` / `data:text/html` a live vector from MDX. Added a scheme
allowlist; unsafe schemes render as inert text.
- Iframe sandboxes in `InlineDemo`, `IframeSwitcher`, and `YouTubeVideo`
combined `allow-scripts` with `allow-same-origin`, which per MDN lets
the iframe remove its own sandbox. Dropped `allow-same-origin`.
- `<InlineDemo>` piped registry `backend_url` into `src` without running
it through `validateIframeSrc` like the other iframe sinks do.
Harmonized.
- `<img>` and `<video>` style-prop handling spread caller props and then
clobbered `style` wholesale, dropping author-supplied keys. Now merges,
with component-enforced layout overrides winning.
- `docs-page-view` rendered the FM title as an `<h1>` and then let the
MDX body's duplicate `# Title` render again, producing stacked H1s.
Strips the body H1 only when it matches the FM title after whitespace
normalization; CRLF-safe.
### Component invariants
Tabs, framework-tabs, framework-provider, framework-selector — edge
cases around duplicate labels, mismatched lengths, and stale state.
- `docs-tabs` picked the active panel via `.filter(label === active)`,
which double-rendered on duplicate labels and blanked out when `items`
length diverged from children. Replaced with a positional `findIndex` so
pairing is strictly by index.
- `framework-tabs` paired `frameworks[i]` with children by index; a
child rendering null shifted the array and every subsequent tab showed
the wrong framework's content. Added a count-mismatch guard with dev
warn. Seeded `active` only at mount, so swapping `frameworks` left it
stale — added a re-sync effect. Removed never-consumed `cell`/`region`
interface props.
- `framework-provider` hydrated `stored` from localStorage without
validating against the registry, so a stale slug (framework removed from
registry) kept redirecting users to a dead route. Added the same
`knownFrameworks.includes(...)` check used elsewhere, including the
cross-tab `storage` handler.
- `framework-selector` dropdown rendered undeployed options as
clickable; clicking persisted the undeployed slug and `router.replace`'d
into a dead route, trapping users in a `RouterPivot` redirect loop.
Gated undeployed options with `disabled`, aria, and a click-handler
guard.
### Route-level content handling
Ag-ui page, docs root, framework route, reference route, brand-nav —
duplicate H1s, stale sidebars, dead links, wrong cross-links.
- `ag-ui/[[...slug]]` H1 strip wasn't CRLF-safe and unconditionally
removed the first body H1 even when it didn't match the FM title (silent
content loss). Now extracts body H1 separately and strips only on
whitespace-normalized match.
- `readTitleForSlug` only read `${slug}.mdx`, but the page handler
already falls back to `${slug}/index.mdx`. Sidebar titles drifted from
actual page content. Mirrored the page handler's precedence.
- Docs overview integration grid rendered undeployed integrations as
active `<Link>`s, handing users dead-link cards. Undeployed now render
as non-link `<div>` with preserved visual treatment. Also filtered
`findFrameworksWithCell` and `previewUrl` derivation to `deployed ===
true`.
- `[framework]/[[...slug]]` alternative-framework banner pulled from all
integrations unfiltered; `slice(0, 3)` could pad with undeployed.
Filtered before slice.
- `reference/[...slug]` fallback `<h1>` showed the raw hyphenated slug
while the breadcrumb was `titleCase`d. Wrapped the fallback too. Added
an `index.mdx` route fallback matching `loadDoc`'s convention.
- `brand-nav` mobile menu's cross-brand link was hardcoded to `/ag-ui`,
so on an ag-ui page the link pointed at itself. Now branches on
`activeBrandFromPath()`.
### Error boundary
The log effect re-fired on every render because the `error` object
identity is fresh each pass, filling console with duplicate reports. Dep
array now tracks `error.message` + `error.digest` (stable strings); the
raw `error` is still passed through in the body for stack preservation.
### Bundler scripts (`bundle-demo-content.ts`, `reference-items.ts`)
Correctness and platform parity.
- Multi-file regions (same region name in multiple files) kept the first
file's span while appending code from all contributors — the caption
said `foo.tsx · L5–L20` while showing content from elsewhere. Track
contributors per region; relabel `file` to `(multiple: a, b, c)` when
more than one.
- Same-file multi-slice regions never extended `endLine`, so captions
under-reported the span.
- `fs.watch({ recursive: true })` is silently ignored on Linux; watch
mode claimed to be watching while missing nested edits. Added a loud
startup warning on Linux.
- Watch-mode path filter used forward slashes only, so on Windows
`fs.watch`'s backslashed paths matched no directory pattern. Normalized
before testing.
- `walkMdx` emitted `<parent>/index` for `parent/index.mdx`,
inconsistent with `loadDoc`'s collapsing to `parent`. Collapsed, and
added a collision guard preferring the flat file when `parent.mdx` also
exists.
### Regenerated `demo-content.json`
Both the shell and shell-docs bundles, carrying the new `(multiple:
...)` caption format and previously-stale committed state.
## Test plan
- [ ] Local typecheck passes on `showcase/shell-docs` and
`showcase/scripts`
- [ ] CI matrix green (Python, Vercel previews, check-binaries, format,
oxlint, package-quality, unit 20/22/24, commitlint)
- [ ] Manual spot-check: MDX pages with nested `<Tabs>` render correctly
(no stranded close tags)
- [ ] Manual spot-check: framework-selector dropdown — undeployed
entries are disabled
- [ ] Manual spot-check: docs overview page — undeployed integration
cards are non-clickable
- [ ] Manual spot-check: trigger showcase_deploy on a path-filter test
change touching `showcase/shared/**` — confirm shell, shell-dashboard,
AND shell-docs rebuild
Directory-plus-index.mdx layout produced two reference entries for
the same logical page: one at `foo/` (the index) and one at `foo`
(a flat sibling if it existed). Collapse index.mdx into the parent
slug so the generated list has a single canonical entry, and bail
out with a clear error when a flat-file collision would shadow the
index. Prevents silently dropping one of the two pages at build
time.
Correctness and portability fixes in the demo-content bundler:
- Track contributor snippets per-file so edits to multiple files in
one commit all get attributed, not just the last one walked.
- Extend endLine when the same file is seen again rather than
dropping the earlier slice — previously a later, smaller region
overwrote a larger one.
- Warn when the watch flag is set on Linux without the recursive
fs.watch support matrix, so the user sees why nothing is firing
instead of assuming silent success.
- Normalize path separators for Windows so bundle manifests use
POSIX paths regardless of the host OS.
The log effect depended on the error object identity, which changes
on every render even when the underlying error is the same — so the
effect fired repeatedly and spammed the log. Depend on
error.message and error.digest (primitive, stable across renders
of the same error) instead. React's exhaustive-deps check is still
satisfied because those are the fields the effect actually reads.
A cluster of UX correctness fixes across the page handlers and
the shared brand nav:
- Filter out undeployed frameworks before rendering so the route
doesn't produce blank pages that users can reach via stale links.
- Strip the leading body H1 with a CRLF-safe regex that only matches
when the body H1 equals the frontmatter title — mirrors ag-ui
route behavior so two stacked titles never render.
- Reference routes now titleCase the slug and resolve via the
index.mdx fallback, matching the directory-plus-index layout the
docs source uses.
- ag-ui title resolver gains a fallback so deep slugs without a
matching registry entry still produce a reasonable title instead
of crashing.
- brand-nav builds the mobile link href dynamically so it points at
the current framework rather than a hard-coded placeholder.
The selector dropdown previously rendered every framework entry as
a clickable option, including ones that had been marked as not
deployed. Selecting an undeployed entry routed to a blank page.
Disable the button for those entries so the dropdown matches the
surrounding tab behavior, which already hides them.
localStorage can hold a value from a previous deploy that no longer
exists in the current frameworks list (package renames, undeploy,
etc.), which pins the provider to a bogus framework identity until
the user manually picks another. Validate the stored value against
the current list both at mount and on cross-tab `storage` events,
and fall back to the default when it doesn't match.
Framework-tabs assumed the items array always matched the frameworks
prop one-to-one. A count mismatch wedged the active index at a stale
value that could fall outside the new frameworks array, rendering a
blank tab. Add a length-mismatch guard that re-syncs state when the
frameworks prop changes between renders, and remove props that were
threaded through but never read so the surface matches what the
component actually uses.
When the tab item array contains duplicate values or has a length
that doesn't match the items prop, looking up the active index by
value conflates tabs. Find the active tab by position instead so
each rendered tab has a stable identity regardless of duplicated
or drifted values.
The img and video overrides spread props, then replaced style wholesale
with the layout defaults — silently discarding any author-provided
style keys from MDX. Merge author style first and layout defaults
last so we preserve author intent (e.g. custom max-width, filters)
while still enforcing our rounded-corner + marginBottom guards.
Per MDN, combining `allow-scripts` with `allow-same-origin` lets the
framed page remove its own sandbox attribute at runtime — that is a
sandbox escape regardless of how much we trust the origin. Drop
`allow-same-origin` from all three iframe sandboxes (InlineDemo,
author-supplied IFrame, YouTube embed). None of them need same-origin
semantics with the parent docs host to function.
Also validate InlineDemo's computed demo URL at the sink instead of
trusting the registry blindly — a malformed backend_url should render
a visible error placeholder, not a silently broken iframe.
MDX-authored hrefs flow into <a> and <Link> without scheme filtering.
`rel="noopener noreferrer"` does not neutralize script-URL schemes
like `javascript:`, `data:text/html`, or `vbscript:`, so a malicious
snippet could land in the rendered page as an XSS vector.
Add sanitizeHref() with an allowlist of http/https/mailto/tel plus
relative / protocol-relative / fragment / query forms. Anything else
returns null and the A/Link overrides render a <span> with the same
visible content instead of an anchor. Also add a protocol-relative
(`//`) early-return in isExternalHref so those URLs correctly route
through a plain <a> rather than next/link.
The title resolution path had two correctness bugs:
- readTitle and loadDoc both ran the H1 fallback regex against the
raw file, so a YAML comment like `# ...` inside frontmatter could
be picked up as the page's H1. Run the regex against the parsed
body (frontmatter stripped) instead. Also log a console.error when
frontmatter parsing blows up so a malformed file doesn't silently
yield a garbage title.
- docs-page-view unconditionally stripped the leading body H1, which
dropped distinct H1 headings whenever the MDX body didn't match
the frontmatter title. Mirror the ag-ui route: only strip when the
body H1 equals the FM title after whitespace normalization, and
use `\r?\n` so CRLF-authored MDX parses correctly too.
MDX authors pass component-map overrides via doubled-brace object
syntax, e.g. `<SharedContent components={{ Foo: Bar }} />`. The
previous regex used `[^}]*` which truncates at the inner `}`, so
the tag never matched and the snippet was never inlined. Match the
doubled-brace object form explicitly with `\{\{...\}\}` and
`[\s\S]*?` so multi-line prop objects still match.
JS regex alternation is leftmost-first, not leftmost-longest, so for
source like `<Tabs>` the pattern `Tab|Tabs` matches "Tab", `[^>]*`
consumes the "s", and the close-side alternation would then pair
with any container's `</...>` regardless of which tag opened. Two
separate bugs fell out of this:
- Prefix collisions (Tab vs Tabs) silently skipped table conversion.
Sort JSX_CONTAINER_TAGS longest-first so the opener always matches
the actual outer tag name.
- Mismatched pairs like `<Tabs><Tab>x</Tab></Tabs>` paired the outer
`<Tabs>` with the inner `</Tab>`, stranding `</Tabs>`. Use a
numbered backref (`\\2`) on the closing tag so only the same tag
name can close the match.
Also tighten the inline comment on the nested-same-tag bailout to
reflect that the non-greedy match now closes on the same tag, not
"any container in the set".
matrix.service.timeout is already a number in the generated matrix
JSON, so wrapping it in fromJSON() was a no-op that only obscured
the expression. Drop the wrapper.
The HTTP_CODE="000" sentinel assigned immediately before the probe
loop is dead: the loop's first iteration unconditionally overwrites
HTTP_CODE before any reader runs. Remove it to avoid implying a
meaningful default where there isn't one.
The shell-family starters bundle shared demo content and tooling at
build time via showcase/scripts/bundle-demo-content.ts, which walks
showcase/shared and showcase/packages/*/manifest.yaml. A change to
any of those inputs can alter the generated bundle without touching
the package directory, so path-filter was under-reporting changes
and skipping deploys that actually needed to rebuild. Extend the
filters to shared/**, scripts/**, and packages/*/manifest.yaml so
those inputs trigger the correct downstream deploys.
- Promote SLACK_WEBHOOK_OSS_ALERTS to job-level env so the step-level
if: gate can actually see it. Step-level env: is not visible to the
same step's if:, which silently disabled every Slack post.
- Drop the service suffix from the concurrency group so two manual
dispatches for different services don't serialize unexpectedly and
push-triggered runs cancel each other deterministically.
- Rewrite the shared-module copy skip guard as an allowlist scoped to
showcase/packages/*. Starters, shell-family, aimock, and shell-dojo
don't consume shared modules, so the previous denylist missed every
new self-contained context (context=".") and polluted the repo root.
Add a card for the new Intelligence Platform explainer to the Learn
index card grid, positioned between Architecture and Threads so the
landing page reads overall architecture → platform → features → protocols.
Without this, the page is reachable only via the sidebar and is
invisible to evaluators browsing /learn top-down.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add the documentation kit for self-hosted CopilotKit Intelligence:
- (root)/premium/self-hosting.mdx — canonical how-to covering platform
prerequisites, values file configuration (AWS ALB/IRSA and on-prem
nginx paths), secret delivery via External Secrets or direct
Kubernetes Secrets, install/verify/upgrade/uninstall steps, and an
inline values.yaml reference
- learn/intelligence-platform.mdx — Diataxis explanation covering the
app-api / app-frontend / realtime-gateway workloads, the
control-plane/data-plane split, the multi-tenancy model
(organization/project/user with auth.deploymentMode postures), the
platform-layer vs application-layer split for self-hosted installs,
secret rotation, scaling, upgrade, error handling, and the
hosted-vs-self-hosted decision
- snippets/shared/premium/self-hosting.mdx — shared snippet that
mirrors the canonical how-to for integration pointer pages
- integrations/*/premium/self-hosting.mdx (12 frameworks) — thin
pointer pages that import the shared snippet, so Self-Hosting is
discoverable from every integration sidebar (a2a has no /premium/
subdir and is intentionally skipped)
- meta.json nav entries in (root)/premium, learn, and every integration
premium sidebar
Also wraps the remaining Threads docs with the ThreadsEarlyAccess gate
(root threads.mdx, shared threads snippet, multi-conversation-chat
tutorial) and replaces the previous "self-hosted Enterprise, configure
your database connection per the Enterprise setup guide" placeholders
with real /premium/self-hosting links — the Threads docs referenced a
setup guide that did not exist until now.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wrap the Threads documentation surface with the existing
InsecurePasswordProtected component via a thin ThreadsEarlyAccess
wrapper that surfaces Threads-specific splash copy and the shareable
early-access URL. Register the wrapper in both route handlers so the
gate works for docs under both (home) and integrations slug routes.
Wraps learn/threads.mdx and reference/v2/hooks/useThreads.mdx; the
remaining Threads pages are wrapped in the follow-up commit that also
fixes the cross-links to the new Self-Hosting docs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- bundle-demo-content: reject highlight: paths that resolve outside
the package root. The bundle output is committed to the repo and
consumed by both shells at build time, so a malicious or
mistake-riddled manifest could otherwise smuggle arbitrary
filesystem contents (../../secrets, absolute paths) into
demo-content.json. Resolve relative to pkgRoot and throw on
escape (finding #19).
- generate-registry: add a runtime guard that manifest.slug is a
non-empty string before path.join(PACKAGES_DIR, manifest.slug).
Schema validation upstream already enforces this, but a
silently-undefined slug fed to path.join yields
"<packages-dir>/undefined" and would produce an empty docs_links
without surfacing any error. Fail loudly instead (finding #20).
- Extend the binary-extension regex to cover .class, .jar, .pyd,
.pyc, .node, .bin, .pdb, .zip, .whl — any of which a build tool
can drop into a PR and none of which the previous regex caught.
- Add set -euo pipefail at the top of the run block. Previously
a transient wc failure (file deleted mid-diff) produced an empty
SIZE, which then failed [ "$SIZE" -gt ... ] with "integer
expression expected" on stderr while the script marched on —
silent false-green possible. Now errors fail the job loudly.
- Guard the wc -c call with an explicit numeric-regex check on
SIZE. A non-numeric or blank result now emits a ::warning:: and
skips the file rather than falling into the arithmetic
comparison and crashing under set -e (finding #17, #18).
- Add a comment to the prior-deploy Railway GraphQL query explaining
that deployments(first: 1) relies on the API default of
createdAt DESC ordering. Railway has no explicit orderBy argument
on this field; documenting the assumption means a future schema
change (or an operator reading the file cold) has something to
point at before debugging stale-deployment guard regressions
(finding #15).
- Harden both truncate_csv shell helpers (one per run block): disable
pathname globbing via set -f for the iteration so a slug
containing a glob char (*, ?, [) cannot expand against the
filesystem, and lock IFS to whitespace for deterministic
word-splitting regardless of caller env. Restore the prior glob
state on the way out so other shell blocks see no side effect
(finding #16).
Extract the near-identical card markup from both route-level
error.tsx files into src/components/error-boundary-card.tsx. The
route-level handlers remain required (Next.js discovers them per
route segment) but are now one-line wrappers that forward props
with a scope label ("docs" / "ag-ui") so console.error entries
stay distinguishable per route segment (finding #14).
- Replace the className="capitalize" breadcrumb span (which only
cased the first character, rendering my-component as
My-component) with an explicit titleCase helper that splits on
hyphens and capitalizes each segment (finding #12).
- Derive the sidebar category list from loadAllReferenceItems
instead of hardcoding ["Components", "Hooks"]. A new
REFERENCE_SUBDIRS entry in reference-items.ts now shows up
automatically without a second edit here (finding #13).
- titleFromSlug now Title-Cases the result (multimodal-inputs →
Multimodal Inputs) instead of returning a lowercased fallback.
The lowercase labels previously clashed with the docs-render
and reference-breadcrumb conventions elsewhere (finding #10).
- Wrap getTitleForSlug in a process-scoped cache with dev-mode
invalidation, mirroring reference-items.ts. Previously every
entry in NAV_DEFINITION re-opened and re-parsed its MDX file on
every request — multiply across ~20 nav entries per render and
it added noticeable fs churn (finding #11).
- Reject RESERVED_ROUTE_SLUGS at the handler as defense-in-depth.
Next.js already prefers exact-match top-level routes over this
catch-all, so /docs, /ag-ui, etc. never reach here under normal
routing — but if the registry ever ships an integration whose slug
collides with a reserved segment, the handler now short-circuits
with notFound() rather than rendering garbage (finding #6).
- Import the shared findFrameworksWithCell helper instead of a
duplicated local copy (finding #3 follow-through).
- Filter out unresolved alternative-framework slugs BEFORE mapping
to React fragments so the ", " separator aligns with the final
rendered count. The pre-filter index previously emitted stray
commas when an entry dropped to null mid-sequence (finding #7).
- FrameworkLandingPage's four hardcoded LandingCards are now gated by
registry integration.features presence: strands (no HITL support)
no longer shows a dead "Human-in-the-Loop" card. The card
definitions are hoisted into a module-level constant with a TODO
pointing at the eventual feature-id → doc-slug mapping
(finding #8).
- Declare export const dynamicParams = true explicitly so a future
migration to output: "export" fails loudly here (under static
export dynamicParams must be false) rather than silently 404ing
every /<framework>/<slug> URL (finding #9).
Jordan Ritter