The inline check-binaries hook broke on Windows Git Bash because lefthook invoked
it via sh.exe -c with the multi-line YAML script as a single argument, and the
nested quotes inside (`echo "$STAGED" | grep -iE '...'`) got mangled during
Windows command-line argument escaping. Move it to scripts/hooks/check-binaries.sh
so lefthook just invokes bash against a file, avoiding the escaping issue.
Also scope the root test script (and test:coverage) to --projects=packages/**,
mirroring check:packages. The previous unscoped nx run-many -t test triggered
showcase starter generation tests that fail on leftover state from prior runs;
these aren't relevant to the pre-commit gate, which is about verifying shipped
packages.
## Summary
Removes all GitHub-issue-creation steps from 3 workflows. Slack alerts
to #oss-alerts are the canonical (and now only) alert channel.
Workflows changed:
- **starter-smoke.yml** — dropped `Create GitHub issue on failure` step
(github-script); removed `issues: write` permission. Slack step
preserved with full FAILURE_SUMMARY context.
- **starter_deployed_smoke.yml** — dropped `Create GitHub issue on
failure` step (github-script); removed `issues: write` permission. Slack
step preserved.
- **showcase_drift-detection.yml** — dropped `Create issue on failure`
(drift-detection job) and `Create drift report issue` (version-drift
job). Slack payload updated to link to workflow run URL (previously
linked to created issue). No `permissions:` block existed, so nothing to
remove.
## Why
Auto-created issues became noise — 13 stale bot-created issues
accumulated (all closed separately). Slack is the single source of truth
going forward.
## Test plan
- [ ] CI green
- [ ] First post-merge failure of any of these workflows emits a Slack
alert in #oss-alerts (no new GitHub issue)
## Summary
- Fixes matrix fan-out bug in Showcase Build & Deploy: single-service
workflow_dispatch was triggering all 37 matrix legs instead of 1
- Root cause: dorny/paths-filter@v3 on workflow_dispatch (no before SHA)
falls back to "all filters matched", producing CHANGES=[all 37
filter_keys]. The old OR-branch `(.filter_key as $fk | $changes |
index($fk) != null)` fired for every service regardless of $dispatch
- Also fixes regression: dispatch=all was routing through paths-filter
(returned empty on dispatch) → manual "deploy all" produced 0-entry
matrix
- New 3-mode filter:
- `dispatch == "all"` → include every service unconditionally (manual
"deploy all")
- `dispatch == <specific>` → narrow to that one service (drift-rebuild,
manual single-service)
- `dispatch == ""` → push event, use paths-filter against CHANGES
## Impact
- Drift rebuild was amplifying CI ~37× per dispatch (19 drifted services
× 37 matrix legs = 703 build jobs instead of 19)
- Manual single-service deploys via `gh workflow run` also fanning out
to full matrix
## Test plan
- [ ] CI green on this branch
- [ ] Post-merge drift-rebuild run produces 1-leg matrix per dispatch,
not 37
- [ ] Post-merge manual `service=all` dispatch deploys all services
- [ ] Post-merge push events continue to use paths-filter correctly
## Summary
- Swap Showcase Build & Deploy build matrix from GitHub-hosted
ubuntu-latest + buildx + GHA cache to Depot managed runners
(depot-ubuntu-24.04-4) + depot/build-push-action@v1
- Depot project `m2kw2wmmcp` (org `bnsr78t2s3`, Startup plan — unlimited
concurrency)
- Auth: OIDC via `id-token: write` permission; no DEPOT_TOKEN secret
needed; project configured to trust CopilotKit/CopilotKit on Depot side
- Drop now-dead `cache_scope` matrix field from all 37 service entries
(Depot provides its own project-scoped cache)
- Lightweight coordination jobs (detect-changes, check-lockfile, notify)
kept on stock ubuntu-latest
## Motivation
- CI queue saturation: CopilotKit's Team plan 60-concurrent-job cap gets
monopolized by drift-rebuild dispatches. Depot runners are outside that
cap (Startup plan = unlimited concurrency).
- Build speed: Depot's persistent remote Docker cache shares layers
across all 37 services and across runs — expected significant reduction
in 3m53s Docker build step.
- Reference: existing `e2e_dojo.yml` has already used
`depot-ubuntu-24.04` successfully; this expands coverage where it
matters most (Notion plan:
https://www.notion.so/3453aa38185281ea952acedda0d322b5).
## Test plan
- [ ] CI green on this branch
- [ ] Post-merge Showcase: Build & Deploy run completes all 37 matrix
legs successfully
- [ ] Measure shell service matrix-leg wall time pre vs post (target:
meaningful reduction from ~10min cap pressure)
- [ ] Verify no auth errors on first run (OIDC trust relationship in
place)
## Summary
- Yesterday's uniform `health_path: /api/health` assignment was wrong
for shell + shell-dojolike — those are Next.js frontends with no
/api/health route
- Verify step polled 404 for 24 attempts (~6min), burned the 10-min
matrix cap, failed Showcase: Build & Deploy on main
- Live-verified: showcase.copilotkit.ai/ → 200; /api/health → 404
- Fix: set health_path to / for those two services only. All other
services keep /api/health (they have it).
## Test plan
- [ ] CI green on this branch
- [ ] Post-merge Showcase: Build & Deploy on main completes shell matrix
leg under 10min with 200 health check
Internal job id was still 'starter-smoke:' and artifact pattern was
'starter-smoke-${{ matrix.starter }}'. Rename to 'smoke-starter' /
'smoke-starter-${{ matrix.starter }}' to match the new
test_<layer>-<target> / smoke-<layer> naming convention. Cosmetic only —
no other workflow references this job id or artifact name.
Asymmetric path filter — .changeset was listed in PR trigger but not push.
The dorny/paths-filter step already includes .changeset in the ts: filter,
so this matches the intent on both triggers.
Showcase-only and sdk-python-only PRs were triggering the full TS unit matrix
(3 Node versions x full monorepo). Neither directory affects TS unit tests.
Now paths-ignore matches the spirit of the existing 'examples/**' exclusion.
## Problem
All 18 stale-service rebuild triggers in [run
24534637524](https://github.com/CopilotKit/CopilotKit/actions/runs/24534637524)
failed with:
```
failed to run git: fatal: not a git repository (or any of the parent directories): .git
```
The image-drift workflow is completely broken — when it detects stale
showcase images, the follow-up `gh workflow run showcase_deploy.yml`
step 500-fails for every service, so no rebuilds actually happen.
## Root cause
The `smoke` job in `.github/workflows/showcase_smoke-monitor.yml` has
**no `actions/checkout@v4` step**. `gh workflow run` (unlike `gh api` or
`gh run list`) resolves the target repository by shelling out to `git`
in the current directory to read the remote. With no checkout present,
`git` aborts and the entire rebuild loop fails.
PR #3968 didn't introduce the missing-checkout problem — it was latent —
but it was the first run to reach the rebuild path with a real stale
list, exposing it.
## Fix
Pass `--repo "${{ github.repository }}"` to `gh workflow run` so the CLI
skips the git-based repo lookup entirely. Minimal, one-line change; no
checkout needed.
```diff
-ERR_OUTPUT=$(gh workflow run showcase_deploy.yml -f service="${SVC}" 2>&1) || RC=$?
+ERR_OUTPUT=$(gh workflow run showcase_deploy.yml --repo "${{ github.repository }}" -f service="${SVC}" 2>&1) || RC=$?
```
## Test plan
- [x] YAML parses
- [x] Local pre-commit (lint + test + check:packages) green
- [ ] Next scheduled smoke-monitor run (or manual `workflow_dispatch`
with a stale service) triggers rebuilds without the `not a git
repository` error
## Summary
The `showcase_docs-sync` workflow flagged two LangGraph interrupt docs
as having showcase-local modifications that blocked the automated
transform pipeline. This PR reconciles them manually.
Flagged files:
-
`docs/content/docs/integrations/langgraph/generative-ui/your-components/interrupt-based.mdx`
-
`docs/content/docs/integrations/langgraph/human-in-the-loop/interrupt-flow.mdx`
## Upstream changes pending sync to showcase/
- Add `useInterrupt` `agentId: "starterAgent"` to all four examples per
file (from #63a6a2733, [docs(langgraph): warn on useInterrupt agentId
and add starterAgent to
examples](https://github.com/CopilotKit/CopilotKit/commit/63a6a2733fce979f15b1a0bd74021d1a139ff2c7))
- Add `<Callout type="warn">` explaining that a mismatched `agentId`
silently never fires the interrupt (expanded in #90d1869ea,
[docs(langgraph): clarify useInterrupt agentId mismatch
consequence](https://github.com/CopilotKit/CopilotKit/commit/90d1869ead822b1e2ca45010e3ad1d63a9192b58))
- Update stale model name `gpt-5.2` -> `gpt-5.4` in code snippets (from
[fix(docs): update stale model names, fix broken quickstart
examples](https://github.com/CopilotKit/CopilotKit/commit/25f3f70ab))
Note: `interrupt-based.mdx` on main received its upstream updates in
both docs/ and showcase/. `interrupt-flow.mdx` was only updated in docs/
— the showcase copy was missed. Both are fixed here.
## Showcase-local divergence found in current files
- **Leftover import fragment at the top of each file** — bytes like
```
TailoredContent,
TailoredContentOption,
} from "@/components/react/tailored-content.tsx";
```
left behind by an earlier version of `stripMdxImports` that failed to
fully strip multi-line imports. The current version of the sync script
strips these cleanly. Garbage, not deliberate content.
- **Prettier-style reformatting** applied ad hoc — line wrapping in the
`<Callout type="info">` intro and full reformatting of the last
`useInterrupt` handler example (double-quotes, trailing commas,
different indentation). No prettier config / lint-staged hook enforces
this on these files; it appears to be IDE-on-save drift.
- **Stale `gpt-5.2`** — pre-dates the upstream bump to `gpt-5.4`.
- **`interrupt-flow.mdx` missed the agentId/Callout additions**
entirely.
## Reconciliation approach
Judgment call: none of the showcase-local changes represent intentional
additive content. They are (a) a stripping bug from an old sync script,
(b) formatter drift, and (c) pre-sync staleness. I rebuilt both showcase
files as a **clean transform** of the current upstream source (applying
`stripMdxImports`, `stripComponentsProps`, and
`replaceLangChainWithLangGraph` exactly as the sync script would),
instead of cherry-picking the upstream diff on top of the
prettier-reformatted showcase files.
Advantages:
- Matches the output the automated sync script will produce on the next
non-conflicting change, eliminating persistent drift.
- Removes the garbage import fragment.
- Incorporates every upstream change (agentId, Callout, model name) that
was trying to land.
- Both files now differ from upstream only by the script's standard
transforms (no local edits) so subsequent auto-syncs will pass cleanly.
Also bumps `showcase/shell/.docs-sync-sha` from `13272bab` to `41b6eb93`
(current main tip) so the next workflow run doesn't re-flag the same
files.
## Files changed
-
`showcase/shell/src/content/docs/integrations/langgraph/generative-ui/your-components/interrupt-based.mdx`
— clean re-transform of upstream; gains agentId (4x), Callout warn,
`gpt-5.4`; loses leftover import fragment, prettier reformatting
-
`showcase/shell/src/content/docs/integrations/langgraph/human-in-the-loop/interrupt-flow.mdx`
— same as above (this one was missing the agentId/Callout updates
entirely)
- `showcase/shell/.docs-sync-sha` — bumped to current main HEAD
## Test plan
- [x] Verified dry-run of `showcase/scripts/sync-docs-from-main.ts`
reports "Needs review: 0" after changes
- [x] Verified both files contain the 4 `agentId: "starterAgent"`
insertions and the `<Callout type="warn">`
- [x] Verified no leftover import fragment at top of either file
- [x] Ran local pre-commit hooks (lefthook: check-binaries, lint-fix,
test-and-check-packages) — all pass
- [x] commitlint validation passes
## Summary
The "files needing manual review" Slack warning in
`showcase_docs-sync.yml` listed flagged files but did not link the
auto-opened PR, forcing reviewers to hunt for it manually. This PR
captures the PR URL (already exposed as `steps.push.outputs.pr_url`) and
threads it into the alert payload so reviewers can click through
directly.
## Changes
- Review-needed alert now includes a `Review: <PR URL>` line
- Split the alert into two variants:
- PR opened (normal case): includes the PR link
- No PR opened (edge case where the clean-transform portion was empty):
posts review items without a link
- Auto-sync and merge-failed alerts already linked the PR — this brings
the review-needed alert to parity
## Test plan
- [ ] YAML validated locally (`python3 -c "import yaml;
yaml.safe_load(...)"`)
- [ ] Next docs sync that produces review items should include the PR
link in Slack
## Summary
Fixes two sources of noisy `❌ FAILED` Slack alerts on
`showcase_deploy.yml` that fired today (aimock run
[24529510571](https://github.com/CopilotKit/CopilotKit/actions/runs/24529510571),
mastra run
[24531762985](https://github.com/CopilotKit/CopilotKit/actions/runs/24531762985),
37-service run
[24531118773](https://github.com/CopilotKit/CopilotKit/actions/runs/24531118773)).
## Root causes (diagnosed from workflow logs)
**Run 24529510571 (aimock)** — Railway reported `status=SUCCESS` for 19
polls, but every poll logged `streak=0`. The `Verify deploy health` step
hardcodes `https://${DOMAIN}/api/health`. aimock is not a Next.js app
and does not expose `/api/health`; it exposes `/health` (confirmed:
`curl .../health` returns `{"status":"ok"}`). The step was eventually
cancelled mid-loop by concurrency group, and Slack posted `FAILED`
because `build=cancelled` falls into the else branch.
**Run 24531762985 (mastra)** — Entire run cancelled mid-flight by
concurrency group (a later push/dispatch superseded it). Not a
regression from PR #3970. Notify still posted `FAILED`.
**Run 24531118773 (37-service, `service=all` dispatch)** —
`google-adk`'s Railway deploy returned `status=FAILED` (not a workflow
bug; the deploy itself crashed on Railway for this specific attempt).
Service is currently healthy — the subsequent drift-detection wave
rebuilt and redeployed fine. No workflow change needed for this one.
## Fixes
1. **`/api/health` → fall back to `/health`.** The probe first tries
`/api/health` (Next.js convention) and falls back to `/health` if that
returns non-200. Tested locally against aimock, google-adk, ag2,
spring-ai — all resolve correctly.
2. **Quiet concurrency-group cancellations.** When any of
`detect-changes`, `check-lockfile`, or `build` has `result=cancelled`,
skip the Slack post entirely (a newer run is handling the same services;
the cancellation is expected, not a failure).
## Test plan
- [ ] CI passes on this PR
- [ ] Next `showcase_deploy.yml` run with aimock converges on `/health`
→ posts `✅` instead of `❌`
- [ ] A concurrent push-cancel-push sequence posts a single success
message (no `❌ FAILED` from the cancelled run)
Addresses three CR findings on PR #3980 — rebuilt from origin/main to
drop stale test-integration-tmp plumbing that was reintroduced in the
prior iteration.
Finding 1 — Scoped cancel-skip for concurrency group cancellations:
- Pre-build cancellation (detect-changes/check-lockfile cancelled):
stay silent, newer run will redo all work from scratch.
- Mid-matrix build cancellation: post a distinct muted info message
so humans can spot anomalies (fail-fast:false matrix legs rolling
up to `cancelled` when some legs silently passed via warn-not-fail
health-probe timeout).
Finding 2 — Per-service `health_path` with no silent fallback:
- Every service in ALL_SERVICES carries an explicit `health_path`
field (`/api/health` for Next.js, `/health` for aimock).
- Verify step fails loud with `::error::` + exit 1 when health_path
is unset, rather than silently defaulting to /api/health. An
unscoped fallback could mask a broken endpoint when an unrelated
catch-all / CDN / actuator happens to 200 at a different path.
Finding 3 — Comment accuracy:
- health_path comment now cites the policy ("no fallback — misconfig
is a config bug, not runtime behavior to hide").
- Cancel-skip comment enumerates all three cancellation causes
(concurrency supersede, manual cancel, upstream failure cascade)
and explains the pre-build vs mid-build split.
The smoke-monitor job has no actions/checkout step, so `gh workflow
run` cannot resolve the repo from a git remote and fails with:
failed to run git: fatal: not a git repository
Pass --repo ${{ github.repository }} so gh CLI skips the git lookup.
All 18 rebuild triggers in run 24534637524 failed for this reason.
Fixes 3 HIGH findings from R1 review on #3988:
1. pr_url empty was used as a proxy for "no PR opened because clean-transform
was empty", but it's also empty on every error path (bot-token failure, gh
pr create failure, push failure). Replace with an explicit pr_opened=true/
false output from the push step — true only after the PR URL is captured,
false only on the deliberate CHANGED=0 path. Error paths leave it unset so
alerts fall through to the failure() handler.
2. review_items_json was string-interpolated raw into a JSON payload inside
triple-backticks. Any filename containing ", \\, or a control character
would break the payload. Moved to a jq-based payload-file-path pattern:
a dedicated Build Slack payloads step writes each payload to disk with jq
--arg, so all values are safely JSON-escaped regardless of content. Slack
steps consume the tmpfiles via payload-file-path.
3. If a notify-* step itself fails (webhook 5xx, rate limit, malformed JSON),
the review-needed alert was silently lost — the existing failure() alert
was gated on pr_url == '' and would not fire. Added an unconditional
fallback step that posts a plain-text "alert machinery failed" message via
curl when any notify-* step's outcome is failure, so we never lose a
review-needed or failure notification.
The docs-sync workflow flagged these two files as having showcase-local
modifications that blocked the automated transform pipeline:
- docs/content/docs/integrations/langgraph/generative-ui/your-components/interrupt-based.mdx
- docs/content/docs/integrations/langgraph/human-in-the-loop/interrupt-flow.mdx
Upstream docs/ changes pending sync to showcase/:
- Add useInterrupt `agentId: "starterAgent"` to all four examples per file
- Add <Callout type="warn"> explaining that a mismatched agentId silently
never fires the interrupt
- Update stale model name gpt-5.2 -> gpt-5.4 in code snippets
Showcase-local divergence present before this PR:
- Leftover import fragment at the top of each file (bytes from an
incomplete multi-line import strip by an older version of the sync
script, not deliberate content)
- Prettier-style reformatting applied ad hoc (line wrapping in the
Callout info block and the last useInterrupt handler example)
- Stale model name gpt-5.2 (pre-dates the upstream bump to gpt-5.4)
- interrupt-flow.mdx never received the agentId / Callout additions
that interrupt-based.mdx received in 63a6a2733 and 90d1869ea
Resolution: rebuild both showcase files as a clean transform of the
current upstream source. None of the showcase-local changes were
intentional additive content — they were stripping bugs, formatter
drift, and pre-sync staleness — so dropping them and taking the full
upstream reconciles both files cleanly.
Also bumps showcase/shell/.docs-sync-sha to the current main tip so the
next docs-sync run does not re-flag the same files.
The "files needing manual review" Slack warning listed files but
didn't link the auto-opened PR, forcing reviewers to hunt for it
in GitHub. Capture the PR URL from the create/merge step output
(already exposed as steps.push.outputs.pr_url) and include it as
a "Review:" line in the payload.
Split the alert into two variants:
- PR opened (normal case): includes the PR link
- No PR opened (edge case where clean-transform portion was empty):
posts review items without a link
Auto-sync and merge-failed alerts already linked the PR — this
brings the review-needed alert to parity.
## Summary
Adds 5 missing QA markdown files (gen-ui-agent,
shared-state-{read,write,streaming}, subagents) to bring
`showcase/packages/ag2/qa/` from 4 files to 9, matching
`langgraph-python/qa/`.
Part of the QA-parity blitz (Bundle 0 in the action inventory).
### Detail
- `gen-ui-agent.md` and `shared-state-read.md` describe the full
implementations — selectors (`task-progress`, `task-step-text`,
`recipe-card`, `add-ingredient-button`, `improve-button`), labels (`AI
Recipe Assistant`), and suggestion button names match the ag2 page
sources verbatim (they mirror LG-py's implementations word-for-word).
- `shared-state-write.md`, `shared-state-streaming.md`, and
`subagents.md` follow the "Note: Stub Demo" pattern from LG-py — the ag2
pages for these demos are stubs identical to LG-py's stubs, so the QA
coverage is identical.
## Test plan
- [ ] Verify `showcase/packages/ag2/qa/` now contains 9 files matching
the file set in `showcase/packages/langgraph-python/qa/`
- [ ] Spot-check selectors in `gen-ui-agent.md` and
`shared-state-read.md` against the ag2 page sources
## Summary
Follow-up hardening to #3971. aimock supports fixture schema validation
at startup via `--validate-on-load`, but the flag is **opt-in** and the
showcase Dockerfile was not passing it. That meant fixtures with
unrecognized response keys (e.g. `"text"` instead of `"content"`) loaded
silently and only surfaced as HTTP 500s at request time — which is
exactly what crashed crewai-crews and triggered #3971.
This PR wires up two independent safety nets so a broken fixture can't
ship again:
1. **Dockerfile (fail-fast at container boot)** —
`showcase/aimock/Dockerfile` now passes `--validate-on-load`. If any
fixture fails the aimock schema, the container exits non-zero instead of
starting and serving 500s. Railway will not promote a bad build.
2. **CI test (fail-fast in PR review)** — new vitest spec at
`showcase/scripts/__tests__/aimock-fixtures.test.ts` imports
`loadFixtureFile` + `validateFixtures` from `@copilotkit/aimock` and
asserts zero errors against both `feature-parity.json` and `smoke.json`.
Runs inside the existing ` Showcase: Validate` workflow
(`showcase/scripts` vitest suite) on every PR that touches
`showcase/**`.
## Verification
**Red-green on the vitest spec:**
- Rebased onto the tip of main *before* #3971 merged: the spec fails
with 5 errors — exactly the 5 broken `"text"` fixtures (`plan`, `steps`,
`mars`, `dashboard`, `report`) that #3971 repaired.
- Rebased forward onto main *after* #3971: spec passes with 0 errors,
all 549 showcase/scripts tests green.
**Red-green on the Dockerfile:**
- Current fixtures + `--validate-on-load`: container boots cleanly, logs
`Loaded 39 fixture(s) from /fixtures`.
- Injecting an intentionally broken fixture (`response: { "text": "..."
}`): container fails to start with `[aimock] Fixture 0: response is not
a recognized type (must have content, toolCalls, error, or embedding)` /
`Validation failed: 1 error(s), 0 warning(s)` and non-zero exit.
## Test plan
- [x] Local: full `showcase/scripts` vitest suite passes (549/549)
- [x] Local: `pnpm run test` (monorepo) passes
- [x] Docker: image builds and starts with `--validate-on-load` against
current fixtures
- [x] Docker red-green: broken fixture fails container start with
non-zero exit
- [ ] CI: ` Showcase: Validate` job runs the new test file on PR
## Summary
Updates LangGraph human-in-the-loop interrupt documentation so
`useInterrupt`
examples set `agentId: "starterAgent"` and readers see a warning that
`agentId`
must match a runtime-registered agent (default assumption is
`"default"`).
## Changes
-
**`docs/content/docs/integrations/langgraph/human-in-the-loop/interrupt-flow.mdx`**
— warning callout; `agentId` on basic, conditional (`enabled`), and
`handler` snippets.
-
**`docs/content/docs/integrations/langgraph/generative-ui/your-components/interrupt-based.mdx`**
— same.
-
**`showcase/shell/src/content/docs/integrations/langgraph/generative-ui/your-components/interrupt-based.mdx`**
— same content alignment; adds the `useInterrupt` import line in the
primary code block for consistency with main docs.
## Notes
- Production apps should replace `"starterAgent"` with the id their
runtime actually registers.
- If you also want showcase `human-in-the-loop/interrupt-flow.mdx`, hook
reference docs, or a longer callout (e.g. `/info`, example error text),
those are not part of this staged set—add them in a follow-up or stage
them before merge.
## Test plan
- [ ] Open the three MDX pages in the docs/showcase preview and confirm
the callout and snippets render as expected.
Brings showcase/packages/ag2/qa/ from 4 files to 9, matching langgraph-python.
Adds: gen-ui-agent, shared-state-read, shared-state-write, shared-state-streaming, subagents.
Content is adapted from the LG-py templates with titles changed to AG2. The
three stub demos (shared-state-write, shared-state-streaming, subagents) keep
the LG-py 'Note: Stub Demo' pattern since the ag2 implementations are stubs
identical in shape to LG-py's. gen-ui-agent and shared-state-read describe
the full implementations (which mirror LG-py's selectors and labels
verbatim).
## Summary
CrewAI's `ChatWithCrewFlow.__init__` (invoked from
`ag_ui_crewai.endpoint.add_crewai_crew_fastapi_endpoint` at module
import in `ag-ui-crewai <= 0.1.5`) makes synchronous blocking LLM calls
via `crewai.cli.crew_chat.generate_input_description_with_ai` and
`generate_crew_description_with_ai`. ANY LLM hiccup — aimock regression,
OpenAI outage, network blip, DNS failure — crashes the Python process
BEFORE uvicorn can bind its port, causing Railway/Kubernetes health
checks to fail and deploys to roll back.
This was the direct cause of the crewai-crews Railway crash fixed
server-side in #3971. That fix patched the aimock response schema, but
the underlying fragility in upstream CrewAI / ag-ui-crewai remained — a
future blip would crash us again.
This PR adds a defensive monkey-patch in `agent_server.py` that replaces
both generator functions with static-string returns BEFORE
`ag_ui_crewai` is imported. The AI-generated descriptions are only
surfaced in the CrewAI chat UI (which the CopilotKit runtime does not
use), so static defaults are functionally equivalent for our showcase.
Upstream issue filed: https://github.com/crewAIInc/crewAI/issues/5510
The long-term fix is deferred construction in `ag-ui-crewai`, which has
landed on ag-ui `main` but is not yet released. Remove this shim once
`ag-ui-crewai > 0.1.5` ships.
## Why a monkey-patch and not lazy-init
`add_crewai_crew_fastapi_endpoint` is the entry point and internally
constructs `ChatWithCrewFlow(crew)` synchronously in `ag-ui-crewai <=
0.1.5`. Deferring that call would require either vendoring the endpoint
function or reimplementing it. The monkey-patch is two lines and removes
cleanly when the upstream fix ships.
## Test plan
Verified locally via Docker build + run with an intentionally broken LLM
endpoint (`OPENAI_BASE_URL=http://invalid-host/v1`):
**Unhardened (negative control):**
```
File "/app/agent_server.py", line 27, in <module>
add_crewai_crew_fastapi_endpoint(app, LatestAiDevelopment(), "/")
File ".../ag_ui_crewai/endpoint.py", line 250, in add_crewai_crew_fastapi_endpoint
add_crewai_flow_fastapi_endpoint(app, ChatWithCrewFlow(crew=crew), path)
File ".../ag_ui_crewai/crews.py", line 56, in __init__
self.crew_chat_inputs = crew_chat_generate_crew_chat_inputs(...)
File ".../crewai/cli/crew_chat.py", line 387, in generate_crew_chat_inputs
description = generate_input_description_with_ai(input_name, crew, chat_llm)
File ".../crewai/cli/crew_chat.py", line 481, in generate_input_description_with_ai
response = chat_llm.call(...)
APIError
```
Container exits with code 1, never binds a port.
**Hardened (this PR):**
```
INFO: Started server process [7]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
```
`curl http://localhost:PORT/api/health` ->
`{"status":"ok","integration":"crewai-crews","agent":"ok","timestamp":"..."}`
(HTTP 200).
### Checklist
- [x] Docker build succeeds locally
- [x] Unhardened build crashes on import with broken LLM endpoint
(negative control)
- [x] Hardened build starts cleanly with broken LLM endpoint and
responds 200 on `/api/health`
- [x] Upstream issue filed on crewAIInc/crewAI
## Summary
Fixes Slack alerts that rendered literal `\n` (backslash + n) instead of
actual newlines — messages looked like `*Starter Deployed Smoke Test
Failed*\nView run` in Slack.
Root cause: `jq -n --arg text "...\n..."` passes the two literal
characters `\` and `n` to jq (bash doesn't interpret `\n` inside double
quotes). `--arg` stores them verbatim; jq then JSON-escapes the
backslash, producing `"\\n"` in the payload, which Slack parses back to
the two-character string `\n` and renders as-is.
## Changes
Switched three alert builders from `jq --arg` with embedded `\n` to the
safer pattern already used in `showcase_smoke-monitor.yml`: write a
message file with real LF bytes via `printf`, then `jq -n --rawfile text
…` for correct JSON escaping.
Affected workflows:
- `starter_deployed_smoke.yml` — Starter Deployed Smoke Test Failed (the
alert from the screenshot)
- `starter-smoke.yml` — Starter smoke test failing: <starter>
- `showcase_drift-detection.yml` — Showcase E2E suite failed
## Enrichment (starter_deployed_smoke.yml)
The deployed-smoke failure alert was just `*Starter Deployed Smoke Test
Failed* | View run`. Now emits a Playwright JSON report, extracts
failures, and builds a richer payload:
- Failed starter slugs listed in the header (parsed from spec titles)
- Direct link to the failed job in addition to the workflow run
- Up to 5 failure entries, each with:
- starter slug
- test-level tags (`@starter-health` / `@starter-agent` /
`@starter-chat` / `@starter-tools`)
- first line of the error message (ANSI-stripped, 240-char cap)
- "…and N more failure(s)" footer when the count exceeds 5
Falls back to the minimal header when no JSON report exists (e.g.
pre-test setup failed) so alerts still fire.
The other two alerts already had a summary but now also include a "View
job" link for direct navigation.
## Test plan
- [ ] Trigger `starter_deployed_smoke.yml` via `workflow_dispatch`
against a starter known to fail (or simulate) and confirm Slack renders
real newlines plus the enriched payload
- [ ] Trigger `starter-smoke.yml` via `workflow_dispatch` (PR run skips
the Slack step) and confirm alerting format when a starter is forced to
fail
- [ ] Trigger `showcase_drift-detection.yml` via `workflow_dispatch` and
confirm Slack renders real newlines and the fenced code block
- [ ] Grep `.github/workflows/` for `jq -n --arg text ".*\\n"` — should
return zero matches
Supersedes #3912 (closed).
## Summary
- The Mastra showcase package registers only `weatherAgent` in its
Mastra config, but the demo pages request names like `agentic_chat`,
`human_in_the_loop`, `tool-rendering`, etc.
`MastraAgent.getLocalAgents({ mastra })` returns agents keyed by
registration name, so the runtime cannot resolve the demo-requested
names. This was causing the L3 mastra chat test to fail.
- Mirror the pattern used in
`showcase/packages/crewai-crews/src/app/api/copilotkit/route.ts`: wrap
`MastraAgent.getLocalAgents` and expose the underlying `weatherAgent`
under every name the demo pages ask for. `weatherAgent` itself is
preserved so the internal `/api/smoke` route (which calls `agentId:
"weatherAgent"`) keeps working.
## Agent names now exposed on the backend
- `weatherAgent` (original Mastra registration — preserved for smoke
test)
- `agentic_chat`
- `human_in_the_loop`
- `tool-rendering`
- `gen-ui-tool-based`
- `gen-ui-agent`
- `shared-state-read`
- `shared-state-write`
- `shared-state-streaming`
- `subagents`
The starter (`showcase/starters/mastra`) was checked and is not affected
— its route uses `LangGraphAgent` and already registers `sample_agent`,
which matches what the starter's page requests.
## Test plan
- [ ] CI L3 mastra chat test passes (the failure this PR targets)
- [ ] `/api/smoke` route still returns 200 (uses `weatherAgent` which is
preserved)
- [ ] Demo pages (`/demos/agentic-chat`, `/demos/hitl`,
`/demos/tool-rendering`, etc.) can reach the runtime without 404
## Summary
The showcase smoke monitor's image-drift alert in `#oss-alerts`
currently lists every stale image on every run — a ~19-line message for
the common, successful case where all rebuilds triggered cleanly. This
PR trims the success case to a one-line summary and reserves the
detailed list for when rebuilds actually fail.
**Before (every run, ~20 lines):**
```
📦 Image drift detected — rebuilds triggered:
⚠️ shell — image stale
⚠️ langgraph-python — image stale
... (17 more)
```
**After — success (common case):**
```
📦 Image drift detected — 19 rebuilds triggered (<run>)
```
**After — failure (only the ones that failed, with reasons):**
```
📦 Image drift detected — 19 rebuilds triggered, some failed:
❌ langgraph-python — HTTP 422: workflow not found
<Workflow run>
```
## Changes
- `Trigger rebuild for stale services` step now captures stderr from
each `gh workflow run` call and writes per-service failure lines (with
the actual error) to `rebuild-failures.txt`
- Step exposes two outputs: `triggered_count` and `has_failures`
- `Alert image drift to Slack` step branches on `has_failures`: summary
line on success, detailed failure block on failure (with fallback if the
rebuild step didn't run)
- No change to the rebuild-triggering logic itself — only the Slack
payload
## Test plan
- [x] `python3 yaml.safe_load` and `yamllint --relaxed` pass on the
modified workflow
- [ ] Manually dispatch `showcase_smoke-monitor.yml` after a deploy
where drift is expected → verify single-line success message
- [ ] Force a rebuild failure (e.g. temporarily remove
`showcase_deploy.yml` workflow_dispatch permissions) → verify detailed
failure message lists only the failed service with the gh error
Addresses CR R1 CRITICAL finding on PR #3974.
setattr() on a Python module always succeeds regardless of prior
attribute existence. Without a guard, an upstream rename of
generate_input_description_with_ai or generate_crew_description_with_ai
in a future crewai release would silently no-op the patch, leaving the
real functions in place. The pre-bind LLM crash bug would quietly
reappear in production with a green PR.
Changes:
- Add hasattr() guard in both agent_server.py files that raises
RuntimeError with an actionable drift message if either symbol
disappears upstream.
- Add post-assignment assert to defend against import-order weirdness
or module re-imports shadowing the reference.
- Add an info-level log line so operators can see the shim is active
and know to remove it after adoption.
- Add upstream issue link (crewAIInc/crewAI#5510) and explicit ag-ui-crewai
release status to the comment block.
- Pin ag-ui-crewai upper bound to <0.1.6 in both requirements.txt files
so the shim's applicability window is enforced by pip — upgrading past
0.1.5 forces the engineer to confront the version mismatch and remove
the shim.
Applied to both showcase/packages/crewai-crews/src/agent_server.py and
showcase/starters/crewai-crews/agent_server.py to keep the demo and
starter trees in sync.
Verified locally:
- Docker build succeeds (showcase/packages/crewai-crews).
- Hardened container starts cleanly with OPENAI_BASE_URL set to an
unreachable host; /health returns 200 on both agent (8000) and
Next.js (10000) ports.
- Negative case: deleting generate_input_description_with_ai from the
installed crewai module inside the container and re-importing
agent_server raises RuntimeError with the upstream-drift message, as
expected.
- Add --validate-on-load to all aimock invocations (4 workflows/scripts
+ 13 integration docker-compose files)
- Replace hardcoded 2-file fixture list with dynamic discovery across
showcase/, examples/integrations/*/, scripts/doc-tests/ (16 fixtures)
- Add sanity check to prevent silent zero-test pass when discovery fails
- Extend showcase_validate.yml path filter to trigger on
examples/integrations/**/fixtures/** and scripts/doc-tests/fixtures/**
- Import and use ValidationResult type for callback parameters
- Fix scripts/doc-tests/fixtures/default.json to use { fixtures: [...] }
envelope shape
Alem Tuzlak