- Promote SLACK_WEBHOOK_OSS_ALERTS to job-level env so the step-level
if: gate can actually see it. Step-level env: is not visible to the
same step's if:, which silently disabled every Slack post.
- Drop the service suffix from the concurrency group so two manual
dispatches for different services don't serialize unexpectedly and
push-triggered runs cancel each other deterministically.
- Rewrite the shared-module copy skip guard as an allowlist scoped to
showcase/packages/*. Starters, shell-family, aimock, and shell-dojo
don't consume shared modules, so the previous denylist missed every
new self-contained context (context=".") and polluted the repo root.
- Extend the binary-extension regex to cover .class, .jar, .pyd,
.pyc, .node, .bin, .pdb, .zip, .whl — any of which a build tool
can drop into a PR and none of which the previous regex caught.
- Add set -euo pipefail at the top of the run block. Previously
a transient wc failure (file deleted mid-diff) produced an empty
SIZE, which then failed [ "$SIZE" -gt ... ] with "integer
expression expected" on stderr while the script marched on —
silent false-green possible. Now errors fail the job loudly.
- Guard the wc -c call with an explicit numeric-regex check on
SIZE. A non-numeric or blank result now emits a ::warning:: and
skips the file rather than falling into the arithmetic
comparison and crashing under set -e (finding #17, #18).
- Add a comment to the prior-deploy Railway GraphQL query explaining
that deployments(first: 1) relies on the API default of
createdAt DESC ordering. Railway has no explicit orderBy argument
on this field; documenting the assumption means a future schema
change (or an operator reading the file cold) has something to
point at before debugging stale-deployment guard regressions
(finding #15).
- Harden both truncate_csv shell helpers (one per run block): disable
pathname globbing via set -f for the iteration so a slug
containing a glob char (*, ?, [) cannot expand against the
filesystem, and lock IFS to whitespace for deterministic
word-splitting regardless of caller env. Restore the prior glob
state on the way out so other shell blocks see no side effect
(finding #16).
## Summary
- Merges #4085 (Atai's no-docker-restructure feature port — merge-ready
variant) as the primary base.
- Renames `showcase/shell-internal/` → `showcase/shell-dashboard/` +
provisions Railway service at `dashboard.showcase.copilotkit.ai`.
- Renames `showcase/shell-dojolike/` → `showcase/shell-dojo/` + renames
Railway service in place (domain unchanged).
- Extracts MDX-docs infrastructure (`/docs`, `/[framework]`, `/ag-ui`,
`/reference` + components, libs, content) from `showcase/shell/` into
new standalone `showcase/shell-docs/` package; provisions Railway
service at `docs.showcase.copilotkit.ai`; adds 301 redirects on shell
for every moved route (including 17 per-framework slugs enumerated from
`registry.json`).
- Fixes `starter-langroid` `/api/health` 503 (sed stdout wrappers +
readiness gate); adds red-green regression guards in
`starter-consistency.test.ts`.
## DNS state
- `dashboard.showcase.copilotkit.ai` CNAME → `6u9icvje.up.railway.app` —
✅ live in Route53.
- `docs.showcase.copilotkit.ai` CNAME → `t6ge0qwv.up.railway.app` —
pending Route53 add.
## Railway services
- `showcase-shell-dashboard` (ID `4d5dfd74-be61-40b2-8564-b53b7dd4c15b`)
— provisioned, image pending first deploy.
- `showcase-shell-docs` (ID `7badfb8d-4228-414c-9145-b4026803714f`) —
provisioned, image pending first deploy.
- `showcase-shell-dojo` (ID `7ad1ece7-2228-49cd-8a78-bddf30322907`) —
renamed from shell-dojolike; already live.
## Closes / supersedes
- Supersedes #4084 (docker-restructure variant; preserved on
`jpr5/ftg-shell-ops` for future consideration, not merged here).
- True-merges #4085 via `--no-ff` — that PR auto-closes on this PR's
merge.
## Known non-blockers
- `/ag-ui/introduction` throws a server-component `onMouseEnter` runtime
error on `next start`. Same issue exists in shell today — pre-existing,
not a regression. Follow-up needed.
- `showcase/shell-dojo` lacks `package-lock.json` and `.gitignore`;
trivial follow-up for parity with siblings.
- `search-modal` retains the flat `search-index.json` shape (not the
extensible `{version:1, sources:[...]}` shape). Generator is the single
point of change if multi-shell search is wanted later.
- oxfmt has a pre-existing oscillation bug on 14 `.mdx` files with
`<Steps>` structures; sidestepped by mirroring the existing
`showcase/shell/src/content/**` ignore policy to the new
`showcase/shell-docs/src/content/**` path.
## Test plan
- [ ] CI green end-to-end (Validate Showcase, unit tests on 20/22/24,
format, oxlint, commitlint, starter-smoke).
- [ ] \`showcase_deploy.yml\` builds shell-dashboard + shell-docs images
to GHCR and Railway pulls them successfully.
- [ ] Add \`docs.showcase\` CNAME in Route53
(\`t6ge0qwv.up.railway.app\`) — without this,
docs.showcase.copilotkit.ai won't resolve and Railway won't provision
its TLS cert.
- [ ] Verify \`dashboard.showcase.copilotkit.ai/\` returns 200.
- [ ] Verify \`docs.showcase.copilotkit.ai/\` returns 200 and
\`/langgraph-python/quickstart\` renders.
- [ ] Verify \`showcase.copilotkit.ai/\` returns grid (200);
\`/docs/foo\` 301s to \`docs.showcase.copilotkit.ai/foo\`;
\`/react/foo\` 301s; \`/integrations\` and \`/matrix\` do NOT redirect.
- [ ] \`starter-langroid\` redeploy via \`showcase-deploy\` workflow:
\`/api/health\` returns 200.
The weekly Showcase pin-drift Slack alert previously read
"📉 Showcase pin-drift (weekly): FAIL=N
(baseline N) [ok]". The token "FAIL=N" read as a regression even when N
matched the ratchet baseline exactly (a stable week), generating recurring
false-alarm fatigue on #oss-alerts.
Reframe the notification text into three explicit states:
- [stable] — DRIFT=N matches baseline AND hash matches
- [REGRESSION] — DRIFT=N grew OR hash mismatch (set drifted); keeps the
🚨 emoji + run URL for actionable alerts
- [IMPROVED, ratchet me] — DRIFT=N shrank; reminder to update the baseline
Presentation-only. The ratchet logic in validate-pins.ts, the baseline file
(showcase/scripts/fail-baseline.json), and the FAIL-line hashing are
unchanged.
The main-deploy step already uses '// empty' on PRIOR_DEPLOY_ID but
the verify step extracted DEPLOY_ID / STATUS / DOMAIN without it, so
when Railway's edges array was transiently empty these became the
literal string 'null'. That breaks the prior-vs-fresh comparison and
the DOMAIN emptiness guard in inconsistent ways. Add '// empty' to
all three jq filters so empty-edge responses normalize to '' the
same way everywhere.
Two small but load-bearing hygiene fixes in the notify job's shell:
1) The matrix-legs collection pipeline used `jq -r '...' 2>/dev/null`,
swallowing any jq parse or schema errors before the surrounding
pipeline substitution could see them. If the gh-api payload shape
ever shifts (schema rev, partial response, encoding glitch), we
now want the error in the job log so an operator knows WHY notify
fell back to softer wording — the existing `-z "$pairs"` guard
still handles the empty-output path without relying on suppression.
2) The duplicated inline `truncate_csv` helper was declaring
`local ws=...` on its OWN line AFTER `local budget=...` on the
function's first line. Redeclaring `local` inside the same
function body is legal but prints 'local: not in a function'
warnings on some bash versions (3.x, noexec edge cases) and is
outright forbidden by `set -eu` hardening profiles. Collapse `ws`
into the single `local` declaration at the top and use a plain
assignment where it's computed.
The BUILD outcome classifier read:
elif [ "$BUILD" = "failure" ] || [ -n "$BUILD" ]; then
After the earlier `success` + `skipped && detect=success` branches
had claimed the obvious green cases, any non-empty BUILD string fell
through the `-n` test and was tagged FAILURE — including values like
"skipped" that reach this branch when detect-changes did NOT emit
`success` (e.g. cancelled pre-stage races or a partial matrix skip).
That means the Slack bot would occasionally cry red at runs that never
failed.
Simplify to an exact string match on "failure"; the final `else`
branch already handles the empty/unknown case by emitting OUTCOME=skip,
which is the correct disposition for indeterminate state.
The rest of the showcase workflows already pin `actions/checkout@v4`.
This one lagged on v3, which is now long-deprecated (Node 16 runner
EOL). Bringing it in line with the fleet avoids a future forced bump
when GitHub retires v3 runners entirely.
Remove every trace of the defunct test-integration-tmp service from the
showcase deploy workflow:
- drop the `test-integration-tmp` entry from the workflow_dispatch
inputs.service dropdown,
- drop the detect-changes `test_integration_tmp` output + paths-filter
entry, and
- delete the entire build-test_integration_tmp job (including its
placeholder RAILWAY_SERVICE_ID literal).
Matches the jpr5/test_integration_tmp_leak cleanup — the test harness
leak has been patched at its source (create-integration.test.ts), but
these workflow vestiges were never part of that fix and only survived
because nothing read them after the regression guard landed.
Adds the shell-docs package to the deploy matrix, paths-filter, and
dispatch options. Railway service `showcase-shell-docs` (ID
7badfb8d-4228-414c-9145-b4026803714f) has been provisioned with the
custom domain `docs.showcase.copilotkit.ai` and the Railway-generated
CNAME target `t6ge0qwv.up.railway.app`.
Env vars seeded on the Railway service: OPENAI_API_KEY,
ANTHROPIC_API_KEY, GOOGLE_API_KEY, POSTHOG_PROJECT_KEY, PORT=10000,
NEXT_PUBLIC_BASE_URL=https://docs.showcase.copilotkit.ai,
NEXT_PUBLIC_SHELL_URL=https://showcase.copilotkit.ai.
Matrix entry uses lfs:true + context:"." + explicit Dockerfile path
(mirrors shell) because the image pulls in shared/ scripts/ packages/
at build time. build_args_sha/branch are included so the build surface
embeds commit metadata, matching shell's pattern.
- Renamed showcase/shell-internal/ → showcase/shell-dashboard/ (git mv for history).
- Updated package name to @copilotkit/showcase-shell-dashboard (+ lockfile).
- Updated external references: showcase/scripts/{probe-docs,generate-status}.ts comments and showcase/README.md.
- Added dispatch + filter + ALL_SERVICES entry for shell-dashboard in .github/workflows/showcase_deploy.yml.
- Provisioned Railway service showcase-shell-dashboard (id 4d5dfd74-be61-40b2-8564-b53b7dd4c15b):
- image source ghcr.io/copilotkit/showcase-shell-dashboard:latest
- env vars OPENAI_API_KEY, ANTHROPIC_API_KEY, PORT copied from showcase-shell
- custom domain dashboard.showcase.copilotkit.ai attached (CNAME → 6u9icvje.up.railway.app)
- image will be built and pushed via showcase_deploy.yml on merge
- Add top-level 'permissions: contents: read' — the workflow performs
no repo writes, so declaring the minimum explicitly closes a
default-token hardening gap.
- Serialize publishes with concurrency: group: vscode-extension-publish,
cancel-in-progress: false. Two rapid pushes to main used to race
straight into vsce publish; the second failed noisily on duplicate
version or, worse, published out of order.
- Remove the 'Lint' step. The Nx target 'copilotkit-vscode-extension:lint'
doesn't exist and there's no 'lint' npm script either, so the step
always failed — 'continue-on-error: true' just hid it while never
actually linting anything. Re-add once a real lint target lands.
check-binaries.sh:
- Replace '[ ] && exit 1 / exit 0' with an explicit 'if [ ]; then
exit 1; fi; exit 0' block. Under 'set -e' the old form was safe
only because the trailing 'exit 0' existed; the explicit form is
robust regardless of what follows.
Critical
- scripts/hooks/check-binaries.sh: restore showcase data-file exclusions
(demo-content / search-index / starter-content >1 MB) that the inline
refactor dropped; add 'set -eu' so silent shell failures don't hide
policy violations.
- packages/vscode-extension/src/extension/preview-panel.ts: drop 'blob:'
from CSP script-src — it lets arbitrary-string JS execute via Blob URL
and defeats most of CSP's XSS protection.
- packages/vscode-extension/package.json: set private: true. The extension
ships as a .vsix via vsce, not npm, and workspace:* devDependencies
would break an accidental 'npm publish'.
Important
- runtime: forward the real agentId from handleConnectAgent into
handleSseConnect / createSseEventResponse so DebugEventBus envelopes
on /connect carry the actual agent name instead of the literal
'connect'. Updates handle-connect.ts, sse/connect.ts.
- hooks/panel.ts CSP: narrow connect-src from 'https:' to just the
Tailwind CDN. The preview path never drives a real CopilotKit runtime
— all hook calls route through the stub — so there's no legitimate
https: fetch to allow from inside bundled user code.
- extension/utils.ts getNonce(): switch to crypto.randomBytes. Math.random()
is not acceptable for a value that gates inline-script execution.
- .github/workflows/vscode-extension.yml:
* Build step uses 'nx run copilotkit-vscode-extension:build' instead
of 'pnpm run build' (targeted build with Nx caching, not full
monorepo rebuild).
* Added explicit 'Type check' step (tsc --noEmit).
* Added Lint step gated with continue-on-error until the Nx target
exists, so a missing target doesn't break the pipeline.
* Publish job now queries the Marketplace for the current published
version and skips 'vsce publish' when the local package.json
version matches — stops every docs/CI-only push to main from
failing on duplicate-version errors.
- hooks/hook-scanner.ts: bound the synchronous walk at 20 000 files so a
pathologically large workspace can't freeze the extension host; flag
kept in module-level constant with a rationale comment.
- sse-response.ts: document that debugEventBus.broadcast intentionally
runs before the stream-closed gate so debug subscribers see trailing
events even after the SSE client disconnects.
- inspector-panel.ts: subscribe to DebugStream lazily on show() rather
than in the constructor; unsubscribe on panel dispose. Avoids firing
the event callback on every envelope when no panel is open.
Suggestions
- fetch-router.ts: document 'debug-events' as a reserved route so it
can't be shadowed by an agent literally named 'debug-events'.
- activate.ts findValuePosition: add optional startOffset parameter,
document the first-occurrence limitation + the follow-up path for
per-fixture precision.
- activate-hook-explorer.ts isInsideWorkspace: fix JSDoc to reflect the
code (root itself is excluded).
I added it to the matrix in c3d105ae0 but the corresponding
examples/integrations/test-integration-tmp/ directory was never
created on this branch — the job fails on every PR run with
"No such file or directory".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Teams get pinged on every failure but never see a positive recovery
signal when these workflows flip back to passing. Mirror the per-service
transition pattern already implemented in showcase_smoke-monitor.yml.
showcase_deploy.yml (notify job):
- Restore previous run's status from actions/cache (key pattern
matches smoke-monitor: prefix-based restore-keys + run_id save key)
- Classify outcome as failure | success | skip
- failure → post existing red alert + update state
- success AND prev=failure → post recovery + clear state
- success AND prev=ok → silent (green→green, matches prior behavior)
- skip (no-changes or cancellation) → no state change, no post
starter_deployed_smoke.yml:
- Restore state before tests run
- After test step, classify via steps.playwright.outcome
- On red→green, post recovery message alongside existing failure path
- On any terminal outcome, save new state to cache
First-ever run (or cache miss): state initializes to ok so we don't
emit a false recovery on the first green run after this ships.
Recovery wording:
✅ *Showcase deploy*: recovered (was down since <iso8601>)
✅ *Starter Deployed Smoke Tests*: recovered (was down since <iso8601>)
Existing red-alert wording is unchanged.
## Summary
Two related defects in the showcase deploy pipeline let stale images sit
live on Railway while Slack stayed green. This PR fixes both.
### Defect 1 — Drift detector skipped all starter services
`.github/workflows/showcase_smoke-monitor.yml` listed only 19
**package** slugs in its `SERVICES=(...)` array (ag2, mastra,
llamaindex, ...). Zero **starter** slugs. As a result:
- GHCR `showcase-starter-<svc>` tags were never checked for drift.
- `gh workflow run showcase_deploy.yml -f service=starter-*` was never
auto-dispatched.
- Starter services could run with weeks-old images and no alert would
fire.
`showcase_deploy.yml` already supports `starter-*` dispatch names and
already calls `serviceInstanceRedeploy` for any service with a
`railway_id`, so no change is required there. The fix is extending
`SERVICES=(...)` to include all 17 starter slugs via a
sparse-checkout-driven filesystem enumeration (no more literal
duplication between workflow and `showcase/starters/`).
### Defect 2 — Silent deploy failures reported green
`.github/workflows/showcase_deploy.yml` emitted `::warning::` and exited
0 when a service never returned 200 on its health path within 360s. The
legacy justification (`# Don't fail — sleep-on-idle services take time
to wake`) no longer applies: Railway is on the Pro tier with no
sleep-on-idle, so a 6-minute failure to become healthy is a real
failure. Changed to `::error::` + `exit 1`.
## Round 2 fixes
Round 2 CR raised six findings against the original smoke-monitor +
validator changes. All fixed in this PR:
- **BLOCKING 1/2 — smoke-monitor guard.** Replaced the magic `-eq 19`
sentinel with `grep -c '^starter-'` so adds/removes to the literal
non-starter list can't silently disable the guard. Added `shopt -s
nullglob` around the `showcase/starters/*/` loop so an empty starters
tree no longer corrupts `SERVICES` with a `starter-*` literal.
- **BLOCKING 3 — GHCR stderr isolation.** Dropped `2>&1` on the `gh api
-i` call; captured stderr to a temp file and surfaced it only when `gh`
returns a non-zero RC with no HTTP status. Auth / rate-limit / network
noise can no longer splice into the HTTP header block and poison
`HTTP_STATUS` / `API_BODY` parsing.
- **BLOCKING 4 — validator tests.** Added
`showcase/scripts/__tests__/validate-workflow-starters.test.ts` (12
specs): happy path, missing-from-options-only, missing-from-matrix-only,
missing-from-both, empty starters dir (exit 3), template/ excluded,
substring-spoof (starter-ag2 vs starter-ag2-extended), missing workflow
file (exit 3). Also extended `VALIDATE_WORKFLOW_STARTERS_REPO_ROOT` to
re-home the starters dir for testability.
- **MEDIUM 1 — YAML parsing.** Replaced the fragile regex-over-YAML
options scanner with a real `yaml.parse()` + typed navigation down
`on.workflow_dispatch.inputs.service.options`. ALL_SERVICES stays
regex-scanned (embedded JSON in a bash heredoc, with `${{ ... }}`
interpolations that aren't valid JSON pre-execution), but the
surrounding step is now located via YAML.
- **MEDIUM 2 — Slack list truncation.** Replaced `cut -c1-200` with a
`truncate_csv` helper that drops whole comma-separated entries until
under budget and appends `…` when truncated. No more
`starter-claude-sdk-pyth` mid-slug corruption.
- **MEDIUM 3 — template/ exclusion cross-references.** Both the TS
validator's `EXCLUDED_DIRS` and `showcase_smoke-monitor.yml`'s `[
"$slug" = "template" ] && continue` now carry `# keep in sync with ...`
comments pointing at each other.
- **NIT 2 — entry-check simplification.** Dropped the
belt-and-suspenders `import.meta.url === \`file://${argv[1]}\`` branch;
kept only the canonical `fileURLToPath(import.meta.url)` form.
- **NIT 3 — jq pipeline collapse.** Single-pass `.jobs[]? | select |
"\(...)"` replaces the three-pass `map | map | .[]` chain in the notify
step.
## Files changed
- `.github/workflows/showcase_deploy.yml` — warning → error + exit 1 on
unhealthy deploy; `truncate_csv` replaces `cut -c1-200` (3 sites);
single-pass jq pipeline in notify step.
- `.github/workflows/showcase_smoke-monitor.yml` — filesystem-driven
`SERVICES=(...)`, starter-count guard, `nullglob` loop, stderr-isolated
`gh api` call, cross-reference comment.
- `.github/workflows/showcase_validate.yml` — wires
`validate-workflow-starters` into CI.
- `showcase/scripts/validate-workflow-starters.ts` — YAML-aware presence
checks; env-var override homes both starters dir and workflow path.
- `showcase/scripts/tsconfig.json` — scripts-local tsconfig for LSP type
resolution.
- `showcase/scripts/__tests__/validate-workflow-starters.test.ts` — 12
specs covering the full matrix of drift scenarios.
## Test plan
- [ ] Next scheduled `showcase_smoke-monitor` run includes starter
services in its drift scan.
- [ ] A deliberately-unhealthy deploy (simulate by pointing health_path
at a 404) fails the job and fires the Slack alert.
- [ ] `showcase_validate` CI job runs `validate-workflow-starters` and
`npx vitest run scripts/__tests__/validate-workflow-starters.test.ts`
green.
Two CI fixes for the showcase_validate python-unit-tests job on PR #4083:
1. Install pytest-asyncio (and pytest-mock) in the shared "minimal test
deps" step. langroid/tests/python/test_agui_adapter.py has 16 tests
annotated with @pytest.mark.asyncio; without the plugin, pytest
reports "async def functions are not natively supported" and skips
them. Locally the user's venv has it; CI did not.
2. Surgically skip the strands package on Python 3.10.
ag_ui_strands==0.1.0 declares requires-python >=3.12,<3.14, so
pip install fails on 3.10 before pytest runs. Prefer the surgical
skip over dropping 3.10 from the matrix so the typing_extensions
NotRequired fallback path in aimock_toggle.py keeps getting
exercised.
Python unit-test workflow previously installed only pytest +
typing_extensions, assuming stdlib-only tests. Tests now import runtime
deps (openai, google.genai, httpx, opentelemetry, etc.) at module load
time, causing ModuleNotFoundError at pytest collection before
conftest-based stub finders run.
Loop each showcase/packages/*/ with a tests/python/ dir, pip install the
package's requirements.txt (if present) before pytest. Re-enable pip
cache keyed on those requirements files. Matches real-runtime parity.
Refs blocker 2 on PR #4083 (fix/showcase-smoke-failures).
Depot Startup plan (unlimited minutes) for consistent runner quality
across test workflows. Bun setup action is runner-agnostic on linux.
Extends the pattern from PR #4018.
Also adds the id-token: write permission required for Depot OIDC
auth, alongside contents: read for least-privilege defaults.
Depot Startup plan (unlimited minutes) for consistent runner quality
across test workflows. Extends the pattern from PR #4018.
Also adds the id-token: write permission required for Depot OIDC
auth, alongside contents: read for least-privilege defaults.
Depot Startup plan (unlimited minutes) for consistent runner quality
across test workflows. Playwright/chromium install is runner-agnostic
on linux. Extends the pattern from PR #4018.
Also adds the id-token: write permission required for Depot OIDC
auth, alongside contents: read for least-privilege defaults.
Depot Startup plan (unlimited minutes) for consistent runner quality
across unit-test workflows. Extends the pattern from PR #4018.
Also adds the id-token: write permission required for Depot OIDC
auth, alongside contents: read for least-privilege defaults.
Depot Startup plan (unlimited minutes) resolves the vitest birpc
onTaskUpdate timeouts on subprocess-heavy suites that standard
ubuntu-latest runners were amplifying. Extends the pattern from
PR #4018 (showcase_validate.yml, showcase_drift-report.yml).
Also adds the id-token: write permission required for Depot OIDC
auth, alongside contents: read for least-privilege defaults.
NIT 3: collapse the per-leg jq pipeline in the notify step. The
previous version went `.jobs // [] | map(filter) | map(project) | .[]`
— three full passes over the list plus a flatten. Rewrite as a single
`.jobs[]? | select | "…"` stream: functionally identical, noticeably
easier to skim. No observable output change.
MEDIUM 3: keep the template/ exclusion cross-referenced in both places
it is enforced — validate-workflow-starters.ts EXCLUDED_DIRS and
showcase_smoke-monitor.yml's per-slug continue. If a future non-service
sibling (e.g. shared/, docs/) gets added, the pair of comments makes
it obvious that both sites need the new entry.
`cut -c1-200` truncated mid-slug, producing reader-hostile output
like `starter-claude-sdk-pyth` or `succeeded: starter-cl`. Replace
with a `truncate_csv` helper that iterates items, appends until the
next item would blow the budget, then emits `…` to signal truncation.
Applied to three list-cap sites (services summary, succeeded_list,
failed_list). Falls back to a hard character cut only when the very
first item is already over budget, so a single pathological 250-char
slug still emits something rather than an empty string.
Helper is inlined per step because YAML run: blocks don't share shell
functions. Comment cross-references steps.legs so the two copies stay
recognisably linked.
`gh api -i 2>&1` merged stderr into stdout, so gh diagnostics (auth
failures, rate limits, network errors) could be spliced ahead of the
HTTP header block. That corrupted HTTP_STATUS/API_BODY parsing — a
failing drift run would look clean while every call actually failed.
Redirect stderr to a mktemp file instead; surface it only when `gh`
returns a non-zero RC with no HTTP status (genuine connectivity/binary
failure). Normal HTTP error paths (401/403/404/5xx) continue to parse
stdout as the canonical header+body response.
Replaces the magic `-eq 19` sentinel that assumed a fixed non-starter
count with a `grep -c '^starter-'` check that counts only the appended
starter entries. Any future add/remove on the literal non-starter list
would have silently disabled the previous guard.
Also enables bash `nullglob` around the `showcase/starters/*/` loop
so an empty/missing starters tree doesn't expand to the literal pattern
and corrupt SERVICES with a bogus `starter-*` entry.
The SERVICES=(...) array in showcase_smoke-monitor.yml's 'Check image
drift' step was a hardcoded copy of the 17 starter slugs already
declared by showcase/starters/*/ directory names. Every new starter
required a manual edit in three places; the parity validator caught
drift after the fact but couldn't prevent it.
This commit:
- Adds a sparse actions/checkout step for showcase/starters/ only.
- Replaces the literal starter-* entries with a filesystem enumeration
(for dir in showcase/starters/*/; do ... done), skipping template/.
- Fails loudly if the enumeration produces zero starters, so a broken
checkout can't silently under-check drift.
- Updates validate-workflow-starters.ts to drop the smoke-monitor check
(drift is now structurally impossible) while keeping the two remaining
literal-list checks against showcase_deploy.yml (workflow_dispatch
options must be literal pre-checkout; ALL_SERVICES matrix carries
per-starter deploy metadata like railway_id).
Non-starter services stay literal — they don't live under
showcase/starters/ and are provisioned differently.
The starter slug list is duplicated across at least three places:
- .github/workflows/showcase_deploy.yml workflow_dispatch options
- .github/workflows/showcase_deploy.yml ALL_SERVICES matrix entries
- .github/workflows/showcase_smoke-monitor.yml SERVICES bash array
Adding a new starter under `showcase/starters/` but forgetting any of
these leaves the service deployable in theory but invisible to the
dispatch UI and/or drift detection — exactly the failure mode this PR
is trying to close.
Add `showcase/scripts/validate-workflow-starters.ts`. It enumerates
every directory under `showcase/starters/` (excluding `template/`)
and confirms `starter-<slug>` is present in each of the three
workflow locations, emitting a precise "missing from: <source>"
diagnostic per gap.
Wire it into showcase_validate.yml right after `validate-parity` so
it gates every PR + main push touching `showcase/**` or the relevant
workflow files. Also extend that workflow's `on.paths` filter to
include `showcase_deploy.yml` and `showcase_smoke-monitor.yml` so
edits to those files trigger the parity check.
The drift detector queried GHCR with `gh api ... || true` and then
skipped the service whenever `$TAGS` was empty. That collapsed three
very different outcomes into one silent continue:
- 200 OK with no published versions yet (legitimate, quiet)
- 404 for a brand-new service (legitimate, quiet)
- 401/403/5xx transient error (NOT legitimate — looks clean and
hides a broken drift check from operators)
Capture the HTTP status via `gh api -i`, route 200-empty and 404 to
a quiet continue with a log line, and emit `::warning::` for any
other status so operators see the problem in the run UI without
failing the whole drift run over one flaky service.
Also tighten the SHA match from substring `grep -q "$SHA"` to word-
boundary `grep -qw "$SHA"`. The previous pattern could false-positive
if one SHA happened to be a prefix of another tag on the same image
(extremely unlikely at 40 chars, but the SHA-8 abbreviation shows up
in tags like `branch-<8char>` which is the failure mode we avoid by
anchoring).
`shell` and `shell-dojolike` use `health_path="/"` because neither
ships an `/api/health` endpoint — they're Next.js app shells serving
the homepage as the liveness signal. With the new `exit 1` on
persistent unhealth, a single transient 5xx or 301-chain glitch at
the homepage during a cold-start would reset the HEALTHY_STREAK and,
at the edge of the 24-attempt budget, could page us for nothing.
Wrap the HTTP probe in a 3-attempt inner retry (2s apart) before the
streak logic observes the code. Services whose homepage is genuinely
dead still fail — all 3 tries must return non-200 — but a one-off
blip no longer trashes accumulated progress. Non-shell services
(which hit `/api/health`, `/health`, etc.) benefit from the same
transient-tolerance for free.
The notify job's failure branch posted "FAILED — N service(s) targeted"
whenever `needs.build.result == 'failure'`, which rolls up to 'failure'
if even a single matrix leg failed. With fail-fast: false, one failed
leg out of 18 can roll up as 'failure' while 17 others shipped fine —
the old wording falsely implied all N failed.
Add a "Compute per-leg build results" step that queries the Actions
API for this run's jobs, buckets each `build (<leg>)` leg by its
conclusion, and exposes failed/succeeded counts + lists as outputs.
The payload step now picks between three shapes:
- Partial failure: "N/M service(s) failed (<failed>) — <succeeded> ok"
- Full failure: existing "FAILED — M service(s) targeted (<list>)"
- API lookup lost: softer "1+ of M service(s) failed (<list> targeted)"
so the alert never lies about how many were affected.
Also reword the stale "sleep-on-idle waking up" comment on the
success-but-no-HTTP-200 branch — Railway is on the Pro tier (see
rationale near line 412) and does not sleep on idle. Clarify the
notify cancel-handling comment accordingly.
Previously, when a deployed service never returned a 200 on its health
path within the 360s window, the workflow emitted `::warning::` and
exited 0. That masked silently-broken deploys: Slack saw green, smoke
monitor saw no failing deploy run, and the stale image sat live.
Emit `::error::` and exit 1 so the deploy job fails and the alert
actually fires. Railway is on the Pro tier with no sleep-on-idle, so a
persistent failure to become healthy in 6 minutes is a real failure,
not a cold start.
Evidence of silent failures: https://github.com/CopilotKit/CopilotKit/actions/runs/24616467033
The smoke monitor's SERVICES array only listed package slugs (mastra,
llamaindex, ...) and did not include any starter-* slugs. As a result,
GHCR showcase-starter-<svc> tags were never checked for drift and the
monitor never dispatched showcase_deploy.yml for starter services when
their images fell behind main.
showcase_deploy.yml already accepts starter-* dispatch names and already
redeploys any service with a railway_id via serviceInstanceRedeploy, so
simply extending SERVICES to include the 17 starter slugs is sufficient
to close the loop.
Evidence of the gap: https://github.com/CopilotKit/CopilotKit/actions/runs/24616467033
The Alert Slack step used `payload-file-path` pointed at an
`mktemp`-produced extensionless tmpfile. slackapi/slack-github-action
v2.1.0 requires the payload file to have a `.json` / `.yaml` / `.yml`
extension and rejects extensionless paths with:
SlackError: Invalid input! Failed to parse file extension /tmp/tmp.XXX
SlackError: Invalid input! Failed to parse contents of the provided payload file
This silent failure (masked by `continue-on-error: true`) meant smoke
results never reached #oss-alerts.
Switch to the inline `payload:` + `toJSON(format(...))` pattern from
showcase_validate.yml (PR #4068, commit d4e75958). This:
- avoids the file-extension trap entirely — no temp file needed
- JSON-encodes dynamic values so quotes/backslashes/newlines in
starter slugs or error excerpts can't break the payload
- carries failed job + first error line per the #oss-alerts
actionable-detail policy (feedback_oss_alerts_detail)
The extraction step is preserved but now emits to `$GITHUB_ENV` with
heredoc delimiters (not `$GITHUB_OUTPUT`) so the Slack step can reference
values via `env.*` inside the `toJSON(format(...))` expression. Four
failure modes are still distinguished (missing_report, jq_parse_failed,
no_failures_in_report, real failures) via the `extraction_error` + truncated
first-failure excerpt emitted as `error_excerpt`.
The `continue-on-error: true` and temp-file cleanup steps are removed —
the inline payload approach no longer creates anything to clean up, and
a genuine Slack failure should surface in the job status.
Bare "❌ Showcase validate: failed | View run" forces a click-through to
triage — the signal that would make the alert actionable is one click
away. Per the oss-alerts policy, red alerts must carry triage-ready
detail in the payload itself.
Adds a pre-notify step that (on failure(), push, webhook set):
- resolves the current job's ID via the runs/{id}/jobs API (matches by
job name; falls back to "first job with a failed step" for rename
drift tolerance)
- extracts the first failed step name from the same jobs response
- pulls the failed-step log via `gh run view --log-failed --job=<id>`,
strips TSV prefix + leading timestamp + ANSI codes, skips runner
header noise (##[group], shell:, env:, Run), then grabs the first
line matching [FAIL]/[ERROR]/Error:/::error and truncates to 300
chars
- emits failed_step and error_excerpt to $GITHUB_ENV via heredoc
delimiters (safe for values containing = or newlines)
- uses set +e and explicit exit 0 so extraction glitches never block
the notify step — fallbacks yield "unknown step" / "see workflow
run for details"
Updates the Slack payload to interpolate both values via toJSON(format())
so dynamic content is JSON-encoded defensively (matches the pattern from
showcase_drift-report.yml). Preserves the ❌ prefix, the "View run"
link, and keeps total length well under the 800-char budget.
Companion to #4065, which quieted routine per-run success posts; this
PR makes the remaining (failure-only) posts actionable at a glance.
Against run 24598654559 (tonight's validate-parity regression), the
extracted excerpt is:
[FAIL] ag2: demo 'hitl-in-chat' declared in manifest but no
src/app/demos/hitl-in-chat/ directory
which is exactly the triage signal that was missing from the bare
"failed" alert.
Blocking fix: aimock serves `/__aimock/health` + `/v1/*` only — probing `/`
returns HTTP 404 which fails `curl -sf`, so the readiness loop never broke
and every run hard-failed at "Start aimock". Switched all three probes
(startup loop, startup verify, pre-Playwright re-probe) to
`/__aimock/health`.
Also tightened the workflow in four smaller ways:
* Added `--max-time 2 --connect-timeout 1` to every curl probe (aimock
readiness + verify, agent readiness + verify, dev-server readiness +
verify, pre-Playwright re-probe). A hung socket can't blow the loop's
iteration budget now.
* Shell slug extractor (`grep -oE`) and job-level `startsWith` gate now
agree on "first token only": the extractor anchors at `^` instead of
allowing a leading-whitespace alternative. Header comment updated so the
two-layer defense-in-depth rationale reflects the now-identical behavior.
* Captured aimock PID into `$GITHUB_ENV` and added an `if: always()`
"Re-check aimock liveness after Playwright" step that `kill -0`s the PID
after tests. If aimock OOM'd mid-Playwright, the job now fails loudly
instead of trusting cached / fall-through results.
* Removed dead `OPENAI_BASE_URL` / `OPENAI_API_KEY` env on the
`Run Playwright tests` step (Next.js was already running from the earlier
step with these inline on that process; env on `npx playwright test`
didn't flow anywhere). Added comment explaining why setting them here
would be misleading.
* Expanded comment on `workflow_dispatch.inputs.slug.type: choice` to
document that single-choice UI preselect IS intentional (refused the
sentinel-value footgun).
- Replace `npm bin -g` with `npm prefix -g` + /bin/aimock. `npm bin` was
removed in npm 9.0.0 and setup-node@v4 with node 22.x ships npm 10+, so
the old form exited with "Unknown command: 'bin'" and the existence
check below fired on every run, hard-failing the workflow.
- Replace `pip install --only-binary :all:` with `--prefer-binary`. CrewAI's
transitive graph (tiktoken / chromadb / litellm cadence releases) ships
sdist-only revisions often enough that `--only-binary :all:` made the
workflow fail-loud with "Could not find a version that satisfies the
requirement" on otherwise-valid requirements.txt. `--prefer-binary`
keeps the wheel-first preference while letting sdist-only deps install;
residual source-build-hook risk is bounded by the author_association gate.
- Document the comment-trigger TOCTOU in the residual-trust-model header:
there is a window between `/test-aimock` (reviewing diff D1) and
`pulls.get` (resolves whatever HEAD is current), so a force-push of
malicious content in between wins. GitHub Actions has no native
comment-time SHA pin, so the mitigation is the author_association gate
+ social contract — documenting accepts the known residual risk.
- Drop dead `else APP_MODULE=agent:app` branch. All currently-dispatchable
slugs ship src/agent_server.py (enforced by the ships_toggle check
above), so the legacy path was unreachable. Fail loud with `::error::`
instead of silently falling through to a guessed module name.
- Capture aimock PID and `kill -0` inside the readiness loop so an aimock
that crashes on startup fails in ~1s instead of burning the full 20s
polling a dead process.
- Add `persist-credentials: false` to the PR-HEAD checkout. Prevents the
workflow's GITHUB_TOKEN from being left in .git/config where attacker-
controlled PR-HEAD code (pip build hooks, Playwright fixtures) could
read it. Defense-in-depth on top of `permissions: contents: read`.
- Guard the Resolve PR HEAD step with `pr.state === 'open'`. A trusted
commenter typing `/test-aimock` on a stale closed PR would otherwise
re-exec the old HEAD — wasting CI or re-running known-bad code.
- Add a residual-trust-model header comment to the workflow documenting
the social contract (trusted commenter reviews PR diff before typing
the command; not a security boundary against a rogue OWNER/MEMBER).
- Tighten the Actions-level gate from `contains(body, '/test-aimock ')`
to `startsWith(body, '/test-aimock ')`. Substring matches fire on
mentions inside code blocks / quoted replies; startsWith requires
`/test-aimock ` at the very start of the comment.
- Replace `npx @copilotkit/aimock@...` with a direct invocation of the
globally-installed binary via `$(npm bin -g)/aimock`. npx may re-resolve
the spec against the registry, defeating both `--ignore-scripts` (which
doesn't propagate to npx's transient install) and the caret pin if a
new patch published between global install and npx.
- Add a second aimock liveness probe right before Playwright runs. If
aimock died between initial readiness and test execution, Playwright
would silently fall through to real OpenAI (since OPENAI_BASE_URL
points at a dead port). Fail loud instead.
- Remove the dead `/` fallback in the Python agent health check. FastAPI's
root is typically a POST endpoint that always fails `curl -sf` — the
fallback was copy-paste residue that added no signal.
- Pin `pnpm/action-setup` to `@v4.4.0` matching showcase_validate.yml
(was floating `@v4`).
- Make workflow_dispatch `slug` input required (no default). A hidden
default to crewai-crews contradicted the "no silent fallback" claim
enforced by the comment-path extractor.
- Reword the Next.js OPENAI_BASE_URL comment to acknowledge it's
defensive-only in the CrewAI showcase (Next proxies via runtime, not
direct OpenAI); the value still prevents a future route-level leak.
- Add `--only-binary :all:` to the pip install in showcase_validate.yml's
python-unit-tests job, mirroring the hardening already in the aimock-e2e
sibling workflow. This job runs on EVERY PR automatically without a
trusted-commenter gate, so the broader exposure deserves the same
wheel-only install defense against PR-controlled setup.py hooks.
- Drop the unconditional per-package requirements.txt install. Today's
tests are stdlib-only (+ typing_extensions fallback path); installing
the full CrewAI graph on every PR run was a ~30-60s tax with no benefit.
Install only pytest + typing_extensions. Comment documents the upgrade
path for future tests that need package runtime deps.
- Drop the now-unused pip cache on setup-python (keyed on a never-used
requirements.txt would be pure overhead).
HIGH fixes from CR5 round:
- H1: `pip install --only-binary :all:` forces wheel-only installs, blocking
source-build hooks (setup.py / PEP 517 build backends) from executing on
the runner. Unlike npm/pnpm there is no `pip --ignore-scripts` equivalent;
wheels are the closest mitigation. Residual risk documented inline.
- H2: Tighten `/test-aimock` match so `/test-aimocker` / `/test-aimock-like-this`
do NOT trigger runs. Require trailing space via job-level `if:` AND enforce
whole-word boundary in the shell-level slug extractor (belt-and-suspenders).
Remove the silent `crewai-crews` default fallback — a missing slug now FAILS
the workflow instead of silently running against the default.
- M4: Scope write perms (pull-requests + issues) to a SEPARATE `post-result`
job that only runs the comment post. The heavy test job now runs with
`contents: read` only, so a compromised transitive dep from `pip install`
on a PR-controlled requirements.txt cannot mutate PRs/issues with the
workflow token.
- M8: Use `npx next dev` directly in "Start dev server" step. `pnpm dev`
would spawn a SECOND uvicorn on :8000 via concurrently, racing the
already-bound agent from the prior step and silently picking whichever
won the port.
LOW fixes:
- Pin `npx` invocation to fully-scoped `@copilotkit/aimock@^1.14.3` so a
squatter on the unscoped `aimock` name can't be silently picked up if the
global install fails.
- Add rationale comment explaining why `steps.slug.outputs.slug` interpolation
is safe (slug is whitelisted + dir-validated before use).
Jordan Ritter