Adds a new step that runs validate-fixture-tool-surface.ts on every PR
and push to main. Sits alongside the existing validate-parity /
validate-workflow-starters / validate-pins steps and follows the same
pnpm-exec-tsx pattern.
Without this, the drift validator only runs locally or via the
vitest suite (which only catches bugs in the validator itself, not
drift in the real fixture/demo state). The CLI invocation against the
committed tree is what would have caught the 2026-04-22 regression
before it reached prod.
## Summary
Two related defects in the showcase deploy pipeline let stale images sit
live on Railway while Slack stayed green. This PR fixes both.
### Defect 1 — Drift detector skipped all starter services
`.github/workflows/showcase_smoke-monitor.yml` listed only 19
**package** slugs in its `SERVICES=(...)` array (ag2, mastra,
llamaindex, ...). Zero **starter** slugs. As a result:
- GHCR `showcase-starter-<svc>` tags were never checked for drift.
- `gh workflow run showcase_deploy.yml -f service=starter-*` was never
auto-dispatched.
- Starter services could run with weeks-old images and no alert would
fire.
`showcase_deploy.yml` already supports `starter-*` dispatch names and
already calls `serviceInstanceRedeploy` for any service with a
`railway_id`, so no change is required there. The fix is extending
`SERVICES=(...)` to include all 17 starter slugs via a
sparse-checkout-driven filesystem enumeration (no more literal
duplication between workflow and `showcase/starters/`).
### Defect 2 — Silent deploy failures reported green
`.github/workflows/showcase_deploy.yml` emitted `::warning::` and exited
0 when a service never returned 200 on its health path within 360s. The
legacy justification (`# Don't fail — sleep-on-idle services take time
to wake`) no longer applies: Railway is on the Pro tier with no
sleep-on-idle, so a 6-minute failure to become healthy is a real
failure. Changed to `::error::` + `exit 1`.
## Round 2 fixes
Round 2 CR raised six findings against the original smoke-monitor +
validator changes. All fixed in this PR:
- **BLOCKING 1/2 — smoke-monitor guard.** Replaced the magic `-eq 19`
sentinel with `grep -c '^starter-'` so adds/removes to the literal
non-starter list can't silently disable the guard. Added `shopt -s
nullglob` around the `showcase/starters/*/` loop so an empty starters
tree no longer corrupts `SERVICES` with a `starter-*` literal.
- **BLOCKING 3 — GHCR stderr isolation.** Dropped `2>&1` on the `gh api
-i` call; captured stderr to a temp file and surfaced it only when `gh`
returns a non-zero RC with no HTTP status. Auth / rate-limit / network
noise can no longer splice into the HTTP header block and poison
`HTTP_STATUS` / `API_BODY` parsing.
- **BLOCKING 4 — validator tests.** Added
`showcase/scripts/__tests__/validate-workflow-starters.test.ts` (12
specs): happy path, missing-from-options-only, missing-from-matrix-only,
missing-from-both, empty starters dir (exit 3), template/ excluded,
substring-spoof (starter-ag2 vs starter-ag2-extended), missing workflow
file (exit 3). Also extended `VALIDATE_WORKFLOW_STARTERS_REPO_ROOT` to
re-home the starters dir for testability.
- **MEDIUM 1 — YAML parsing.** Replaced the fragile regex-over-YAML
options scanner with a real `yaml.parse()` + typed navigation down
`on.workflow_dispatch.inputs.service.options`. ALL_SERVICES stays
regex-scanned (embedded JSON in a bash heredoc, with `${{ ... }}`
interpolations that aren't valid JSON pre-execution), but the
surrounding step is now located via YAML.
- **MEDIUM 2 — Slack list truncation.** Replaced `cut -c1-200` with a
`truncate_csv` helper that drops whole comma-separated entries until
under budget and appends `…` when truncated. No more
`starter-claude-sdk-pyth` mid-slug corruption.
- **MEDIUM 3 — template/ exclusion cross-references.** Both the TS
validator's `EXCLUDED_DIRS` and `showcase_smoke-monitor.yml`'s `[
"$slug" = "template" ] && continue` now carry `# keep in sync with ...`
comments pointing at each other.
- **NIT 2 — entry-check simplification.** Dropped the
belt-and-suspenders `import.meta.url === \`file://${argv[1]}\`` branch;
kept only the canonical `fileURLToPath(import.meta.url)` form.
- **NIT 3 — jq pipeline collapse.** Single-pass `.jobs[]? | select |
"\(...)"` replaces the three-pass `map | map | .[]` chain in the notify
step.
## Files changed
- `.github/workflows/showcase_deploy.yml` — warning → error + exit 1 on
unhealthy deploy; `truncate_csv` replaces `cut -c1-200` (3 sites);
single-pass jq pipeline in notify step.
- `.github/workflows/showcase_smoke-monitor.yml` — filesystem-driven
`SERVICES=(...)`, starter-count guard, `nullglob` loop, stderr-isolated
`gh api` call, cross-reference comment.
- `.github/workflows/showcase_validate.yml` — wires
`validate-workflow-starters` into CI.
- `showcase/scripts/validate-workflow-starters.ts` — YAML-aware presence
checks; env-var override homes both starters dir and workflow path.
- `showcase/scripts/tsconfig.json` — scripts-local tsconfig for LSP type
resolution.
- `showcase/scripts/__tests__/validate-workflow-starters.test.ts` — 12
specs covering the full matrix of drift scenarios.
## Test plan
- [ ] Next scheduled `showcase_smoke-monitor` run includes starter
services in its drift scan.
- [ ] A deliberately-unhealthy deploy (simulate by pointing health_path
at a 404) fails the job and fires the Slack alert.
- [ ] `showcase_validate` CI job runs `validate-workflow-starters` and
`npx vitest run scripts/__tests__/validate-workflow-starters.test.ts`
green.
Two CI fixes for the showcase_validate python-unit-tests job on PR #4083:
1. Install pytest-asyncio (and pytest-mock) in the shared "minimal test
deps" step. langroid/tests/python/test_agui_adapter.py has 16 tests
annotated with @pytest.mark.asyncio; without the plugin, pytest
reports "async def functions are not natively supported" and skips
them. Locally the user's venv has it; CI did not.
2. Surgically skip the strands package on Python 3.10.
ag_ui_strands==0.1.0 declares requires-python >=3.12,<3.14, so
pip install fails on 3.10 before pytest runs. Prefer the surgical
skip over dropping 3.10 from the matrix so the typing_extensions
NotRequired fallback path in aimock_toggle.py keeps getting
exercised.
Python unit-test workflow previously installed only pytest +
typing_extensions, assuming stdlib-only tests. Tests now import runtime
deps (openai, google.genai, httpx, opentelemetry, etc.) at module load
time, causing ModuleNotFoundError at pytest collection before
conftest-based stub finders run.
Loop each showcase/packages/*/ with a tests/python/ dir, pip install the
package's requirements.txt (if present) before pytest. Re-enable pip
cache keyed on those requirements files. Matches real-runtime parity.
Refs blocker 2 on PR #4083 (fix/showcase-smoke-failures).
The starter slug list is duplicated across at least three places:
- .github/workflows/showcase_deploy.yml workflow_dispatch options
- .github/workflows/showcase_deploy.yml ALL_SERVICES matrix entries
- .github/workflows/showcase_smoke-monitor.yml SERVICES bash array
Adding a new starter under `showcase/starters/` but forgetting any of
these leaves the service deployable in theory but invisible to the
dispatch UI and/or drift detection — exactly the failure mode this PR
is trying to close.
Add `showcase/scripts/validate-workflow-starters.ts`. It enumerates
every directory under `showcase/starters/` (excluding `template/`)
and confirms `starter-<slug>` is present in each of the three
workflow locations, emitting a precise "missing from: <source>"
diagnostic per gap.
Wire it into showcase_validate.yml right after `validate-parity` so
it gates every PR + main push touching `showcase/**` or the relevant
workflow files. Also extend that workflow's `on.paths` filter to
include `showcase_deploy.yml` and `showcase_smoke-monitor.yml` so
edits to those files trigger the parity check.
Bare "❌ Showcase validate: failed | View run" forces a click-through to
triage — the signal that would make the alert actionable is one click
away. Per the oss-alerts policy, red alerts must carry triage-ready
detail in the payload itself.
Adds a pre-notify step that (on failure(), push, webhook set):
- resolves the current job's ID via the runs/{id}/jobs API (matches by
job name; falls back to "first job with a failed step" for rename
drift tolerance)
- extracts the first failed step name from the same jobs response
- pulls the failed-step log via `gh run view --log-failed --job=<id>`,
strips TSV prefix + leading timestamp + ANSI codes, skips runner
header noise (##[group], shell:, env:, Run), then grabs the first
line matching [FAIL]/[ERROR]/Error:/::error and truncates to 300
chars
- emits failed_step and error_excerpt to $GITHUB_ENV via heredoc
delimiters (safe for values containing = or newlines)
- uses set +e and explicit exit 0 so extraction glitches never block
the notify step — fallbacks yield "unknown step" / "see workflow
run for details"
Updates the Slack payload to interpolate both values via toJSON(format())
so dynamic content is JSON-encoded defensively (matches the pattern from
showcase_drift-report.yml). Preserves the ❌ prefix, the "View run"
link, and keeps total length well under the 800-char budget.
Companion to #4065, which quieted routine per-run success posts; this
PR makes the remaining (failure-only) posts actionable at a glance.
Against run 24598654559 (tonight's validate-parity regression), the
extracted excerpt is:
[FAIL] ag2: demo 'hitl-in-chat' declared in manifest but no
src/app/demos/hitl-in-chat/ directory
which is exactly the triage signal that was missing from the bare
"failed" alert.
- Add `--only-binary :all:` to the pip install in showcase_validate.yml's
python-unit-tests job, mirroring the hardening already in the aimock-e2e
sibling workflow. This job runs on EVERY PR automatically without a
trusted-commenter gate, so the broader exposure deserves the same
wheel-only install defense against PR-controlled setup.py hooks.
- Drop the unconditional per-package requirements.txt install. Today's
tests are stdlib-only (+ typing_extensions fallback path); installing
the full CrewAI graph on every PR run was a ~30-60s tax with no benefit.
Install only pytest + typing_extensions. Comment documents the upgrade
path for future tests that need package runtime deps.
- Drop the now-unused pip cache on setup-python (keyed on a never-used
requirements.txt would be pure overhead).
showcase_aimock-e2e.yml:
- workflow_dispatch slug is now a choice-type enum restricted to Python
packages that ship aimock_toggle.py; a TS (mastra) or Java (spring-ai)
slug would have skipped the Python agent start step and then failed with
a misleading Playwright timeout. For comment-trigger paths, the pkg-type
step short-circuits with a clear ::error:: when the slug does not ship
aimock_toggle.py.
- Slug parsing replaced PCRE grep -oP '\\K\\S+' with a POSIX-safe
grep -oE + sed pipeline so BSD/Alpine grep works too (future-proof
against runner image changes).
- aimock pinned to @copilotkit/aimock@^1.14.3 (was @latest) with
--ignore-scripts; unpinned @latest let a bad aimock publish silently
poison CI for everyone.
- pnpm install now runs with --ignore-scripts — trusted commenter triggers
/test-aimock on untrusted PR content, so postinstall scripts must not
run on the runner.
- OPENAI_BASE_URL is no longer pre-set for Python agent start on packages
that ship aimock_toggle.py — forcing the toggle itself to do the
redirection proves the toggle works rather than masking it with env
already set.
- Dead AIMOCK_URL export removed from the Next.js dev-server step (the
runtime only reads OPENAI_BASE_URL).
- Python agent health-check extended 60s -> 90s (45 iter * 2s) for CrewAI
cold imports.
- actions/setup-python caches pip keyed on requirements.txt.
- github-script Post-result step passes slug + job status via env rather
than ${{ }} interpolation — even though the slug is already whitelisted,
the env pattern is the defensive default for dynamic values.
showcase_validate.yml:
- python-unit-tests is now a matrix on Python 3.10 + 3.12 (fail-fast
disabled) so the typing_extensions fallback branch gets CI coverage;
previously only 3.12 ran.
- pip install per package is a HARD FAIL (was ::warning:: + continue);
hiding broken requirements.txt behind a warning let a package ship green
with unresolvable runtime deps. A missing requirements.txt is still
handled gracefully via the -f guard.
- pip caching added via actions/setup-python.
The `secrets.*` context is not a valid named-value inside step-level
`if:` expressions on push events — GitHub Actions rejects it at
workflow parse time with "Unrecognized named-value: 'secrets'", which
caused both showcase_validate.yml and showcase_drift-report.yml to
fail at startup with zero jobs spawned after #4018 + #4060 merged.
Hoist the webhook into a job-level `env: SLACK_WEBHOOK` and reference
`env.SLACK_WEBHOOK` in every step-level `if:`. The `with: webhook:`
keys still use `secrets.*` directly (valid in that context).
PR CI didn't catch this because pull_request events parse if:
expressions less strictly than push events.
showcase_validate.yml and showcase_drift-report.yml were flipped to
runs-on: depot-ubuntu-24.04-4 in #4018 but inherited the repo's
default least-privilege permissions (contents: read only). Depot
runner provisioning uses OIDC and requires id-token: write, so the
jobs failed to spawn on the first main-branch push — matching the
pattern already used by showcase_deploy.yml's Depot job.
Two GitHub Actions workflows that consume the three validators in
showcase/scripts/ and surface drift to CI and Slack.
showcase_validate.yml — runs on pull_request and push-to-main:
- Enforces per-package e2e spec count against
fail-baseline.json.baselineDemoCount, with find failures
aggregated (not exit-on-first) so multiple package issues surface
in one run. Preserves find's exit status by capturing via command
substitution rather than process substitution (mapfile does not
propagate exit codes through < <(...)).
- Runs validate-pins.ts with both count + content-hash ratchet:
the sorted-uniqued [FAIL] set is SHA-256 hashed so a
count-preserving set change ('one fail healed, another regressed')
is still flagged.
- Separates stdout from stderr before hashing so progress chatter
cannot corrupt the ratchet hash.
- Preserves validator exit codes distinctly (1 drift, 2 internal,
3 unreadable, 4+ future) so downstream consumers can distinguish
crashes from legitimate drift.
- set -euo pipefail throughout, with { grep || true; } scoped to
tolerate grep no-match without masking producer failures.
- Slack notifications gated on push events and secret presence;
payload values wrapped via toJSON(format(...)) for injection
safety.
showcase_drift-report.yml — weekly Monday 10:00 UTC + dispatch:
- Same ratchet + hash logic applied in report mode: computes
set_status (OK / SET DRIFTED / COUNT DRIFTED) and posts to Slack.
- Mirrors validate.yml's pipefail + grep-scope + toJSON discipline.
- Add --validate-on-load to all aimock invocations (4 workflows/scripts
+ 13 integration docker-compose files)
- Replace hardcoded 2-file fixture list with dynamic discovery across
showcase/, examples/integrations/*/, scripts/doc-tests/ (16 fixtures)
- Add sanity check to prevent silent zero-test pass when discovery fails
- Extend showcase_validate.yml path filter to trigger on
examples/integrations/**/fixtures/** and scripts/doc-tests/fixtures/**
- Import and use ValidationResult type for callback parameters
- Fix scripts/doc-tests/fixtures/default.json to use { fixtures: [...] }
envelope shape
- Node 20 → 22 in all Dockerfiles + CI workflows
- CI copies shared_python, shared_frontend/src, shared_typescript/tools
- tsconfig paths for @copilotkit/showcase-shared
- .gitignore for CI artifacts
Jordan Ritter