When a promote fails, the succeeded-service set is empty, so verify-prod
hits its skip branch (`exit 0`). The GitHub job result is therefore
`success`, and the notify step rendered `verify-prod=success` in the
#oss-alerts Slack message — a misleading green, since prod was never
probed.
verify-prod now exports a `status` output: `success` after a real probe
passes, `skipped` on the empty-CSV skip. notify reads that output (via
the new bats-tested verify-prod-display.sh) instead of the raw job
result, so the Slack line accurately reads `verify-prod=skipped` vs
`success` vs `failure`. A genuine probe failure / contract violation
exits non-zero (job result `failure`, status never written), and the
display falls back to the job result. Slack formatting is unchanged.
Extracts the display mapping into showcase/scripts/verify-prod-display.sh
(mirroring promote-fleet.sh) with red-green bats coverage, and adds it to
the showcase_validate.yml shellcheck step.
GitHub Actions expression string literals don't interpret `\n`, so
`toJSON(format('...\n...'))` emits literal `\\n` and Slack renders the
two characters backslash-n instead of a line break. Inject real newlines
via a `fromJSON('"\n"')` placeholder, matching the starter-smoke fix.
Fixes the "all builds failed" and "Showcase Build Failed" alerts in
showcase_build.yml and the multi-line "showcase_validate failed" alert
in showcase_validate.yml. showcase_promote.yml already used the
fromJSON placeholder; the single-line validate alert has no newlines
and was left untouched.
the per-service promote loop ran under `set -euo pipefail`, so the first failing
service aborted the whole `all` fleet promote; extracted to promote-fleet.sh
which attempts every service, accumulates succeeded/failed sets, exits non-zero
only after attempting all, and exports succeeded_csv. verify-prod now runs
`if: !cancelled()` and scopes --services to the succeeded set; the staging
precondition is advisory (promote runs even when it reports red — bin/railway
enforces staging-green per-service); notify success keys on PROMOTE && PROD.
Adds a shell-script-tests CI job (bats + shellcheck) and input-validation
hardening (fail-loud on empty/all-empty CSV, RAILWAY_BIN check, whitespace trim).
Add showcase/scripts/sync-promote-service-options.ts: generates the
promote workflow's service `choice` options from the SSOT
(railway-envs.ts), spliced between BEGIN/END markers in
showcase_promote.yml. Fail-loud throughout — every emitted token must
resolve to exactly one service under the resolve-step predicate
(name|dispatchName match AND probe.prod), tokens are YAML-safe, args are
strict (a typo'd flag cannot trigger a destructive write), and markers
are validated before any rewrite.
Wire it into a lefthook pre-commit hook (regenerate + restage; set -e so
a failed regen blocks the commit) and an advisory (never-failing) drift
check in showcase_validate.yml. Vitest coverage for ordering, exclusion,
collision/ambiguity guards, marker errors, exit codes, idempotency, and
the import-side-effect guard.
The new no-webhook log step interpolated github.ref directly into a shell
echo, which zizmor flags as template-injection (HIGH). Bind it to a step
env var REF and reference $REF in the script instead. Message text is
unchanged; no Co-Authored-By.
Collapse the promote workflow to an input-agnostic concurrency group so promotes can't race the
same Railway service; add #oss-alerts failure notifications to the build and validate workflows
(build via extended needs, validate via a new workflow-level notify job).
Comprehensive CI/CD security hardening pass over all 33 workflows.
Action pinning
- Every `uses:` is now pinned to a 40-char commit SHA with a `# vX.Y.Z`
comment alongside (167 occurrences resolved). Tag-style refs like `@v4`
are mutable and have been used in past supply-chain attacks (e.g.
tj-actions/changed-files in March 2025) to repoint widely-used actions
to malicious commits.
- Removed redundant `version: "10.13.1"` hardcodes from `pnpm/action-setup`
call sites so the action inherits from package.json `packageManager`
(one source of truth).
Automated maintenance
- Added `.github/dependabot.yml` for the `github-actions` ecosystem so
SHA pins stay current. Without this, pins go stale fast and new
upstream advisories never reach us. Minor/patch bumps are grouped;
major bumps stay separate so they get a real review.
Static analysis
- Added `.github/zizmor.yml` configuration and
`.github/workflows/security_zizmor.yml` (blocking on PR, runs on push
to main, weekly schedule for advisory drift). zizmor catches the
well-known classes of Actions footguns: template injection from
untrusted input, dangerous triggers, unpinned uses, excessive token
scopes, secret exfil patterns.
- All 28 high-severity and 54 medium-severity findings from the baseline
scan are remediated. Each suppression in zizmor.yml carries a
per-finding justification comment so future maintainers can audit the
trust assumption.
Workflow hardening (from zizmor + manual audit)
- Added `persist-credentials: false` to every `actions/checkout` except
the 7 workflows that legitimately push back to the repo via the
workflow token (release tagging, auto-formatting, docs-sync, registry
updates). Each retained credential persistence carries a
`persist-credentials required: ...` comment explaining the call site.
- Routed every attacker-controllable expansion (`github.head_ref`,
`github.event.pull_request.head.repo.full_name`, `inputs.*`,
step outputs) through `env:` and referenced as quoted shell variables.
Eliminates 17 template-injection vectors in fork-PR-reachable
workflows.
- Added per-job `permissions:` blocks across 14 workflows; demoted
broad workflow-level `id-token: write` to the specific Depot-runner
jobs that need it; narrowed `pull-requests: write` /
`actions: write` to the jobs that actually call those APIs.
Audit-driven fixes
- `publish-release.yml` build job: dropped `token:` and added
`persist-credentials: false`. The subsequent `Upload workspace` step
was packing `.git/config` (with the persisted GITHUB_TOKEN) into a
1-day-retention artifact downloadable by anyone with `actions:read`.
- `auto_merge_showcases.yml`: team-membership check now authorizes on
the PR AUTHOR (`pull_request.user.login`), never `context.actor` —
the actor is whoever triggered the latest event, so a team member
synchronizing or reopening an outsider's PR would otherwise
green-light auto-merge of code they didn't author.
- `static_quality.yml`: pinned ruff to a specific version so a
compromised release can't land on the next PR run with the
persisted-credentials write token in the format job.
- `showcase_capture-previews.yml`: switched the args-string construction
to a bash array so a slug or demo value containing whitespace or shell
metacharacters stays a single argument rather than being re-tokenized
by the shell.
Adds a new step that runs validate-fixture-tool-surface.ts on every PR
and push to main. Sits alongside the existing validate-parity /
validate-workflow-starters / validate-pins steps and follows the same
pnpm-exec-tsx pattern.
Without this, the drift validator only runs locally or via the
vitest suite (which only catches bugs in the validator itself, not
drift in the real fixture/demo state). The CLI invocation against the
committed tree is what would have caught the 2026-04-22 regression
before it reached prod.
## Summary
Two related defects in the showcase deploy pipeline let stale images sit
live on Railway while Slack stayed green. This PR fixes both.
### Defect 1 — Drift detector skipped all starter services
`.github/workflows/showcase_smoke-monitor.yml` listed only 19
**package** slugs in its `SERVICES=(...)` array (ag2, mastra,
llamaindex, ...). Zero **starter** slugs. As a result:
- GHCR `showcase-starter-<svc>` tags were never checked for drift.
- `gh workflow run showcase_deploy.yml -f service=starter-*` was never
auto-dispatched.
- Starter services could run with weeks-old images and no alert would
fire.
`showcase_deploy.yml` already supports `starter-*` dispatch names and
already calls `serviceInstanceRedeploy` for any service with a
`railway_id`, so no change is required there. The fix is extending
`SERVICES=(...)` to include all 17 starter slugs via a
sparse-checkout-driven filesystem enumeration (no more literal
duplication between workflow and `showcase/starters/`).
### Defect 2 — Silent deploy failures reported green
`.github/workflows/showcase_deploy.yml` emitted `::warning::` and exited
0 when a service never returned 200 on its health path within 360s. The
legacy justification (`# Don't fail — sleep-on-idle services take time
to wake`) no longer applies: Railway is on the Pro tier with no
sleep-on-idle, so a 6-minute failure to become healthy is a real
failure. Changed to `::error::` + `exit 1`.
## Round 2 fixes
Round 2 CR raised six findings against the original smoke-monitor +
validator changes. All fixed in this PR:
- **BLOCKING 1/2 — smoke-monitor guard.** Replaced the magic `-eq 19`
sentinel with `grep -c '^starter-'` so adds/removes to the literal
non-starter list can't silently disable the guard. Added `shopt -s
nullglob` around the `showcase/starters/*/` loop so an empty starters
tree no longer corrupts `SERVICES` with a `starter-*` literal.
- **BLOCKING 3 — GHCR stderr isolation.** Dropped `2>&1` on the `gh api
-i` call; captured stderr to a temp file and surfaced it only when `gh`
returns a non-zero RC with no HTTP status. Auth / rate-limit / network
noise can no longer splice into the HTTP header block and poison
`HTTP_STATUS` / `API_BODY` parsing.
- **BLOCKING 4 — validator tests.** Added
`showcase/scripts/__tests__/validate-workflow-starters.test.ts` (12
specs): happy path, missing-from-options-only, missing-from-matrix-only,
missing-from-both, empty starters dir (exit 3), template/ excluded,
substring-spoof (starter-ag2 vs starter-ag2-extended), missing workflow
file (exit 3). Also extended `VALIDATE_WORKFLOW_STARTERS_REPO_ROOT` to
re-home the starters dir for testability.
- **MEDIUM 1 — YAML parsing.** Replaced the fragile regex-over-YAML
options scanner with a real `yaml.parse()` + typed navigation down
`on.workflow_dispatch.inputs.service.options`. ALL_SERVICES stays
regex-scanned (embedded JSON in a bash heredoc, with `${{ ... }}`
interpolations that aren't valid JSON pre-execution), but the
surrounding step is now located via YAML.
- **MEDIUM 2 — Slack list truncation.** Replaced `cut -c1-200` with a
`truncate_csv` helper that drops whole comma-separated entries until
under budget and appends `…` when truncated. No more
`starter-claude-sdk-pyth` mid-slug corruption.
- **MEDIUM 3 — template/ exclusion cross-references.** Both the TS
validator's `EXCLUDED_DIRS` and `showcase_smoke-monitor.yml`'s `[
"$slug" = "template" ] && continue` now carry `# keep in sync with ...`
comments pointing at each other.
- **NIT 2 — entry-check simplification.** Dropped the
belt-and-suspenders `import.meta.url === \`file://${argv[1]}\`` branch;
kept only the canonical `fileURLToPath(import.meta.url)` form.
- **NIT 3 — jq pipeline collapse.** Single-pass `.jobs[]? | select |
"\(...)"` replaces the three-pass `map | map | .[]` chain in the notify
step.
## Files changed
- `.github/workflows/showcase_deploy.yml` — warning → error + exit 1 on
unhealthy deploy; `truncate_csv` replaces `cut -c1-200` (3 sites);
single-pass jq pipeline in notify step.
- `.github/workflows/showcase_smoke-monitor.yml` — filesystem-driven
`SERVICES=(...)`, starter-count guard, `nullglob` loop, stderr-isolated
`gh api` call, cross-reference comment.
- `.github/workflows/showcase_validate.yml` — wires
`validate-workflow-starters` into CI.
- `showcase/scripts/validate-workflow-starters.ts` — YAML-aware presence
checks; env-var override homes both starters dir and workflow path.
- `showcase/scripts/tsconfig.json` — scripts-local tsconfig for LSP type
resolution.
- `showcase/scripts/__tests__/validate-workflow-starters.test.ts` — 12
specs covering the full matrix of drift scenarios.
## Test plan
- [ ] Next scheduled `showcase_smoke-monitor` run includes starter
services in its drift scan.
- [ ] A deliberately-unhealthy deploy (simulate by pointing health_path
at a 404) fails the job and fires the Slack alert.
- [ ] `showcase_validate` CI job runs `validate-workflow-starters` and
`npx vitest run scripts/__tests__/validate-workflow-starters.test.ts`
green.
Two CI fixes for the showcase_validate python-unit-tests job on PR #4083:
1. Install pytest-asyncio (and pytest-mock) in the shared "minimal test
deps" step. langroid/tests/python/test_agui_adapter.py has 16 tests
annotated with @pytest.mark.asyncio; without the plugin, pytest
reports "async def functions are not natively supported" and skips
them. Locally the user's venv has it; CI did not.
2. Surgically skip the strands package on Python 3.10.
ag_ui_strands==0.1.0 declares requires-python >=3.12,<3.14, so
pip install fails on 3.10 before pytest runs. Prefer the surgical
skip over dropping 3.10 from the matrix so the typing_extensions
NotRequired fallback path in aimock_toggle.py keeps getting
exercised.
Python unit-test workflow previously installed only pytest +
typing_extensions, assuming stdlib-only tests. Tests now import runtime
deps (openai, google.genai, httpx, opentelemetry, etc.) at module load
time, causing ModuleNotFoundError at pytest collection before
conftest-based stub finders run.
Loop each showcase/packages/*/ with a tests/python/ dir, pip install the
package's requirements.txt (if present) before pytest. Re-enable pip
cache keyed on those requirements files. Matches real-runtime parity.
Refs blocker 2 on PR #4083 (fix/showcase-smoke-failures).
The starter slug list is duplicated across at least three places:
- .github/workflows/showcase_deploy.yml workflow_dispatch options
- .github/workflows/showcase_deploy.yml ALL_SERVICES matrix entries
- .github/workflows/showcase_smoke-monitor.yml SERVICES bash array
Adding a new starter under `showcase/starters/` but forgetting any of
these leaves the service deployable in theory but invisible to the
dispatch UI and/or drift detection — exactly the failure mode this PR
is trying to close.
Add `showcase/scripts/validate-workflow-starters.ts`. It enumerates
every directory under `showcase/starters/` (excluding `template/`)
and confirms `starter-<slug>` is present in each of the three
workflow locations, emitting a precise "missing from: <source>"
diagnostic per gap.
Wire it into showcase_validate.yml right after `validate-parity` so
it gates every PR + main push touching `showcase/**` or the relevant
workflow files. Also extend that workflow's `on.paths` filter to
include `showcase_deploy.yml` and `showcase_smoke-monitor.yml` so
edits to those files trigger the parity check.
Bare "❌ Showcase validate: failed | View run" forces a click-through to
triage — the signal that would make the alert actionable is one click
away. Per the oss-alerts policy, red alerts must carry triage-ready
detail in the payload itself.
Adds a pre-notify step that (on failure(), push, webhook set):
- resolves the current job's ID via the runs/{id}/jobs API (matches by
job name; falls back to "first job with a failed step" for rename
drift tolerance)
- extracts the first failed step name from the same jobs response
- pulls the failed-step log via `gh run view --log-failed --job=<id>`,
strips TSV prefix + leading timestamp + ANSI codes, skips runner
header noise (##[group], shell:, env:, Run), then grabs the first
line matching [FAIL]/[ERROR]/Error:/::error and truncates to 300
chars
- emits failed_step and error_excerpt to $GITHUB_ENV via heredoc
delimiters (safe for values containing = or newlines)
- uses set +e and explicit exit 0 so extraction glitches never block
the notify step — fallbacks yield "unknown step" / "see workflow
run for details"
Updates the Slack payload to interpolate both values via toJSON(format())
so dynamic content is JSON-encoded defensively (matches the pattern from
showcase_drift-report.yml). Preserves the ❌ prefix, the "View run"
link, and keeps total length well under the 800-char budget.
Companion to #4065, which quieted routine per-run success posts; this
PR makes the remaining (failure-only) posts actionable at a glance.
Against run 24598654559 (tonight's validate-parity regression), the
extracted excerpt is:
[FAIL] ag2: demo 'hitl-in-chat' declared in manifest but no
src/app/demos/hitl-in-chat/ directory
which is exactly the triage signal that was missing from the bare
"failed" alert.
- Add `--only-binary :all:` to the pip install in showcase_validate.yml's
python-unit-tests job, mirroring the hardening already in the aimock-e2e
sibling workflow. This job runs on EVERY PR automatically without a
trusted-commenter gate, so the broader exposure deserves the same
wheel-only install defense against PR-controlled setup.py hooks.
- Drop the unconditional per-package requirements.txt install. Today's
tests are stdlib-only (+ typing_extensions fallback path); installing
the full CrewAI graph on every PR run was a ~30-60s tax with no benefit.
Install only pytest + typing_extensions. Comment documents the upgrade
path for future tests that need package runtime deps.
- Drop the now-unused pip cache on setup-python (keyed on a never-used
requirements.txt would be pure overhead).
showcase_aimock-e2e.yml:
- workflow_dispatch slug is now a choice-type enum restricted to Python
packages that ship aimock_toggle.py; a TS (mastra) or Java (spring-ai)
slug would have skipped the Python agent start step and then failed with
a misleading Playwright timeout. For comment-trigger paths, the pkg-type
step short-circuits with a clear ::error:: when the slug does not ship
aimock_toggle.py.
- Slug parsing replaced PCRE grep -oP '\\K\\S+' with a POSIX-safe
grep -oE + sed pipeline so BSD/Alpine grep works too (future-proof
against runner image changes).
- aimock pinned to @copilotkit/aimock@^1.14.3 (was @latest) with
--ignore-scripts; unpinned @latest let a bad aimock publish silently
poison CI for everyone.
- pnpm install now runs with --ignore-scripts — trusted commenter triggers
/test-aimock on untrusted PR content, so postinstall scripts must not
run on the runner.
- OPENAI_BASE_URL is no longer pre-set for Python agent start on packages
that ship aimock_toggle.py — forcing the toggle itself to do the
redirection proves the toggle works rather than masking it with env
already set.
- Dead AIMOCK_URL export removed from the Next.js dev-server step (the
runtime only reads OPENAI_BASE_URL).
- Python agent health-check extended 60s -> 90s (45 iter * 2s) for CrewAI
cold imports.
- actions/setup-python caches pip keyed on requirements.txt.
- github-script Post-result step passes slug + job status via env rather
than ${{ }} interpolation — even though the slug is already whitelisted,
the env pattern is the defensive default for dynamic values.
showcase_validate.yml:
- python-unit-tests is now a matrix on Python 3.10 + 3.12 (fail-fast
disabled) so the typing_extensions fallback branch gets CI coverage;
previously only 3.12 ran.
- pip install per package is a HARD FAIL (was ::warning:: + continue);
hiding broken requirements.txt behind a warning let a package ship green
with unresolvable runtime deps. A missing requirements.txt is still
handled gracefully via the -f guard.
- pip caching added via actions/setup-python.
The `secrets.*` context is not a valid named-value inside step-level
`if:` expressions on push events — GitHub Actions rejects it at
workflow parse time with "Unrecognized named-value: 'secrets'", which
caused both showcase_validate.yml and showcase_drift-report.yml to
fail at startup with zero jobs spawned after #4018 + #4060 merged.
Hoist the webhook into a job-level `env: SLACK_WEBHOOK` and reference
`env.SLACK_WEBHOOK` in every step-level `if:`. The `with: webhook:`
keys still use `secrets.*` directly (valid in that context).
PR CI didn't catch this because pull_request events parse if:
expressions less strictly than push events.
showcase_validate.yml and showcase_drift-report.yml were flipped to
runs-on: depot-ubuntu-24.04-4 in #4018 but inherited the repo's
default least-privilege permissions (contents: read only). Depot
runner provisioning uses OIDC and requires id-token: write, so the
jobs failed to spawn on the first main-branch push — matching the
pattern already used by showcase_deploy.yml's Depot job.
Two GitHub Actions workflows that consume the three validators in
showcase/scripts/ and surface drift to CI and Slack.
showcase_validate.yml — runs on pull_request and push-to-main:
- Enforces per-package e2e spec count against
fail-baseline.json.baselineDemoCount, with find failures
aggregated (not exit-on-first) so multiple package issues surface
in one run. Preserves find's exit status by capturing via command
substitution rather than process substitution (mapfile does not
propagate exit codes through < <(...)).
- Runs validate-pins.ts with both count + content-hash ratchet:
the sorted-uniqued [FAIL] set is SHA-256 hashed so a
count-preserving set change ('one fail healed, another regressed')
is still flagged.
- Separates stdout from stderr before hashing so progress chatter
cannot corrupt the ratchet hash.
- Preserves validator exit codes distinctly (1 drift, 2 internal,
3 unreadable, 4+ future) so downstream consumers can distinguish
crashes from legitimate drift.
- set -euo pipefail throughout, with { grep || true; } scoped to
tolerate grep no-match without masking producer failures.
- Slack notifications gated on push events and secret presence;
payload values wrapped via toJSON(format(...)) for injection
safety.
showcase_drift-report.yml — weekly Monday 10:00 UTC + dispatch:
- Same ratchet + hash logic applied in report mode: computes
set_status (OK / SET DRIFTED / COUNT DRIFTED) and posts to Slack.
- Mirrors validate.yml's pipefail + grep-scope + toJSON discipline.
- Add --validate-on-load to all aimock invocations (4 workflows/scripts
+ 13 integration docker-compose files)
- Replace hardcoded 2-file fixture list with dynamic discovery across
showcase/, examples/integrations/*/, scripts/doc-tests/ (16 fixtures)
- Add sanity check to prevent silent zero-test pass when discovery fails
- Extend showcase_validate.yml path filter to trigger on
examples/integrations/**/fixtures/** and scripts/doc-tests/fixtures/**
- Import and use ValidationResult type for callback parameters
- Fix scripts/doc-tests/fixtures/default.json to use { fixtures: [...] }
envelope shape
- Node 20 → 22 in all Dockerfiles + CI workflows
- CI copies shared_python, shared_frontend/src, shared_typescript/tools
- tsconfig paths for @copilotkit/showcase-shared
- .gitignore for CI artifacts
Jordan Ritter