Three merges landed on main within 34 seconds on 2026-07-26. The build
workflow has no concurrency group by design, so all three ran at once and
raced to push the same `:latest` tags. The NEWEST commit's build finished
FIRST, so the two older builds overwrote it:
run 30190815370 7b28934387 (#6162) 06:18:05 -> 06:29:44
run 30190823203 59f275eedc (#6161) 06:18:21 -> 06:29:44
run 30190831480 db75a04837 (#6158) 06:18:39 -> 06:29:13 <- NEWEST
Per-service, the older commit beat the newer one on every shared slot:
shell-dashboard (+1s), showcase-harness (+4s), shell (+11s), shell-dojo
(+16s). All three runs reported `success`. Staging served pre-#6158 code
while CI, the redeploy gate and deploy verification all looked clean.
A concurrency group is the wrong fix. `detect-changes` builds a per-push,
path-filtered matrix, so concurrent runs build overlapping but NON-IDENTICAL
service sets -- 7b28934387 was the only run building ag2, agno, langroid,
spring-ai, strands and 10 others. `cancel-in-progress: true` would have
dropped those entirely, trading a stale-image bug for a never-shipped bug.
`cancel-in-progress: false` is no better: GitHub keeps at most one pending
run per group and cancels any previously-pending one.
Instead make the mutable pointer monotonic. The build step now pushes only
the immutable `:<sha>` tag plus an `org.opencontainers.image.revision`
label. A new guard then advances `:latest` unless the tag already points at
a DESCENDANT of the commit being built, which is exactly the regression
case. Every ambiguous state (no tag, unlabelled legacy image, unreachable
API, diverged history) advances -- a stuck `:latest` is the failure mode
being fixed, so the guard declines only on positive proof of regression.
The guard runs in redeploy-staging / redeploy-staging-starters, immediately
before the Railway pull that consumes `:latest`, over the same
matrix-intersect-build-success set that decides what gets redeployed. That
placement keeps the read->retag window minimal and avoids an unpinned
`npx tsx` fetch on ~50 parallel build slots that have no Node.
Also adds showcase_build.yml to showcase_validate.yml's trigger paths: the
new test asserts against that file's live text, and without the path a PR
re-adding `:latest` to the push step would never run the test that catches it.
Prod is unaffected -- verify-railway-image-refs.ts already pins prod to
`@sha256:<digest>`; only staging consumes the mutable tag.
PR #6163 converted showcase/integrations/*/public/demo-files/* to Git LFS.
showcase_validate.yml's python-unit-tests job checks out without LFS, so the
working tree holds 129-byte pointer text instead of the real assets. The new
ms-agent-python multimodal test is the first Python test to read those bytes:
pypdf reads the pointer, extracts nothing, and the test fails with
"real pypdf text extraction produced nothing".
Fetch only the demo-file assets (40 objects, ~320 KB) rather than setting
lfs: true, which would pull all ~248 tracked objects (~475 MB, including
13-26 MB README gifs) into a 2-4 minute job under a 10 minute cap. A job that
hits timeout-minutes reports 'cancelled' -- neither success nor failure -- and
silently suppresses alerting, so the timeout margin is worth protecting.
The repo is public, so LFS downloads resolve anonymously and the pull works
with persist-credentials: false. A post-pull %PDF- header check fails the step
immediately if the pull ever no-ops, instead of surfacing minutes later as a
misleading pytest assertion.
Restore the 12 Python integration tools/ dirs to symlinks into
shared/python/tools. They had eroded to real, drifting copies via an
accidental stage_shared() leak (commit 534cd1efa7) — the structural root
cause of showcase divergence bugs. Symlinking re-establishes the single
source of truth; content is identical to shared (only render_a2ui naming
and the shared roll_dice/sanitize additions are adopted).
Add showcase/AGENTS.md documenting the 4 iron rules and the single-source
symlink mechanism, plus a validate-shared-symlinks CI guard (shrink-only
baseline) that fails on any NEW erosion.
The boot-smoke previously failed only when the thrown error carried a
module-RESOLUTION code (ERR_MODULE_NOT_FOUND + siblings, walked through
the cause chain / AggregateError members) and passed everything else.
That defaults-to-pass on module-EVALUATION crashes — a top-level throw,
an await-rejection, a bad named binding, or a SyntaxError — none of which
carry a resolution code, so a real boot-crashing regression of that class
would ship green.
The smoke runs `node -e "import('./dist/orchestrator.js')"` with
process.argv[1] UNSET, so bootFleet() (the env/PocketBase validation that
legitimately throws) never runs — only the module graph is linked and
evaluated. A clean build therefore loads with no thrown error, so ANY
error thrown by import() here is a boot regression and must fail the gate.
Now: any rejection -> BOOT_FAIL / exit 1 (resolution AND
evaluation/link/binding/syntax/top-level-throw). Successful load ->
BOOT_OK / exit 0. The collectErrorCodes cause/AggregateError walk is
retained ONLY to label the failure ("module-resolution failure" vs
"boot/evaluation failure") — both exit 1, richer diagnostics preserved.
Kept process.exit(0) on success, the timeout 120s wrapper, and
timeout-minutes: 5 (a hang still fails).
No-false-red proof: built the real harness dist and ran the strict guard
against the real dist/orchestrator.js under node -e (argv[1] unset) —
BOOT_OK, exit 0, ~0.28s, no hang, confirming a clean graph loads without
throwing and the strict guard does not false-red real CI.
The boot-smoke gate classified pass/fail using only the top-level `e.code`.
A module-resolution error that arrives WRAPPED — nested in `e.cause`
(possibly a chain), bundled inside an `AggregateError` (`e.errors[]`), or
rethrown without preserving `.code` at the top — showed no code to the
`MODULE_RESOLUTION_CODES.has(e.code)` check and was misclassified as
BOOT_OK, defeating the gate.
Add a `collectErrorCodes` helper that gathers every code reachable from
the thrown error: the error itself, its cause chain (recursively), and any
AggregateError members (recursively), with a depth cap to bound cause
cycles. If ANY collected code is a module-resolution code -> BOOT_FAIL /
exit 1. Purely additive to the FAIL set: direct top-level codes still
redden, and a benign non-resolution runtime error (e.g. the
`HARNESS_ROLE must be set` guard, which carries no such code anywhere)
still passes as BOOT_OK / exit 0. The `timeout 120s` wrapper,
`timeout-minutes: 5`, and success/expected-error `process.exit(0)` are
unchanged.
Local red-green (classifier extracted to a temp file, driven against
synthetic errors):
- RED (top-level-only): wrapped cause -> BOOT_OK exit 0 (swallowed);
AggregateError member -> BOOT_OK exit 0 (swallowed).
- GREEN (hardened): wrapped -> exit 1; aggregate -> exit 1; direct
ERR_MODULE_NOT_FOUND -> still exit 1; benign ERR_INVALID_ARG_TYPE and
HARNESS_ROLE error -> BOOT_OK exit 0; real built dist/orchestrator.js ->
BOOT_OK exit 0 in <200ms (prompt exit, no hang).
The boot-smoke step only treated ERR_MODULE_NOT_FOUND as failure, so other
module-resolution regressions (ERR_UNSUPPORTED_DIR_IMPORT,
ERR_PACKAGE_PATH_NOT_EXPORTED, ERR_UNKNOWN_FILE_EXTENSION,
ERR_INVALID_MODULE_SPECIFIER) were swallowed as BOOT_OK/exit 0 — the very
class of bug this gate exists to catch could slip through. It also had no
process.exit(0) on the success/expected-error paths and no bounded timeout,
so a future open handle at import time could hang node -e to the job's
25-minute ceiling.
- Broaden the failure condition to a MODULE_RESOLUTION_CODES set (any of the
five codes => BOOT_FAIL, exit 1). Non-module-resolution runtime errors
(e.g. the HARNESS_ROLE env guard, no such code) stay BOOT_OK/exit 0.
- Add explicit process.exit(0) on both the success and expected-error paths.
- Wrap the node invocation in `timeout 120s` (non-zero on timeout => step
fails) and add step-level timeout-minutes: 5.
Red-green proof (extracted guard logic vs synthetic modules): current logic
passes ERR_UNSUPPORTED_DIR_IMPORT / ERR_PACKAGE_PATH_NOT_EXPORTED at exit 0
(RED gap); hardened logic fails all five codes at exit 1, keeps benign
runtime error at exit 0, and against the real built dist/orchestrator.js
reports BOOT_OK and exits promptly (779ms, no hang).
CI missed the extensionless-import regression because tsc (bundler
resolution), vitest, and tsx all resolve extensionless relative
specifiers fine — no existing step ever ran the real node dist module
graph, which is what the container actually does at boot.
Add a boot-smoke to the Validate Showcase job (already gated on
showcase/harness/**): after building the harness dist, load
dist/orchestrator.js via a node import() and fail hard on
ERR_MODULE_NOT_FOUND. A later runtime error from missing env/PocketBase
is expected and passes — only a module-resolution failure reddens the
build. Verified red-green: the guard exits 1 on the pre-fix
extensionless imports and 0 once the .js extensions are added.
Round-3 CR fixes for the LGT persistence-disable preload.
HIGH-1: after installing the fs-write patches, import the node:fs/promises
namespace and assert each patched member is identity-equal to the installed
function; throw (fail boot) naming any mismatched member. Catches the
load-order case where fs/promises was linked before the reassignment and the
namespace snapshotted the original fn (silent bypass -> disk-growth recurrence).
HIGH-2: make the real-package behavioral test non-skippable under
LGT_REQUIRE_BEHAVIORAL=1 (missing runtime fails, not skips), and wire the
python-unit-tests job to set up Node, npm install the agent deps, and run the
langgraph-typescript pytest with that flag so a green check proves interception.
LOW: tolerant writer-shape guard regex (quote/whitespace/alias agnostic; still
trips on a named-import switch); read-only open/openSync reject ENOENT for
suppressed paths (write-intent still no-ops); mkdir recursive returns the
topmost-created dir per the real fs contract.
Adds a HIGH-1 guard-fires regression test.
The shell-script-tests job installs bats via apt. GitHub's ubuntu-latest runner
image preconfigures third-party apt repos (Microsoft / azure-cli) for
preinstalled tooling this job never uses. When one of those repos serves invalid
release metadata, `apt-get update` exits non-zero and `bash -e` aborts the step
before bats installs — even though bats comes from Ubuntu's own `universe` repo,
which is unaffected.
This job only needs Ubuntu packages, so remove those unused third-party repos
before `apt-get update`.
Lever 1 of the promote-reliability hardening plan. The showcase deploy
model is staging=mutable :latest (continuously rebuilt), prod=immutable
@sha256: (advances only on explicit promote), so a prod column can
silently fall BEHIND a green staging — drift today is only noticed by
eyeballing a dead column. This adds proactive, automatic detection.
- bin/railway reconcile-prod: for every prod-eligible (probe.prod==true)
service, compares the prod SERVING digest (LintProd snapshot path) vs
the staging RUNNING digest (reuses PromoteCommand#staging_running_digest).
Classifies green/stale/gray, prints a table + summary, exits 1 iff any
stale. --json for machine output. Read-only: no promotes/mutations.
- scripts/reconcile-prod-gate.sh: wrapper mirroring lint-prod-gate.sh —
surfaces the table to the GH step summary, captures JSON for the Slack
builder, propagates the exit-code verdict.
- .github/workflows/showcase_reconcile.yml: daily cron + workflow_dispatch;
runs the gate; on stale services posts the stale-column list to
#oss-alerts (SLACK_WEBHOOK_OSS_ALERTS) via the fromJSON('"\n"') idiom.
- Tests: Ruby minitest (classification + exit-code, RED-anchored on a
drift-blind classifier) and a bats gate test. Wired the gate script
into the showcase_validate.yml shellcheck list.
Post-promote convergence verification is deferred to a fast-follow.
When a promote fails, the succeeded-service set is empty, so verify-prod
hits its skip branch (`exit 0`). The GitHub job result is therefore
`success`, and the notify step rendered `verify-prod=success` in the
#oss-alerts Slack message — a misleading green, since prod was never
probed.
verify-prod now exports a `status` output: `success` after a real probe
passes, `skipped` on the empty-CSV skip. notify reads that output (via
the new bats-tested verify-prod-display.sh) instead of the raw job
result, so the Slack line accurately reads `verify-prod=skipped` vs
`success` vs `failure`. A genuine probe failure / contract violation
exits non-zero (job result `failure`, status never written), and the
display falls back to the job result. Slack formatting is unchanged.
Extracts the display mapping into showcase/scripts/verify-prod-display.sh
(mirroring promote-fleet.sh) with red-green bats coverage, and adds it to
the showcase_validate.yml shellcheck step.
GitHub Actions expression string literals don't interpret `\n`, so
`toJSON(format('...\n...'))` emits literal `\\n` and Slack renders the
two characters backslash-n instead of a line break. Inject real newlines
via a `fromJSON('"\n"')` placeholder, matching the starter-smoke fix.
Fixes the "all builds failed" and "Showcase Build Failed" alerts in
showcase_build.yml and the multi-line "showcase_validate failed" alert
in showcase_validate.yml. showcase_promote.yml already used the
fromJSON placeholder; the single-line validate alert has no newlines
and was left untouched.
the per-service promote loop ran under `set -euo pipefail`, so the first failing
service aborted the whole `all` fleet promote; extracted to promote-fleet.sh
which attempts every service, accumulates succeeded/failed sets, exits non-zero
only after attempting all, and exports succeeded_csv. verify-prod now runs
`if: !cancelled()` and scopes --services to the succeeded set; the staging
precondition is advisory (promote runs even when it reports red — bin/railway
enforces staging-green per-service); notify success keys on PROMOTE && PROD.
Adds a shell-script-tests CI job (bats + shellcheck) and input-validation
hardening (fail-loud on empty/all-empty CSV, RAILWAY_BIN check, whitespace trim).
Add showcase/scripts/sync-promote-service-options.ts: generates the
promote workflow's service `choice` options from the SSOT
(railway-envs.ts), spliced between BEGIN/END markers in
showcase_promote.yml. Fail-loud throughout — every emitted token must
resolve to exactly one service under the resolve-step predicate
(name|dispatchName match AND probe.prod), tokens are YAML-safe, args are
strict (a typo'd flag cannot trigger a destructive write), and markers
are validated before any rewrite.
Wire it into a lefthook pre-commit hook (regenerate + restage; set -e so
a failed regen blocks the commit) and an advisory (never-failing) drift
check in showcase_validate.yml. Vitest coverage for ordering, exclusion,
collision/ambiguity guards, marker errors, exit codes, idempotency, and
the import-side-effect guard.
The new no-webhook log step interpolated github.ref directly into a shell
echo, which zizmor flags as template-injection (HIGH). Bind it to a step
env var REF and reference $REF in the script instead. Message text is
unchanged; no Co-Authored-By.
Collapse the promote workflow to an input-agnostic concurrency group so promotes can't race the
same Railway service; add #oss-alerts failure notifications to the build and validate workflows
(build via extended needs, validate via a new workflow-level notify job).
Comprehensive CI/CD security hardening pass over all 33 workflows.
Action pinning
- Every `uses:` is now pinned to a 40-char commit SHA with a `# vX.Y.Z`
comment alongside (167 occurrences resolved). Tag-style refs like `@v4`
are mutable and have been used in past supply-chain attacks (e.g.
tj-actions/changed-files in March 2025) to repoint widely-used actions
to malicious commits.
- Removed redundant `version: "10.13.1"` hardcodes from `pnpm/action-setup`
call sites so the action inherits from package.json `packageManager`
(one source of truth).
Automated maintenance
- Added `.github/dependabot.yml` for the `github-actions` ecosystem so
SHA pins stay current. Without this, pins go stale fast and new
upstream advisories never reach us. Minor/patch bumps are grouped;
major bumps stay separate so they get a real review.
Static analysis
- Added `.github/zizmor.yml` configuration and
`.github/workflows/security_zizmor.yml` (blocking on PR, runs on push
to main, weekly schedule for advisory drift). zizmor catches the
well-known classes of Actions footguns: template injection from
untrusted input, dangerous triggers, unpinned uses, excessive token
scopes, secret exfil patterns.
- All 28 high-severity and 54 medium-severity findings from the baseline
scan are remediated. Each suppression in zizmor.yml carries a
per-finding justification comment so future maintainers can audit the
trust assumption.
Workflow hardening (from zizmor + manual audit)
- Added `persist-credentials: false` to every `actions/checkout` except
the 7 workflows that legitimately push back to the repo via the
workflow token (release tagging, auto-formatting, docs-sync, registry
updates). Each retained credential persistence carries a
`persist-credentials required: ...` comment explaining the call site.
- Routed every attacker-controllable expansion (`github.head_ref`,
`github.event.pull_request.head.repo.full_name`, `inputs.*`,
step outputs) through `env:` and referenced as quoted shell variables.
Eliminates 17 template-injection vectors in fork-PR-reachable
workflows.
- Added per-job `permissions:` blocks across 14 workflows; demoted
broad workflow-level `id-token: write` to the specific Depot-runner
jobs that need it; narrowed `pull-requests: write` /
`actions: write` to the jobs that actually call those APIs.
Audit-driven fixes
- `publish-release.yml` build job: dropped `token:` and added
`persist-credentials: false`. The subsequent `Upload workspace` step
was packing `.git/config` (with the persisted GITHUB_TOKEN) into a
1-day-retention artifact downloadable by anyone with `actions:read`.
- `auto_merge_showcases.yml`: team-membership check now authorizes on
the PR AUTHOR (`pull_request.user.login`), never `context.actor` —
the actor is whoever triggered the latest event, so a team member
synchronizing or reopening an outsider's PR would otherwise
green-light auto-merge of code they didn't author.
- `static_quality.yml`: pinned ruff to a specific version so a
compromised release can't land on the next PR run with the
persisted-credentials write token in the format job.
- `showcase_capture-previews.yml`: switched the args-string construction
to a bash array so a slug or demo value containing whitespace or shell
metacharacters stays a single argument rather than being re-tokenized
by the shell.
Adds a new step that runs validate-fixture-tool-surface.ts on every PR
and push to main. Sits alongside the existing validate-parity /
validate-workflow-starters / validate-pins steps and follows the same
pnpm-exec-tsx pattern.
Without this, the drift validator only runs locally or via the
vitest suite (which only catches bugs in the validator itself, not
drift in the real fixture/demo state). The CLI invocation against the
committed tree is what would have caught the 2026-04-22 regression
before it reached prod.
## Summary
Two related defects in the showcase deploy pipeline let stale images sit
live on Railway while Slack stayed green. This PR fixes both.
### Defect 1 — Drift detector skipped all starter services
`.github/workflows/showcase_smoke-monitor.yml` listed only 19
**package** slugs in its `SERVICES=(...)` array (ag2, mastra,
llamaindex, ...). Zero **starter** slugs. As a result:
- GHCR `showcase-starter-<svc>` tags were never checked for drift.
- `gh workflow run showcase_deploy.yml -f service=starter-*` was never
auto-dispatched.
- Starter services could run with weeks-old images and no alert would
fire.
`showcase_deploy.yml` already supports `starter-*` dispatch names and
already calls `serviceInstanceRedeploy` for any service with a
`railway_id`, so no change is required there. The fix is extending
`SERVICES=(...)` to include all 17 starter slugs via a
sparse-checkout-driven filesystem enumeration (no more literal
duplication between workflow and `showcase/starters/`).
### Defect 2 — Silent deploy failures reported green
`.github/workflows/showcase_deploy.yml` emitted `::warning::` and exited
0 when a service never returned 200 on its health path within 360s. The
legacy justification (`# Don't fail — sleep-on-idle services take time
to wake`) no longer applies: Railway is on the Pro tier with no
sleep-on-idle, so a 6-minute failure to become healthy is a real
failure. Changed to `::error::` + `exit 1`.
## Round 2 fixes
Round 2 CR raised six findings against the original smoke-monitor +
validator changes. All fixed in this PR:
- **BLOCKING 1/2 — smoke-monitor guard.** Replaced the magic `-eq 19`
sentinel with `grep -c '^starter-'` so adds/removes to the literal
non-starter list can't silently disable the guard. Added `shopt -s
nullglob` around the `showcase/starters/*/` loop so an empty starters
tree no longer corrupts `SERVICES` with a `starter-*` literal.
- **BLOCKING 3 — GHCR stderr isolation.** Dropped `2>&1` on the `gh api
-i` call; captured stderr to a temp file and surfaced it only when `gh`
returns a non-zero RC with no HTTP status. Auth / rate-limit / network
noise can no longer splice into the HTTP header block and poison
`HTTP_STATUS` / `API_BODY` parsing.
- **BLOCKING 4 — validator tests.** Added
`showcase/scripts/__tests__/validate-workflow-starters.test.ts` (12
specs): happy path, missing-from-options-only, missing-from-matrix-only,
missing-from-both, empty starters dir (exit 3), template/ excluded,
substring-spoof (starter-ag2 vs starter-ag2-extended), missing workflow
file (exit 3). Also extended `VALIDATE_WORKFLOW_STARTERS_REPO_ROOT` to
re-home the starters dir for testability.
- **MEDIUM 1 — YAML parsing.** Replaced the fragile regex-over-YAML
options scanner with a real `yaml.parse()` + typed navigation down
`on.workflow_dispatch.inputs.service.options`. ALL_SERVICES stays
regex-scanned (embedded JSON in a bash heredoc, with `${{ ... }}`
interpolations that aren't valid JSON pre-execution), but the
surrounding step is now located via YAML.
- **MEDIUM 2 — Slack list truncation.** Replaced `cut -c1-200` with a
`truncate_csv` helper that drops whole comma-separated entries until
under budget and appends `…` when truncated. No more
`starter-claude-sdk-pyth` mid-slug corruption.
- **MEDIUM 3 — template/ exclusion cross-references.** Both the TS
validator's `EXCLUDED_DIRS` and `showcase_smoke-monitor.yml`'s `[
"$slug" = "template" ] && continue` now carry `# keep in sync with ...`
comments pointing at each other.
- **NIT 2 — entry-check simplification.** Dropped the
belt-and-suspenders `import.meta.url === \`file://${argv[1]}\`` branch;
kept only the canonical `fileURLToPath(import.meta.url)` form.
- **NIT 3 — jq pipeline collapse.** Single-pass `.jobs[]? | select |
"\(...)"` replaces the three-pass `map | map | .[]` chain in the notify
step.
## Files changed
- `.github/workflows/showcase_deploy.yml` — warning → error + exit 1 on
unhealthy deploy; `truncate_csv` replaces `cut -c1-200` (3 sites);
single-pass jq pipeline in notify step.
- `.github/workflows/showcase_smoke-monitor.yml` — filesystem-driven
`SERVICES=(...)`, starter-count guard, `nullglob` loop, stderr-isolated
`gh api` call, cross-reference comment.
- `.github/workflows/showcase_validate.yml` — wires
`validate-workflow-starters` into CI.
- `showcase/scripts/validate-workflow-starters.ts` — YAML-aware presence
checks; env-var override homes both starters dir and workflow path.
- `showcase/scripts/tsconfig.json` — scripts-local tsconfig for LSP type
resolution.
- `showcase/scripts/__tests__/validate-workflow-starters.test.ts` — 12
specs covering the full matrix of drift scenarios.
## Test plan
- [ ] Next scheduled `showcase_smoke-monitor` run includes starter
services in its drift scan.
- [ ] A deliberately-unhealthy deploy (simulate by pointing health_path
at a 404) fails the job and fires the Slack alert.
- [ ] `showcase_validate` CI job runs `validate-workflow-starters` and
`npx vitest run scripts/__tests__/validate-workflow-starters.test.ts`
green.
Two CI fixes for the showcase_validate python-unit-tests job on PR #4083:
1. Install pytest-asyncio (and pytest-mock) in the shared "minimal test
deps" step. langroid/tests/python/test_agui_adapter.py has 16 tests
annotated with @pytest.mark.asyncio; without the plugin, pytest
reports "async def functions are not natively supported" and skips
them. Locally the user's venv has it; CI did not.
2. Surgically skip the strands package on Python 3.10.
ag_ui_strands==0.1.0 declares requires-python >=3.12,<3.14, so
pip install fails on 3.10 before pytest runs. Prefer the surgical
skip over dropping 3.10 from the matrix so the typing_extensions
NotRequired fallback path in aimock_toggle.py keeps getting
exercised.
Python unit-test workflow previously installed only pytest +
typing_extensions, assuming stdlib-only tests. Tests now import runtime
deps (openai, google.genai, httpx, opentelemetry, etc.) at module load
time, causing ModuleNotFoundError at pytest collection before
conftest-based stub finders run.
Loop each showcase/packages/*/ with a tests/python/ dir, pip install the
package's requirements.txt (if present) before pytest. Re-enable pip
cache keyed on those requirements files. Matches real-runtime parity.
Refs blocker 2 on PR #4083 (fix/showcase-smoke-failures).
The starter slug list is duplicated across at least three places:
- .github/workflows/showcase_deploy.yml workflow_dispatch options
- .github/workflows/showcase_deploy.yml ALL_SERVICES matrix entries
- .github/workflows/showcase_smoke-monitor.yml SERVICES bash array
Adding a new starter under `showcase/starters/` but forgetting any of
these leaves the service deployable in theory but invisible to the
dispatch UI and/or drift detection — exactly the failure mode this PR
is trying to close.
Add `showcase/scripts/validate-workflow-starters.ts`. It enumerates
every directory under `showcase/starters/` (excluding `template/`)
and confirms `starter-<slug>` is present in each of the three
workflow locations, emitting a precise "missing from: <source>"
diagnostic per gap.
Wire it into showcase_validate.yml right after `validate-parity` so
it gates every PR + main push touching `showcase/**` or the relevant
workflow files. Also extend that workflow's `on.paths` filter to
include `showcase_deploy.yml` and `showcase_smoke-monitor.yml` so
edits to those files trigger the parity check.
Bare "❌ Showcase validate: failed | View run" forces a click-through to
triage — the signal that would make the alert actionable is one click
away. Per the oss-alerts policy, red alerts must carry triage-ready
detail in the payload itself.
Adds a pre-notify step that (on failure(), push, webhook set):
- resolves the current job's ID via the runs/{id}/jobs API (matches by
job name; falls back to "first job with a failed step" for rename
drift tolerance)
- extracts the first failed step name from the same jobs response
- pulls the failed-step log via `gh run view --log-failed --job=<id>`,
strips TSV prefix + leading timestamp + ANSI codes, skips runner
header noise (##[group], shell:, env:, Run), then grabs the first
line matching [FAIL]/[ERROR]/Error:/::error and truncates to 300
chars
- emits failed_step and error_excerpt to $GITHUB_ENV via heredoc
delimiters (safe for values containing = or newlines)
- uses set +e and explicit exit 0 so extraction glitches never block
the notify step — fallbacks yield "unknown step" / "see workflow
run for details"
Updates the Slack payload to interpolate both values via toJSON(format())
so dynamic content is JSON-encoded defensively (matches the pattern from
showcase_drift-report.yml). Preserves the ❌ prefix, the "View run"
link, and keeps total length well under the 800-char budget.
Companion to #4065, which quieted routine per-run success posts; this
PR makes the remaining (failure-only) posts actionable at a glance.
Against run 24598654559 (tonight's validate-parity regression), the
extracted excerpt is:
[FAIL] ag2: demo 'hitl-in-chat' declared in manifest but no
src/app/demos/hitl-in-chat/ directory
which is exactly the triage signal that was missing from the bare
"failed" alert.
- Add `--only-binary :all:` to the pip install in showcase_validate.yml's
python-unit-tests job, mirroring the hardening already in the aimock-e2e
sibling workflow. This job runs on EVERY PR automatically without a
trusted-commenter gate, so the broader exposure deserves the same
wheel-only install defense against PR-controlled setup.py hooks.
- Drop the unconditional per-package requirements.txt install. Today's
tests are stdlib-only (+ typing_extensions fallback path); installing
the full CrewAI graph on every PR run was a ~30-60s tax with no benefit.
Install only pytest + typing_extensions. Comment documents the upgrade
path for future tests that need package runtime deps.
- Drop the now-unused pip cache on setup-python (keyed on a never-used
requirements.txt would be pure overhead).
showcase_aimock-e2e.yml:
- workflow_dispatch slug is now a choice-type enum restricted to Python
packages that ship aimock_toggle.py; a TS (mastra) or Java (spring-ai)
slug would have skipped the Python agent start step and then failed with
a misleading Playwright timeout. For comment-trigger paths, the pkg-type
step short-circuits with a clear ::error:: when the slug does not ship
aimock_toggle.py.
- Slug parsing replaced PCRE grep -oP '\\K\\S+' with a POSIX-safe
grep -oE + sed pipeline so BSD/Alpine grep works too (future-proof
against runner image changes).
- aimock pinned to @copilotkit/aimock@^1.14.3 (was @latest) with
--ignore-scripts; unpinned @latest let a bad aimock publish silently
poison CI for everyone.
- pnpm install now runs with --ignore-scripts — trusted commenter triggers
/test-aimock on untrusted PR content, so postinstall scripts must not
run on the runner.
- OPENAI_BASE_URL is no longer pre-set for Python agent start on packages
that ship aimock_toggle.py — forcing the toggle itself to do the
redirection proves the toggle works rather than masking it with env
already set.
- Dead AIMOCK_URL export removed from the Next.js dev-server step (the
runtime only reads OPENAI_BASE_URL).
- Python agent health-check extended 60s -> 90s (45 iter * 2s) for CrewAI
cold imports.
- actions/setup-python caches pip keyed on requirements.txt.
- github-script Post-result step passes slug + job status via env rather
than ${{ }} interpolation — even though the slug is already whitelisted,
the env pattern is the defensive default for dynamic values.
showcase_validate.yml:
- python-unit-tests is now a matrix on Python 3.10 + 3.12 (fail-fast
disabled) so the typing_extensions fallback branch gets CI coverage;
previously only 3.12 ran.
- pip install per package is a HARD FAIL (was ::warning:: + continue);
hiding broken requirements.txt behind a warning let a package ship green
with unresolvable runtime deps. A missing requirements.txt is still
handled gracefully via the -f guard.
- pip caching added via actions/setup-python.
The `secrets.*` context is not a valid named-value inside step-level
`if:` expressions on push events — GitHub Actions rejects it at
workflow parse time with "Unrecognized named-value: 'secrets'", which
caused both showcase_validate.yml and showcase_drift-report.yml to
fail at startup with zero jobs spawned after #4018 + #4060 merged.
Hoist the webhook into a job-level `env: SLACK_WEBHOOK` and reference
`env.SLACK_WEBHOOK` in every step-level `if:`. The `with: webhook:`
keys still use `secrets.*` directly (valid in that context).
PR CI didn't catch this because pull_request events parse if:
expressions less strictly than push events.
showcase_validate.yml and showcase_drift-report.yml were flipped to
runs-on: depot-ubuntu-24.04-4 in #4018 but inherited the repo's
default least-privilege permissions (contents: read only). Depot
runner provisioning uses OIDC and requires id-token: write, so the
jobs failed to spawn on the first main-branch push — matching the
pattern already used by showcase_deploy.yml's Depot job.
Two GitHub Actions workflows that consume the three validators in
showcase/scripts/ and surface drift to CI and Slack.
showcase_validate.yml — runs on pull_request and push-to-main:
- Enforces per-package e2e spec count against
fail-baseline.json.baselineDemoCount, with find failures
aggregated (not exit-on-first) so multiple package issues surface
in one run. Preserves find's exit status by capturing via command
substitution rather than process substitution (mapfile does not
propagate exit codes through < <(...)).
- Runs validate-pins.ts with both count + content-hash ratchet:
the sorted-uniqued [FAIL] set is SHA-256 hashed so a
count-preserving set change ('one fail healed, another regressed')
is still flagged.
- Separates stdout from stderr before hashing so progress chatter
cannot corrupt the ratchet hash.
- Preserves validator exit codes distinctly (1 drift, 2 internal,
3 unreadable, 4+ future) so downstream consumers can distinguish
crashes from legitimate drift.
- set -euo pipefail throughout, with { grep || true; } scoped to
tolerate grep no-match without masking producer failures.
- Slack notifications gated on push events and secret presence;
payload values wrapped via toJSON(format(...)) for injection
safety.
showcase_drift-report.yml — weekly Monday 10:00 UTC + dispatch:
- Same ratchet + hash logic applied in report mode: computes
set_status (OK / SET DRIFTED / COUNT DRIFTED) and posts to Slack.
- Mirrors validate.yml's pipefail + grep-scope + toJSON discipline.
- Add --validate-on-load to all aimock invocations (4 workflows/scripts
+ 13 integration docker-compose files)
- Replace hardcoded 2-file fixture list with dynamic discovery across
showcase/, examples/integrations/*/, scripts/doc-tests/ (16 fixtures)
- Add sanity check to prevent silent zero-test pass when discovery fails
- Extend showcase_validate.yml path filter to trigger on
examples/integrations/**/fixtures/** and scripts/doc-tests/fixtures/**
- Import and use ValidationResult type for callback parameters
- Fix scripts/doc-tests/fixtures/default.json to use { fixtures: [...] }
envelope shape
Tyler Slaton