Depot Startup plan (unlimited minutes) for consistent runner quality
across test workflows. Extends the pattern from PR #4018.
Also adds the id-token: write permission required for Depot OIDC
auth, alongside contents: read for least-privilege defaults.
Depot Startup plan (unlimited minutes) for consistent runner quality
across test workflows. Playwright/chromium install is runner-agnostic
on linux. Extends the pattern from PR #4018.
Also adds the id-token: write permission required for Depot OIDC
auth, alongside contents: read for least-privilege defaults.
Depot Startup plan (unlimited minutes) for consistent runner quality
across unit-test workflows. Extends the pattern from PR #4018.
Also adds the id-token: write permission required for Depot OIDC
auth, alongside contents: read for least-privilege defaults.
Depot Startup plan (unlimited minutes) resolves the vitest birpc
onTaskUpdate timeouts on subprocess-heavy suites that standard
ubuntu-latest runners were amplifying. Extends the pattern from
PR #4018 (showcase_validate.yml, showcase_drift-report.yml).
Also adds the id-token: write permission required for Depot OIDC
auth, alongside contents: read for least-privilege defaults.
NIT 3: collapse the per-leg jq pipeline in the notify step. The
previous version went `.jobs // [] | map(filter) | map(project) | .[]`
— three full passes over the list plus a flatten. Rewrite as a single
`.jobs[]? | select | "…"` stream: functionally identical, noticeably
easier to skim. No observable output change.
MEDIUM 3: keep the template/ exclusion cross-referenced in both places
it is enforced — validate-workflow-starters.ts EXCLUDED_DIRS and
showcase_smoke-monitor.yml's per-slug continue. If a future non-service
sibling (e.g. shared/, docs/) gets added, the pair of comments makes
it obvious that both sites need the new entry.
`cut -c1-200` truncated mid-slug, producing reader-hostile output
like `starter-claude-sdk-pyth` or `succeeded: starter-cl`. Replace
with a `truncate_csv` helper that iterates items, appends until the
next item would blow the budget, then emits `…` to signal truncation.
Applied to three list-cap sites (services summary, succeeded_list,
failed_list). Falls back to a hard character cut only when the very
first item is already over budget, so a single pathological 250-char
slug still emits something rather than an empty string.
Helper is inlined per step because YAML run: blocks don't share shell
functions. Comment cross-references steps.legs so the two copies stay
recognisably linked.
`gh api -i 2>&1` merged stderr into stdout, so gh diagnostics (auth
failures, rate limits, network errors) could be spliced ahead of the
HTTP header block. That corrupted HTTP_STATUS/API_BODY parsing — a
failing drift run would look clean while every call actually failed.
Redirect stderr to a mktemp file instead; surface it only when `gh`
returns a non-zero RC with no HTTP status (genuine connectivity/binary
failure). Normal HTTP error paths (401/403/404/5xx) continue to parse
stdout as the canonical header+body response.
Replaces the magic `-eq 19` sentinel that assumed a fixed non-starter
count with a `grep -c '^starter-'` check that counts only the appended
starter entries. Any future add/remove on the literal non-starter list
would have silently disabled the previous guard.
Also enables bash `nullglob` around the `showcase/starters/*/` loop
so an empty/missing starters tree doesn't expand to the literal pattern
and corrupt SERVICES with a bogus `starter-*` entry.
The SERVICES=(...) array in showcase_smoke-monitor.yml's 'Check image
drift' step was a hardcoded copy of the 17 starter slugs already
declared by showcase/starters/*/ directory names. Every new starter
required a manual edit in three places; the parity validator caught
drift after the fact but couldn't prevent it.
This commit:
- Adds a sparse actions/checkout step for showcase/starters/ only.
- Replaces the literal starter-* entries with a filesystem enumeration
(for dir in showcase/starters/*/; do ... done), skipping template/.
- Fails loudly if the enumeration produces zero starters, so a broken
checkout can't silently under-check drift.
- Updates validate-workflow-starters.ts to drop the smoke-monitor check
(drift is now structurally impossible) while keeping the two remaining
literal-list checks against showcase_deploy.yml (workflow_dispatch
options must be literal pre-checkout; ALL_SERVICES matrix carries
per-starter deploy metadata like railway_id).
Non-starter services stay literal — they don't live under
showcase/starters/ and are provisioned differently.
The starter slug list is duplicated across at least three places:
- .github/workflows/showcase_deploy.yml workflow_dispatch options
- .github/workflows/showcase_deploy.yml ALL_SERVICES matrix entries
- .github/workflows/showcase_smoke-monitor.yml SERVICES bash array
Adding a new starter under `showcase/starters/` but forgetting any of
these leaves the service deployable in theory but invisible to the
dispatch UI and/or drift detection — exactly the failure mode this PR
is trying to close.
Add `showcase/scripts/validate-workflow-starters.ts`. It enumerates
every directory under `showcase/starters/` (excluding `template/`)
and confirms `starter-<slug>` is present in each of the three
workflow locations, emitting a precise "missing from: <source>"
diagnostic per gap.
Wire it into showcase_validate.yml right after `validate-parity` so
it gates every PR + main push touching `showcase/**` or the relevant
workflow files. Also extend that workflow's `on.paths` filter to
include `showcase_deploy.yml` and `showcase_smoke-monitor.yml` so
edits to those files trigger the parity check.
The drift detector queried GHCR with `gh api ... || true` and then
skipped the service whenever `$TAGS` was empty. That collapsed three
very different outcomes into one silent continue:
- 200 OK with no published versions yet (legitimate, quiet)
- 404 for a brand-new service (legitimate, quiet)
- 401/403/5xx transient error (NOT legitimate — looks clean and
hides a broken drift check from operators)
Capture the HTTP status via `gh api -i`, route 200-empty and 404 to
a quiet continue with a log line, and emit `::warning::` for any
other status so operators see the problem in the run UI without
failing the whole drift run over one flaky service.
Also tighten the SHA match from substring `grep -q "$SHA"` to word-
boundary `grep -qw "$SHA"`. The previous pattern could false-positive
if one SHA happened to be a prefix of another tag on the same image
(extremely unlikely at 40 chars, but the SHA-8 abbreviation shows up
in tags like `branch-<8char>` which is the failure mode we avoid by
anchoring).
`shell` and `shell-dojolike` use `health_path="/"` because neither
ships an `/api/health` endpoint — they're Next.js app shells serving
the homepage as the liveness signal. With the new `exit 1` on
persistent unhealth, a single transient 5xx or 301-chain glitch at
the homepage during a cold-start would reset the HEALTHY_STREAK and,
at the edge of the 24-attempt budget, could page us for nothing.
Wrap the HTTP probe in a 3-attempt inner retry (2s apart) before the
streak logic observes the code. Services whose homepage is genuinely
dead still fail — all 3 tries must return non-200 — but a one-off
blip no longer trashes accumulated progress. Non-shell services
(which hit `/api/health`, `/health`, etc.) benefit from the same
transient-tolerance for free.
The notify job's failure branch posted "FAILED — N service(s) targeted"
whenever `needs.build.result == 'failure'`, which rolls up to 'failure'
if even a single matrix leg failed. With fail-fast: false, one failed
leg out of 18 can roll up as 'failure' while 17 others shipped fine —
the old wording falsely implied all N failed.
Add a "Compute per-leg build results" step that queries the Actions
API for this run's jobs, buckets each `build (<leg>)` leg by its
conclusion, and exposes failed/succeeded counts + lists as outputs.
The payload step now picks between three shapes:
- Partial failure: "N/M service(s) failed (<failed>) — <succeeded> ok"
- Full failure: existing "FAILED — M service(s) targeted (<list>)"
- API lookup lost: softer "1+ of M service(s) failed (<list> targeted)"
so the alert never lies about how many were affected.
Also reword the stale "sleep-on-idle waking up" comment on the
success-but-no-HTTP-200 branch — Railway is on the Pro tier (see
rationale near line 412) and does not sleep on idle. Clarify the
notify cancel-handling comment accordingly.
Previously, when a deployed service never returned a 200 on its health
path within the 360s window, the workflow emitted `::warning::` and
exited 0. That masked silently-broken deploys: Slack saw green, smoke
monitor saw no failing deploy run, and the stale image sat live.
Emit `::error::` and exit 1 so the deploy job fails and the alert
actually fires. Railway is on the Pro tier with no sleep-on-idle, so a
persistent failure to become healthy in 6 minutes is a real failure,
not a cold start.
Evidence of silent failures: https://github.com/CopilotKit/CopilotKit/actions/runs/24616467033
The smoke monitor's SERVICES array only listed package slugs (mastra,
llamaindex, ...) and did not include any starter-* slugs. As a result,
GHCR showcase-starter-<svc> tags were never checked for drift and the
monitor never dispatched showcase_deploy.yml for starter services when
their images fell behind main.
showcase_deploy.yml already accepts starter-* dispatch names and already
redeploys any service with a railway_id via serviceInstanceRedeploy, so
simply extending SERVICES to include the 17 starter slugs is sufficient
to close the loop.
Evidence of the gap: https://github.com/CopilotKit/CopilotKit/actions/runs/24616467033
The Alert Slack step used `payload-file-path` pointed at an
`mktemp`-produced extensionless tmpfile. slackapi/slack-github-action
v2.1.0 requires the payload file to have a `.json` / `.yaml` / `.yml`
extension and rejects extensionless paths with:
SlackError: Invalid input! Failed to parse file extension /tmp/tmp.XXX
SlackError: Invalid input! Failed to parse contents of the provided payload file
This silent failure (masked by `continue-on-error: true`) meant smoke
results never reached #oss-alerts.
Switch to the inline `payload:` + `toJSON(format(...))` pattern from
showcase_validate.yml (PR #4068, commit d4e75958). This:
- avoids the file-extension trap entirely — no temp file needed
- JSON-encodes dynamic values so quotes/backslashes/newlines in
starter slugs or error excerpts can't break the payload
- carries failed job + first error line per the #oss-alerts
actionable-detail policy (feedback_oss_alerts_detail)
The extraction step is preserved but now emits to `$GITHUB_ENV` with
heredoc delimiters (not `$GITHUB_OUTPUT`) so the Slack step can reference
values via `env.*` inside the `toJSON(format(...))` expression. Four
failure modes are still distinguished (missing_report, jq_parse_failed,
no_failures_in_report, real failures) via the `extraction_error` + truncated
first-failure excerpt emitted as `error_excerpt`.
The `continue-on-error: true` and temp-file cleanup steps are removed —
the inline payload approach no longer creates anything to clean up, and
a genuine Slack failure should surface in the job status.
Bare "❌ Showcase validate: failed | View run" forces a click-through to
triage — the signal that would make the alert actionable is one click
away. Per the oss-alerts policy, red alerts must carry triage-ready
detail in the payload itself.
Adds a pre-notify step that (on failure(), push, webhook set):
- resolves the current job's ID via the runs/{id}/jobs API (matches by
job name; falls back to "first job with a failed step" for rename
drift tolerance)
- extracts the first failed step name from the same jobs response
- pulls the failed-step log via `gh run view --log-failed --job=<id>`,
strips TSV prefix + leading timestamp + ANSI codes, skips runner
header noise (##[group], shell:, env:, Run), then grabs the first
line matching [FAIL]/[ERROR]/Error:/::error and truncates to 300
chars
- emits failed_step and error_excerpt to $GITHUB_ENV via heredoc
delimiters (safe for values containing = or newlines)
- uses set +e and explicit exit 0 so extraction glitches never block
the notify step — fallbacks yield "unknown step" / "see workflow
run for details"
Updates the Slack payload to interpolate both values via toJSON(format())
so dynamic content is JSON-encoded defensively (matches the pattern from
showcase_drift-report.yml). Preserves the ❌ prefix, the "View run"
link, and keeps total length well under the 800-char budget.
Companion to #4065, which quieted routine per-run success posts; this
PR makes the remaining (failure-only) posts actionable at a glance.
Against run 24598654559 (tonight's validate-parity regression), the
extracted excerpt is:
[FAIL] ag2: demo 'hitl-in-chat' declared in manifest but no
src/app/demos/hitl-in-chat/ directory
which is exactly the triage signal that was missing from the bare
"failed" alert.
Blocking fix: aimock serves `/__aimock/health` + `/v1/*` only — probing `/`
returns HTTP 404 which fails `curl -sf`, so the readiness loop never broke
and every run hard-failed at "Start aimock". Switched all three probes
(startup loop, startup verify, pre-Playwright re-probe) to
`/__aimock/health`.
Also tightened the workflow in four smaller ways:
* Added `--max-time 2 --connect-timeout 1` to every curl probe (aimock
readiness + verify, agent readiness + verify, dev-server readiness +
verify, pre-Playwright re-probe). A hung socket can't blow the loop's
iteration budget now.
* Shell slug extractor (`grep -oE`) and job-level `startsWith` gate now
agree on "first token only": the extractor anchors at `^` instead of
allowing a leading-whitespace alternative. Header comment updated so the
two-layer defense-in-depth rationale reflects the now-identical behavior.
* Captured aimock PID into `$GITHUB_ENV` and added an `if: always()`
"Re-check aimock liveness after Playwright" step that `kill -0`s the PID
after tests. If aimock OOM'd mid-Playwright, the job now fails loudly
instead of trusting cached / fall-through results.
* Removed dead `OPENAI_BASE_URL` / `OPENAI_API_KEY` env on the
`Run Playwright tests` step (Next.js was already running from the earlier
step with these inline on that process; env on `npx playwright test`
didn't flow anywhere). Added comment explaining why setting them here
would be misleading.
* Expanded comment on `workflow_dispatch.inputs.slug.type: choice` to
document that single-choice UI preselect IS intentional (refused the
sentinel-value footgun).
- Replace `npm bin -g` with `npm prefix -g` + /bin/aimock. `npm bin` was
removed in npm 9.0.0 and setup-node@v4 with node 22.x ships npm 10+, so
the old form exited with "Unknown command: 'bin'" and the existence
check below fired on every run, hard-failing the workflow.
- Replace `pip install --only-binary :all:` with `--prefer-binary`. CrewAI's
transitive graph (tiktoken / chromadb / litellm cadence releases) ships
sdist-only revisions often enough that `--only-binary :all:` made the
workflow fail-loud with "Could not find a version that satisfies the
requirement" on otherwise-valid requirements.txt. `--prefer-binary`
keeps the wheel-first preference while letting sdist-only deps install;
residual source-build-hook risk is bounded by the author_association gate.
- Document the comment-trigger TOCTOU in the residual-trust-model header:
there is a window between `/test-aimock` (reviewing diff D1) and
`pulls.get` (resolves whatever HEAD is current), so a force-push of
malicious content in between wins. GitHub Actions has no native
comment-time SHA pin, so the mitigation is the author_association gate
+ social contract — documenting accepts the known residual risk.
- Drop dead `else APP_MODULE=agent:app` branch. All currently-dispatchable
slugs ship src/agent_server.py (enforced by the ships_toggle check
above), so the legacy path was unreachable. Fail loud with `::error::`
instead of silently falling through to a guessed module name.
- Capture aimock PID and `kill -0` inside the readiness loop so an aimock
that crashes on startup fails in ~1s instead of burning the full 20s
polling a dead process.
- Add `persist-credentials: false` to the PR-HEAD checkout. Prevents the
workflow's GITHUB_TOKEN from being left in .git/config where attacker-
controlled PR-HEAD code (pip build hooks, Playwright fixtures) could
read it. Defense-in-depth on top of `permissions: contents: read`.
- Guard the Resolve PR HEAD step with `pr.state === 'open'`. A trusted
commenter typing `/test-aimock` on a stale closed PR would otherwise
re-exec the old HEAD — wasting CI or re-running known-bad code.
- Add a residual-trust-model header comment to the workflow documenting
the social contract (trusted commenter reviews PR diff before typing
the command; not a security boundary against a rogue OWNER/MEMBER).
- Tighten the Actions-level gate from `contains(body, '/test-aimock ')`
to `startsWith(body, '/test-aimock ')`. Substring matches fire on
mentions inside code blocks / quoted replies; startsWith requires
`/test-aimock ` at the very start of the comment.
- Replace `npx @copilotkit/aimock@...` with a direct invocation of the
globally-installed binary via `$(npm bin -g)/aimock`. npx may re-resolve
the spec against the registry, defeating both `--ignore-scripts` (which
doesn't propagate to npx's transient install) and the caret pin if a
new patch published between global install and npx.
- Add a second aimock liveness probe right before Playwright runs. If
aimock died between initial readiness and test execution, Playwright
would silently fall through to real OpenAI (since OPENAI_BASE_URL
points at a dead port). Fail loud instead.
- Remove the dead `/` fallback in the Python agent health check. FastAPI's
root is typically a POST endpoint that always fails `curl -sf` — the
fallback was copy-paste residue that added no signal.
- Pin `pnpm/action-setup` to `@v4.4.0` matching showcase_validate.yml
(was floating `@v4`).
- Make workflow_dispatch `slug` input required (no default). A hidden
default to crewai-crews contradicted the "no silent fallback" claim
enforced by the comment-path extractor.
- Reword the Next.js OPENAI_BASE_URL comment to acknowledge it's
defensive-only in the CrewAI showcase (Next proxies via runtime, not
direct OpenAI); the value still prevents a future route-level leak.
- Add `--only-binary :all:` to the pip install in showcase_validate.yml's
python-unit-tests job, mirroring the hardening already in the aimock-e2e
sibling workflow. This job runs on EVERY PR automatically without a
trusted-commenter gate, so the broader exposure deserves the same
wheel-only install defense against PR-controlled setup.py hooks.
- Drop the unconditional per-package requirements.txt install. Today's
tests are stdlib-only (+ typing_extensions fallback path); installing
the full CrewAI graph on every PR run was a ~30-60s tax with no benefit.
Install only pytest + typing_extensions. Comment documents the upgrade
path for future tests that need package runtime deps.
- Drop the now-unused pip cache on setup-python (keyed on a never-used
requirements.txt would be pure overhead).
HIGH fixes from CR5 round:
- H1: `pip install --only-binary :all:` forces wheel-only installs, blocking
source-build hooks (setup.py / PEP 517 build backends) from executing on
the runner. Unlike npm/pnpm there is no `pip --ignore-scripts` equivalent;
wheels are the closest mitigation. Residual risk documented inline.
- H2: Tighten `/test-aimock` match so `/test-aimocker` / `/test-aimock-like-this`
do NOT trigger runs. Require trailing space via job-level `if:` AND enforce
whole-word boundary in the shell-level slug extractor (belt-and-suspenders).
Remove the silent `crewai-crews` default fallback — a missing slug now FAILS
the workflow instead of silently running against the default.
- M4: Scope write perms (pull-requests + issues) to a SEPARATE `post-result`
job that only runs the comment post. The heavy test job now runs with
`contents: read` only, so a compromised transitive dep from `pip install`
on a PR-controlled requirements.txt cannot mutate PRs/issues with the
workflow token.
- M8: Use `npx next dev` directly in "Start dev server" step. `pnpm dev`
would spawn a SECOND uvicorn on :8000 via concurrently, racing the
already-bound agent from the prior step and silently picking whichever
won the port.
LOW fixes:
- Pin `npx` invocation to fully-scoped `@copilotkit/aimock@^1.14.3` so a
squatter on the unscoped `aimock` name can't be silently picked up if the
global install fails.
- Add rationale comment explaining why `steps.slug.outputs.slug` interpolation
is safe (slug is whitelisted + dir-validated before use).
showcase_aimock-e2e.yml:
- workflow_dispatch slug is now a choice-type enum restricted to Python
packages that ship aimock_toggle.py; a TS (mastra) or Java (spring-ai)
slug would have skipped the Python agent start step and then failed with
a misleading Playwright timeout. For comment-trigger paths, the pkg-type
step short-circuits with a clear ::error:: when the slug does not ship
aimock_toggle.py.
- Slug parsing replaced PCRE grep -oP '\\K\\S+' with a POSIX-safe
grep -oE + sed pipeline so BSD/Alpine grep works too (future-proof
against runner image changes).
- aimock pinned to @copilotkit/aimock@^1.14.3 (was @latest) with
--ignore-scripts; unpinned @latest let a bad aimock publish silently
poison CI for everyone.
- pnpm install now runs with --ignore-scripts — trusted commenter triggers
/test-aimock on untrusted PR content, so postinstall scripts must not
run on the runner.
- OPENAI_BASE_URL is no longer pre-set for Python agent start on packages
that ship aimock_toggle.py — forcing the toggle itself to do the
redirection proves the toggle works rather than masking it with env
already set.
- Dead AIMOCK_URL export removed from the Next.js dev-server step (the
runtime only reads OPENAI_BASE_URL).
- Python agent health-check extended 60s -> 90s (45 iter * 2s) for CrewAI
cold imports.
- actions/setup-python caches pip keyed on requirements.txt.
- github-script Post-result step passes slug + job status via env rather
than ${{ }} interpolation — even though the slug is already whitelisted,
the env pattern is the defensive default for dynamic values.
showcase_validate.yml:
- python-unit-tests is now a matrix on Python 3.10 + 3.12 (fail-fast
disabled) so the typing_extensions fallback branch gets CI coverage;
previously only 3.12 ran.
- pip install per package is a HARD FAIL (was ::warning:: + continue);
hiding broken requirements.txt behind a warning let a package ship green
with unresolvable runtime deps. A missing requirements.txt is still
handled gracefully via the -f guard.
- pip caching added via actions/setup-python.
The showcase_deploy.yml notify job posted a green success message on
every run. Bulk drift rebuilds fan out one showcase_deploy.yml run per
stale service (up to ~18), so a single drift cycle produced ~18
"Showcase deploy: 1 service(s) deployed to Railway" messages in
#oss-alerts — pure noise.
Align with the channel policy: surface only actionable state
(failures + state transitions). The bulk-rebuild aggregate is already
posted by showcase_smoke-monitor.yml as "📦 Image drift detected
— N rebuilds triggered". Ad-hoc single-service pushes/dispatches stay
quiet on success — the Actions UI is the source of truth.
Kept as-is: failure messages (pre-build and build), mid-matrix
cancellation info (state transition worth humans seeing). Also added an
empty-webhook guard on the Post step so a missing SLACK_WEBHOOK_OSS_ALERTS
secret fails closed instead of erroring out the step.
The `secrets.*` context is not a valid named-value inside step-level
`if:` expressions on push events — GitHub Actions rejects it at
workflow parse time with "Unrecognized named-value: 'secrets'", which
caused both showcase_validate.yml and showcase_drift-report.yml to
fail at startup with zero jobs spawned after #4018 + #4060 merged.
Hoist the webhook into a job-level `env: SLACK_WEBHOOK` and reference
`env.SLACK_WEBHOOK` in every step-level `if:`. The `with: webhook:`
keys still use `secrets.*` directly (valid in that context).
PR CI didn't catch this because pull_request events parse if:
expressions less strictly than push events.
showcase_validate.yml and showcase_drift-report.yml were flipped to
runs-on: depot-ubuntu-24.04-4 in #4018 but inherited the repo's
default least-privilege permissions (contents: read only). Depot
runner provisioning uses OIDC and requires id-token: write, so the
jobs failed to spawn on the first main-branch push — matching the
pattern already used by showcase_deploy.yml's Depot job.
## Showcase validation tooling (Bundle 3)
Ships three CLI validators, a shared parsing lib, and two CI workflows
that enforce consistency across the 17 showcase packages and detect
drift before it lands on main. Consolidates four earlier tooling PRs
(#3985, #3987, #3995, #3996).
## What's in the box
### `showcase/scripts/` — three validators
| Tool | Purpose | Exit codes |
|------|---------|-----------|
| `audit.ts` | Cross-checks manifest-declared demos against
`tests/e2e/*.spec.ts` and `qa/*.md`, plus `examples/integrations/`
provenance via `SLUG_TO_EXAMPLES` / `FALLBACK_MAP` | 0 ok, 1 anomalies,
2 invalid-input, 3 unreadable, 4 internal, 5 strict-warnings |
| `validate-pins.ts` | Framework-dep pin-drift between
`showcase/packages/*/` and their dojo `examples/integrations/*/`
counterparts. Parses package.json, requirements.txt, pyproject.toml
(Poetry + PEP 621) | 0 ok, 1 drift, 2 internal, 3 unreadable |
| `validate-parity.ts` | Enforces demo ↔ spec ↔ qa coverage per package
with a monotonic demo-count floor | 0 ok, 1 warnings, 2 invalid-input, 3
unreadable, 4 internal, 5 must-failure |
### `showcase/scripts/lib/` — shared primitives
- **`slug-map.ts`** — single-source-of-truth `ENTRIES` for the showcase
slug taxonomy;
`BORN_IN_SHOWCASE`/`SLUG_MAP`/`SLUG_TO_EXAMPLES`/`FALLBACK_MAP` derived
and frozen at module load. `SlugEntry` is a discriminated union that
makes illegal states (born-in-showcase with non-empty examples)
unrepresentable. `freezeSet`/`freezeMap` helpers install throwing
replacements via `Object.defineProperty({writable:false,
configurable:false})` so `Set.add` / `Map.set` truly fail at runtime.
- **`manifest.ts`** — `parseManifest` returns a tagged `ParsedManifest`
union (`ok` | `missing` | `malformed{subkind: "syntax"|"shape"}` |
`unreadable`) with a never-throws content contract. Uses `statSync` +
errno inspection (not `existsSync`, which conflates ENOENT with EACCES).
`DemoId` is a branded string minted only via `createDemoId`.
### `.github/workflows/` — CI enforcement
- **`showcase_validate.yml`** — runs on PR and push-to-main. Enforces
the e2e-spec floor, runs the validators, and drives the pin-drift
ratchet.
- **`showcase_drift-report.yml`** — weekly Monday 10:00 UTC +
workflow_dispatch. Computes `set_status` (OK / SET DRIFTED / COUNT
DRIFTED) and posts to Slack.
Both workflows run on `depot-ubuntu-24.04-4` (Startup plan, unlimited)
for persistent pnpm/npm cache across runs.
## The pin-drift ratchet
`validate-pins.ts` currently finds **111 existing pin-drift failures**
across 12 showcase packages. Rather than block the PR on those, we
baseline them in `showcase/scripts/fail-baseline.json` and ratchet:
- `validatePinsFailCount` must not increase; CI tells you to ratchet
down when it decreases.
- `validatePinsFailHash` is SHA-256 of the sorted-uniqued `[FAIL]` set.
When the count is equal but the hash differs, a fail healed AND a new
one regressed — CI prints the diff and fails.
- `baselineDemoCount` (9) is the single source of truth for the e2e-spec
floor; consumed by both the workflow and `validate-parity.ts` with sync
enforced by a dedicated regression test.
Tracked in #4047. The 111 failures are mostly showcase packages pinning
`@copilotkit/*` to the `next` dist-tag while dojo pins concrete
versions; direction of fix (align showcase → dojo vs. bump dojo →
showcase) is a separate versioning decision outside this PR.
## Correctness posture
- **977 tests**, 13 files, covering every `Anomaly` / `PackageIssue` /
`ParsedManifest` variant in-process and via subprocess CLI for every
exit code. EACCES/ENOTDIR/TOCTOU paths are exercised via chmod probes
(with `it.skipIf` fallback when CI runs as root) and path-filtered
`vi.spyOn` fall-throughs.
- **`fs.statSync` + errno everywhere** — `fs.existsSync` silently
collapses ENOENT with EACCES and is a known anti-pattern in validation
tooling; the codebase uses structured errno discrimination throughout.
- **Tagged discriminated unions with exhaustive `switch` + `never`
guards** — `bucketFor` in `audit.ts`, `deriveMessage` in
`validate-parity.ts`. Adding a new variant without wiring every site is
a compile error.
- **Partial-report preservation** — when an infra error hits
mid-slug-loop, `UnreadableInputError.partialReport` carries
already-collected drift findings so the top-level catch prints them
before exiting 3. One bad package never orphans signal for the rest.
- **Per-slug isolation** — in `validate-parity.ts runParityImpl`, each
slug's audit is wrapped; a crash surfaces as a `crashed` `PackageIssue`
and forces `EXIT_INTERNAL` without aborting siblings.
- **Pipefail + scoped `|| true`** — every workflow step uses `set -euo
pipefail` with grep's no-match tolerance wrapped in `{ grep || true; }`
so producer failures (sort, shasum, cut) still surface.
## Diff
+16,455 / −18 across 34 files (26 source + 5 fixture trees + 2 workflows
+ 1 baseline).
Commits grouped by purpose:
1. `chore(showcase/scripts)`: vitest config + test deps
2. `feat(showcase/scripts)`: shared slug-map and manifest parsing lib
3. `feat(showcase/scripts)`: audit.ts coverage auditor
4. `feat(showcase/scripts)`: validate-pins.ts pin-drift validator
5. `feat(showcase/scripts)`: validate-parity.ts demo/spec/qa parity
validator
6. `ci(showcase)`: validation + weekly drift-report workflows (Depot
runners)
## Test plan
- [x] `pnpm vitest run` in `showcase/scripts/` — 977/977 green
- [x] Exit-code taxonomy verified end-to-end via subprocess tests for
every documented code
- [x] EACCES/ENOENT/ENOTDIR routing verified in all three validators
- [x] Partial-report preservation verified in both in-process and
subprocess paths
- [x] Per-slug crash isolation verified (one broken slug does not orphan
siblings)
- [x] Baseline sync contract (`BASELINE_DEMO_COUNT` ↔
`fail-baseline.json.baselineDemoCount`) pinned by test
- [ ] First CI run on Depot to confirm cold-cache timing (expected 5–8m
vs. 18–20m on ubuntu-latest)
Refs: [Full Action
Inventory](https://www.notion.so/3443aa38185281b5a1dfc6e0890264e1),
#4047
Two GitHub Actions workflows that consume the three validators in
showcase/scripts/ and surface drift to CI and Slack.
showcase_validate.yml — runs on pull_request and push-to-main:
- Enforces per-package e2e spec count against
fail-baseline.json.baselineDemoCount, with find failures
aggregated (not exit-on-first) so multiple package issues surface
in one run. Preserves find's exit status by capturing via command
substitution rather than process substitution (mapfile does not
propagate exit codes through < <(...)).
- Runs validate-pins.ts with both count + content-hash ratchet:
the sorted-uniqued [FAIL] set is SHA-256 hashed so a
count-preserving set change ('one fail healed, another regressed')
is still flagged.
- Separates stdout from stderr before hashing so progress chatter
cannot corrupt the ratchet hash.
- Preserves validator exit codes distinctly (1 drift, 2 internal,
3 unreadable, 4+ future) so downstream consumers can distinguish
crashes from legitimate drift.
- set -euo pipefail throughout, with { grep || true; } scoped to
tolerate grep no-match without masking producer failures.
- Slack notifications gated on push events and secret presence;
payload values wrapped via toJSON(format(...)) for injection
safety.
showcase_drift-report.yml — weekly Monday 10:00 UTC + dispatch:
- Same ratchet + hash logic applied in report mode: computes
set_status (OK / SET DRIFTED / COUNT DRIFTED) and posts to Slack.
- Mirrors validate.yml's pipefail + grep-scope + toJSON discipline.
pull_request runs in the fork context with restricted permissions, which
would block the updateBranch API call. pull_request_target runs in the
base repo with full permissions — the label trigger is safe since no
checkout of PR code happens here.
Also treat 422 responses (already up to date, nothing to merge) as
informational rather than job failures.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Triggers on the qa:update-branch label being added to a PR, calls the
GitHub update-branch API to merge the base into the PR, and removes the
label so it can be re-applied. Enables the QA bot to keep PRs current
without human intervention.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Internal job id was still 'starter-smoke:' and artifact pattern was
'starter-smoke-${{ matrix.starter }}'. Rename to 'smoke-starter' /
'smoke-starter-${{ matrix.starter }}' to match the new
test_<layer>-<target> / smoke-<layer> naming convention. Cosmetic only —
no other workflow references this job id or artifact name.
Asymmetric path filter — .changeset was listed in PR trigger but not push.
The dorny/paths-filter step already includes .changeset in the ts: filter,
so this matches the intent on both triggers.
Showcase-only and sdk-python-only PRs were triggering the full TS unit matrix
(3 Node versions x full monorepo). Neither directory affects TS unit tests.
Now paths-ignore matches the spirit of the existing 'examples/**' exclusion.
## Problem
All 18 stale-service rebuild triggers in [run
24534637524](https://github.com/CopilotKit/CopilotKit/actions/runs/24534637524)
failed with:
```
failed to run git: fatal: not a git repository (or any of the parent directories): .git
```
The image-drift workflow is completely broken — when it detects stale
showcase images, the follow-up `gh workflow run showcase_deploy.yml`
step 500-fails for every service, so no rebuilds actually happen.
## Root cause
The `smoke` job in `.github/workflows/showcase_smoke-monitor.yml` has
**no `actions/checkout@v4` step**. `gh workflow run` (unlike `gh api` or
`gh run list`) resolves the target repository by shelling out to `git`
in the current directory to read the remote. With no checkout present,
`git` aborts and the entire rebuild loop fails.
PR #3968 didn't introduce the missing-checkout problem — it was latent —
but it was the first run to reach the rebuild path with a real stale
list, exposing it.
## Fix
Pass `--repo "${{ github.repository }}"` to `gh workflow run` so the CLI
skips the git-based repo lookup entirely. Minimal, one-line change; no
checkout needed.
```diff
-ERR_OUTPUT=$(gh workflow run showcase_deploy.yml -f service="${SVC}" 2>&1) || RC=$?
+ERR_OUTPUT=$(gh workflow run showcase_deploy.yml --repo "${{ github.repository }}" -f service="${SVC}" 2>&1) || RC=$?
```
## Test plan
- [x] YAML parses
- [x] Local pre-commit (lint + test + check:packages) green
- [ ] Next scheduled smoke-monitor run (or manual `workflow_dispatch`
with a stale service) triggers rebuilds without the `not a git
repository` error
## Summary
The "files needing manual review" Slack warning in
`showcase_docs-sync.yml` listed flagged files but did not link the
auto-opened PR, forcing reviewers to hunt for it manually. This PR
captures the PR URL (already exposed as `steps.push.outputs.pr_url`) and
threads it into the alert payload so reviewers can click through
directly.
## Changes
- Review-needed alert now includes a `Review: <PR URL>` line
- Split the alert into two variants:
- PR opened (normal case): includes the PR link
- No PR opened (edge case where the clean-transform portion was empty):
posts review items without a link
- Auto-sync and merge-failed alerts already linked the PR — this brings
the review-needed alert to parity
## Test plan
- [ ] YAML validated locally (`python3 -c "import yaml;
yaml.safe_load(...)"`)
- [ ] Next docs sync that produces review items should include the PR
link in Slack
Addresses three CR findings on PR #3980 — rebuilt from origin/main to
drop stale test-integration-tmp plumbing that was reintroduced in the
prior iteration.
Finding 1 — Scoped cancel-skip for concurrency group cancellations:
- Pre-build cancellation (detect-changes/check-lockfile cancelled):
stay silent, newer run will redo all work from scratch.
- Mid-matrix build cancellation: post a distinct muted info message
so humans can spot anomalies (fail-fast:false matrix legs rolling
up to `cancelled` when some legs silently passed via warn-not-fail
health-probe timeout).
Finding 2 — Per-service `health_path` with no silent fallback:
- Every service in ALL_SERVICES carries an explicit `health_path`
field (`/api/health` for Next.js, `/health` for aimock).
- Verify step fails loud with `::error::` + exit 1 when health_path
is unset, rather than silently defaulting to /api/health. An
unscoped fallback could mask a broken endpoint when an unrelated
catch-all / CDN / actuator happens to 200 at a different path.
Finding 3 — Comment accuracy:
- health_path comment now cites the policy ("no fallback — misconfig
is a config bug, not runtime behavior to hide").
- Cancel-skip comment enumerates all three cancellation causes
(concurrency supersede, manual cancel, upstream failure cascade)
and explains the pre-build vs mid-build split.
The smoke-monitor job has no actions/checkout step, so `gh workflow
run` cannot resolve the repo from a git remote and fails with:
failed to run git: fatal: not a git repository
Pass --repo ${{ github.repository }} so gh CLI skips the git lookup.
All 18 rebuild triggers in run 24534637524 failed for this reason.
Jordan Ritter