Adds .github/workflows/showcase_promote_notify.yml — workflow_dispatch-only
Slack notify workflow for promote results. Posts initiation + threaded reply
to #team-showcase; cross-posts to #oss-alerts on partial/total failure.
Also adds showcase_promote_notify.dry-run.sh — a local render-logic mirror
that decodes the same payload and prints the messages the workflow would
send to Slack, without making any Slack API calls. Used by the pre-merge
runbook and by future CI fixture tests.
## Summary
Ports
[ag-ui-protocol/ag-ui#1914](https://github.com/ag-ui-protocol/ag-ui/pull/1914)
to CopilotKit — plus the two supporting guard files ag-ui already had:
- **`.github/workflows/canary.yml`** — discoverable **`canary /
publish`** `workflow_dispatch` orchestrator. Any maintainer can publish
a prerelease of the branch they're on straight from the Actions tab. It
is a thin orchestrator — it does **not** publish to npm itself:
1. Guards against `main` and non-branch refs.
2. Mints the devops-bot App token (app-id `1108748`,
`DEVOPS_BOT_PRIVATE_KEY`) with scoped `contents:write` +
`actions:write`.
3. Mirrors the dispatched ref to a unique
`canary/<slug>-<run_id>-<attempt>` branch via the GitHub API (no
checkout).
4. Dispatches **`publish-release.yml --ref canary/<slug> -f
mode=prerelease …`**, locates the run, and waits (`gh run watch
--exit-status` + explicit conclusion check).
5. Deletes the canary ref — status-gated (never yanks the ref under a
still-running delegated run) with a fresh cleanup token (90-min job
ceiling exceeds the 1h App-token TTL).
- **`scripts/release/verify-release-scope-dropdowns.sh`** — drift guard:
the hand-maintained `scope` dropdowns in `publish-release.yml` /
`stable-release.yml` / `canary.yml` must exactly match
`release.config.json`'s `.scopes` keys. Parsers fail loud and distinct
on structural changes instead of silently passing.
- **`.github/workflows/lint-release-workflows.yml`** — actionlint +
shellcheck + the dropdown-sync job over the release pipelines.
### Why a separate orchestrator (and not a flag in publish-release.yml)
- A GitHub Environment's deployment-branch policy is evaluated against
the ref a run is **triggered on** — not branches created mid-run. The
orchestrator exists to get the publish run *onto* a `canary/*` ref.
- `publish-release.yml` holds the **single npm OIDC trusted-publisher
binding**; a second publishing entry point would break OIDC for every
`@copilotkit/*` package. The orchestrator never touches npm.
- The cross-workflow dispatch uses the **App token, not `GITHUB_TOKEN`**
— `GITHUB_TOKEN`-authenticated events never start new workflow runs.
**Note:** the `npm` environment currently has *no* deployment-branch
policy, so the orchestrator is a convenience wrapper today. Tightening
the policy to `main` + `canary/*` + `release/publish/*` (matching
ag-ui's security posture) is being applied as repo configuration
alongside this PR — requires admin. This PR includes the prerequisite:
`publish-commit.yml` (pkg-pr-new) is removed from the `npm` environment,
since it runs on every PR and would be blocked by the policy (it
publishes to pkg.pr.new, not npm, and uses no environment secrets).
## Testing done
- Drift guard: positive run against all three real workflows; negative
tests (scope removed → drift FAIL with diff; bogus scope → FAIL; `case
"${SCOPE}"` quoting refactor → loud parser-degradation FAIL; whole case
block deleted → loud zero-block FAIL; quoted arm `"angular")` →
accepted; blank/comment lines inside `options:` → still parsed; prose
comments mentioning case/SCOPE/in → no false positive).
- `shellcheck` clean at all severities; `bash -n` on every workflow
`run:` block; YAML parses.
- 3 rounds of 7-agent code review converged to zero load-bearing
findings.
## ⚠️ Still to verify before first real use
- [ ] devops-bot App (id 1108748) has **Actions: write** — required for
the in-workflow `gh workflow run`. Safe first test: dispatch once with
`dry_run=true`.
- [ ] First `dry_run=false` run clears the `npm` environment end-to-end
via the App token once the deployment-branch policy is tightened.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Post-merge follow-ups (maintainer action required)
These need repo **admin** rights and must happen **in this order**:
1. **Merge this PR first.** `main`'s current `publish-commit.yml`
(pkg-pr-new) still sits in the `npm` environment and runs on every PR
touching `packages/**` — applying the branch policy before this PR lands
would block every snapshot publish. This PR removes that environment
association.
2. **Tighten the `npm` environment's deployment-branch policy** to
`main` + `canary/*` + `release/publish/*` (matching ag-ui). With an
admin-scoped token:
```bash
gh api --method PUT repos/CopilotKit/CopilotKit/environments/npm \
-F "deployment_branch_policy[protected_branches]=false" \
-F "deployment_branch_policy[custom_branch_policies]=true"
gh api --method POST
repos/CopilotKit/CopilotKit/environments/npm/deployment-branch-policies
-f name="main" -f type=branch
gh api --method POST
repos/CopilotKit/CopilotKit/environments/npm/deployment-branch-policies
-f name="canary/*" -f type=branch
gh api --method POST
repos/CopilotKit/CopilotKit/environments/npm/deployment-branch-policies
-f name="release/publish/*" -f type=branch
```
Or via UI: Settings → Environments → npm → Deployment branches and tags
→ "Selected branches and tags" → add the three patterns above.
Why these three: `main` covers stable `workflow_dispatch` retries and
`stable-release.yml`; `release/publish/*` covers the merged-release-PR
runs (the run's head branch is the release PR branch); `canary/*` covers
the orchestrator's delegated prerelease runs. After this, direct
`mode=prerelease` dispatches from arbitrary feature branches stop
working — the `canary / publish` orchestrator becomes the one-click path
(by design).
3. **Verify the devops-bot App (id `1108748`) has `Actions: write`**
(org/App settings). The orchestrator's `gh workflow run` dispatch 403s
without it. Safe end-to-end test, after step 2: Actions tab → **canary /
publish** → pick any feature branch, any scope, **`dry_run=true`** →
confirm the delegated `release / publish` run is created, watched, and
the `canary/*` ref is deleted afterward.
4. **First real canary** (`dry_run=false`) confirms the npm OIDC publish
clears the environment gate end-to-end on a `canary/*` ref.
Run `test / unit` over only the packages affected since the base instead
of building + testing every package 3× across the Node 20/22/24 matrix on
every PR.
- fetch-depth: 0 so affected has a merge-base to diff against.
- Derive NX_BASE/NX_HEAD: PR → merge-base with the base branch tip; push →
github.event.before with a HEAD~1 fallback.
- Select packages via `nx show projects --affected --projects='packages/**'`
fed to run-many (the `nx affected` run form ignores --projects and pulls
in downstream examples/storybook — hence the show-projects → run-many split).
- workflow_dispatch still runs all packages (manual/full run).
- Editing this workflow can't surface as an affected package, so a change to
test_unit.yml in the range now forces a full all-packages run — this keeps
the build/test path exercised on the PR that changes it.
- GitHub context passed via env: (not inline ${{ }}) to satisfy zizmor;
NX_VERBOSE_LOGGING forced off for the JSON-parsing step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The repo migrated off @changesets/* to conventional-commit-driven releases.
scripts/release/lib/changes.ts::getChangesSummary reads `git log <lastTag>..HEAD`
commit subjects and never touches .changeset/. No .changeset/config.json,
no @changesets/* in any package.json, no npm scripts reference it.
The 11 .changeset/*.md files describe changes that have either already
shipped (via commit subjects in prior releases) or will ship in the next
release (via the current commit subjects in the v1.60.1..main window) —
the .changeset/ files are inert.
Also removes the dead workflow `paths:` filters in
test_e2e-dojo.yml and test_e2e-legacy-v1.yml that re-fired e2e on
.changeset/ changes — nothing to fire on after the directory is gone.
Bumps the dojo e2e matrix from `depot-ubuntu-24.04` (2 vCPU) to
`depot-ubuntu-24.04-4` (4 vCPU) and `NX_PARALLEL: 4` so the build uses
the extra cores. This is the non-serializing way to cut dojo wall-clock
(the build-once dedup tried in #5450 regressed wall-clock and was
reverted).
## Result: −40% wall-clock (measured on CI)
Dojo wall-clock = the single slowest suite (the 15 run in parallel).
Comparison vs the 2-vCPU baseline:
| metric | 2-vCPU baseline | 4-vCPU | Δ |
|---|---|---|---|
| **wall-clock** (long pole `langgraph-python`) | 623s (10.4m) | **373s
(6.2m)** | **−40%** |
| runner-minutes (wall summed, 15 suites) | 105m | 76m | −28% |
| **billed compute** (vCPU-min; 4-vCPU ≈ 2× rate) | ~210 | ~304 |
**+45%** |
Every suite got faster; the long-pole suites benefited most:
| suite | 2-vCPU | 4-vCPU |
|---|---|---|
| langgraph-python | 623s | 373s |
| langgraph-typescript | 547s | 362s |
| langgraph-fastapi | 500s | 337s |
| adk-middleware | 414s | 286s |
| (… all 15 faster …) | | |
Long-pole `langgraph-python` step breakdown:
| phase | 2-vCPU | 4-vCPU |
|---|---|---|
| Build cpk | 82s | 48s |
| Prep dojo | 94s | 52s |
| **Run tests (Playwright)** | **271s** | **117s** |
| total | 623s | 373s |
**Key finding:** the Playwright phase more than halved → the e2e suites
are **CPU/worker-bound, not LLM-latency-bound**. A bigger runner is the
right lever; test sharding is not needed to reach ~6 min.
## Trade-off
−40% wall-clock for **~+45% billed compute** (4-vCPU costs ~2×/min,
partly offset by finishing 28% sooner). If the cost bump isn't worth it
across all 15 suites, a follow-up can scope `-4` to just the slow suites
via a per-matrix `runner` field (wall ~6.5m, smaller cost increase).
Companion to #5450 (unit-test `nx affected`).
Measure wall-clock impact of a larger Depot runner on the e2e suites.
The long pole (langgraph-python, ~10.4min) spends ~50% on build/prep
(CPU-bound) and ~43% on the Playwright run. 2->4 vCPU + NX_PARALLEL 4
should speed build/prep; the test-phase gain reveals whether it is
CPU-bound (big win) or LLM-latency-bound (then sharding is the lever).
The injected/streamed a2ui fixtures all included catalogId, so aimock
replay never exercised the basic-catalog fallback that broke production
(real models omit catalogId per the tool-usage guide). Strip catalogId
from the langgraph-python sales-dashboard secondary-call fixtures and
hard-assert "Catalog not found" is absent outside the charts-rendered
soft branch, so the spec fails without a route defaultCatalogId.
Also repoint the on-demand e2e workflow at the d4/d5-recorded/d6/shared
fixture dirs — it still referenced feature-parity.json, deleted in the
1e66a5f8d fixture reorg, so every /test-aimock run died at aimock start.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pkg-pr-new publishes snapshot builds to pkg.pr.new, not the npm
registry, and uses no environment-scoped secrets or variables. The
job runs on every push/PR touching packages/**, so the npm
environment's tightened deployment-branch policy (main, canary/*,
release/publish/*) would block every snapshot publish. Removing the
environment association is a prerequisite for the policy.
actionlint over stable-release.yml, publish-release.yml, canary.yml,
and itself; shellcheck (severity warning) over scripts/release shell
scripts; and the release-scope-dropdown-sync job running
verify-release-scope-dropdowns.sh. Scope intentionally narrow to the
release pipelines to avoid drowning unrelated changes in lint noise.
Ported from ag-ui-protocol/ag-ui.
canary / publish: a workflow_dispatch orchestrator that mirrors the
dispatched ref to a unique short-lived canary/<slug>-<run_id>-<attempt>
branch via the GitHub API (devops-bot App token, app-id 1108748),
dispatches publish-release.yml on that ref with mode=prerelease, waits
for the delegated run (gh run watch + explicit conclusion check), and
deletes the ref afterward.
It does NOT publish to npm itself — publish-release.yml holds the
single npm OIDC trusted-publisher binding. Failure paths covered:
suffix validated before any side effect, ref kept when a dispatched
run was never located, ref deleted when the dispatch itself failed,
status-gated deletion + fresh cleanup token for cancellation/timeout
(90-min ceiling exceeds the 1h App-token TTL and queueing behind
publish-release's global concurrency group).
Ported from ag-ui-protocol/ag-ui PR #1914 with cpk adaptations
(scopes from release.config.json; dry_run -> dry-run input mapping).
When a promote fails, the succeeded-service set is empty, so verify-prod
hits its skip branch (`exit 0`). The GitHub job result is therefore
`success`, and the notify step rendered `verify-prod=success` in the
#oss-alerts Slack message — a misleading green, since prod was never
probed.
verify-prod now exports a `status` output: `success` after a real probe
passes, `skipped` on the empty-CSV skip. notify reads that output (via
the new bats-tested verify-prod-display.sh) instead of the raw job
result, so the Slack line accurately reads `verify-prod=skipped` vs
`success` vs `failure`. A genuine probe failure / contract violation
exits non-zero (job result `failure`, status never written), and the
display falls back to the job result. Slack formatting is unchanged.
Extracts the display mapping into showcase/scripts/verify-prod-display.sh
(mirroring promote-fleet.sh) with red-green bats coverage, and adds it to
the showcase_validate.yml shellcheck step.
The standalone threads template is retired: the ENT-679 rollout was
reverted from main (#5215/#5216/#5217) and the settled model is base
langgraph-python + Intelligence activation overlay. The Intelligence
CLI (main and published copilotkit@3.0.2) no longer references it.
Also drops its two entries from .github/config-allowlist.txt.
The check-config-files job blocked the strands-crm showcase PR
(examples/showcases/strands-crm/frontend/next.config.ts was not in
the build-config allowlist). Adding it next to strands-file-analyzer.
## Summary
- Removes three project-level Claude Code config files from the
checked-out PR code immediately after checkout, before any other step
runs
- Closes three pre-model-load code execution vectors that a malicious PR
could exploit when a maintainer triggers the social copy generator
## What's removed and why
| File | Vector |
|------|--------|
| `.claude/settings.json` | `SessionStart` hooks execute as shell
commands before the model loads |
| `.claude/settings.local.json` | Same hook vector — gitignored but can
be force-committed to a branch |
| `.mcp.json` | MCP servers are launched as subprocesses before the
model loads in non-`--bare` mode |
All three fire before the model processes any prompt, so
`--allowedTools` restrictions don't protect against them.
## Context
The social copy generator checks out the PR's HEAD SHA and then runs
`claude -p` with `ANTHROPIC_API_KEY` in scope. An existing write-access
permission gate (a maintainer must manually check a checkbox to trigger
generation) limits who can pull the trigger — but doesn't prevent a
social-engineering attack where a seemingly legitimate PR contains a
malicious config file.
The step is placed immediately after `actions/checkout` so nothing in
the checked-out workspace can influence Claude Code before the
sanitization runs.
## Test plan
- [ ] Trigger the social copy generator on a normal PR and confirm it
still produces output
GitHub Actions expression string literals don't interpret `\n`, so
`toJSON(format('...\n...'))` emits literal `\\n` and Slack renders the
two characters backslash-n instead of a line break. Inject real newlines
via a `fromJSON('"\n"')` placeholder, matching the starter-smoke fix.
Fixes the "all builds failed" and "Showcase Build Failed" alerts in
showcase_build.yml and the multi-line "showcase_validate failed" alert
in showcase_validate.yml. showcase_promote.yml already used the
fromJSON placeholder; the single-line validate alert has no newlines
and was left untouched.
The starter smoke failure alert built its message with literal `\n`
inside a GitHub Actions `format()` call. GHA expression string literals
do not interpret `\n` as an escape, so `format()` emitted the two
characters backslash+n, which `toJSON()` then encoded as `\\n` — Slack
rendered a literal "\n" and the triple-backtick fence as plain text
instead of a line break and a code block.
Inject real newlines via `fromJSON('"\n"')` so `toJSON` encodes them as
a single `\n` Slack honors, and place the code-fence delimiters on their
own lines so the failure summary renders as a proper code block.
PocketBase had no CI build path: `ghcr.io/copilotkit/showcase-pocketbase`
was a stale April `:latest`, and there was no way to ship pb_migrations /
pb_hooks changes without an ad-hoc manual build. Add a `pocketbase` slot to
showcase_build.yml's build matrix, mirroring the harness/aimock entries:
- dispatch_name `showcase-pocketbase`, context `showcase/pocketbase`, its
own Dockerfile, health `/api/health`, railway_id from the SSOT.
- a paths-filter key gated to `showcase/pocketbase/**` so the slot only
rebuilds when PB's own files change (the image is self-contained — no
shared-module copy), not on every showcase push.
- the workflow_dispatch service choice so PB is human-targetable.
Flip the SSOT entry (railway-envs.ts) to `ciBuilt: true` with
`dispatchName: "showcase-pocketbase"` so it is built+pushed (`:sha` +
`:latest`) and joins the default staging-redeploy scope; the build's
redeploy step only touches the matrix-intersect-success set, so PB still
only redeploys when its own files change. Regenerate
railway-envs.generated.json and the showcase_promote.yml service dropdown,
and update the SSOT/redeploy tests that pinned PB as out-of-band
(CI_BUILT_SERVICES 25 -> 26; webhooks stays the only non-CI-built service).
Three pre-existing showcase_promote.yml issues surfaced by code review:
1. verify-prod's empty-SERVICES_CSV branch exited 0 vacuously, relying on the
unenforced assumption "promote already failed if the succeeded set is
empty". Pass PROMOTE_RESULT into the step env and fail loud on contract
violation: if promote reported success yet emitted no CSV, emit
"::error::promote reported success but succeeded_csv is empty — contract
violation" and exit 1. The genuine all-failed case keeps the ::notice:: +
exit 0.
2. Both verify-deploy.ts invocations (verify-staging-precondition, verify-prod)
ran `npm ci` in showcase/scripts but then `npx tsx showcase/scripts/...`
from the repo root, so npx could network-fetch tsx instead of using the
local install. Give those steps working-directory: showcase/scripts and the
relative path verify-deploy.ts, matching how resolve-targets/promote run
emit-railway-envs-json.ts. verify-deploy.ts takes no cwd-relative file args
(only --env/--services) and imports ./railway-envs relative to its own file,
so the cwd change is safe.
3. The notify job's Slack payloads embedded literal '\n' inside
toJSON(format('...\n...')). GitHub Actions expression string literals do not
interpret backslash escapes, so the '\n' survived toJSON as the two chars
\\n and Slack rendered it verbatim as "\n" instead of a line break.
EMPIRICALLY CONFIRMED against the live #team-showcase webhook: an existing
production post from a real promote run showed literal "\n", and a corrected
payload (real LF) rendered proper line breaks. Fixed BOTH posting steps by
injecting newlines via fromJSON('"\n"') (a real LF char) as a format()
argument instead of a literal '\n' in the template.
Mirror the failure-path 'no Slack' observability fallback for the new
#team-showcase success post. When SLACK_WEBHOOK_TEAM_SHOWCASE is unset
(current default), a successful prod promote now emits a ::notice:: log
line instead of a silent green.
Add a success-only Slack post to #team-showcase in the showcase_promote
notify job, mirroring the existing failure→#oss-alerts step. Guarded by
env.SLACK_WEBHOOK_TS so it safely no-ops until the SLACK_WEBHOOK_TEAM_SHOWCASE
org secret exists. Failures continue to route only to #oss-alerts.
showcase_build.yml's existing detect-changes job shared the workflow_dispatch
`service` input with the new detect-starter-changes job, so dispatching
`service=starter-<slug>` tripped detect-changes's fail-loud `exit 1` ("did
not match any entry in ALL_SERVICES") and reddened the run even though
build-starters published fine. Scope the showcase fail-loud to non-`starter-*`
inputs via a case statement (mirroring how the starter job scopes its own
fail-loud) so a `starter-*` dispatch resolves to an empty showcase matrix and
SKIPS; typo'd showcase service names still fail loud.
Restore the 6h `schedule` cron in test_smoke-starter.yml. The PR had removed
it, leaving NO post-merge floating-dependency breakage detector for starters
— and its harness-probe replacement depends on S5 Railway services that don't
exist yet. Keep the cron until S5 starter-service probing is confirmed live.
showcase_build.yml: add detect-starter-changes + build-starters jobs building
each of the 12 starters from examples/integrations/<slug>/Dockerfile via Depot
(--platform linux/amd64) and pushing ghcr.io/copilotkit/starter-<slug>:latest +
:<sha>. The starter- prefix is disjoint from showcase-* so harness discovery
stays clean.
test_smoke-starter.yml: drop the 6h schedule cron (live signal now comes from
the harness probing deployed Railway services + the harness alert path); keep
the examples/integrations/** PR build-sanity gate + offline aimock smoke run.
Workflows slot (S4) of the starter-row-group spec (model B). Railway
provisioning (S5) is intentionally out of scope, gated on cost approval.
## Summary
The showcase Railway promote CI flow could not promote the fleet: an
`all` promote was blocked end-to-end whenever a single service was red,
and the reported failure traced to `showcase-ag2` being chronically red
on staging.
Root causes and fixes:
- **ag2 crash-on-import (the red service).** `gen_ui_agent.py` carried
`from __future__ import annotations`, which stringified the `set_steps`
tool's `context_variables: ContextVariables` parameter into an
unresolved `ForwardRef`. AG2's tool-schema generation then raised
`PydanticUserError` at import time, so the process never came up and the
staging healthcheck failed on every deploy since 2026-05-31. Removed the
import (matching the working sibling agents) and added a regression test
that statically asserts the future-import stays absent
(version-independent) plus a live import check.
- **Promote loop was all-or-nothing.** The per-service loop ran under
`set -euo pipefail`, so the first failing service aborted the whole
`all` promote, leaving the rest unpromoted. Extracted the loop into
`showcase/scripts/promote-fleet.sh`, which attempts every service,
accumulates succeeded/failed sets, exits non-zero only after attempting
all, and exports `succeeded_csv`.
- **`verify-prod` defeated the best-effort design.** It was skipped on
any non-zero promote and verified the full requested set. It now runs
`if: !cancelled()` and scopes `--services` to the succeeded set.
- **Staging precondition blocked the fleet.**
`verify-staging-precondition` failed the whole `all` promote when any
one service was staging-red. It is now advisory — `promote` runs
regardless, and `bin/railway`'s per-service P2/P3 staging-green gates
authoritatively refuse red services while green services promote.
`notify` success keys on PROMOTE && PROD.
- **Regression tests now gate in CI.** Added a `shell-script-tests` job
(bats + shellcheck) to `showcase_validate.yml`, plus input-validation
hardening in the script (fail-loud on empty / all-empty CSV,
`RAILWAY_BIN` executability check, whitespace trim).
## Test plan
- [x] ag2 regression test passes in the 3.12 venv (`PYTHONPATH=".:src"
pytest tests/python/`) — 2/2
- [x] `promote-fleet.bats` — 12/12 (best-effort loop, succeeded_csv
export, empty/whitespace/missing-binary guards, digest forwarding)
- [x] `shellcheck promote-fleet.sh` clean; `actionlint` clean on both
workflows
- [ ] CI green on this PR
Add a notify job that runs after both publish lanes via always() and
computes release intent directly from the github.event payload,
independent of the build jobs, so an infra failure in a lane can't
swallow the alert. The job posts the builder's rendered message to
#oss-alerts and includes a best-effort self-watchdog Slack post.
the per-service promote loop ran under `set -euo pipefail`, so the first failing
service aborted the whole `all` fleet promote; extracted to promote-fleet.sh
which attempts every service, accumulates succeeded/failed sets, exits non-zero
only after attempting all, and exports succeeded_csv. verify-prod now runs
`if: !cancelled()` and scopes --services to the succeeded set; the staging
precondition is advisory (promote runs even when it reports red — bin/railway
enforces staging-green per-service); notify success keys on PROMOTE && PROD.
Adds a shell-script-tests CI job (bats + shellcheck) and input-validation
hardening (fail-loud on empty/all-empty CSV, RAILWAY_BIN check, whitespace trim).
The starter smoke-test Slack alert had no indication of where it came
from, which is ambiguous when the same workflow runs across multiple
repos (e.g. the public CopilotKit/CopilotKit repo vs the internal
testybara fork). Prepend a `[ci:<owner/repo>]` tag derived from
github.repository so triage is unambiguous about the source. These CI
alerts test example source and carry no staging/production dimension,
so [ci] is the meaningful source axis. Existing message format is
otherwise preserved.
Convert the promote workflow's `service` input from a freeform string
(default "all", the accidental-fleet-promote footgun) to a generated
`type: choice` dropdown whose first/default option is a rejected sentinel
so a blind "Run workflow" aborts instead of promoting.
resolve-targets: reject the sentinel, fail loud on ambiguous matches
(no silent head -n1), independently re-filter probe.prod, and reject
--digest combined with `all`. promote: run unattended in CI
(--yes --non-interactive) — the manual dispatch + service selection is
the human authorization. notify: empty-webhook guard + fallback warning,
neutral state for sentinel-abort and any cancellation, and surface
resolve-targets.result in the failure alert for triage.
Add showcase/scripts/sync-promote-service-options.ts: generates the
promote workflow's service `choice` options from the SSOT
(railway-envs.ts), spliced between BEGIN/END markers in
showcase_promote.yml. Fail-loud throughout — every emitted token must
resolve to exactly one service under the resolve-step predicate
(name|dispatchName match AND probe.prod), tokens are YAML-safe, args are
strict (a typo'd flag cannot trigger a destructive write), and markers
are validated before any rewrite.
Wire it into a lefthook pre-commit hook (regenerate + restage; set -e so
a failed regen blocks the commit) and an advisory (never-failing) drift
check in showcase_validate.yml. Vitest coverage for ordering, exclusion,
collision/ambiguity guards, marker errors, exit codes, idempotency, and
the import-side-effect guard.
The format job checked out github.head_ref without setting repository,
so it defaulted to the base repo. For fork PRs the head branch only
exists on the fork, making checkout fail with 'a branch or tag with the
name <branch> could not be found' (e.g. PR #5099). Resolve repository to
the head repo on pull_request events; same-repo PRs are unchanged, so
the same-repo-guarded auto-format push-back still works.
deploy.yml's notify-harness step was sending {state, services, ...},
but the harness ingest schema at showcase/harness/src/http/webhooks/
deploy.ts is .strict() and requires succeeded/failed/cancelled arrays
+ bool. Every push-to-main notify-harness POST has been 400ing as
invalid-payload (unknown key 'state') since the schema landed.
Build the payload from the redeploy gate's success-and-error buckets:
- Emit failed_services alongside ok_services from the redeploy-gate
step (services whose status==error), and expose both as resolve-matrix
job outputs.
- Rewrite the notify-harness payload to drop state and emit:
succeeded=ok_services, failed=failed_services, cancelled=(verify
conclusion==cancelled), services=union of succeeded and failed.
- Omit buildRunId/buildRunUrl entirely when empty (the schema runs
.url() on buildRunUrl, so an empty string would be rejected). Same
treatment for runUrl as defense-in-depth.
- Group the no-summary echoes through a single >>GITHUB_OUTPUT block
so shellcheck SC2129 stays clean once a third echo is added.
## Summary
Hardening pass over the showcase deploy pipeline and its operator
tooling. Five focused areas:
- **`bin/railway` rollback shell-injection fix** —
`RollbackCommitCommand` shelled out via backtick subshells interpolating
the `--sha`/`--env` flags. Replaced with `IO.popen` argument arrays (no
shell), with `--sha` validated against an anchored hex regex and `--env`
validated against the known env set before any git invocation. Both `git
ls-tree` and `git show` now gate on `$?.exitstatus` before parsing their
output (a failed `git show` previously merged stderr into stdout and fed
it to `YAML.safe_load`). New spec covers injection rejection, git-show
failure, and empty-snapshot paths.
- **Promote snapshot encapsulation** — replaced the bare `@full_*`/`@*`
snapshot ivars in `PromoteCommand` with named `fleet_*`/`target_*`
accessors so fleet-scoped vs per-service snapshot selection is
name-enforced rather than comment-enforced.
- **Branded `Host` type** — `ProbeTarget.host` is now a branded `Host`
produced only by `asHost` (rejects scheme, path, empty, whitespace,
userinfo, query, fragment), branded at the verify-pipeline ingress;
`ProbeTarget` fields are `readonly`.
- **`deploy-to-railway.ts` SSOT migration** — uses the shared
`resolveRailwayToken` and sources project/env IDs from the railway-envs
SSOT; reads the GHCR username from env (fail-loud) instead of a
hardcoded handle.
- **Workflow safety + observability** — collapsed the promote workflow
to an input-agnostic concurrency group so promotes can't race the same
Railway service; added `#oss-alerts` failure notifications to the build
and validate workflows (which previously had silent red paths).
- **Harness probe test fixes** — repointed the aimock fixture-coverage
probe to the per-integration D4/D6/shared layout and aligned
d5-multimodal assertions to the current auto-send sentinels.
## Test plan
- [ ] CI: Validate Showcase, build-checks, python tests green
- [ ] CI: static/quality (oxfmt + lint + typecheck) green
- [ ] Local: Ruby promote suite (`ruby spec/all_tests.rb`) — 103/342
green
- [ ] Local: showcase-scripts vitest green; harness probe tests green
- [ ] Local: oxfmt --check clean; actionlint clean
Adds two CI signals for keeping the published packages small and broadly compatible:
- Bundle size: size-limit file-mode config across packages plus a
CopilotChat import-size regression signal (gzip) so growth in the
headline consumer entrypoint is visible on every PR. A bundle-size
workflow comments results on the PR (Phase 1: no hard-fail).
- ES compatibility: a compat-check (es-check) script across 9 packages
with a root .browserslistrc, validating built .mjs/.cjs against the
es2022 build target.
The measure script is importable (measureBundle) and unit-tested. Dev
docs live under dev-docs/ (bundle-size.md, browser-compat.md). All
action refs are pinned to full commit SHAs for supply-chain safety.
The new no-webhook log step interpolated github.ref directly into a shell
echo, which zizmor flags as template-injection (HIGH). Bind it to a step
env var REF and reference $REF in the script instead. Message text is
unchanged; no Co-Authored-By.
Collapse the promote workflow to an input-agnostic concurrency group so promotes can't race the
same Railway service; add #oss-alerts failure notifications to the build and validate workflows
(build via extended needs, validate via a new workflow-level notify job).
Jordan Ritter