Wire the strands-typescript showcase integration for staging deployment,
mirroring how the Python strands integration is deployed.
- manifest: flip deployed: true so the shell lists it in the integration menu
- railway-envs.ts: add showcase-strands-typescript SSOT entry (staging-only
for now: prod instance not yet provisioned, so it omits the prod env and is
gateIgnore'd until promoted dual-env); regenerate railway-envs.generated.json
- showcase_build.yml + showcase_build_check.yml: add the strands-typescript
build matrix entry, change-detection filter, and dispatch option (railway_id
is the new Railway service id)
- golden fixture + image-ref-gate inventory tests updated for the new service
Railway staging service showcase-strands-typescript provisioned
(showcase-strands-typescript-staging.up.railway.app, health /api/health,
OpenAI-via-aimock env). Prod is added later via the promote pipeline.
Fail loud on a dropped #oss-alerts page: the failure-alert cross-post no longer
swallows a 200/ok:false Slack response, so a dropped page-the-humans alert reds
the renderer job instead of vanishing on a green run. The thread reply stays
warn-only. Both posts capture the response via a shared slack_alert_posted_ok
predicate, mirrored byte-identically across the live workflow and the dry-run
helper.
Debt cleanup: drop a dead failed_count var, correct a misleading gha_url comment,
and validate the decoded blob run_id against ^[0-9a-f]{6}$ in the render step so
a malformed run_id can't reach Slack or the run name.
Tests: predicate edge cases (non-JSON, malformed, missing/null ok), an anti-drift
parity guard asserting the predicate is identical in both files, and call-site
tests locking the #oss-alerts fail-loud vs thread warn-only exit semantics.
Promote-notify Slack message: name the promoted AND failed services (one
Failed: header + bullets), legible "(N): <names>" count, real wall-clock
elapsed (integer-coerced), and drop the constant verify-prod legend line.
Durable healthcheckPath: track it per-service/env in the SSOT (railway-envs),
re-assert it on the promote pin path (omit-when-absent, never null), and route
deploy-to-railway provisioning through isTrackedService/resolveProvisionHealthcheck
so a tracked-null service omits the healthcheck while an untracked one keeps the
/api/health default — fixing the silent prod-healthcheck drift that refused aimock.
Tests: ruby pin-reassert spec + deploy-to-railway healthcheck spec + emit/golden/accessor.
Repairs TypeScript check-types across the monorepo and adds a CI gate so
regressions are caught going forward:
- core: bundler module resolution and strict-mode fixes
- sdk-js: bundler module resolution; keep codegen, formatter, packaging working
- react-core: fixes across components, hooks, and tests
- react-native: restore catch binding referenced by TypeError cause
- runtime: repair check-types and bound AI SDK schema inference
- web-inspector: nodenext import extensions, export Anchor
- remaining packages and node example: assorted check-types repairs
- deps: add missing type-only devDependencies
- license context driven from /info licenseStatus
- ci: run check-types in the static quality workflow
Squashed from 12 commits for a single, easily-revertable change.
## Summary
- Mint a GitHub App token for the stable release workflow and reuse it
for PR creation and follow-up API calls
- Disable lefthook during automation commits so release PR generation
does not depend on local developer hooks
- Relax the CopilotChat perf regression test to assert correctness
without a hard 5s wall-clock check
## Testing
- Unit/UI test updated to allow longer async rendering while still
verifying 100 messages render successfully
- Not run (not requested)
promote-fleet.sh now emits a base64 results JSON (schema_version=1) with
both succeeded[] and failed[] alongside the existing succeeded_csv /
staging_drift outputs. showcase_promote.yml's notify job replaces the old
inline two-state (success/failure) notifier — which dumped the full
requested CSV and mislabeled any partial promote as a blanket Failed — with
a single dispatch of the previously-orphaned showcase_promote_notify.yml
renderer (success / partial / total). The dispatch enriches promote-fleet's
results blob with run context (6-hex run_id, trigger=workflow, operator,
pre_staging) and authenticates via the devops-bot App token (actions:write),
mirroring canary.yml — the default GITHUB_TOKEN cannot start new workflow runs.
promote-fleet has no failure taxonomy, so each failed entry uses the default
category "promote-failed".
## Summary
An **on-demand** tool to answer "is prod caught up with staging right
now?". The showcase deploy model is **staging = mutable `:latest`**
(continuously rebuilt) and **prod = immutable `@sha256:`** (advances
only on an explicit promote), so a prod column can sit **behind** a
green staging.
**There is no scheduled drift alert — by design.** Prod lagging staging
is **often intentional**: changes are batched and promoted deliberately,
so a recurring "N columns stale" alert would be pure noise. This tool is
therefore manual-only: a maintainer runs it when they want to check, and
it tells them the current state.
- **`bin/railway reconcile-prod`** — for every prod-eligible
(`probe.prod == true`) service, compares the **prod serving digest**
(the `@sha256:` from
`SnapshotCommand.build_snapshot(PRODUCTION_ENV_ID)`) against the
**staging running digest** (reuses
`PromoteCommand#staging_running_digest`, the same source the promote pin
uses). Classifies each:
- `green` — prod == staging (in sync)
- `stale` — prod != staging **and** staging is resolvable (prod is
behind a green staging)
- `gray` — staging running digest not resolvable, or the service has no
prod snapshot entry yet — informational, **not** stale
- Prints a readable per-service table + summary; **exits nonzero iff any
service is stale**; `--json` for machine output. **Read-only — no
promotes/mutations.**
- **`showcase/scripts/reconcile-prod-gate.sh`** — wrapper mirroring
`lint-prod-gate.sh`: surfaces the table into `$GITHUB_STEP_SUMMARY`,
optionally captures `--json` to `RECONCILE_JSON`, and propagates the
exit-code verdict (never swallows a non-zero).
- **`.github/workflows/showcase_reconcile.yml`** — **`workflow_dispatch`
only** (no cron). Regenerates the SSOT JSON (`EMIT_SKIP_OXFMT=1`, same
as the promote workflow's resolve/promote jobs), runs the gate with the
Railway/GHCR auth env, renders the reconcile table to the **GH step
summary**, and uploads the `--json` as a `reconcile-json` artifact. **No
Slack.** The run exits nonzero on a stale column so a manual run visibly
flags drift. `timeout-minutes: 10`.
- **Tests** — Ruby minitest (`test_reconcile_prod.rb`: classification +
exit-code + `--json` shape + dispatcher registration) and a bats gate
test (`reconcile-prod-gate.bats`). Wired the gate script into the
`showcase_validate.yml` shellcheck list.
### What changed from the original scheduled-alert design
The first cut of this PR shipped a daily cron + auto-post to #oss-alerts
on any stale column. Per owner feedback, that was reshaped to on-demand
only: the `schedule:` trigger and the Slack-on-stale step were removed
(intentional/deliberate staleness is not a bug, so an unsolicited
recurring alert is noise). The CLI command, the gate wrapper, and all
tests are unchanged.
## Gates
- `ruby showcase/bin/spec/test_reconcile_prod.rb` → **9 runs, 20
assertions, 0 failures**
- `bats showcase/scripts/__tests__/reconcile-prod-gate.bats` → **6 ok**
- `shellcheck -s bash showcase/scripts/reconcile-prod-gate.sh` →
**clean**
- `actionlint .github/workflows/showcase_reconcile.yml` → **clean** (the
pre-existing `depot-ubuntu-24.04-4` custom-runner-label warning is on
`showcase_validate.yml`, predates this PR — my only change there is one
line in the shellcheck list)
## Test plan
- [ ] CI green (Ruby suite, bats suite, actionlint/shellcheck,
commitlint)
- [ ] Optional: read-only `workflow_dispatch` run of
`showcase_reconcile.yml` to confirm it runs against live prod/staging
(safe — no mutations)
## What
Adds **`@copilotkit/bot-whatsapp`** — a WhatsApp Business **Cloud API**
`PlatformAdapter` for the platform-agnostic `@copilotkit/bot` engine —
plus a runnable **`examples/whatsapp`** app and docs. This brings
WhatsApp to the bots ecosystem alongside the existing Slack support,
reusing the engine, the `@copilotkit/bot-ui` IR, and the pluggable
`ActionStore` untouched.
## How it works
- **Ingress:** the adapter owns its own HTTP server — GET verification
handshake (`hub.challenge`) + POST intake validated by
`X-Hub-Signature-256` HMAC (timing-safe), acked `200` immediately then
processed async.
- **No streaming:** WhatsApp messages are immutable, so the run renderer
**buffers** text and sends once on `TEXT_MESSAGE_END`
(`supportsStreaming: false`; `update()` posts fresh, `delete()` no-ops).
- **Interactive mapping:** text/section → text; ≤3 buttons →
reply-button message; `Select` or 4–10 actions → list message; >10 →
numbered-text fallback. A control's `value` round-trips by encoding it
into the reply id (`ck:…::<json>`), since WhatsApp replies carry no
value field; oversized encodings fail loud rather than corrupt silently.
- **Memory:** WhatsApp exposes no readable history, so a pluggable
**`HistoryStore`** (default `InMemoryHistoryStore`) holds it and replays
it into `agent.messages` each turn (fresh threadId per turn, mirroring
`bot-slack`). Swap in a durable backend to persist across restarts.
- **Commands:** leading-keyword matching (`commandPrefix`, default `/`);
the command text is injected via the engine's `runAgent({ prompt })`
path (not persisted at ingress).
- **Inbound media** → AG-UI multimodal content parts; **HITL** via
interactive replies.
## Example
`examples/whatsapp` mirrors `examples/slack`: a CopilotKit
`BuiltInAgent` over MCP (Linear + Notion), with `issue_list`, an
interactive `show_incident`, and a `confirm_write` HITL gate.
## Tests & verification
- 62 unit tests across the package (render mapping, markdown→WhatsApp,
signature verification incl. wrong-but-equal-length, interaction
decode/round-trip, buffered renderer, webhook listener/server, stores,
media, adapter).
- `build` ✅, package `check-types` ✅, `publint`/`attw` (ESM-only) ✅,
example `check-types` ✅. Full `nx run-many -t test
--projects=packages/**` passes.
- Two rounds of code review (APPROVE) — fixed slash-command history
double-append and silent value-truncation; minors (HMAC over raw bytes,
conversationKey invariant, offset-correct Blob, unused-dep pruning,
added tests).
## Docs
Package `README.md` + `ARCHITECTURE.md`, example setup guide (Meta app +
webhook + tunnel), and a `shell-docs` WhatsApp guide page (registered in
`meta.json` + early-access gate).
## Notes / out of scope (v1)
- No template-send path for messaging outside WhatsApp's 24-hour
customer-service window (documented limitation).
- Pre-existing, unrelated `@copilotkit/core` `phoenix-observable.ts`
typecheck error exists on the branch base (missing `@types/phoenix`) —
not introduced here.
Prod sitting behind staging is often intentional (changes are batched and
promoted deliberately), so a recurring drift alert is noise. Reshape the
reconcile workflow to manual-only:
- Remove the daily `schedule:` cron trigger — leave only `workflow_dispatch`.
- Remove the auto-Slack-on-stale step (and its SLACK_WEBHOOK env / stale_line
output derivation) — no unsolicited #oss-alerts post on mere staleness.
- A manual run surfaces the reconcile table to the GH step summary, keeps the
cheap `--json` capture as an uploaded artifact, and still exits nonzero on a
stale column so a manual run visibly flags drift.
- De-noise the gate script + bats comments that referenced the removed
scheduled/Slack behavior.
The on-demand CLI (`bin/railway reconcile-prod`), the gate wrapper, and the
Ruby + bats tests are unchanged.
Lever 1 of the promote-reliability hardening plan. The showcase deploy
model is staging=mutable :latest (continuously rebuilt), prod=immutable
@sha256: (advances only on explicit promote), so a prod column can
silently fall BEHIND a green staging — drift today is only noticed by
eyeballing a dead column. This adds proactive, automatic detection.
- bin/railway reconcile-prod: for every prod-eligible (probe.prod==true)
service, compares the prod SERVING digest (LintProd snapshot path) vs
the staging RUNNING digest (reuses PromoteCommand#staging_running_digest).
Classifies green/stale/gray, prints a table + summary, exits 1 iff any
stale. --json for machine output. Read-only: no promotes/mutations.
- scripts/reconcile-prod-gate.sh: wrapper mirroring lint-prod-gate.sh —
surfaces the table to the GH step summary, captures JSON for the Slack
builder, propagates the exit-code verdict.
- .github/workflows/showcase_reconcile.yml: daily cron + workflow_dispatch;
runs the gate; on stale services posts the stale-column list to
#oss-alerts (SLACK_WEBHOOK_OSS_ALERTS) via the fromJSON('"\n"') idiom.
- Tests: Ruby minitest (classification + exit-code, RED-anchored on a
drift-blind classifier) and a bats gate test. Wired the gate script
into the showcase_validate.yml shellcheck list.
Post-promote convergence verification is deferred to a fast-follow.
## Summary
- `upload-artifact`'s `!**/node_modules/**` filters are post-walk: the
action still descends into every `node_modules` and stats every file
(~6M with pnpm's `.pnpm/` symlink farm) before applying negations. That
enumeration is the actual bottleneck — `Upload workspace` runs 10+
minutes even with the filters added in #5044.
- Replace the filtered upload with: `rm -rf` the heavy dirs
(`node_modules`, `.nx`, `.turbo`, `.next`), `tar -czf /tmp/workspace.tgz
.`, upload that single file. Publish job `tar -xzf`'s it after download
and continues unchanged.
- Applied symmetrically to `prerelease.yml` and `publish-release.yml`.
## Measured impact
Verified on a dry-run dispatch of `release / pre` against this branch
([run
26531785850](https://github.com/CopilotKit/CopilotKit/actions/runs/26531785850)):
| Step | Before (run 26529550757) | After (this PR) |
| ------------------------------- | ------------------------ |
--------------- |
| Upload workspace | ~800s (cancelled) | **3s** |
| Pack workspace | — | 9s |
| Download workspace | — | 1s |
| Unpack workspace | — | 1s |
| **Total artifact round-trip** | **~800s** | **14s** |
- `Upload workspace` step alone: **~99.6% reduction (~267× faster)**.
- Full pack/upload/download/unpack pipeline vs the prior single upload:
**~98% reduction (~57× faster)**.
The 800s baseline is from a cancelled run, so both numbers are
conservative.
## Test plan
- [x] Dispatch `release / pre` against this branch with `dry_run=true`
- [x] Confirm `Upload workspace` completes in seconds instead of 10+ min
- [x] Confirm publish job `Unpack workspace` restores the tree and `pnpm
install` succeeds
- [x] Confirm dry-run publish step exits clean (no missing files from
the tarball round-trip)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## What
Adds **cvdiag** — a permanent, always-available observability subsystem
for the showcase, built to diagnose the red↔green cell flap on the
staging dashboard and to make that diagnosis a dashboard query rather
than a multi-day forensic hunt in the future.
Captures the full request path with `X-Test-Id` correlation across
**probe → backend → aimock → edge**, across every integration
(TypeScript, Python, Java/spring-ai, .NET):
- Per-language backend emitters (canonical + staged/compile-linked
mirrors), all sharing one schema (`schema.json`, closed-world
`additionalProperties:false`).
- CREATE-only writes to two new PocketBase collections: `cvdiag_events`
and `cvdiag_raw_byte_samples` (additive migrations — no existing data
touched).
- An 8-class flap classifier mapping to the observed failure signatures
(`sse-missing` / `text-unstable` / `dom-missing`).
- DEBUG-tier raw-byte capture (secret-scrubbed) and HMAC-guarded A/B
edge-interference detection.
## Why
The runId flap-fix (`cdc1e90e`, 2026-06-09) did **not** fully resolve
the flap — it was still observed 2026-06-19. cvdiag exists so the
*remaining* cause is observed live with full correlation instead of
inferred.
## Safety / enablement
- **Inert by default.** With `CVDIAG_BACKEND_EMITTER` unset the
subsystem performs zero host mutation (no logging-config changes, no
threads/tasks, no stdout) — verified by
`test_cvdiag_inert_when_disabled`. **To accumulate data, set
`CVDIAG_BACKEND_EMITTER=1` on the showcase services.**
- All per-language scrubbers match the canonical `scrubSecrets`
(sk-/base64url, Bearer, colon-less URL userinfo, size-guard) — verified
with real toolchains (vitest / mvn / dotnet).
- Merged latest `main` (only conflict: a clean `.csproj` include union).
## Verification
- harness `tsc --noEmit` ✓ · `src/cvdiag` vitest 251/251 ✓ ·
`cvdiag-stage-ts --check` in-sync ✓
- Java MessageScrubber 17/17 (mvn) ✓ · .NET CvdiagBackend 5/5 (dotnet
sdk:9.0) ✓ · Python emitters 93/93 (3.12) ✓
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The showcase_promote.yml resolve-targets job runs
`emit-railway-envs-json.ts`, which shells out to the repo-root
`node_modules/.bin/oxfmt` to produce oxfmt-canonical JSON. That job's
`npm ci` runs in showcase/scripts only and never installs the root oxfmt
binary, so every promote dispatch died at "Generate SSOT artifact" with
`spawnSync .../node_modules/.bin/oxfmt ENOENT` (exit 1) — blocking ALL
promotes, including the team's regular shell-docs promote, since the last
green run on 2026-06-18.
The emitted JSON on the resolve-targets / promote path is EPHEMERAL: it
is parsed in-memory by jq (resolve-promote-targets.sh) and bin/railway to
pick the promote target and is NEVER committed, so oxfmt-canonical
formatting is irrelevant there. Add an explicit `EMIT_SKIP_OXFMT=1`
opt-out that returns the raw `JSON.stringify` form, and set it on both
ephemeral workflow steps.
The DEFAULT (committed-artifact) path is unchanged: oxfmt stays REQUIRED
and fails loud if the binary is absent, because the committed
railway-envs.generated.json must stay oxfmt-canonical or CI's
static_quality.yml `oxfmt --check` auto-format bot fires on the drift.
This is opt-IN-to-skip, never silent-on-absence.
Call sites of emit-railway-envs-json.ts:
- showcase_promote.yml resolve-targets — EMIT_SKIP_OXFMT=1 (this fix).
- showcase_promote.yml promote — EMIT_SKIP_OXFMT=1 (this fix).
- static_quality.yml committed-artifact `--check` — unset, oxfmt required.
- resolve-verify-matrix.ts (showcase_deploy.yml) — only invokes the
emitter when the committed JSON is absent; the checkout always has it,
so the default (oxfmt) path is correct and unchanged.
Tests: 2037 showcase/scripts tests pass; 2 new EMIT_SKIP_OXFMT unit tests
assert the skip path emits valid (raw) JSON; the existing oxfmt-canonical
golden tests still gate the committed path.
Regenerated the stale committed showcase_promote.yml so the 12 starter-*
services (+ shell-docs and all existing targets) appear in the service
dispatch choice list (fixes HTTP 422 on
gh workflow run -f service=starter-*). Reverted isProdPromotable to
env-map-only (environments.prod.probe), equivalent to the workflow resolve
predicate. Added a regression test asserting shell-docs + all 12 starters
remain in the generated AND committed dropdown.
## Summary
Fixes two correctness bugs in the showcase staging→prod promote path
(`showcase/bin/railway`), discovered + live-validated while promoting
the full 19-service cluster.
**Bug #1 — promote pinned the wrong digest.** `resolved_prod_image`
re-resolved the mutable `:latest` tag against *current* GHCR instead of
pinning the digest staging is actually *running*
(`latestDeployment.meta.imageDigest`). When `:latest` drifted after
staging deployed, promote pushed an unvalidated (and once, regressed)
image to prod. Now pins staging's running digest. Adds a loud `⚠️
STAGING DRIFT` warning (promote stdout + `STAGING_DRIFT_MARKER:` →
`promote-fleet.sh` aggregation → both Slack payloads) when staging's
running digest ≠ current `:latest`, so the gap is visible without
blocking the promote.
**Bug #2 — the pin never activated.** `pin_and_verify` used
`serviceInstanceRedeploy`, which replays the *existing* deployment's old
image rather than the just-pinned `source.image`. Config showed the new
digest while prod kept serving the old one (this is why earlier promotes
"succeeded" while prod stayed broken). Switched to
`serviceInstanceDeployV2` + a new `verify_serving_digest!` gate that
polls the new deployment to SUCCESS and **fails loud** if the running
digest ≠ pinned.
Plus CR-round hardening: P2 in-flight race check now reads
`meta.imageDigest` (was dead on tag-form staging) and skips on
`--digest` override; `detect_staging_drift` fails loud (WARN) on
GHCR-resolve failure instead of swallowing; `--digest` override
suppresses spurious drift; `drift_line` LF-stripped at the
`GITHUB_OUTPUT` boundary; multi-service drift join fixed; fallback-log
drift preserved.
## Commits
1. `fix(showcase): pin prod to staging's running digest + loud
staging-drift warning`
2. `fix(showcase): activate prod pin via serviceInstanceDeployV2 +
verify running==pinned`
3. `fix(showcase): harden promote P2 race check + refresh ivar-lint
allowlist`
## Validation
- Live red-green on real Railway: claude-sdk-python prod flipped from
stale → pinned digest via the fixed CLI; all 19 cluster services
promoted green; 5 previously-degraded backends (ag2, llamaindex,
pydantic-ai, ms-agent-python, strands) recovered.
- 15/15 Ruby specs + 15/15 bats green; ruby -c, shellcheck (CI
invocation), actionlint clean.
- 3-round cr-loop (7 agents/round) converged to zero bucket-(a).
## Follow-ups (not in this PR)
- Defensive `.to_s.empty?` on `meta.imageDigest` extraction
(can't-happen on real Railway).
- Test-quality nits (capture_io scoping, weak bats glob, shared mock
fixture).
- Pre-existing verify-prod `succeeded_csv`/`GITHUB_OUTPUT` coupling
(graceful-degraded, audited STAY_IN_C).
- `deploy-to-railway.ts` births prod on `:latest` (the root provisioning
gap); CLI↔workflow notify equivalence (PR2).
## Test plan
- [ ] CI green on PR HEAD
- [ ] (post-merge) a real `gh workflow run showcase_promote.yml` shows
the drift line in the #team-showcase notification when staging is behind
:latest
## Summary
- remove the retired top-level `docs/` Next app and the disabled
docs-sync workflow/script
- add `docs -> showcase/shell-docs` as a top-level symlink for `cd docs`
muscle memory without restoring the old docs tree
- move the docs model allowlist into `showcase/shell-docs/` and retarget
docs validation/doctest extraction to shell-docs content
- update docs/agent guidance and CI path filters so `docs/` is treated
as an alias, not an active separate docs surface
- tighten the pre-commit package check so non-package docs/tooling
changes do not fan out into the full package matrix
## Validation
- `pnpm exec tsx scripts/validate-doc-model-names.ts`
- `pnpm exec tsx scripts/doc-tests/extract.ts`
- `pnpm exec vitest run
scripts/__tests__/validate-doc-model-names.test.ts
scripts/doc-tests/__tests__/extract.test.ts
showcase/harness/src/cli/eval/scope.test.ts`
- `pnpm exec oxlint showcase/harness/src/cli/eval/scope.test.ts
scripts/doc-tests/extract.ts`
- `git diff --cached --check` before follow-up commit
- `test "$(readlink docs)" = "showcase/shell-docs"`
- `test -f docs/package.json`
- `pnpm exec oxfmt --check .claude/docs/documentation.md
.claude/docs/hooks.md AGENTS.md CLAUDE.md CONTRIBUTING.md
showcase/shell-docs/README.md`
- commit hooks passed
## Notes
- historical docs remain recoverable from
`archive/docs-save-do-not-prune` and `archive/docs-retired-2026-06-17`
- I intentionally left Vercel/project teardown out of this PR; this is
repo cleanup only
The stable-release (create-pr), publish-release, and canary workflows all
used a target-agnostic concurrency group, so releasing one scope (e.g.
`monorepo`) would queue behind an unrelated scope's run (e.g. `angular`)
despite touching disjoint packages and tags. Fold the release target into
each concurrency key so different scopes run in independent lanes while
same-scope runs stay serialized (cancel-in-progress: false), preserving the
version-bump / tag-push / npm-publish race protection.
publish-release also fires on merged release PRs where `inputs.scope` is
empty; the PR branch `release/publish/<scope>/v<version>` carries the scope
via `github.head_ref`, so the key stays scope-distinct on that path too.
ENT-950
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bug #1: resolved_prod_image re-resolved :latest at promote time, so prod
could be pinned to a digest different from what staging is actually serving.
Now resolve via staging_running_digest (latestDeployment meta.imageDigest)
and REFUSE when unavailable. Add detect_staging_drift +
emit_staging_drift_warnings to surface (non-fatally) when :latest has moved
past staging's running digest; aggregate markers across the fleet in
promote-fleet.sh (printf join) and plumb the drift_line through
showcase_promote.yml (both Slack payloads + fallback log + GITHUB_OUTPUT
single-line guard). Skip drift detection on the --digest override path.
The plugin version pins to packages/runtime/package.json, but neither the
lefthook glob nor the plugin-skills-check workflow path filters watched
that file. A routine release bump to the runtime version therefore matched
no trigger and the pin silently rotted between releases (the root cause of
the version drift). Add packages/runtime/package.json to the lefthook glob
and to both the push and pull_request path filters so a version bump
re-runs check:plugin-skills.
Add a bot-whatsapp release scope to release.config.json and the matching
workflow_dispatch scope dropdowns in publish-release / stable-release / canary,
so the package can be released via the manual CI trigger like bot-slack.
Adds .github/workflows/showcase_promote_notify.yml — workflow_dispatch-only
Slack notify workflow for promote results. Posts initiation + threaded reply
to #team-showcase; cross-posts to #oss-alerts on partial/total failure.
Also adds showcase_promote_notify.dry-run.sh — a local render-logic mirror
that decodes the same payload and prints the messages the workflow would
send to Slack, without making any Slack API calls. Used by the pre-merge
runbook and by future CI fixture tests.
## Summary
Ports
[ag-ui-protocol/ag-ui#1914](https://github.com/ag-ui-protocol/ag-ui/pull/1914)
to CopilotKit — plus the two supporting guard files ag-ui already had:
- **`.github/workflows/canary.yml`** — discoverable **`canary /
publish`** `workflow_dispatch` orchestrator. Any maintainer can publish
a prerelease of the branch they're on straight from the Actions tab. It
is a thin orchestrator — it does **not** publish to npm itself:
1. Guards against `main` and non-branch refs.
2. Mints the devops-bot App token (app-id `1108748`,
`DEVOPS_BOT_PRIVATE_KEY`) with scoped `contents:write` +
`actions:write`.
3. Mirrors the dispatched ref to a unique
`canary/<slug>-<run_id>-<attempt>` branch via the GitHub API (no
checkout).
4. Dispatches **`publish-release.yml --ref canary/<slug> -f
mode=prerelease …`**, locates the run, and waits (`gh run watch
--exit-status` + explicit conclusion check).
5. Deletes the canary ref — status-gated (never yanks the ref under a
still-running delegated run) with a fresh cleanup token (90-min job
ceiling exceeds the 1h App-token TTL).
- **`scripts/release/verify-release-scope-dropdowns.sh`** — drift guard:
the hand-maintained `scope` dropdowns in `publish-release.yml` /
`stable-release.yml` / `canary.yml` must exactly match
`release.config.json`'s `.scopes` keys. Parsers fail loud and distinct
on structural changes instead of silently passing.
- **`.github/workflows/lint-release-workflows.yml`** — actionlint +
shellcheck + the dropdown-sync job over the release pipelines.
### Why a separate orchestrator (and not a flag in publish-release.yml)
- A GitHub Environment's deployment-branch policy is evaluated against
the ref a run is **triggered on** — not branches created mid-run. The
orchestrator exists to get the publish run *onto* a `canary/*` ref.
- `publish-release.yml` holds the **single npm OIDC trusted-publisher
binding**; a second publishing entry point would break OIDC for every
`@copilotkit/*` package. The orchestrator never touches npm.
- The cross-workflow dispatch uses the **App token, not `GITHUB_TOKEN`**
— `GITHUB_TOKEN`-authenticated events never start new workflow runs.
**Note:** the `npm` environment currently has *no* deployment-branch
policy, so the orchestrator is a convenience wrapper today. Tightening
the policy to `main` + `canary/*` + `release/publish/*` (matching
ag-ui's security posture) is being applied as repo configuration
alongside this PR — requires admin. This PR includes the prerequisite:
`publish-commit.yml` (pkg-pr-new) is removed from the `npm` environment,
since it runs on every PR and would be blocked by the policy (it
publishes to pkg.pr.new, not npm, and uses no environment secrets).
## Testing done
- Drift guard: positive run against all three real workflows; negative
tests (scope removed → drift FAIL with diff; bogus scope → FAIL; `case
"${SCOPE}"` quoting refactor → loud parser-degradation FAIL; whole case
block deleted → loud zero-block FAIL; quoted arm `"angular")` →
accepted; blank/comment lines inside `options:` → still parsed; prose
comments mentioning case/SCOPE/in → no false positive).
- `shellcheck` clean at all severities; `bash -n` on every workflow
`run:` block; YAML parses.
- 3 rounds of 7-agent code review converged to zero load-bearing
findings.
## ⚠️ Still to verify before first real use
- [ ] devops-bot App (id 1108748) has **Actions: write** — required for
the in-workflow `gh workflow run`. Safe first test: dispatch once with
`dry_run=true`.
- [ ] First `dry_run=false` run clears the `npm` environment end-to-end
via the App token once the deployment-branch policy is tightened.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Post-merge follow-ups (maintainer action required)
These need repo **admin** rights and must happen **in this order**:
1. **Merge this PR first.** `main`'s current `publish-commit.yml`
(pkg-pr-new) still sits in the `npm` environment and runs on every PR
touching `packages/**` — applying the branch policy before this PR lands
would block every snapshot publish. This PR removes that environment
association.
2. **Tighten the `npm` environment's deployment-branch policy** to
`main` + `canary/*` + `release/publish/*` (matching ag-ui). With an
admin-scoped token:
```bash
gh api --method PUT repos/CopilotKit/CopilotKit/environments/npm \
-F "deployment_branch_policy[protected_branches]=false" \
-F "deployment_branch_policy[custom_branch_policies]=true"
gh api --method POST
repos/CopilotKit/CopilotKit/environments/npm/deployment-branch-policies
-f name="main" -f type=branch
gh api --method POST
repos/CopilotKit/CopilotKit/environments/npm/deployment-branch-policies
-f name="canary/*" -f type=branch
gh api --method POST
repos/CopilotKit/CopilotKit/environments/npm/deployment-branch-policies
-f name="release/publish/*" -f type=branch
```
Or via UI: Settings → Environments → npm → Deployment branches and tags
→ "Selected branches and tags" → add the three patterns above.
Why these three: `main` covers stable `workflow_dispatch` retries and
`stable-release.yml`; `release/publish/*` covers the merged-release-PR
runs (the run's head branch is the release PR branch); `canary/*` covers
the orchestrator's delegated prerelease runs. After this, direct
`mode=prerelease` dispatches from arbitrary feature branches stop
working — the `canary / publish` orchestrator becomes the one-click path
(by design).
3. **Verify the devops-bot App (id `1108748`) has `Actions: write`**
(org/App settings). The orchestrator's `gh workflow run` dispatch 403s
without it. Safe end-to-end test, after step 2: Actions tab → **canary /
publish** → pick any feature branch, any scope, **`dry_run=true`** →
confirm the delegated `release / publish` run is created, watched, and
the `canary/*` ref is deleted afterward.
4. **First real canary** (`dry_run=false`) confirms the npm OIDC publish
clears the environment gate end-to-end on a `canary/*` ref.
Run `test / unit` over only the packages affected since the base instead
of building + testing every package 3× across the Node 20/22/24 matrix on
every PR.
- fetch-depth: 0 so affected has a merge-base to diff against.
- Derive NX_BASE/NX_HEAD: PR → merge-base with the base branch tip; push →
github.event.before with a HEAD~1 fallback.
- Select packages via `nx show projects --affected --projects='packages/**'`
fed to run-many (the `nx affected` run form ignores --projects and pulls
in downstream examples/storybook — hence the show-projects → run-many split).
- workflow_dispatch still runs all packages (manual/full run).
- Editing this workflow can't surface as an affected package, so a change to
test_unit.yml in the range now forces a full all-packages run — this keeps
the build/test path exercised on the PR that changes it.
- GitHub context passed via env: (not inline ${{ }}) to satisfy zizmor;
NX_VERBOSE_LOGGING forced off for the JSON-parsing step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The repo migrated off @changesets/* to conventional-commit-driven releases.
scripts/release/lib/changes.ts::getChangesSummary reads `git log <lastTag>..HEAD`
commit subjects and never touches .changeset/. No .changeset/config.json,
no @changesets/* in any package.json, no npm scripts reference it.
The 11 .changeset/*.md files describe changes that have either already
shipped (via commit subjects in prior releases) or will ship in the next
release (via the current commit subjects in the v1.60.1..main window) —
the .changeset/ files are inert.
Also removes the dead workflow `paths:` filters in
test_e2e-dojo.yml and test_e2e-legacy-v1.yml that re-fired e2e on
.changeset/ changes — nothing to fire on after the directory is gone.
Bumps the dojo e2e matrix from `depot-ubuntu-24.04` (2 vCPU) to
`depot-ubuntu-24.04-4` (4 vCPU) and `NX_PARALLEL: 4` so the build uses
the extra cores. This is the non-serializing way to cut dojo wall-clock
(the build-once dedup tried in #5450 regressed wall-clock and was
reverted).
## Result: −40% wall-clock (measured on CI)
Dojo wall-clock = the single slowest suite (the 15 run in parallel).
Comparison vs the 2-vCPU baseline:
| metric | 2-vCPU baseline | 4-vCPU | Δ |
|---|---|---|---|
| **wall-clock** (long pole `langgraph-python`) | 623s (10.4m) | **373s
(6.2m)** | **−40%** |
| runner-minutes (wall summed, 15 suites) | 105m | 76m | −28% |
| **billed compute** (vCPU-min; 4-vCPU ≈ 2× rate) | ~210 | ~304 |
**+45%** |
Every suite got faster; the long-pole suites benefited most:
| suite | 2-vCPU | 4-vCPU |
|---|---|---|
| langgraph-python | 623s | 373s |
| langgraph-typescript | 547s | 362s |
| langgraph-fastapi | 500s | 337s |
| adk-middleware | 414s | 286s |
| (… all 15 faster …) | | |
Long-pole `langgraph-python` step breakdown:
| phase | 2-vCPU | 4-vCPU |
|---|---|---|
| Build cpk | 82s | 48s |
| Prep dojo | 94s | 52s |
| **Run tests (Playwright)** | **271s** | **117s** |
| total | 623s | 373s |
**Key finding:** the Playwright phase more than halved → the e2e suites
are **CPU/worker-bound, not LLM-latency-bound**. A bigger runner is the
right lever; test sharding is not needed to reach ~6 min.
## Trade-off
−40% wall-clock for **~+45% billed compute** (4-vCPU costs ~2×/min,
partly offset by finishing 28% sooner). If the cost bump isn't worth it
across all 15 suites, a follow-up can scope `-4` to just the slow suites
via a per-matrix `runner` field (wall ~6.5m, smaller cost increase).
Companion to #5450 (unit-test `nx affected`).
Measure wall-clock impact of a larger Depot runner on the e2e suites.
The long pole (langgraph-python, ~10.4min) spends ~50% on build/prep
(CPU-bound) and ~43% on the Playwright run. 2->4 vCPU + NX_PARALLEL 4
should speed build/prep; the test-phase gain reveals whether it is
CPU-bound (big win) or LLM-latency-bound (then sharding is the lever).
The injected/streamed a2ui fixtures all included catalogId, so aimock
replay never exercised the basic-catalog fallback that broke production
(real models omit catalogId per the tool-usage guide). Strip catalogId
from the langgraph-python sales-dashboard secondary-call fixtures and
hard-assert "Catalog not found" is absent outside the charts-rendered
soft branch, so the spec fails without a route defaultCatalogId.
Also repoint the on-demand e2e workflow at the d4/d5-recorded/d6/shared
fixture dirs — it still referenced feature-parity.json, deleted in the
1e66a5f8d fixture reorg, so every /test-aimock run died at aimock start.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ran Shemtov