## Two silent-failure gaps in the showcase build/deploy/notify pipeline
These are **pre-existing** silent-failure holes surfaced in code review
(not
caused by any recent PR). This PR fixes the two load-bearing ones.
### 1. Green-but-zero-redeploy (silent "we thought we shipped but
didn't")
The `redeploy-staging` job computes the redeploy set as the intersection
of the
build matrix and the build-success set. This job **only runs when
`aggregate-build-results.outputs.any_success == 'true'`** (job-level
`if:`
guard). So if that intersection comes back **EMPTY**, it does NOT mean
"nothing
to deploy" — it means at least one slot built successfully yet none of
those
successes maps back to a matrix `dispatch_name`. That's a
`dispatch_name`↔
`service` contract skew (the aggregator's `service` values and the
matrix's
`dispatch_name` values drifted apart).
The old code emitted `services=` (empty) and exited 0 → the build went
**GREEN
while redeploying NOTHING**, silently.
**Fix:** on an empty intersection in this any_success-guaranteed step,
fail loud
(`::error::` + `exit 1`) with a diagnostic naming both sides of the
skew.
The legitimate "nothing changed / nothing succeeded" no-op paths are
guarded at
the **job level** (`has_changes=='true' && any_success=='true'`), so the
fixed
step never runs there — no false-red.
### 2. Starter build failures had no alert surface (invisible failures)
The `notify` job's `needs` (and its `if: failure()`) omitted
`detect-starter-changes` and `build-starters`, and `build-starters`
wrote no
per-slot build-result artifact. So a **failed starter image build
produced NO
Slack alert and NO PR comment** — it shipped silently.
**Fix:**
- Added `detect-starter-changes` + `build-starters` to `notify.needs` so
`if: failure()` sees a starter build failure → Slack alert + PR comment.
- Gave `build-starters` a per-slot build-result artifact **mirroring the
main
`build` matrix** (same `{service,status}` shape, `cancelled→skipped`
normalization, `if: always()`, `if-no-files-found: error`), using a
**distinct `starter-build-result-*` prefix** so it never matches the
aggregator's `build-result-*` download pattern (starters must not
pollute the
showcase redeploy set keyed by `dispatch_name`).
### Red / Green
**Finding #1** — extracted the step's shell/jq logic and drove it with
synthetic
inputs:
RED (pre-fix), any_success=true + empty intersection:
```
No services in matrix ∩ success-set — skipping redeploy.
Computed services CSV (matrix ∩ build-success):
EXIT=0 # $GITHUB_OUTPUT: services= -> silent pass, redeploys NOTHING
```
GREEN (post-fix), same inputs:
```
::error::Build succeeded (any_success=true) but matrix ∩ success-set is EMPTY — dispatch_name/service contract skew; nothing would be redeployed.
Successful build service values: ["shell-RENAMED","mastra-RENAMED"]
Scheduled matrix dispatch_name values: ["shell","mastra"]
EXIT=1 # fails loud
```
No-regression: non-empty intersection → `EXIT=0 ; services=shell`. The
nothing-changed/nothing-succeeded paths are skipped at the job level
(never
reach the step) → no false-red.
**Finding #2** — modeled `if: failure()` (fires iff any `needs` job
result is
`failure`):
```
BEFORE (starters NOT in needs), starter=failure -> notify fires = False (INVISIBLE, the bug)
AFTER (starters IN needs), starter=failure -> notify fires = True (FIXED)
AFTER no-regression, starters=skipped, all green -> notify fires = False (quiet)
```
### Validation
- `python3 yaml.safe_load` parses OK.
- `actionlint`: only pre-existing findings remain (matrix jq SC2086 +
the known
`depot-ubuntu-24.04-4` runner-label warning); no new errors in edited
regions.
- `yamllint`: only pre-existing line-length/document-start/truthy
warnings.
### Scope
Touches **only** `.github/workflows/showcase_build.yml`, and only these
two
concerns. Does NOT touch the `shell_dashboard` paths-filter region (PR
#5955's
domain), nor the other backlog debt (false-root-cause comment,
double-alert,
check-lockfile guard). Self-contained; not stacked on #5955.
Two pre-existing silent-failure gaps in the showcase build/deploy/notify
pipeline (surfaced in code review):
1. Green-but-zero-redeploy: the redeploy-staging job computes the redeploy
set as (build matrix ∩ build-success). This job only runs when
any_success=='true', so an EMPTY intersection means builds succeeded but
none maps to a matrix dispatch_name — a dispatch_name/service contract
skew. The old code emitted an empty services= and exited 0, going GREEN
while redeploying nothing. Now it fails loud with a diagnostic naming both
sides of the skew. The legitimate nothing-changed/nothing-succeeded no-ops
stay guarded at the job level, so they are unaffected.
2. Starter-failure-invisible: the notify job's needs omitted build-starters,
so a failed starter image build produced no Slack alert and no PR comment.
Added detect-starter-changes + build-starters to notify.needs, and gave
build-starters a per-slot build-result artifact mirroring the main build
matrix (distinct starter-build-result-* prefix so it never pollutes the
showcase aggregator's build-result-* set).
PR #5952 (9a8cf615) added explicit `.js` extensions to the relative imports
inside the harness's shared cell-model fold
(showcase/harness/src/shared/cell-model/{cell-model,live-status,staleness}.ts)
— REQUIRED for the harness's pure-Node-ESM runtime and correct as-is.
But the dashboard re-exports that fold via shims
(showcase/shell-dashboard/src/lib/{cell-model,live-status,staleness,format-ts}.ts
`export * from "../../../harness/src/shared/cell-model/*"`), pulling the fold
into the dashboard's `next build`. `export *` does not rewrite the fold's
INTERNAL `.js` edges, and the dashboard's empty next.config.ts had no
extensionAlias, so webpack resolved `./live-status.js` literally, found only
the `.ts` source, and failed:
Module not found: Can't resolve './live-status.js'
Module not found: Can't resolve './staleness.js'
Module not found: Can't resolve './format-ts.js'
> Build failed because of webpack errors
Two-part fix (one coherent subject):
1. Resolution: add `webpack.resolve.extensionAlias` to
showcase/shell-dashboard/next.config.ts so `.js`/`.mjs` specifiers resolve
to `.ts`/`.tsx`/`.mts` sources — the bundler complement to TS NodeNext's
`.js`-import convention. Covers the `next build` (webpack) path CI uses.
The harness fold's `.js` imports are left untouched (they are correct).
2. CI gap: the dashboard build did not run on #5952 because the build matrix
is path-filtered and #5952 only touched `showcase/harness/**`, which
selects `showcase_harness` but not `shell_dashboard`. Add
`showcase/harness/src/shared/**` to the `shell_dashboard` paths-filter so
any change to the shared fold the dashboard compiles in also selects the
dashboard build — a fold change can never again ship an unbuilt dashboard.
Local red-green proof:
- RED (main, before fix): `next build` in showcase/shell-dashboard emitted the
4 fold-resolve errors above.
- GREEN (after extensionAlias): same build → 0 fold-resolve errors; the fold
resolves. Remaining `@/data/*.json` errors are the prebuild-generated files
(generate-registry/probe-docs) skipped in the local repro, produced in CI's
Docker build — unrelated to this fix.
The boot-smoke previously failed only when the thrown error carried a
module-RESOLUTION code (ERR_MODULE_NOT_FOUND + siblings, walked through
the cause chain / AggregateError members) and passed everything else.
That defaults-to-pass on module-EVALUATION crashes — a top-level throw,
an await-rejection, a bad named binding, or a SyntaxError — none of which
carry a resolution code, so a real boot-crashing regression of that class
would ship green.
The smoke runs `node -e "import('./dist/orchestrator.js')"` with
process.argv[1] UNSET, so bootFleet() (the env/PocketBase validation that
legitimately throws) never runs — only the module graph is linked and
evaluated. A clean build therefore loads with no thrown error, so ANY
error thrown by import() here is a boot regression and must fail the gate.
Now: any rejection -> BOOT_FAIL / exit 1 (resolution AND
evaluation/link/binding/syntax/top-level-throw). Successful load ->
BOOT_OK / exit 0. The collectErrorCodes cause/AggregateError walk is
retained ONLY to label the failure ("module-resolution failure" vs
"boot/evaluation failure") — both exit 1, richer diagnostics preserved.
Kept process.exit(0) on success, the timeout 120s wrapper, and
timeout-minutes: 5 (a hang still fails).
No-false-red proof: built the real harness dist and ran the strict guard
against the real dist/orchestrator.js under node -e (argv[1] unset) —
BOOT_OK, exit 0, ~0.28s, no hang, confirming a clean graph loads without
throwing and the strict guard does not false-red real CI.
The boot-smoke gate classified pass/fail using only the top-level `e.code`.
A module-resolution error that arrives WRAPPED — nested in `e.cause`
(possibly a chain), bundled inside an `AggregateError` (`e.errors[]`), or
rethrown without preserving `.code` at the top — showed no code to the
`MODULE_RESOLUTION_CODES.has(e.code)` check and was misclassified as
BOOT_OK, defeating the gate.
Add a `collectErrorCodes` helper that gathers every code reachable from
the thrown error: the error itself, its cause chain (recursively), and any
AggregateError members (recursively), with a depth cap to bound cause
cycles. If ANY collected code is a module-resolution code -> BOOT_FAIL /
exit 1. Purely additive to the FAIL set: direct top-level codes still
redden, and a benign non-resolution runtime error (e.g. the
`HARNESS_ROLE must be set` guard, which carries no such code anywhere)
still passes as BOOT_OK / exit 0. The `timeout 120s` wrapper,
`timeout-minutes: 5`, and success/expected-error `process.exit(0)` are
unchanged.
Local red-green (classifier extracted to a temp file, driven against
synthetic errors):
- RED (top-level-only): wrapped cause -> BOOT_OK exit 0 (swallowed);
AggregateError member -> BOOT_OK exit 0 (swallowed).
- GREEN (hardened): wrapped -> exit 1; aggregate -> exit 1; direct
ERR_MODULE_NOT_FOUND -> still exit 1; benign ERR_INVALID_ARG_TYPE and
HARNESS_ROLE error -> BOOT_OK exit 0; real built dist/orchestrator.js ->
BOOT_OK exit 0 in <200ms (prompt exit, no hang).
The boot-smoke step only treated ERR_MODULE_NOT_FOUND as failure, so other
module-resolution regressions (ERR_UNSUPPORTED_DIR_IMPORT,
ERR_PACKAGE_PATH_NOT_EXPORTED, ERR_UNKNOWN_FILE_EXTENSION,
ERR_INVALID_MODULE_SPECIFIER) were swallowed as BOOT_OK/exit 0 — the very
class of bug this gate exists to catch could slip through. It also had no
process.exit(0) on the success/expected-error paths and no bounded timeout,
so a future open handle at import time could hang node -e to the job's
25-minute ceiling.
- Broaden the failure condition to a MODULE_RESOLUTION_CODES set (any of the
five codes => BOOT_FAIL, exit 1). Non-module-resolution runtime errors
(e.g. the HARNESS_ROLE env guard, no such code) stay BOOT_OK/exit 0.
- Add explicit process.exit(0) on both the success and expected-error paths.
- Wrap the node invocation in `timeout 120s` (non-zero on timeout => step
fails) and add step-level timeout-minutes: 5.
Red-green proof (extracted guard logic vs synthetic modules): current logic
passes ERR_UNSUPPORTED_DIR_IMPORT / ERR_PACKAGE_PATH_NOT_EXPORTED at exit 0
(RED gap); hardened logic fails all five codes at exit 1, keeps benign
runtime error at exit 0, and against the real built dist/orchestrator.js
reports BOOT_OK and exits promptly (779ms, no hang).
CI missed the extensionless-import regression because tsc (bundler
resolution), vitest, and tsx all resolve extensionless relative
specifiers fine — no existing step ever ran the real node dist module
graph, which is what the container actually does at boot.
Add a boot-smoke to the Validate Showcase job (already gated on
showcase/harness/**): after building the harness dist, load
dist/orchestrator.js via a node import() and fail hard on
ERR_MODULE_NOT_FOUND. A later runtime error from missing env/PocketBase
is expected and passes — only a module-resolution failure reddens the
build. Verified red-green: the guard exits 1 on the pre-fix
extensionless imports and 0 once the .js extensions are added.
Round-3 CR fixes for the LGT persistence-disable preload.
HIGH-1: after installing the fs-write patches, import the node:fs/promises
namespace and assert each patched member is identity-equal to the installed
function; throw (fail boot) naming any mismatched member. Catches the
load-order case where fs/promises was linked before the reassignment and the
namespace snapshotted the original fn (silent bypass -> disk-growth recurrence).
HIGH-2: make the real-package behavioral test non-skippable under
LGT_REQUIRE_BEHAVIORAL=1 (missing runtime fails, not skips), and wire the
python-unit-tests job to set up Node, npm install the agent deps, and run the
langgraph-typescript pytest with that flag so a green check proves interception.
LOW: tolerant writer-shape guard regex (quote/whitespace/alias agnostic; still
trips on a named-import switch); read-only open/openSync reject ENOENT for
suppressed paths (write-intent still no-ops); mkdir recursive returns the
topmost-created dir per the real fs contract.
Adds a HIGH-1 guard-fires regression test.
Four workflow steps installed CLI tools ad-hoc via `npm install -g`, which
zizmor's `adhoc-packages` audit flags (install outside a lockfile). Replace
each with a lockfile-managed or pinned-action install, preserving behavior:
- aimock (test_integration-docs, test_e2e-showcase-on-demand): invoke the
workspace-pinned @copilotkit/aimock `llmock` bin from the frozen lockfile
(already a dep of @copilotkit/showcase-scripts) instead of `npm install -g`.
Kept lockfile-devDep rather than the CopilotKit/aimock composite action:
the action wraps the newer config-only `aimock` CLI and can't do the
multi-`--fixtures` / `--validate-on-load` / `/__aimock/health` invocation
these jobs need.
- claude-code (social_copy-generator): pin @anthropic-ai/claude-code as a root
devDependency, install from the frozen lockfile, invoke via its documented
cli-wrapper.cjs entrypoint. Kept lockfile-devDep rather than
anthropics/claude-code-action: the job uses claude as a scripted `-p` CLI,
not PR/issue automation.
- oxfmt (static_quality): already a root devDependency; install from the frozen
lockfile and put node_modules/.bin on PATH instead of `npm install -g`.
- ruff (static_quality): switch `pipx install` to the pinned official
astral-sh/ruff-action@278981a (v4.1.0) with the same 0.15.13 version.
zizmor --min-severity low --config .github/zizmor.yml .github/workflows:
before: exit 12, 4 adhoc-packages findings
after: exit 0, 0 adhoc-packages findings, 0 unpinned-uses (no findings)
OSS-451 shipped because nothing linked a demo page's CopilotKit runtimeUrl
to the existence of the /api route it names. The only automatic pre-merge
gate for showcase/** is a Docker build, which compiles a page that
references a non-existent route just fine (runtimeUrl is an unchecked
string) — so the page-404-on-load class was invisible.
Add a static validator (validate-runtime-routes.ts) that, for every SHIPPED
demo (a demo listed in its integration's manifest `features`), asserts its
runtimeUrl resolves to a real route dir under src/app/api. Unshipped /
experimental demos (not in `features`) and not_supported_features are
skipped, so incomplete placeholders don't fail the gate — but promoting one
into `features` immediately starts enforcing it. A baseline file can
grandfather pre-existing violations; the fleet is currently clean (0).
Wire it into a new pre-merge workflow (showcase_validate-wiring.yml) that
runs on every showcase/integrations PR alongside the build check. Add it to
branch-protection required checks to make it blocking.
Regression test proves it flags the exact OSS-451 shape (shipped demo,
missing route) while passing existing/base routes and skipping unshipped.
Verified: npm run validate-routes -> clean fleet-wide; removing the 3
OSS-451 routes -> flags exactly those 3; full showcase/scripts vitest suite
(2151 tests) green.
Refs OSS-451
Renames the Bots SDK to the Channels SDK. Names only — no behavior change.
- 8 packages @copilotkit/bot* -> @copilotkit/channels* (git mv dirs, names,
workspace: cross-deps). Now includes @copilotkit/bot-intelligence ->
@copilotkit/channels-intelligence (landed on main via #5761; unpublished, so
renamed fresh with the family).
- release.config.json scope keys + versionSource; ReleaseScope union;
canary/stable-release/publish-release scope dropdowns; verify script
- examples/slack (Kite) + examples/teams: deps, jsxImportSource, imports
- showcase/shell-docs: content dirs docs/bots->docs/channels and
reference/bot->reference/channels, nav registry, redirects
createBot and other API names unchanged. Old @copilotkit/bot* to be deprecated
after the new packages publish (bot-intelligence was never published).
Re-derived onto latest main (was conflicting after #5761 landed).
Refs OSS-438
Brings the 499-commit-stale foundations branch up to date with main so #5761
has a clean diff and no stale reverts (e.g. forwardHeaders). Conflicts:
- CopilotThreadsDrawer.tsx: took main's (main renamed CopilotDrawer -> ThreadsDrawer
+ added the collapse feature; the branch's edit was a no-op import-type split).
- pnpm-lock.yaml: regenerated with the pinned pnpm 10.33.4 (adds @copilotkit/bot-intelligence).
The shell-script-tests job installs bats via apt. GitHub's ubuntu-latest runner
image preconfigures third-party apt repos (Microsoft / azure-cli) for
preinstalled tooling this job never uses. When one of those repos serves invalid
release metadata, `apt-get update` exits non-zero and `bash -e` aborts the step
before bats installs — even though bats comes from Ubuntu's own `universe` repo,
which is unaffected.
This job only needs Ubuntu packages, so remove those unused third-party repos
before `apt-get update`.
Remove the Dependabot `github-actions` ecosystem config plus its
companion `dependabot-auto-merge` and `dependabot-major-analysis`
workflows. Renovate (via `renovate.json` → `local>CopilotKit/renovate`,
Dependency Dashboard #592) now owns github-actions updates.
Also cleans stale references to the deleted files:
- `.github/zizmor.yml`: drop the `dangerous-triggers` ignores for the
two dependabot workflows, remove the now-empty `dependabot-cooldown`
rule, and update the `unpinned-uses` comment to reference Renovate.
- `.github/workflows/security_zizmor.yml`: drop the
`.github/dependabot.yml` path triggers.
`.github/dependabot.yml` contained ONLY the github-actions ecosystem, so
it is deleted in full. No npm/pip/docker or other ecosystem was touched
— npm is untouched.
Rebased onto current main; all CI green (zizmor pass, commitlint pass,
build/types/unit/package-quality all pass).
Remove the Dependabot github-actions ecosystem config and its companion
auto-merge / major-analysis workflows. Renovate (via
renovate.json -> local>CopilotKit/renovate, Dependency Dashboard #592)
now owns github-actions updates.
Also clean stale references to the deleted files:
- zizmor.yml: drop dangerous-triggers ignores for the two dependabot
workflows, remove the now-empty dependabot-cooldown rule, and update
the unpinned-uses comment to reference Renovate.
- security_zizmor.yml: drop the .github/dependabot.yml path trigger.
npm and other ecosystems are untouched (dependabot.yml had only the
github-actions ecosystem).
Move the Intelligence-delivered managed-bot surface out of @copilotkit/bot into
its own package so the adapter, transports, contracts, and lifecycle ship
independently of bot core.
- New @copilotkit/bot-intelligence: intelligenceAdapter + DeliverySource/EgressSink
(+ in-memory impls) + placeholder contracts + startManagedBots/validation/
activation metadata. Production code imports only types from @copilotkit/bot
and @copilotkit/bot-ui.
- @copilotkit/bot keeps the generic PlatformCodec seam (moved to src/codec.ts) and
all core createBot changes (addAdapter, deferred store, id fields,
__managed/skipIngressDedup, exclusive guard). It now also exports the
FakeAdapter/FakeAgent test utilities for downstream adapter-package tests.
- Registered the new release scope: release.config.json, scripts/release/lib/
config.ts, and the canary/publish/stable release workflows.
Tests preserved: bot 150 + bot-intelligence 19 (= the prior 169); bot-slack 261.
Builds typecheck across bot/bot-intelligence/bot-slack/runtime; publint/attw/
oxlint/oxfmt clean.
The format job set format_fixed=true from a whole-tree git diff and then
git-committed only the scoped PR files. When a PR's own files are already
formatter-clean but the runner's working tree is dirty for an unrelated
reason (e.g. an LFS smudge on examples/teams/appPackage/*.png, which are
*.png filter=lfs), the whole-tree diff falsely triggered the commit path
while the scoped git add staged nothing, so git commit exited 1 and failed
the job. This intermittently red-flagged any PR depending on per-runner
LFS-smudge state (cf. #5715).
- Trigger format_fixed only when a SCOPED file actually changed.
- Guard the commit so an empty staged set is a no-op (exit 0) instead of a
hard failure.
The showcase-strands-typescript integration was staging-only: it had no
production Railway serviceInstance, so the prod D6 dashboard column showed
a uniform false-red (every cell errorClass=goto-error, backendUrl="") —
the probe navigated a bare relative path because the harness had no prod
health record / backendUrl to discover.
Provisions the prod serviceInstance (8a50728e-6119-43c4-b59c-d9535b6717a4,
domain showcase-strands-typescript-production.up.railway.app, healthcheck
/api/health, image pinned to the GHCR @sha256 digest, OPENAI_BASE_URL at
prod aimock) and brings the SSOT to the dual-env showcase-strands shape:
- railway-envs.ts: add the prod env entry with the real instanceId,
gateValidated:true, drop gateIgnore, remove the legacyJsonCompat
prod-domain placeholder.
- railway-envs.generated.json: regenerated (prod instanceId/domain, probe.prod
true, prod healthcheck; moved into the promote closure, tier 2).
- railway-envs.golden.json: regenerated to include the new prod (service,env)
pair (intentional behavior change, not a refactor regression).
- showcase_promote.yml: dropdown regenerated to list strands-typescript.
- verify-railway-image-refs.test.ts / redeploy-env.test.ts: update the
gateValidated/scope counts (39->40 gate targets, prod default 38->39) and the
now-stale staging-only comments.
RED->GREEN (live prod): BEFORE /api/health 404, prod PocketBase
health:strands-typescript totalItems:0, the 3 named D6 cells all
errorClass=goto-error backendUrl="". AFTER /api/health 200, prod PocketBase
health:strands-typescript present (status:200, valid url),
verify-railway-image-refs OK 80 instances.
Wire the strands-typescript showcase integration for staging deployment,
mirroring how the Python strands integration is deployed.
- manifest: flip deployed: true so the shell lists it in the integration menu
- railway-envs.ts: add showcase-strands-typescript SSOT entry (staging-only
for now: prod instance not yet provisioned, so it omits the prod env and is
gateIgnore'd until promoted dual-env); regenerate railway-envs.generated.json
- showcase_build.yml + showcase_build_check.yml: add the strands-typescript
build matrix entry, change-detection filter, and dispatch option (railway_id
is the new Railway service id)
- golden fixture + image-ref-gate inventory tests updated for the new service
Railway staging service showcase-strands-typescript provisioned
(showcase-strands-typescript-staging.up.railway.app, health /api/health,
OpenAI-via-aimock env). Prod is added later via the promote pipeline.
Fail loud on a dropped #oss-alerts page: the failure-alert cross-post no longer
swallows a 200/ok:false Slack response, so a dropped page-the-humans alert reds
the renderer job instead of vanishing on a green run. The thread reply stays
warn-only. Both posts capture the response via a shared slack_alert_posted_ok
predicate, mirrored byte-identically across the live workflow and the dry-run
helper.
Debt cleanup: drop a dead failed_count var, correct a misleading gha_url comment,
and validate the decoded blob run_id against ^[0-9a-f]{6}$ in the render step so
a malformed run_id can't reach Slack or the run name.
Tests: predicate edge cases (non-JSON, malformed, missing/null ok), an anti-drift
parity guard asserting the predicate is identical in both files, and call-site
tests locking the #oss-alerts fail-loud vs thread warn-only exit semantics.
Promote-notify Slack message: name the promoted AND failed services (one
Failed: header + bullets), legible "(N): <names>" count, real wall-clock
elapsed (integer-coerced), and drop the constant verify-prod legend line.
Durable healthcheckPath: track it per-service/env in the SSOT (railway-envs),
re-assert it on the promote pin path (omit-when-absent, never null), and route
deploy-to-railway provisioning through isTrackedService/resolveProvisionHealthcheck
so a tracked-null service omits the healthcheck while an untracked one keeps the
/api/health default — fixing the silent prod-healthcheck drift that refused aimock.
Tests: ruby pin-reassert spec + deploy-to-railway healthcheck spec + emit/golden/accessor.
Repairs TypeScript check-types across the monorepo and adds a CI gate so
regressions are caught going forward:
- core: bundler module resolution and strict-mode fixes
- sdk-js: bundler module resolution; keep codegen, formatter, packaging working
- react-core: fixes across components, hooks, and tests
- react-native: restore catch binding referenced by TypeError cause
- runtime: repair check-types and bound AI SDK schema inference
- web-inspector: nodenext import extensions, export Anchor
- remaining packages and node example: assorted check-types repairs
- deps: add missing type-only devDependencies
- license context driven from /info licenseStatus
- ci: run check-types in the static quality workflow
Squashed from 12 commits for a single, easily-revertable change.
## Summary
- Mint a GitHub App token for the stable release workflow and reuse it
for PR creation and follow-up API calls
- Disable lefthook during automation commits so release PR generation
does not depend on local developer hooks
- Relax the CopilotChat perf regression test to assert correctness
without a hard 5s wall-clock check
## Testing
- Unit/UI test updated to allow longer async rendering while still
verifying 100 messages render successfully
- Not run (not requested)
promote-fleet.sh now emits a base64 results JSON (schema_version=1) with
both succeeded[] and failed[] alongside the existing succeeded_csv /
staging_drift outputs. showcase_promote.yml's notify job replaces the old
inline two-state (success/failure) notifier — which dumped the full
requested CSV and mislabeled any partial promote as a blanket Failed — with
a single dispatch of the previously-orphaned showcase_promote_notify.yml
renderer (success / partial / total). The dispatch enriches promote-fleet's
results blob with run context (6-hex run_id, trigger=workflow, operator,
pre_staging) and authenticates via the devops-bot App token (actions:write),
mirroring canary.yml — the default GITHUB_TOKEN cannot start new workflow runs.
promote-fleet has no failure taxonomy, so each failed entry uses the default
category "promote-failed".
## Summary
An **on-demand** tool to answer "is prod caught up with staging right
now?". The showcase deploy model is **staging = mutable `:latest`**
(continuously rebuilt) and **prod = immutable `@sha256:`** (advances
only on an explicit promote), so a prod column can sit **behind** a
green staging.
**There is no scheduled drift alert — by design.** Prod lagging staging
is **often intentional**: changes are batched and promoted deliberately,
so a recurring "N columns stale" alert would be pure noise. This tool is
therefore manual-only: a maintainer runs it when they want to check, and
it tells them the current state.
- **`bin/railway reconcile-prod`** — for every prod-eligible
(`probe.prod == true`) service, compares the **prod serving digest**
(the `@sha256:` from
`SnapshotCommand.build_snapshot(PRODUCTION_ENV_ID)`) against the
**staging running digest** (reuses
`PromoteCommand#staging_running_digest`, the same source the promote pin
uses). Classifies each:
- `green` — prod == staging (in sync)
- `stale` — prod != staging **and** staging is resolvable (prod is
behind a green staging)
- `gray` — staging running digest not resolvable, or the service has no
prod snapshot entry yet — informational, **not** stale
- Prints a readable per-service table + summary; **exits nonzero iff any
service is stale**; `--json` for machine output. **Read-only — no
promotes/mutations.**
- **`showcase/scripts/reconcile-prod-gate.sh`** — wrapper mirroring
`lint-prod-gate.sh`: surfaces the table into `$GITHUB_STEP_SUMMARY`,
optionally captures `--json` to `RECONCILE_JSON`, and propagates the
exit-code verdict (never swallows a non-zero).
- **`.github/workflows/showcase_reconcile.yml`** — **`workflow_dispatch`
only** (no cron). Regenerates the SSOT JSON (`EMIT_SKIP_OXFMT=1`, same
as the promote workflow's resolve/promote jobs), runs the gate with the
Railway/GHCR auth env, renders the reconcile table to the **GH step
summary**, and uploads the `--json` as a `reconcile-json` artifact. **No
Slack.** The run exits nonzero on a stale column so a manual run visibly
flags drift. `timeout-minutes: 10`.
- **Tests** — Ruby minitest (`test_reconcile_prod.rb`: classification +
exit-code + `--json` shape + dispatcher registration) and a bats gate
test (`reconcile-prod-gate.bats`). Wired the gate script into the
`showcase_validate.yml` shellcheck list.
### What changed from the original scheduled-alert design
The first cut of this PR shipped a daily cron + auto-post to #oss-alerts
on any stale column. Per owner feedback, that was reshaped to on-demand
only: the `schedule:` trigger and the Slack-on-stale step were removed
(intentional/deliberate staleness is not a bug, so an unsolicited
recurring alert is noise). The CLI command, the gate wrapper, and all
tests are unchanged.
## Gates
- `ruby showcase/bin/spec/test_reconcile_prod.rb` → **9 runs, 20
assertions, 0 failures**
- `bats showcase/scripts/__tests__/reconcile-prod-gate.bats` → **6 ok**
- `shellcheck -s bash showcase/scripts/reconcile-prod-gate.sh` →
**clean**
- `actionlint .github/workflows/showcase_reconcile.yml` → **clean** (the
pre-existing `depot-ubuntu-24.04-4` custom-runner-label warning is on
`showcase_validate.yml`, predates this PR — my only change there is one
line in the shellcheck list)
## Test plan
- [ ] CI green (Ruby suite, bats suite, actionlint/shellcheck,
commitlint)
- [ ] Optional: read-only `workflow_dispatch` run of
`showcase_reconcile.yml` to confirm it runs against live prod/staging
(safe — no mutations)
## What
Adds **`@copilotkit/bot-whatsapp`** — a WhatsApp Business **Cloud API**
`PlatformAdapter` for the platform-agnostic `@copilotkit/bot` engine —
plus a runnable **`examples/whatsapp`** app and docs. This brings
WhatsApp to the bots ecosystem alongside the existing Slack support,
reusing the engine, the `@copilotkit/bot-ui` IR, and the pluggable
`ActionStore` untouched.
## How it works
- **Ingress:** the adapter owns its own HTTP server — GET verification
handshake (`hub.challenge`) + POST intake validated by
`X-Hub-Signature-256` HMAC (timing-safe), acked `200` immediately then
processed async.
- **No streaming:** WhatsApp messages are immutable, so the run renderer
**buffers** text and sends once on `TEXT_MESSAGE_END`
(`supportsStreaming: false`; `update()` posts fresh, `delete()` no-ops).
- **Interactive mapping:** text/section → text; ≤3 buttons →
reply-button message; `Select` or 4–10 actions → list message; >10 →
numbered-text fallback. A control's `value` round-trips by encoding it
into the reply id (`ck:…::<json>`), since WhatsApp replies carry no
value field; oversized encodings fail loud rather than corrupt silently.
- **Memory:** WhatsApp exposes no readable history, so a pluggable
**`HistoryStore`** (default `InMemoryHistoryStore`) holds it and replays
it into `agent.messages` each turn (fresh threadId per turn, mirroring
`bot-slack`). Swap in a durable backend to persist across restarts.
- **Commands:** leading-keyword matching (`commandPrefix`, default `/`);
the command text is injected via the engine's `runAgent({ prompt })`
path (not persisted at ingress).
- **Inbound media** → AG-UI multimodal content parts; **HITL** via
interactive replies.
## Example
`examples/whatsapp` mirrors `examples/slack`: a CopilotKit
`BuiltInAgent` over MCP (Linear + Notion), with `issue_list`, an
interactive `show_incident`, and a `confirm_write` HITL gate.
## Tests & verification
- 62 unit tests across the package (render mapping, markdown→WhatsApp,
signature verification incl. wrong-but-equal-length, interaction
decode/round-trip, buffered renderer, webhook listener/server, stores,
media, adapter).
- `build` ✅, package `check-types` ✅, `publint`/`attw` (ESM-only) ✅,
example `check-types` ✅. Full `nx run-many -t test
--projects=packages/**` passes.
- Two rounds of code review (APPROVE) — fixed slash-command history
double-append and silent value-truncation; minors (HMAC over raw bytes,
conversationKey invariant, offset-correct Blob, unused-dep pruning,
added tests).
## Docs
Package `README.md` + `ARCHITECTURE.md`, example setup guide (Meta app +
webhook + tunnel), and a `shell-docs` WhatsApp guide page (registered in
`meta.json` + early-access gate).
## Notes / out of scope (v1)
- No template-send path for messaging outside WhatsApp's 24-hour
customer-service window (documented limitation).
- Pre-existing, unrelated `@copilotkit/core` `phoenix-observable.ts`
typecheck error exists on the branch base (missing `@types/phoenix`) —
not introduced here.
Prod sitting behind staging is often intentional (changes are batched and
promoted deliberately), so a recurring drift alert is noise. Reshape the
reconcile workflow to manual-only:
- Remove the daily `schedule:` cron trigger — leave only `workflow_dispatch`.
- Remove the auto-Slack-on-stale step (and its SLACK_WEBHOOK env / stale_line
output derivation) — no unsolicited #oss-alerts post on mere staleness.
- A manual run surfaces the reconcile table to the GH step summary, keeps the
cheap `--json` capture as an uploaded artifact, and still exits nonzero on a
stale column so a manual run visibly flags drift.
- De-noise the gate script + bats comments that referenced the removed
scheduled/Slack behavior.
The on-demand CLI (`bin/railway reconcile-prod`), the gate wrapper, and the
Ruby + bats tests are unchanged.
Lever 1 of the promote-reliability hardening plan. The showcase deploy
model is staging=mutable :latest (continuously rebuilt), prod=immutable
@sha256: (advances only on explicit promote), so a prod column can
silently fall BEHIND a green staging — drift today is only noticed by
eyeballing a dead column. This adds proactive, automatic detection.
- bin/railway reconcile-prod: for every prod-eligible (probe.prod==true)
service, compares the prod SERVING digest (LintProd snapshot path) vs
the staging RUNNING digest (reuses PromoteCommand#staging_running_digest).
Classifies green/stale/gray, prints a table + summary, exits 1 iff any
stale. --json for machine output. Read-only: no promotes/mutations.
- scripts/reconcile-prod-gate.sh: wrapper mirroring lint-prod-gate.sh —
surfaces the table to the GH step summary, captures JSON for the Slack
builder, propagates the exit-code verdict.
- .github/workflows/showcase_reconcile.yml: daily cron + workflow_dispatch;
runs the gate; on stale services posts the stale-column list to
#oss-alerts (SLACK_WEBHOOK_OSS_ALERTS) via the fromJSON('"\n"') idiom.
- Tests: Ruby minitest (classification + exit-code, RED-anchored on a
drift-blind classifier) and a bats gate test. Wired the gate script
into the showcase_validate.yml shellcheck list.
Post-promote convergence verification is deferred to a fast-follow.
## Summary
- `upload-artifact`'s `!**/node_modules/**` filters are post-walk: the
action still descends into every `node_modules` and stats every file
(~6M with pnpm's `.pnpm/` symlink farm) before applying negations. That
enumeration is the actual bottleneck — `Upload workspace` runs 10+
minutes even with the filters added in #5044.
- Replace the filtered upload with: `rm -rf` the heavy dirs
(`node_modules`, `.nx`, `.turbo`, `.next`), `tar -czf /tmp/workspace.tgz
.`, upload that single file. Publish job `tar -xzf`'s it after download
and continues unchanged.
- Applied symmetrically to `prerelease.yml` and `publish-release.yml`.
## Measured impact
Verified on a dry-run dispatch of `release / pre` against this branch
([run
26531785850](https://github.com/CopilotKit/CopilotKit/actions/runs/26531785850)):
| Step | Before (run 26529550757) | After (this PR) |
| ------------------------------- | ------------------------ |
--------------- |
| Upload workspace | ~800s (cancelled) | **3s** |
| Pack workspace | — | 9s |
| Download workspace | — | 1s |
| Unpack workspace | — | 1s |
| **Total artifact round-trip** | **~800s** | **14s** |
- `Upload workspace` step alone: **~99.6% reduction (~267× faster)**.
- Full pack/upload/download/unpack pipeline vs the prior single upload:
**~98% reduction (~57× faster)**.
The 800s baseline is from a cancelled run, so both numbers are
conservative.
## Test plan
- [x] Dispatch `release / pre` against this branch with `dry_run=true`
- [x] Confirm `Upload workspace` completes in seconds instead of 10+ min
- [x] Confirm publish job `Unpack workspace` restores the tree and `pnpm
install` succeeds
- [x] Confirm dry-run publish step exits clean (no missing files from
the tarball round-trip)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## What
Adds **cvdiag** — a permanent, always-available observability subsystem
for the showcase, built to diagnose the red↔green cell flap on the
staging dashboard and to make that diagnosis a dashboard query rather
than a multi-day forensic hunt in the future.
Captures the full request path with `X-Test-Id` correlation across
**probe → backend → aimock → edge**, across every integration
(TypeScript, Python, Java/spring-ai, .NET):
- Per-language backend emitters (canonical + staged/compile-linked
mirrors), all sharing one schema (`schema.json`, closed-world
`additionalProperties:false`).
- CREATE-only writes to two new PocketBase collections: `cvdiag_events`
and `cvdiag_raw_byte_samples` (additive migrations — no existing data
touched).
- An 8-class flap classifier mapping to the observed failure signatures
(`sse-missing` / `text-unstable` / `dom-missing`).
- DEBUG-tier raw-byte capture (secret-scrubbed) and HMAC-guarded A/B
edge-interference detection.
## Why
The runId flap-fix (`cdc1e90e`, 2026-06-09) did **not** fully resolve
the flap — it was still observed 2026-06-19. cvdiag exists so the
*remaining* cause is observed live with full correlation instead of
inferred.
## Safety / enablement
- **Inert by default.** With `CVDIAG_BACKEND_EMITTER` unset the
subsystem performs zero host mutation (no logging-config changes, no
threads/tasks, no stdout) — verified by
`test_cvdiag_inert_when_disabled`. **To accumulate data, set
`CVDIAG_BACKEND_EMITTER=1` on the showcase services.**
- All per-language scrubbers match the canonical `scrubSecrets`
(sk-/base64url, Bearer, colon-less URL userinfo, size-guard) — verified
with real toolchains (vitest / mvn / dotnet).
- Merged latest `main` (only conflict: a clean `.csproj` include union).
## Verification
- harness `tsc --noEmit` ✓ · `src/cvdiag` vitest 251/251 ✓ ·
`cvdiag-stage-ts --check` in-sync ✓
- Java MessageScrubber 17/17 (mvn) ✓ · .NET CvdiagBackend 5/5 (dotnet
sdk:9.0) ✓ · Python emitters 93/93 (3.12) ✓
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The showcase_promote.yml resolve-targets job runs
`emit-railway-envs-json.ts`, which shells out to the repo-root
`node_modules/.bin/oxfmt` to produce oxfmt-canonical JSON. That job's
`npm ci` runs in showcase/scripts only and never installs the root oxfmt
binary, so every promote dispatch died at "Generate SSOT artifact" with
`spawnSync .../node_modules/.bin/oxfmt ENOENT` (exit 1) — blocking ALL
promotes, including the team's regular shell-docs promote, since the last
green run on 2026-06-18.
The emitted JSON on the resolve-targets / promote path is EPHEMERAL: it
is parsed in-memory by jq (resolve-promote-targets.sh) and bin/railway to
pick the promote target and is NEVER committed, so oxfmt-canonical
formatting is irrelevant there. Add an explicit `EMIT_SKIP_OXFMT=1`
opt-out that returns the raw `JSON.stringify` form, and set it on both
ephemeral workflow steps.
The DEFAULT (committed-artifact) path is unchanged: oxfmt stays REQUIRED
and fails loud if the binary is absent, because the committed
railway-envs.generated.json must stay oxfmt-canonical or CI's
static_quality.yml `oxfmt --check` auto-format bot fires on the drift.
This is opt-IN-to-skip, never silent-on-absence.
Call sites of emit-railway-envs-json.ts:
- showcase_promote.yml resolve-targets — EMIT_SKIP_OXFMT=1 (this fix).
- showcase_promote.yml promote — EMIT_SKIP_OXFMT=1 (this fix).
- static_quality.yml committed-artifact `--check` — unset, oxfmt required.
- resolve-verify-matrix.ts (showcase_deploy.yml) — only invokes the
emitter when the committed JSON is absent; the checkout always has it,
so the default (oxfmt) path is correct and unchanged.
Tests: 2037 showcase/scripts tests pass; 2 new EMIT_SKIP_OXFMT unit tests
assert the skip path emits valid (raw) JSON; the existing oxfmt-canonical
golden tests still gate the committed path.
Regenerated the stale committed showcase_promote.yml so the 12 starter-*
services (+ shell-docs and all existing targets) appear in the service
dispatch choice list (fixes HTTP 422 on
gh workflow run -f service=starter-*). Reverted isProdPromotable to
env-map-only (environments.prod.probe), equivalent to the workflow resolve
predicate. Added a regression test asserting shell-docs + all 12 starters
remain in the generated AND committed dropdown.
## Summary
Fixes two correctness bugs in the showcase staging→prod promote path
(`showcase/bin/railway`), discovered + live-validated while promoting
the full 19-service cluster.
**Bug #1 — promote pinned the wrong digest.** `resolved_prod_image`
re-resolved the mutable `:latest` tag against *current* GHCR instead of
pinning the digest staging is actually *running*
(`latestDeployment.meta.imageDigest`). When `:latest` drifted after
staging deployed, promote pushed an unvalidated (and once, regressed)
image to prod. Now pins staging's running digest. Adds a loud `⚠️
STAGING DRIFT` warning (promote stdout + `STAGING_DRIFT_MARKER:` →
`promote-fleet.sh` aggregation → both Slack payloads) when staging's
running digest ≠ current `:latest`, so the gap is visible without
blocking the promote.
**Bug #2 — the pin never activated.** `pin_and_verify` used
`serviceInstanceRedeploy`, which replays the *existing* deployment's old
image rather than the just-pinned `source.image`. Config showed the new
digest while prod kept serving the old one (this is why earlier promotes
"succeeded" while prod stayed broken). Switched to
`serviceInstanceDeployV2` + a new `verify_serving_digest!` gate that
polls the new deployment to SUCCESS and **fails loud** if the running
digest ≠ pinned.
Plus CR-round hardening: P2 in-flight race check now reads
`meta.imageDigest` (was dead on tag-form staging) and skips on
`--digest` override; `detect_staging_drift` fails loud (WARN) on
GHCR-resolve failure instead of swallowing; `--digest` override
suppresses spurious drift; `drift_line` LF-stripped at the
`GITHUB_OUTPUT` boundary; multi-service drift join fixed; fallback-log
drift preserved.
## Commits
1. `fix(showcase): pin prod to staging's running digest + loud
staging-drift warning`
2. `fix(showcase): activate prod pin via serviceInstanceDeployV2 +
verify running==pinned`
3. `fix(showcase): harden promote P2 race check + refresh ivar-lint
allowlist`
## Validation
- Live red-green on real Railway: claude-sdk-python prod flipped from
stale → pinned digest via the fixed CLI; all 19 cluster services
promoted green; 5 previously-degraded backends (ag2, llamaindex,
pydantic-ai, ms-agent-python, strands) recovered.
- 15/15 Ruby specs + 15/15 bats green; ruby -c, shellcheck (CI
invocation), actionlint clean.
- 3-round cr-loop (7 agents/round) converged to zero bucket-(a).
## Follow-ups (not in this PR)
- Defensive `.to_s.empty?` on `meta.imageDigest` extraction
(can't-happen on real Railway).
- Test-quality nits (capture_io scoping, weak bats glob, shared mock
fixture).
- Pre-existing verify-prod `succeeded_csv`/`GITHUB_OUTPUT` coupling
(graceful-degraded, audited STAY_IN_C).
- `deploy-to-railway.ts` births prod on `:latest` (the root provisioning
gap); CLI↔workflow notify equivalence (PR2).
## Test plan
- [ ] CI green on PR HEAD
- [ ] (post-merge) a real `gh workflow run showcase_promote.yml` shows
the drift line in the #team-showcase notification when staging is behind
:latest
renovate[bot]