Dry-run by default (lists the plan, changes nothing); --force executes,
--all ignores TTL/keep, --include-live opts into reaping a live-owner
target, <name|slot> targets one. Identifies harness-owned projects via
the slot-record / run-dir / showcase-iso<N> / self-id-label union, and
never touches the base 'showcase' stack or BuildKit resources. Real
docker bats prove dry-run/--force/--all + the base/buildkit guards.
A --keep'd isolated stack whose owning process had exited (but whose
containers kept running) was classified 'live' forever and never reaped,
leaking Docker stacks indefinitely. Introduce a start-time-verified
_owner_liveness probe and a new 'kept' state, an ISOLATE_KEEP_TTL (4h,
SHOWCASE_ISOLATE_KEEP_TTL-overridable) that flips an over-age kept slot
to 'stale' so the sweep reclaims it, a com.copilotkit.showcase.isolate
self-id label stamped by apply_isolation, a 'slots --reapable' filter,
and a macOS lsof COMMAND-truncation fix in the own-project port filter.
Real-surface bats cover the liveness false-positive and TTL reaping.
Emit CVDIAG backend boundary markers from the agent process for strands-typescript (byteLength fix on sseChunkByteLength), enable the emitter in docker-compose.local.yml, vendor src/cvdiag, and exclude tests from tsconfig.
## Root cause
Prod's `harness-workers` fleet worker runs a **stale `showcase-harness`
image** because it had **no `prod` env entry** in the railway-envs SSOT.
- The worker (`serviceId c2aa8a0b-350e-4b76-8541-3012dfac41d0`, prod
instance `7c48ee43-6df4-457b-b977-10f1f1ac1680`, `HARNESS_ROLE=worker`)
consumes the shared `showcase-harness` image via `imageOf: "harness"`.
- `expandImageConsumers(names, env)` is **env-aware**: a consumer only
enters an env's redeploy scope if it declares that env
(`redeploy-env.ts:278` — `if (!Object.hasOwn(entry.environments, env))
continue;`).
- Because the worker modeled **staging only**, a rebuilt
`showcase-harness:latest` bounced the prod control-plane but **silently
skipped the prod worker**, which kept its stale **2026-06-19** image.
- That stale image bakes a **1-demo `registry.json`** for
`ms-agent-harness-dotnet` (only `beautiful-chat`). The hourly
`e2e_demos` driver runs on the worker → resolves 1 demo → writes only
`e2e:ms-agent-harness-dotnet/beautiful-chat`. The other 38 feature rows
never exist in prod PocketBase → `resolveD3.exists === false` → `UI`
badge omitted → broken D3 rung collapses the ladder → **D0**.
(d5/d6 populate fully in prod because the D5/D6 drivers enumerate from a
**compiled-in** script registry, not the `registry.json` data file —
only `e2e_demos` is data-driven, which is why only `UI` was affected.)
## Fix
Backfill the live prod worker as a real `prod` env entry in
`scripts/railway-envs.ts` (real serviceInstance ID `7c48ee43…`), flip
`gateIgnore` off, and set `gateValidated: true`. The env-aware `imageOf`
expansion now pulls the prod worker into the **prod** redeploy scope on
every `showcase-harness` rebuild, so it can no longer drift onto a stale
image.
Also regenerates `railway-envs.generated.json` (Ruby/jq boundary
artifact) and the golden behavior-preservation fixture, and updates the
two gate-count assertions (`gateValidated` services 40→41;
`harness-workers` removed from the gateIgnore set).
## Local RED → GREEN proof
Failure surface: the real `expandImageConsumers("harness", "prod")`
against the real SSOT must include `harness-workers`.
**RED** (prod env entry absent from SSOT — the bug):
```
× includes harness-workers in the PROD redeploy scope when showcase-harness rebuilds
AssertionError: expected [ 'harness' ] to include 'harness-workers'
at __tests__/redeploy-env.harness-worker-prod-scope.test.ts:31:19
Tests 1 failed | 1 passed (2)
```
**GREEN** (after adding the prod `harness-workers` env entry):
```
✓ includes harness-workers in the PROD redeploy scope when showcase-harness rebuilds
✓ still includes harness-workers in the STAGING redeploy scope (no regression)
Tests 2 passed (2)
```
Full SSOT-dependent suite (golden snapshot, emit-json, image-ref gate,
promote closure, verify-matrix, redeploy-env): **140 passed**.
## Note (out of scope for this PR)
This SSOT change ensures the prod worker is bounced on **future**
rebuilds. The currently-live prod worker still needs a one-time
redeploy/restart onto the current `showcase-harness:latest` (39-demo
registry) to immediately backfill the 38 missing rows; that is an
operational step, not a code change.
Port the google-adk a2ui-recovery demo to langgraph (python, fastapi,
typescript) and aws-strands (python, typescript). Each ships a dedicated
recovery agent, route, demo page/chat/suggestions, manifest entry, aimock
d6 fixtures, e2e spec, and QA doc.
Backend-owned recovery on langgraph via get_a2ui_tools / getA2UITools
(injectA2UITool=false); auto-inject recovery on the strands adapter path.
Heal stages an invalid-then-valid render via aimock sequenceIndex (the
toolkit validate->retry loop rejects the whole surface, so a single-pass
parse_and_fix heal is ADK-specific and does not apply here). Recovery
prompts are unique per framework and the fixtures carry no context match
field, so they fire for real browser (dojo) traffic, not just the harness.
Also harden the strands declarative-gen-ui composition guide to name the
exact catalog component (Metric, not MetricTile) and update the
generate-catalog + aimock-fixtures test expectations.
The prod harness-workers backfill (e88d01a) inverts the old
"harness-workers is staging-only" invariant. Update the 8 stale
assertions across 3 test files that still encoded staging-only,
deriving the new expected values from the SSOT (railway-envs.ts) and
the regenerated railway-envs.generated.json:
- healthcheckPathFor/emit healthcheckPath: prod now /health (was undefined/omitted)
- repoNameFor(prod): now resolves showcase-harness (was throw)
- envsFor: now [prod, staging] (was [staging])
- the worker-shape test: dual-env, domainless+probe-disabled in BOTH
envs, gateValidated:true / gateIgnore dropped (per SSOT)
- computePromoteClosure: harness-workers now Tier-1 promoted, not
skipped; the always-Tier-1 set no longer filters it out
- expandImageConsumers(prod) / default prod redeploy scope (39->40):
the dual-env worker now joins the prod showcase-harness redeploy scope
The prod `harness-workers` fleet worker (serviceId
c2aa8a0b-350e-4b76-8541-3012dfac41d0, instance
7c48ee43-6df4-457b-b977-10f1f1ac1680) runs the shared `showcase-harness`
image (`imageOf: "harness"`) but had NO `prod` env entry in the
railway-envs SSOT. `expandImageConsumers` is env-aware — a consumer only
joins an env's redeploy scope if it declares that env — so a rebuilt
`showcase-harness:latest` bounced the prod control-plane but SILENTLY
SKIPPED the prod worker, leaving it pinned to a stale 2026-06-19 image.
That stale worker image carries a 1-demo `registry.json` for
`ms-agent-harness-dotnet` (only `beautiful-chat`), so the hourly
`e2e_demos` driver running on it produced only 1 of 39 `e2e:` rows in
prod PocketBase. The other 38 feature rows were absent → `resolveD3`
exists=false → `UI` badge omitted → broken D3 rung → D0.
Backfill the live prod worker as a `prod` env entry (real
serviceInstance ID), flip `gateIgnore` off, and set `gateValidated:
true` so the env-aware `imageOf` expansion now pulls the prod worker
into the prod redeploy scope on every `showcase-harness` rebuild.
Regenerate the emitted JSON + golden fixture and update the two
gate-count assertions accordingly.
The showcase-strands-typescript integration was staging-only: it had no
production Railway serviceInstance, so the prod D6 dashboard column showed
a uniform false-red (every cell errorClass=goto-error, backendUrl="") —
the probe navigated a bare relative path because the harness had no prod
health record / backendUrl to discover.
Provisions the prod serviceInstance (8a50728e-6119-43c4-b59c-d9535b6717a4,
domain showcase-strands-typescript-production.up.railway.app, healthcheck
/api/health, image pinned to the GHCR @sha256 digest, OPENAI_BASE_URL at
prod aimock) and brings the SSOT to the dual-env showcase-strands shape:
- railway-envs.ts: add the prod env entry with the real instanceId,
gateValidated:true, drop gateIgnore, remove the legacyJsonCompat
prod-domain placeholder.
- railway-envs.generated.json: regenerated (prod instanceId/domain, probe.prod
true, prod healthcheck; moved into the promote closure, tier 2).
- railway-envs.golden.json: regenerated to include the new prod (service,env)
pair (intentional behavior change, not a refactor regression).
- showcase_promote.yml: dropdown regenerated to list strands-typescript.
- verify-railway-image-refs.test.ts / redeploy-env.test.ts: update the
gateValidated/scope counts (39->40 gate targets, prod default 38->39) and the
now-stale staging-only comments.
RED->GREEN (live prod): BEFORE /api/health 404, prod PocketBase
health:strands-typescript totalItems:0, the 3 named D6 cells all
errorClass=goto-error backendUrl="". AFTER /api/health 200, prod PocketBase
health:strands-typescript present (status:200, valid url),
verify-railway-image-refs OK 80 instances.
Bump the canonical CopilotKit pin across all showcase integrations + shell
to 1.61.2 (canonical-pins.json, every package.json + package-lock.json),
which carries CopilotKit#5611: passing a catalog to the provider
(`<CopilotKit a2ui={{ catalog }}>`) now auto-enables A2UI and defaults tool
injection on, so the runtime no longer needs an explicit `a2ui` config.
Demonstrate the feature on the A2UI dynamic (declarative-gen-ui) demos by
removing the now-redundant runtime `a2ui` block (`injectA2UITool: true` +
`defaultCatalogId`) from:
- langgraph-python, langgraph-fastapi, langgraph-typescript
- strands, strands-typescript
- google-adk
The forwarded catalog supplies its own catalogId (sdk-js A2UI middleware
auto-derives `defaultCatalogId` from it), so the previous "Catalog not found"
fallback no longer applies.
Verified: validate-pins drift ratchet unchanged (38 / same hash);
langgraph-python D6 `gen-ui-declarative` green end-to-end (no Catalog-not-found).
The dojo's preview iframe built its src from `integration.backend_url`,
which generate-registry.ts bakes into registry.json at Docker BUILD time
(default `showcase-{slug}-production.up.railway.app`). So the staging
dojo iframed PROD integration backends — the exact staging->prod leakage
the shell's SU-13 runtime-derivation fix already prevents, but which was
never ported to shell-dojo.
Port the `backendHostPattern` slice of SU-13:
- copy shell's backend-url.ts verbatim (resolveBackendUrl + the
NEXT_PUBLIC_LOCAL_BACKENDS local-dev override); a scripts drift-guard
test keeps it byte-identical to the shell's and pins the default
pattern across backend-url.ts and generate-registry.ts.
- add `backendHostPattern` to shell-dojo's RuntimeConfig (server reads
SHOWCASE_BACKEND_HOST_PATTERN at request time; client carries the SSR
sentinel) — the existing layout injection picks it up automatically.
- page.tsx derives previewUrl via resolveBackendUrl at request time,
gated on a `mounted` flag so the SSR-phase sentinel host never reaches
an iframe src (shell-dojo loads the registry synchronously, so unlike
the shell it has no data-loading guard to defer the read past
hydration).
Staging dojo's SHOWCASE_BACKEND_HOST_PATTERN is set to
`showcase-{slug}-staging.up.railway.app`; prod stays unset (= default
prod pattern), so prod behavior is byte-identical.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The strands-typescript SSOT entry is ciBuilt:true but staging-only (prod
instance not yet provisioned). redeploy-env.ts's default scope was the full
CI_BUILT_SERVICES set for BOTH envs, so a staging-only ciBuilt service would
wrongly enter the prod default scope and fail a manual `redeploy-env.ts prod`
(no prod instance). Filter the default scope by env declaration (explicit
--services stays unfiltered, preserving the contract-pin that an operator can
force a named service in an env it does not declare). imageOf expansion was
already env-aware; this extends the same invariant to the base scope.
Update the inventory-lock test counts for the new service (total 40->41,
CI_BUILT 38->39, staging default scope 39->40; prod default scope stays 38 as
the staging-only service is now correctly excluded).
Wire the strands-typescript showcase integration for staging deployment,
mirroring how the Python strands integration is deployed.
- manifest: flip deployed: true so the shell lists it in the integration menu
- railway-envs.ts: add showcase-strands-typescript SSOT entry (staging-only
for now: prod instance not yet provisioned, so it omits the prod env and is
gateIgnore'd until promoted dual-env); regenerate railway-envs.generated.json
- showcase_build.yml + showcase_build_check.yml: add the strands-typescript
build matrix entry, change-detection filter, and dispatch option (railway_id
is the new Railway service id)
- golden fixture + image-ref-gate inventory tests updated for the new service
Railway staging service showcase-strands-typescript provisioned
(showcase-strands-typescript-staging.up.railway.app, health /api/health,
OpenAI-via-aimock env). Prod is added later via the promote pipeline.
Fail loud on a dropped #oss-alerts page: the failure-alert cross-post no longer
swallows a 200/ok:false Slack response, so a dropped page-the-humans alert reds
the renderer job instead of vanishing on a green run. The thread reply stays
warn-only. Both posts capture the response via a shared slack_alert_posted_ok
predicate, mirrored byte-identically across the live workflow and the dry-run
helper.
Debt cleanup: drop a dead failed_count var, correct a misleading gha_url comment,
and validate the decoded blob run_id against ^[0-9a-f]{6}$ in the render step so
a malformed run_id can't reach Slack or the run name.
Tests: predicate edge cases (non-JSON, malformed, missing/null ok), an anti-drift
parity guard asserting the predicate is identical in both files, and call-site
tests locking the #oss-alerts fail-loud vs thread warn-only exit semantics.
Bump every @copilotkit/* dependency across the showcase integrations and
the shell from 1.60.2 (and stray "latest" override pins) to an exact
1.61.1 pin, and move the canonical pin source of truth to match.
Regenerate each standalone npm package-lock.json with the same
--legacy-peer-deps flag the Dockerfiles use for "npm ci".
- showcase/integrations/*/package.json + package-lock.json
- showcase/integrations/langgraph-typescript/src/agent/*
- showcase/shell/package.json + package-lock.json
- showcase/scripts/showcase-canonical-pins.json: canonical 1.60.2 to 1.61.1
aimock stays on its own version line (1.26.1). The Python copilotkit SDK
was already 0.1.94 across every requirements.txt, so no change there.
validate-pins ratchet is unchanged (FAIL=38, identical hash);
validate-parity, validate-fixture-tool-surface, and the showcase/scripts
vitest suite (2102 tests) all pass.
Adding a2ui-recovery to the shared feature-registry cross-joins it across all
20 integrations (wired for google-adk, unshipped elsewhere), so the catalog
generator's hardcoded counts move: cross-join 920->940, metadata.total_cells
900->920, LGP 46->47 (unshipped 7->8). docs_only stays 20; the relative
sum-checks are unchanged. Fixes the Validate Showcase failure on #5662.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pool-fleet migration is complete: the control-plane harness plus the
prod workers (deployed 2026-06-19, HARNESS_ROLE=worker, pool count 2) now
cover every probe dimension the interim `harness-legacy` fleet-migration
bridge was holding live, so `harness-legacy` is dead config.
This is the code-side cleanup only:
- Remove the `harness-legacy` entry from the railway-envs SSOT and the
now-dead `key === "harness-legacy"` special-case in computePromoteClosure.
- Regenerate railway-envs.generated.json (41 -> 40 services).
- Drop harness-legacy from the golden snapshot, the gateIgnore expectation
sets, the promote-notify fixtures, and the redeploy-env doc comments;
update the service-count assertions (41 -> 40).
- Fix the stale "STAGING-ONLY" harness-workers comment: prod workers are
live on Railway, though this SSOT entry still models the staging
instance only (no prod env backfilled here yet).
The live Railway `harness-legacy` service is torn down separately as a
follow-up infra step.
Promote-notify Slack message: name the promoted AND failed services (one
Failed: header + bullets), legible "(N): <names>" count, real wall-clock
elapsed (integer-coerced), and drop the constant verify-prod legend line.
Durable healthcheckPath: track it per-service/env in the SSOT (railway-envs),
re-assert it on the promote pin path (omit-when-absent, never null), and route
deploy-to-railway provisioning through isTrackedService/resolveProvisionHealthcheck
so a tracked-null service omits the healthcheck while an untracked one keeps the
/api/health default — fixing the silent prod-healthcheck drift that refused aimock.
Tests: ruby pin-reassert spec + deploy-to-railway healthcheck spec + emit/golden/accessor.
Flip the 12 starter-* services to behave like every other managed
showcase service: always-on and staging-probed.
- railway-envs SSOT: staging probe false -> true for all 12 starters
(prod was already on); refresh the stale "staging probe OFF" /
fail-loud-placeholder doc-comments to reflect baseline-driver verification.
- provision-starter-fleet: sleepApplication true -> false so the
provisioner creates always-on (not sleepable) Railway instances; update
the "the whole point — sleepable" header/inline comments + log lines.
- Regenerate railway-envs.generated.json (emit-railway-envs-json.ts) and
the golden fixture; update the SSOT probe + provisioner sleep assertions.
Replace the fail-loud `case "starter"` stub in verify-deploy's dispatch
switch with a real baseline liveness driver. The starter-template fleet
EXPOSEs only its Next.js frontend (serving `/` and `/api/copilotkit`,
NO `/api/health`), so the driver healthchecks `/` via probeBaseline,
exactly like the Next.js shells: deployment-SUCCESS + HTTP 200 on `/`.
promote-fleet.sh now emits a base64 results JSON (schema_version=1) with
both succeeded[] and failed[] alongside the existing succeeded_csv /
staging_drift outputs. showcase_promote.yml's notify job replaces the old
inline two-state (success/failure) notifier — which dumped the full
requested CSV and mislabeled any partial promote as a blanket Failed — with
a single dispatch of the previously-orphaned showcase_promote_notify.yml
renderer (success / partial / total). The dispatch enriches promote-fleet's
results blob with run context (6-hex run_id, trigger=workflow, operator,
pre_staging) and authenticates via the devops-bot App token (actions:write),
mirroring canary.yml — the default GITHUB_TOKEN cannot start new workflow runs.
promote-fleet has no failure taxonomy, so each failed entry uses the default
category "promote-failed".
Adds a durable persistence layer for @copilotkit/bot, replacing the
in-memory-only ActionStore with a pluggable StateStore.
- StateStore interface (kv/list/lock/dedup/queue) with a shared
conformance suite; MemoryStore default plus @copilotkit/bot-store-redis
and @copilotkit/bot-store-postgres backends.
- createBot({ store }): typed per-thread state via Standard Schema,
action snapshots persisted through the store, per-conversation turn
lock (onLockConflict drop|force), and inbound-event dedup keyed on a
stable eventId. ActionStore is kept as a deprecated alias.
- Cross-platform transcripts (bot.transcripts + identity resolver) with
age-bounded retention (prune on append + filter on read), and
runAgent({ transcript: true }) to auto-inject history and capture the
reply.
- createBot({ components }) re-registers components so durable actions
re-fire after a restart; restart-durability demo in examples/slack.
- Dedup is marked seen only after the turn lock is acquired, so a turn
dropped on lock-conflict does not burn its eventId (no lost retries).
- Release lockstep: bot-store-redis/postgres version with bot + bot-ui.
## Summary
An **on-demand** tool to answer "is prod caught up with staging right
now?". The showcase deploy model is **staging = mutable `:latest`**
(continuously rebuilt) and **prod = immutable `@sha256:`** (advances
only on an explicit promote), so a prod column can sit **behind** a
green staging.
**There is no scheduled drift alert — by design.** Prod lagging staging
is **often intentional**: changes are batched and promoted deliberately,
so a recurring "N columns stale" alert would be pure noise. This tool is
therefore manual-only: a maintainer runs it when they want to check, and
it tells them the current state.
- **`bin/railway reconcile-prod`** — for every prod-eligible
(`probe.prod == true`) service, compares the **prod serving digest**
(the `@sha256:` from
`SnapshotCommand.build_snapshot(PRODUCTION_ENV_ID)`) against the
**staging running digest** (reuses
`PromoteCommand#staging_running_digest`, the same source the promote pin
uses). Classifies each:
- `green` — prod == staging (in sync)
- `stale` — prod != staging **and** staging is resolvable (prod is
behind a green staging)
- `gray` — staging running digest not resolvable, or the service has no
prod snapshot entry yet — informational, **not** stale
- Prints a readable per-service table + summary; **exits nonzero iff any
service is stale**; `--json` for machine output. **Read-only — no
promotes/mutations.**
- **`showcase/scripts/reconcile-prod-gate.sh`** — wrapper mirroring
`lint-prod-gate.sh`: surfaces the table into `$GITHUB_STEP_SUMMARY`,
optionally captures `--json` to `RECONCILE_JSON`, and propagates the
exit-code verdict (never swallows a non-zero).
- **`.github/workflows/showcase_reconcile.yml`** — **`workflow_dispatch`
only** (no cron). Regenerates the SSOT JSON (`EMIT_SKIP_OXFMT=1`, same
as the promote workflow's resolve/promote jobs), runs the gate with the
Railway/GHCR auth env, renders the reconcile table to the **GH step
summary**, and uploads the `--json` as a `reconcile-json` artifact. **No
Slack.** The run exits nonzero on a stale column so a manual run visibly
flags drift. `timeout-minutes: 10`.
- **Tests** — Ruby minitest (`test_reconcile_prod.rb`: classification +
exit-code + `--json` shape + dispatcher registration) and a bats gate
test (`reconcile-prod-gate.bats`). Wired the gate script into the
`showcase_validate.yml` shellcheck list.
### What changed from the original scheduled-alert design
The first cut of this PR shipped a daily cron + auto-post to #oss-alerts
on any stale column. Per owner feedback, that was reshaped to on-demand
only: the `schedule:` trigger and the Slack-on-stale step were removed
(intentional/deliberate staleness is not a bug, so an unsolicited
recurring alert is noise). The CLI command, the gate wrapper, and all
tests are unchanged.
## Gates
- `ruby showcase/bin/spec/test_reconcile_prod.rb` → **9 runs, 20
assertions, 0 failures**
- `bats showcase/scripts/__tests__/reconcile-prod-gate.bats` → **6 ok**
- `shellcheck -s bash showcase/scripts/reconcile-prod-gate.sh` →
**clean**
- `actionlint .github/workflows/showcase_reconcile.yml` → **clean** (the
pre-existing `depot-ubuntu-24.04-4` custom-runner-label warning is on
`showcase_validate.yml`, predates this PR — my only change there is one
line in the shellcheck list)
## Test plan
- [ ] CI green (Ruby suite, bats suite, actionlint/shellcheck,
commitlint)
- [ ] Optional: read-only `workflow_dispatch` run of
`showcase_reconcile.yml` to confirm it runs against live prod/staging
(safe — no mutations)
A `shell-docs` promote expanded its dependency closure to include the
always-on Tier-1 verification set (harness, dashboard, …); harness then
tripped a P6 env-divergence WARN-refusal and the tier barrier gated docs
as NOT-ATTEMPTED. docs has no runtime dependency on the control plane.
Introduce a declarative `standalone` service class: a leaf that neither
depends on anything nor gates on anything. computePromoteClosure (TS) and
the resolve-promote-targets jq both skip the Tier-1 union when the request
is entirely standalone, so a standalone request promotes ONLY itself; the
fleet runner promotes `s:`-marked services ungated (never NOT-ATTEMPTED on
an unrelated failure, never gating a tier). Flag `docs` standalone.
Red-green: railway-envs.test.ts (4), resolve-promote-targets.bats (5),
promote-fleet.bats (3) fail on origin/main (docs closure pulls harness;
`s:` is an unknown tier) and pass after the fix.
Prod sitting behind staging is often intentional (changes are batched and
promoted deliberately), so a recurring drift alert is noise. Reshape the
reconcile workflow to manual-only:
- Remove the daily `schedule:` cron trigger — leave only `workflow_dispatch`.
- Remove the auto-Slack-on-stale step (and its SLACK_WEBHOOK env / stale_line
output derivation) — no unsolicited #oss-alerts post on mere staleness.
- A manual run surfaces the reconcile table to the GH step summary, keeps the
cheap `--json` capture as an uploaded artifact, and still exits nonzero on a
stale column so a manual run visibly flags drift.
- De-noise the gate script + bats comments that referenced the removed
scheduled/Slack behavior.
The on-demand CLI (`bin/railway reconcile-prod`), the gate wrapper, and the
Ruby + bats tests are unchanged.
Lever 1 of the promote-reliability hardening plan. The showcase deploy
model is staging=mutable :latest (continuously rebuilt), prod=immutable
@sha256: (advances only on explicit promote), so a prod column can
silently fall BEHIND a green staging — drift today is only noticed by
eyeballing a dead column. This adds proactive, automatic detection.
- bin/railway reconcile-prod: for every prod-eligible (probe.prod==true)
service, compares the prod SERVING digest (LintProd snapshot path) vs
the staging RUNNING digest (reuses PromoteCommand#staging_running_digest).
Classifies green/stale/gray, prints a table + summary, exits 1 iff any
stale. --json for machine output. Read-only: no promotes/mutations.
- scripts/reconcile-prod-gate.sh: wrapper mirroring lint-prod-gate.sh —
surfaces the table to the GH step summary, captures JSON for the Slack
builder, propagates the exit-code verdict.
- .github/workflows/showcase_reconcile.yml: daily cron + workflow_dispatch;
runs the gate; on stale services posts the stale-column list to
#oss-alerts (SLACK_WEBHOOK_OSS_ALERTS) via the fromJSON('"\n"') idiom.
- Tests: Ruby minitest (classification + exit-code, RED-anchored on a
drift-blind classifier) and a bats gate test. Wired the gate script
into the showcase_validate.yml shellcheck list.
Post-promote convergence verification is deferred to a fast-follow.
Two PR checks were red on the new integration:
- check-config-files: add strands-typescript/next.config.ts to the build-
config allowlist.
- Validate Showcase: update the new-integration guard pins that intentionally
trip when an integration is added — BORN_IN_SHOWCASE 6→7, calculator
_from-feature-parity count 18→19, catalog cross-join 874→920 / total_cells
855→900 / docs_only 19→20 (46 features × 20 integrations), and the aimock
substring-shadow ceiling 132→133 (+1 from the strands-typescript calculator
fixture).
Also drop the premature deploy wiring: strands-typescript is removed from
showcase_build.yml (matrix + path filter + ALL_SERVICES) because it has no
Railway service yet (deployed: false) and the railway-envs SSOT test requires
a real service entry. It re-enters the deploy pipeline when the Railway
service is provisioned (external setup per INTEGRATION-CHECKLIST).
## What
Adds **cvdiag** — a permanent, always-available observability subsystem
for the showcase, built to diagnose the red↔green cell flap on the
staging dashboard and to make that diagnosis a dashboard query rather
than a multi-day forensic hunt in the future.
Captures the full request path with `X-Test-Id` correlation across
**probe → backend → aimock → edge**, across every integration
(TypeScript, Python, Java/spring-ai, .NET):
- Per-language backend emitters (canonical + staged/compile-linked
mirrors), all sharing one schema (`schema.json`, closed-world
`additionalProperties:false`).
- CREATE-only writes to two new PocketBase collections: `cvdiag_events`
and `cvdiag_raw_byte_samples` (additive migrations — no existing data
touched).
- An 8-class flap classifier mapping to the observed failure signatures
(`sse-missing` / `text-unstable` / `dom-missing`).
- DEBUG-tier raw-byte capture (secret-scrubbed) and HMAC-guarded A/B
edge-interference detection.
## Why
The runId flap-fix (`cdc1e90e`, 2026-06-09) did **not** fully resolve
the flap — it was still observed 2026-06-19. cvdiag exists so the
*remaining* cause is observed live with full correlation instead of
inferred.
## Safety / enablement
- **Inert by default.** With `CVDIAG_BACKEND_EMITTER` unset the
subsystem performs zero host mutation (no logging-config changes, no
threads/tasks, no stdout) — verified by
`test_cvdiag_inert_when_disabled`. **To accumulate data, set
`CVDIAG_BACKEND_EMITTER=1` on the showcase services.**
- All per-language scrubbers match the canonical `scrubSecrets`
(sk-/base64url, Bearer, colon-less URL userinfo, size-guard) — verified
with real toolchains (vitest / mvn / dotnet).
- Merged latest `main` (only conflict: a clean `.csproj` include union).
## Verification
- harness `tsc --noEmit` ✓ · `src/cvdiag` vitest 251/251 ✓ ·
`cvdiag-stage-ts --check` in-sync ✓
- Java MessageScrubber 17/17 (mvn) ✓ · .NET CvdiagBackend 5/5 (dotnet
sdk:9.0) ✓ · Python emitters 93/93 (3.12) ✓
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Adds a minimal Angular quick-start guide and wires Angular into the docs
**frontend picker** introduced in #5586. Connects directly to an AG-UI
agent via `HttpAgent`, no runtime required.
Closes
[OSS-252](https://linear.app/copilotkit/issue/OSS-252/add-angular-quick-start-guide).
### Angular in the frontend picker
- Guide lives at `content/docs/frontends/angular.mdx`, served at
`/angular` like the other frontends
- Registered `angular` in `frontend-options` (+ `SiAngular` logo),
`search-hrefs`, the search-index pages, and the typed reference-slug
record
- Not flagged early access (parity with Vue / React Native)
### Verified end-to-end
Built a fresh Angular 21 app, followed the guide verbatim, and got a
live streamed reply. shell-docs typecheck and the frontend unit tests
(8/8) pass.
Key specifics from that validation: install **`@copilotkit/angular`**
(the deprecated `@copilotkitnext/angular` is renamed), no TS7016
workaround needed (the package ships types), and pin `@angular/cli@21`
since `@latest` (v22) is outside the `19-21` peer range.
> Reference-docs link points at `/reference` for now; it should switch
to `/reference/angular` once the Angular reference PR (#5585) lands.
> The in-repo package is still `@copilotkitnext/angular`; the
`@copilotkit/angular` rename only landed on npm 2026-06-18. Renaming in
source is a separate follow-up.
_Screenshots live on the `assets/oss-252-angular-quickstart` branch to
keep this diff scoped._
The showcase_promote.yml resolve-targets job runs
`emit-railway-envs-json.ts`, which shells out to the repo-root
`node_modules/.bin/oxfmt` to produce oxfmt-canonical JSON. That job's
`npm ci` runs in showcase/scripts only and never installs the root oxfmt
binary, so every promote dispatch died at "Generate SSOT artifact" with
`spawnSync .../node_modules/.bin/oxfmt ENOENT` (exit 1) — blocking ALL
promotes, including the team's regular shell-docs promote, since the last
green run on 2026-06-18.
The emitted JSON on the resolve-targets / promote path is EPHEMERAL: it
is parsed in-memory by jq (resolve-promote-targets.sh) and bin/railway to
pick the promote target and is NEVER committed, so oxfmt-canonical
formatting is irrelevant there. Add an explicit `EMIT_SKIP_OXFMT=1`
opt-out that returns the raw `JSON.stringify` form, and set it on both
ephemeral workflow steps.
The DEFAULT (committed-artifact) path is unchanged: oxfmt stays REQUIRED
and fails loud if the binary is absent, because the committed
railway-envs.generated.json must stay oxfmt-canonical or CI's
static_quality.yml `oxfmt --check` auto-format bot fires on the drift.
This is opt-IN-to-skip, never silent-on-absence.
Call sites of emit-railway-envs-json.ts:
- showcase_promote.yml resolve-targets — EMIT_SKIP_OXFMT=1 (this fix).
- showcase_promote.yml promote — EMIT_SKIP_OXFMT=1 (this fix).
- static_quality.yml committed-artifact `--check` — unset, oxfmt required.
- resolve-verify-matrix.ts (showcase_deploy.yml) — only invokes the
emitter when the committed JSON is absent; the checkout always has it,
so the default (oxfmt) path is correct and unchanged.
Tests: 2037 showcase/scripts tests pass; 2 new EMIT_SKIP_OXFMT unit tests
assert the skip path emits valid (raw) JSON; the existing oxfmt-canonical
golden tests still gate the committed path.
Add the Angular frontend quick-start at content/docs/frontends/angular.mdx
and wire it into the frontend picker (options, logo, page content, search
hrefs, search-index generation).
Regenerated the stale committed showcase_promote.yml so the 12 starter-*
services (+ shell-docs and all existing targets) appear in the service
dispatch choice list (fixes HTTP 422 on
gh workflow run -f service=starter-*). Reverted isProdPromotable to
env-map-only (environments.prod.probe), equivalent to the workflow resolve
predicate. Added a regression test asserting shell-docs + all 12 starters
remain in the generated AND committed dropdown.
## Summary
- Fix frontend selection, logo, and docs navigation behavior in shell
docs
- Update link rewriting, search href generation, SEO redirects, and
sitemap handling for frontend docs paths
- Refresh related docs pages and tests across framework and frontend
routes
## Testing
- Added and updated unit tests for frontend options, link rewriting,
search hrefs, SEO redirects, and framework shell layout
- Added route-level coverage for the llms-mdx endpoint
Jordan Ritter
