## Summary
Aligns showcase integration dependencies to current released minor
versions for the **1.60.2** cycle. Closes the version-coherence gap that
was blocking the `react-core@1.60.2` resume-path / gen-ui-interrupt
fixes from taking effect on staging.
## Package families bumped
| Family | From | To | Scope |
|---|---|---|---|
| `@copilotkit/{a2ui-renderer, react-core, react-ui, runtime, shared,
sdk-js, voice}` | `1.59.4` (18 integrations) / `1.57.2`
(ms-agent-harness-dotnet) | **`1.60.2`** | 19 integrations |
| `@ag-ui/{client, core, encoder}` | `0.0.55` | **`0.0.57`** | 15
integrations |
| `@ag-ui/mastra` | `0.2.1-beta.2` | **`0.2.4`** | mastra only |
`@ag-ui/mastra@1.0.x` (major) **deliberately not** bumped — major jump
held back per the broad-scope dep-bump policy.
## Integrations covered (19/19)
`ag2`, `agno`, `built-in-agent`, `claude-sdk-python`,
`claude-sdk-typescript`, `crewai-crews`, `google-adk`,
`langgraph-fastapi`, `langgraph-python`, `langgraph-typescript`,
`langroid`, `llamaindex`, `mastra`, `ms-agent-dotnet`,
**`ms-agent-harness-dotnet`** (newly added — was missed by the prior
18-integration staging and jumps two minor lines), `ms-agent-python`,
`pydantic-ai`, `spring-ai`, `strands`.
`built-in-agent`, `langgraph-fastapi`, `langgraph-python`,
`langgraph-typescript` carry no `@ag-ui/*` deps directly (the langgraph
trio gets the protocol via `@copilotkit/runtime`'s nested resolution,
which has been verified at `0.0.57` post-install).
## Coherence note
`@copilotkit/react-core@1.60.2` does not declare a hard peer-dep on
`@ag-ui/core` at the package-manifest level; it bundles its own copy via
nested `node_modules`. Lockfile inspection confirms nested
`@copilotkit/{react-core,runtime,shared}/node_modules/@ag-ui/client`
resolved at `0.0.57` across every integration that ships them,
satisfying the `feedback_agui_client_bump_scope` rule (`@ag-ui/core >=
0.0.48`).
## Reconciliation method
Per-integration `npm install --package-lock-only --legacy-peer-deps` (no
`node_modules` mutation). The `--legacy-peer-deps` flag is required to
step past a **pre-existing** `cmdk@0.2.1` ↔ `react@^19` peer conflict
that long predates this bump; lockfile contents are otherwise unchanged
in shape (only dep-version touchups). 30 files in the visible diff
because 8 of the 38 changed files were identical between staged-index
and working-tree.
## Red / Green proof
**RED** (pre-merge staging):
- `gen-ui-interrupt` cells on the langgraph trio (LGP / LGTS /
LG-FastAPI) and across the broader integration matrix are RED on the
showcase dashboard pending the `@copilotkit/react-core@1.60.2`
resume-path fix landing on every integration container.
**GREEN** (expected post-merge):
- `showcase_deploy` will rebuild every integration container on push to
`main`. The dashboard `gen-ui-interrupt` + `resume-path` cells should
flip GREEN once those containers redeploy.
- Per-cell empirical value-test (`bin/showcase test
<slug>:gen-ui-interrupt --d6 --direct`) on ≥3 cells across LGP / LGTS /
LG-FastAPI is queued for the post-merge verification window.
## Out of scope
- No source-code changes (TS/Python/.NET/Java/Go).
- No fixture changes.
- No frontend page changes.
- No `packages/`, `examples/`, `showcase/shell-*` touched.
- No Railway worker restart (separate effort).
- No langgraph-typescript backend agent changes (separate effort).
## Test plan
- [ ] CI green on this PR (lint / format / build / publint / attw on the
affected workspaces)
- [ ] Admin-merge once cr-loop converges to zero findings
- [ ] Post-merge: confirm `showcase_deploy` rebuilds integration
containers
- [ ] Post-merge value-test: `gen-ui-interrupt` + `resume-path` cells
flip GREEN on staging dashboard for LGP, LGTS, LG-FastAPI (≥3 cells)
## Summary
- New `.github/workflows/showcase_promote_notify.yml` — Slack notify
workflow (workflow_dispatch only) for promote results. Posts initiation
+ threaded reply to `#team-showcase`; cross-posts to `#oss-alerts` on
partial/total failure.
- New `showcase_promote_notify.dry-run.sh` — local render-logic mirror
(no Slack API calls); validates payload + emits expected Slack messages
for manual review.
- New `showcase/test-fixtures/promote-notify/` — three canonical
fixtures (success / partial / total-failure), a strict schema validator,
and a README.
- New `docs/runbooks/showcase-promote-notify-pr1-checklist.md` —
pre-merge runbook with Slack-membership checks, dispatch-fixture
commands, and schema-mismatch test.
## Why this is PR1
PR1 lands the notify workflow with no callers. The CLI (`bin/railway
promote --notify`) lands in PR2. Splitting is required because `gh
workflow run` resolves the workflow file from the repo's default branch
— PR2's CLI cannot dispatch a workflow that doesn't yet exist on `main`.
## Test plan
- [ ] Pre-merge checklist passes (see
`docs/runbooks/showcase-promote-notify-pr1-checklist.md`)
- [ ] Required `SLACK_BOT_TOKEN` secret is set with scopes `chat:write`,
`chat:write.public`, `users:read.email`
- [ ] Bot is a member of `#team-showcase` AND `#oss-alerts` (or scope
sufficient for public posts)
- [ ] All 3 canonical fixtures dispatched via `gh workflow run --ref
<pr-branch>` produce the expected Slack messages
- [ ] Schema-mismatch test (step 5 of runbook) produces `::warning::`
annotation with NO Slack API call
## Follow-up
A bucket(d) list of defensive-hardening items was deferred to a
follow-up PR (notify workflow defensive hardening). The CR loop (4
rounds, 7 unbiased agents each) identified ~50 items across categories:
silent-Slack-failure exit codes, validator strictness gaps (ISO-8601
fractional seconds, enum constraints), dry-run/workflow parity
(lookup-failure simulation, permalink-empty fallback), payload
defensive-validation (cross-field, run_id source-of-truth). Per cr-loop
convergence-audit: these are PR1-adjacent but their own subject; they'll
land in a focused follow-up PR.
Additionally, local `actionlint` (not currently in CI for this workflow)
flags one SC2034 (`failed_count` retained for workflow/dry-run parity
but not echoed) and two SC2016 (intentional single-quoted Slack-mrkdwn
backticks in `trigger_label`). Both fold into the bucket(d) follow-up.
## CR rounds
4 rounds × 7 agents = 28 reviews. Final convergence: ALL bucket(a)
findings fixed; bucket(b)/(c)/(d) preserved for follow-up. Integration
HEAD: `b65b8452d9b847f73c885854c146547ce82c3706`.
- showcase-canonical-pins.json: bump canonicalCopilotKitVersion 1.59.4 -> 1.60.2;
remove ms-agent-harness-dotnet override (caught up to canonical in prior commit).
- fail-baseline.json: re-ratchet validatePinsFailCount 39 -> 38 and hash to match
the one-item drop (ms-agent-harness-dotnet override no longer counted).
- @ag-ui/mastra: revert 0.2.4 -> 0.2.1-beta.2. 0.2.4 imports
'@mastra/core/runtime-context' which the pinned @mastra/core@1.41.0 does not
export, breaking 'next build' (failing mastra build-check in CI). Holding
@ag-ui/mastra at the prior pin until a coordinated @mastra/core upgrade lands.
Aligns dependency versions across all 19 showcase integrations to current
released minor versions for the 1.60.2 release cycle.
Package families:
- @copilotkit/{a2ui-renderer, react-core, react-ui, runtime, shared, sdk-js, voice}
1.59.4 -> 1.60.2 (18 integrations already staged; ms-agent-harness-dotnet
catches up from 1.57.2)
- @ag-ui/{client, core, encoder} 0.0.55 -> 0.0.57
- @ag-ui/mastra 0.2.1-beta.2 -> 0.2.4 (stable on 0.x; 1.0.x major held back)
Includes the previously-missed ms-agent-harness-dotnet integration in the
@copilotkit/* bump, plus the @copilotkit/web-inspector override pin.
Lockfile-only reconciliation via npm install --package-lock-only
--legacy-peer-deps (cmdk@0.2.1 pre-existing react^18 peer-dep is unaffected).
Pre-merge checklist for the promote-notify workflow: Slack-membership
checks (#team-showcase + #oss-alerts), SLACK_BOT_TOKEN scope verification,
gh workflow run dispatch commands for all three canonical fixtures, and
a schema-mismatch test that verifies the workflow emits a ::warning::
annotation with NO Slack API call.
Adds three canonical fixtures under showcase/test-fixtures/promote-notify/:
success.json (29 services green), partial.json (26 green / 3 red with
mixed exit codes + categories), and total-failure.json (29 red fleet
abort with truncation-suffix sentinel).
Includes validate.sh — strict schema enforcement (rc propagation, enum +
regex assertions for run_id, pre_staging, abort_reason, category, exit
codes). And a README documenting the fixture contract + how to dispatch
them via gh workflow run.
Adds .github/workflows/showcase_promote_notify.yml — workflow_dispatch-only
Slack notify workflow for promote results. Posts initiation + threaded reply
to #team-showcase; cross-posts to #oss-alerts on partial/total failure.
Also adds showcase_promote_notify.dry-run.sh — a local render-logic mirror
that decodes the same payload and prints the messages the workflow would
send to Slack, without making any Slack API calls. Used by the pre-merge
runbook and by future CI fixture tests.
Throw when a catalog definition has no matching renderer so createCatalog
satisfies strict check-types under Angular's tsconfig.
Co-authored-by: Cursor <cursoragent@cursor.com>
Modern @a2ui/web_core and @angular/cdk subpath exports require bundler
resolution so angular:check-types can resolve the new A2UI imports.
Co-authored-by: Cursor <cursoragent@cursor.com>
Align surface operation normalization and catalog test fixtures with the
RendererProps contract so the changed A2UI web-components pass tsc.
Co-authored-by: Cursor <cursoragent@cursor.com>
Remove the hand-written types entry from exports["."] so ng-packagr can
generate deterministic published package metadata without warnings.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Implemented audio transcription capabilities with error handling.
- Refactored CopilotChat component to utilize a directive for handling attachments.
- Improved CopilotChatReasoningMessage to manage streaming state and elapsed time more efficiently.
- Added new scroll view component for better message display and auto-scrolling behavior.
- Updated styles for A2UI surface components to enhance layout and scrolling.
- Enhanced tests for OpenGenerativeUIRenderer to ensure proper height measurement.
Apply the common Angular CLI schematics to code which has been added
by Soverius.
We can do a follow-up for the high amount of effects, but we have to be
careful. We need tests first to do a refactoring for that.
## Release monorepo v1.60.2
**Scope:** `monorepo` | **Bump:** `patch`
---
### How this release process works
1. **This PR was created automatically** by the "release / create-pr"
workflow.
It bumped the `monorepo` packages to `1.60.2`
and generated AI-enhanced release notes.
2. **CI runs on this PR** — the full test suite (unit tests, lint, type
checks, build)
must pass before merging. This is the review gate.
3. **Review the release notes** in `release-notes.md` in this PR.
If a Notion draft was created, you can edit the release notes there
before merging.
4. **When this PR is merged**, the `release / publish` workflow
automatically:
- Builds all packages
- Publishes the `monorepo` packages to npm at version `1.60.2`
- Creates git tag `monorepo/v1.60.2`
- Creates a GitHub Release with the final release notes
### Before merging
- [ ] CI is green (tests, lint, types, build)
- [ ] Version bumps look correct
- [ ] Release notes are accurate (edit in Notion if a draft was created)
---
> **Do not merge until CI is fully green.** The full test suite runs
automatically on this PR.
## Summary
- document the shell-docs hybrid docs architecture and `docs_mode`
meanings
- add a shell-docs README with npm local dev, validation commands, and
authoring recipes
- keep retired top-level `docs/` guidance intact while making shell-docs
the clear source of truth
## Verification
- `git diff --check -- .claude/docs/documentation.md
showcase/shell-docs/README.md`
- pre-commit: `nx run-many -t test --projects=packages/**` passed after
rerun; Nx flagged `@copilotkit/react-core:test` as flaky from an earlier
timeout
- pre-commit: `nx run-many -t publint,attw --projects=packages/**`
passed
## Notes
- Archive refs were created before this branch:
`archive/docs-save-do-not-prune` and `archive/docs-retired-2026-06-17`.
- This PR intentionally does not delete the retired top-level `docs/`
tree; that should be a follow-up cleanup PR.
## Summary
Harden the showcase harness producer against transient Railway-GQL 429 /
Cloudflare-WAF flaps. Today's incident: a ~25-min Cloudflare WAF
burst-block on `backboard.railway.com/graphql/v2` caused the producer's
catalog-enumerate to hard-fail every cron tick, zeroing out D4/D5/D6
writes and turning the entire staging dashboard red within one tick
window.
Three discrete behavior changes, one PR:
1. **Retry with exponential backoff in `source.enumerate`** — three
retries at 1s/4s/16s on HTTP 429, 5xx, Cloudflare 1015/1020/1022
markers, or transport-level errors. Does NOT retry on
`DiscoverySourceAuthError`, non-429 4xx, or schema errors
(operator-actionable, fail loud). Lives in
`showcase/harness/src/fleet/control-plane/catalog-enumerator.ts` (the
seam every family enumerator passes through).
2. **Cached-catalog fallback** — per-enumerator in-memory cache of the
last successful `services[]`. On persistent failure (all retries
exhausted) AND a cache present, the wrapper logs
`fleet.producer.enumerate-failed-using-cache` (warn, with `services`
count, `ageMs`, and `reason`) and returns the cached catalog. With NO
cache (fresh-boot first enumerate fails), the wrapper re-throws so the
producer's `enumerate-failed` short-circuit still runs — without a
catalog there's nothing to enqueue.
3. **3-tick family-silence threshold** —
`SILENCE_CONSECUTIVE_TICK_THRESHOLD = 3` layered ON TOP of the existing
`3 × period` elapsed-time gate. The silence alert now requires BOTH:
`now - lastSuccessAt > 3 × period` AND three consecutive evaluation
cycles observed silent. A single bad tick on a stale `lastSuccessAt` can
no longer page every family at once.
## Red-Green proof
**RED on main (`5a62acbf`)** — observed BEFORE the fix. The RED file
asserts the BUG (`calls === 1` after a 429 throw; `posts.length === 1`
after a single silent tick):
```
> vitest run src/fleet/control-plane/red-baseline-railway-gql-resilience.test.ts
RUN v3.2.4 .../showcase/harness
✓ src/fleet/control-plane/red-baseline-railway-gql-resilience.test.ts (2 tests) 3ms
✓ [BUG] one 429 + Cloudflare 1015 from the source aborts the whole enumerate (no retry)
✓ [BUG] silence alert posts on the FIRST silent evaluation tick (no consecutive-tick threshold)
Test Files 1 passed (1)
Tests 2 passed (2)
```
**GREEN on fix branch** — observed AFTER the fix (assertions inverted to
the fixed behavior; `calls === 4` after retries; `posts === []` until
the third silent tick):
```
> vitest run src/fleet/control-plane/red-baseline-railway-gql-resilience.test.ts src/fleet/control-plane/catalog-enumerator.test.ts src/fleet/control-plane/family-silence-monitor.test.ts
RUN v3.2.4 .../showcase/harness
✓ src/fleet/control-plane/red-baseline-railway-gql-resilience.test.ts (2 tests) 4ms
✓ src/fleet/control-plane/family-silence-monitor.test.ts (19 tests) 8ms
✓ src/fleet/control-plane/catalog-enumerator.test.ts (29 tests) 6ms
Test Files 3 passed (3)
Tests 50 passed (50)
```
Full harness suite — **131 files / 2812 tests pass** (`pnpm -F
@copilotkit/showcase-harness test`).
## Test plan
- [x] RED proof captured on `main` (single-attempt enumerate;
single-tick silence alert)
- [x] GREEN proof on this branch (retry to 4 calls; cached fallback;
3-tick threshold)
- [x] `pnpm -F @copilotkit/showcase-harness test` — 2812 passed
- [x] `pnpm -F @copilotkit/showcase-harness typecheck` — clean
- [x] `pnpm -F @copilotkit/showcase-harness build` — clean
- [x] `oxfmt --write` applied; `oxlint` clean
- [ ] CI green
- [ ] Staging redeploy verified via Railway CLI + Playwright dashboard
snapshot
## Files touched
- `showcase/harness/src/fleet/control-plane/catalog-enumerator.ts`
(+296, -3): retry+cache wrapper, exports `ENUMERATE_RETRY_BACKOFF_MS` +
`isRetryableEnumerateError` + `SleepFn`
- `showcase/harness/src/fleet/control-plane/catalog-enumerator.test.ts`
(+295): retry/cache/auth-not-retried/backoff-SSOT tests
- `showcase/harness/src/fleet/control-plane/family-silence-monitor.ts`
(+61): `SILENCE_CONSECUTIVE_TICK_THRESHOLD = 3`, per-family counter,
counter reset on healthy
-
`showcase/harness/src/fleet/control-plane/family-silence-monitor.test.ts`
(+218, -39): 3-tick threshold + counter-reset gate tests; updated
existing tests to advance through 3 silent ticks
-
`showcase/harness/src/fleet/control-plane/red-baseline-railway-gql-resilience.test.ts`
(new, +212): the literal RED→GREEN gate
## Operational notes
- No new env vars, feature flags, or backward-compat shims (per scope
directive).
- The cached-catalog warn surfaces in observability via
`fleet.producer.enumerate-failed-using-cache` (services count, ageMs,
reason).
- `SLACK_WEBHOOK_OSS_ALERTS` is intentionally unset (user config); not
touched.
Pre-existing repo-wide lefthook failures (`@copilotkit/core`,
`@copilotkit/runtime`, `@copilotkit/shared` etc.) reproduce on `main`
without my changes and are unrelated to harness code; harness-scoped
quality gates all passed before commit.
## Summary
Ports
[ag-ui-protocol/ag-ui#1914](https://github.com/ag-ui-protocol/ag-ui/pull/1914)
to CopilotKit — plus the two supporting guard files ag-ui already had:
- **`.github/workflows/canary.yml`** — discoverable **`canary /
publish`** `workflow_dispatch` orchestrator. Any maintainer can publish
a prerelease of the branch they're on straight from the Actions tab. It
is a thin orchestrator — it does **not** publish to npm itself:
1. Guards against `main` and non-branch refs.
2. Mints the devops-bot App token (app-id `1108748`,
`DEVOPS_BOT_PRIVATE_KEY`) with scoped `contents:write` +
`actions:write`.
3. Mirrors the dispatched ref to a unique
`canary/<slug>-<run_id>-<attempt>` branch via the GitHub API (no
checkout).
4. Dispatches **`publish-release.yml --ref canary/<slug> -f
mode=prerelease …`**, locates the run, and waits (`gh run watch
--exit-status` + explicit conclusion check).
5. Deletes the canary ref — status-gated (never yanks the ref under a
still-running delegated run) with a fresh cleanup token (90-min job
ceiling exceeds the 1h App-token TTL).
- **`scripts/release/verify-release-scope-dropdowns.sh`** — drift guard:
the hand-maintained `scope` dropdowns in `publish-release.yml` /
`stable-release.yml` / `canary.yml` must exactly match
`release.config.json`'s `.scopes` keys. Parsers fail loud and distinct
on structural changes instead of silently passing.
- **`.github/workflows/lint-release-workflows.yml`** — actionlint +
shellcheck + the dropdown-sync job over the release pipelines.
### Why a separate orchestrator (and not a flag in publish-release.yml)
- A GitHub Environment's deployment-branch policy is evaluated against
the ref a run is **triggered on** — not branches created mid-run. The
orchestrator exists to get the publish run *onto* a `canary/*` ref.
- `publish-release.yml` holds the **single npm OIDC trusted-publisher
binding**; a second publishing entry point would break OIDC for every
`@copilotkit/*` package. The orchestrator never touches npm.
- The cross-workflow dispatch uses the **App token, not `GITHUB_TOKEN`**
— `GITHUB_TOKEN`-authenticated events never start new workflow runs.
**Note:** the `npm` environment currently has *no* deployment-branch
policy, so the orchestrator is a convenience wrapper today. Tightening
the policy to `main` + `canary/*` + `release/publish/*` (matching
ag-ui's security posture) is being applied as repo configuration
alongside this PR — requires admin. This PR includes the prerequisite:
`publish-commit.yml` (pkg-pr-new) is removed from the `npm` environment,
since it runs on every PR and would be blocked by the policy (it
publishes to pkg.pr.new, not npm, and uses no environment secrets).
## Testing done
- Drift guard: positive run against all three real workflows; negative
tests (scope removed → drift FAIL with diff; bogus scope → FAIL; `case
"${SCOPE}"` quoting refactor → loud parser-degradation FAIL; whole case
block deleted → loud zero-block FAIL; quoted arm `"angular")` →
accepted; blank/comment lines inside `options:` → still parsed; prose
comments mentioning case/SCOPE/in → no false positive).
- `shellcheck` clean at all severities; `bash -n` on every workflow
`run:` block; YAML parses.
- 3 rounds of 7-agent code review converged to zero load-bearing
findings.
## ⚠️ Still to verify before first real use
- [ ] devops-bot App (id 1108748) has **Actions: write** — required for
the in-workflow `gh workflow run`. Safe first test: dispatch once with
`dry_run=true`.
- [ ] First `dry_run=false` run clears the `npm` environment end-to-end
via the App token once the deployment-branch policy is tightened.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Post-merge follow-ups (maintainer action required)
These need repo **admin** rights and must happen **in this order**:
1. **Merge this PR first.** `main`'s current `publish-commit.yml`
(pkg-pr-new) still sits in the `npm` environment and runs on every PR
touching `packages/**` — applying the branch policy before this PR lands
would block every snapshot publish. This PR removes that environment
association.
2. **Tighten the `npm` environment's deployment-branch policy** to
`main` + `canary/*` + `release/publish/*` (matching ag-ui). With an
admin-scoped token:
```bash
gh api --method PUT repos/CopilotKit/CopilotKit/environments/npm \
-F "deployment_branch_policy[protected_branches]=false" \
-F "deployment_branch_policy[custom_branch_policies]=true"
gh api --method POST
repos/CopilotKit/CopilotKit/environments/npm/deployment-branch-policies
-f name="main" -f type=branch
gh api --method POST
repos/CopilotKit/CopilotKit/environments/npm/deployment-branch-policies
-f name="canary/*" -f type=branch
gh api --method POST
repos/CopilotKit/CopilotKit/environments/npm/deployment-branch-policies
-f name="release/publish/*" -f type=branch
```
Or via UI: Settings → Environments → npm → Deployment branches and tags
→ "Selected branches and tags" → add the three patterns above.
Why these three: `main` covers stable `workflow_dispatch` retries and
`stable-release.yml`; `release/publish/*` covers the merged-release-PR
runs (the run's head branch is the release PR branch); `canary/*` covers
the orchestrator's delegated prerelease runs. After this, direct
`mode=prerelease` dispatches from arbitrary feature branches stop
working — the `canary / publish` orchestrator becomes the one-click path
(by design).
3. **Verify the devops-bot App (id `1108748`) has `Actions: write`**
(org/App settings). The orchestrator's `gh workflow run` dispatch 403s
without it. Safe end-to-end test, after step 2: Actions tab → **canary /
publish** → pick any feature branch, any scope, **`dry_run=true`** →
confirm the delegated `release / publish` run is created, watched, and
the `canary/*` ref is deleted afterward.
4. **First real canary** (`dry_run=false`) confirms the npm OIDC publish
clears the environment gate end-to-end on a `canary/*` ref.
Adds an integration-style test file that pins the BOTH layers of the
2026-06-17 Cloudflare-WAF-burst incident fix together:
- the enumerator retries 3× on a 429+Cloudflare-1015 burst before
bubbling (the original bug let one 429 abort the whole enumerate);
- the silence monitor requires THREE consecutive silent evaluation
cycles before posting an alert (the original bug fired on tick #1).
These assertions were the LITERAL red proof on `main`:
- on main both `it` blocks PASSED while asserting the buggy behavior
(calls === 1, posts.length === 1 after a single silent tick),
- on this branch the same gates re-pin the fixed behavior (calls === 4
after retries; posts === [] until the third silent tick).
Run-time output captured for the PR body confirms the inversion.
Layer a per-family consecutive-silent-tick counter ON TOP of the existing
3×period elapsed-time gate (`SILENCE_PERIOD_MULTIPLIER`). The silence
alert now requires BOTH:
- `now - lastSuccessAt > 3 × period` (existing elapsed-time gate), AND
- `SILENCE_CONSECUTIVE_TICK_THRESHOLD = 3` consecutive evaluation cycles
observed silent (new — the counter resets on any successful evaluation).
Without the new gate a single bad cron tick on a family whose
`lastSuccessAt` was already stale (e.g. after a long quiet window or a
deploy gap) tripped the alert immediately — the failure mode the
2026-06-17 Cloudflare-WAF-burst incident exposed where one ~25 min flap
on backboard.railway.com/graphql/v2 paged every family at once.
The counter is NOT incremented during boot grace, so a cold-start cycle
can't alone push it to threshold. The meta-alert (`family-silence-eval`)
path keeps its own clock and is unaffected. Existing tests advance
through three consecutive silent ticks before asserting the post.
Three-retry exponential backoff (1s/4s/16s) on `source.enumerate` against
Railway-GQL when the underlying error is transient (HTTP 429, 5xx, or a
Cloudflare 1015/1020/1022 WAF marker, or a transport-level reject). On
persistent failure, fall back to the last successful catalog from a
per-enumerator in-memory cache — LOUDLY logged via
`fleet.producer.enumerate-failed-using-cache` so the cache-use shows up
in observability.
A fresh-boot process with no cached entry preserves the current
hard-fail behavior (the producer's `enumerate-failed` short-circuit) —
without a catalog there is nothing to enqueue. Real config errors
(`DiscoverySourceAuthError`, non-429 4xx, schema rot) are NOT retried so
operator-actionable failures surface immediately.
Context: 2026-06-17 Cloudflare WAF burst-blocked
backboard.railway.com/graphql/v2 for ~25 min, hard-failing the producer
enumerate on every cron tick and zeroing out D4/D5/D6 writes — the
entire staging dashboard went red within one tick. Retries + cache ride
out the burst on the same tick and preserve job production across
longer outages.
Pre-existing repo-wide lefthook test failures (@copilotkit/core,
@copilotkit/runtime, @copilotkit/shared, etc.) are unrelated to the
harness; verified by stashing my changes and running the same hook on
main with identical failures. Harness suite (131 files / 2812 tests),
typecheck, and build pass on this branch.
- Add control-plane-run.test.ts (17 tests) covering buildLocalServicesJson,
expectedKeys, dedupeScopes, and runViaControlPlane error surfacing
- Export SlugScope, buildLocalServicesJson, expectedKeys, dedupeScopes
for unit-test coverage (factored inline dedup loop into dedupeScopes
helper at the same time)
- runViaControlPlane: surface scopeLabel (demo-aware) in the 0-enqueue
error instead of the bare-slug join, with an empty-targets guard so
the error never renders with a double-space gap
- runViaControlPlane: treat tick.enqueueFailures > 0 as fatal — partial
enqueue used to silently proceed and either mask missing cells or
hang the poll loop to timeout
- Eliminate a stray literal NUL byte in the source by switching the
dedup key separator to a \x00 escape
- lifecycle.up(): name the compose call (infra-up vs target rebuild)
in the health-fail error so an operator can tell which call left a
service unhealthy
(cherry picked from commit 9f35c64adfdf7f5ff2bf0a5ae57ed03818cea607)
CR Finding 1 (BLOCKER): d5-tool-rendering-default-catchall.ts used the
broken page.evaluate(fn, arg) second-arg form to pass the leak-phrase
needle into the browser-side closure. A11 proved empirically that the
arg arrives as undefined inside the closure, making 'if (needle)' guard
the entire leak-detection cascade as dead code — customLeakPhrasePresent
stayed false forever, rendering validateDefaultCatchall's leak branch
dead code as well. Mirrors the A11 fix on the sibling custom-catchall
probe by inlining the needle as a JS string literal inside the closure;
no page.evaluate(fn, arg) dependency at all. Both probes now share the
same inline-needle pattern and keep the canonical literal in lock-step
with their exported phrase constant.
CR Finding 2 (MAJOR): the A11 inline-needle fix on the sibling
custom-catchall probe had no regression test — fake Page.evaluate in
makePageReturning never executes the probe closure, so reverting the
fix would not be caught. Added regression tests that capture the
probe's function source via toString() and assert (a) the canonical
phrase appears as a literal inside the page.evaluate(...) closure and
(b) the closure takes no parameter / the evaluate call has no
second arg. Added the same coverage to default-catchall to protect
the new A25a fix.
CR Finding 3 (MAJOR): A7's requireContentPhrase=true branch in
validateCustomCatchall / assertCustomCatchall had zero coverage —
tests omitted the third arg and exercised only the default false
branch. Added coverage for the true branch (pass on phrase present,
fail on phrase absent, fail on phrase undefined, default-branch
preserved) plus assertCustomCatchall plumbing through the options
form. Also added coverage for default-catchall's customLeakPhrasePresent
branch in validateDefaultCatchall for symmetry.
Local proof:
- RED (fix reverted via git stash): 2 inline-needle regression tests
fail on d5-tool-rendering-default-catchall.test.ts
- GREEN (fix restored): 34/34 tests pass across both files
Out of scope (NOT touched this commit): showcase/harness/src/cli/
control-plane-run.ts and lifecycle.ts (A25b's scope).
(cherry picked from commit c409e3a8d99e16ad0bb05ee3c2e2051e5792049f)
Re-tier the showcase docs tree to be an agent entry point: README.md
opens with a 'when X, see Y' fanout table that routes to the right
procedural doc; each procedural doc gets a one-line tagline answering
'what does this answer'.
Consolidation:
- DELETE showcase/RUNBOOK.md — operational content merged into DEBUGGING.md
(Integration Patterns, Docker Compose Environment, Production Debugging,
Anti-Patterns, Aimock Fixture Deployment, Dev Iteration Speed). The
--isolate mechanics + CLI rules were already duplicated in DEBUGGING.md.
- DELETE showcase/QA-COVERAGE.md — per-demo coverage matrix + starter hero
matrix + probe depth + infra locations + gaps folded into TESTING.md as
the 'Per-Demo Coverage Matrix' section.
Taglines added (no behavioral change to content): TESTING.md, DEBUGGING.md,
GOTCHAS.md, INTEGRATION-CHECKLIST.md, STYLING-GUIDE.md, FRONTEND-STRATEGY.md,
RAILWAY.md, bin/README.md, aimock/README.md, aimock/RAILWAY.md,
harness/README.md, harness/docs/rotation-drill.md.
Cross-link fixups: FRONTEND-STRATEGY.md (was QA-COVERAGE.md →
TESTING.md#per-demo-coverage-matrix), TESTING.md (removed dangling RUNBOOK
companion reference), README.md (rewritten as fanout entry + retained
from-scratch setup + dashboard SOPs below the fanout).
PARITY_NOTES.md × 12 left alone (per-slug context, not redundant).
(cherry picked from commit 75c9d9755c9118c8abc1fa52deda2012b768cab1)
(cherry picked from commit b64189bae0fe2c9e3a5e3ca440013deb4121f23b)
New content:
- TESTING.md: add 10-step cell red→green SOP + bin/showcase test invocation
table (control-plane vs --direct, per-demo scoping matrix); retain
existing CI gating matrix below.
- GOTCHAS.md: add operational gotchas — aimock caches fixtures at container
startup (warm-slot reuse needs docker restart) + --isolate slot collisions
with foreign Docker projects.
- README.md: cross-link to TESTING.md SOP from CLI section; flesh out
--isolate / --direct in test options table; update use cases.
- RUNBOOK.md: update Verifying a Slug's D6 State to use auto-named --isolate;
note A21+A21b per-slug rebuild scoping; rewrite Fixture Matching to teach
picking the backend-id-invariant discriminator (turnIndex post-A12/A13/A20);
modernize Debugging Sequence to --isolate flow.
- DEBUGGING.md: lead with TESTING.md SOP cross-link; update Phase 1 to
--isolate canonical; soften turnIndex-only log-line description; note
aimock startup caching in Phase 5; switch Strategy 5 gold-standard check
to --isolate.
Pruned/updated stale claims (post-A11/A12/A13/A18/A20/A21/A21b):
- RUNBOOK.md "Do not use turnIndex in new fixtures" — turnIndex is now
the canonical backend-id-invariant alternative when toolCallId is fragile
(Anthropic / TanStack Responses API ID rewrites). Replaced with discriminator
selection guidance.
- RUNBOOK.md anti-pattern "NEVER use turnIndex" — replaced with NEVER
anchor on toolCallId strict equality against ID-rewriting backends, and
NEVER use --direct for value-tests.
- RUNBOOK.md bin/showcase test <slug> --d5 (no --isolate) as canonical SOP
— replaced with --isolate canonical, no manual name required.
- README.md --d5 option description claiming "subagents/tool-rendering/agentic-chat"
fixed slate — replaced with "defaults to agentic-chat representative; :demo
qualifier honored post-A18".
- DEBUGGING.md Phase 1 "showcase up aimock <slug> && showcase test <slug> --d5"
as primary — kept as legacy alternative; --isolate is now lead.
- DEBUGGING.md Phase 5 "fixtures baked into Docker image" — clarified that
aimock additionally caches fixtures in memory at startup (volume-mounted
isolated stack still requires docker restart for warm-slot edits).
- DEBUGGING.md Strategy 5 "showcase test langgraph-python --d5" — replaced
with :demo + --isolate so the gold-standard check exercises the same cell.
(cherry picked from commit 0e548455043396972f7fb5b96f8c0ea8abdf1d98)
(cherry picked from commit 592c02d392350d02cc5e17544e663a6605b8da65)
Mirrors A19b's BIA fix pattern for the Anthropic-family csdkts integration.
Root cause: csdkts uses Anthropic SDK which generates its own toolCallIds (toolu_*) rather than echoing aimock's prescribed call_d6_cc_*. The fixture's toolCallId-gated narration entries never matched on turn-2, causing fall-through to less-specific entries (or 503/no-match).
Fix: replace toolCallId discriminator with turnIndex (count of role:assistant messages). turnIndex is backend-id-invariant — it works regardless of how the backend rewrites tool_call_id values. Same shape as A19b BIA fix.
- Tokyo narration: toolCallId → turnIndex: 1
- AAPL narration: toolCallId → turnIndex: 3
- AAPL emit: added turnIndex: 2
- Tokyo emit: turnIndex: 0
response.content + canonical phrase ("rendered through the custom wildcard catchall") and response.toolCalls UNTOUCHED.
Verified locally on cr5495/fix-a20-csdkts-green at HEAD d178e6730 (post-A21b):
- /tmp/cr/a20v6-green-csdkts.log: 1 passed, INNER_EXIT=0
- iso2 slot, full infra healthy (aimock+pocketbase+dashboard+csdkts)
(cherry picked from commit e66e0eb0ce72c970348183eeb4f4b57c3f5b1d29)
A21 (a9114a831) regression: positional slug after 'up' restricted which services start (only target+depends_on came up; infra profile services never started). With concurrent sibling stack on same host ports, health checks crossed to foreign pocketbase → cell results misrouted → 0.0s red. Fix: split into 2 compose calls — (1) compose infra up -d, then (2) compose --profile slug up -d --build slug. Preserves A21's target-only rebuild + restores full infra stack.
(cherry picked from commit d71ff46edb4a7b0a89f89d8126fd5fe3d5c7d24e)
Tyler Slaton