The smoke-test app Dockerfiles ran `RUN npm install next@latest` right after
installing the pinned deps from package.json. This overrode the deliberate
`next` pin with a floating version (non-deterministic builds) and added a
flaky network round-trip that broke the agno smoke build with ECONNRESET on
2026-06-04 (run 26972381969). Every affected starter already pins `next` in
package.json (agno 16.0.7, adk 16.1.1, crewai ^15.5.15, llamaindex 16.0.8,
ms-agent-framework-python 16.0.8, pydantic-ai 16.0.7), so the extra install
is pure harm. Removing it makes builds deterministic and removes the network
fragility.
GitHub Actions expression string literals don't interpret `\n`, so
`toJSON(format('...\n...'))` emits literal `\\n` and Slack renders the
two characters backslash-n instead of a line break. Inject real newlines
via a `fromJSON('"\n"')` placeholder, matching the starter-smoke fix.
Fixes the "all builds failed" and "Showcase Build Failed" alerts in
showcase_build.yml and the multi-line "showcase_validate failed" alert
in showcase_validate.yml. showcase_promote.yml already used the
fromJSON placeholder; the single-line validate alert has no newlines
and was left untouched.
The starter smoke failure alert built its message with literal `\n`
inside a GitHub Actions `format()` call. GHA expression string literals
do not interpret `\n` as an escape, so `format()` emitted the two
characters backslash+n, which `toJSON()` then encoded as `\\n` — Slack
rendered a literal "\n" and the triple-backtick fence as plain text
instead of a line break and a code block.
Inject real newlines via `fromJSON('"\n"')` so `toJSON` encodes them as
a single `\n` Slack honors, and place the code-fence delimiters on their
own lines so the failure summary renders as a proper code block.
PocketBase had no CI build path: `ghcr.io/copilotkit/showcase-pocketbase`
was a stale April `:latest`, and there was no way to ship pb_migrations /
pb_hooks changes without an ad-hoc manual build. Add a `pocketbase` slot to
showcase_build.yml's build matrix, mirroring the harness/aimock entries:
- dispatch_name `showcase-pocketbase`, context `showcase/pocketbase`, its
own Dockerfile, health `/api/health`, railway_id from the SSOT.
- a paths-filter key gated to `showcase/pocketbase/**` so the slot only
rebuilds when PB's own files change (the image is self-contained — no
shared-module copy), not on every showcase push.
- the workflow_dispatch service choice so PB is human-targetable.
Flip the SSOT entry (railway-envs.ts) to `ciBuilt: true` with
`dispatchName: "showcase-pocketbase"` so it is built+pushed (`:sha` +
`:latest`) and joins the default staging-redeploy scope; the build's
redeploy step only touches the matrix-intersect-success set, so PB still
only redeploys when its own files change. Regenerate
railway-envs.generated.json and the showcase_promote.yml service dropdown,
and update the SSOT/redeploy tests that pinned PB as out-of-band
(CI_BUILT_SERVICES 25 -> 26; webhooks stays the only non-CI-built service).
A PB image freshly built from main crash-loops staging PocketBase (502s)
because of two latent defects, both verified by booting the built image
against a real PB 0.22.21 binary on a volume that already has the
collections but has NOT recorded their migrations in `_migrations`.
1. Hook API. `pb_hooks/main.pb.js` registered its CORS middleware via the
bare global `onBeforeServe(...)`, which is undefined in PB 0.22.x JSVM
(only the `$app.onBeforeServe()` Go method exists) — it throws
`ReferenceError: onBeforeServe is not defined` at hook load and crashes
the server. Switch to the documented global `routerUse((next) => (c) =>
…)` entry point. Separately, the per-request closure runs in PB's pooled
goja runtime where top-level helpers/consts are out of scope, so calling
them throws per request and the router returns HTTP 400 on EVERY route;
inline the entire allowlist/env/match logic into the closure to fix that
second regression. Verified: health 200, collection reads 200, the
allowlisted origin is echoed on `Access-Control-Allow-Origin`, a
non-allowlisted origin is not, and OPTIONS preflight returns 204.
2. Migration idempotency. `1777700000_create_baseline.js` and the three
original `1745193*` creators (status, status_history, alert_state) called
`saveCollection(new Collection(...))` unconditionally, so on a volume
where the collection already exists they throw
`UNIQUE constraint failed: _collections.name`, aborting the ENTIRE
migration chain before later migrations (resource_snapshots, future fleet
collections) can run. Guard each with the proven find-or-skip pattern
already used by probe_runs / resource_snapshots, and harden their down
arms to tolerate an already-absent collection. Verified end to end:
deleting those migrations' `_migrations` rows while leaving the
collections in place (the exact staging state), then rebooting the built
image — boots healthy, re-records the migrations cleanly with no UNIQUE
abort and no duplicate collections, and a brand-new collection migration
still applies through the now-clean chain (the pool-fleet path).
Bump canonicalCopilotKitVersion 1.59.2 -> 1.59.4 and pin every
integration's @copilotkit/* to 1.59.4 (locks regenerated). Keeps the
whole showcase on one version instead of letting the langgraph A2UI
demos deviate. Existing per-slug overrides (built-in-agent pkg.pr.new,
ms-agent-harness-dotnet 1.57.2) unchanged.
The A2UI middleware now stamps recovery.debugExposure onto every a2ui_recovery
activity, so the server (covering Python and TS agents alike, since the middleware
is the single emitter) can drive how much retry/error detail the renderer surfaces.
Resolve debugExposure per-activity inside render() with precedence
content > client option > "collapsed" default, and declare it on
RecoveryContentSchema. Adds a test asserting the server value wins.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the hand-rolled render_a2ui/generate_a2ui + a2ui_prompt framing.
Document the two real paths: prebuilt agent (add CopilotKitMiddleware)
and graph agent (wire get_a2ui_tools). injectA2UITool stays the single
on/off switch. Fixes langgraph + deepagents + the generic page;
corrects the streamed op names (createSurface/updateComponents/
updateDataModel).
## Summary
Hardens a set of pre-existing `showcase` promote/`bin/railway` bugs
surfaced during review of the GHCR bearer fix (#5239) and the
#team-showcase notification (#5240). Each is an independent, real
defect; all changes are covered by tests (Ruby `bin/spec` **127 runs / 0
failures**, TS `verify-deploy` drivers **82 passing**, `actionlint`
clean).
## Fixes
1. **`rollback` could roll back to the wrong deployment.**
`find_previous_deployment` selected the second-newest SUCCESS from an
*unsorted* GraphQL result, and assumed the head deploy was always
SUCCESS — so when the latest deploy FAILED/CRASHED (the exact case
rollback is for) it rolled back one good deploy too far. Now sorts by
`createdAt` desc, selects the newest SUCCESS strictly older than the
current head (`sorted.drop(1).find { SUCCESS }`), and **fails loud**
(rather than silently mis-rolling) when the `first: N` window is
saturated with no valid target — telling the operator to pass `--to`.
2. **`env-diff` was dishonest.** It advertised custom-domain comparison
it never performed, and exposed a `--ignore-env-scoped` flag that was
parsed but never read. Now actually diffs `custom_domains` (the snapshot
already carried them), removes the dead flag/helper, and nil-guards
**every** accessor in `diff_services` (`services`, `env_keys`,
`custom_domains`) consistent with the rest of the file.
3. **`verify-prod` could vacuously pass.** Its empty-`succeeded_csv`
branch `exit 0`'d unconditionally. Now fails loud if `promote` reported
success but produced no succeeded set (contract violation), while still
skipping cleanly when promote genuinely failed.
4. **`verify-prod` raced the prod rollout.** It failed instantly when
the just-promoted deploy was still `DEPLOYING` (observed live: promote
succeeded, verify-prod failed ~17s later mid-rollout). `verify-deploy`
now polls in-progress statuses
(`DEPLOYING`/`BUILDING`/`INITIALIZING`/`WAITING`/`QUEUED`/`NEEDS_APPROVAL`)
until terminal (~150s budget), still failing fast on
`FAILED`/`CRASHED`/`REMOVED`.
5. **`npx tsx` ran from the wrong cwd** in the verify jobs (deps
installed in `showcase/scripts`, invoked from repo root → could fetch
`tsx` from the network). Now runs with `working-directory:
showcase/scripts`, matching the resolve/promote jobs.
6. **Slack payloads rendered literal `\n`.** Both promote Slack posts
used `toJSON(format('...\n...'))`, where the literal `\n` survives as
backslash-n (verified live in #team-showcase). Now uses the
`fromJSON('"\n"')` idiom for real line breaks.
## Test plan
- [x] `showcase/bin/spec` — 127 runs, 0 failures (new: rollback
head-FAILED + saturated-window, env-diff custom-domain + nil-guard
cases)
- [x] `showcase/scripts` `tsc --noEmit` clean; `verify-deploy` driver
tests 82 passing (in-progress→SUCCESS, →timeout, fast-fail-on-terminal)
- [x] `actionlint` clean
- [ ] CI green
## Out of scope (separate follow-up)
Review surfaced further pre-existing items intentionally NOT fixed here
(no diff overlap): `run_staging_probe`'s `IO.popen` nests its options
hash inside the argv array (stderr redirect verified working; the
`child_env` hash isn't applied but the probe inherits the parent env in
CI); `image_shape`/`parse_image_ref`/`PinCommand` colon-splitting for
registry-port refs (latent — portless `ghcr.io` only); the deeper
`DEPLOYMENTS_QUERY first:10` truncation beyond the new fail-loud guard;
`succeeded_csv` integrity under a 20-min promote-job timeout; and a few
comment/test-coverage nits.
## Release monorepo v1.59.4
**Scope:** `monorepo` | **Bump:** `patch`
---
### How this release process works
1. **This PR was created automatically** by the "release / create-pr"
workflow.
It bumped the `monorepo` packages to `1.59.4`
and generated AI-enhanced release notes.
2. **CI runs on this PR** — the full test suite (unit tests, lint, type
checks, build)
must pass before merging. This is the review gate.
3. **Review the release notes** in `release-notes.md` in this PR.
If a Notion draft was created, you can edit the release notes there
before merging.
4. **When this PR is merged**, the `release / publish` workflow
automatically:
- Builds all packages
- Publishes the `monorepo` packages to npm at version `1.59.4`
- Creates git tag `monorepo/v1.59.4`
- Creates a GitHub Release with the final release notes
### Before merging
- [ ] CI is green (tests, lint, types, build)
- [ ] Version bumps look correct
- [ ] Release notes are accurate (edit in Notion if a draft was created)
---
> **Do not merge until CI is fully green.** The full test suite runs
automatically on this PR.
## Summary
Three-tier wiring on the CopilotKit side, mirroring `useThreads`, to
surface user UI signals into CopilotKit Intelligence's self-learning
loop. Companion change in `CopilotKit/Intelligence` (PR #192) lands the
connector + schema.
- **Runtime client** — `CopilotKitIntelligence.recordUserAction(...)`
hits the idempotent platform endpoint
`${apiUrl}/connector/user-actions/record/:clientEventId`. Auth via the
deployment-level Intel API key (Bearer); the Intel key never reaches the
browser.
- **Runtime handler** — `handleRecordUserAction` resolves the Intel user
via `resolveIntelligenceUser`, forwards to the platform client, returns
`{ id, duplicate }`.
- **Fetch router** — `POST /user-actions` wired in
(`user-actions/record` `RouteInfo` variant + dispatch case).
- **React hook** — `useRecordUserAction()` and
`useRecordUserActionInCurrentThread()` in `@copilotkit/react-core/v2`.
Auto-generates a UUID `clientEventId` per call so retries are idempotent
by default. Throws when `runtimeUrl` is absent.
Linear: CPK-7587
## Test plan
- [ ] CI green on this PR
- [ ] Companion Intelligence PR #192 merged or coordinated
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Prebuilt agents get dynamic A2UI with no extra wiring — adding the
middleware is enough. When the frontend registers an A2UI catalog
(surfaced by the runtime into state["ag-ui"].a2ui_schema), the
middleware infers the agent's own model, advertises the generate_a2ui
tool in the model-call hook, and executes it in the tool-call hook. No
catalog → the tool is never advertised.
Covers both @copilotkit/sdk-js and the copilotkit Python SDK. Bumps the
A2UI tool-factory dependency to where get_a2ui_tools ships
(@ag-ui/langgraph 0.0.35, ag-ui-langgraph >=0.0.37).
find_previous_deployment returned the second-newest SUCCESS regardless of the
head deploy's status. When the head deploy FAILED/CRASHED (precisely when
rollback is invoked) the newest SUCCESS is the last-known-good target, but the
old code skipped it and rolled back one good deploy too far. Now select the
newest SUCCESS strictly older than the current head (drop head, first SUCCESS
in the remainder), and fail loud when the fetched window is saturated and no
target is found — directing the operator to pass --to explicitly rather than
silently returning nil.
Also guard the one unguarded accessor in EnvDiffCommand#diff_services with
`|| []` so a snapshot lacking "services" hits the documented exit-2 path
instead of a raw NoMethodError.
Renumber the ivar-lint allowlist to track the added lines in bin/railway.
verify-prod commonly runs seconds after a promote pins a new image digest,
while Railway is still rolling the container out. checkDeploymentSuccess (in
verify-deploy.drivers.baseline.ts — the SSOT for the deployment-SUCCESS gate
shared by every driver) treated a transient in-progress status as a hard FAIL:
promote run 26966193624's predecessor pinned the docs digest, then verify-prod
fired ~17s later and FAILED with status="DEPLOYING" — a race, not a failure.
Now the deployment-status check polls when the latest deployment sits in any
non-terminal Railway status (QUEUED/BUILDING/INITIALIZING/DEPLOYING/WAITING/
NEEDS_APPROVAL), re-querying every 5s up to a 150s budget until it reaches a
terminal state, then asserts SUCCESS. Terminal-failure statuses (FAILED/
CRASHED/REMOVED/any non-SUCCESS-non-in-progress) still fail FAST with the
original error-string shape — no waiting. Infra/contract errors (network,
GraphQL errors[], missing edge) also fail fast.
The poll loop is fully seamed for tests (injectable sleep/now/budget); the
signature stays backward-compatible (trailing optional pollOpts). probeBaseline
forwards an optional deployPoll through, and both --env staging and --env prod
go through the identical code path, so staging behavior is unchanged.
Red-green tests: in-progress-then-SUCCESS passes (and polls), in-progress-until-
timeout fails with "still in progress", terminal FAILED fails fast with zero
sleeps.
Three pre-existing showcase_promote.yml issues surfaced by code review:
1. verify-prod's empty-SERVICES_CSV branch exited 0 vacuously, relying on the
unenforced assumption "promote already failed if the succeeded set is
empty". Pass PROMOTE_RESULT into the step env and fail loud on contract
violation: if promote reported success yet emitted no CSV, emit
"::error::promote reported success but succeeded_csv is empty — contract
violation" and exit 1. The genuine all-failed case keeps the ::notice:: +
exit 0.
2. Both verify-deploy.ts invocations (verify-staging-precondition, verify-prod)
ran `npm ci` in showcase/scripts but then `npx tsx showcase/scripts/...`
from the repo root, so npx could network-fetch tsx instead of using the
local install. Give those steps working-directory: showcase/scripts and the
relative path verify-deploy.ts, matching how resolve-targets/promote run
emit-railway-envs-json.ts. verify-deploy.ts takes no cwd-relative file args
(only --env/--services) and imports ./railway-envs relative to its own file,
so the cwd change is safe.
3. The notify job's Slack payloads embedded literal '\n' inside
toJSON(format('...\n...')). GitHub Actions expression string literals do not
interpret backslash escapes, so the '\n' survived toJSON as the two chars
\\n and Slack rendered it verbatim as "\n" instead of a line break.
EMPIRICALLY CONFIRMED against the live #team-showcase webhook: an existing
production post from a real promote run showed literal "\n", and a corrected
payload (real LF) rendered proper line breaks. Fixed BOTH posting steps by
injecting newlines via fromJSON('"\n"') (a real LF char) as a format()
argument instead of a literal '\n' in the template.
Two pre-existing bin/railway bugs surfaced by code review:
1. RollbackCommand#find_previous_deployment selected "the second SUCCESS in
reverse-chronological order" but never sorted the deployments — it trusted
Railway's arbitrary GraphQL connection order, so the wrong deploy could be
chosen as the rollback target. Now sort by createdAt descending before
selecting, mirroring the sibling fetch_latest_staging_deployments which
documents and guards against this exact hazard.
2. EnvDiffCommand advertised behavior it never implemented:
- The banner claimed it compares "custom domains", but run() only diffed
digest/startCommand/env_keys. custom_domains was captured in the snapshot
yet never compared. FIX: extract a pure diff_services helper and add a
custom-domain set comparison (both directions).
- --ignore-env-scoped was parsed into options[:ignore_env_scoped] but never
read, and the env_scoped? helper (+ ENV_SCOPED_URL_MARKERS) it relied on
was defined but never called. The flag had nothing meaningful to act on:
env-diff only compares env-var KEY names (snapshots never carry values),
and custom domains are compared as exact sets (silently dropping
"env-scoped" domains would defeat the comparison). DECISION: remove the
dead flag, the unused env_scoped? helper, and the now-orphan
ENV_SCOPED_URL_MARKERS constant rather than leave false advertising.
Tests: TDD red->green for both. test_rollback_sort feeds deployments in
non-chronological edge order (with a newer FAILED deploy as a decoy) and
asserts the true second-newest SUCCESS is returned. test_env_diff exercises
diff_services directly for identical and differing custom_domains. The
snapshot-ivar lint allowlist is renumbered (numbers only) to track the line
shifts from removing the dead code and adding the rollback comment.
## What does this PR do?
Bumps the default Gemini model across the Google ADK showcase from
`gemini-2.5-flash` to `gemini-3.1-flash-lite` — the closest GA model in
the Gemini 3.x line. The Gemini 2.5 family is slated for deprecation in
June 2026, so this gets ahead of the cutoff.
**Files touched (14):**
- Agent factories — `DEFAULT_MODEL`, `_DEFAULT_A2UI_MODEL`, `_SUB_MODEL`
in `shared_chat.py`, `main.py`, `beautiful_chat_agent.py`,
`subagents_agent.py`
- Prose references — docstrings in `shared_chat.py`,
`multimodal_agent.py`, `tool_rendering_reasoning_chain_agent.py`,
`registry.py`; comment in `entrypoint.sh`
- Test fixtures — `test_generate_a2ui.py` default-model assertion,
`test_stop_on_terminal_text.py` docstring
- Docs/config — `manifest.yaml` reasoning-demo description, QA prereqs
in `multimodal.md`, `headless-complete.md`, `headless-simple.md`
### Why Flash-Lite and not Flash?
There's no standalone `gemini-3.1-flash` identifier. The 3.1 Flash
family only ships as Flash-Lite (GA), Flash Live (preview), Flash Image
(preview), or Flash TTS (preview). Flash-Lite is the only stable choice;
the non-Lite `gemini-3-flash-preview` is still preview-only.
### Worth verifying empirically
A few behaviors that may differ from 2.5-flash and warrant a smoke test
before merging:
- **Thinking mode** (`build_thinking_chat_agent`, used by
`reasoning-custom` and `tool-rendering-reasoning-chain`) — Gemini 3.x
uses `thinkingLevel` instead of `thinkingBudget`, and Lite variants
historically have reduced reasoning vs full Flash/Pro
- **Forced tool-calling**
(`ToolConfig.function_calling_config.mode="ANY"` in A2UI planner)
- **Multimodal** image + PDF (Flash-Lite docs explicitly support
text/image/video/audio/PDF, so this should work)
- **`stop_on_terminal_text` loop-stopper** — the quirk was originally
observed on 2.5-flash; whether Flash-Lite still needs the guard is open,
but leaving it in is harmless
## Related PRs and Issues
- N/A — proactive model bump ahead of Gemini 2.5 deprecation
## Checklist
- [x] I have read the [Contribution
Guide](https://github.com/copilotkit/copilotkit/blob/master/CONTRIBUTING.md)
- [x] If the PR changes or adds functionality, I have updated the
relevant documentation
- [x] "Allow edits by maintainers" is checked
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Red-green coverage for the writer-hardening fixes:
- in-flight cap DROP emits dropped-over-inflight and the counter does not leak
- a hung create times out, counts as a failure, and escalates after the threshold
- once writes are timing out, concurrent drops contribute to the escalation
- a drop while a write succeeds (healthy backpressure) does NOT escalate
- a no-progress deleteByFilter over cap backs off (gated by interval, not per insert)
- a cold start far over cap prunes in bounded batches (<= 200), never a thousands-row list/filter
- a hung Slack webhook is aborted at the timeout so the serialized health-signal chain isn't stalled
Three durable-logging robustness fixes in the resource-snapshot writer:
- PRUNE BATCH CAP + NO-PROGRESS BACKOFF: a cold start over a large backlog
(or lowering RESOURCE_SNAPSHOT_MAX_ROWS on a full collection) made
`surplus` thousands large, used directly as a `perPage` and a
one-clause-per-row id-delete filter — a giant request/URL that PB
rejects (400/414) so the prune never converged. Cap each sweep to
PRUNE_BATCH_MAX (200) oldest rows and let knownOverCap drain the rest
across inserts. When a sweep makes NO structural progress while still
over cap (PB delete-rule 403, filter 400, or 0 matched), back off via a
suppression window so the retry is gated to once-per-interval instead of
re-arming knownOverCap and re-running the failing sweep on every insert.
- DROP→ESCALATION SEAM: an in-flight-cap DROP did not count toward the
consecutive-failure escalation, so a 100%-broken PB whose writes hang
could keep new snapshots on the DROP path and delay/dodge the loud
`write-failing-systematically` error. A drop now contributes to the
failure signal when writes are already failing (consecutiveFailures > 0),
with no false positive on healthy backpressure (a drop while writes
succeed does not escalate).
- ORPHAN-CREATE REJECTION: on a write timeout the race rejects but the
orphaned `pb.create` keeps running; a later rejection from a client that
ignores the AbortSignal would be unhandled. Attach a no-op catch.
formatGauges had no production callers after the fix round removed the
gauge-string logging; only its own test referenced it. Delete the dead
function and the describe("formatGauges") block.
Add coverage (slot-6 F1) that `writeUnrecoverable` NAMES the proven wedge signal
in the Slack/alarm message when the give-up snapshot measured the cgroup PID
ceiling (cgroupPidsCurrent >= 0) — pids.current/pids.max + threads — and OMITS
the clause when the gauges are the off-Linux `-1` sentinel.
Documentation the diff invalidated, swept to match reality:
- browser-pool.ts JSDoc/comments + browser-pool.test.ts comments claimed
hard-recovery "purges the stale /tmp/playwright_* profile dirs". The code is
purge-free — hardRecover is a PACED cold relaunch that gives the
thread-exhausted kernel time to relax. Rewrite to match: paced cold relaunch,
NO /tmp purge (the wedge is the cgroup pids ceiling, mitigated demand-side;
/tmp is a proven non-cause).
- probe-invoker.ts stagger comment said "8 simultaneous workers", "~400 thread
spawns", "10-minute D6 budget" → updated to the current 5 / 20-min reality.
- e2e-deep.yml: the global cap note (40, "D6 peak 32 + D5 peak 8") → 24, with
D6 peak now 5×4=20; 20+8=28 > 24 so a d6+d5 overlap serializes under the
global cap (intended demand-side back-pressure).
- d6-all-pills-e2e.yml: the worked example (max_concurrency 8, ceil(18/8)=3
rounds) → 5 / ceil(18/5)=4 rounds, cross-referencing the overlap block that
justifies the concurrency itself.
Robustness/bounding nits a 7-agent CR flagged:
- browser-pool: the hot acquire/release path emitted a SECOND `hot-gauges` info
log duplicating inUse/available already on the acquire/release line. Fold the
two cheap cgroup-pid fields INTO the existing acquire/release log line (one
line, not two); keep the cheap two-file-read sampling (now `readHotGauges`).
- probe-invoker: `logTickGauges` double-encoded the sample as both a formatted
`gauges:` string AND the structured `...g` spread. Drop the redundant string;
keep the queryable spread.
- orchestrator: the best-effort Slack ping in `writeUnrecoverable` had no
timeout, so a hung webhook could stall the serialized health-signal write
chain. Add an AbortController timeout on the fetch.
Normalize the migration schema comments to reality: the heartbeat is 45s (not
the stale "~30-60s"), retention prunes by stable row id (robust to
same-millisecond observed_at ties) rather than a bare timestamp cutoff, and
document the null-vs-unavailable convention — the nullable number fields store
`null` for the `-1` "unavailable" sentinel so post-wedge queries cleanly
separate a measured reading from an unavailable one.
The snapshot writer had several robustness gaps a 7-agent CR flagged:
- PRUNE TIES: the ring prune deleted `observed_at < cutoff` (strict), but
observed_at is ms-resolution and several snapshots can fire in the same
millisecond (degraded+heartbeat+crash). If surplus rows tied the boundary
timestamp, strict `<` deleted nothing → knownOverCap stayed true → every
insert re-ran a full no-op sweep → unbounded growth + per-insert cost. Now
the prune deletes the oldest `surplus` rows by stable row id, which converges
regardless of ties, with a deleted==0 safety valve that can't spin.
- RATE-LIMIT CATCH-UP: a local row-count estimate (re-anchored to the real
total every sweep) forces a catch-up prune the moment we cross the cap, so the
rate-limit only suppresses sweeps while UNDER cap and never strands us over.
- FAILURE ESCALATION: swallow+warn alone hid a systematic PB rejection (unrun
migration / outage / schema drift). Now consecutive failures are counted, the
per-failure warn is throttled, and after a threshold the writer escalates ONCE
to logger.error (latched, reset on first success).
- BOUNDING: a best-effort in-flight cap drops snapshots past N concurrent
unawaited creates (launch-crash-loop burst), and a Promise.race write timeout
frees the in-flight slot if a PB write hangs.
- NULL SENTINELS: the `-1` "unavailable" gauge sentinel is written as `null`
(PB number fields are nullable) so post-wedge queries separate measured-vs-
unavailable.
- DOCS: cadence prose + JSDoc updated to the real 45s heartbeat, and a
single-writer note that the fleet case needs an elected pruner.
Red-green tests added: identical-boundary-timestamp prune converges to maxRows;
rate-limit keeps prune off the steady-state path but knownOverCap catches up
over cap; N consecutive failures escalate to error once and reset on success;
`-1` gauge written as null.
browser-pool.ts imported `./resource-gauges` without a `.js` extension while
sibling files use `.js`. The harness builds with tsc (module: esnext) and runs
the emitted ESM directly (`node dist/orchestrator.js`); moduleResolution:
bundler lets it typecheck but tsc emits the specifier verbatim and Node ESM
does NO extension resolution, so the dist would fail at boot with
ERR_MODULE_NOT_FOUND. Append `.js` to the extensionless relative specifiers in
browser-pool.ts and resource-gauges.test.ts. Verified by building the dist and
boot-importing dist/orchestrator.js (BOOT-IMPORT-OK, no module-not-found).
The live wedge peaked 998/1000 against the cgroup pids.max=1000 ceiling
specifically during a d6+d5 overlap — the simultaneous chromium renderers of
both probes summed past the ceiling. Fewer concurrent d6 feature-workers cut
the simultaneous-renderer thread peak, leaving headroom for a co-firing d5 (or
a recovery relaunch) without crossing 1000. Complementary to the
MAX_CONTEXTS 40->24 lever already landed. Comment updated to the pids-ceiling
reasoning; 5 x 4 = 20 concurrent contexts stays under the 24-context cap.
Previously full gauges were logged on only 3 sites (launch, self-heal-launch-
failed, probe-tick). Extend forensic gauge coverage across the whole pool
lifecycle so a recurring wedge has copious history to look back at.
- Full forensic snapshot (sample + log + durable onSnapshot hook) on every
meaningful transition: init, recycle (crash/hygiene), crash/disconnect,
set-empty degraded, self-heal-launch-failed, recovered, unrecoverable, and
shutdown.
- Periodic heartbeat (~45s, env BROWSER_POOL_HEARTBEAT_MS) via a
shutdown-signal-gated self-rescheduling loop — NOT a raw setInterval that
leaks past shutdown — for a baseline trend between events.
- Cheap cgroup-PID-only subset (logHotGauges) on the hot acquire/release path:
high-resolution view of the headline pids.current/pids.max signal without
the full-sample /proc-walk + df cost, and no durable write on the hot path.
- onSnapshot is best-effort: a throwing hook is caught + logged, never crashes
the pool.
The browser-pool wedge ends in a container restart that clears in-memory
state, and Railway's stdout window rolls off — so stdout/in-memory gauges
are not retrievable post-wedge. Persist the OS resource gauges to a new
`resource_snapshots` PB collection so the forensic history survives the
restart and the PID/thread-ceiling exhaustion is reconstructable after.
- New PB migration `1779989300_create_resource_snapshots.js` (mirrors the
probe_runs idempotent create + public-read pattern).
- `resource-snapshot-writer.ts`: best-effort writer (swallows + logs every PB
error so a missing migration / PB hiccup never breaks the pool) with
ring-style retention (cap last N rows, default 5000, env-overridable).
- Orchestrator wires the writer to the pool's onSnapshot hook, reusing the
shared pb client.
The Dockerfile FIX#3 comment claimed the in-image `ulimit -u` raised the cgroup
pids.max (target 16384). It cannot: `ulimit -u` only lifts this process's
RLIMIT_NPROC; the cgroup pids.max is set by the container runtime
(--pids-limit), is platform-fixed at 1000 on staging, and is not raisable from
the image. Rewrite the comment to state this accurately and point at the real
demand-side mitigation (fewer concurrent contexts + gauge/alarm). The harmless
`ulimit -u` line itself is retained. Also update the env test to assert the new
MAX_CONTEXTS default of 24.
The wedge was reproduced as cgroup PID/thread-ceiling exhaustion (pthread EAGAIN
-> "has been closed" -> crash-loop with pids.current pegged at pids.max=1000).
Every /tmp candidate was refuted by measurement, so this removes the /tmp
machinery that targeted a non-cause and replaces it with the demand-side fix +
observability:
- Remove the proactive stale-profile sweep (init + recycle), defaultPurgeProfileDirs,
the PurgeProfileDirs type, the staleProfileDirTtlMs / PURGE_MIN_AGE_MS options
and env, and the breaker's /tmp purge — the hard recovery is now just the paced
cold relaunch. Corresponding purge/sweep tests removed.
- KEEP the circuit-breaker failure-counting -> give-up -> onUnrecoverable alarm
(root-cause-agnostic backstop) and the footgun clamp; the breaker tests are
adapted to drive the wedge via the launcher instead of the purge.
- Log the resource gauges on every launchBrowser(), on the self-heal-launch-failed
path, and at probe-tick start/end so a burst approaching pids.max is observable
and an EAGAIN correlates to a measured pids.current.
- Include pids.current/pids.max + thread count in the onUnrecoverable payload and
the orchestrator's terminal alarm message so the alert names the real signal.
- Lower the BROWSER_POOL_MAX_CONTEXTS default 40 -> 24 to cap thread demand under
the platform-fixed ceiling (still env-overridable).
Standalone, unit-tested gauge sampler that reads the cgroup PID controller
(pids.current / pids.max, v2 with v1 fallback) plus the process-tree thread/proc
counts, FDs, RSS, /dev/shm, and /tmp inode/space from /proc + /sys/fs/cgroup.
The cgroup PID counters are the headline signal for the proven browser-pool
wedge (PID/thread-ceiling exhaustion). Degrades gracefully to -1 off-Linux so it
is safe to import anywhere. Consumed by the browser pool and probe invoker in
follow-up commits.
The recurring browser-pool wedge's ROOT CAUSE: every chromium.launch() leaves
a /tmp/playwright_* profile dir + playwright-artifacts-* dir whose Playwright
close-time cleanup is best-effort (swallowed on failure). In a long-lived
harness container under sustained d6 cron load these orphans accumulate
monotonically until /tmp exhausts inodes/space — then every launch throws
"...has been closed" and self-heal relaunches into the same exhausted /tmp
forever (only a restart, which gives a fresh /tmp, clears it).
PREVENTION: sweep the age-gated stale dirs PROACTIVELY on a cadence — once at
init() and on EVERY recycle relaunch (crash + hygiene) — so /tmp never reaches
exhaustion. This is the durable complement to the circuit-breaker ESCAPE: the
sweep prevents the wedge; the breaker escapes it if it ever still happens.
Unifies the purge into ONE canonical helper — defaultPurgeProfileDirs now takes
an age threshold (minAgeMs) as its first arg. The proactive sweep passes the
conservative staleProfileDirTtlMs (5min default, env
BROWSER_POOL_STALE_PROFILE_DIR_TTL_MS); the breaker's hard-recovery keeps its
aggressive PURGE_MIN_AGE_MS (60s). Same hardened (symlink-safe, realpath-checked)
implementation, different age — no duplicate purge logic.
Adds 4 prevention tests (init-sweep, every-recycle-sweep, bounded-accumulation
over 30 cycles, real-FS age gate) and adapts the existing breaker tests to gate
their purge-count assertions on the breaker's age so the shared cadence does not
inflate them.
defaultPurgeProfileDirs (the self-heal hard-recovery profile-dir purge) ran a
blind readdir + rm(recursive, force) against the world-writable OS temp dir:
- A planted playwright_* SYMLINK had its TARGET recursively deleted. Now read
dirents (withFileTypes), SKIP symlinks + non-directories, and verify the
realpath stays under the scan dir before removing.
- A browser launching CONCURRENTLY with the purge had its FRESH profile dir
nuked, re-wedging the very launch being revived. Now only dirs older than
PURGE_MIN_AGE_MS (60s) are eligible (mtime via lstat).
The real purge previously had ZERO coverage (the pool tests inject a no-op).
Parameterize the scan dir so it is unit-testable against a temp fixture, export
it, and add tests: removes old playwright_*/playwright-artifacts-* dirs, leaves
non-matching dirs, SKIPS symlinks (target untouched), leaves too-fresh dirs, and
swallows ENOENT. (Also pulls in the resolvePositive helper used by the breaker
clamp, which lives adjacent to this function.)
Two circuit-breaker robustness fixes:
Latch second-episode silence: this.unrecoverable was cleared ONLY on a
successful launch, so a permanently-wedged container that re-empties after the
first give-up hit `if (this.unrecoverable) return` and went silent on every
subsequent degraded episode — the exact silent-spin the breaker exists to kill.
Remove the instance latch entirely; the loop-local consecutiveHardRecoveries
guard already gives once-per-episode (the loop returns immediately after firing),
and each fresh degraded episode spawns its own self-heal loop that can alarm.
Stale JSDoc updated to match. The onUnrecoverable hook now carries breaker
counters so the wired alarm can report how hard the pool tried.
Disable footgun: a selfHealHardRecoveryThreshold / selfHealMaxHardRecoveries of
0 silently disabled both the hard-recovery escape AND the give-up (guards are
`> 0`), reverting to the infinite silent spin. Clamp resolved breaker thresholds
to >= 1 (resolvePositive) so a config typo can't disable the safety net.
Tests: a second degraded episode ALSO fires onUnrecoverable (red→green: prior
latch silenced it); a 0 threshold stays armed (red→green: prior it spun
forever); plus a browsers:2 multi-browser test locking in the partial-revive
counter-reset / breaker pacing that was previously unverified.
The self-heal circuit-breaker's terminal give-up shipped a mechanism with no
operator signal — the pool's onUnrecoverable hook was never wired, so a
breaker give-up silently stopped spinning without telling anyone a redeploy is
required. Add createBrowserPoolHealthSignals.writeUnrecoverable and wire it to
the pool's onUnrecoverable hook:
- Unconditional health-signal writes: a DISTINCT system:browser-pool-unrecoverable
key (red, terminal, critical, redeployRequired + breaker counters) AND an
escalation of the shared degraded key to severity=critical/terminal, so a
give-up is distinguishable from a transient self-heal-degraded.
- Best-effort, env-guarded Slack ping on SLACK_WEBHOOK_BROWSER_POOL_UNRECOVERABLE
(unset by default per alerting discipline — deploy the code, wire the URL
separately); a failing/absent webhook never blocks the health-signal write.
The pool's onUnrecoverable hook now passes breaker counters so the alarm can
report how hard the pool tried before giving up.
The recurring staging BrowserPool collapse (#5185/#5221/#5225 each chipped at it but it kept recurring): after the long-lived harness container runs ~hours under sustained d6 cron load, chromium enters a launch crash-loop where every chromium.launch() throws 'Target page, context or browser has been closed'. The set empties, startSelfHeal() kicks in, and its loop just relaunches into the SAME wedged state forever (self-heal-launch-failed repeating) without ever escaping — the stale /tmp/playwright_* profile dirs / FD-shm pressure persist across every relaunch. acquire() therefore has no contexts forever and blocks to timeout fleet-wide; only a container restart cleared it (reactive).
Adds a circuit-breaker to the self-heal loop: after selfHealHardRecoveryThreshold (default 4) consecutive launch failures, instead of looping another identical relaunch the pool performs a HARD recovery — purges the stale ${tmpdir()}/playwright_* profile/temp dirs the crash-looping processes left behind, then cold-launches fresh. Any successful launch resets the counter. If selfHealMaxHardRecoveries (default 3) consecutive hard recoveries also fail, it fires a loud browser-pool.pool-unrecoverable alarm (onUnrecoverable hook) and stops the heal loop rather than spinning silently — the operator signal that a redeploy is genuinely required. All tunable via BROWSER_POOL_SELF_HEAL_* env vars; the purge is injectable for tests and defaults to a hermetic no-op when a fake launcher is injected.
## Summary
Adds a Slack notification to **#team-showcase** whenever a showcase
staging→prod promotion **succeeds**, matching our existing routing
convention (releases → #engr, PRs/promote-failures → #oss-alerts). Today
the promote workflow only posts to #oss-alerts on failure; the `success`
state was computed but unused.
## Changes (all in the `notify` job of `showcase_promote.yml`)
- New **success-only** "Post to #team-showcase" step, gated on `state ==
'success'`, mirroring the existing #oss-alerts step exactly (same pinned
`slackapi/slack-github-action@b0fa283 # v2.1.0`, `incoming-webhook`,
`toJSON(format(...))` payload).
- Job-level `SLACK_WEBHOOK_TS: ${{ secrets.SLACK_WEBHOOK_TEAM_SHOWCASE
}}` so the step `if:` can gate on `env.SLACK_WEBHOOK_TS != ''` (step
`if:` can't read `secrets.*` directly — same pattern as the existing
`SLACK_WEBHOOK`).
- Symmetric "Log (no Slack — team-showcase webhook unset)" `::notice::`
fallback, so a successful promote with the webhook unset still leaves a
trace (mirrors the failure path's fallback).
- Failures still route **only** to #oss-alerts — unchanged.
## Activation (required before notifications fire)
The new step **no-ops until** an org secret
`SLACK_WEBHOOK_TEAM_SHOWCASE` exists, so this is safe to merge now. To
activate: create the `#team-showcase` Slack incoming webhook, then set
it as a CopilotKit **org** secret (visibility=all), matching the
convention for `SLACK_WEBHOOK_OSS_ALERTS` / `SLACK_WEBHOOK_ENGR`.
## Test plan
- [x] `actionlint` exit 0
- [x] YAML parses
- [x] Step is mutually exclusive with the webhook-unset log fallback
(one fires per successful run)
- [ ] CI green
- [ ] After the org secret is set: a real promote posts to
#team-showcase
## Out of scope (separate follow-up)
Review surfaced several pre-existing `notify`/`verify-prod` robustness
items not touched here (verify-prod empty-CSV vacuous-green hardening,
`npx tsx` working-dir in the verify jobs, env-vs-secrets
webhook-reference fragility, literal `\n` rendering shared with the
existing #oss-alerts step). To be addressed in a separate
showcase-promote hardening PR.
Jordan Ritter