mirror of
https://github.com/CopilotKit/CopilotKit.git
synced 2026-09-14 16:26:20 +08:00
Jordan Ritter
3998f7fbdb
## Summary Two confirmed bugs in the showcase harness + operator dashboard that together produced a false-green D3 on the dashboard while the e2e probe pipeline was actually dead. ### Bug 1 — Browser-pool death spiral (`showcase/harness/src/probes/helpers/browser-pool.ts`) When a chromium process crashed, `recycleSlot` relaunched it; if that launch threw, the slot was permanently removed via `slots.splice`. A single transient launch failure (OOM spike, fd exhaustion) thus monotonically shrank the pool to empty with no self-heal, after which every e2e probe failed with `launcher-error` / `BrowserPool acquire timeout`. Fix: - The recycle retries the relaunch with bounded backoff (3 attempts, doubling backoff). - On exhaustion the slot is **kept** (capacity preserved) and parked as `relaunchPending`; the next `acquire()` lazily re-attempts the launch. - `acquire()` re-initializes the pool as a backstop when every slot has been lost (`slots.length === 0`). - Existing acquire/release/recycle semantics and logging style preserved. ### Bug 2 — Stale-green e2e rows read as healthy (`shell-dashboard/src/lib/cell-model.ts`, `src/components/depth-utils.ts`) When the e2e driver stops writing `e2e:<slug>/<feature>` rows, the last row freezes. A green row then reads as a healthy D3 forever, so the depth ladder shows a false-green D3 that masks a dead probe pipeline instead of surfacing it. Fix: - `buildCellModel` and `deriveDepth` treat a green e2e row whose `observed_at` is older than `E2E_STALE_AFTER_MS` (6h, matching the original stale-window model) as degraded/amber, so it no longer credits D3. - Only green is downgraded — a stale red/degraded row already signals a problem and is left as-is. - An unparseable timestamp is treated as not-stale, so staleness is never inferred from bad data. ## Test plan - [x] New unit test: pool recovers after a transient relaunch failure (does NOT shrink to 0) and re-inits when emptied — confirmed red→green. - [x] New unit tests: a stale green `e2e:` row downgrades to amber (not green) and does not credit D3; a fresh green row stays green; a stale red row stays red — confirmed red→green. - [x] Existing fixtures with hardcoded old `observed_at` updated to default to a recent timestamp so legitimate green rows aren't treated as stale. - [x] `showcase/harness` full suite: 1646 tests pass. - [x] `shell-dashboard` full suite: 630 pass (1 env-gated build-spike skipped). - [x] `packages/**` test graph: 17 projects pass. - [x] typecheck clean (both packages); oxlint 0 errors; oxfmt clean. Note: there is a separate operational lever (reduce `BROWSER_POOL_SIZE` / e2e concurrency, raise harness memory) handled outside this PR; this is the code fix only.