A red/BE✗ dashboard cell is often staleness, not a broken feature. The
per-cell BE flag is resolveD4 = worst-of(chat,tools) folded by a staleness
window; a green row older than its window folds to stale-red even when the
app is healthy. Add D5 Strategy 10 with a diagnostic checklist (harness
/api/runs sweep-duration vs window, observed_at age vs *_STALE_AFTER_MS,
numReplicas × HARNESS_POOL_COUNT concurrency) plus an anti-pattern entry.
Evidence from the 2026-06-26 prod incident: D4 sweep (127m d6 / 97m e2e-demos)
blew past the 60m window with a starved worker pool, stale-reddening ~13
integration columns while apps were fine.
Re-tier the showcase docs tree to be an agent entry point: README.md
opens with a 'when X, see Y' fanout table that routes to the right
procedural doc; each procedural doc gets a one-line tagline answering
'what does this answer'.
Consolidation:
- DELETE showcase/RUNBOOK.md — operational content merged into DEBUGGING.md
(Integration Patterns, Docker Compose Environment, Production Debugging,
Anti-Patterns, Aimock Fixture Deployment, Dev Iteration Speed). The
--isolate mechanics + CLI rules were already duplicated in DEBUGGING.md.
- DELETE showcase/QA-COVERAGE.md — per-demo coverage matrix + starter hero
matrix + probe depth + infra locations + gaps folded into TESTING.md as
the 'Per-Demo Coverage Matrix' section.
Taglines added (no behavioral change to content): TESTING.md, DEBUGGING.md,
GOTCHAS.md, INTEGRATION-CHECKLIST.md, STYLING-GUIDE.md, FRONTEND-STRATEGY.md,
RAILWAY.md, bin/README.md, aimock/README.md, aimock/RAILWAY.md,
harness/README.md, harness/docs/rotation-drill.md.
Cross-link fixups: FRONTEND-STRATEGY.md (was QA-COVERAGE.md →
TESTING.md#per-demo-coverage-matrix), TESTING.md (removed dangling RUNBOOK
companion reference), README.md (rewritten as fanout entry + retained
from-scratch setup + dashboard SOPs below the fanout).
PARITY_NOTES.md × 12 left alone (per-slug context, not redundant).
(cherry picked from commit 75c9d9755c9118c8abc1fa52deda2012b768cab1)
(cherry picked from commit b64189bae0fe2c9e3a5e3ca440013deb4121f23b)
New content:
- TESTING.md: add 10-step cell red→green SOP + bin/showcase test invocation
table (control-plane vs --direct, per-demo scoping matrix); retain
existing CI gating matrix below.
- GOTCHAS.md: add operational gotchas — aimock caches fixtures at container
startup (warm-slot reuse needs docker restart) + --isolate slot collisions
with foreign Docker projects.
- README.md: cross-link to TESTING.md SOP from CLI section; flesh out
--isolate / --direct in test options table; update use cases.
- RUNBOOK.md: update Verifying a Slug's D6 State to use auto-named --isolate;
note A21+A21b per-slug rebuild scoping; rewrite Fixture Matching to teach
picking the backend-id-invariant discriminator (turnIndex post-A12/A13/A20);
modernize Debugging Sequence to --isolate flow.
- DEBUGGING.md: lead with TESTING.md SOP cross-link; update Phase 1 to
--isolate canonical; soften turnIndex-only log-line description; note
aimock startup caching in Phase 5; switch Strategy 5 gold-standard check
to --isolate.
Pruned/updated stale claims (post-A11/A12/A13/A18/A20/A21/A21b):
- RUNBOOK.md "Do not use turnIndex in new fixtures" — turnIndex is now
the canonical backend-id-invariant alternative when toolCallId is fragile
(Anthropic / TanStack Responses API ID rewrites). Replaced with discriminator
selection guidance.
- RUNBOOK.md anti-pattern "NEVER use turnIndex" — replaced with NEVER
anchor on toolCallId strict equality against ID-rewriting backends, and
NEVER use --direct for value-tests.
- RUNBOOK.md bin/showcase test <slug> --d5 (no --isolate) as canonical SOP
— replaced with --isolate canonical, no manual name required.
- README.md --d5 option description claiming "subagents/tool-rendering/agentic-chat"
fixed slate — replaced with "defaults to agentic-chat representative; :demo
qualifier honored post-A18".
- DEBUGGING.md Phase 1 "showcase up aimock <slug> && showcase test <slug> --d5"
as primary — kept as legacy alternative; --isolate is now lead.
- DEBUGGING.md Phase 5 "fixtures baked into Docker image" — clarified that
aimock additionally caches fixtures in memory at startup (volume-mounted
isolated stack still requires docker restart for warm-slot edits).
- DEBUGGING.md Strategy 5 "showcase test langgraph-python --d5" — replaced
with :demo + --isolate so the gold-standard check exercises the same cell.
(cherry picked from commit 0e548455043396972f7fb5b96f8c0ea8abdf1d98)
(cherry picked from commit 592c02d392350d02cc5e17544e663a6605b8da65)
Cross-session review fixes for the --isolate machinery (one concern:
source + test + docs).
1) Reaper reserved-name guard (critical): _reap_isolate_slot trusted
slot records — a record naming 'showcase' (corrupt, or written by an
older CLI version before apply_isolation reserved the name) passes
the charset regex, so the reap ran `docker compose -p showcase down
--remove-orphans --volumes` against the LIVE default stack,
destroying the PocketBase named volume. The reserved name now gets
the same treatment as the path-traversal guard: warn (naming the
record and why it is dangerous) and leave the slot intact for manual
inspection — no compose-down, no state removal.
Call-site enumeration: _reap_isolate_slot's sole caller is
_sweep_isolate_slots, at 3 sites (dead-PID reap, project-recorded/
no-owner reap, age-fallback reap), all passing
"$slot_entry" "$slot_proj" — all three flow through the new guard
identically.
Red-green: the new bats test ("a slot whose project record reads the
RESERVED 'showcase' is left intact...") was run against the UNFIXED
code first and FAILED — the sweep logged "Attempting to reclaim
stale slot 0 (project showcase has no live containers and no
recorded owner)" and reaped the slot. It passes with the guard.
2) .iso-bak restore race: two concurrent runs can both see a stale
backup; the loser's mv is the FINAL command of its `[ -f ] && mv`
AND-list, so its failure trips set -e and kills the CLI pre-claim
with a raw error. Both mv's now carry `2>/dev/null || true` — the
survivor's restore wins, the loser proceeds with restored originals.
3) Keep-test absence regexes greped only the `--project-name <name>
down` spelling; the reaper's own downs use `-p <name> down`, so a
keep-branch regression via the -p form passed undetected. Both keep
absence assertions now match `(--project-name|-p) <name> down`.
Mutation-verified: a temporary -p-form compose-down added to the
keep branch made BOTH broadened tests FAIL; reverted, suite green.
(All other absence assertions use the word-matched generic
`compose ... down` regex, which already covers both spellings.)
4) RUNBOOK.md/DEBUGGING.md contradicted shipped code: the manual
teardown was quoted without --volumes plus notes claiming
`down --remove-orphans` leaves named volumes (the shipped survival
notice and every teardown path include --volumes), and the name rule
was documented as `[a-z0-9_-]+` (actual: starts with [a-z0-9], then
[a-z0-9_-], uppercase normalized with a warn, 'showcase' reserved).
Both updated to the shipped semantics; the now-redundant separate
`down --volumes` snippets removed.
Verification: full `bats showcase/scripts/__tests__/` green (60 tests);
shellcheck on _common.sh shows no new warnings vs baseline
(pre-existing SC2034/SC2115 only, line-shifted).
RUNBOOK.md + DEBUGGING.md establish `bin/showcase test <slug> --d6 --isolate <name>` as the default way to verify a slug's d6 state (isolated stack on offset ports, own compose project, doesn't touch the shared stack), replacing the stale isolated-run sections.
Slice 3 (#5022) renamed the e2e-deep probe driver to d6-all-pills
(also d4-chat-roundtrip + d5-single-pill in the same family). The
two operator-facing docs still referenced the old names. Fix.
No probe behavior change.
The --isolate flag was rewritten in PR #4570 to use temp overlay
files instead of mutating originals in place. Add sections to
DEBUGGING.md (how it works, troubleshooting) and RUNBOOK.md
(operational notes for parallel runs and cleanup).
Strategy 9: pull all D5 status records from PocketBase in one request,
categorize errors, cross-reference with deploy history, and compute
per-service flapping rates to distinguish real bugs from transient
production issues.
Covers: failure triage (text-only vs tool features, probe text extraction,
message disappearing, chatMemory pollution), framework-specific gotchas
(LlamaIndex, Spring AI, Built-in Agent, MS Agent, AG2), and production
vs local parity checklist.
Jordan Ritter