Needed for turnIndex/sequenceIndex fixture matching in D5 multi-turn
conversations. Updated in both packages/runtime (devDep) and
showcase/scripts (dep).
Add demo entries for hitl, hitl-in-app, hitl-in-chat, tool-rendering,
shared-state-read-write, and gen-ui-tool-based across 14 integrations.
Ensure every demo ID also appears in the features list so the showcase
matrix and D5 probes discover them correctly.
STATE_SNAPSHOT can deliver a Preferences object with interests undefined,
crashing .includes(), .filter(), and spread at 4 sites per file. Add
(value.interests ?? []) guards across all 17 integrations.
Add 3 fixtures for the independent LLM calls made by each sub-agent
tool (research_agent, writing_agent, critique_agent) so the demo works
without --proxy-only mode. Interleave them with supervisor fixtures for
readability. Rebuild d5-all.json bundle.
The first parameter of Mastra ToolAction.execute is the raw
Zod-validated input, not a destructured wrapper. Changed all 4
tools from ({ context }, executionContext) to (inputData,
executionContext) and updated field accesses accordingly.
Renders ComposedCell with real CellStatus, DocsRow, DepthChip, and Badge
components (only data layer mocked) to verify that overlay toggles
actually control visible content. Covers all 5 overlays individually,
key combinations (Health+Docs, Health-without-Docs), all 3 presets
(Catalog, Assessment, Parity Review), testing-kind D5/D6 hiding, and
the critical regression test proving Health-only no longer leaks docs
indicators after the CellStatus DocsRow removal fix.
All 17 showcase-starter-* Railway services were stopped and deployments
halted (PR #4390), but the probe discovery still enumerated them via the
namePrefix: "showcase-" filter. This caused probes to scan dead endpoints
and generate false-red alerts.
Add all 17 starter service names to nameExcludes in every probe YAML
that uses railway-services discovery (smoke, e2e-smoke, e2e-deep,
e2e-demos, e2e-parity, image-drift) and to the EXCLUDE_SERVICES set
in aimock-wiring.ts.
Add shared formatTs() utility that converts ISO 8601 UTC strings to
the user's local timezone via toLocaleString(). Apply it to all raw
timestamp interpolation sites in tooltip/badge/drilldown rendering:
- live-status.ts formatTooltip() (4 sites)
- cell-pieces.tsx formatTransitionLine() (3 sites)
- level-strip.tsx resolveBadge() (1 site)
- cell-drilldown.tsx refactored to use shared formatTs()
Update existing live-status tests to expect formatted timestamps.
Add unit tests for formatTs() covering valid/invalid inputs.
CellStatus unconditionally rendered DocsRow for non-testing features,
causing docs indicators (docs-og, docs-shell) to appear in matrix cells
whenever the Health overlay was active — even when the Docs overlay was
toggled OFF. DocsLayer in ComposedCell already handles docs rendering
correctly, gated on overlays.has("docs"). Remove the redundant DocsRow
from CellStatus so docs indicators respect the overlay toggle.
## Summary
Migrates all D5 (e2e-deep) fixtures from the `toolCallId` workaround (PR
#4398) to aimock's native `turnIndex` matching shipped in v1.16.0.
- Removes all `toolCallId` match criteria from 9 fixture files (8 source
+ 1 bundle)
- Adds `turnIndex` values (counts assistant messages in request history)
- Reorders fixtures to natural 0→N reading order
- Response bodies and tool call IDs in responses untouched
## Why
`toolCallId` was a stateless workaround for the broken `sequenceIndex`
global counter. It required agent frameworks to return the exact
`tool_call_id` from the fixture response — if they generated their own
ID, the second-leg fixture never matched. This was the likely cause of
the remaining 10 e2e-deep failures (shared-state: 9, subagents: 8).
`turnIndex` counts `role: "assistant"` messages in the request's message
array. Zero server-side state, concurrent-safe,
fixture-ordering-insensitive, and agnostic to what `tool_call_id` the
agent framework generates.
## Test plan
- [ ] All JSON files parse successfully (validated locally)
- [ ] Merge to main → aimock restarts and fetches updated fixtures
- [ ] Trigger e2e-deep probe, verify improvement (expect shared-state +
subagents to fix)
aimock v1.16.0 ships native turnIndex matching — counts assistant
messages in the request's message array instead of relying on
toolCallId in the last tool message. Stateless, concurrent-safe,
fixture-ordering-insensitive.
Removes all toolCallId match criteria from 9 D5 fixture files (8
source + 1 bundle), adds turnIndex values, and reorders fixtures
to natural 0→N reading order.
The rename PR (#4388) deleted smoke-red-tick.yml from config/alerts/
and test/fixtures/rules/valid/ but left 16 tests referencing it:
- render-red-tick.test.ts: removed smoke-red-tick.yml from YAMLS array
and deleted two smoke-specific tests (endpoint link rendering)
- rule-loader.test.ts: replaced smoke-red-tick.yml fixture references
with agent-red-tick.yml; created matching valid fixture; removed the
smoke-red-tick describe block from full YAML contract coverage
- e2e-deep.test.ts: increased timeout on failure_turn test from 5s to
15s -- fillAndVerifySend retry loop (3 x 2s) exceeds 5s when
stallEvaluate prevents user message count growth
Add missing `commander` dependency to showcase-harness package.json
(imported in cli.ts but never declared). Add index signatures to
SmokeInput and ChatToolsInput interfaces in targets.ts so they satisfy
the `{ [k: string]: unknown }` constraint from the Zod `.passthrough()`
schemas used by the liveness and e2e-chat-tools drivers.
Remove conditional early return that rendered different DOM structures
on server vs client (bare fragment vs div>button wrapper). All hooks
now run unconditionally; interaction is disabled via guards when items
array is empty. Fixes React errors #418 and #310.
## Summary
- Pre-launches 4 Chromium instances at service boot; probes check
out/return `BrowserContext` handles instead of spawning fresh processes
per tick
- Eliminates `pthread_create: EAGAIN` failures when multiple e2e probes
fire simultaneously (root cause of crewai-crews + langgraph-fastapi
timeouts at aligned cron ticks)
- Recycles browsers after 100 contexts (configurable via
`BROWSER_POOL_RECYCLE_AFTER`) to prevent native memory drift
- Staggers e2e-demos cron from minute 0 → minute 5 to reduce probe
alignment overlap
## Validation
- **Benchmark**: 1.16x speedup, +46MB resident cost, recycling at 55ms
with zero degradation
- **Docker red-green test**: Pool of 4 browsers serving 12 simultaneous
contexts succeeds under the same `ulimit nproc` constraint that causes
12 simultaneous `chromium.launch()` to fail with EAGAIN
- **Test suite**: 1320 tests pass, typecheck + build clean
## Test plan
- [ ] Verify showcase-ops builds and deploys on Railway
- [ ] Monitor next e2e-smoke + e2e-deep overlap tick for absence of
EAGAIN errors
- [ ] Confirm pool stats visible in /metrics endpoint (future: expose
pool.stats())
- [ ] Verify e2e-demos fires at :05 past the hour instead of :00
## Summary
- Adds a local CLI (`showcase/harness/src/cli.ts`) that lets developers
run Dn parity tests against Docker Compose containers without Railway or
CI
- Hardens `docker-compose.local.yml` with YAML anchors, healthchecks on
all 20 services, profiles, and switches aimock to the published GHCR
image
- CLI supports `up`/`down`/`test`/`rebuild`/`ps`/`logs`/`status`
commands with `--smoke`/`--d4`/`--d5` shorthand flags
## Why
Debugging Dn failures currently requires pushing to a branch and waiting
for CI + Railway. This gives Claude and developers a fully local loop:
`showcase test --d5 langgraph-python` spins up the container, runs
probes, and reports results in seconds. Three use cases: (1) Claude
debugging Dn failures by cutting Railway out of the loop, (2) developers
building new columns/cells/rows and testing iteratively, (3) background
automation testing during development.
## What changed
| Commit | Area |
|--------|------|
| `fix(showcase): harden docker-compose.local.yml` |
x-integration-defaults anchor, aimock→GHCR, healthchecks, profiles,
pocketbase/dashboard infra |
| `feat: CLI entry point + config` | Commander CLI, flags, mutual
exclusion, config loader |
| `feat: lifecycle manager` | Profile-aware compose wrapper, ENOENT
detection, caching |
| `feat: target resolution` | Driver input builders, manifest validation
|
| `feat: test runner` | Probe orchestration, Playwright adapter, SIGINT,
PB reuse |
| `feat: results formatter` | Console table output, createPbWriter
factory |
| `chore: format` | oxfmt cleanup |
## Test plan
- [ ] `docker compose -f showcase/docker-compose.local.yml --profile
infra up -d` starts aimock + pocketbase + dashboard with healthchecks
passing
- [ ] `docker compose -f showcase/docker-compose.local.yml --profile
langgraph-python up -d` starts the integration container and depends_on
aimock healthy
- [ ] CLI `--smoke` and `--level smoke` produce equivalent behavior
- [ ] CLI `--smoke --level d4` exits with mutual exclusion error
- [ ] `down` with no args tears down all profiled services
- [ ] `isRunning` correctly reports infra services (which have
`["infra", "all"]` profiles, not per-name profiles)
## Summary
- Hovering/clicking a tally count (✓ N, ~ N, × N) in column headers now
shows a popover listing which features/signals fall in that bucket
- `computeColumnTallyDetail()` returns per-bucket feature lists (health
+ e2e dimensions) alongside the existing aggregate counts
- `TallyTrigger` wrapper handles hover-to-preview (200ms delay) +
click-to-pin semantics with click-outside-to-close
- 16 new unit tests (6 for the data function, 10 for the popover
component)
## Test plan
- [ ] Verify tally breakdown popover appears on hover/click of ✓/~/×
counts
- [ ] Verify popover lists correct feature names per bucket
- [ ] Verify click-outside closes the popover
- [ ] Verify 0-count items show no interaction
- [ ] Run `npx vitest run src/components/__tests__/` — 110 tests pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Document the parity-testing CLI (commands, test levels, options, config,
architecture diagram, use cases). Reorder so the CLI appears before the
lower-level docker-compose workflow it wraps.
TallyTrigger wraps each tally count (green/amber/red) in OverlayColumnHeader
with hover-open (200ms delay) and click-pin semantics. The popover lists
which features/signals fall in each bucket with colored dots and dimension tags.
Add companion function to computeColumnTally() that returns per-bucket
TallyItem arrays instead of numeric counts. Wire a tallyDetails memo
into FeatureGrid and pass tallyDetail prop to OverlayColumnHeader.
Includes 6 unit tests covering connection error, health routing,
e2e tone bucketing, and demo-less feature filtering.
Resolves CLI slug arguments to probe driver inputs for liveness, D4,
and D5 levels. Validates manifest demos and features arrays. Filters
empty feature lists from deep inputs.
Profile-aware compose wrapper with ENOENT detection for missing Docker.
Handles up/down/rebuild/ps/logs with correct --profile flags for both
targeted slugs and full-stack operations. Caches loadPortsFile result.
stageSharedModules wrapped in try/finally for cleanup on failure.
Commander-based CLI with up/down/test/rebuild/ps/logs/status commands.
Shorthand flags (--smoke, --d4, --d5) with mutual exclusion against
--level. Default test level is smoke. Config loader reads
showcase.local.json for PocketBase and port overrides.
Add x-integration-defaults YAML anchor to DRY 17 service definitions.
Switch aimock from local build to ghcr.io/copilotkit/aimock:latest.
Add healthchecks to all 20 services (internal ports, not host-mapped).
Add pocketbase and dashboard infra services with proper depends_on
conditions. Add profiles to every service so compose enumeration works.
aimock sequenceIndex counter is server-global, causing fixture match
failures when multiple integrations run concurrently. Replace with
stateless toolCallId matching.
Move the isShutdown check before slot mutation in recycleSlot to prevent
a double-close race where shutdown() closes a browser that was already
assigned to the slot and mapped in browserToSlot.
Always call browserPool.shutdown() in the orchestrator stop path
regardless of browserPoolReady flag, so partially-initialized pools
(where init threw after launching some browsers) get cleaned up.
Jordan Ritter