Add 3 fixtures for the independent LLM calls made by each sub-agent
tool (research_agent, writing_agent, critique_agent) so the demo works
without --proxy-only mode. Interleave them with supervisor fixtures for
readability. Rebuild d5-all.json bundle.
## Summary
Migrates all D5 (e2e-deep) fixtures from the `toolCallId` workaround (PR
#4398) to aimock's native `turnIndex` matching shipped in v1.16.0.
- Removes all `toolCallId` match criteria from 9 fixture files (8 source
+ 1 bundle)
- Adds `turnIndex` values (counts assistant messages in request history)
- Reorders fixtures to natural 0→N reading order
- Response bodies and tool call IDs in responses untouched
## Why
`toolCallId` was a stateless workaround for the broken `sequenceIndex`
global counter. It required agent frameworks to return the exact
`tool_call_id` from the fixture response — if they generated their own
ID, the second-leg fixture never matched. This was the likely cause of
the remaining 10 e2e-deep failures (shared-state: 9, subagents: 8).
`turnIndex` counts `role: "assistant"` messages in the request's message
array. Zero server-side state, concurrent-safe,
fixture-ordering-insensitive, and agnostic to what `tool_call_id` the
agent framework generates.
## Test plan
- [ ] All JSON files parse successfully (validated locally)
- [ ] Merge to main → aimock restarts and fetches updated fixtures
- [ ] Trigger e2e-deep probe, verify improvement (expect shared-state +
subagents to fix)
aimock v1.16.0 ships native turnIndex matching — counts assistant
messages in the request's message array instead of relying on
toolCallId in the last tool message. Stateless, concurrent-safe,
fixture-ordering-insensitive.
Removes all toolCallId match criteria from 9 D5 fixture files (8
source + 1 bundle), adds turnIndex values, and reorders fixtures
to natural 0→N reading order.
The rename PR (#4388) deleted smoke-red-tick.yml from config/alerts/
and test/fixtures/rules/valid/ but left 16 tests referencing it:
- render-red-tick.test.ts: removed smoke-red-tick.yml from YAMLS array
and deleted two smoke-specific tests (endpoint link rendering)
- rule-loader.test.ts: replaced smoke-red-tick.yml fixture references
with agent-red-tick.yml; created matching valid fixture; removed the
smoke-red-tick describe block from full YAML contract coverage
- e2e-deep.test.ts: increased timeout on failure_turn test from 5s to
15s -- fillAndVerifySend retry loop (3 x 2s) exceeds 5s when
stallEvaluate prevents user message count growth
Add missing `commander` dependency to showcase-harness package.json
(imported in cli.ts but never declared). Add index signatures to
SmokeInput and ChatToolsInput interfaces in targets.ts so they satisfy
the `{ [k: string]: unknown }` constraint from the Zod `.passthrough()`
schemas used by the liveness and e2e-chat-tools drivers.
## Summary
- Pre-launches 4 Chromium instances at service boot; probes check
out/return `BrowserContext` handles instead of spawning fresh processes
per tick
- Eliminates `pthread_create: EAGAIN` failures when multiple e2e probes
fire simultaneously (root cause of crewai-crews + langgraph-fastapi
timeouts at aligned cron ticks)
- Recycles browsers after 100 contexts (configurable via
`BROWSER_POOL_RECYCLE_AFTER`) to prevent native memory drift
- Staggers e2e-demos cron from minute 0 → minute 5 to reduce probe
alignment overlap
## Validation
- **Benchmark**: 1.16x speedup, +46MB resident cost, recycling at 55ms
with zero degradation
- **Docker red-green test**: Pool of 4 browsers serving 12 simultaneous
contexts succeeds under the same `ulimit nproc` constraint that causes
12 simultaneous `chromium.launch()` to fail with EAGAIN
- **Test suite**: 1320 tests pass, typecheck + build clean
## Test plan
- [ ] Verify showcase-ops builds and deploys on Railway
- [ ] Monitor next e2e-smoke + e2e-deep overlap tick for absence of
EAGAIN errors
- [ ] Confirm pool stats visible in /metrics endpoint (future: expose
pool.stats())
- [ ] Verify e2e-demos fires at :05 past the hour instead of :00
## Summary
- Adds a local CLI (`showcase/harness/src/cli.ts`) that lets developers
run Dn parity tests against Docker Compose containers without Railway or
CI
- Hardens `docker-compose.local.yml` with YAML anchors, healthchecks on
all 20 services, profiles, and switches aimock to the published GHCR
image
- CLI supports `up`/`down`/`test`/`rebuild`/`ps`/`logs`/`status`
commands with `--smoke`/`--d4`/`--d5` shorthand flags
## Why
Debugging Dn failures currently requires pushing to a branch and waiting
for CI + Railway. This gives Claude and developers a fully local loop:
`showcase test --d5 langgraph-python` spins up the container, runs
probes, and reports results in seconds. Three use cases: (1) Claude
debugging Dn failures by cutting Railway out of the loop, (2) developers
building new columns/cells/rows and testing iteratively, (3) background
automation testing during development.
## What changed
| Commit | Area |
|--------|------|
| `fix(showcase): harden docker-compose.local.yml` |
x-integration-defaults anchor, aimock→GHCR, healthchecks, profiles,
pocketbase/dashboard infra |
| `feat: CLI entry point + config` | Commander CLI, flags, mutual
exclusion, config loader |
| `feat: lifecycle manager` | Profile-aware compose wrapper, ENOENT
detection, caching |
| `feat: target resolution` | Driver input builders, manifest validation
|
| `feat: test runner` | Probe orchestration, Playwright adapter, SIGINT,
PB reuse |
| `feat: results formatter` | Console table output, createPbWriter
factory |
| `chore: format` | oxfmt cleanup |
## Test plan
- [ ] `docker compose -f showcase/docker-compose.local.yml --profile
infra up -d` starts aimock + pocketbase + dashboard with healthchecks
passing
- [ ] `docker compose -f showcase/docker-compose.local.yml --profile
langgraph-python up -d` starts the integration container and depends_on
aimock healthy
- [ ] CLI `--smoke` and `--level smoke` produce equivalent behavior
- [ ] CLI `--smoke --level d4` exits with mutual exclusion error
- [ ] `down` with no args tears down all profiled services
- [ ] `isRunning` correctly reports infra services (which have
`["infra", "all"]` profiles, not per-name profiles)
Resolves CLI slug arguments to probe driver inputs for liveness, D4,
and D5 levels. Validates manifest demos and features arrays. Filters
empty feature lists from deep inputs.
Profile-aware compose wrapper with ENOENT detection for missing Docker.
Handles up/down/rebuild/ps/logs with correct --profile flags for both
targeted slugs and full-stack operations. Caches loadPortsFile result.
stageSharedModules wrapped in try/finally for cleanup on failure.
Commander-based CLI with up/down/test/rebuild/ps/logs/status commands.
Shorthand flags (--smoke, --d4, --d5) with mutual exclusion against
--level. Default test level is smoke. Config loader reads
showcase.local.json for PocketBase and port overrides.
aimock sequenceIndex counter is server-global, causing fixture match
failures when multiple integrations run concurrently. Replace with
stateless toolCallId matching.
Move the isShutdown check before slot mutation in recycleSlot to prevent
a double-close race where shutdown() closes a browser that was already
assigned to the slot and mapped in browserToSlot.
Always call browserPool.shutdown() in the orchestrator stop path
regardless of browserPoolReady flag, so partially-initialized pools
(where init threw after launching some browsers) get cleaned up.
Pre-launch 4 Chromium instances at service boot and check out/return
BrowserContext handles instead of spawning fresh processes per probe.
Eliminates the pthread_create EAGAIN failures when multiple e2e probes
fire simultaneously by capping concurrent Chromium processes at 4.
Pool recycles browsers after 100 contexts (configurable via
BROWSER_POOL_RECYCLE_AFTER) to prevent native memory drift.
Also staggers e2e-demos cron from minute 0 to minute 5 to reduce
probe alignment overlap.
Remove the "Tokyo" city label check from d5-tool-rendering's
validateProbe — some integrations render the weather card with
only temperature and condition, omitting the city name from the
card element's textContent. The card structure (selector matched,
numeric temperature, childCount >= 1) is the real D5 signal.
Reduce d5-mcp-subagents EXPECTED_REPLY_FRAGMENTS from 5 exact
phrases to 2 ("remote work", "talent pool") — some integration
runtimes truncate or rephrase the streamed text so the original
verbose fragments caused false negatives. The D5 signal is chain
completion, not exact phrase fidelity.
Tests updated to match the relaxed assertions.
The monitoring/alerting service is a test harness (probes, assertions,
alerting), not an operations service. Rename the directory, package
name (@copilotkit/showcase-ops → @copilotkit/showcase-harness), all
internal references (Dockerfile, Prometheus metric prefix
showcase_ops_ → showcase_harness_, orchestrator log messages, probe
YAML nameExcludes, test fixtures), and regenerate pnpm-lock.yaml.
Wire protocol names (X-Ops-* headers) and shell-dashboard internal
API naming (OPS_BASE_URL, ops-api.ts) are intentionally unchanged —
they are stable contracts between sender and receiver.
Jordan Ritter