--isolate <name> <slug> (name before slug) mis-parsed the name as the
slug, then died "Unexpected argument"; a bare --isolate= silently
fell through to auto-pick. Defer the ambiguous post---isolate token to
pending_iso_name and resolve it after the parse loop (slug present =>
token was the name; no slug => token was the slug); reject an empty
--isolate= loudly; --isolate=<name> (non-numeric) now binds an explicit
isolate name. The --isolate=<N> numeric pin is unchanged.
A --keep'd isolated stack whose owning process had exited (but whose
containers kept running) was classified 'live' forever and never reaped,
leaking Docker stacks indefinitely. Introduce a start-time-verified
_owner_liveness probe and a new 'kept' state, an ISOLATE_KEEP_TTL (4h,
SHOWCASE_ISOLATE_KEEP_TTL-overridable) that flips an over-age kept slot
to 'stale' so the sweep reclaims it, a com.copilotkit.showcase.isolate
self-id label stamped by apply_isolation, a 'slots --reapable' filter,
and a macOS lsof COMMAND-truncation fix in the own-project port filter.
Real-surface bats cover the liveness false-positive and TTL reaping.
Adds a sugar form of the --isolate flag that pins the isolation slot
directly from the command line:
bin/showcase test agno --d5 --isolate=9
# equivalent to:
SHOWCASE_ISO_SLOT=9 bin/showcase test agno --d5 --isolate
The arg parser splits --isolate=<N> into setting use_isolate=true plus
exporting SHOWCASE_ISO_SLOT=<N>; the existing picker
(_claim_isolate_slot in _common.sh) handles all validation — positive
integer, slot 0 reserved, 1<=N<=ISOLATE_MAX_SLOT, port probe, liveness.
No validation logic is duplicated.
Tests:
- replays the parser branch and verifies SHOWCASE_ISO_SLOT export +
picker pinning
- drives the picker's reserved-slot (N=0) and out-of-range (N=99)
rejections through the arg form to pin the parser->env->picker wiring
- drift guard: sources the REAL cmd-test.sh, stubs apply_isolation, and
snapshots SHOWCASE_ISO_SLOT to catch any future regression of the
parser branch
Help text and TESTING.md updated in a follow-up commit.
The persistent stack's docker-compose.local.yml hardcodes LOCAL_SERVICES_JSON
to the langgraph-python sample for fast N=1 local demos. When --isolate
spawns an iso1 stack with a different slug (e.g. ms-agent-python), the
iso1 harness container inherited that hardcoded value, causing
discovery.railway-services.local-injection to enumerate the wrong service
(showcase-langgraph-python instead of showcase-<requested-slug>). The iso1
probe then targeted the wrong container, broke red-green verification, and
left D5 cells unwritten.
Inject a per-slug LOCAL_SERVICES_JSON override into the iso1 compose
generator so iso1 always probes the slug passed via --isolate.
- ISOLATE_KEEP promoted to a global so --keep survives cmd_test return
into the trap scope
- early-die and default-stack protection: failed --isolate setup no
longer tears down the default stack; half-initialized state is
cleaned up on the way out
- liveness/PID/age reaping signals with a sweep lock: heartbeat
updates, own-pid lock release, tombstones, and a claim-then-verify
duplicate-name guard close slot-registry races (TOCTOU, lock
takeover, reap order)
- teardown robustness: --volumes on every compose down, failed-down
runs preserve state for diagnosis, reap remnants get a compose-down,
path-traversal guard, uniform rm guards under set -e
- name validation: --isolate names must start with a lowercase letter
or digit; reserved name 'showcase' rejected (it aliases the default
stack)
- fail-loud warning before pre-down of an existing stack; help text
updated
Result of an 8-round, 7-agent code-review loop with red-green
verified fixes.
Previously the EXIT-trap restore_isolation always tore down the isolated
stack, ignoring --keep. Now restore_isolation reads a keep flag (set in
cmd-test.sh when --keep is parsed): when kept it skips compose down, the run-dir
removal, and the slot release, and instead prints a survival notice with the
project, slot, the three offset host ports, and the exact manual teardown
command. The kept stack's live containers keep its slot from being reaped.
Make the showcase dev tool faithful to staging by construction. Two changes:
1. `showcase test --d5/--d6` now drives the fleet CONTROL-PLANE (producer ->
probe_jobs queue -> worker -> result-aggregator) instead of the legacy
in-process runLevel() driver. The new cli/control-plane-run.ts replicates
the deep/full producer tick exactly as runControlPlane wires it
(createE2eDeepServiceEnumerator / createServiceEnumerator over
createJobProducer + createFleetQueueClient), enqueues one operator-triggered
tick, and polls local PocketBase for the run's terminal cells. The running
worker fleet claims + runs the driver + the aggregator writes the d5/d6
status cells, so the dev tool exercises the IDENTICAL wiring + concurrency
as staging. The old in-process path stays available behind `--direct`.
2. `showcase up --dev` adds a docker-compose.dev.yml overlay that bind-mounts
each integration's source and overrides the run command with a stack-aware
hot-reload entrypoint (shared/dev/dev-entrypoint.sh: uvicorn --reload for
FastAPI agents, langgraph dev for graphs, next dev for the frontend). Edit a
source file and the component reloads in place with no image rebuild. The
built-image mode remains the faithful/staging-equivalent default.
Add fixture analysis/split/merge scripts for the D6 restructure,
update aimock fixture collision detection tests for the new directory
layout, update GOTCHAS.md and QA-COVERAGE.md with D6 notes, and
add --d6 flag support to the showcase test CLI.
apply_isolation previously mutated docker-compose.local.yml and
local-ports.json in-place with .iso-bak backups. If the process crashed
the originals stayed corrupted with +200 port offsets, breaking all
subsequent showcase commands.
Now writes modified copies to a temp directory and overrides
COMPOSE_FILE/PORTS_FILE shell variables so downstream code reads from
the overlay. Originals are never touched. restore_isolation just removes
the temp dir.
Also replaces hardcoded +200 port offset with atomic mkdir-based slot
allocation. Two parallel --isolate runs now get different port ranges
(slot 0 = +200, slot 1 = +400, etc.) instead of colliding on the same
ports. Container names include the slot number for collision-free Docker
naming. Stale slots from crashed runs are reclaimed via PID liveness
checks and a 2-hour age fallback.
TS harness files (config.ts, lifecycle.ts, doctor.ts) honor
LOCAL_PORTS_FILE env var so they read offset ports from the temp overlay.
When passed to `showcase test`, creates an isolated Docker Compose
project with offset ports (+200) and renamed containers, allowing
multiple agents/sessions to run showcase tests simultaneously without
container conflicts.
Usage:
showcase test agno --d5 --isolate # auto-names isolate-<PID>
showcase test agno --d5 --isolate d5verify # explicit name
Jordan Ritter