The summarised pipeline (parse → bucket → cluster) is the right shape when
you trust the bucketing, but sometimes you want full-fidelity log data and
ad-hoc jq exploration. --raw-capture spawns 'log stream --style ndjson' and
dumps lines verbatim to raw.ndjson, bypassing the clustering pipeline.
Garbage collection is automatic and layered so a year of daily use ≈ a day:
- Per-session size cap (--max-size-mb, default 10) — worker stops cleanly on
cap hit, writes size_cap_hit marker, sets extras.truncated.
- Gzip on stop (skippable with --no-gzip) — NDJSON compresses ~15-19x in
practice. Verified live: 0.96 MB raw → 0.05 MB gzipped on Grapla today.
- TTL prune (existing IOS_SIM_HANG_SESSION_TTL_HOURS=24) runs on every
--start. New aggregate cap IOS_SIM_HANG_TOTAL_CAP_MB (default 100) runs
alongside, evicting oldest sessions first when total skill state exceeds.
Verified end-to-end on Grapla: a 12s session captured 737 raw NDJSON lines.
'jq -s group_by(.processImagePath)' shows runningboardd=544, SpringBoard=83,
mediaremoted=39, locationd=39, grapla=28. The grapla-process share (4%) is
exactly why the --bundle-id AND clause in #83 was wrong: it filtered out
96% of useful hang signal originating from system daemons.
Implementation:
- New args: --raw-capture, --max-size-mb, --no-gzip; new env var
IOS_SIM_HANG_TOTAL_CAP_MB. Plumbed through start_args into meta.args.
- run_worker grew a raw-capture branch that reuses the #84 restart/crash
machinery: spawn log stream --style ndjson, feed to _read_stream_into_raw,
which dumps lines verbatim and bails on size cap.
- Non-JSON banners (log stream's 'Filtering the log data ...' first line,
pwuid_r warnings) are skipped so consumers get strict NDJSON for jq.
- _stop_raw_session gzips raw.ndjson → raw.ndjson.gz (via stdlib gzip +
shutil.copyfileobj), removes the uncompressed file, records bytes in
meta.extras. No summary.json for raw sessions.
- _mark_truncated sets extras.truncated when the size cap fires.
- SessionStore.session_total_bytes + prune_to_aggregate_cap: sum sizes
per session, evict oldest first until total ≤ cap.
- --list-sessions surfaces raw size, truncated flag, capture duration.
- --get-details on a raw session points to the file with a zcat | jq hint.
Tests: 7 new (in tests/test_worker_loop.py) — raw line writing, size-cap
trigger, gzip round-trip + content preservation, --no-gzip preserves uncompressed,
aggregate-cap eviction order, session_total_bytes. 131 → 138 passing.
Pre-fix: log stream EOF after 30s of silence broke the read loop, wrote a
stream_ended event, and exited. SessionStore.persist_worker_counters then
overwrote the status back to 'running' — sessions appeared live forever in
--list-sessions even though the worker had been dead for hours. Verified on
Grapla: a 19s session showed counters.total=1, status='running' indefinitely.
Fix split across three concerns:
1. Crash detection (run_worker rewrite):
- Removed the 30s silence bail. A quiet log stream is not a dead one;
bailing on silence was a false-positive death signal.
- EOF on stdin or proc.poll() != None now triggers a stream_died event
with exit_code and attempt number.
- try/finally wrapping ensures mark_crashed runs whenever the worker
exits involuntarily (subprocess died, restart budget exhausted).
- Inner read loop extracted into _read_stream_into_events for clarity
and direct testability.
2. Bounded auto-restart:
- New env var IOS_SIM_HANG_MAX_RESTARTS (default 3). Each restart logs
a stream_restart event with attempt number and a 2s backoff prevents
tight respawn loops.
- counters now tracks stream_restarts; surfaces in summary + --list-sessions.
- SIGTERM during backoff or read loop exits cleanly without restart.
3. Capture-duration tracking (hang_sessions.py):
- mark_crashed now also records stopped_at/stopped_at_ms — so crashed
sessions have a real end-of-capture timestamp, not just a status flip.
- persist_worker_counters preserves CRASHED status (alongside STOPPED)
and no longer clobbers it back to RUNNING.
- build_summary prefers meta.stopped_at_ms over now() so summaries
reflect the capture window, not the time of inspection.
- --list-sessions surfaces 'capture=Xs' and 'restarts=N' columns.
Tests: 8 new (5 in tests/test_worker_loop.py, 3 in tests/test_hang_sessions.py).
Fake subprocess.Popen drives EOF/restart/exhaustion scenarios deterministically.
123 → 131 passing.
Conor Luddy