roles/gateway.md carries the settled bar: a ping means waiting-on-Joel
and aging, important first-notice breakage, or an answer he asked for.
One incident one thread, one ping per crossing, worker receipts never
ping, runbook cadence is evidence. Ack-first applies to addressed
inbound only; ambient gets observe and zero outbound. Morning digest
beat: idempotent 07:30 LA wake with registry readback in the driver.
The heartbeat fiber logs only on verdict change, so the one moment worth seeing
— it refreshing the key while the work pass sits inside a 300s poke — left no
trace at all. The fix was invisible in production exactly where it matters.
A work pass older than the TTL can only mean it is blocked, so refreshes taken
in that window now carry coveringBlockedPass and how stale the pass had gone.
Still silent the rest of the time; the receipt log is 25MB per five days and
does not need every ordinary tick twice.
The transport reads gateway:agent:heartbeat to answer one question: will a
live agent session pick this message up? The driver was answering a different
one — "is the session idle this instant" — and withholding the key otherwise.
Two failure modes, both measured on the live receipt log 2026-07-27:
1. The pass that wrote the key was the same pass that blocked on promptAgent.
The poke deadline is 300s; the key lives 60s. 51 of the last 200 pokes
outlived the TTL, leaving the gateway invisible 12.6% of the day — precisely
while it was busiest. Fixed by giving the heartbeat its own fiber that takes
its own fresh observation and never blocks on a poke.
2. Every ordinary busy turn read as an outage, so the transport delivered raw
text and bypassed the agent's judgment. Fixed by heartbeatVerdict(), which
withholds only on real evidence: no session, degeneration, a failed poke,
an in-flight retire, or a session that has stopped answering entirely.
The last of those is what keeps this honest. A mid-turn session is vouched for
only within unresponsiveGraceMs (default = poke deadline), so a wedged session
still loses the key and the kill drill still has something to catch. A retired
session drops the key in the same pass that stops it, never on pre-kill
evidence.
Also fixes the restart gap: a driver restarting mid-turn used to score the
running session unresponsive on sight and withhold, which is why restarting the
driver guaranteed a raw-delivery window.
45 driver tests pass (7 new), 252 across gateway + driver.
Six days of the message event log (3978 events, the gateway's whole life)
say the same thing three ways: it talks more and Joel talks less.
Deliveries went 15/day on cutover to 76/day; Joel's own messages went
36/day to 3/day.
Three causes, two of them mine from 2026-07-25.
1. The 4h session retire stranded every open aggregate. A batch lives in
the agent's head, so the successor can never close one it did not
open. Aggregation stopped dead at 07-25 21:00 and never resumed across
11 retires — and aggregation was absorbing 694 of 844 health-check
inputs. shouldRetire now waits for an empty batch set, bounded by
aggregateGraceMs so a wedged batch cannot keep a rotting session alive.
2. The Front reader emitted only channel/message.received (indexing).
front-message-received-notify listens for front/message.received, so
from 07-23 12:13 every inbound email was indexed and none announced.
The reader now emits both, inbound only, with a Redis claim so the
5-minute overlap window cannot ping twice.
3. check-system-health was 844 of 1321 gateway inputs — the loudest
speaker by 6x — because the only suppressor was "does a task exist".
Joel was told about Front Projection 46 times in 40 hours. It now
notifies on transitions of the degraded set, with a 6h reminder for
real outages and one message on recovery.
Also retries Front 200s with unparseable bodies instead of burning an
Inngest attempt on "Unexpected end of JSON input".
11 new tests. Pre-existing 4 failures in system-bus are unrelated and
unchanged (verified against a clean tree).
Session age was tracked as first-sighting in driver memory, so restarting the
driver handed the running session a fresh 4h lease. Caught on the live restart:
a session that had been up since 2026-07-22 reported sessionAgeMs 0.
That is backwards — a driver restart is exactly when a rotting session most
needs retiring. Age now comes from the Claude transcript named by the herdr
agent session id, falling back to first sighting only when it cannot be read.
The live session reads 3.5 days old against a 4h limit.
Two long sessions degenerated into repeated "court" tokens — one reply emitted
it 32,000 times — at ~394k and ~886k context on --model opus --effort low,
after running for days. The brief's own Vocabulary always said a cycle should
"retire at ~150k tokens or a quiet moment." Nothing implemented it: the driver
had no age or size signal at all, so the session ran until the model broke.
Adds two retire paths:
- Clean age retire (default 4h, GATEWAY_MAX_SESSION_AGE_MS, 0 disables) fires
only when the session is healthy, idle, and has no unhandled work. Retiring
with queued work or an outstanding poke would defeat the point.
- Forced retire on degeneration. Repeated-token collapse is poison — once it
appears the model re-samples it — so this one preempts queue state. The
detector is deliberately boring: the known court burst, or one token
dominating a short recent window.
Wall-clock age is the honest signal. A scraped token count would be a fragile
guess about someone else's internals.
This exposed a real gap in the successor spawn: it returned early whenever a
pane already carried the target label, assuming that pane held a live session.
After a retire the pane exists with nothing in it, so nothing would have
relaunched. Spawn now checks for a live agent and relaunches in place, still
never opening a second pane for the same label.
shitratgit[bot]