First outage message immediate, later inputs join a rolling 10-min
batch summary (sources + counts, urgent/critical non-private subjects
only). Front/email/slack NEVER contribute raw text on fallback.
Recovery cancels the pending summary and leaves canonical inputs for
the gateway. Per-input fallback.delivered receipts preserved. 07-27
burst replay: 33 raw sends become 6 summaries, zero private bodies.
212 gateway tests re-run green by steering.
gateway/external-delivery-canary: host-only Inngest function, weekly
Tue 09:17 PT, GATEWAY_EXTERNAL_CANARY_MODE=off by default so
registration is inert. Proves both paths (immediate delivery.confirmed
within 10min SLO; quiet aggregate close-deliver to one visible digest),
snapshots pending-input age + aggregate deadlines from the canonical
log, 0600 JSONL receipts + OTEL pass/fail, failure alarm via the
mechanical pane scheduler (no gateway judgment, no listeners, never
restarts). First live run is a separate approved action. Tests re-run
green by steering.
New @joelclaw/gateway-incident-latch reconstructs (source, anomalyId)
incident state from canonical gateway.decision.recorded receipts; Redis
out of the truth path. Stream tool rewrites incident-tagged decisions:
opens deliver + aggregate, identical repeats join, one material change
per PT day, resolution close-delivers once, reopens link follows.
Producer contract documented in docs/gateway.md + messaging skill.
Audited pulse replay: 15 DMs (12 transitions + 3 digests) from 42
inputs. 45 tests re-run green by steering; companion joelclaw-api
boundary change committed separately.
The checker diagnoses, writes an idempotent operator-action receipt,
emits gateway.health.operator_action.requested via the OTEL outbox, and
pushes gateway.health.degraded with automaticRestart:false. 72h audit
showed 2 auto-restarts, both errored, 0 successes. Only remaining
restart callers are operator-invoked (cli gateway.ts --restart flag,
cutover-rollback script). Tests re-run green by steering.
check-system-health now latches (component, anomalyId) incidents in
Redis and DMs only open/material-change/resolved transitions; every
observation still lands in OTEL and the daily aggregate; one digest at
00:05 PT; all-green silent. Replay of the audited 285-check window:
10 DMs where 72 were sent. New systemHealthDailyDigest function
registered — NOT live until the system-bus host reloads (registration
is not routing). 42 targeted tests re-run green by steering; worker
reports 794/794 package suite.
Flow projection gains fallback.delivered and aggregate-closure
terminals; notify wait resolves the flow from the event log with the
Redis correlation key as cache only; producer correlation preserved
through decision execution. Fixes false deliveryProven:false on
confirmed deliveries (daily-flow receipts 07-25..07-29). 31 targeted
tests green, verified by steering. Step + ledger recorded in
aihero-support .brain/projects/telegram-quiet.
roles/gateway.md carries the settled bar: a ping means waiting-on-Joel
and aging, important first-notice breakage, or an answer he asked for.
One incident one thread, one ping per crossing, worker receipts never
ping, runbook cadence is evidence. Ack-first applies to addressed
inbound only; ambient gets observe and zero outbound. Morning digest
beat: idempotent 07:30 LA wake with registry readback in the driver.
Transport stamps addressed/ambient on inbound.received at ingress: reply
to a gateway flow, @mention, or DM is addressed; Slack channel messages
without a mention are ambient. The plugin refuses deliver/close-deliver
on ambient without a prior escalate receipt, requires observe/escalate
as the first ambient decision, and scopes ack-first to addressed only.
Joel told the gateway 'my slack messages are not commands unless noted'
on 2026-07-27 and prose lost; this is the tooth.
Daily 08:00 cron counts delivery.requested + fallback.delivered over 24h.
At 24/day (2x the 12/day settled target) it pages once with top sources;
a Redis NX latch resets only after a full day back at target. Stable
notification eventId keeps Inngest retries from double-paging.
The heartbeat fiber logs only on verdict change, so the one moment worth seeing
— it refreshing the key while the work pass sits inside a 300s poke — left no
trace at all. The fix was invisible in production exactly where it matters.
A work pass older than the TTL can only mean it is blocked, so refreshes taken
in that window now carry coveringBlockedPass and how stale the pass had gone.
Still silent the rest of the time; the receipt log is 25MB per five days and
does not need every ordinary tick twice.
The transport reads gateway:agent:heartbeat to answer one question: will a
live agent session pick this message up? The driver was answering a different
one — "is the session idle this instant" — and withholding the key otherwise.
Two failure modes, both measured on the live receipt log 2026-07-27:
1. The pass that wrote the key was the same pass that blocked on promptAgent.
The poke deadline is 300s; the key lives 60s. 51 of the last 200 pokes
outlived the TTL, leaving the gateway invisible 12.6% of the day — precisely
while it was busiest. Fixed by giving the heartbeat its own fiber that takes
its own fresh observation and never blocks on a poke.
2. Every ordinary busy turn read as an outage, so the transport delivered raw
text and bypassed the agent's judgment. Fixed by heartbeatVerdict(), which
withholds only on real evidence: no session, degeneration, a failed poke,
an in-flight retire, or a session that has stopped answering entirely.
The last of those is what keeps this honest. A mid-turn session is vouched for
only within unresponsiveGraceMs (default = poke deadline), so a wedged session
still loses the key and the kill drill still has something to catch. A retired
session drops the key in the same pass that stops it, never on pre-kill
evidence.
Also fixes the restart gap: a driver restarting mid-turn used to score the
running session unresponsive on sight and withhold, which is why restarting the
driver guaranteed a raw-delivery window.
45 driver tests pass (7 new), 252 across gateway + driver.
Restoring the webhook's notify seam restored its behavior too: every
inbound email announced individually. Seven landed on Joel's phone inside
a minute, ads included, while aggregation was still cold.
check/email-triage already batches inbound mail into one "N emails need
you" digest, so per-message announcement was always the redundant half.
The reader now announces inbound VIP senders only. Everything else is
still indexed and still reaches Joel through the digest; it just does not
ring the phone.
Six days of the message event log (3978 events, the gateway's whole life)
say the same thing three ways: it talks more and Joel talks less.
Deliveries went 15/day on cutover to 76/day; Joel's own messages went
36/day to 3/day.
Three causes, two of them mine from 2026-07-25.
1. The 4h session retire stranded every open aggregate. A batch lives in
the agent's head, so the successor can never close one it did not
open. Aggregation stopped dead at 07-25 21:00 and never resumed across
11 retires — and aggregation was absorbing 694 of 844 health-check
inputs. shouldRetire now waits for an empty batch set, bounded by
aggregateGraceMs so a wedged batch cannot keep a rotting session alive.
2. The Front reader emitted only channel/message.received (indexing).
front-message-received-notify listens for front/message.received, so
from 07-23 12:13 every inbound email was indexed and none announced.
The reader now emits both, inbound only, with a Redis claim so the
5-minute overlap window cannot ping twice.
3. check-system-health was 844 of 1321 gateway inputs — the loudest
speaker by 6x — because the only suppressor was "does a task exist".
Joel was told about Front Projection 46 times in 40 hours. It now
notifies on transitions of the degraded set, with a 6h reminder for
real outages and one message on recovery.
Also retries Front 200s with unparseable bodies instead of burning an
Inngest attempt on "Unexpected end of JSON input".
11 new tests. Pre-existing 4 failures in system-bus are unrelated and
unchanged (verified against a clean tree).
The beat bypass reused a lane by matching its pane label. It never matched.
The `herdr-name-sync` pi extension renames a worker's pane to the pi session
name on every turn, so the "⏰ Campaign Pulse" label this code wrote became
"📈 Campaign Pulse" (and later "📡 Campaign Pulse") within one turn. Every
firing then failed to find its own lane and opened a new pane: ten Campaign
Pulse panes in ten tabs in [jc] scheduled beats, hourly, exactly the sprawl the
bypass exists to prevent.
A lane is now keyed by its brief path — the schedule names it and never changes
it — and the pane id is persisted in a Redis hash. Matching goes registry
first, label only as a fallback for lanes created before the registry existed.
Lesson worth keeping: a display label is not an identity. Anything cosmetic can
be rewritten by another process, and here one deliberately does.
Also makes SpawnBeatPorts.runCommand optional with a real default, so callers
can supply only the registry ports.
content-sync declared both a cron trigger and a debounce. In Inngest v1.28.0 an
accepted debounce makes the scheduler retry the cron item, so it looped — 124
scheduling errors today, ramping from 8/hr to 54/hr. Split the concerns: a
cron-only scheduling function emits content/seed.requested, and the debounced
worker stays event-driven. Burst collapse is preserved.
nas-backup shelled out to kubectl, which is not installed on flagg and never
was. It failed 174 times while its runs reported COMPLETED. The last real Redis
backup is 2026-05-16 — dump-20260516.rdb, 115,761 bytes, REDIS0012 header. Two
months of green checkmarks over nothing. It now fails loudly through the shared
incident latch and resolves only on a confirmed backup.
The transport is NOT settled: panda has exhausted its ephemeral ports (16,008
TIME_WAIT of a 16,384 range, 9,753 to its own k8s API), so nothing there can
open an outbound TCP socket, including kubectl. The ssh path stays in-tree but
is not deployable until that is fixed. An in-cluster CronJob is the proposed
durable answer.
The reader shipped against GET /events. Our token cannot read it, so every
scheduled run failed and the watermark never advanced — Front stayed exactly as
stale as it was with dead webhooks, just louder in the worker log:
front events 403: Missing required scopes: [events:*:read]
Probing the live token (status codes only, never the value): /events 403,
/conversations 200, /inboxes 200, /me 200. So the key is fine and the endpoint
choice was wrong. vip-email-backfill.ts already reads Front the working way.
Rebuilt on GET /conversations + GET /conversations/{id}/messages, which need
nothing new from Joel. The event contract, watermark, overlap, bounded lookback
and pagination, backoff, and per-run telemetry are unchanged, and timestamp is
still Front's real created_at so overlapping polls keep producing stable
document ids. The dead /events path is deleted rather than left beside it.
A 403 now fails loudly naming the endpoint and the missing scope. This one was
invisible for an hour because a scope error looked exactly like "no new
messages" — the same way a dead webhook looks like a quiet inbox.
The worker-role uniqueness test has been red on main, flagging
webhook-subscription-dispatch-generic as a duplicate. It is not an accident:
commit 2810c4d6 registered it in both roles on purpose, because "the Front
incident proved cluster-only registration never runs on flagg."
Deleting the test would give up catching the accidental duplicates it exists
for, so the intentional ones are allowlisted by name instead, and a second test
asserts every allowlisted id really is registered in both roles — a stale
entry is how the next accident would hide.
packages/system-bus now runs green (768 pass), so a real regression is visible
instead of being buried under known-red tests.
getStoryDiff runs `git diff main...HEAD`. When the project is ON main that
compares a branch to itself: exit 0, empty output. The try succeeded, so the
HEAD~1 fallback never ran and the caller got "".
That matters because the reviewer and judge consume this diff as evidence. An
empty string reads identically as "this story changed nothing" and "I could not
read what changed" — so a story could pass review on no evidence at all.
Empty output now falls through to the single-commit diff instead of being
returned as a result. The no-main-ref case still falls through as before.
Found via three long-red tests in packages/system-bus, which turn out to have
been reporting a real bug rather than being broken themselves.
Joel, via the gateway on Telegram 2026-07-25T21:31Z: "we don't need 5min email
crons, every hour or two is more than enough."
Overlap stays at 5 minutes — that is the re-read window for clock skew and late
indexing, independent of poll cadence, and it stays conservative under hourly
runs. Authored by a gateway-dispatched worker; steering-reviewed.
Note: the Front freshness threshold is 60m, so an hourly poll sits right at it.
Webhooks remain the fast path; if they stay dead the threshold wants raising
rather than the cadence dropping back.
Front's inbound projection died at 2026-07-21T18:44Z and stayed dead for ~98
hours. Nothing downstream was broken — front-notify, channel-message-ingest,
and the Typesense projection all still worked. The webhooks simply stopped
arriving. A webhook is a delivery promise somebody else makes; a reader is ours.
Adds a cron reader (*/5) that pulls Front with FRONT_API_TOKEN and emits the
same `channel/message.received` event the webhook path already emits, so
classification, thread aggregation, gateway context, and the health freshness
probe all keep working unchanged. A durable Redis watermark tracks progress with
5 minutes of deliberate overlap — re-reading slightly is free, missing a message
is not — bounded to a 7-day lookback and 15 pages per run so one pass cannot
walk all of Front. Retryable statuses back off rather than hammering.
The webhook stays as the fast path. Two sources feeding one seam is the point.
That required fixing idempotency, which was only theoretical: channel-message-
ingest derives its document id by hashing the message fields including
`timestamp`, and front-notify passed `Date.now()` — receipt time, not message
time. The same Front message ingested twice produced two different documents.
Harmless when a webhook fires once; ruinous for a poller that overlaps on
purpose. Both producers now pass Front's real message created_at, so a given
message has a stable id no matter which path delivers it or how often.
Pre-existing and NOT introduced here: `webhook-subscription-dispatch-generic` is
registered in both index.host.ts and index.cluster.ts, so the worker-role id
uniqueness test is already red on main.
Session age was tracked as first-sighting in driver memory, so restarting the
driver handed the running session a fresh 4h lease. Caught on the live restart:
a session that had been up since 2026-07-22 reported sessionAgeMs 0.
That is backwards — a driver restart is exactly when a rotting session most
needs retiring. Age now comes from the Claude transcript named by the herdr
agent session id, falling back to first sighting only when it cannot be read.
The live session reads 3.5 days old against a 4h limit.
185 fallbacks since cutover, 77 on 2026-07-25 alone, while the Claude gateway
process had been up since Wednesday. 161 of the 185 sources went on to receive
a real gateway decision anyway — so Joel got the raw dump first for nothing.
The gate was `EXISTS gateway:agent:heartbeat`: a key-presence check, not
liveness. A key missing for one poll during a normal session cycle read
identically to a dead loop. Worse, every receipt recorded heartbeatStaleForMs
as the configured TTL constant rather than measured staleness, so nobody could
tell a blip from an outage after the fact.
Now the receipt carries real measured staleness, a momentary gap has to
actually persist before the loop is declared down, and routine machine noise
falls back as one line and a pointer instead of a wall of health markdown.
Repeated identical fallbacks coalesce.
The safety property is unchanged and deliberately so: fallback still never
depends on the agent being healthy enough to report its own health. That
circularity is exactly what the kill drill exists to catch.
Two long sessions degenerated into repeated "court" tokens — one reply emitted
it 32,000 times — at ~394k and ~886k context on --model opus --effort low,
after running for days. The brief's own Vocabulary always said a cycle should
"retire at ~150k tokens or a quiet moment." Nothing implemented it: the driver
had no age or size signal at all, so the session ran until the model broke.
Adds two retire paths:
- Clean age retire (default 4h, GATEWAY_MAX_SESSION_AGE_MS, 0 disables) fires
only when the session is healthy, idle, and has no unhandled work. Retiring
with queued work or an outstanding poke would defeat the point.
- Forced retire on degeneration. Repeated-token collapse is poison — once it
appears the model re-samples it — so this one preempts queue state. The
detector is deliberately boring: the known court burst, or one token
dominating a short recent window.
Wall-clock age is the honest signal. A scraped token count would be a fragile
guess about someone else's internals.
This exposed a real gap in the successor spawn: it returned early whenever a
pane already carried the target label, assuming that pane held a live session.
After a retire the pane exists with nothing in it, so nothing would have
relaunched. Spawn now checks for a live agent and relaunches in place, still
never opening a second pane for the same label.
`joelclaw messages audit` has failed fleet-wide since cutover with
MESSAGE_JOURNAL_CONFIG_MISSING. The diagnosis "provision the reader
credentials" was wrong: the legacy ClickHouse journal has no listener on flagg
at all, and the events the command wanted were sitting in Convex the whole
time — the same stream `messages trace` already reads successfully.
Audit now falls back to the canonical stream when the legacy journal is
unconfigured, and labels which source answered. A journal failure that is not
a missing-credential problem still surfaces as an error rather than being
quietly papered over.
Verified live: audit --since 6h returns 100 events with source "convex".
Carries three gateway-worker changes that were reviewed and left uncommitted:
- Front Projection drops out of CRITICAL_COMPONENTS. A stale inbox projection
is worth knowing about and is not worth forcing overall health critical and
paging Joel. Health alerts were the single largest producer in the message
stream (740 of 1083 machine messages) and the largest fallback source.
- serve.ts runs the function-health registration check at startup, so the
assertion still runs when the durable function's own registration was lost
while Inngest was unavailable.
- replicate-critical-db.sh freezes one source generation before hashing it.
The builder renames critical.db atomically, so reading the live path twice
could pair the old file's checksum with the new file's built_at.
Also lands the three investigation briefs those findings came from.
mlx_whisper always writes a transcript file, and without --output-dir it
writes to the worker's cwd. The system-bus worker runs from packages/system-bus,
so every Telegram voice note left the contents of Joel's message there as a
UUID-named .txt. One was sitting untracked in the working tree.
Writes the side-effect file beside the audio it came from instead. stdout is
still what the function returns.
A pane.schedule.due event with verb:spawn already names its own brief, its
own prompt, and its own cadence. It was reaching the comms gateway as
evidence anyway, and the gateway re-executed it by dispatching a worker —
copying the brief path into a worker file and opening a tab. Every hour,
forever. That is a mechanical spawn wearing an LLM costume, and it produced
most of the pane sprawl.
verb:spawn now executes into a stable herdr beat lane, keyed by the brief's
title, reusing an idle or done pane and refusing to stack a second one while
the lane is working. verb:wake and verb:revive keep the gateway due-signal
path untouched. The beat still reports its result through notify, so the
gateway keeps deciding whether Joel hears about it — it just stops being the
thing that runs it.
Beat lanes never land in the gateway's own workspace. That workspace is
resolved from the pane hosting the gateway loop rather than a hardcoded
herdr id, so the guard survives a herdr restart.
Authored by a gateway-dispatched worker; steering-reviewed.
Five close-deliver decisions on cutover day were recorded and never sent.
Four of them carried their text on decision.rewrite while the executor only
read payload.rewrite, so Joel simply never received those messages; the
transport log shows the skips.
The executor now reads either location, preferring the top level. The plugin
validator still requires the top-level field, so new receipts stay canonical
while old and off-spec ones still reach Joel — a message he should have
received outranks schema purity about where the agent put it.
The executor had no test file at all. Adds one covering both rewrite
locations, precedence, the no-text skip, and non-delivery passthrough.
New host function mega-journey-watch: on github/workflow_run.completed
for MEGA-DOT-DEV/mega-dev pushes to main, run one tick of the repo's
journey-watch.sh (post-merge Playwright buy-flow proof on flagg). The
script owns sha dedupe, single-flight, retry-once flake law, evidence
preservation, and joelclaw notification; a 30-minute LaunchAgent poll
stays as the delivery backstop.
Registration success proves nothing: on 2026-07-24 a PUT returned
modified:false while three crons were completely dead (upstream v1.27.0 dropped
Function.EnvID and minted unrunnable queue items). Verify runs, not registration.
Startup + hourly check: every serve.ts function must exist in Inngest (bounded
PUT retries with backoff and readback if not), and every cron function must have
a run newer than 2x its real cadence — parsed from the actual expression, so
offsets like the ai-hero reconciler 17 * * * * are handled, not assumed :00.
Alerts name the function, its cron, and its last successful run, through the
shared incident latch with resolve-on-recovery.
Two chain deaths this week, two causes:
- 2026-07-21 campaign-pulse: the spawned run died on a WebSocket error before
arming its heir. Fire-and-forget arming left no trace. `joelclaw wake
assert-successor` now re-reads the registry, requires exactly one future
successor, and alerts by chain name when verification fails.
- 2026-07-22 daily-flow (schedule 853efe38): the dispatcher logged "attempt 1 of
3" and dropped the schedule. Pending schedules now survive failures 1-2 for the
reconciler to re-emit; only exhaustion clears and alerts, exactly once.
Application webhooks wrap the same Front Event object as Rules webhooks,
so the message resource lives at payload.target.data — the parser was
reading the event wrapper and producing empty messages, freezing Front
Projection ingestion since 2026-07-21.
Also hardens text extraction (text -> blurb -> stripped HTML body ->
attachment placeholder) and quarantines events with no usable text via
front/message.quarantined instead of emitting blanks.
Registers message/event-consumer on the host worker (event + cron triggers),
adds the message/event.consume.requested event type, a live proof script, and
the Brain work receipt. Convex deploy of the backing schema is still blocked;
step status set to blocked with receipts.
shitratgit[bot]