mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
flow-visual-settle
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 filip131311
|
7f85f996aa |
fix(tool-server): defer to an existing server on EADDRINUSE instead of crash-looping (#533)
## Summary Fixes **bucket B** of the 0.16.0 crash telemetry — the `toolserver_bind` / `EADDRINUSE` restart loop. It's the largest crash **volume** (~817 events, one machine alone = 785) even though it's a narrow user set (10). Companion to #531 (which fixed the widespread multi-user bucket A). ## Root cause (reproduced live) The main HTTP listener's `server.on("error")` handler routed **every** bind error through `crashShutdown` (exit 1). When a second tool-server starts on an already-owned port, that exit just feeds a supervisor respawn → re-bind → crash, forever. The telemetry signature was puzzling: all these crashes are `crash_phase=serving` with ~1s uptime and **0 tool calls** — i.e. the listener *bound successfully* and then died. I reproduced it: on macOS, a **second tool-server process's** `listen()` momentarily **succeeds** (the `listening` callback runs → `listening=true` → phase `serving`), and **then** the server emits an asynchronous `EADDRINUSE`. So the redundant instance binds, immediately errors, crashes, and the supervisor loops it (~1s each, 785× on one machine). Live A/B on the compiled `dist` (isolated port, telemetry disabled), server A up then a second server B on the same port: ``` buggy (current 0.16.0): B: "Tools server listening ..." → "Failed to bind ... EADDRINUSE" → exit 1 (→ respawn loop) fixed: B: "Tools server listening ..." → "deferring to it (redundant instance)" → exit 0 ``` A survives in both. ## Fix Handle `EADDRINUSE` specifically instead of crash-looping: - Probe the port for a healthy argent tool-server (`GET /tools` → 200, or 401 when a token is configured). - **If a healthy peer answers**, this instance is redundant → shut down **cleanly** (`shutdown(0)`) and defer to it. This removes it from the `reason:"crash"` restart-loop population entirely. - **If nothing healthy owns the port** (a foreign holder, or a wedged peer that doesn't answer within the probe timeout) → still crash, so a genuinely stuck port stays visible. - `EACCES` and all other server errors crash exactly as before. It also corrects the now-false comment claiming these crashes are phased `startup` ("the listen callback never ran"). Telemetry and the live repro prove the bind succeeds first, so `listening=true` and the phase is genuinely `serving`. ## Testing - **New unit tests** for `probeArgentToolServer`: peer answering 200 or 401 → `true`; foreign 500, connection-refused, and wedged/timeout → `false`. - Full tool-server suite **2731 green**; `tsc` + `eslint` clean. - **Live A/B** above, on the built dist. ## Note on scope This is a robustness fix that breaks the loop for the redundant-instance case (the dominant one). The precise reason macOS lets a second process's bind succeed-then-error asynchronously (vs. failing immediately, as same-process contention does) is a kernel/loopback timing detail; the fix does not depend on it — any `EADDRINUSE`, immediate or deferred, now defers-or-crashes deterministically rather than hot-looping. |
||
|
filip131311
|
2196cc18f0 |
feat(telemetry): anonymous crash diagnostics on toolserver:stop (#494)
## Why
PostHog shows **~9,785 `toolserver:stop` crashes over 90 days**, but
every one collapses into a single bucket — `error_kind=crash`,
`error_code=TOOLSERVER_UNCAUGHT_EXCEPTION`. We know *that* the tool
server died on a top-level throw, never *which* throw. The population is
also two very different things we can't tell apart today:
- **~99%** crash at **<5s uptime with 0 tool calls** — startup faults,
and heavily concentrated per-user (top 10 users ≈ 31% of all crashes;
one user logged 1,027), i.e. **boot → crash → relaunch loops**.
- **~1,000** crash after **>10 min** of real serving — genuine runtime
bugs.
Without the actual exception, all 9,785 rows are unrankable and
unfixable.
## What
Four **crash-only, provably-anonymous** fields on `toolserver:stop`,
enforced by the existing coded-allowlist sanitizer so **no free text can
ever reach PostHog**:
| field | value | why |
|---|---|---|
| `error_name` | error class name (`TypeError`, …) | "what kind" |
| `error_syscall` | Node's `err.code` (`EADDRINUSE`, `ECONNREFUSED`, …)
| the key discriminator for the startup-loop population (port/socket
conflicts) |
| `crash_fingerprint` | 16-hex hash of the **de-identified** top stack
frames | clusters the same crash across users; input carries no
identity, output is opaque |
| `crash_phase` | `"startup"` \| `"serving"` | splits the two
populations directly |
Turns 9,785 opaque rows into a handful of rankable clusters.
## Anonymity (the hard requirement)
- **No error message, no raw stack.** Those routinely embed
`/Users/<name>/…`, URLs, argv, hostnames, interpolated values — the
highest-PII surface and impossible to reliably scrub. The fingerprint
replaces the message for clustering; `error_name` + `error_syscall` give
the "what kind" without free text.
- **Fingerprint input is de-identified before hashing**
(defense-in-depth on top of the hash): first-party frames → basename
only (drops the home-dir/project prefix), `node_modules` frames →
package-relative tail (drops the user prefix; also collapses the pnpm
`.pnpm/pkg@ver/` layout), `node:` internals kept verbatim, column
dropped. So the *same crash on two machines produces the same
fingerprint*, and no username/path survives even into the hash input.
- **Sanitizer is the final gate.** `error_name`
`^[A-Za-z][A-Za-z0-9_]{0,63}$`, `error_syscall` `^E[A-Z0-9_]{1,31}$`,
`crash_fingerprint` `^[0-9a-f]{16}$`, `crash_phase` a 2-value enum. Any
path/message/lowercase/oversize value is dropped before it can leave the
machine.
## Scope & safety (no leaking crashes out of scope)
- Fields are populated **only** in `crashShutdown` (which forces
`reason="crash"`), so clean idle/signal stops carry none. Verified by
the existing startup-telemetry test (the pre-listener watcher-rejection
crash path stays the bare `{reason,uptime_ms,total_tool_calls}` shape).
- The four keys exist only on `ToolserverStopProps` and only in the
`toolserver:stop` allowlist entry — they can't ride any other event.
- `describeCrash` is **total**: every field extraction is isolated, so a
hostile throwing getter on `.name`/`.code`/`.stack` yields the phase
(and whatever else is readable) instead of turning crash reporting into
a second crash. The crash/exit guarantee — re-entrancy guard, force-exit
timer, `finalExitCode` escalation — is untouched.
## Tests
- New `crash-diagnostics.test.ts`: class-name/constructor-name/primitive
extraction, syscall capture, cross-user determinism (incl. **Windows**
drive-letter paths and **pnpm** layout), node_modules tail,
line-sensitivity, no-stack omission, hostile-getter resilience, and
"never emits a message/stack" confinement.
- `sanitize.test.ts`: crash fields pass when well-formed; dropped for
message-as-name, path/lowercase syscall, wrong-length/uppercase
fingerprint, unknown phase; `ERR_*` accepted.
- Full telemetry suite (278) + tool-server startup test green;
lint/prettier clean.
## Reviewed
Ran an adversarial review pass (anonymity leak-hunt, determinism,
crash-path safety, scope). Ship-ready; all findings were Low/Nit and the
actionable ones (hostile-getter hardening, Windows/pnpm test coverage,
`error_syscall` doc clarity) are folded in.
### Deliberately deferred (documented, not blockers)
- Sanitizer doesn't *cross-gate* the crash fields on `reason==="crash"`
— the per-field validator design has no sibling access; coupling is
enforced in emit code. The fields are non-identifying regardless, so a
hypothetical stray `crash_phase` on an idle stop is a data nit, not a
leak.
- First-party frames reduce to basename; if `dist` ever ships unbundled
with duplicate basenames, two sites at the same line could collide.
Non-issue for the bundled tool-server.
|
||
|
Hubert Gancarczyk
|
f002fc9540 |
feat(telemetry): track Argent Lens design-review funnel (#478)
## What Adds three privacy-safe telemetry events for the Argent Lens design-review flow, capturing the human-decision funnel that the generic `tool:*` path can't see (submission is an HTTP POST, and the `await_user_selection` tool is hidden in `argent lens` CLI sessions): - **`lens:preview_opened`** — a human loaded the preview UI (`GET /preview/`), deduped per round. - **`lens:round_completed`** — the human submitted their picks (once per round, never on a resubmit). - **`lens:round_abandoned`** — a staged round was discarded before submit (window closed, `argent lens` exited mid-review, or the round superseded). ## Privacy Payloads carry only aggregate counts, booleans, durations, and the device `platform` enum — never element names, comment text, variant code, file paths, or raw device ids. The store emits privacy-safe aggregate structs that are decoupled from the telemetry package; structural assignability to the event props is enforced at the `track()` call site (drift is a compile error). ## Event fields - **`preview_opened`**: `round, element_count, variant_count, is_cli_session, platform` - **`round_completed`**: `round, element_count, variant_count, annotation_count, element_comment_count, skipped_comment_count, has_global_comment, is_cli_session, had_parked_await, round_duration_ms, platform` - **`round_abandoned`**: `round, element_count, variant_count, had_parked_await, is_cli_session, platform` ## Testing - Unit tests for all three events: per-round dedup, exactly-once completion (no resubmit re-emit), abandoned vs happy-path roll, CLI up-front open, and content-agnostic privacy assertions. - Verified end-to-end against a live tool-server: all three events fire with correct aggregate values and zero content leakage through the real sanitize pipeline. --------- Co-authored-by: Ignacy Łątka <latkaignacy@gmail.com> |
||
|
Stanisław Chmiela
|
a7775e4ba6 |
feat(tool-server): add tool-server event log (#367)
We want to display a nicely parsed and rendered log of events that happened over a span of an Argent session. Current log file is not designed to be machine-readable, so adding a JSONL-formatted log of events (gated by `tool-server-event-log`). I eventually want this 😋 https://x.com/j3nnings/status/2069072177584849192 When enabled, the tool server writes Bunyan-compatible records to `ARGENT_EVENT_LOG` or `~/.argent/tool-server-events.jsonl`. The log captures tool-server lifecycle events and registry lifecycle events, including tool invocations and others. To make messages nicer, added `interaction` object to tool definition. This allows tools to define a nicer message in log (print "Screenshot capture completed." instead of "Tool `screenshot` completed.") Adding these to the existing tools is a follow up PR matter I think. |
||
|
Hubert Gancarczyk
|
f38f0e3876 |
feat(telemetry): distinguish tvOS and Android TV from iOS/Android (#477)
## Summary Telemetry inferred the device platform purely from device-id **shape** on the per-tool-call hot path, so a tvOS simulator (same 8-4-4-4-12 UUID as an iPhone) and an Android TV emulator (same `emulator-NNNN` serial as a phone) were both reported as `ios` / `android` — TV usage was invisible in analytics. This adds `tvos` and `android-tv` to the **telemetry** `PLATFORMS` enum and refines the inference so a TV target is attributed correctly. ## Approach - The runtime kind (`mobile` / `tv`) is already resolved and memoized in two module-level caches (`runtimeKindCache` in `ios-devices.ts`, `androidRuntimeKindCache` in `adb.ts`), warmed by the usual TV-path tools (describe, screenshot, keyboard, launch/restart-app, the streaming blueprint, tv-control). - New synchronous, **cache-only** readers (`getCachedSimulatorRuntimeKind`, `getCachedAndroidRuntimeKind`) let `refineTvPlatform` in `http.ts` map `ios → tvos` / `android → android-tv` with **no new `simctl`/`adb` round-trip** on the hot path. - The registry **device** `Platform` (`@argent/registry`) is intentionally left unchanged — a TV stays a `runtimeKind`, not its own device platform, so capability gating and dispatch remain TV-agnostic. The two new values are confined to the telemetry boundary (`tool:invoke` / `tool:complete` / `tool:fail` and child sub-tool attribution). ## Behavior notes - The **first** tool call on a device may land before the cache is warm and reports the coarse base platform (`ios`/`android`); subsequent calls, once describe/screenshot/streaming have run, report the TV variant. This is best-effort coarse telemetry by design. - For an `emulator-NNNN` slot reclaimed by a different-form-factor AVD within one tool-server lifetime, the synchronous reader can momentarily report the prior occupant's kind until an async, avdName-checked path re-probes. Documented inline; no crash and no dispatch impact. ## Verification - **Live E2E** against real booted devices (real `simctl`/`adb` → real cache → real http telemetry recording): Apple TV → `tvos` (cold-cache `ios`, warm `tvos`); iPhone → `ios`; Android phone → `android`. The `android-tv` path is symmetric and covered by unit/integration tests with real leanback-feature probing. - Unit + integration tests added: sanitize allowlist, http platform inference (cold/warm/TV/child), and cache readers. Telemetry suite 192/192; tool-server affected suites green. - `tsc` build, `typecheck:tests`, eslint (`--max-warnings 0`), and prettier all clean. |
||
|
Hubert Gancarczyk
|
3393d22edd |
feat(telemetry): derive distinct_id from a stable host fingerprint (#444)
## Summary Replaces the random-UUID telemetry `distinct_id` with a stable per-machine id derived from the host fingerprint, so PostHog's native "Unique users" counts machines (funnels / retention / breakdowns work with no HogQL). Every Argent entry point - installer, CLI, tool-server, MCP - derives the id from one shared code path, so a given machine reports a single identity across every event and across install / uninstall / reinstall. ## What changed - `distinct_id = uuidv5(hostFingerprint, NAMESPACE)`, where the fingerprint comes from `simulator-server fingerprint`. Deterministic, keeps the UUID shape, and never stores the raw hardware value in PostHog (only the SHA-1-derived UUID leaves the process). - The fingerprint is resolved in `buildPayload` - the single `capture()` path shared by every entry point - so installer / CLI / tool-server / MCP all produce the same id. Resolved lazily, once per process, memoized. - One-time, local-only migration: the first time a fingerprint resolves, `~/.argent/telemetry-id` is rewritten from the legacy random UUID to the derived id. No PostHog `alias` / `$identify` event is emitted. - Fallback: if the fingerprint can't be resolved (unsupported platform, binary absent), the stored or a fresh random UUID is used - identically for every entry point on that machine, self-healing to the derived id on the next successful resolve. - Hardening: never throws; the fingerprint is validated (exactly 64 hex, case-insensitive) before hashing; corrupt / empty / unreadable id files self-heal; the binary is spawned at most once per process. ## Why the id is stable across install -> uninstall -> reinstall Uninstall deletes `~/.argent/telemetry-id`. Because the id is `uuidv5(fingerprint)` and the fingerprint is deterministic per machine, a fresh install recomputes the identical id. Previously only the tool-server resolved the fingerprint while the installer emitted under a random UUID, so this cycle produced a different random id each time. ## Bundling note for reviewers `@argent/telemetry` now depends on `@argent/native-devtools-ios` to locate the binary. That package computes its bin dir from `__dirname` at module load, which is undefined in an ES module. The installer / CLI / MCP ship as ESM bundles, so `bundle-tools.cjs`'s ESM banner is extended to shim `__dirname` / `__filename` (the CJS tool-server bundle already has them natively). Without this shim those bundles would crash on load, so the banner change is load-bearing. ## Testing - Unit: 219 telemetry tests - RFC 4122 v5 vectors, migration, idempotency, case/length validation, corrupt-file self-heal, spawn-at-most-once, and all fallbacks. - E2E on a real machine + booted iPhone 16 Pro: installer, CLI, and tool-server runtimes all emit the same derived id; the id is stable across a delete/recreate (uninstall/reinstall) cycle; a legacy v4 id migrates locally with no alias/identify emitted; verified from an ESM bundle placed in the real install's `dist/` so the binary resolves via the bundled `__dirname` with no env override; tool calls against the device carry the same id. |
||
|
filip131311
|
1aaa94d118 |
feat(telemetry): detect cloud/remote AI-agent environment and tag every event (#418)
## What
Adds a **`cloud_agent`** field to the base telemetry props — alongside
`is_ci` — naming the **cloud/remote AI coding-agent environment** the
process is running in, or `null`. Since it lives in the base props, it's
attached to **every** telemetry event automatically.
Crucially this is **cloud-only**: it fires for vendor-hosted / remote
autonomous runs, **not** for a developer running an agent CLI locally. A
laptop `cursor-agent`/`claude` session returns `null`; only a genuine
remote environment resolves to a slug. (Generic cloud/CI execution stays
`is_ci`; this names the specific agent.)
## How
- **New `packages/telemetry/src/cloud-agent-detect.ts`** —
`detectCloudAgent(env?)` returns a canonical slug or `null`. Modelled on
`ci-detect.ts`: takes `env` lazily for tests, runs nothing at import
time, and only ever emits a fixed allowlist slug — a raw env value is
never copied into telemetry.
- **`base-props.ts`** — adds `cloud_agent: CloudAgent | null` next to
`is_ci`, computed once in the memoized invariant block (no per-event
cost in the long-lived tool-server).
## Signals (only high-confidence remote markers; local CLI use → `null`)
| Slug | Cloud marker | Basis |
|---|---|---|
| `claude_code` | `CLAUDE_CODE_ENVIRONMENT_KIND ∈ {byoc,
anthropic_cloud}`, a `remote_*`/`claude-in-teams` entrypoint, or
`CLAUDE_CODE_REMOTE_SESSION_ID` | Verified against the Claude Code
binary's own internal cloud check |
| `cursor` | `CURSOR_AGENT_WORKER_ID` / `CURSOR_WORKER_POOL_NAME`
(worker/pool exec) | Verified `cursor-agent` reads it from env; the
local CLI sets `CURSOR_AGENT` + `CURSOR_AGENT_CLI_LOCAL_MODE` instead |
| `copilot` | `GITHUB_ACTIONS` + `copilot-swe-agent` actor/workflow |
Confirmed actor identity (also `is_ci`) |
| `replit` | `REPLIT_AGENT` | Agent-specific (the broad `REPL_ID` fires
in any Repl and is **not** used) |
| `devin` | `/opt/.devin` VM marker | Confirmed (`@vercel/detect-agent`)
|
| `jules` | `/opt/environment_summary.sh` VM marker | Confirmed
(official Jules docs) |
### Deliberately excluded
- **Codex cloud** — no reliable cloud-specific marker exists
(`CODEX_SANDBOX` is the *local* sandbox signal; OpenAI declined an
`AGENT=codex` flag in `openai/codex#13416`). Not detected rather than
guessed.
- **Local agent CLIs** — `CURSOR_AGENT`, `CLAUDECODE` with
`entrypoint=cli`, plain `REPL_ID` workspaces: intentionally `null`.
## Privacy
Reads only a fixed allowlist of env-var **names** plus two filesystem
markers (`/opt/.devin`, `/opt/environment_summary.sh`); emits only a
fixed slug. The filesystem checks are injectable and guarded so a
permission error can never break telemetry.
## Tests
- New `test/cloud-agent-detect.test.ts` — every vendor signal, plus the
key **negatives** (local Claude `entrypoint=cli`, local `cursor-agent`,
plain Repl `REPL_ID`, normal CI run) asserting `null`.
- Updated `test/base-props.test.ts` for the renamed key.
- Full telemetry suite **191 passing**, typecheck + build clean. Live
check confirms this local Claude Code session resolves to `null` while a
simulated `byoc`/remote env resolves to `claude_code`.
---
> Scope note: started as a broad "is an agent in the loop" flag
(`agent_env`); narrowed to cloud-only per review — a human using an
agent locally shouldn't look like a cloud run.
|
||
|
Hubert Gancarczyk
|
0edefebe06 |
fix(telemetry): orchestator sub-tools invocations was missing ai_client and platform attributes (#400)
## Summary Fixes telemetry attribution for tools dispatched by orchestrators. `run-sequence`, `flow-execute`, and `flow-add-step` run their steps via `registry.invokeTool()` directly, which minted a fresh, unregistered invocation id per nested call — so the `ai_client` / `platform` the HTTP layer captured for the outer request never reached the nested gestures, and they were recorded as **anonymous** in telemetry. Remove `opt_out` event that is a leftover. ## Changes - Add an optional `recordChildInvocation` recorder to `InvokeToolOptions` / `ToolContext`, bound by the HTTP layer to the outer request's attribution. - New `invokeSubTool` helper: registers each sub-tool under a minted id inheriting the request's AI client, with the platform re-derived from the sub-tool's **own** device arg (`udid` / `device_id` / `avdName`); forwards the recorder so propagation survives arbitrary nesting (e.g. `flow-execute → run-sequence → gesture-tap`). - Route the three orchestrators' sub-dispatch through - Refactor platform inference in the HTTP layer into a shared `platformFromArgs` helper (also hardened to not throw on a malformed device arg). - Remove opt_out event leftover Why re-derive platform per child: an orchestrator like `flow-execute` has no platform of its own and a single flow can target several devices, so the child's device arg is the only correct platform source; the parent's platform is the fallback. |
||
|
Hubert Gancarczyk
|
8483fe23f9 |
feat: add telemetry (#274)
## Summary Adds anonymous, opt-out telemetry for Argent installer, update/uninstall flows, CLI telemetry controls, tool-server lifecycle, and tool invocation health signals. ## What Changed - Added a new private `@argent/telemetry` package - Introduced typed telemetry events with runtime property allowlisting/sanitization. - Added lazy anonymous identity creation in `~/.argent/telemetry-id`. - Added consent handling via: `argent telemetry enable`, `argent telemetry disable`, `argent telemetry status` - Added local debug logging with `ARGENT_TELEMETRY_DEBUG=1`. - Tracks installer funnel events for init, global install/update decisions, editor selection, allowlist decisions, skill installation, rules/agents copy, update, and uninstall. - Tracks tool-server start/stop and tool invocation/complete/fail events. - Adds per-tool invocation IDs so request metadata can be correlated without relying on ordering. - Records only coarse platform metadata (`ios` / `android`) for device tools - Ensures short-lived commands drain telemetry with bounded shutdown before exit. - Wires telemetry package into bundling and injects full Argent CLI version metadata at build time. - Added central `FAILURE_CODES` and `FailureSignal` helpers in `@argent/registry`. - Attached structured, sanitized failure metadata across registry errors, tool-server tools/services, HTTP validation/dependency gates, installer flows, and `argent run`. - Added telemetry allowlisting for safe failure fields only: static error codes/stages, coarse area/kind/command, bounded exit codes, allowlisted signals/spawn codes, and coarse network failure classes. - Split CLI-local `argent run` failures into `cli:run_fail` so they are not double-counted as tool failures. - Added installer telemetry for failed package actions, update/uninstall failures, rules/agents copy results, and skill refresh results. - Made telemetry consent writes atomic and kept telemetry status side-effect free via `peekAnonId()`. - Adds a coarse, anonymous signal to tool telemetry: which AI coding tool is driving the MCP server (Claude Code, Codex, Cursor, Gemini, VS Code, Windsurf, Zed, opencode, Copilot, or other). ## Tests Added / Updated - Telemetry unit coverage for consent precedence, identity creation, base props, sanitization, PostHog config, debug logging, shutdown, opt-out, and forget behavior. - Registry/tool-server tests for invocation IDs, platform metadata boundaries, startup failure flushing, and tool lifecycle telemetry. - Installer tests for final telemetry drain, update action classification, and uninstall identity cleanup ordering. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |