mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
stack/721-base-legacy
5 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ignacy Łątka
|
0a6dbec87c |
feat(telemetry): export events via OpenTelemetry OTLP logs instead of PostHog (#570)
## What
Swaps the `posthog-node` transport for an OpenTelemetry Logs exporter.
Everything above the transport stays as it is - opt-out consent,
per-machine identity and fingerprint, the sanitizer allowlist, the typed
event surface, crash diagnostics. Only the ~3-method client boundary
changes.
One analytics event becomes one OTLP/HTTP log record:
| PostHog | OpenTelemetry |
| --- | --- |
| `capture({ event, distinctId, properties })` | `logger.emit({ body:
event, attributes: { distinct_id, "event.name", ...props } })` |
| event name | log-record `body` |
| `distinctId` | `distinct_id` attribute |
| event/base properties | log-record attributes |
| `client.shutdown()` | `LoggerProvider.shutdown()` (force-flush +
teardown) |
Records go to a hard-coded Software Mansion collector. The endpoint and
the authorization header are passed to the exporter explicitly in code,
and an explicit value beats the corresponding `OTEL_EXPORTER_OTLP_*`
variable - the same anti-exfiltration property the fixed PostHog host
had.
## Changes
- `packages/telemetry/src/posthog.ts` becomes `otel.ts`:
`LoggerProvider` + `BatchLogRecordProcessor` + `OTLPLogExporter`
(HTTP/JSON). Queue size 20 and a 10 s flush interval mirror the old
client; the per-export deadline is 1.5 s, kept at or below `index.ts`'s
drain budget so a stalled export can't hold a short-lived command open.
- Dependencies: drop `posthog-node`, add
`@opentelemetry/{api,api-logs,sdk-logs,exporter-logs-otlp-http,resources}`.
They inline cleanly into the esbuild bundles.
- Build-time ingest token via esbuild `define`
(`ARGENT_OTEL_INGEST_TOKEN`), same mechanism as `ARGENT_CLI_VERSION`.
Unset in a dev build leaves the client unconstructed and telemetry
inert.
- Base props: drop `$process_person_profile` (a PostHog directive,
meaningless to a collector) and rename `$session_id` to `session_id`.
Null-valued attributes are dropped, since OTel rejects them -
`cloud_agent` is null on the common path, and absence means the same
thing.
- `Telemetry.md`: the "who we disclose to" and "international transfers"
sections describe a Software Mansion-operated OTLP collector instead of
PostHog as a third-party processor.
## A hang this uncovered
`timeoutMillis` only bounds a request once its socket is connected,
because the exporter applies it with `req.setTimeout()`. Against a
collector address that drops packets rather than refusing them - a
corporate egress filter, a dead host behind a firewall - the socket sits
in the connecting state where no export deadline reaches it, and the
process stays alive for the OS connect timeout: 75 s past a `shutdown()`
that had already resolved at 1.75 s.
Pairing the export deadline with an agent-level socket timeout, armed at
socket creation so it covers connection establishment too, fixes it:
| collector state | before | after |
| --- | --- | --- |
| healthy | 0.18 s, delivered | 0.18 s, delivered |
| connection refused | 0.94 s | 1.03 s |
| blackholed (http and https) | 75.0 s | 1.6 s |
| connected but never responds | - | 1.6 s |
`posthog-node` exited in 10.6 s under the same blackhole (undici aborts
a connecting socket), so this was a regression as well as a hang. A unit
test that fails without the fix guards it.
## Found in review
Each of these was reproduced before it was changed, and each is pinned
by a test that fails without the fix.
- **The environment's OTLP headers were forwarded to the collector.**
The SDK merges `OTEL_EXPORTER_OTLP_HEADERS` / `_LOGS_HEADERS` into every
request, keeping any key the code does not set itself. Out in the world
that variable holds the developer's *own* observability credential - a
Honeycomb team key, a Dynatrace `Api-Token`, Grafana Cloud basic auth -
so any machine already running OpenTelemetry shipped that third-party
secret here on every batch, without it passing the sanitizer or
appearing in `Telemetry.md`. Reproduced by watching `x-honeycomb-team:
hcaik_…` arrive at a capture server. The variables are now cleared
across the exporter constructor (where the SDK resolves its header set)
and restored after.
- **`httpAgentOptions` silently disabled connection reuse.** Supplying
it replaces the agent the SDK would build, whose default is `keepAlive:
true`. Measured on loopback: three batches shared one socket by default,
and opened three sockets with `httpAgentOptions` - a fresh TCP+TLS
handshake per 10 s batch in the long-lived tool-server. `keepAlive` is
restated alongside the timeout.
- **`markDisabled()`'s drain had no grace period.** It raced flat
against `SHORT_FLUSH_TIMEOUT_MS` while `shutdown()` allowed `timeoutMs +
250`. Since the exporter's own deadline equals that budget, the flat
race fired at the same instant and always abandoned the batch instead of
letting it finish. Both call sites share one `raceDrain()` now.
- **Export failures reached nobody.** The batch processor hands errors
to OpenTelemetry's global handler, so `LoggerProvider.shutdown()`
resolves identically for a delivered batch and a rejected token - making
`shutdown()`'s `catch` unreachable. The SDK's diagnostics are now routed
into the existing debug channel under `ARGENT_TELEMETRY_DEBUG=1`; a run
without the flag stays silent so no host application's diag logger is
disturbed.
- **The build-time token now beats the `__ARGENT_OTEL_TOKEN_TEST` seam**
rather than losing to it, so the seam reaches unbundled source only.
Verified with esbuild across all three build modes.
- **The endpoint guard test mocked the exporter it guards.** It could
only show that the code passes a `url`; an SDK upgrade that changed the
precedence would have kept it green. `otel-endpoint-live.test.ts` drives
the real exporter against loopback servers - dropping the explicit
`url`, the explicit `authorization`, or the header clearing each fails
it.
- **The wire shape had no pins at all.** Bypassing the sanitizer
entirely, sending an explicit `null`, renaming `service.name` and
dropping `event.name` all left the suite green. Four tests now fail on
those mutations. The sanitizer bypass was unpinned on `main` too.
## Before this ships
1. ~~**Collector URL**~~ - done in
|
||
|
filip131311
|
4975ff490c |
fix(debugger): cut debugger tool fail rate — structured not-connected results, classified CDP failures, narrow recovery (#610)
## Why
PostHog (30d) shows the debugger tool family is argent's least reliable
surface:
| tool | fail rate | dominant causes |
|---|---|---|
| `debugger-status` | **37%** | METRO_NO_TARGETS /
SERVICE_INITIALIZATION_FAILED / zod |
| `debugger-connect` | **24%** | SERVICE_INITIALIZATION_FAILED (~7k, >1k
users), METRO_NO_TARGETS |
| `debugger-log-registry` | 41.7% raw / 4.2% excl. storms | one user
retry-looped it **12,742×** on NO_TARGETS |
Three design issues drive this: (1) "app/Metro not ready yet" — an
expected precondition — is reported as a tool *error*, which both
inflates fail rates and invites agent retry storms; (2) every CDP
init/lifecycle fault lands in telemetry as the catch-all
`REGISTRY_SERVICE_INITIALIZATION_FAILED` with `error_kind=unknown`; (3)
~600 users/30d hit zod validation failures with zero visibility into
which parameter was wrong.
## What
**1. Classified failures** (`d71efa43`) — six former plain-`Error` sites
in `cdp-client.ts` / `target-selection.ts` now carry dedicated
`DEBUGGER_CDP_*` / `DEBUGGER_TARGET_DEVICE_MISMATCH` signals (messages
byte-identical). The registry's two causeless "terminating window"
`ServiceInitializationError`s get their own
`REGISTRY_SERVICE_TERMINATING` code.
**2. Narrow recovery** (`ff9c1955`) — `recoverable()` on both debugger
blueprints, scoped to windows where retry is provably safe: Metro
recovers only `DEBUGGER_CDP_NOT_CONNECTED` (request never left the host;
a detected socket death already self-heals for the *next* call via the
terminated cascade — recovering `CONNECTION_CLOSED` there would be dead
code, see the blueprint comment). Chromium also recovers
`CONNECTION_CLOSED` for the tab-switch reconnect window.
`REQUEST_TIMEOUT` stays unrecoverable (double-execution risk). A
reachability test proves the retry actually fires for a RUNNING node.
**3. Structured not-connected results** (`48b91ffd`) — `debugger-status`
and `debugger-log-registry` become factories that return `{ status:
"not_connected", reason, detail, guidance }` for classified
preconditions (`metro_not_running | no_app_connected | device_mismatch |
cdp_unreachable | stale_connection | reconnecting`) and still throw on
unexpected faults. `detail` preserves the original error text for
string-matching agents. `debugger-status` gains a socket-state gate
(Metro: dispose stale node; Chromium: report transient reconnect, no
dispose — the client belongs to the ChromiumCdp dependency).
log-registry keeps **no** gate on purpose: captured logs are readable
over a dead socket, and disposal would destroy the post-crash log file.
Flows/run-sequence map a not_connected result to a **failed step**
(mirrors `isUnmetUiWaitResult`) so recorded connectivity gates don't
silently pass. `debugger-evaluate` re-codes agent-expression throws as
`DEBUGGER_EVALUATE_EXPRESSION_THREW`.
**4. Telemetry + guidance + docs** (`7602a70d`) —
`tool:fail.invalid_params` (schema-declared names only; unknown keys →
literal `"unrecognized_keys"`; capped emit-side at 16), new
`debugger:tool_outcome` event (coded enum; joins
`tool:invoke`/`complete` via `tool_invocation_id` —
`ai_client`/`duration` live on the joined row), anti-retry-loop guidance
appended to the Metro discovery errors (first sentences verbatim —
skills match on them), six skills updated, Telemetry.md bumped to v1.03
with the new coded diagnostics disclosed.
## Dashboard migration note
`debugger-status` / `debugger-log-registry` precondition failures stop
emitting `tool:fail` — track them via `debugger:tool_outcome` (`outcome
!= 'connected'`). Per-AI-client breakdowns of those outcomes need a join
on `tool_invocation_id` against `tool:complete`.
## Testing
- Full suites green: tool-server 3,142, telemetry 290, registry 87. New
coverage: recoverable truth tables (incl. non-Error inputs), status
not-connected matrix
(Metro/Chromium/terminating-window/stale/dispose-side-effect/fall-through-rethrow),
log-registry rows, zod `invalid_params` derivation + emission-level
guard, flow gate, evaluate rewrap, sanitize-level enum checks.
- Live-tested against a booted iOS simulator + RN app and an Electron
target (details in PR comments).
Plan hardened through 3 adversarial review rounds (46 confirmed findings
addressed) before implementation.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Hubert Gancarczyk
|
f002fc9540 |
feat(telemetry): track Argent Lens design-review funnel (#478)
## What Adds three privacy-safe telemetry events for the Argent Lens design-review flow, capturing the human-decision funnel that the generic `tool:*` path can't see (submission is an HTTP POST, and the `await_user_selection` tool is hidden in `argent lens` CLI sessions): - **`lens:preview_opened`** — a human loaded the preview UI (`GET /preview/`), deduped per round. - **`lens:round_completed`** — the human submitted their picks (once per round, never on a resubmit). - **`lens:round_abandoned`** — a staged round was discarded before submit (window closed, `argent lens` exited mid-review, or the round superseded). ## Privacy Payloads carry only aggregate counts, booleans, durations, and the device `platform` enum — never element names, comment text, variant code, file paths, or raw device ids. The store emits privacy-safe aggregate structs that are decoupled from the telemetry package; structural assignability to the event props is enforced at the `track()` call site (drift is a compile error). ## Event fields - **`preview_opened`**: `round, element_count, variant_count, is_cli_session, platform` - **`round_completed`**: `round, element_count, variant_count, annotation_count, element_comment_count, skipped_comment_count, has_global_comment, is_cli_session, had_parked_await, round_duration_ms, platform` - **`round_abandoned`**: `round, element_count, variant_count, had_parked_await, is_cli_session, platform` ## Testing - Unit tests for all three events: per-round dedup, exactly-once completion (no resubmit re-emit), abandoned vs happy-path roll, CLI up-front open, and content-agnostic privacy assertions. - Verified end-to-end against a live tool-server: all three events fire with correct aggregate values and zero content leakage through the real sanitize pipeline. --------- Co-authored-by: Ignacy Łątka <latkaignacy@gmail.com> |
||
|
Hubert Gancarczyk
|
3393d22edd |
feat(telemetry): derive distinct_id from a stable host fingerprint (#444)
## Summary Replaces the random-UUID telemetry `distinct_id` with a stable per-machine id derived from the host fingerprint, so PostHog's native "Unique users" counts machines (funnels / retention / breakdowns work with no HogQL). Every Argent entry point - installer, CLI, tool-server, MCP - derives the id from one shared code path, so a given machine reports a single identity across every event and across install / uninstall / reinstall. ## What changed - `distinct_id = uuidv5(hostFingerprint, NAMESPACE)`, where the fingerprint comes from `simulator-server fingerprint`. Deterministic, keeps the UUID shape, and never stores the raw hardware value in PostHog (only the SHA-1-derived UUID leaves the process). - The fingerprint is resolved in `buildPayload` - the single `capture()` path shared by every entry point - so installer / CLI / tool-server / MCP all produce the same id. Resolved lazily, once per process, memoized. - One-time, local-only migration: the first time a fingerprint resolves, `~/.argent/telemetry-id` is rewritten from the legacy random UUID to the derived id. No PostHog `alias` / `$identify` event is emitted. - Fallback: if the fingerprint can't be resolved (unsupported platform, binary absent), the stored or a fresh random UUID is used - identically for every entry point on that machine, self-healing to the derived id on the next successful resolve. - Hardening: never throws; the fingerprint is validated (exactly 64 hex, case-insensitive) before hashing; corrupt / empty / unreadable id files self-heal; the binary is spawned at most once per process. ## Why the id is stable across install -> uninstall -> reinstall Uninstall deletes `~/.argent/telemetry-id`. Because the id is `uuidv5(fingerprint)` and the fingerprint is deterministic per machine, a fresh install recomputes the identical id. Previously only the tool-server resolved the fingerprint while the installer emitted under a random UUID, so this cycle produced a different random id each time. ## Bundling note for reviewers `@argent/telemetry` now depends on `@argent/native-devtools-ios` to locate the binary. That package computes its bin dir from `__dirname` at module load, which is undefined in an ES module. The installer / CLI / MCP ship as ESM bundles, so `bundle-tools.cjs`'s ESM banner is extended to shim `__dirname` / `__filename` (the CJS tool-server bundle already has them natively). Without this shim those bundles would crash on load, so the banner change is load-bearing. ## Testing - Unit: 219 telemetry tests - RFC 4122 v5 vectors, migration, idempotency, case/length validation, corrupt-file self-heal, spawn-at-most-once, and all fallbacks. - E2E on a real machine + booted iPhone 16 Pro: installer, CLI, and tool-server runtimes all emit the same derived id; the id is stable across a delete/recreate (uninstall/reinstall) cycle; a legacy v4 id migrates locally with no alias/identify emitted; verified from an ESM bundle placed in the real install's `dist/` so the binary resolves via the bundled `__dirname` with no env override; tool calls against the device carry the same id. |
||
|
Hubert Gancarczyk
|
8483fe23f9 |
feat: add telemetry (#274)
## Summary Adds anonymous, opt-out telemetry for Argent installer, update/uninstall flows, CLI telemetry controls, tool-server lifecycle, and tool invocation health signals. ## What Changed - Added a new private `@argent/telemetry` package - Introduced typed telemetry events with runtime property allowlisting/sanitization. - Added lazy anonymous identity creation in `~/.argent/telemetry-id`. - Added consent handling via: `argent telemetry enable`, `argent telemetry disable`, `argent telemetry status` - Added local debug logging with `ARGENT_TELEMETRY_DEBUG=1`. - Tracks installer funnel events for init, global install/update decisions, editor selection, allowlist decisions, skill installation, rules/agents copy, update, and uninstall. - Tracks tool-server start/stop and tool invocation/complete/fail events. - Adds per-tool invocation IDs so request metadata can be correlated without relying on ordering. - Records only coarse platform metadata (`ios` / `android`) for device tools - Ensures short-lived commands drain telemetry with bounded shutdown before exit. - Wires telemetry package into bundling and injects full Argent CLI version metadata at build time. - Added central `FAILURE_CODES` and `FailureSignal` helpers in `@argent/registry`. - Attached structured, sanitized failure metadata across registry errors, tool-server tools/services, HTTP validation/dependency gates, installer flows, and `argent run`. - Added telemetry allowlisting for safe failure fields only: static error codes/stages, coarse area/kind/command, bounded exit codes, allowlisted signals/spawn codes, and coarse network failure classes. - Split CLI-local `argent run` failures into `cli:run_fail` so they are not double-counted as tool failures. - Added installer telemetry for failed package actions, update/uninstall failures, rules/agents copy results, and skill refresh results. - Made telemetry consent writes atomic and kept telemetry status side-effect free via `peekAnonId()`. - Adds a coarse, anonymous signal to tool telemetry: which AI coding tool is driving the MCP server (Claude Code, Codex, Cursor, Gemini, VS Code, Windsurf, Zed, opencode, Copilot, or other). ## Tests Added / Updated - Telemetry unit coverage for consent precedence, identity creation, base props, sanitization, PostHog config, debug logging, shutdown, opt-out, and forget behavior. - Registry/tool-server tests for invocation IDs, platform metadata boundaries, startup failure flushing, and tool lifecycle telemetry. - Installer tests for final telemetry drain, update action classification, and uninstall identity cleanup ordering. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |