Files
software-mansion__argent/packages/tool-server/test/debugger/not-connected-map.test.ts
filip131311 4975ff490c fix(debugger): cut debugger tool fail rate — structured not-connected results, classified CDP failures, narrow recovery (#610)
## Why

PostHog (30d) shows the debugger tool family is argent's least reliable
surface:

| tool | fail rate | dominant causes |
|---|---|---|
| `debugger-status` | **37%** | METRO_NO_TARGETS /
SERVICE_INITIALIZATION_FAILED / zod |
| `debugger-connect` | **24%** | SERVICE_INITIALIZATION_FAILED (~7k, >1k
users), METRO_NO_TARGETS |
| `debugger-log-registry` | 41.7% raw / 4.2% excl. storms | one user
retry-looped it **12,742×** on NO_TARGETS |

Three design issues drive this: (1) "app/Metro not ready yet" — an
expected precondition — is reported as a tool *error*, which both
inflates fail rates and invites agent retry storms; (2) every CDP
init/lifecycle fault lands in telemetry as the catch-all
`REGISTRY_SERVICE_INITIALIZATION_FAILED` with `error_kind=unknown`; (3)
~600 users/30d hit zod validation failures with zero visibility into
which parameter was wrong.

## What

**1. Classified failures** (`d71efa43`) — six former plain-`Error` sites
in `cdp-client.ts` / `target-selection.ts` now carry dedicated
`DEBUGGER_CDP_*` / `DEBUGGER_TARGET_DEVICE_MISMATCH` signals (messages
byte-identical). The registry's two causeless "terminating window"
`ServiceInitializationError`s get their own
`REGISTRY_SERVICE_TERMINATING` code.

**2. Narrow recovery** (`ff9c1955`) — `recoverable()` on both debugger
blueprints, scoped to windows where retry is provably safe: Metro
recovers only `DEBUGGER_CDP_NOT_CONNECTED` (request never left the host;
a detected socket death already self-heals for the *next* call via the
terminated cascade — recovering `CONNECTION_CLOSED` there would be dead
code, see the blueprint comment). Chromium also recovers
`CONNECTION_CLOSED` for the tab-switch reconnect window.
`REQUEST_TIMEOUT` stays unrecoverable (double-execution risk). A
reachability test proves the retry actually fires for a RUNNING node.

**3. Structured not-connected results** (`48b91ffd`) — `debugger-status`
and `debugger-log-registry` become factories that return `{ status:
"not_connected", reason, detail, guidance }` for classified
preconditions (`metro_not_running | no_app_connected | device_mismatch |
cdp_unreachable | stale_connection | reconnecting`) and still throw on
unexpected faults. `detail` preserves the original error text for
string-matching agents. `debugger-status` gains a socket-state gate
(Metro: dispose stale node; Chromium: report transient reconnect, no
dispose — the client belongs to the ChromiumCdp dependency).
log-registry keeps **no** gate on purpose: captured logs are readable
over a dead socket, and disposal would destroy the post-crash log file.
Flows/run-sequence map a not_connected result to a **failed step**
(mirrors `isUnmetUiWaitResult`) so recorded connectivity gates don't
silently pass. `debugger-evaluate` re-codes agent-expression throws as
`DEBUGGER_EVALUATE_EXPRESSION_THREW`.

**4. Telemetry + guidance + docs** (`7602a70d`) —
`tool:fail.invalid_params` (schema-declared names only; unknown keys →
literal `"unrecognized_keys"`; capped emit-side at 16), new
`debugger:tool_outcome` event (coded enum; joins
`tool:invoke`/`complete` via `tool_invocation_id` —
`ai_client`/`duration` live on the joined row), anti-retry-loop guidance
appended to the Metro discovery errors (first sentences verbatim —
skills match on them), six skills updated, Telemetry.md bumped to v1.03
with the new coded diagnostics disclosed.

## Dashboard migration note

`debugger-status` / `debugger-log-registry` precondition failures stop
emitting `tool:fail` — track them via `debugger:tool_outcome` (`outcome
!= 'connected'`). Per-AI-client breakdowns of those outcomes need a join
on `tool_invocation_id` against `tool:complete`.

## Testing

- Full suites green: tool-server 3,142, telemetry 290, registry 87. New
coverage: recoverable truth tables (incl. non-Error inputs), status
not-connected matrix
(Metro/Chromium/terminating-window/stale/dispose-side-effect/fall-through-rethrow),
log-registry rows, zod `invalid_params` derivation + emission-level
guard, flow gate, evaluate rewrap, sanitize-level enum checks.
- Live-tested against a booted iOS simulator + RN app and an Electron
target (details in PR comments).

Plan hardened through 3 adversarial review rounds (46 confirmed findings
addressed) before implementation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-08-10 11:51:45 +02:00

102 lines
4.1 KiB
TypeScript

import { describe, it, expect } from "vitest";
import { FAILURE_CODES, FailureError, type FailureSignal } from "@argent/registry";
import { classifyNotConnected, buildNotConnected } from "../../src/tools/debugger/not-connected";
/**
* Pins EVERY entry of NOT_CONNECTED_CODE_MAP. The map is the contract that
* turns a classified resolution failure into a structured not_connected result
* — deleting any single entry silently reverts that code to a thrown tool
* failure (the regression the map exists to prevent), so each row is asserted
* individually here.
*/
function coded(
error_code: FailureSignal["error_code"],
message = "x",
error_kind: FailureSignal["error_kind"] = "network"
) {
return new FailureError(message, {
error_code,
failure_stage: "test_stage",
failure_area: "tool_server",
error_kind,
});
}
const MAP: Array<[FailureSignal["error_code"], string]> = [
[FAILURE_CODES.DEBUGGER_METRO_NOT_RUNNING, "metro_not_running"],
[FAILURE_CODES.DEBUGGER_METRO_NO_TARGETS, "no_app_connected"],
[FAILURE_CODES.DEBUGGER_TARGET_DEVICE_MISMATCH, "device_mismatch"],
[FAILURE_CODES.DEBUGGER_CDP_CONNECT_FAILED, "cdp_unreachable"],
[FAILURE_CODES.DEBUGGER_CDP_SOCKET_CLOSED_BEFORE_OPEN, "cdp_unreachable"],
[FAILURE_CODES.DEBUGGER_CDP_NOT_CONNECTED, "cdp_unreachable"],
[FAILURE_CODES.DEBUGGER_CDP_CONNECTION_CLOSED, "cdp_unreachable"],
[FAILURE_CODES.DEBUGGER_CDP_REQUEST_TIMEOUT, "runtime_unresponsive"],
[FAILURE_CODES.CHROMIUM_CDP_UNREACHABLE, "cdp_unreachable"],
[FAILURE_CODES.CHROMIUM_CDP_INVALID_RESPONSE, "cdp_unreachable"],
[FAILURE_CODES.CHROMIUM_CDP_NO_PAGE_TARGET, "cdp_unreachable"],
[FAILURE_CODES.REGISTRY_SERVICE_TERMINATING, "reconnecting"],
];
describe("classifyNotConnected code map", () => {
it.each(MAP)("%s → %s", (code, reason) => {
expect(classifyNotConnected(coded(code))).toBe(reason);
});
it("an unmapped classified code stays unclassified (rethrow path)", () => {
expect(
classifyNotConnected(coded(FAILURE_CODES.REGISTRY_SERVICE_INITIALIZATION_FAILED))
).toBeUndefined();
});
it("a plain Error stays unclassified (rethrow path)", () => {
expect(classifyNotConnected(new Error("boom"))).toBeUndefined();
});
});
describe("guidance platform-correctness", () => {
it("chromium cdp_unreachable guidance never points at launch-app (a documented no-op on Chromium)", () => {
const result = buildNotConnected(
"cdp_unreachable",
coded(FAILURE_CODES.CHROMIUM_CDP_UNREACHABLE),
{ port: 8081, device_id: "chromium-cdp-9222" }
);
// The Metro phrasing "Verify the app is running (launch-app)" must not
// appear — following it on Chromium manufactures a guaranteed second
// failure. The actionable path is a relaunch with --remote-debugging-port.
expect(result.guidance).not.toMatch(/\(launch-app\)/);
expect(result.guidance).toContain("--remote-debugging-port");
expect(result.guidance).toContain("launch-app cannot start a Chromium app");
});
it("Metro cdp_unreachable keeps the launch-app guidance (it IS actionable there)", () => {
const result = buildNotConnected(
"cdp_unreachable",
coded(FAILURE_CODES.DEBUGGER_CDP_CONNECT_FAILED),
{ port: 8081, device_id: "emulator-5554" }
);
expect(result.guidance).toContain("launch-app");
expect(result.guidance).not.toContain("--remote-debugging-port");
});
it("runtime_unresponsive guidance warns about the per-attempt timeout on both platforms", () => {
const metro = buildNotConnected(
"runtime_unresponsive",
coded(FAILURE_CODES.DEBUGGER_CDP_REQUEST_TIMEOUT),
{ port: 8081, device_id: "emulator-5554" }
);
const chromium = buildNotConnected(
"runtime_unresponsive",
coded(FAILURE_CODES.DEBUGGER_CDP_REQUEST_TIMEOUT),
{ port: 8081, device_id: "chromium-cdp-9222" }
);
for (const r of [metro, chromium]) {
expect(r.guidance).toMatch(/Do not retry in a loop/);
expect(r.guidance).toMatch(/timeout/);
}
expect(metro.guidance).toContain("restart-app");
expect(chromium.guidance).not.toContain("restart-app");
expect(chromium.guidance).toContain("electronAppPath");
});
});