Files
software-mansion__argent/packages/tool-server/test/http-dep-gate.test.ts
filip131311 4975ff490c fix(debugger): cut debugger tool fail rate — structured not-connected results, classified CDP failures, narrow recovery (#610)
## Why

PostHog (30d) shows the debugger tool family is argent's least reliable
surface:

| tool | fail rate | dominant causes |
|---|---|---|
| `debugger-status` | **37%** | METRO_NO_TARGETS /
SERVICE_INITIALIZATION_FAILED / zod |
| `debugger-connect` | **24%** | SERVICE_INITIALIZATION_FAILED (~7k, >1k
users), METRO_NO_TARGETS |
| `debugger-log-registry` | 41.7% raw / 4.2% excl. storms | one user
retry-looped it **12,742×** on NO_TARGETS |

Three design issues drive this: (1) "app/Metro not ready yet" — an
expected precondition — is reported as a tool *error*, which both
inflates fail rates and invites agent retry storms; (2) every CDP
init/lifecycle fault lands in telemetry as the catch-all
`REGISTRY_SERVICE_INITIALIZATION_FAILED` with `error_kind=unknown`; (3)
~600 users/30d hit zod validation failures with zero visibility into
which parameter was wrong.

## What

**1. Classified failures** (`d71efa43`) — six former plain-`Error` sites
in `cdp-client.ts` / `target-selection.ts` now carry dedicated
`DEBUGGER_CDP_*` / `DEBUGGER_TARGET_DEVICE_MISMATCH` signals (messages
byte-identical). The registry's two causeless "terminating window"
`ServiceInitializationError`s get their own
`REGISTRY_SERVICE_TERMINATING` code.

**2. Narrow recovery** (`ff9c1955`) — `recoverable()` on both debugger
blueprints, scoped to windows where retry is provably safe: Metro
recovers only `DEBUGGER_CDP_NOT_CONNECTED` (request never left the host;
a detected socket death already self-heals for the *next* call via the
terminated cascade — recovering `CONNECTION_CLOSED` there would be dead
code, see the blueprint comment). Chromium also recovers
`CONNECTION_CLOSED` for the tab-switch reconnect window.
`REQUEST_TIMEOUT` stays unrecoverable (double-execution risk). A
reachability test proves the retry actually fires for a RUNNING node.

**3. Structured not-connected results** (`48b91ffd`) — `debugger-status`
and `debugger-log-registry` become factories that return `{ status:
"not_connected", reason, detail, guidance }` for classified
preconditions (`metro_not_running | no_app_connected | device_mismatch |
cdp_unreachable | stale_connection | reconnecting`) and still throw on
unexpected faults. `detail` preserves the original error text for
string-matching agents. `debugger-status` gains a socket-state gate
(Metro: dispose stale node; Chromium: report transient reconnect, no
dispose — the client belongs to the ChromiumCdp dependency).
log-registry keeps **no** gate on purpose: captured logs are readable
over a dead socket, and disposal would destroy the post-crash log file.
Flows/run-sequence map a not_connected result to a **failed step**
(mirrors `isUnmetUiWaitResult`) so recorded connectivity gates don't
silently pass. `debugger-evaluate` re-codes agent-expression throws as
`DEBUGGER_EVALUATE_EXPRESSION_THREW`.

**4. Telemetry + guidance + docs** (`7602a70d`) —
`tool:fail.invalid_params` (schema-declared names only; unknown keys →
literal `"unrecognized_keys"`; capped emit-side at 16), new
`debugger:tool_outcome` event (coded enum; joins
`tool:invoke`/`complete` via `tool_invocation_id` —
`ai_client`/`duration` live on the joined row), anti-retry-loop guidance
appended to the Metro discovery errors (first sentences verbatim —
skills match on them), six skills updated, Telemetry.md bumped to v1.03
with the new coded diagnostics disclosed.

## Dashboard migration note

`debugger-status` / `debugger-log-registry` precondition failures stop
emitting `tool:fail` — track them via `debugger:tool_outcome` (`outcome
!= 'connected'`). Per-AI-client breakdowns of those outcomes need a join
on `tool_invocation_id` against `tool:complete`.

## Testing

- Full suites green: tool-server 3,142, telemetry 290, registry 87. New
coverage: recoverable truth tables (incl. non-Error inputs), status
not-connected matrix
(Metro/Chromium/terminating-window/stale/dispose-side-effect/fall-through-rethrow),
log-registry rows, zod `invalid_params` derivation + emission-level
guard, flow gate, evaluate rewrap, sanitize-level enum checks.
- Live-tested against a booted iOS simulator + RN app and an Electron
target (details in PR comments).

Plan hardened through 3 adversarial review rounds (46 confirmed findings
addressed) before implementation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-08-10 11:51:45 +02:00

291 lines
11 KiB
TypeScript

import { describe, it, expect, beforeEach, vi } from "vitest";
import request from "supertest";
import { Registry } from "@argent/registry";
import { z } from "zod";
// PATH deps (xcrun) resolve through `commandOnPath`; mock it so the gate test
// is platform-agnostic rather than asserting a `/bin/sh` call shape.
const commandOnPathMock = vi.fn();
vi.mock("../src/utils/command-on-path", () => ({
commandOnPath: (name: string) => commandOnPathMock(name),
}));
// `probe()` special-cases adb/emulator to use `resolveAndroidBinary`
// (which adds an `$ANDROID_HOME` fallback to PATH). Mock the resolver so
// each test controls availability per-dep instead of fighting the host's
// real $ANDROID_HOME.
const resolveAndroidBinaryMock = vi.fn();
vi.mock("../src/utils/android-binary", () => ({
resolveAndroidBinary: (name: "adb" | "emulator") => resolveAndroidBinaryMock(name),
__resetAndroidBinaryCacheForTesting: () => {},
}));
import { createHttpApp } from "../src/http";
import {
DependencyMissingError,
__resetDepCacheForTests,
ensureDep,
} from "../src/utils/check-deps";
import { InvalidToolInputError } from "../src/utils/capability";
function stubProbe(missing: readonly string[]): void {
commandOnPathMock.mockImplementation(async (name: string) =>
missing.includes(name) ? null : `/usr/bin/${name}`
);
resolveAndroidBinaryMock.mockImplementation(async (name: string) => {
return missing.includes(name) ? null : `/usr/bin/${name}`;
});
}
describe("http dependency gate", () => {
beforeEach(() => {
__resetDepCacheForTests();
commandOnPathMock.mockReset();
resolveAndroidBinaryMock.mockReset();
});
it("returns 424 with a pretty message when a pre-flight dep is missing", async () => {
stubProbe(["adb"]);
const registry = new Registry();
registry.registerTool({
id: "android-thing",
requires: ["adb"],
zodSchema: z.object({}),
services: () => ({}),
async execute() {
// Should never run — the dep gate blocks the request before execute.
throw new Error("execute should have been skipped");
},
});
const { app } = createHttpApp(registry);
const res = await request(app).post("/tools/android-thing").send({});
expect(res.status).toBe(424);
expect(res.body.error).toMatch(/android-platform-tools/);
});
it("records a static failure signal when a pre-flight dep is missing", async () => {
stubProbe(["adb"]);
const recordFailure = vi.fn();
const registry = new Registry();
registry.registerTool({
id: "android-thing",
requires: ["adb"],
zodSchema: z.object({}),
services: () => ({}),
async execute() {
throw new Error("execute should have been skipped");
},
});
const { app } = createHttpApp(registry, { recordFailure });
const res = await request(app).post("/tools/android-thing").send({});
expect(res.status).toBe(424);
expect(recordFailure).toHaveBeenCalledWith(
"android-thing",
{},
{
error_code: "HTTP_DEPENDENCY_PREFLIGHT_MISSING",
failure_stage: "http_dependency_preflight",
failure_area: "http",
error_kind: "dependency_missing",
},
expect.any(Number)
);
});
it("records a static failure signal when request validation fails", async () => {
stubProbe([]);
const recordFailure = vi.fn();
const registry = new Registry();
registry.registerTool({
id: "validated-thing",
zodSchema: z.object({ count: z.number() }),
services: () => ({}),
async execute() {
throw new Error("execute should have been skipped");
},
});
const { app } = createHttpApp(registry, { recordFailure });
const res = await request(app).post("/tools/validated-thing").send({ count: "nope" });
expect(res.status).toBe(400);
expect(recordFailure).toHaveBeenCalledWith(
"validated-thing",
// Schema-declared names of the failing params only — never values.
{ invalid_params: ["count"] },
{
error_code: "HTTP_ZOD_VALIDATION_FAILED",
failure_stage: "http_zod_validation",
failure_area: "http",
error_kind: "validation",
},
expect.any(Number)
);
expect(JSON.stringify(recordFailure.mock.calls)).not.toContain("Expected number");
});
it("reports strict-schema unknown keys as the literal token, never the user-typed names", async () => {
stubProbe([]);
const recordFailure = vi.fn();
const registry = new Registry();
registry.registerTool({
id: "strict-thing",
zodSchema: z.object({ port: z.number() }).strict(),
services: () => ({}),
async execute() {
throw new Error("execute should have been skipped");
},
});
const { app } = createHttpApp(registry, { recordFailure });
const res = await request(app)
.post("/tools/strict-thing")
.send({ port: 1, secret_user_key: "hunter2" });
expect(res.status).toBe(400);
expect(recordFailure).toHaveBeenCalledWith(
"strict-thing",
{ invalid_params: ["unrecognized_keys"] },
expect.objectContaining({ error_code: "HTTP_ZOD_VALIDATION_FAILED" }),
expect.any(Number)
);
expect(JSON.stringify(recordFailure.mock.calls)).not.toContain("secret_user_key");
});
it("invokes the tool normally when declared deps are present", async () => {
stubProbe([]);
const registry = new Registry();
registry.registerTool({
id: "ios-thing",
requires: ["xcrun"],
zodSchema: z.object({}),
services: () => ({}),
async execute() {
return { ran: true };
},
});
const { app } = createHttpApp(registry);
const res = await request(app).post("/tools/ios-thing").send({});
expect(res.status).toBe(200);
expect(res.body.data).toEqual({ ran: true });
});
it("surfaces a DependencyMissingError thrown from inside execute (post-classify path) as 424", async () => {
// Two probes expected: the first stubs all missing; the second call (for
// the cross-platform tool's in-execute ensureDep) re-probes adb and finds
// it still missing. This is the cross-platform tool pattern: `requires`
// is absent so the pre-flight gate doesn't fire, and the dep check
// happens after classifyDevice has picked android.
stubProbe(["adb"]);
const registry = new Registry();
registry.registerTool({
id: "cross-platform-thing",
zodSchema: z.object({}),
services: () => ({}),
async execute() {
// Simulate the classify → ensureDep pattern used by launch-app etc.
await ensureDep("adb");
return { ran: true };
},
});
const { app } = createHttpApp(registry);
const res = await request(app).post("/tools/cross-platform-thing").send({});
expect(res.status).toBe(424);
expect(res.body.error).toMatch(/android-platform-tools/);
});
it("maps an InvalidToolInputError thrown from execute to 400, not 500", async () => {
// A tool rejecting its (well-typed) arguments — e.g. a newline / non-ASCII
// char in Android `keyboard` text, an unknown named key — is a client input
// error, not an internal fault. It reaches the dispatcher wrapped in
// ToolExecutionError, so the mapping must walk the cause chain.
stubProbe([]);
const recordFailure = vi.fn();
const registry = new Registry();
registry.registerTool({
id: "picky-input",
zodSchema: z.object({}),
services: () => ({}),
async execute() {
throw new InvalidToolInputError("that argument can't be carried out on this device");
},
});
const { app } = createHttpApp(registry, { recordFailure });
const res = await request(app).post("/tools/picky-input").send({});
expect(res.status).toBe(400);
expect(res.body.error).toBe("that argument can't be carried out on this device");
// An InvalidToolInputError thrown FROM execute() is recorded once, by the
// registry's failure listener (the error carries its own signal out of
// execute) — the HTTP layer must NOT also record it, or the failure
// double-counts. `emitHttpFailure` fires only on pre-execute HTTP-layer
// faults (zod / capability / device-resolution / dep-preflight), never on
// this execute-catch 400 path.
expect(recordFailure).not.toHaveBeenCalled();
});
it("does not call the dep probe for tools without a `requires` declaration", async () => {
stubProbe([]);
const registry = new Registry();
registry.registerTool({
id: "no-deps",
zodSchema: z.object({}),
services: () => ({}),
async execute() {
return { ran: true };
},
});
const { app } = createHttpApp(registry);
const res = await request(app).post("/tools/no-deps").send({});
expect(res.status).toBe(200);
expect(commandOnPathMock).not.toHaveBeenCalled();
});
it("prefers 424 over 400 when a dependency error is nested under an InvalidToolInputError", async () => {
// Pins the ordering invariant documented at the dispatcher's
// InvalidToolInputError → 400 check (http.ts): `findDependencyMissing`
// scans the WHOLE cause chain first, so for a chain carrying both classes
// the 424 wins. Reordering the two checks — or a throw site nesting a
// DependencyMissingError under an InvalidToolInputError — flips this test
// instead of silently flipping the status code.
stubProbe([]);
const registry = new Registry();
registry.registerTool({
id: "dual-class-error",
zodSchema: z.object({}),
services: () => ({}),
async execute() {
const err = new InvalidToolInputError(
"that argument needs a missing dependency"
) as InvalidToolInputError & { cause?: unknown };
err.cause = new DependencyMissingError(["adb"], "install adb please");
throw err;
},
});
const { app } = createHttpApp(registry);
const res = await request(app).post("/tools/dual-class-error").send({});
expect(res.status).toBe(424);
expect(res.body.error).toBe("install adb please");
});
it("still returns 424 when the DependencyMissingError is buried two levels deep in the cause chain", async () => {
// The registry wraps execute() errors in ToolExecutionError with `cause`.
// If a future middleware adds a second wrap (or something else does), a
// naive one-level `.cause` check regresses 424 → 500. Walk the chain.
stubProbe([]);
const registry = new Registry();
registry.registerTool({
id: "double-wrap",
zodSchema: z.object({}),
services: () => ({}),
async execute() {
const inner = new DependencyMissingError(["adb"], "install adb please");
const middle = new Error("outer wrap") as Error & { cause?: unknown };
middle.cause = inner;
const outer = new Error("tool failed") as Error & { cause?: unknown };
outer.cause = middle;
throw outer;
},
});
const { app } = createHttpApp(registry);
const res = await request(app).post("/tools/double-wrap").send({});
expect(res.status).toBe(424);
expect(res.body.error).toBe("install adb please");
});
});