mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
4975ff490c
## Why
PostHog (30d) shows the debugger tool family is argent's least reliable
surface:
| tool | fail rate | dominant causes |
|---|---|---|
| `debugger-status` | **37%** | METRO_NO_TARGETS /
SERVICE_INITIALIZATION_FAILED / zod |
| `debugger-connect` | **24%** | SERVICE_INITIALIZATION_FAILED (~7k, >1k
users), METRO_NO_TARGETS |
| `debugger-log-registry` | 41.7% raw / 4.2% excl. storms | one user
retry-looped it **12,742×** on NO_TARGETS |
Three design issues drive this: (1) "app/Metro not ready yet" — an
expected precondition — is reported as a tool *error*, which both
inflates fail rates and invites agent retry storms; (2) every CDP
init/lifecycle fault lands in telemetry as the catch-all
`REGISTRY_SERVICE_INITIALIZATION_FAILED` with `error_kind=unknown`; (3)
~600 users/30d hit zod validation failures with zero visibility into
which parameter was wrong.
## What
**1. Classified failures** (`d71efa43`) — six former plain-`Error` sites
in `cdp-client.ts` / `target-selection.ts` now carry dedicated
`DEBUGGER_CDP_*` / `DEBUGGER_TARGET_DEVICE_MISMATCH` signals (messages
byte-identical). The registry's two causeless "terminating window"
`ServiceInitializationError`s get their own
`REGISTRY_SERVICE_TERMINATING` code.
**2. Narrow recovery** (`ff9c1955`) — `recoverable()` on both debugger
blueprints, scoped to windows where retry is provably safe: Metro
recovers only `DEBUGGER_CDP_NOT_CONNECTED` (request never left the host;
a detected socket death already self-heals for the *next* call via the
terminated cascade — recovering `CONNECTION_CLOSED` there would be dead
code, see the blueprint comment). Chromium also recovers
`CONNECTION_CLOSED` for the tab-switch reconnect window.
`REQUEST_TIMEOUT` stays unrecoverable (double-execution risk). A
reachability test proves the retry actually fires for a RUNNING node.
**3. Structured not-connected results** (`48b91ffd`) — `debugger-status`
and `debugger-log-registry` become factories that return `{ status:
"not_connected", reason, detail, guidance }` for classified
preconditions (`metro_not_running | no_app_connected | device_mismatch |
cdp_unreachable | stale_connection | reconnecting`) and still throw on
unexpected faults. `detail` preserves the original error text for
string-matching agents. `debugger-status` gains a socket-state gate
(Metro: dispose stale node; Chromium: report transient reconnect, no
dispose — the client belongs to the ChromiumCdp dependency).
log-registry keeps **no** gate on purpose: captured logs are readable
over a dead socket, and disposal would destroy the post-crash log file.
Flows/run-sequence map a not_connected result to a **failed step**
(mirrors `isUnmetUiWaitResult`) so recorded connectivity gates don't
silently pass. `debugger-evaluate` re-codes agent-expression throws as
`DEBUGGER_EVALUATE_EXPRESSION_THREW`.
**4. Telemetry + guidance + docs** (`7602a70d`) —
`tool:fail.invalid_params` (schema-declared names only; unknown keys →
literal `"unrecognized_keys"`; capped emit-side at 16), new
`debugger:tool_outcome` event (coded enum; joins
`tool:invoke`/`complete` via `tool_invocation_id` —
`ai_client`/`duration` live on the joined row), anti-retry-loop guidance
appended to the Metro discovery errors (first sentences verbatim —
skills match on them), six skills updated, Telemetry.md bumped to v1.03
with the new coded diagnostics disclosed.
## Dashboard migration note
`debugger-status` / `debugger-log-registry` precondition failures stop
emitting `tool:fail` — track them via `debugger:tool_outcome` (`outcome
!= 'connected'`). Per-AI-client breakdowns of those outcomes need a join
on `tool_invocation_id` against `tool:complete`.
## Testing
- Full suites green: tool-server 3,142, telemetry 290, registry 87. New
coverage: recoverable truth tables (incl. non-Error inputs), status
not-connected matrix
(Metro/Chromium/terminating-window/stale/dispose-side-effect/fall-through-rethrow),
log-registry rows, zod `invalid_params` derivation + emission-level
guard, flow gate, evaluate rewrap, sanitize-level enum checks.
- Live-tested against a booted iOS simulator + RN app and an Electron
target (details in PR comments).
Plan hardened through 3 adversarial review rounds (46 confirmed findings
addressed) before implementation.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
291 lines
11 KiB
TypeScript
291 lines
11 KiB
TypeScript
import { describe, it, expect, beforeEach, vi } from "vitest";
|
|
import request from "supertest";
|
|
import { Registry } from "@argent/registry";
|
|
import { z } from "zod";
|
|
|
|
// PATH deps (xcrun) resolve through `commandOnPath`; mock it so the gate test
|
|
// is platform-agnostic rather than asserting a `/bin/sh` call shape.
|
|
const commandOnPathMock = vi.fn();
|
|
vi.mock("../src/utils/command-on-path", () => ({
|
|
commandOnPath: (name: string) => commandOnPathMock(name),
|
|
}));
|
|
|
|
// `probe()` special-cases adb/emulator to use `resolveAndroidBinary`
|
|
// (which adds an `$ANDROID_HOME` fallback to PATH). Mock the resolver so
|
|
// each test controls availability per-dep instead of fighting the host's
|
|
// real $ANDROID_HOME.
|
|
const resolveAndroidBinaryMock = vi.fn();
|
|
vi.mock("../src/utils/android-binary", () => ({
|
|
resolveAndroidBinary: (name: "adb" | "emulator") => resolveAndroidBinaryMock(name),
|
|
__resetAndroidBinaryCacheForTesting: () => {},
|
|
}));
|
|
|
|
import { createHttpApp } from "../src/http";
|
|
import {
|
|
DependencyMissingError,
|
|
__resetDepCacheForTests,
|
|
ensureDep,
|
|
} from "../src/utils/check-deps";
|
|
import { InvalidToolInputError } from "../src/utils/capability";
|
|
|
|
function stubProbe(missing: readonly string[]): void {
|
|
commandOnPathMock.mockImplementation(async (name: string) =>
|
|
missing.includes(name) ? null : `/usr/bin/${name}`
|
|
);
|
|
resolveAndroidBinaryMock.mockImplementation(async (name: string) => {
|
|
return missing.includes(name) ? null : `/usr/bin/${name}`;
|
|
});
|
|
}
|
|
|
|
describe("http dependency gate", () => {
|
|
beforeEach(() => {
|
|
__resetDepCacheForTests();
|
|
commandOnPathMock.mockReset();
|
|
resolveAndroidBinaryMock.mockReset();
|
|
});
|
|
|
|
it("returns 424 with a pretty message when a pre-flight dep is missing", async () => {
|
|
stubProbe(["adb"]);
|
|
const registry = new Registry();
|
|
registry.registerTool({
|
|
id: "android-thing",
|
|
requires: ["adb"],
|
|
zodSchema: z.object({}),
|
|
services: () => ({}),
|
|
async execute() {
|
|
// Should never run — the dep gate blocks the request before execute.
|
|
throw new Error("execute should have been skipped");
|
|
},
|
|
});
|
|
const { app } = createHttpApp(registry);
|
|
const res = await request(app).post("/tools/android-thing").send({});
|
|
expect(res.status).toBe(424);
|
|
expect(res.body.error).toMatch(/android-platform-tools/);
|
|
});
|
|
|
|
it("records a static failure signal when a pre-flight dep is missing", async () => {
|
|
stubProbe(["adb"]);
|
|
const recordFailure = vi.fn();
|
|
const registry = new Registry();
|
|
registry.registerTool({
|
|
id: "android-thing",
|
|
requires: ["adb"],
|
|
zodSchema: z.object({}),
|
|
services: () => ({}),
|
|
async execute() {
|
|
throw new Error("execute should have been skipped");
|
|
},
|
|
});
|
|
const { app } = createHttpApp(registry, { recordFailure });
|
|
const res = await request(app).post("/tools/android-thing").send({});
|
|
expect(res.status).toBe(424);
|
|
expect(recordFailure).toHaveBeenCalledWith(
|
|
"android-thing",
|
|
{},
|
|
{
|
|
error_code: "HTTP_DEPENDENCY_PREFLIGHT_MISSING",
|
|
failure_stage: "http_dependency_preflight",
|
|
failure_area: "http",
|
|
error_kind: "dependency_missing",
|
|
},
|
|
expect.any(Number)
|
|
);
|
|
});
|
|
|
|
it("records a static failure signal when request validation fails", async () => {
|
|
stubProbe([]);
|
|
const recordFailure = vi.fn();
|
|
const registry = new Registry();
|
|
registry.registerTool({
|
|
id: "validated-thing",
|
|
zodSchema: z.object({ count: z.number() }),
|
|
services: () => ({}),
|
|
async execute() {
|
|
throw new Error("execute should have been skipped");
|
|
},
|
|
});
|
|
const { app } = createHttpApp(registry, { recordFailure });
|
|
const res = await request(app).post("/tools/validated-thing").send({ count: "nope" });
|
|
expect(res.status).toBe(400);
|
|
expect(recordFailure).toHaveBeenCalledWith(
|
|
"validated-thing",
|
|
// Schema-declared names of the failing params only — never values.
|
|
{ invalid_params: ["count"] },
|
|
{
|
|
error_code: "HTTP_ZOD_VALIDATION_FAILED",
|
|
failure_stage: "http_zod_validation",
|
|
failure_area: "http",
|
|
error_kind: "validation",
|
|
},
|
|
expect.any(Number)
|
|
);
|
|
expect(JSON.stringify(recordFailure.mock.calls)).not.toContain("Expected number");
|
|
});
|
|
|
|
it("reports strict-schema unknown keys as the literal token, never the user-typed names", async () => {
|
|
stubProbe([]);
|
|
const recordFailure = vi.fn();
|
|
const registry = new Registry();
|
|
registry.registerTool({
|
|
id: "strict-thing",
|
|
zodSchema: z.object({ port: z.number() }).strict(),
|
|
services: () => ({}),
|
|
async execute() {
|
|
throw new Error("execute should have been skipped");
|
|
},
|
|
});
|
|
const { app } = createHttpApp(registry, { recordFailure });
|
|
const res = await request(app)
|
|
.post("/tools/strict-thing")
|
|
.send({ port: 1, secret_user_key: "hunter2" });
|
|
expect(res.status).toBe(400);
|
|
expect(recordFailure).toHaveBeenCalledWith(
|
|
"strict-thing",
|
|
{ invalid_params: ["unrecognized_keys"] },
|
|
expect.objectContaining({ error_code: "HTTP_ZOD_VALIDATION_FAILED" }),
|
|
expect.any(Number)
|
|
);
|
|
expect(JSON.stringify(recordFailure.mock.calls)).not.toContain("secret_user_key");
|
|
});
|
|
|
|
it("invokes the tool normally when declared deps are present", async () => {
|
|
stubProbe([]);
|
|
const registry = new Registry();
|
|
registry.registerTool({
|
|
id: "ios-thing",
|
|
requires: ["xcrun"],
|
|
zodSchema: z.object({}),
|
|
services: () => ({}),
|
|
async execute() {
|
|
return { ran: true };
|
|
},
|
|
});
|
|
const { app } = createHttpApp(registry);
|
|
const res = await request(app).post("/tools/ios-thing").send({});
|
|
expect(res.status).toBe(200);
|
|
expect(res.body.data).toEqual({ ran: true });
|
|
});
|
|
|
|
it("surfaces a DependencyMissingError thrown from inside execute (post-classify path) as 424", async () => {
|
|
// Two probes expected: the first stubs all missing; the second call (for
|
|
// the cross-platform tool's in-execute ensureDep) re-probes adb and finds
|
|
// it still missing. This is the cross-platform tool pattern: `requires`
|
|
// is absent so the pre-flight gate doesn't fire, and the dep check
|
|
// happens after classifyDevice has picked android.
|
|
stubProbe(["adb"]);
|
|
const registry = new Registry();
|
|
registry.registerTool({
|
|
id: "cross-platform-thing",
|
|
zodSchema: z.object({}),
|
|
services: () => ({}),
|
|
async execute() {
|
|
// Simulate the classify → ensureDep pattern used by launch-app etc.
|
|
await ensureDep("adb");
|
|
return { ran: true };
|
|
},
|
|
});
|
|
const { app } = createHttpApp(registry);
|
|
const res = await request(app).post("/tools/cross-platform-thing").send({});
|
|
expect(res.status).toBe(424);
|
|
expect(res.body.error).toMatch(/android-platform-tools/);
|
|
});
|
|
|
|
it("maps an InvalidToolInputError thrown from execute to 400, not 500", async () => {
|
|
// A tool rejecting its (well-typed) arguments — e.g. a newline / non-ASCII
|
|
// char in Android `keyboard` text, an unknown named key — is a client input
|
|
// error, not an internal fault. It reaches the dispatcher wrapped in
|
|
// ToolExecutionError, so the mapping must walk the cause chain.
|
|
stubProbe([]);
|
|
const recordFailure = vi.fn();
|
|
const registry = new Registry();
|
|
registry.registerTool({
|
|
id: "picky-input",
|
|
zodSchema: z.object({}),
|
|
services: () => ({}),
|
|
async execute() {
|
|
throw new InvalidToolInputError("that argument can't be carried out on this device");
|
|
},
|
|
});
|
|
const { app } = createHttpApp(registry, { recordFailure });
|
|
const res = await request(app).post("/tools/picky-input").send({});
|
|
expect(res.status).toBe(400);
|
|
expect(res.body.error).toBe("that argument can't be carried out on this device");
|
|
// An InvalidToolInputError thrown FROM execute() is recorded once, by the
|
|
// registry's failure listener (the error carries its own signal out of
|
|
// execute) — the HTTP layer must NOT also record it, or the failure
|
|
// double-counts. `emitHttpFailure` fires only on pre-execute HTTP-layer
|
|
// faults (zod / capability / device-resolution / dep-preflight), never on
|
|
// this execute-catch 400 path.
|
|
expect(recordFailure).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("does not call the dep probe for tools without a `requires` declaration", async () => {
|
|
stubProbe([]);
|
|
const registry = new Registry();
|
|
registry.registerTool({
|
|
id: "no-deps",
|
|
zodSchema: z.object({}),
|
|
services: () => ({}),
|
|
async execute() {
|
|
return { ran: true };
|
|
},
|
|
});
|
|
const { app } = createHttpApp(registry);
|
|
const res = await request(app).post("/tools/no-deps").send({});
|
|
expect(res.status).toBe(200);
|
|
expect(commandOnPathMock).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("prefers 424 over 400 when a dependency error is nested under an InvalidToolInputError", async () => {
|
|
// Pins the ordering invariant documented at the dispatcher's
|
|
// InvalidToolInputError → 400 check (http.ts): `findDependencyMissing`
|
|
// scans the WHOLE cause chain first, so for a chain carrying both classes
|
|
// the 424 wins. Reordering the two checks — or a throw site nesting a
|
|
// DependencyMissingError under an InvalidToolInputError — flips this test
|
|
// instead of silently flipping the status code.
|
|
stubProbe([]);
|
|
const registry = new Registry();
|
|
registry.registerTool({
|
|
id: "dual-class-error",
|
|
zodSchema: z.object({}),
|
|
services: () => ({}),
|
|
async execute() {
|
|
const err = new InvalidToolInputError(
|
|
"that argument needs a missing dependency"
|
|
) as InvalidToolInputError & { cause?: unknown };
|
|
err.cause = new DependencyMissingError(["adb"], "install adb please");
|
|
throw err;
|
|
},
|
|
});
|
|
const { app } = createHttpApp(registry);
|
|
const res = await request(app).post("/tools/dual-class-error").send({});
|
|
expect(res.status).toBe(424);
|
|
expect(res.body.error).toBe("install adb please");
|
|
});
|
|
|
|
it("still returns 424 when the DependencyMissingError is buried two levels deep in the cause chain", async () => {
|
|
// The registry wraps execute() errors in ToolExecutionError with `cause`.
|
|
// If a future middleware adds a second wrap (or something else does), a
|
|
// naive one-level `.cause` check regresses 424 → 500. Walk the chain.
|
|
stubProbe([]);
|
|
const registry = new Registry();
|
|
registry.registerTool({
|
|
id: "double-wrap",
|
|
zodSchema: z.object({}),
|
|
services: () => ({}),
|
|
async execute() {
|
|
const inner = new DependencyMissingError(["adb"], "install adb please");
|
|
const middle = new Error("outer wrap") as Error & { cause?: unknown };
|
|
middle.cause = inner;
|
|
const outer = new Error("tool failed") as Error & { cause?: unknown };
|
|
outer.cause = middle;
|
|
throw outer;
|
|
},
|
|
});
|
|
const { app } = createHttpApp(registry);
|
|
const res = await request(app).post("/tools/double-wrap").send({});
|
|
expect(res.status).toBe(424);
|
|
expect(res.body.error).toBe("install adb please");
|
|
});
|
|
});
|