Files
software-mansion__argent/packages/tool-server/test/list-devices-deadline.test.ts
Hubert Gancarczyk 6bc5376746 fix(tool-server): stop list-devices from hanging - wedged-VVD stall and CLI socket leak (#419)
## Summary

`list-devices` (an `alwaysLoad` tool called at session start and
frequently after)
could appear to **hang**. Investigation found **two independent
causes**, both fixed here:

1. **Server-side (Vega):** a wedged Vega VVD turned `list-devices` into
a ~40s stall
   and leaked `vega` CLI subprocesses.
2. **Client-side (CLI):** every `argent run …` lingered a constant **~6s
after it had
already printed its result**, because a health-check `fetch` never
drained its
response body and kept an undici keep-alive socket ref'd until the
server closed it.

Net effect for `argent run list-devices`: **6.15s → 0.91s**, and the
worst-case
wedged-VVD stall drops from ~40s to a bounded short call.

---

## Problem 1 — wedged Vega VVD stalls `list-devices` (~40s) and leaks
processes

On a host with the Vega SDK installed and a **running-but-unresponsive
VVD**,
`list-devices`' Vega discovery path made **two serial 20s CLI calls**
that both hung:

- `vega device list` → SIGKILL at 20s
- then (recovery) `vega device info` → SIGKILL at another 20s → **~40s
total**

Three compounding defects:

- `execFileAsync` resolves on stdio **close**, not child **exit**. The
`vega` launcher
forks a `python3 → node → vda` worker tree; an orphaned worker holds the
stdout pipe
open, so the call stayed pending for the full timeout even after the
launcher was
  killed.
- The `device info` recovery fired even when `device list` had **hung**,
doubling the
  cost and spawning a second hanging subprocess.
- On timeout we killed only the **direct child**, so the `python3 → node
→ vda` worker
tree was **orphaned, not reaped** — leaking a process tree per call (a
wedged host had
  accumulated ~269 of them).

### Fix

- **`runVega` (`vega-cli.ts`):** switched from `execFileAsync` to
`spawn` with
**`detached: true`**, making the launcher its own **process-group
leader**, and on
timeout SIGKILL the **whole process group** (`process.kill(-pid)`) —
which reaps the
entire `python3 → node → vda` tree, **not just the launcher**. We also
destroy our
stdio stream ends so a pipe-holding worker can't keep the call pending.
(`execFile`
silently drops `detached`, which is why the switch to `spawn` was
required.)
- **Success path doesn't depend on stdio EOF either:** we prefer `close`
(fully
drained), but if a grandchild holds the stdout pipe open after a **clean
exit**, we
fall back to `exit` + a short (~1s) drain grace and resolve with the
captured
output — so a finished call can't be stalled into the timeout and then
rejected,
discarding valid data. The output cap is measured in real UTF-8 bytes.
For that
guarantee to hold at the boundary, the `exit` handler **disarms the main
timeout
timer**: once the child has terminated the wall-clock deadline is moot,
so a clean
exit landing within the drain grace of the deadline resolves with its
output instead
of being mis-rejected as a `timeout` (which would also wrongly suppress
the
    `listVegaDevices` recovery for a still-running VVD).
- **Belt-and-suspenders for a worker that escapes the group:** an
earlier
investigation reported the worker tree `setsid()`-ing into its own
session (which
would defeat a group-only SIGKILL). To be safe regardless of which is
true, on
timeout we **snapshot the launcher's descendants from the process table
*before*
killing** (a setsid'd worker keeps its `ppid` pointing at the launcher
until the
launcher dies, so a `ppid` tree-walk still reaches it) and SIGKILL those
pids
individually after the group kill. The snapshot is **precise** — only
this
launcher's own descendants, by pid, so a concurrent `vega` call's
workers are never
touched — and bounded, and the group kill runs regardless of whether it
succeeds.
- A real-subprocess test confirms **0 orphans** after a timeout,
including a worker
deliberately spawned into its **own** process group (so the group kill
alone cannot
    reach it — the test fails without the descendant sweep).
- **`listVegaDevices` (`vega-devices.ts`):** skip the `device info`
recovery only when
  `device list` **timed out** (a wedged agent — keyed off the failure's
`error_kind: "timeout"`), so a hung list no longer pays for a second
timeout wait. A
*fast* (non-timeout) list failure still recovers a genuinely-running VVD
rather than
mis-reporting it as stopped. Tightened discovery timeouts (20s → 6s/4s —
a generous
multiple of the ~1s healthy time so a cold-start `vega` launcher isn't
spuriously
timed out, with their sum (10s) kept under the 15s branch deadline with
margin).
- **`list-devices.ts`:** added a 15s `withDeadline` backstop around the
Android and Vega
branches so no single branch can stall the whole fan-out (returns
partial results +
logs a note). Sized to sit ~5s above the Vega branch's ~10s worst case,
so the
backstop never truncates a slow-but-completing branch (which would drop
a real
  device); an invariant test guards that ordering.
- **`readAvdName` (`adb.ts`):** run the two `getprop` probes
concurrently so a wedged
  emulator costs 5s, not 10s.

---

## Problem 2 — `argent run` hangs ~6s after printing output (CLI socket
leak)

Independent of Vega. The tool-server returns `list-devices` in <1s, but
the CLI process
took a **constant ~6s** to exit. Any path that calls `process.exit()`
was instant; any
path that returned naturally (`argent run …`, `argent run … --help`,
`argent tools`)
lingered ~6s.

**Root cause:** `isToolsServerHealthy` (`launcher.ts`) did
`const res = await fetch('/tools'); return res.ok;` and **never drained
the body**.
`/tools` is a large (~112KB) payload, so undici keeps the keep-alive
socket **ref'd**
until the tool-server's idle `keepAliveTimeout` (~5s) closes it.
`ensureToolsServer`
runs this health check on every invocation, and `run()`'s success path
has no
`process.exit(0)`, so the ref'd socket holds the event loop open.
(Confirmed
telemetry-independent — `DO_NOT_TRACK` made no difference.) The same
undrained-body
pattern existed in `link.ts` (`argent link`).

### Fix

Drain the response body before returning in both health-check-style
fetches:

```ts
await res.body?.cancel().catch(() => {});
return res.ok;
```

(`fetchTools`/`callTool` were already fine — they `await res.json()`.)

---

## Testing

**Empirical, end-to-end** (on a machine reproducing the wedged VVD):

| Command | Before | After |
| --- | --- | --- |
| `argent run list-devices` | 6.15s | **0.91s** |
| `argent run list-devices --help` | 6.14s | **0.16s** |
| `argent tools` | 6.15s | **0.16s** |
| handle probe after run | 2 sockets ref'd for ~6s | **none — immediate
exit** |

- Raw `POST /tools/list-devices` was ~0.7s throughout — confirming the
~5.4s was pure
  CLI teardown, not tool work.
- **Process-group reap proven empirically:** the live `vega → bash →
python3 → node`
tree all shares the detached launcher's PGID; a group-SIGKILL left **0
survivors**.
A real-subprocess test (unmocked `runVega` vs. a fake hanging `vega` on
`PATH`) confirms
  it settles at its deadline **and** leaves **0 orphaned processes**.

**Automated:**

- `vega-cli-timeout.test.ts` (real-subprocess test) — `runVega` rejects
on its deadline,
**reaps the entire worker tree (0 orphans)** — the fake's hung worker is
spawned into its
**own** process group, so the test fails without the descendant sweep —
and returns
normally when the CLI responds. Also covers the boundary case: a **clean
exit whose
drain grace outlasts the deadline** resolves with its captured output
rather than being
rejected as a `timeout` (guards the `exit`-handler timer disarm above).
- `vega-adb-connect-recovery.test.ts` — a *timed-out* `device list` does
not trigger a
second `device info` call (no stacking); a *fast, non-timeout* failure
**still** recovers
  a running VVD.
- `list-devices-deadline.test.ts` — `withDeadline` returns the branch
value on the happy
path and the fallback (with a logged note) when a branch exceeds the
deadline.
- `list-devices.test.ts` — added a `spawn` mock (vega moved off
`execFile`).
- `launcher-helpers.test.ts` — `isToolsServerHealthy` cancels (drains)
its response body.
- Suites green: **tool-server (1610), argent-tools-client (109),
argent-cli (84)**.
- eslint (`--max-warnings 0`), prettier, and `tsc` clean.

---
2026-07-01 11:02:33 +02:00

70 lines
3.7 KiB
TypeScript

import { describe, it, expect, vi, afterEach } from "vitest";
import { withDeadline, BRANCH_DEADLINE_MS } from "../src/tools/devices/list-devices";
import {
VEGA_DISCOVERY_LIST_TIMEOUT_MS,
VEGA_DISCOVERY_INFO_TIMEOUT_MS,
} from "../src/utils/vega-devices";
import { VVD_PS_PROBE_TIMEOUT_MS } from "../src/utils/vega-process";
import { ADB_DEVICES_TIMEOUT_MS, ENRICH_TIMEOUT_MS } from "../src/utils/adb";
// Guards the timeout/backstop ordering: the backstop must sit ABOVE every branch's
// FULL per-call worst case, or it stops being a last-resort and starts truncating a
// branch that would have completed — dropping a real device from the list. Each
// branch worst case is summed from the SAME exported timeout constants the code
// uses, so a future edit that bumps any of them past the backstop fails here loudly.
//
// The earlier version of this test counted only the two Vega *device* timeouts
// (list + info) and silently omitted the two serial `ps` probes the recovery path
// also runs — which made the asserted margin fictional. These sums now include them.
const MIN_MARGIN_MS = 3_000;
describe("discovery timeout vs backstop invariant", () => {
it("the Vega branch's full worst case stays comfortably under the branch deadline", () => {
// The recovery path runs, serially: the `device list` timeout, the recovery-gate
// `ps` probe, the `-d emulator-<port>` selector `ps` probe (inside runVegaDevice),
// and the `device info` timeout. (A *timed-out* list skips recovery, so this is the
// non-timeout-failure / empty-list ceiling — the true long pole.)
const vegaWorstCase =
VEGA_DISCOVERY_LIST_TIMEOUT_MS + 2 * VVD_PS_PROBE_TIMEOUT_MS + VEGA_DISCOVERY_INFO_TIMEOUT_MS;
expect(vegaWorstCase).toBeLessThan(BRANCH_DEADLINE_MS);
// A real margin (not just "<"), so a slightly-slow-but-completing branch on a
// loaded machine isn't cut off at the deadline.
expect(BRANCH_DEADLINE_MS - vegaWorstCase).toBeGreaterThanOrEqual(MIN_MARGIN_MS);
});
it("the Android branch's full worst case stays comfortably under the branch deadline", () => {
// One bounded `adb devices` call, then concurrent getprop enrichment (all devices
// and all per-device props run in parallel, so it caps at one ENRICH_TIMEOUT_MS).
const androidWorstCase = ADB_DEVICES_TIMEOUT_MS + ENRICH_TIMEOUT_MS;
expect(androidWorstCase).toBeLessThan(BRANCH_DEADLINE_MS);
expect(BRANCH_DEADLINE_MS - androidWorstCase).toBeGreaterThanOrEqual(MIN_MARGIN_MS);
});
});
// Unit cover for the hard backstop that keeps a single wedged discovery branch from
// stalling the `alwaysLoad` list-devices tool. The per-call subprocess timeouts are
// the first line of defence; this asserts the defence-in-depth race itself returns a
// partial result (the fallback) instead of hanging, and stays out of the way on the
// happy path.
describe("withDeadline backstop", () => {
afterEach(() => vi.useRealTimers());
it("returns the branch's value untouched when it resolves before the deadline", async () => {
await expect(withDeadline(Promise.resolve(["a", "b"]), [], "vega")).resolves.toEqual([
"a",
"b",
]);
});
it("returns the fallback (and logs) when the branch exceeds the deadline", async () => {
vi.useFakeTimers();
const stderr = vi.spyOn(process.stderr, "write").mockReturnValue(true);
const never = new Promise<string[]>(() => {}); // a wedged branch that never settles
const result = withDeadline(never, ["fallback"], "android");
await vi.advanceTimersByTimeAsync(BRANCH_DEADLINE_MS);
await expect(result).resolves.toEqual(["fallback"]);
expect(stderr).toHaveBeenCalledWith(expect.stringContaining("android discovery exceeded"));
stderr.mockRestore();
});
});