mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
2196cc18f0
## Why
PostHog shows **~9,785 `toolserver:stop` crashes over 90 days**, but
every one collapses into a single bucket — `error_kind=crash`,
`error_code=TOOLSERVER_UNCAUGHT_EXCEPTION`. We know *that* the tool
server died on a top-level throw, never *which* throw. The population is
also two very different things we can't tell apart today:
- **~99%** crash at **<5s uptime with 0 tool calls** — startup faults,
and heavily concentrated per-user (top 10 users ≈ 31% of all crashes;
one user logged 1,027), i.e. **boot → crash → relaunch loops**.
- **~1,000** crash after **>10 min** of real serving — genuine runtime
bugs.
Without the actual exception, all 9,785 rows are unrankable and
unfixable.
## What
Four **crash-only, provably-anonymous** fields on `toolserver:stop`,
enforced by the existing coded-allowlist sanitizer so **no free text can
ever reach PostHog**:
| field | value | why |
|---|---|---|
| `error_name` | error class name (`TypeError`, …) | "what kind" |
| `error_syscall` | Node's `err.code` (`EADDRINUSE`, `ECONNREFUSED`, …)
| the key discriminator for the startup-loop population (port/socket
conflicts) |
| `crash_fingerprint` | 16-hex hash of the **de-identified** top stack
frames | clusters the same crash across users; input carries no
identity, output is opaque |
| `crash_phase` | `"startup"` \| `"serving"` | splits the two
populations directly |
Turns 9,785 opaque rows into a handful of rankable clusters.
## Anonymity (the hard requirement)
- **No error message, no raw stack.** Those routinely embed
`/Users/<name>/…`, URLs, argv, hostnames, interpolated values — the
highest-PII surface and impossible to reliably scrub. The fingerprint
replaces the message for clustering; `error_name` + `error_syscall` give
the "what kind" without free text.
- **Fingerprint input is de-identified before hashing**
(defense-in-depth on top of the hash): first-party frames → basename
only (drops the home-dir/project prefix), `node_modules` frames →
package-relative tail (drops the user prefix; also collapses the pnpm
`.pnpm/pkg@ver/` layout), `node:` internals kept verbatim, column
dropped. So the *same crash on two machines produces the same
fingerprint*, and no username/path survives even into the hash input.
- **Sanitizer is the final gate.** `error_name`
`^[A-Za-z][A-Za-z0-9_]{0,63}$`, `error_syscall` `^E[A-Z0-9_]{1,31}$`,
`crash_fingerprint` `^[0-9a-f]{16}$`, `crash_phase` a 2-value enum. Any
path/message/lowercase/oversize value is dropped before it can leave the
machine.
## Scope & safety (no leaking crashes out of scope)
- Fields are populated **only** in `crashShutdown` (which forces
`reason="crash"`), so clean idle/signal stops carry none. Verified by
the existing startup-telemetry test (the pre-listener watcher-rejection
crash path stays the bare `{reason,uptime_ms,total_tool_calls}` shape).
- The four keys exist only on `ToolserverStopProps` and only in the
`toolserver:stop` allowlist entry — they can't ride any other event.
- `describeCrash` is **total**: every field extraction is isolated, so a
hostile throwing getter on `.name`/`.code`/`.stack` yields the phase
(and whatever else is readable) instead of turning crash reporting into
a second crash. The crash/exit guarantee — re-entrancy guard, force-exit
timer, `finalExitCode` escalation — is untouched.
## Tests
- New `crash-diagnostics.test.ts`: class-name/constructor-name/primitive
extraction, syscall capture, cross-user determinism (incl. **Windows**
drive-letter paths and **pnpm** layout), node_modules tail,
line-sensitivity, no-stack omission, hostile-getter resilience, and
"never emits a message/stack" confinement.
- `sanitize.test.ts`: crash fields pass when well-formed; dropped for
message-as-name, path/lowercase syscall, wrong-length/uppercase
fingerprint, unknown phase; `ERR_*` accepted.
- Full telemetry suite (278) + tool-server startup test green;
lint/prettier clean.
## Reviewed
Ran an adversarial review pass (anonymity leak-hunt, determinism,
crash-path safety, scope). Ship-ready; all findings were Low/Nit and the
actionable ones (hostile-getter hardening, Windows/pnpm test coverage,
`error_syscall` doc clarity) are folded in.
### Deliberately deferred (documented, not blockers)
- Sanitizer doesn't *cross-gate* the crash fields on `reason==="crash"`
— the per-field validator design has no sibling access; coupling is
enforced in emit code. The fields are non-identifying regardless, so a
hypothetical stray `crash_phase` on an idle stop is a data nit, not a
leak.
- First-party frames reduce to basename; if `dist` ever ships unbundled
with duplicate basenames, two sites at the same line could collide.
Non-issue for the bundled tool-server.
172 lines
7.6 KiB
TypeScript
172 lines
7.6 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
|
import { describeCrash } from "../src/crash-diagnostics.js";
|
|
|
|
/** Build an Error with a fully controlled stack so frame parsing is exercised. */
|
|
function errorWithStack(message: string, stackLines: string[]): Error {
|
|
const err = new Error(message);
|
|
err.stack = [`Error: ${message}`, ...stackLines].join("\n");
|
|
return err;
|
|
}
|
|
|
|
describe("describeCrash", () => {
|
|
describe("crash_phase", () => {
|
|
it("passes the phase through verbatim", () => {
|
|
expect(describeCrash(new Error("x"), "startup").crash_phase).toBe("startup");
|
|
expect(describeCrash(new Error("x"), "serving").crash_phase).toBe("serving");
|
|
});
|
|
});
|
|
|
|
describe("error_name", () => {
|
|
it("reads the class name of a built-in error subclass", () => {
|
|
expect(describeCrash(new TypeError("bad"), "startup").error_name).toBe("TypeError");
|
|
expect(describeCrash(new RangeError("bad"), "startup").error_name).toBe("RangeError");
|
|
});
|
|
|
|
it("reads the constructor name of a thrown plain object", () => {
|
|
class CustomFailure {}
|
|
expect(describeCrash(new CustomFailure(), "serving").error_name).toBe("CustomFailure");
|
|
});
|
|
|
|
it("is omitted for a thrown primitive (no class to name)", () => {
|
|
expect(describeCrash("boom", "startup").error_name).toBeUndefined();
|
|
expect(describeCrash(undefined, "startup").error_name).toBeUndefined();
|
|
});
|
|
});
|
|
|
|
describe("error_syscall", () => {
|
|
it("captures a Node system-error code", () => {
|
|
const err = Object.assign(new Error("bind EADDRINUSE 127.0.0.1:3001"), {
|
|
code: "EADDRINUSE",
|
|
syscall: "listen",
|
|
});
|
|
expect(describeCrash(err, "startup").error_syscall).toBe("EADDRINUSE");
|
|
});
|
|
|
|
it("is omitted when there is no code", () => {
|
|
expect(describeCrash(new Error("plain"), "startup").error_syscall).toBeUndefined();
|
|
});
|
|
});
|
|
|
|
describe("crash_fingerprint", () => {
|
|
it("is 16 lowercase hex chars for a stack with frames", () => {
|
|
const err = errorWithStack("boom", [
|
|
" at Server.<anonymous> (/Users/alice/project/dist/index.js:120:15)",
|
|
]);
|
|
expect(describeCrash(err, "startup").crash_fingerprint).toMatch(/^[0-9a-f]{16}$/);
|
|
});
|
|
|
|
it("is identical across users — the home-dir prefix and message are excluded", () => {
|
|
// Same crash on two machines: only the absolute path prefix and the
|
|
// PII-bearing message differ. The fingerprint must not.
|
|
const alice = errorWithStack("failed for /Users/alice/secret.txt", [
|
|
" at Server.<anonymous> (/Users/alice/project/dist/index.js:120:15)",
|
|
" at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
|
|
]);
|
|
const bob = errorWithStack("failed for /home/bob/other.txt", [
|
|
" at Server.<anonymous> (/home/bob/app/dist/index.js:120:15)",
|
|
" at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
|
|
]);
|
|
const fpA = describeCrash(alice, "startup").crash_fingerprint;
|
|
expect(fpA).toBeDefined();
|
|
expect(describeCrash(bob, "startup").crash_fingerprint).toBe(fpA);
|
|
});
|
|
|
|
it("keeps the package-relative tail of node_modules frames (drops the user prefix)", () => {
|
|
const alice = errorWithStack("x", [
|
|
" at Layer.handle (/Users/alice/proj/node_modules/express/lib/router/layer.js:95:5)",
|
|
]);
|
|
const bob = errorWithStack("x", [
|
|
" at Layer.handle (/opt/ci/build/node_modules/express/lib/router/layer.js:95:5)",
|
|
]);
|
|
expect(describeCrash(bob, "startup").crash_fingerprint).toBe(
|
|
describeCrash(alice, "startup").crash_fingerprint
|
|
);
|
|
});
|
|
|
|
it("de-identifies Windows paths (drive letter + backslashes)", () => {
|
|
const alice = errorWithStack("x", [" at f (C:\\Users\\alice\\app\\dist\\index.js:12:34)"]);
|
|
const bob = errorWithStack("x", [" at f (D:\\work\\bob\\build\\dist\\index.js:12:34)"]);
|
|
const fp = describeCrash(alice, "startup").crash_fingerprint;
|
|
expect(fp).toMatch(/^[0-9a-f]{16}$/);
|
|
expect(describeCrash(bob, "startup").crash_fingerprint).toBe(fp);
|
|
});
|
|
|
|
it("collapses pnpm and npm node_modules layouts to the same fingerprint", () => {
|
|
const npm = errorWithStack("x", [
|
|
" at handle (/Users/alice/proj/node_modules/express/lib/router/layer.js:95:5)",
|
|
]);
|
|
const pnpm = errorWithStack("x", [
|
|
" at handle (/home/bob/proj/node_modules/.pnpm/express@4.18.2/node_modules/express/lib/router/layer.js:95:5)",
|
|
]);
|
|
// lastIndexOf("node_modules/") keeps only the final package-relative tail,
|
|
// dropping the .pnpm version-hash segment.
|
|
expect(describeCrash(pnpm, "startup").crash_fingerprint).toBe(
|
|
describeCrash(npm, "startup").crash_fingerprint
|
|
);
|
|
});
|
|
|
|
it("ignores non-frame and anonymous lines without throwing", () => {
|
|
const err = errorWithStack("x", [
|
|
" at <anonymous>",
|
|
" at new Promise (<anonymous>)",
|
|
" at Server.<anonymous> (/app/dist/index.js:10:5)",
|
|
]);
|
|
// The two anonymous lines don't match FRAME_RE and are skipped; the real
|
|
// frame still yields a fingerprint.
|
|
expect(describeCrash(err, "startup").crash_fingerprint).toMatch(/^[0-9a-f]{16}$/);
|
|
});
|
|
|
|
it("changes when the crash site changes (different line)", () => {
|
|
const a = errorWithStack("x", [" at f (/app/dist/index.js:120:15)"]);
|
|
const b = errorWithStack("x", [" at f (/app/dist/index.js:200:15)"]);
|
|
expect(describeCrash(a, "startup").crash_fingerprint).not.toBe(
|
|
describeCrash(b, "startup").crash_fingerprint
|
|
);
|
|
});
|
|
|
|
it("is omitted when the error carries no usable stack", () => {
|
|
expect(describeCrash("just a string", "startup").crash_fingerprint).toBeUndefined();
|
|
const noFrames = errorWithStack("boom", []); // header line only, no `at` frames
|
|
expect(describeCrash(noFrames, "startup").crash_fingerprint).toBeUndefined();
|
|
});
|
|
});
|
|
|
|
describe("resilience to hostile errors", () => {
|
|
it("never throws, even when name/code/stack getters throw", () => {
|
|
// `Object.create(Error.prototype)` (not `new Error()`) keeps
|
|
// `instanceof Error` true — so errorName/stackOf take the `.name`/`.stack`
|
|
// branch and hit the throwing getters — while avoiding V8's real-Error
|
|
// lazy-stack machinery, which on some Node versions would run the `name`
|
|
// getter (and throw) during the `defineProperty(…, "stack", …)` setup
|
|
// itself, before describeCrash is ever called.
|
|
const hostile = Object.create(Error.prototype) as Record<string, unknown>;
|
|
const throwing = () => {
|
|
throw new Error("boom in getter");
|
|
};
|
|
Object.defineProperty(hostile, "name", { get: throwing });
|
|
Object.defineProperty(hostile, "code", { get: throwing });
|
|
Object.defineProperty(hostile, "stack", { get: throwing });
|
|
let result: ReturnType<typeof describeCrash>;
|
|
expect(() => {
|
|
result = describeCrash(hostile, "serving");
|
|
}).not.toThrow();
|
|
// Only the phase survives; every throwing field is dropped.
|
|
expect(result!).toEqual({ crash_phase: "serving" });
|
|
});
|
|
});
|
|
|
|
it("never emits a message or raw-stack field", () => {
|
|
const err = errorWithStack("secret /Users/alice/id_rsa", [
|
|
" at f (/Users/alice/app/index.js:1:1)",
|
|
]);
|
|
const keys = Object.keys(describeCrash(err, "startup"));
|
|
expect(keys).not.toContain("error_message");
|
|
expect(keys).not.toContain("message");
|
|
expect(keys).not.toContain("stack");
|
|
// Whole record is confined to the four coded fields.
|
|
for (const key of keys) {
|
|
expect(["error_name", "error_syscall", "crash_fingerprint", "crash_phase"]).toContain(key);
|
|
}
|
|
});
|
|
});
|