Hubert Gancarczyk ef955e604d fix(telemetry): add missing failure codes (#420) ...

## Summary

Adds the missing **failure codes** across the tool-server and converts
raw
`throw new Error(...)` sites into classified `FailureError`s, so
operational
failures land in named telemetry buckets instead of falling through to
the
generic `REGISTRY_TOOL_FAILURE_UNCLASSIFIED` bucket.

- **39 new `FAILURE_CODES`** plus a new `"electron"` entry in
`FAILURE_COMMANDS`.
- Throw sites migrated to `FailureError`, each carrying
`error_code` + `failure_stage` + `failure_area` + `error_kind`, and
where
  relevant `failure_command` / `network_failure` / subprocess metadata
  (`failure_exit_code` / `failure_signal` / `failure_spawn_code`).

## Addressing review feedback

The review found that several proposed codes were thrown only on paths
that are
caught / reformatted before the telemetry boundary (a code only buckets
if its
`FailureError` propagates out of a registered tool's `execute`), so they
could
never bucket a real failure. Those are removed rather than left as dead
codes:

- **Dropped
`VEGA_PAGE_SOURCE_{FETCH_FAILED,HTTP_ERROR,RPC_ERROR,TIMEOUT,PARSE_FAILED}`**
- `describeVega` deliberately swallows every toolkit failure (empty tree
+
  relaunch hint) except `MultipleVegaDevicesError`, so these never reach
`toolFailed`. The throw sites revert to plain `Error` (behaviour
unchanged).
- **Dropped `CHROMIUM_CLIPBOARD_FAILED`** (the chromium-server Express
route
reformats the error to a 500 body; the image-copy path has no production
  caller) and **`CHROMIUM_BUTTON_UNSUPPORTED`** (the WS `button` handler
  reformats it; the registered `button` tool is apple+android only).
- **Fixed the `CHROMIUM_CDP_INVALID_RESPONSE` gap** - a reachable
HTTP-200 with a
non-JSON body made `res.json()` reject with a raw `SyntaxError` that
escaped
`fetchJson` unclassified; it is now classified
`CHROMIUM_CDP_INVALID_RESPONSE`
(matching the non-2xx and missing-`webSocketDebuggerUrl` paths).
`AbortError`
  still surfaces untouched.
- **Unified two codes that carried two `error_kind`s** - both
`REACT_PROFILER_NO_ACTIVE_SESSION` sites and both
`PROFILER_NATIVE_TRACE_MISSING`
sites now use `not_found` (a missing session / a trace file missing on
disk),
  keeping telemetry keyed on `error_code` from splitting by platform.
- **Reverted an out-of-scope edit** - the in-flight Vega-boot coalescing
Map-key
  separator is back to its original NUL byte.
- **Added `failure-classification.test.ts` coverage** - expanded from 7
to 17
  pinned codes, reading each back exactly how telemetry does
(`getFailureSignal(err).error_code`) across the chromium CDP discovery
family
(incl. the malformed-200 regression guard), vega input, and
native-profiler
  metadata / stack-query.

This branch also merges the latest `main`; the keyboard tool was
refactored on
`main` into per-platform modules + a `createKeyboardTool` factory, so
the
`KEYBOARD_KEY_UNSUPPORTED` / `KEYBOARD_CHARACTER_UNSUPPORTED`
classifications were
re-homed onto its new throw sites (`platforms/chromium.ts`,
`simulator-server-keys.ts`; the vega path keeps its classification in
`vega-input.ts`).

## What changed

By area (the 39 codes):

- **Chromium / CDP (16)** - discovery, input, viewport, storage, tabs,
describe,
and Electron boot: `CHROMIUM_CDP_UNREACHABLE`,
`CHROMIUM_CDP_NO_PAGE_TARGET`,
  `CHROMIUM_CDP_INVALID_RESPONSE`, `CHROMIUM_PARAM_INVALID`,
  `CHROMIUM_INPUT_INVALID`, `CHROMIUM_VIEWPORT_READ_FAILED`,
  `CHROMIUM_SCREENSHOT_FAILED`, `CHROMIUM_STORAGE_EVAL_FAILED`,
`CHROMIUM_TAB_NOT_FOUND`, `CHROMIUM_TAB_OPEN_FAILED`,
`CHROMIUM_DESCRIBE_FAILED`,
  `CHROMIUM_DEVICE_ID_INVALID`, `CHROMIUM_ELECTRON_APP_PATH_INVALID`,
  `CHROMIUM_ELECTRON_SPAWN_FAILED`, `CHROMIUM_ELECTRON_CDP_TIMEOUT`,
  `CHROMIUM_ELECTRON_EXITED_BEFORE_READY`.
  - Unreachable debug ports are distinguished from reached-but-malformed
responses; the `undici`-wrapped `ECONN*` code is read off `err.cause`
and
    mapped to a precise `network_failure` instead of always reporting
    `connection_refused`.
- **Vega (8)** - device discovery/registration, boot timeouts, image
lookup,
input, and screenshot: `VEGA_DEVICE_NOT_FOUND`,
`VEGA_DEVICE_NOT_REGISTERED`,
  `VEGA_MULTIPLE_DEVICES`, `VEGA_BOOT_TIMEOUT`, `VEGA_IMAGE_NOT_FOUND`,
  `VEGA_ALREADY_RUNNING`, `VEGA_SCREENSHOT_FAILED`, `VEGA_TEXT_INVALID`.
- `MultipleVegaDevicesError` extends `FailureError` so callers that
re-throw it
    (e.g. `describe`) surface a classified signal.
- **Native profiler (8)** - perfetto start failures (mirroring the iOS
xctrace
set), app-process detection, the metadata sidecar, and the shared
analyze
  guard: `NATIVE_PROFILER_PERFETTO_PROCESS_ERROR`,
  `NATIVE_PROFILER_PERFETTO_READY_TIMEOUT`,
  `NATIVE_PROFILER_PERFETTO_READY_EXITED`,
`NATIVE_PROFILER_APP_PROCESS_NOT_FOUND`,
`NATIVE_PROFILER_NO_EXPORTED_TRACE`,
`NATIVE_PROFILER_TRACE_TEMPLATE_MISSING`,
`PROFILER_NATIVE_METADATA_MISSING`,
  `PROFILER_NATIVE_METADATA_INVALID`.
- The iOS `analyze` guard moves off `PROFILER_NATIVE_TRACE_MISSING` onto
the
shared `NATIVE_PROFILER_NO_EXPORTED_TRACE` so the same logical failure
isn't
split by platform; `PROFILER_NATIVE_TRACE_MISSING` now means strictly a
trace
file missing on disk (and both of its sites use `error_kind:
not_found`).
- **Android / simulator / screenshot-diff (3)** -
  `ANDROID_UIAUTOMATOR_CAPTURE_FAILED`, `SIMULATOR_SCREENSHOT_FAILED`,
  `SCREENSHOT_DIFF_INPUT_INVALID`.
- **Flows / keyboard (4)** - `FLOW_PROJECT_ROOT_INVALID` (split from
`_REQUIRED`,
  which now means strictly "no root set yet"), `FLOW_NAME_INVALID`,
  `KEYBOARD_KEY_UNSUPPORTED`, `KEYBOARD_CHARACTER_UNSUPPORTED`.

`error_kind` is reserved consistently: `validation` for checks against
caller
input or a schema we own; `unknown` for malformed output from an
uncontrolled
source (renderer main world, toolkit RPC); `network` with a
`network_failure`
class for transport and reached-but-malformed responses; `not_found` for
a
missing resource / session / on-disk artifact.

## Testing

- `tsc --build` clean - `error_kind` / `failure_area` /
`failure_command` /
`network_failure` and every `FAILURE_CODES.X` are typed, so the compiler
  enforces structural validity of every signal.
- Unit tests: registry **83/83**, tool-server green (the one red
`await-ui-element` case is a pre-existing local-only timing flake
unrelated to
  this change - it fails identically on `main` and passes in CI).
- `failure-classification.test.ts` pins **17** codes, reading each back
via
`getFailureSignal(err).error_code`; the `CHROMIUM_CDP_INVALID_RESPONSE`
  malformed-200 fix is reproduced against a local HTTP server.
- `prettier --check` and `eslint` clean on all changed files.

## Notes

- No new codes are orphaned - every added `FAILURE_CODES` entry has at
least one
  throw site that can reach the telemetry boundary.
- A handful of internal-invariant throws (SQL-template/identifier
guards,
Tesseract-TSV parsing, unreachable `default:` branches) are
intentionally left
  as plain `Error`s - they represent programmer/tooling errors, not
  user-reachable failure modes, so they don't warrant telemetry codes.

2026-07-03 16:56:15 +02:00

13 KiB

Raw Permalink Blame History

View Raw