Files
software-mansion__argent/scripts/pack-mcp.cjs
Paweł Fornagiel 3ccd89f333 feat: add native android profiling (#275)
# Android native profiling via in-process Perfetto WASM

Connected with:
https://github.com/software-mansion/argent-private/pull/16

## TL;DR

Adds **Android** to the native profiler (previously iOS-only) and does
it with a
brand-new engine: instead of shipping Perfetto's `trace_processor_shell`
native
binary and forking it per query, we now run **Perfetto's trace-processor
compiled
to WebAssembly, in-process** inside the tool-server.

- **One ~13 MB `trace_processor.wasm`** runs on every OS/arch — no
per-platform
  binary, no subprocess, fully offline.
- The native profiler is refactored from an iOS-only implementation into
a
**platform-dispatch facade** (`platforms/ios.ts` +
`platforms/android.ts`) over
a new **`profiler-shared/`** layer, so iOS and Android share the
aggregator,
  types, lifecycle, and time-alignment code.
- New **release/pack pipeline** fetches + sha256-verifies the WASM
bundle from
`argent-private-releases`, and bundles it (plus SQL queries and the
Perfetto
  TraceConfig) into the published `@swmansion/argent` package.

End-to-end on a 76 MB trace with 1013 jank rows: **6.3 s** total analyze
(the older per-hang-subprocess approach was projected at ~47 minutes and
never
finished within the tool-call deadline).

---

## TESTING MANUALLY

To test manually run the following packing command checked-out on this
branch:
```
npm run pack:mcp -- argent-@pFornagiel-native-android-devtools-perfetto
```

This will bundle the package with the new binaries. After this PR is
fixed and the binaries are resolved from the main release, the
`argent-@pFornagiel-native-android-devtools-perfetto` release can be
deleted.

## Why

Android needed native CPU/hang/memory profiling to reach parity with
iOS. The
natural engine is Perfetto's trace-processor, but the obvious ways to
ship it are
both bad:

- **`trace_processor_shell` binary** — per-platform (a mac-arm64 pack
shipped a
mac-arm64-only tarball), pinned an OS/arch matrix, and cost a fresh ~1.3
s
subprocess fork **per query**. With one subprocess forked per detected
hang, a
  large trace projected to ~47 minutes.
- **`get.perfetto.dev/trace_processor` Python launcher** — adds a
runtime Python 3
dependency, resolves "latest" at first run (defeats version pinning),
and makes
  the first analyze slow via a network fetch.

The WASM engine is one cross-platform artifact, in-process and
warm-cached: the
trace is parsed **once** and reused across the whole analyze run plus
every
drill-down query.

---

## The three big pieces

### 1. In-process Perfetto WASM engine

New package surface in `@argent/native-devtools-android`:

| File | Role |
| --- | --- |
| `src/wasm-trace-processor.ts` (+476) | The whole engine: boot, glue
patching, warm-cache, fault containment, chunked parse, cell decoding |
| `src/perfetto-engine.d.ts` (+49) | Hand-written types for the
version-agnostic RPC decoder (imported by path, so tsc can't see its own
`.d.ts`) |
| `src/errors.ts` (+68) | `TraceProcessorUnavailableError`
(`wasm_load_failed` / `wasm_path_invalid`) |
| `src/bundled-meta.ts` (+12) | `PERFETTO_VERSION` stamp, regenerated at
pack time |
| `src/index.ts` | Re-exports + package-root accessors
(`traceProcessorQueriesDir`, `traceConfigPath`) |

Public surface: `resolveTraceProcessorAssets()`,
`queryWarm<Row>(tracePath, sql)`,
`ensureTraceProcessorReady(tracePath)`, `disposeWarmEngine(tracePath)`.

Highlights of how it works:

- **Boots Google's prebuilt web/worker wasm under Node.** Installs a
minimal
worker scope (`self`, `WorkerGlobalScope`, `location`), reads the wasm
bytes,
patches the Emscripten glue (`useMemory64=false`, hand the bytes in
directly,
expose the bridge), and drives it over a `MessageChannel` + `EngineBase`
subclass. Glue patches are applied **by unique string anchors that throw
if they
  don't match** — the canary for glue-format drift on a Perfetto bump.
- **Warm-engine cache, one per trace path.** LRU with `MAX_WARM_ENGINES
= 3`
(bounds resident memory at ~26–76 MB trace + wasm heap each), 5-min idle
dispose
  (timer `unref()`'d), no failed-promise caching.
- **Fault containment (`fatal` promise).** `EngineBase.fail()` throws
synchronously
inside the `MessagePort.onmessage` handler (e.g. an RPC framing error);
left
unhandled it becomes a Node `uncaughtException` and would **kill the
entire
tool-server**, wiping every device + profiler session. Each engine
exposes a
`fatal: Promise<never>`; the handler try/catches, rejects `fatal`,
evicts the
engine, and every load/query is `Promise.race`'d against it — so a
contained
  fault is a clean rejection and the server stays up.
- **Chunked trace parse.** Perfetto's RPC ring buffer rejects any frame
> 64 MiB,
  so the trace is fed in 32 MiB chunks before `notifyEof()`.

### 2. Android native profiler + the shared/dispatch refactor

**Capture** (`utils/android-profiler/capture.ts`, `detect-app.ts`):
spawns
`adb shell perfetto --txt -c - --background-wait -o
/data/.../argent-<ts>.pftrace`,
piping the TraceConfig on **stdin** (SELinux denies `shell:s0` writes
under
`/data/misc/perfetto-traces/`, so the config can't be staged as a file),
waits for
the PID on stdout, then on stop sends `kill -TERM`, polls `/proc/<pid>`,
and
`adb pull`s the trace.

**Pipeline** (`utils/android-profiler/pipeline/`): PerfettoSQL does the
parsing and
aggregation, so the Android pipeline is intentionally **two files, not
iOS's four**.
The per-hang annotation pass is **batched** — `runBatchedHangFolds`
inlines all
hang windows as a `VALUES` table into one SQL script (two derived views
unioned in
a single terminal SELECT, because the engine returns only the final
statement's
rows) and runs it once against the warm engine.

**The platform-dispatch refactor** —
`native-profiler-start/stop/analyze` shrank
from monolithic iOS files (~160–330 lines each) into **thin routers**
(~60–110
lines) that branch on `api.platform`:

```
native-profiler-start ─┬─ ios?    → platforms/ios.ts      (xctrace)
                        └─ android → platforms/android.ts  (adb perfetto)
```

A new **`utils/profiler-shared/`** layer holds everything both backends
use:

| File | What was hoisted |
| --- | --- |
| `types.ts` | Unified `Bottleneck = CpuHotspot \| UiHang \| MemoryLeak
\| MemoryRssGrowth`, each now carrying a `platform` field; shared
`NativeProfilerAnalyzeResult`; `RECORDING_CAP_MS` |
| `aggregate.ts` | Generic `aggregateCpuHotspots(AggregatorInputRow[])`
— burst windowing (`BURST_GAP_MS=500`), severity banding (RED >15%,
YELLOW 3–15%, <3% dropped). iOS feeds it via a pre-pass; Android feeds
it precomputed SQL rows |
| `lifecycle.ts` | `shutdownChild` SIGINT→SIGTERM→SIGKILL ladder +
`waitForChildExit` (no PID-reuse hazard) |
| `thread.ts` | `normalizeThreadName` for both xctrace and Perfetto
thread names |
| `time-align.ts` | New `buildPerfettoAnchor()` + `windowsOverlap()`
alongside the existing iOS/React anchors |
| `format.ts` | `formatBytes` (moved out of the combined report) |

iOS was refactored to fit this shape: `ios-profiler/lifecycle.ts` and
`types.ts`
become thin re-exports; `pipeline/02-aggregate.ts` shrank from ~200 to
~60 lines
(keeps its iOS-only dominant-function pre-pass, delegates
grouping/bursting/
severity to the shared aggregator). **No iOS behavior change** — same
outputs,
shared internals.

**Session blueprint** (`native-profiler-session.ts`): generic
`capturePid` /
`captureProcess` (was `xctracePid` / `xctraceProcess`), new `platform`
and
`androidOnDeviceTracePath` fields, and **platform-branched dispose**
(iOS SIGKILLs
xctrace; Android `kill -KILL`s perfetto, `rm`s the device trace, and
disposes the
warm WASM engine).

**Drill-down tools**: `profiler-load.ts` learns to restore Android
`.pftrace`
sessions (with an optional metadata sidecar / `app_process` override);
`profiler-stack-query.ts` splits into iOS and Android branches. Unlike
iOS (which
caches `parsedData` in memory), Android **re-queries the `.pftrace`**
per
drill-down — the warm engine is the cache, so each query costs only SQL
execution
time.

### 3. Packaging & release pipeline

The WASM bundle's three third-party artifacts (`trace_processor.wasm`,
`engine_bundle.node.js`, `engine.mjs`) + `LICENSE` are **never
committed**
(`.gitignore` lists the generated files but not the dir). They're built
and
sha256-checksummed in `argent-private` CI, published to
`argent-private-releases`, and on our side:

- **`scripts/download-trace-processor.sh`** (+88, new) — `gh release
download` →
verify tarball sha256 (**FATAL on mismatch** — the blobs are unsigned) →
extract
  → re-verify via `SHA256SUMS` → assert all four present.
- **`scripts/pack-mcp.cjs`** (+60, new) — single pack entry point:
downloads
simulator-server, native binaries, **and** the trace-processor bundle,
then
builds and `npm pack`s `@swmansion/argent`. `package.json`'s `pack` /
`pack:mcp`
  now call this script.
- **`packages/argent/scripts/bundle-tools.cjs`** (heavily reworked,
+574/-...) —
copies `assets/trace-processor/`, `assets/queries/`, and
`argent.tracecfg.pbtxt`
into `packages/argent/assets/`, and regenerates `bundled-meta.ts` to
stamp the
  Perfetto version.
- **`publish.yml` / `publish-next.yml`** — run the download step and
then **verify**:
  all four WASM files present, the wasm magic-byte check
(`head -c4 … | grep $'\x00asm'`), plus the queries dir, TraceConfig,
manifest,
  and Android helper APK.

Asset reorg: `native-devtools-android` now ships `dist/`, `bin/`, and
`assets/`
(SQL queries, TraceConfig, manifest moved under `assets/`); `.gitignore`
swaps the
old per-file `packages/argent/manifest.json` entry for the whole
`packages/argent/assets/` dir.

---

## Testing

15 new test files under `packages/tool-server/test/android-perfetto/`
cover the
engine and pipeline, including:

- `dispatch.test.ts` — start/stop/analyze route to the right platform
backend.
- `aggregate-shared.test.ts` — the shared aggregator on both iOS and
Android-shaped input.
- `hang-folds-batched.test.ts` / `hang-fold.test.ts` — the batched
per-hang SQL fold.
- `hang-severity.test.ts` — jank-reason → RED/YELLOW classification.
- `dispose.test.ts` / `dispose-warm-engine.test.ts` — session dispose
kills the daemon, rms the device trace, and tears down the warm engine.
- `stop-recovery.test.ts` — `profilingActive` is forced false even if
`adb pull` fails (so a retry-start doesn't wedge).
- `run-tp.test.ts` — `renderSqlTemplate` token rendering + the "unused
substitution" guard.
- `manifest-hint.test.ts` — zero-callstack detection emits the manifest
hint.
- `profiler-load.test.ts` — `.pftrace` session restore with/without the
metadata sidecar.

Plus `native-profiler-analyze-failure.test.ts` for the
engine-unavailable banner,
and updated iOS tests for the shared-lifecycle refactor.
`@argent/native-devtools-android` gains its own vitest setup
(`vitest.config.ts`, `tsconfig.test.json`, `test/errors.test.ts`).

---

## Notes & follow-ups

- **Manifest requirement.** Perfetto's `linux.perf` data source needs
`/proc/<pid>/mem` access, granted only to a **debuggable** or
`<profileable
android:shell="true"/>` app. A release build without either silently
produces
zero callstacks; the pipeline detects this and emits a **manifest hint**
instead
  of an "All clear" report.
- **Version pinning lives in the `argent-private` submodule**
(`PERFETTO_VERSION`
/ `PERFETTO_WASM_TAG` / `LYNX_TRACE_PROCESSOR_VERSION`). Bumping is
documented in
`argent-private/docs/ANDROID_PERFETTO.md`. JS↔engine version skew is
structurally
  impossible (version-agnostic decoder + vendored wasm).
- **`disposeWarmEngine` is exported but not wired into session-stop** —
warm
engines are reclaimed only by the idle timer + LRU cap. Acceptable today
(bounded
memory); wire it into session dispose if explicit teardown is ever
needed.
- **Submodule bump.** This PR moves `packages/argent-private` to
`22e55f90` — see the companion `argent-private` PR description,
including its
merge-hygiene note (that branch should pick up latest `argent-private`
main
  before the pointer is finalized).

### Companion docs added in this branch
- `utils/android-profiler/ANDROID_PROFILER_REFERENCE.md` — stack,
capture, queries, manifest requirement.
- `utils/android-profiler/PIPELINE_DESIGN.md` — the "why" (two-file
pipeline, shared-aggregator hoist, re-query-don't-cache, batched fold).

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 16:06:24 +02:00

61 lines
2.5 KiB
JavaScript
Executable File
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
#!/usr/bin/env node
// @ts-check
"use strict";
// Wraps the @swmansion/argent pack pipeline so callers can pin which
// argent-private-releases tag the native binaries come from:
//
// npm run pack:mcp # default: argent-main
// npm run pack:mcp -- argent-v0.7.1 # specific release tag
// npm run pack:mcp -- argent-my-branch # tag produced by triggering
// # build-native-binaries.yml on a
// # branch
//
// Only `download-native-binaries.sh` takes the tag; simulator-server lives in
// a separate release stream with its own default and is left untouched.
const { spawnSync } = require("node:child_process");
const path = require("node:path");
const WORKSPACE_ROOT = path.resolve(__dirname, "..");
// First positional after `--` becomes the tag. Anything else is rejected so
// typos like `npm run pack:mcp -- --tag=foo` don't silently get ignored.
const args = process.argv.slice(2);
if (args.length > 1) {
console.error(`Unexpected extra arguments: ${args.slice(1).join(" ")}`);
console.error("Usage: npm run pack:mcp -- [argent-release-tag]");
process.exit(1);
}
const tag = args[0];
// The Android trace-processor engine ships as three third-party WASM artifacts
// (trace_processor.wasm + emscripten glue + the EngineBase decoder) plus the
// Perfetto LICENSE. They are NOT committed to the repo; download-trace-processor.sh
// fetches and sha256-verifies them from argent-private-releases at pack time
// (same tag as the native binaries) so they land in the published tarball under
// packages/native-devtools-android/assets/trace-processor/. download-native-binaries.sh
// still fetches the iOS binaries and the Android helper APK.
/** @param {string} cmd @param {string[]} cmdArgs */
function run(cmd, cmdArgs) {
const display = [cmd, ...cmdArgs].join(" ");
console.log(`\n› ${display}`);
const result = spawnSync(cmd, cmdArgs, {
cwd: WORKSPACE_ROOT,
stdio: "inherit",
shell: false,
});
if (result.status !== 0) {
console.error(`\nCommand failed (${result.status}): ${display}`);
process.exit(result.status ?? 1);
}
}
run("bash", ["scripts/download-simulator-server.sh"]);
run("bash", ["scripts/download-native-binaries.sh", ...(tag ? [tag] : [])]);
run("bash", ["scripts/download-trace-processor.sh", ...(tag ? [tag] : [])]);
run("npm", ["run", "build"]);
run("npm", ["run", "build", "-w", "@swmansion/argent"]);
run("npm", ["pack", "-w", "@swmansion/argent", "--pack-destination", "."]);