# Android native profiling via in-process Perfetto WASM Connected with: https://github.com/software-mansion/argent-private/pull/16 ## TL;DR Adds **Android** to the native profiler (previously iOS-only) and does it with a brand-new engine: instead of shipping Perfetto's `trace_processor_shell` native binary and forking it per query, we now run **Perfetto's trace-processor compiled to WebAssembly, in-process** inside the tool-server. - **One ~13 MB `trace_processor.wasm`** runs on every OS/arch — no per-platform binary, no subprocess, fully offline. - The native profiler is refactored from an iOS-only implementation into a **platform-dispatch facade** (`platforms/ios.ts` + `platforms/android.ts`) over a new **`profiler-shared/`** layer, so iOS and Android share the aggregator, types, lifecycle, and time-alignment code. - New **release/pack pipeline** fetches + sha256-verifies the WASM bundle from `argent-private-releases`, and bundles it (plus SQL queries and the Perfetto TraceConfig) into the published `@swmansion/argent` package. End-to-end on a 76 MB trace with 1013 jank rows: **6.3 s** total analyze (the older per-hang-subprocess approach was projected at ~47 minutes and never finished within the tool-call deadline). --- ## TESTING MANUALLY To test manually run the following packing command checked-out on this branch: ``` npm run pack:mcp -- argent-@pFornagiel-native-android-devtools-perfetto ``` This will bundle the package with the new binaries. After this PR is fixed and the binaries are resolved from the main release, the `argent-@pFornagiel-native-android-devtools-perfetto` release can be deleted. ## Why Android needed native CPU/hang/memory profiling to reach parity with iOS. The natural engine is Perfetto's trace-processor, but the obvious ways to ship it are both bad: - **`trace_processor_shell` binary** — per-platform (a mac-arm64 pack shipped a mac-arm64-only tarball), pinned an OS/arch matrix, and cost a fresh ~1.3 s subprocess fork **per query**. With one subprocess forked per detected hang, a large trace projected to ~47 minutes. - **`get.perfetto.dev/trace_processor` Python launcher** — adds a runtime Python 3 dependency, resolves "latest" at first run (defeats version pinning), and makes the first analyze slow via a network fetch. The WASM engine is one cross-platform artifact, in-process and warm-cached: the trace is parsed **once** and reused across the whole analyze run plus every drill-down query. --- ## The three big pieces ### 1. In-process Perfetto WASM engine New package surface in `@argent/native-devtools-android`: | File | Role | | --- | --- | | `src/wasm-trace-processor.ts` (+476) | The whole engine: boot, glue patching, warm-cache, fault containment, chunked parse, cell decoding | | `src/perfetto-engine.d.ts` (+49) | Hand-written types for the version-agnostic RPC decoder (imported by path, so tsc can't see its own `.d.ts`) | | `src/errors.ts` (+68) | `TraceProcessorUnavailableError` (`wasm_load_failed` / `wasm_path_invalid`) | | `src/bundled-meta.ts` (+12) | `PERFETTO_VERSION` stamp, regenerated at pack time | | `src/index.ts` | Re-exports + package-root accessors (`traceProcessorQueriesDir`, `traceConfigPath`) | Public surface: `resolveTraceProcessorAssets()`, `queryWarm<Row>(tracePath, sql)`, `ensureTraceProcessorReady(tracePath)`, `disposeWarmEngine(tracePath)`. Highlights of how it works: - **Boots Google's prebuilt web/worker wasm under Node.** Installs a minimal worker scope (`self`, `WorkerGlobalScope`, `location`), reads the wasm bytes, patches the Emscripten glue (`useMemory64=false`, hand the bytes in directly, expose the bridge), and drives it over a `MessageChannel` + `EngineBase` subclass. Glue patches are applied **by unique string anchors that throw if they don't match** — the canary for glue-format drift on a Perfetto bump. - **Warm-engine cache, one per trace path.** LRU with `MAX_WARM_ENGINES = 3` (bounds resident memory at ~26–76 MB trace + wasm heap each), 5-min idle dispose (timer `unref()`'d), no failed-promise caching. - **Fault containment (`fatal` promise).** `EngineBase.fail()` throws synchronously inside the `MessagePort.onmessage` handler (e.g. an RPC framing error); left unhandled it becomes a Node `uncaughtException` and would **kill the entire tool-server**, wiping every device + profiler session. Each engine exposes a `fatal: Promise<never>`; the handler try/catches, rejects `fatal`, evicts the engine, and every load/query is `Promise.race`'d against it — so a contained fault is a clean rejection and the server stays up. - **Chunked trace parse.** Perfetto's RPC ring buffer rejects any frame > 64 MiB, so the trace is fed in 32 MiB chunks before `notifyEof()`. ### 2. Android native profiler + the shared/dispatch refactor **Capture** (`utils/android-profiler/capture.ts`, `detect-app.ts`): spawns `adb shell perfetto --txt -c - --background-wait -o /data/.../argent-<ts>.pftrace`, piping the TraceConfig on **stdin** (SELinux denies `shell:s0` writes under `/data/misc/perfetto-traces/`, so the config can't be staged as a file), waits for the PID on stdout, then on stop sends `kill -TERM`, polls `/proc/<pid>`, and `adb pull`s the trace. **Pipeline** (`utils/android-profiler/pipeline/`): PerfettoSQL does the parsing and aggregation, so the Android pipeline is intentionally **two files, not iOS's four**. The per-hang annotation pass is **batched** — `runBatchedHangFolds` inlines all hang windows as a `VALUES` table into one SQL script (two derived views unioned in a single terminal SELECT, because the engine returns only the final statement's rows) and runs it once against the warm engine. **The platform-dispatch refactor** — `native-profiler-start/stop/analyze` shrank from monolithic iOS files (~160–330 lines each) into **thin routers** (~60–110 lines) that branch on `api.platform`: ``` native-profiler-start ─┬─ ios? → platforms/ios.ts (xctrace) └─ android → platforms/android.ts (adb perfetto) ``` A new **`utils/profiler-shared/`** layer holds everything both backends use: | File | What was hoisted | | --- | --- | | `types.ts` | Unified `Bottleneck = CpuHotspot \| UiHang \| MemoryLeak \| MemoryRssGrowth`, each now carrying a `platform` field; shared `NativeProfilerAnalyzeResult`; `RECORDING_CAP_MS` | | `aggregate.ts` | Generic `aggregateCpuHotspots(AggregatorInputRow[])` — burst windowing (`BURST_GAP_MS=500`), severity banding (RED >15%, YELLOW 3–15%, <3% dropped). iOS feeds it via a pre-pass; Android feeds it precomputed SQL rows | | `lifecycle.ts` | `shutdownChild` SIGINT→SIGTERM→SIGKILL ladder + `waitForChildExit` (no PID-reuse hazard) | | `thread.ts` | `normalizeThreadName` for both xctrace and Perfetto thread names | | `time-align.ts` | New `buildPerfettoAnchor()` + `windowsOverlap()` alongside the existing iOS/React anchors | | `format.ts` | `formatBytes` (moved out of the combined report) | iOS was refactored to fit this shape: `ios-profiler/lifecycle.ts` and `types.ts` become thin re-exports; `pipeline/02-aggregate.ts` shrank from ~200 to ~60 lines (keeps its iOS-only dominant-function pre-pass, delegates grouping/bursting/ severity to the shared aggregator). **No iOS behavior change** — same outputs, shared internals. **Session blueprint** (`native-profiler-session.ts`): generic `capturePid` / `captureProcess` (was `xctracePid` / `xctraceProcess`), new `platform` and `androidOnDeviceTracePath` fields, and **platform-branched dispose** (iOS SIGKILLs xctrace; Android `kill -KILL`s perfetto, `rm`s the device trace, and disposes the warm WASM engine). **Drill-down tools**: `profiler-load.ts` learns to restore Android `.pftrace` sessions (with an optional metadata sidecar / `app_process` override); `profiler-stack-query.ts` splits into iOS and Android branches. Unlike iOS (which caches `parsedData` in memory), Android **re-queries the `.pftrace`** per drill-down — the warm engine is the cache, so each query costs only SQL execution time. ### 3. Packaging & release pipeline The WASM bundle's three third-party artifacts (`trace_processor.wasm`, `engine_bundle.node.js`, `engine.mjs`) + `LICENSE` are **never committed** (`.gitignore` lists the generated files but not the dir). They're built and sha256-checksummed in `argent-private` CI, published to `argent-private-releases`, and on our side: - **`scripts/download-trace-processor.sh`** (+88, new) — `gh release download` → verify tarball sha256 (**FATAL on mismatch** — the blobs are unsigned) → extract → re-verify via `SHA256SUMS` → assert all four present. - **`scripts/pack-mcp.cjs`** (+60, new) — single pack entry point: downloads simulator-server, native binaries, **and** the trace-processor bundle, then builds and `npm pack`s `@swmansion/argent`. `package.json`'s `pack` / `pack:mcp` now call this script. - **`packages/argent/scripts/bundle-tools.cjs`** (heavily reworked, +574/-...) — copies `assets/trace-processor/`, `assets/queries/`, and `argent.tracecfg.pbtxt` into `packages/argent/assets/`, and regenerates `bundled-meta.ts` to stamp the Perfetto version. - **`publish.yml` / `publish-next.yml`** — run the download step and then **verify**: all four WASM files present, the wasm magic-byte check (`head -c4 … | grep $'\x00asm'`), plus the queries dir, TraceConfig, manifest, and Android helper APK. Asset reorg: `native-devtools-android` now ships `dist/`, `bin/`, and `assets/` (SQL queries, TraceConfig, manifest moved under `assets/`); `.gitignore` swaps the old per-file `packages/argent/manifest.json` entry for the whole `packages/argent/assets/` dir. --- ## Testing 15 new test files under `packages/tool-server/test/android-perfetto/` cover the engine and pipeline, including: - `dispatch.test.ts` — start/stop/analyze route to the right platform backend. - `aggregate-shared.test.ts` — the shared aggregator on both iOS and Android-shaped input. - `hang-folds-batched.test.ts` / `hang-fold.test.ts` — the batched per-hang SQL fold. - `hang-severity.test.ts` — jank-reason → RED/YELLOW classification. - `dispose.test.ts` / `dispose-warm-engine.test.ts` — session dispose kills the daemon, rms the device trace, and tears down the warm engine. - `stop-recovery.test.ts` — `profilingActive` is forced false even if `adb pull` fails (so a retry-start doesn't wedge). - `run-tp.test.ts` — `renderSqlTemplate` token rendering + the "unused substitution" guard. - `manifest-hint.test.ts` — zero-callstack detection emits the manifest hint. - `profiler-load.test.ts` — `.pftrace` session restore with/without the metadata sidecar. Plus `native-profiler-analyze-failure.test.ts` for the engine-unavailable banner, and updated iOS tests for the shared-lifecycle refactor. `@argent/native-devtools-android` gains its own vitest setup (`vitest.config.ts`, `tsconfig.test.json`, `test/errors.test.ts`). --- ## Notes & follow-ups - **Manifest requirement.** Perfetto's `linux.perf` data source needs `/proc/<pid>/mem` access, granted only to a **debuggable** or `<profileable android:shell="true"/>` app. A release build without either silently produces zero callstacks; the pipeline detects this and emits a **manifest hint** instead of an "All clear" report. - **Version pinning lives in the `argent-private` submodule** (`PERFETTO_VERSION` / `PERFETTO_WASM_TAG` / `LYNX_TRACE_PROCESSOR_VERSION`). Bumping is documented in `argent-private/docs/ANDROID_PERFETTO.md`. JS↔engine version skew is structurally impossible (version-agnostic decoder + vendored wasm). - **`disposeWarmEngine` is exported but not wired into session-stop** — warm engines are reclaimed only by the idle timer + LRU cap. Acceptable today (bounded memory); wire it into session dispose if explicit teardown is ever needed. - **Submodule bump.** This PR moves `packages/argent-private` to `22e55f90` — see the companion `argent-private` PR description, including its merge-hygiene note (that branch should pick up latest `argent-private` main before the pointer is finalized). ### Companion docs added in this branch - `utils/android-profiler/ANDROID_PROFILER_REFERENCE.md` — stack, capture, queries, manifest requirement. - `utils/android-profiler/PIPELINE_DESIGN.md` — the "why" (two-file pipeline, shared-aggregator hoist, re-query-don't-cache, batched fold). --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
5.1 KiB
Argent Android profiler — PerfettoSQL queries
These *.sql files are the source of truth for every query the Argent Android
native profiler runs against a captured .pftrace. They run through the
in-process Perfetto WASM trace-processor from the tool-server pipeline
(runTpQuery / runTpInline in
tool-server/src/utils/android-profiler/pipeline/run-tp.ts).
The directory lives in native-devtools-android; the bundler copies it next to
the bundled tool-server at publish time, so traceProcessorQueriesDir() resolves
the same path in dev and packaged builds.
Each .sql header documents only what's specific to that query. The shared
conventions live here so they aren't repeated nine times.
What each file is for
| File | Consumed by | Purpose |
|---|---|---|
trace-bounds.sql |
every analyze run | Trace start timestamp anchor (see Timestamps). |
ui-hangs.sql |
analyze | ANRs + app-jank frames → one hang per frame. |
cpu-hotspots.sql |
analyze | Per-thread hottest leaf functions + burst windows. |
thread-breakdown.sql |
profiler-stack-query mode=thread_breakdown |
Per-thread sample share. |
hang-folds-batched.sql |
batched analyze | State breakdown + GC overlap for ALL hangs in one batched query. |
hang-state-breakdown.sql |
drill-down (single hang) | Main-thread state breakdown for one hang window. |
hang-main-thread-samples.sql |
profiler-stack-query mode=hang_stacks |
Main-thread CPU samples inside one hang window. |
function-callers.sql |
profiler-stack-query mode=function_callers |
Callsites that hit one hot function. |
memory-rss.sql |
analyze | RSS-growth weak signal (not leak detection). |
Conventions
Parameters via the _argent_args view
Each query declares its runtime parameters once in a small _argent_args
PERFETTO VIEW at the top, then references them by name:
DROP VIEW IF EXISTS _argent_args;
CREATE PERFETTO VIEW _argent_args AS
SELECT '{{TARGET_PROCESS}}' AS target_process;
...
WHERE p.name = (SELECT target_process FROM _argent_args)
This keeps each value at one self-documenting site, and the body reads like normal SQL instead of scattering bare tokens through it.
{{NAME}} template tokens
{{NAME}} placeholders are resolved by renderSqlTemplate (run-tp.ts) before
the query runs. It throws on a mismatch either way: a {{NAME}} with no
substitution, or a substitution the template never uses — catching forgotten or
stale tokens early.
Values are not escaped for SQL injection — they're interpolated into the
query string passed to the in-process engine, so callers must validate them
(numeric for *_ns; identifier-shaped for process/thread/function names) — see
hang-folds-batched.ts for the strictest example.
Most queries render through runTpQuery. hang-folds-batched.sql is the
exception: pipeline/hang-folds-batched.ts loads it directly, builds the
{{HANG_WINDOWS_VALUES}} tuple list, and resolves it through the same renderer.
Timestamps are CLOCK_MONOTONIC nanoseconds
Perfetto's ts columns are CLOCK_MONOTONIC nanoseconds since device boot — not
trace-relative. trace-bounds.sql returns the earliest ts; the JS side
subtracts it (traceStartMs) to normalise every emitted timestamp to
trace-relative ns. Any native ms/ns a query emits (burst windows, hang bounds)
stays native until JS does that subtraction.
One trace parse per warm engine → batch
Re-parsing the whole trace on every query is expensive (~1.3 s for 76 MB), so
one query per item is quadratic — the per-hang loop this replaced took ~47 min
for 1013 hangs. Instead, fold many per-item queries into a single script with
CREATE PERFETTO VIEW/TABLE + a terminal UNION ALL SELECT, joining over a
runtime-built table. See hang-folds-batched.sql: one ~1.7 s run regardless of
hang count. Only the final SELECT reaches stdout.
Two copies of the hang state breakdown — keep them in sync
The main-thread state-breakdown logic lives in two places:
hang-state-breakdown.sql— single window, drill-down path;- the
argent_hang_stateview inhang-folds-batched.sql— all windows.
Keep the window-clipping math in sync: each thread_state slice is clipped to
the hang window (MIN(ts.ts + ts.dur, end) - MAX(ts.ts, start)) so a slice
straddling a boundary only counts its overlap. A plain SUM(dur) over slices
that merely start inside the window can overshoot its length.
There's no standalone GC query — GC overlap lives only in the batched file.