mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
3ccd89f333
# Android native profiling via in-process Perfetto WASM Connected with: https://github.com/software-mansion/argent-private/pull/16 ## TL;DR Adds **Android** to the native profiler (previously iOS-only) and does it with a brand-new engine: instead of shipping Perfetto's `trace_processor_shell` native binary and forking it per query, we now run **Perfetto's trace-processor compiled to WebAssembly, in-process** inside the tool-server. - **One ~13 MB `trace_processor.wasm`** runs on every OS/arch — no per-platform binary, no subprocess, fully offline. - The native profiler is refactored from an iOS-only implementation into a **platform-dispatch facade** (`platforms/ios.ts` + `platforms/android.ts`) over a new **`profiler-shared/`** layer, so iOS and Android share the aggregator, types, lifecycle, and time-alignment code. - New **release/pack pipeline** fetches + sha256-verifies the WASM bundle from `argent-private-releases`, and bundles it (plus SQL queries and the Perfetto TraceConfig) into the published `@swmansion/argent` package. End-to-end on a 76 MB trace with 1013 jank rows: **6.3 s** total analyze (the older per-hang-subprocess approach was projected at ~47 minutes and never finished within the tool-call deadline). --- ## TESTING MANUALLY To test manually run the following packing command checked-out on this branch: ``` npm run pack:mcp -- argent-@pFornagiel-native-android-devtools-perfetto ``` This will bundle the package with the new binaries. After this PR is fixed and the binaries are resolved from the main release, the `argent-@pFornagiel-native-android-devtools-perfetto` release can be deleted. ## Why Android needed native CPU/hang/memory profiling to reach parity with iOS. The natural engine is Perfetto's trace-processor, but the obvious ways to ship it are both bad: - **`trace_processor_shell` binary** — per-platform (a mac-arm64 pack shipped a mac-arm64-only tarball), pinned an OS/arch matrix, and cost a fresh ~1.3 s subprocess fork **per query**. With one subprocess forked per detected hang, a large trace projected to ~47 minutes. - **`get.perfetto.dev/trace_processor` Python launcher** — adds a runtime Python 3 dependency, resolves "latest" at first run (defeats version pinning), and makes the first analyze slow via a network fetch. The WASM engine is one cross-platform artifact, in-process and warm-cached: the trace is parsed **once** and reused across the whole analyze run plus every drill-down query. --- ## The three big pieces ### 1. In-process Perfetto WASM engine New package surface in `@argent/native-devtools-android`: | File | Role | | --- | --- | | `src/wasm-trace-processor.ts` (+476) | The whole engine: boot, glue patching, warm-cache, fault containment, chunked parse, cell decoding | | `src/perfetto-engine.d.ts` (+49) | Hand-written types for the version-agnostic RPC decoder (imported by path, so tsc can't see its own `.d.ts`) | | `src/errors.ts` (+68) | `TraceProcessorUnavailableError` (`wasm_load_failed` / `wasm_path_invalid`) | | `src/bundled-meta.ts` (+12) | `PERFETTO_VERSION` stamp, regenerated at pack time | | `src/index.ts` | Re-exports + package-root accessors (`traceProcessorQueriesDir`, `traceConfigPath`) | Public surface: `resolveTraceProcessorAssets()`, `queryWarm<Row>(tracePath, sql)`, `ensureTraceProcessorReady(tracePath)`, `disposeWarmEngine(tracePath)`. Highlights of how it works: - **Boots Google's prebuilt web/worker wasm under Node.** Installs a minimal worker scope (`self`, `WorkerGlobalScope`, `location`), reads the wasm bytes, patches the Emscripten glue (`useMemory64=false`, hand the bytes in directly, expose the bridge), and drives it over a `MessageChannel` + `EngineBase` subclass. Glue patches are applied **by unique string anchors that throw if they don't match** — the canary for glue-format drift on a Perfetto bump. - **Warm-engine cache, one per trace path.** LRU with `MAX_WARM_ENGINES = 3` (bounds resident memory at ~26–76 MB trace + wasm heap each), 5-min idle dispose (timer `unref()`'d), no failed-promise caching. - **Fault containment (`fatal` promise).** `EngineBase.fail()` throws synchronously inside the `MessagePort.onmessage` handler (e.g. an RPC framing error); left unhandled it becomes a Node `uncaughtException` and would **kill the entire tool-server**, wiping every device + profiler session. Each engine exposes a `fatal: Promise<never>`; the handler try/catches, rejects `fatal`, evicts the engine, and every load/query is `Promise.race`'d against it — so a contained fault is a clean rejection and the server stays up. - **Chunked trace parse.** Perfetto's RPC ring buffer rejects any frame > 64 MiB, so the trace is fed in 32 MiB chunks before `notifyEof()`. ### 2. Android native profiler + the shared/dispatch refactor **Capture** (`utils/android-profiler/capture.ts`, `detect-app.ts`): spawns `adb shell perfetto --txt -c - --background-wait -o /data/.../argent-<ts>.pftrace`, piping the TraceConfig on **stdin** (SELinux denies `shell:s0` writes under `/data/misc/perfetto-traces/`, so the config can't be staged as a file), waits for the PID on stdout, then on stop sends `kill -TERM`, polls `/proc/<pid>`, and `adb pull`s the trace. **Pipeline** (`utils/android-profiler/pipeline/`): PerfettoSQL does the parsing and aggregation, so the Android pipeline is intentionally **two files, not iOS's four**. The per-hang annotation pass is **batched** — `runBatchedHangFolds` inlines all hang windows as a `VALUES` table into one SQL script (two derived views unioned in a single terminal SELECT, because the engine returns only the final statement's rows) and runs it once against the warm engine. **The platform-dispatch refactor** — `native-profiler-start/stop/analyze` shrank from monolithic iOS files (~160–330 lines each) into **thin routers** (~60–110 lines) that branch on `api.platform`: ``` native-profiler-start ─┬─ ios? → platforms/ios.ts (xctrace) └─ android → platforms/android.ts (adb perfetto) ``` A new **`utils/profiler-shared/`** layer holds everything both backends use: | File | What was hoisted | | --- | --- | | `types.ts` | Unified `Bottleneck = CpuHotspot \| UiHang \| MemoryLeak \| MemoryRssGrowth`, each now carrying a `platform` field; shared `NativeProfilerAnalyzeResult`; `RECORDING_CAP_MS` | | `aggregate.ts` | Generic `aggregateCpuHotspots(AggregatorInputRow[])` — burst windowing (`BURST_GAP_MS=500`), severity banding (RED >15%, YELLOW 3–15%, <3% dropped). iOS feeds it via a pre-pass; Android feeds it precomputed SQL rows | | `lifecycle.ts` | `shutdownChild` SIGINT→SIGTERM→SIGKILL ladder + `waitForChildExit` (no PID-reuse hazard) | | `thread.ts` | `normalizeThreadName` for both xctrace and Perfetto thread names | | `time-align.ts` | New `buildPerfettoAnchor()` + `windowsOverlap()` alongside the existing iOS/React anchors | | `format.ts` | `formatBytes` (moved out of the combined report) | iOS was refactored to fit this shape: `ios-profiler/lifecycle.ts` and `types.ts` become thin re-exports; `pipeline/02-aggregate.ts` shrank from ~200 to ~60 lines (keeps its iOS-only dominant-function pre-pass, delegates grouping/bursting/ severity to the shared aggregator). **No iOS behavior change** — same outputs, shared internals. **Session blueprint** (`native-profiler-session.ts`): generic `capturePid` / `captureProcess` (was `xctracePid` / `xctraceProcess`), new `platform` and `androidOnDeviceTracePath` fields, and **platform-branched dispose** (iOS SIGKILLs xctrace; Android `kill -KILL`s perfetto, `rm`s the device trace, and disposes the warm WASM engine). **Drill-down tools**: `profiler-load.ts` learns to restore Android `.pftrace` sessions (with an optional metadata sidecar / `app_process` override); `profiler-stack-query.ts` splits into iOS and Android branches. Unlike iOS (which caches `parsedData` in memory), Android **re-queries the `.pftrace`** per drill-down — the warm engine is the cache, so each query costs only SQL execution time. ### 3. Packaging & release pipeline The WASM bundle's three third-party artifacts (`trace_processor.wasm`, `engine_bundle.node.js`, `engine.mjs`) + `LICENSE` are **never committed** (`.gitignore` lists the generated files but not the dir). They're built and sha256-checksummed in `argent-private` CI, published to `argent-private-releases`, and on our side: - **`scripts/download-trace-processor.sh`** (+88, new) — `gh release download` → verify tarball sha256 (**FATAL on mismatch** — the blobs are unsigned) → extract → re-verify via `SHA256SUMS` → assert all four present. - **`scripts/pack-mcp.cjs`** (+60, new) — single pack entry point: downloads simulator-server, native binaries, **and** the trace-processor bundle, then builds and `npm pack`s `@swmansion/argent`. `package.json`'s `pack` / `pack:mcp` now call this script. - **`packages/argent/scripts/bundle-tools.cjs`** (heavily reworked, +574/-...) — copies `assets/trace-processor/`, `assets/queries/`, and `argent.tracecfg.pbtxt` into `packages/argent/assets/`, and regenerates `bundled-meta.ts` to stamp the Perfetto version. - **`publish.yml` / `publish-next.yml`** — run the download step and then **verify**: all four WASM files present, the wasm magic-byte check (`head -c4 … | grep $'\x00asm'`), plus the queries dir, TraceConfig, manifest, and Android helper APK. Asset reorg: `native-devtools-android` now ships `dist/`, `bin/`, and `assets/` (SQL queries, TraceConfig, manifest moved under `assets/`); `.gitignore` swaps the old per-file `packages/argent/manifest.json` entry for the whole `packages/argent/assets/` dir. --- ## Testing 15 new test files under `packages/tool-server/test/android-perfetto/` cover the engine and pipeline, including: - `dispatch.test.ts` — start/stop/analyze route to the right platform backend. - `aggregate-shared.test.ts` — the shared aggregator on both iOS and Android-shaped input. - `hang-folds-batched.test.ts` / `hang-fold.test.ts` — the batched per-hang SQL fold. - `hang-severity.test.ts` — jank-reason → RED/YELLOW classification. - `dispose.test.ts` / `dispose-warm-engine.test.ts` — session dispose kills the daemon, rms the device trace, and tears down the warm engine. - `stop-recovery.test.ts` — `profilingActive` is forced false even if `adb pull` fails (so a retry-start doesn't wedge). - `run-tp.test.ts` — `renderSqlTemplate` token rendering + the "unused substitution" guard. - `manifest-hint.test.ts` — zero-callstack detection emits the manifest hint. - `profiler-load.test.ts` — `.pftrace` session restore with/without the metadata sidecar. Plus `native-profiler-analyze-failure.test.ts` for the engine-unavailable banner, and updated iOS tests for the shared-lifecycle refactor. `@argent/native-devtools-android` gains its own vitest setup (`vitest.config.ts`, `tsconfig.test.json`, `test/errors.test.ts`). --- ## Notes & follow-ups - **Manifest requirement.** Perfetto's `linux.perf` data source needs `/proc/<pid>/mem` access, granted only to a **debuggable** or `<profileable android:shell="true"/>` app. A release build without either silently produces zero callstacks; the pipeline detects this and emits a **manifest hint** instead of an "All clear" report. - **Version pinning lives in the `argent-private` submodule** (`PERFETTO_VERSION` / `PERFETTO_WASM_TAG` / `LYNX_TRACE_PROCESSOR_VERSION`). Bumping is documented in `argent-private/docs/ANDROID_PERFETTO.md`. JS↔engine version skew is structurally impossible (version-agnostic decoder + vendored wasm). - **`disposeWarmEngine` is exported but not wired into session-stop** — warm engines are reclaimed only by the idle timer + LRU cap. Acceptable today (bounded memory); wire it into session dispose if explicit teardown is ever needed. - **Submodule bump.** This PR moves `packages/argent-private` to `22e55f90` — see the companion `argent-private` PR description, including its merge-hygiene note (that branch should pick up latest `argent-private` main before the pointer is finalized). ### Companion docs added in this branch - `utils/android-profiler/ANDROID_PROFILER_REFERENCE.md` — stack, capture, queries, manifest requirement. - `utils/android-profiler/PIPELINE_DESIGN.md` — the "why" (two-file pipeline, shared-aggregator hoist, re-query-don't-cache, batched fold). --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
94 lines
5.1 KiB
Markdown
94 lines
5.1 KiB
Markdown
# Argent Android profiler — PerfettoSQL queries
|
|
|
|
These `*.sql` files are the source of truth for every query the Argent Android
|
|
native profiler runs against a captured `.pftrace`. They run through the
|
|
in-process Perfetto WASM trace-processor from the tool-server pipeline
|
|
(`runTpQuery` / `runTpInline` in
|
|
`tool-server/src/utils/android-profiler/pipeline/run-tp.ts`).
|
|
|
|
The directory lives in `native-devtools-android`; the bundler copies it next to
|
|
the bundled tool-server at publish time, so `traceProcessorQueriesDir()` resolves
|
|
the same path in dev and packaged builds.
|
|
|
|
Each `.sql` header documents only what's specific to that query. The shared
|
|
conventions live here so they aren't repeated nine times.
|
|
|
|
## What each file is for
|
|
|
|
| File | Consumed by | Purpose |
|
|
| ------------------------------ | -------------------------------------------- | ---------------------------------------------------------------- |
|
|
| `trace-bounds.sql` | every analyze run | Trace start timestamp anchor (see _Timestamps_). |
|
|
| `ui-hangs.sql` | analyze | ANRs + app-jank frames → one hang per frame. |
|
|
| `cpu-hotspots.sql` | analyze | Per-thread hottest leaf functions + burst windows. |
|
|
| `thread-breakdown.sql` | profiler-stack-query `mode=thread_breakdown` | Per-thread sample share. |
|
|
| `hang-folds-batched.sql` | batched analyze | State breakdown + GC overlap for ALL hangs in one batched query. |
|
|
| `hang-state-breakdown.sql` | drill-down (single hang) | Main-thread state breakdown for one hang window. |
|
|
| `hang-main-thread-samples.sql` | profiler-stack-query `mode=hang_stacks` | Main-thread CPU samples inside one hang window. |
|
|
| `function-callers.sql` | profiler-stack-query `mode=function_callers` | Callsites that hit one hot function. |
|
|
| `memory-rss.sql` | analyze | RSS-growth weak signal (not leak detection). |
|
|
|
|
## Conventions
|
|
|
|
### Parameters via the `_argent_args` view
|
|
|
|
Each query declares its runtime parameters once in a small `_argent_args`
|
|
PERFETTO VIEW at the top, then references them by name:
|
|
|
|
```sql
|
|
DROP VIEW IF EXISTS _argent_args;
|
|
CREATE PERFETTO VIEW _argent_args AS
|
|
SELECT '{{TARGET_PROCESS}}' AS target_process;
|
|
...
|
|
WHERE p.name = (SELECT target_process FROM _argent_args)
|
|
```
|
|
|
|
This keeps each value at one self-documenting site, and the body reads like
|
|
normal SQL instead of scattering bare tokens through it.
|
|
|
|
### `{{NAME}}` template tokens
|
|
|
|
`{{NAME}}` placeholders are resolved by `renderSqlTemplate` (`run-tp.ts`) before
|
|
the query runs. It throws on a mismatch either way: a `{{NAME}}` with no
|
|
substitution, or a substitution the template never uses — catching forgotten or
|
|
stale tokens early.
|
|
|
|
Values are **not** escaped for SQL injection — they're interpolated into the
|
|
query string passed to the in-process engine, so callers must validate them
|
|
(numeric for `*_ns`; identifier-shaped for process/thread/function names) — see
|
|
`hang-folds-batched.ts` for the strictest example.
|
|
|
|
Most queries render through `runTpQuery`. `hang-folds-batched.sql` is the
|
|
exception: `pipeline/hang-folds-batched.ts` loads it directly, builds the
|
|
`{{HANG_WINDOWS_VALUES}}` tuple list, and resolves it through the same renderer.
|
|
|
|
### Timestamps are CLOCK_MONOTONIC nanoseconds
|
|
|
|
Perfetto's `ts` columns are CLOCK_MONOTONIC nanoseconds since device boot — not
|
|
trace-relative. `trace-bounds.sql` returns the earliest `ts`; the JS side
|
|
subtracts it (`traceStartMs`) to normalise every emitted timestamp to
|
|
trace-relative ns. Any native ms/ns a query emits (burst windows, hang bounds)
|
|
stays native until JS does that subtraction.
|
|
|
|
### One trace parse per warm engine → batch
|
|
|
|
Re-parsing the whole trace on every query is expensive (~1.3 s for 76 MB), so
|
|
one query per item is quadratic — the per-hang loop this replaced took ~47 min
|
|
for 1013 hangs. Instead, fold many per-item queries into a single script with
|
|
`CREATE PERFETTO VIEW`/`TABLE` + a terminal `UNION ALL SELECT`, joining over a
|
|
runtime-built table. See `hang-folds-batched.sql`: one ~1.7 s run regardless of
|
|
hang count. Only the final SELECT reaches stdout.
|
|
|
|
### Two copies of the hang state breakdown — keep them in sync
|
|
|
|
The main-thread state-breakdown logic lives in two places:
|
|
|
|
- `hang-state-breakdown.sql` — single window, drill-down path;
|
|
- the `argent_hang_state` view in `hang-folds-batched.sql` — all windows.
|
|
|
|
Keep the window-clipping math in sync: each `thread_state` slice is clipped to
|
|
the hang window (`MIN(ts.ts + ts.dur, end) - MAX(ts.ts, start)`) so a slice
|
|
straddling a boundary only counts its overlap. A plain `SUM(dur)` over slices
|
|
that merely start inside the window can overshoot its length.
|
|
|
|
There's no standalone GC query — GC overlap lives only in the batched file.
|