mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
3ccd89f333
# Android native profiling via in-process Perfetto WASM Connected with: https://github.com/software-mansion/argent-private/pull/16 ## TL;DR Adds **Android** to the native profiler (previously iOS-only) and does it with a brand-new engine: instead of shipping Perfetto's `trace_processor_shell` native binary and forking it per query, we now run **Perfetto's trace-processor compiled to WebAssembly, in-process** inside the tool-server. - **One ~13 MB `trace_processor.wasm`** runs on every OS/arch — no per-platform binary, no subprocess, fully offline. - The native profiler is refactored from an iOS-only implementation into a **platform-dispatch facade** (`platforms/ios.ts` + `platforms/android.ts`) over a new **`profiler-shared/`** layer, so iOS and Android share the aggregator, types, lifecycle, and time-alignment code. - New **release/pack pipeline** fetches + sha256-verifies the WASM bundle from `argent-private-releases`, and bundles it (plus SQL queries and the Perfetto TraceConfig) into the published `@swmansion/argent` package. End-to-end on a 76 MB trace with 1013 jank rows: **6.3 s** total analyze (the older per-hang-subprocess approach was projected at ~47 minutes and never finished within the tool-call deadline). --- ## TESTING MANUALLY To test manually run the following packing command checked-out on this branch: ``` npm run pack:mcp -- argent-@pFornagiel-native-android-devtools-perfetto ``` This will bundle the package with the new binaries. After this PR is fixed and the binaries are resolved from the main release, the `argent-@pFornagiel-native-android-devtools-perfetto` release can be deleted. ## Why Android needed native CPU/hang/memory profiling to reach parity with iOS. The natural engine is Perfetto's trace-processor, but the obvious ways to ship it are both bad: - **`trace_processor_shell` binary** — per-platform (a mac-arm64 pack shipped a mac-arm64-only tarball), pinned an OS/arch matrix, and cost a fresh ~1.3 s subprocess fork **per query**. With one subprocess forked per detected hang, a large trace projected to ~47 minutes. - **`get.perfetto.dev/trace_processor` Python launcher** — adds a runtime Python 3 dependency, resolves "latest" at first run (defeats version pinning), and makes the first analyze slow via a network fetch. The WASM engine is one cross-platform artifact, in-process and warm-cached: the trace is parsed **once** and reused across the whole analyze run plus every drill-down query. --- ## The three big pieces ### 1. In-process Perfetto WASM engine New package surface in `@argent/native-devtools-android`: | File | Role | | --- | --- | | `src/wasm-trace-processor.ts` (+476) | The whole engine: boot, glue patching, warm-cache, fault containment, chunked parse, cell decoding | | `src/perfetto-engine.d.ts` (+49) | Hand-written types for the version-agnostic RPC decoder (imported by path, so tsc can't see its own `.d.ts`) | | `src/errors.ts` (+68) | `TraceProcessorUnavailableError` (`wasm_load_failed` / `wasm_path_invalid`) | | `src/bundled-meta.ts` (+12) | `PERFETTO_VERSION` stamp, regenerated at pack time | | `src/index.ts` | Re-exports + package-root accessors (`traceProcessorQueriesDir`, `traceConfigPath`) | Public surface: `resolveTraceProcessorAssets()`, `queryWarm<Row>(tracePath, sql)`, `ensureTraceProcessorReady(tracePath)`, `disposeWarmEngine(tracePath)`. Highlights of how it works: - **Boots Google's prebuilt web/worker wasm under Node.** Installs a minimal worker scope (`self`, `WorkerGlobalScope`, `location`), reads the wasm bytes, patches the Emscripten glue (`useMemory64=false`, hand the bytes in directly, expose the bridge), and drives it over a `MessageChannel` + `EngineBase` subclass. Glue patches are applied **by unique string anchors that throw if they don't match** — the canary for glue-format drift on a Perfetto bump. - **Warm-engine cache, one per trace path.** LRU with `MAX_WARM_ENGINES = 3` (bounds resident memory at ~26–76 MB trace + wasm heap each), 5-min idle dispose (timer `unref()`'d), no failed-promise caching. - **Fault containment (`fatal` promise).** `EngineBase.fail()` throws synchronously inside the `MessagePort.onmessage` handler (e.g. an RPC framing error); left unhandled it becomes a Node `uncaughtException` and would **kill the entire tool-server**, wiping every device + profiler session. Each engine exposes a `fatal: Promise<never>`; the handler try/catches, rejects `fatal`, evicts the engine, and every load/query is `Promise.race`'d against it — so a contained fault is a clean rejection and the server stays up. - **Chunked trace parse.** Perfetto's RPC ring buffer rejects any frame > 64 MiB, so the trace is fed in 32 MiB chunks before `notifyEof()`. ### 2. Android native profiler + the shared/dispatch refactor **Capture** (`utils/android-profiler/capture.ts`, `detect-app.ts`): spawns `adb shell perfetto --txt -c - --background-wait -o /data/.../argent-<ts>.pftrace`, piping the TraceConfig on **stdin** (SELinux denies `shell:s0` writes under `/data/misc/perfetto-traces/`, so the config can't be staged as a file), waits for the PID on stdout, then on stop sends `kill -TERM`, polls `/proc/<pid>`, and `adb pull`s the trace. **Pipeline** (`utils/android-profiler/pipeline/`): PerfettoSQL does the parsing and aggregation, so the Android pipeline is intentionally **two files, not iOS's four**. The per-hang annotation pass is **batched** — `runBatchedHangFolds` inlines all hang windows as a `VALUES` table into one SQL script (two derived views unioned in a single terminal SELECT, because the engine returns only the final statement's rows) and runs it once against the warm engine. **The platform-dispatch refactor** — `native-profiler-start/stop/analyze` shrank from monolithic iOS files (~160–330 lines each) into **thin routers** (~60–110 lines) that branch on `api.platform`: ``` native-profiler-start ─┬─ ios? → platforms/ios.ts (xctrace) └─ android → platforms/android.ts (adb perfetto) ``` A new **`utils/profiler-shared/`** layer holds everything both backends use: | File | What was hoisted | | --- | --- | | `types.ts` | Unified `Bottleneck = CpuHotspot \| UiHang \| MemoryLeak \| MemoryRssGrowth`, each now carrying a `platform` field; shared `NativeProfilerAnalyzeResult`; `RECORDING_CAP_MS` | | `aggregate.ts` | Generic `aggregateCpuHotspots(AggregatorInputRow[])` — burst windowing (`BURST_GAP_MS=500`), severity banding (RED >15%, YELLOW 3–15%, <3% dropped). iOS feeds it via a pre-pass; Android feeds it precomputed SQL rows | | `lifecycle.ts` | `shutdownChild` SIGINT→SIGTERM→SIGKILL ladder + `waitForChildExit` (no PID-reuse hazard) | | `thread.ts` | `normalizeThreadName` for both xctrace and Perfetto thread names | | `time-align.ts` | New `buildPerfettoAnchor()` + `windowsOverlap()` alongside the existing iOS/React anchors | | `format.ts` | `formatBytes` (moved out of the combined report) | iOS was refactored to fit this shape: `ios-profiler/lifecycle.ts` and `types.ts` become thin re-exports; `pipeline/02-aggregate.ts` shrank from ~200 to ~60 lines (keeps its iOS-only dominant-function pre-pass, delegates grouping/bursting/ severity to the shared aggregator). **No iOS behavior change** — same outputs, shared internals. **Session blueprint** (`native-profiler-session.ts`): generic `capturePid` / `captureProcess` (was `xctracePid` / `xctraceProcess`), new `platform` and `androidOnDeviceTracePath` fields, and **platform-branched dispose** (iOS SIGKILLs xctrace; Android `kill -KILL`s perfetto, `rm`s the device trace, and disposes the warm WASM engine). **Drill-down tools**: `profiler-load.ts` learns to restore Android `.pftrace` sessions (with an optional metadata sidecar / `app_process` override); `profiler-stack-query.ts` splits into iOS and Android branches. Unlike iOS (which caches `parsedData` in memory), Android **re-queries the `.pftrace`** per drill-down — the warm engine is the cache, so each query costs only SQL execution time. ### 3. Packaging & release pipeline The WASM bundle's three third-party artifacts (`trace_processor.wasm`, `engine_bundle.node.js`, `engine.mjs`) + `LICENSE` are **never committed** (`.gitignore` lists the generated files but not the dir). They're built and sha256-checksummed in `argent-private` CI, published to `argent-private-releases`, and on our side: - **`scripts/download-trace-processor.sh`** (+88, new) — `gh release download` → verify tarball sha256 (**FATAL on mismatch** — the blobs are unsigned) → extract → re-verify via `SHA256SUMS` → assert all four present. - **`scripts/pack-mcp.cjs`** (+60, new) — single pack entry point: downloads simulator-server, native binaries, **and** the trace-processor bundle, then builds and `npm pack`s `@swmansion/argent`. `package.json`'s `pack` / `pack:mcp` now call this script. - **`packages/argent/scripts/bundle-tools.cjs`** (heavily reworked, +574/-...) — copies `assets/trace-processor/`, `assets/queries/`, and `argent.tracecfg.pbtxt` into `packages/argent/assets/`, and regenerates `bundled-meta.ts` to stamp the Perfetto version. - **`publish.yml` / `publish-next.yml`** — run the download step and then **verify**: all four WASM files present, the wasm magic-byte check (`head -c4 … | grep $'\x00asm'`), plus the queries dir, TraceConfig, manifest, and Android helper APK. Asset reorg: `native-devtools-android` now ships `dist/`, `bin/`, and `assets/` (SQL queries, TraceConfig, manifest moved under `assets/`); `.gitignore` swaps the old per-file `packages/argent/manifest.json` entry for the whole `packages/argent/assets/` dir. --- ## Testing 15 new test files under `packages/tool-server/test/android-perfetto/` cover the engine and pipeline, including: - `dispatch.test.ts` — start/stop/analyze route to the right platform backend. - `aggregate-shared.test.ts` — the shared aggregator on both iOS and Android-shaped input. - `hang-folds-batched.test.ts` / `hang-fold.test.ts` — the batched per-hang SQL fold. - `hang-severity.test.ts` — jank-reason → RED/YELLOW classification. - `dispose.test.ts` / `dispose-warm-engine.test.ts` — session dispose kills the daemon, rms the device trace, and tears down the warm engine. - `stop-recovery.test.ts` — `profilingActive` is forced false even if `adb pull` fails (so a retry-start doesn't wedge). - `run-tp.test.ts` — `renderSqlTemplate` token rendering + the "unused substitution" guard. - `manifest-hint.test.ts` — zero-callstack detection emits the manifest hint. - `profiler-load.test.ts` — `.pftrace` session restore with/without the metadata sidecar. Plus `native-profiler-analyze-failure.test.ts` for the engine-unavailable banner, and updated iOS tests for the shared-lifecycle refactor. `@argent/native-devtools-android` gains its own vitest setup (`vitest.config.ts`, `tsconfig.test.json`, `test/errors.test.ts`). --- ## Notes & follow-ups - **Manifest requirement.** Perfetto's `linux.perf` data source needs `/proc/<pid>/mem` access, granted only to a **debuggable** or `<profileable android:shell="true"/>` app. A release build without either silently produces zero callstacks; the pipeline detects this and emits a **manifest hint** instead of an "All clear" report. - **Version pinning lives in the `argent-private` submodule** (`PERFETTO_VERSION` / `PERFETTO_WASM_TAG` / `LYNX_TRACE_PROCESSOR_VERSION`). Bumping is documented in `argent-private/docs/ANDROID_PERFETTO.md`. JS↔engine version skew is structurally impossible (version-agnostic decoder + vendored wasm). - **`disposeWarmEngine` is exported but not wired into session-stop** — warm engines are reclaimed only by the idle timer + LRU cap. Acceptable today (bounded memory); wire it into session dispose if explicit teardown is ever needed. - **Submodule bump.** This PR moves `packages/argent-private` to `22e55f90` — see the companion `argent-private` PR description, including its merge-hygiene note (that branch should pick up latest `argent-private` main before the pointer is finalized). ### Companion docs added in this branch - `utils/android-profiler/ANDROID_PROFILER_REFERENCE.md` — stack, capture, queries, manifest requirement. - `utils/android-profiler/PIPELINE_DESIGN.md` — the "why" (two-file pipeline, shared-aggregator hoist, re-query-don't-cache, batched fold). --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
189 lines
6.9 KiB
TypeScript
189 lines
6.9 KiB
TypeScript
import { describe, it, expect, vi, beforeEach } from "vitest";
|
|
|
|
// Top-level queries (cpu-hotspots, ui-hangs, memory-rss, trace-bounds) still
|
|
// flow through runTpQuery — each test plans the queries it expects in order
|
|
// and the mock pops them.
|
|
const queryResponses: Array<{ name: string; rows: unknown[] }> = [];
|
|
|
|
// The per-hang fold path now goes through runTpInline (one batched call),
|
|
// so we capture the rendered SQL and respond with a precanned row set.
|
|
const inlineCalls: Array<{ sql: string }> = [];
|
|
let inlineResponse: unknown[] = [];
|
|
|
|
vi.mock("@argent/native-devtools-android", () => {
|
|
const path = require("node:path");
|
|
return {
|
|
// The pipeline pre-warms the in-process WASM engine up front; pretend it's ready.
|
|
ensureTraceProcessorReady: vi.fn(async () => {}),
|
|
// Real queries dir — runBatchedHangFolds loads hang-folds-batched.sql from
|
|
// here and substitutes the windows/target before calling runTpInline.
|
|
traceProcessorQueriesDir: () =>
|
|
path.resolve(__dirname, "../../../native-devtools-android/assets/queries"),
|
|
};
|
|
});
|
|
vi.mock("../../src/utils/android-profiler/pipeline/run-tp", async (importActual) => ({
|
|
// Keep the real renderSqlTemplate so the batched hang-fold path renders the
|
|
// on-disk template; only the WASM engine query calls are stubbed.
|
|
...(await importActual<typeof import("../../src/utils/android-profiler/pipeline/run-tp")>()),
|
|
runTpQuery: vi.fn(async (opts: { query: string; substitutions: Record<string, string> }) => {
|
|
const next = queryResponses.shift();
|
|
if (!next) throw new Error(`runTpQuery called for "${opts.query}" with no queued response`);
|
|
if (next.name !== opts.query) {
|
|
throw new Error(
|
|
`runTpQuery expected to be called for "${next.name}" but got "${opts.query}"`
|
|
);
|
|
}
|
|
return next.rows;
|
|
}),
|
|
runTpInline: vi.fn(async (opts: { sql: string }) => {
|
|
inlineCalls.push({ sql: opts.sql });
|
|
return inlineResponse;
|
|
}),
|
|
parseTpJsonOutput: vi.fn(),
|
|
}));
|
|
|
|
import { runAndroidProfilerPipeline } from "../../src/utils/android-profiler/pipeline/index";
|
|
|
|
const TRACE_START_NS = 5_000_000_000_000; // realistic monotonic-since-boot offset
|
|
const HANG_START_NATIVE = TRACE_START_NS + 1_000_000_000; // 1s into the trace
|
|
const HANG_DUR_NS = 500_000_000; // 500ms
|
|
|
|
describe("runAndroidProfilerPipeline timestamp normalisation", () => {
|
|
beforeEach(() => {
|
|
queryResponses.length = 0;
|
|
inlineCalls.length = 0;
|
|
inlineResponse = [];
|
|
});
|
|
|
|
it("subtracts trace_bounds.start_ts from hang ts before storing UiHang.startNs/endNs", async () => {
|
|
queryResponses.push(
|
|
{ name: "trace-bounds.sql", rows: [{ start_ts: TRACE_START_NS }] },
|
|
{ name: "cpu-hotspots.sql", rows: [] },
|
|
{
|
|
name: "ui-hangs.sql",
|
|
rows: [
|
|
{
|
|
kind: "jank",
|
|
ts_ns: HANG_START_NATIVE,
|
|
dur_ns: HANG_DUR_NS,
|
|
process_name: "com.example.app",
|
|
reason: "App Deadline Missed",
|
|
error_id: null,
|
|
},
|
|
],
|
|
},
|
|
{ name: "memory-rss.sql", rows: [] }
|
|
);
|
|
|
|
const result = await runAndroidProfilerPipeline("/fake.pftrace", "com.example.app");
|
|
|
|
expect(result.uiHangs.length).toBe(1);
|
|
const hang = result.uiHangs[0]!;
|
|
// Trace-relative: 1_000_000_000 ns (1s) — NOT 5_001_000_000_000 (the native value)
|
|
expect(hang.startNs).toBe(1_000_000_000);
|
|
expect(hang.endNs).toBe(1_000_000_000 + HANG_DUR_NS);
|
|
});
|
|
|
|
it("inlines NATIVE-domain ns bounds into the batched hang-fold SQL", async () => {
|
|
queryResponses.push(
|
|
{ name: "trace-bounds.sql", rows: [{ start_ts: TRACE_START_NS }] },
|
|
{ name: "cpu-hotspots.sql", rows: [] },
|
|
{
|
|
name: "ui-hangs.sql",
|
|
rows: [
|
|
{
|
|
kind: "jank",
|
|
ts_ns: HANG_START_NATIVE,
|
|
dur_ns: HANG_DUR_NS,
|
|
process_name: "com.example.app",
|
|
reason: "App Deadline Missed",
|
|
error_id: null,
|
|
},
|
|
],
|
|
},
|
|
{ name: "memory-rss.sql", rows: [] }
|
|
);
|
|
|
|
await runAndroidProfilerPipeline("/fake.pftrace", "com.example.app");
|
|
|
|
expect(inlineCalls).toHaveLength(1);
|
|
const sql = inlineCalls[0]!.sql;
|
|
// The VALUES row for the single hang must carry the native (not
|
|
// trace-relative) ns bounds, since the JOIN matches against thread_state.ts
|
|
// and slice.ts, both of which are in the native CLOCK_MONOTONIC domain.
|
|
expect(sql).toContain(String(HANG_START_NATIVE));
|
|
expect(sql).toContain(String(HANG_START_NATIVE + HANG_DUR_NS));
|
|
// And the negative-test: the trace-relative value (1s) must NOT appear as
|
|
// a bound — that would mean we mixed domains and the JOIN finds nothing.
|
|
expect(sql).not.toMatch(/,1000000000,1500000000\)/);
|
|
});
|
|
|
|
it("accepts trace_bounds.start_ts emitted as a JSON string (tp version drift)", async () => {
|
|
queryResponses.push(
|
|
// String-typed start_ts mimics a hypothetical future tp version that
|
|
// emits 64-bit ns values as strings to avoid JS-number precision loss.
|
|
{ name: "trace-bounds.sql", rows: [{ start_ts: String(TRACE_START_NS) }] },
|
|
{ name: "cpu-hotspots.sql", rows: [] },
|
|
{
|
|
name: "ui-hangs.sql",
|
|
rows: [
|
|
{
|
|
kind: "jank",
|
|
ts_ns: HANG_START_NATIVE,
|
|
dur_ns: HANG_DUR_NS,
|
|
process_name: "com.example.app",
|
|
reason: null,
|
|
error_id: null,
|
|
},
|
|
],
|
|
},
|
|
{ name: "memory-rss.sql", rows: [] }
|
|
);
|
|
|
|
const result = await runAndroidProfilerPipeline("/fake.pftrace", "com.example.app");
|
|
// If string coercion silently failed, the test would see startNs=HANG_START_NATIVE (5e12).
|
|
expect(result.uiHangs[0]!.startNs).toBe(1_000_000_000);
|
|
});
|
|
|
|
it("computes gcOverlapMs against trace-relative hang bounds (no domain mix-up)", async () => {
|
|
queryResponses.push(
|
|
{ name: "trace-bounds.sql", rows: [{ start_ts: TRACE_START_NS }] },
|
|
{ name: "cpu-hotspots.sql", rows: [] },
|
|
{
|
|
name: "ui-hangs.sql",
|
|
rows: [
|
|
{
|
|
kind: "jank",
|
|
ts_ns: HANG_START_NATIVE,
|
|
dur_ns: HANG_DUR_NS,
|
|
process_name: "com.example.app",
|
|
reason: null,
|
|
error_id: null,
|
|
},
|
|
],
|
|
},
|
|
{ name: "memory-rss.sql", rows: [] }
|
|
);
|
|
// The batched-fold call returns the GC slice in NATIVE ns — the pipeline
|
|
// must normalise to trace-relative before handing it to foldHangAnnotations.
|
|
inlineResponse = [
|
|
{
|
|
hang_index: 0,
|
|
row_kind: "gc",
|
|
state_v: null,
|
|
blocked_function_v: null,
|
|
total_dur_ns_v: null,
|
|
occurrences_v: null,
|
|
gc_reason_v: "GC: concurrent copying",
|
|
// 100ms after hang start, 200ms duration → 200ms overlap
|
|
gc_ts_ns_v: String(HANG_START_NATIVE + 100_000_000),
|
|
gc_dur_ns_v: String(200_000_000),
|
|
},
|
|
];
|
|
|
|
const result = await runAndroidProfilerPipeline("/fake.pftrace", "com.example.app");
|
|
|
|
expect(result.uiHangs[0]!.gcOverlapMs).toBe(200);
|
|
});
|
|
});
|