mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
3ccd89f333
# Android native profiling via in-process Perfetto WASM Connected with: https://github.com/software-mansion/argent-private/pull/16 ## TL;DR Adds **Android** to the native profiler (previously iOS-only) and does it with a brand-new engine: instead of shipping Perfetto's `trace_processor_shell` native binary and forking it per query, we now run **Perfetto's trace-processor compiled to WebAssembly, in-process** inside the tool-server. - **One ~13 MB `trace_processor.wasm`** runs on every OS/arch — no per-platform binary, no subprocess, fully offline. - The native profiler is refactored from an iOS-only implementation into a **platform-dispatch facade** (`platforms/ios.ts` + `platforms/android.ts`) over a new **`profiler-shared/`** layer, so iOS and Android share the aggregator, types, lifecycle, and time-alignment code. - New **release/pack pipeline** fetches + sha256-verifies the WASM bundle from `argent-private-releases`, and bundles it (plus SQL queries and the Perfetto TraceConfig) into the published `@swmansion/argent` package. End-to-end on a 76 MB trace with 1013 jank rows: **6.3 s** total analyze (the older per-hang-subprocess approach was projected at ~47 minutes and never finished within the tool-call deadline). --- ## TESTING MANUALLY To test manually run the following packing command checked-out on this branch: ``` npm run pack:mcp -- argent-@pFornagiel-native-android-devtools-perfetto ``` This will bundle the package with the new binaries. After this PR is fixed and the binaries are resolved from the main release, the `argent-@pFornagiel-native-android-devtools-perfetto` release can be deleted. ## Why Android needed native CPU/hang/memory profiling to reach parity with iOS. The natural engine is Perfetto's trace-processor, but the obvious ways to ship it are both bad: - **`trace_processor_shell` binary** — per-platform (a mac-arm64 pack shipped a mac-arm64-only tarball), pinned an OS/arch matrix, and cost a fresh ~1.3 s subprocess fork **per query**. With one subprocess forked per detected hang, a large trace projected to ~47 minutes. - **`get.perfetto.dev/trace_processor` Python launcher** — adds a runtime Python 3 dependency, resolves "latest" at first run (defeats version pinning), and makes the first analyze slow via a network fetch. The WASM engine is one cross-platform artifact, in-process and warm-cached: the trace is parsed **once** and reused across the whole analyze run plus every drill-down query. --- ## The three big pieces ### 1. In-process Perfetto WASM engine New package surface in `@argent/native-devtools-android`: | File | Role | | --- | --- | | `src/wasm-trace-processor.ts` (+476) | The whole engine: boot, glue patching, warm-cache, fault containment, chunked parse, cell decoding | | `src/perfetto-engine.d.ts` (+49) | Hand-written types for the version-agnostic RPC decoder (imported by path, so tsc can't see its own `.d.ts`) | | `src/errors.ts` (+68) | `TraceProcessorUnavailableError` (`wasm_load_failed` / `wasm_path_invalid`) | | `src/bundled-meta.ts` (+12) | `PERFETTO_VERSION` stamp, regenerated at pack time | | `src/index.ts` | Re-exports + package-root accessors (`traceProcessorQueriesDir`, `traceConfigPath`) | Public surface: `resolveTraceProcessorAssets()`, `queryWarm<Row>(tracePath, sql)`, `ensureTraceProcessorReady(tracePath)`, `disposeWarmEngine(tracePath)`. Highlights of how it works: - **Boots Google's prebuilt web/worker wasm under Node.** Installs a minimal worker scope (`self`, `WorkerGlobalScope`, `location`), reads the wasm bytes, patches the Emscripten glue (`useMemory64=false`, hand the bytes in directly, expose the bridge), and drives it over a `MessageChannel` + `EngineBase` subclass. Glue patches are applied **by unique string anchors that throw if they don't match** — the canary for glue-format drift on a Perfetto bump. - **Warm-engine cache, one per trace path.** LRU with `MAX_WARM_ENGINES = 3` (bounds resident memory at ~26–76 MB trace + wasm heap each), 5-min idle dispose (timer `unref()`'d), no failed-promise caching. - **Fault containment (`fatal` promise).** `EngineBase.fail()` throws synchronously inside the `MessagePort.onmessage` handler (e.g. an RPC framing error); left unhandled it becomes a Node `uncaughtException` and would **kill the entire tool-server**, wiping every device + profiler session. Each engine exposes a `fatal: Promise<never>`; the handler try/catches, rejects `fatal`, evicts the engine, and every load/query is `Promise.race`'d against it — so a contained fault is a clean rejection and the server stays up. - **Chunked trace parse.** Perfetto's RPC ring buffer rejects any frame > 64 MiB, so the trace is fed in 32 MiB chunks before `notifyEof()`. ### 2. Android native profiler + the shared/dispatch refactor **Capture** (`utils/android-profiler/capture.ts`, `detect-app.ts`): spawns `adb shell perfetto --txt -c - --background-wait -o /data/.../argent-<ts>.pftrace`, piping the TraceConfig on **stdin** (SELinux denies `shell:s0` writes under `/data/misc/perfetto-traces/`, so the config can't be staged as a file), waits for the PID on stdout, then on stop sends `kill -TERM`, polls `/proc/<pid>`, and `adb pull`s the trace. **Pipeline** (`utils/android-profiler/pipeline/`): PerfettoSQL does the parsing and aggregation, so the Android pipeline is intentionally **two files, not iOS's four**. The per-hang annotation pass is **batched** — `runBatchedHangFolds` inlines all hang windows as a `VALUES` table into one SQL script (two derived views unioned in a single terminal SELECT, because the engine returns only the final statement's rows) and runs it once against the warm engine. **The platform-dispatch refactor** — `native-profiler-start/stop/analyze` shrank from monolithic iOS files (~160–330 lines each) into **thin routers** (~60–110 lines) that branch on `api.platform`: ``` native-profiler-start ─┬─ ios? → platforms/ios.ts (xctrace) └─ android → platforms/android.ts (adb perfetto) ``` A new **`utils/profiler-shared/`** layer holds everything both backends use: | File | What was hoisted | | --- | --- | | `types.ts` | Unified `Bottleneck = CpuHotspot \| UiHang \| MemoryLeak \| MemoryRssGrowth`, each now carrying a `platform` field; shared `NativeProfilerAnalyzeResult`; `RECORDING_CAP_MS` | | `aggregate.ts` | Generic `aggregateCpuHotspots(AggregatorInputRow[])` — burst windowing (`BURST_GAP_MS=500`), severity banding (RED >15%, YELLOW 3–15%, <3% dropped). iOS feeds it via a pre-pass; Android feeds it precomputed SQL rows | | `lifecycle.ts` | `shutdownChild` SIGINT→SIGTERM→SIGKILL ladder + `waitForChildExit` (no PID-reuse hazard) | | `thread.ts` | `normalizeThreadName` for both xctrace and Perfetto thread names | | `time-align.ts` | New `buildPerfettoAnchor()` + `windowsOverlap()` alongside the existing iOS/React anchors | | `format.ts` | `formatBytes` (moved out of the combined report) | iOS was refactored to fit this shape: `ios-profiler/lifecycle.ts` and `types.ts` become thin re-exports; `pipeline/02-aggregate.ts` shrank from ~200 to ~60 lines (keeps its iOS-only dominant-function pre-pass, delegates grouping/bursting/ severity to the shared aggregator). **No iOS behavior change** — same outputs, shared internals. **Session blueprint** (`native-profiler-session.ts`): generic `capturePid` / `captureProcess` (was `xctracePid` / `xctraceProcess`), new `platform` and `androidOnDeviceTracePath` fields, and **platform-branched dispose** (iOS SIGKILLs xctrace; Android `kill -KILL`s perfetto, `rm`s the device trace, and disposes the warm WASM engine). **Drill-down tools**: `profiler-load.ts` learns to restore Android `.pftrace` sessions (with an optional metadata sidecar / `app_process` override); `profiler-stack-query.ts` splits into iOS and Android branches. Unlike iOS (which caches `parsedData` in memory), Android **re-queries the `.pftrace`** per drill-down — the warm engine is the cache, so each query costs only SQL execution time. ### 3. Packaging & release pipeline The WASM bundle's three third-party artifacts (`trace_processor.wasm`, `engine_bundle.node.js`, `engine.mjs`) + `LICENSE` are **never committed** (`.gitignore` lists the generated files but not the dir). They're built and sha256-checksummed in `argent-private` CI, published to `argent-private-releases`, and on our side: - **`scripts/download-trace-processor.sh`** (+88, new) — `gh release download` → verify tarball sha256 (**FATAL on mismatch** — the blobs are unsigned) → extract → re-verify via `SHA256SUMS` → assert all four present. - **`scripts/pack-mcp.cjs`** (+60, new) — single pack entry point: downloads simulator-server, native binaries, **and** the trace-processor bundle, then builds and `npm pack`s `@swmansion/argent`. `package.json`'s `pack` / `pack:mcp` now call this script. - **`packages/argent/scripts/bundle-tools.cjs`** (heavily reworked, +574/-...) — copies `assets/trace-processor/`, `assets/queries/`, and `argent.tracecfg.pbtxt` into `packages/argent/assets/`, and regenerates `bundled-meta.ts` to stamp the Perfetto version. - **`publish.yml` / `publish-next.yml`** — run the download step and then **verify**: all four WASM files present, the wasm magic-byte check (`head -c4 … | grep $'\x00asm'`), plus the queries dir, TraceConfig, manifest, and Android helper APK. Asset reorg: `native-devtools-android` now ships `dist/`, `bin/`, and `assets/` (SQL queries, TraceConfig, manifest moved under `assets/`); `.gitignore` swaps the old per-file `packages/argent/manifest.json` entry for the whole `packages/argent/assets/` dir. --- ## Testing 15 new test files under `packages/tool-server/test/android-perfetto/` cover the engine and pipeline, including: - `dispatch.test.ts` — start/stop/analyze route to the right platform backend. - `aggregate-shared.test.ts` — the shared aggregator on both iOS and Android-shaped input. - `hang-folds-batched.test.ts` / `hang-fold.test.ts` — the batched per-hang SQL fold. - `hang-severity.test.ts` — jank-reason → RED/YELLOW classification. - `dispose.test.ts` / `dispose-warm-engine.test.ts` — session dispose kills the daemon, rms the device trace, and tears down the warm engine. - `stop-recovery.test.ts` — `profilingActive` is forced false even if `adb pull` fails (so a retry-start doesn't wedge). - `run-tp.test.ts` — `renderSqlTemplate` token rendering + the "unused substitution" guard. - `manifest-hint.test.ts` — zero-callstack detection emits the manifest hint. - `profiler-load.test.ts` — `.pftrace` session restore with/without the metadata sidecar. Plus `native-profiler-analyze-failure.test.ts` for the engine-unavailable banner, and updated iOS tests for the shared-lifecycle refactor. `@argent/native-devtools-android` gains its own vitest setup (`vitest.config.ts`, `tsconfig.test.json`, `test/errors.test.ts`). --- ## Notes & follow-ups - **Manifest requirement.** Perfetto's `linux.perf` data source needs `/proc/<pid>/mem` access, granted only to a **debuggable** or `<profileable android:shell="true"/>` app. A release build without either silently produces zero callstacks; the pipeline detects this and emits a **manifest hint** instead of an "All clear" report. - **Version pinning lives in the `argent-private` submodule** (`PERFETTO_VERSION` / `PERFETTO_WASM_TAG` / `LYNX_TRACE_PROCESSOR_VERSION`). Bumping is documented in `argent-private/docs/ANDROID_PERFETTO.md`. JS↔engine version skew is structurally impossible (version-agnostic decoder + vendored wasm). - **`disposeWarmEngine` is exported but not wired into session-stop** — warm engines are reclaimed only by the idle timer + LRU cap. Acceptable today (bounded memory); wire it into session dispose if explicit teardown is ever needed. - **Submodule bump.** This PR moves `packages/argent-private` to `22e55f90` — see the companion `argent-private` PR description, including its merge-hygiene note (that branch should pick up latest `argent-private` main before the pointer is finalized). ### Companion docs added in this branch - `utils/android-profiler/ANDROID_PROFILER_REFERENCE.md` — stack, capture, queries, manifest requirement. - `utils/android-profiler/PIPELINE_DESIGN.md` — the "why" (two-file pipeline, shared-aggregator hoist, re-query-don't-cache, batched fold). --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
153 lines
5.2 KiB
Bash
Executable File
153 lines
5.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Build native-devtools-android APK for Argent.
|
|
# Java sources live in the argent-private submodule at
|
|
# packages/argent-private/packages/native-devtools-android/Sources/AndroidDevtools/
|
|
# Mirrors `packages/native-devtools-ios/scripts/build.sh` shape.
|
|
#
|
|
# Run from the workspace root: bash packages/native-devtools-android/scripts/build.sh
|
|
# Or from this package: bash scripts/build.sh
|
|
#
|
|
# `versionName` and `versionCode` are sourced from manifest.json — bump that
|
|
# file in git when releasing a new helper, then rebuild.
|
|
#
|
|
# Environment:
|
|
# PREBUILT_ANDROID_DEVTOOLS_APK if set, copy this path to bin/ instead of
|
|
# building (used by CI / non-Android-SDK hosts)
|
|
#
|
|
# Required tools (resolved via $ANDROID_HOME):
|
|
# - javac (system)
|
|
# - $ANDROID_HOME/build-tools/$BUILD_TOOLS/{d8,aapt2,zipalign,apksigner}
|
|
# - $ANDROID_HOME/platforms/android-$COMPILE_SDK/android.jar
|
|
set -euo pipefail
|
|
|
|
PKG_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
WORKSPACE_DIR="$(cd "${PKG_DIR}/../.." && pwd)"
|
|
SUBMODULE_DIR="${WORKSPACE_DIR}/packages/argent-private/packages/native-devtools-android"
|
|
SRC_DIR="${SUBMODULE_DIR}/Sources/AndroidDevtools"
|
|
BIN_DIR="${PKG_DIR}/bin"
|
|
BUILD_DIR="${PKG_DIR}/.build"
|
|
|
|
VERSION="$(node -p "require('${PKG_DIR}/assets/manifest.json').versionName")"
|
|
VERSION_CODE="$(node -p "require('${PKG_DIR}/assets/manifest.json').versionCode")"
|
|
|
|
APK_OUT="${BIN_DIR}/argent-android-devtools-${VERSION}.apk"
|
|
mkdir -p "${BIN_DIR}"
|
|
|
|
# If a prebuilt APK is provided (CI on a non-Android host, or local PREBUILT
|
|
# override), copy it and skip the build pipeline.
|
|
if [[ -n "${PREBUILT_ANDROID_DEVTOOLS_APK:-}" ]]; then
|
|
echo "Using prebuilt APK from PREBUILT_ANDROID_DEVTOOLS_APK"
|
|
cp "${PREBUILT_ANDROID_DEVTOOLS_APK}" "${APK_OUT}"
|
|
echo "Done. APK at ${APK_OUT}"
|
|
exit 0
|
|
fi
|
|
|
|
# Verify the submodule is initialised before trying to build.
|
|
if [[ ! -d "${SRC_DIR}" ]]; then
|
|
echo "Error: Android source not found at ${SRC_DIR}" >&2
|
|
echo " Run: git submodule update --init packages/argent-private" >&2
|
|
echo " Or set PREBUILT_ANDROID_DEVTOOLS_APK=/path/to/argent-android-devtools.apk" >&2
|
|
exit 1
|
|
fi
|
|
|
|
if [[ -z "${ANDROID_HOME:-}" ]]; then
|
|
if [[ -d "${HOME}/Library/Android/sdk" ]]; then
|
|
export ANDROID_HOME="${HOME}/Library/Android/sdk"
|
|
elif [[ -d "${HOME}/Android/Sdk" ]]; then
|
|
export ANDROID_HOME="${HOME}/Android/Sdk"
|
|
else
|
|
echo "ANDROID_HOME is not set and no default SDK location found." >&2
|
|
echo "Set ANDROID_HOME or PREBUILT_ANDROID_DEVTOOLS_APK." >&2
|
|
exit 1
|
|
fi
|
|
fi
|
|
|
|
# Pick the highest-numbered build-tools directory that has d8 + aapt2 + zipalign + apksigner.
|
|
BUILD_TOOLS_DIR=""
|
|
for candidate in $(ls -1 "${ANDROID_HOME}/build-tools" 2>/dev/null | sort -V -r); do
|
|
bt="${ANDROID_HOME}/build-tools/${candidate}"
|
|
if [[ -x "${bt}/d8" && -x "${bt}/aapt2" && -x "${bt}/zipalign" && -x "${bt}/apksigner" ]]; then
|
|
BUILD_TOOLS_DIR="${bt}"
|
|
break
|
|
fi
|
|
done
|
|
|
|
if [[ -z "${BUILD_TOOLS_DIR}" ]]; then
|
|
echo "Could not locate a build-tools directory under ${ANDROID_HOME}/build-tools that contains d8 + aapt2 + zipalign + apksigner." >&2
|
|
exit 1
|
|
fi
|
|
|
|
# Pick the highest android-XX platform that ships android.jar.
|
|
ANDROID_JAR=""
|
|
for candidate in $(ls -1 "${ANDROID_HOME}/platforms" 2>/dev/null | sed 's/android-//' | sort -nr); do
|
|
jar="${ANDROID_HOME}/platforms/android-${candidate}/android.jar"
|
|
if [[ -f "${jar}" ]]; then
|
|
ANDROID_JAR="${jar}"
|
|
break
|
|
fi
|
|
done
|
|
|
|
if [[ -z "${ANDROID_JAR}" ]]; then
|
|
echo "Could not locate android.jar under ${ANDROID_HOME}/platforms/android-*/." >&2
|
|
exit 1
|
|
fi
|
|
|
|
# Keystore lives in the submodule. Rotating it requires a major version bump
|
|
# because `adb install -r` enforces signature match across upgrades.
|
|
KEYSTORE="${SRC_DIR}/debug.keystore"
|
|
if [[ ! -f "${KEYSTORE}" ]]; then
|
|
echo "Error: keystore missing from submodule at ${KEYSTORE}" >&2
|
|
exit 1
|
|
fi
|
|
|
|
echo "→ ANDROID_HOME=${ANDROID_HOME}"
|
|
echo "→ build-tools=${BUILD_TOOLS_DIR}"
|
|
echo "→ android.jar=${ANDROID_JAR}"
|
|
echo "→ submodule src=${SRC_DIR}"
|
|
echo "→ version=${VERSION} (code ${VERSION_CODE})"
|
|
|
|
rm -rf "${BUILD_DIR}"
|
|
mkdir -p "${BUILD_DIR}/classes" "${BUILD_DIR}/dex"
|
|
|
|
echo "→ javac"
|
|
JAVA_SOURCES=$(find "${SRC_DIR}" -name '*.java')
|
|
javac --release 11 \
|
|
-classpath "${ANDROID_JAR}" \
|
|
-d "${BUILD_DIR}/classes" \
|
|
${JAVA_SOURCES}
|
|
|
|
echo "→ d8"
|
|
"${BUILD_TOOLS_DIR}/d8" \
|
|
--min-api 23 \
|
|
--classpath "${ANDROID_JAR}" \
|
|
--output "${BUILD_DIR}/dex" \
|
|
$(find "${BUILD_DIR}/classes" -name '*.class')
|
|
|
|
echo "→ aapt2 link"
|
|
"${BUILD_TOOLS_DIR}/aapt2" link \
|
|
--manifest "${SRC_DIR}/AndroidManifest.xml" \
|
|
-I "${ANDROID_JAR}" \
|
|
--min-sdk-version 23 \
|
|
--target-sdk-version 36 \
|
|
--version-code "${VERSION_CODE}" \
|
|
--version-name "${VERSION}" \
|
|
-o "${BUILD_DIR}/unsigned.apk"
|
|
|
|
echo "→ zip classes.dex into APK"
|
|
(cd "${BUILD_DIR}/dex" && zip -j -q "${BUILD_DIR}/unsigned.apk" classes.dex)
|
|
|
|
echo "→ zipalign"
|
|
"${BUILD_TOOLS_DIR}/zipalign" -f 4 "${BUILD_DIR}/unsigned.apk" "${BUILD_DIR}/aligned.apk"
|
|
|
|
echo "→ apksigner"
|
|
"${BUILD_TOOLS_DIR}/apksigner" sign \
|
|
--ks "${KEYSTORE}" \
|
|
--ks-pass pass:android \
|
|
--key-pass pass:android \
|
|
--out "${APK_OUT}" \
|
|
"${BUILD_DIR}/aligned.apk"
|
|
|
|
echo "→ wrote ${APK_OUT} ($(stat -f%z "${APK_OUT}" 2>/dev/null || stat -c%s "${APK_OUT}") bytes)"
|
|
|
|
rm -rf "${BUILD_DIR}"
|