mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
2251c7cf46
## What Adds a `screen-recording-start` / `screen-recording-stop` tool pair (naming follows `native-profiler-start/stop`) that records the device screen to an mp4 and returns it as a downloadable artifact. Capture runs through **simulator-server** — the same backend `screenshot` and every interaction tool already drive. simulator-server publishes the device screen as an MJPEG stream (`/stream.mjpeg`), so recording needs no platform-specific recorder: the tool-server subscribes to that stream, paces the frames onto a fixed 30 fps timeline, and pipes them into a single ffmpeg process that encodes — and watermarks — straight to the final mp4. The stream is multi-subscriber, so a preview window can stay open on the same device while recording. - **One path for every device simulator-server drives** — iOS simulators, Android emulators, and physical Android devices. - **Native resolution, constant 30 fps** — h264, `+faststart`, no post-processing pass. - Session state lives in a per-device `ScreenRecordingSession` blueprint (mirrors the native-profiler session): time-limit cap with auto-finalize, unexpected-exit recovery (a capture that ended on its own is still retrievable via stop), always-startable state reset on stop, and dispose-on-shutdown cleanup through the registry. ## Why the frames are paced host-side A device only emits a frame when the screen **changes**. Pointing ffmpeg straight at the stream therefore produces a video whose length is the sum of the moments something moved — a still screen collapses to a fraction of a second — and an ffmpeg blocked on a silent socket does not answer a stop signal promptly either (measured: 12 s late). Re-emitting the last frame on a wall-clock schedule fixes both: **video duration always equals real elapsed time**, and stop is a stdin EOF that finalizes in ~60 ms. Duplicate frames cost almost nothing once encoded (a 10 s recording of a static screen is ~1 MB). ## Finishing pass folded into capture The Argent watermark (logo + `By @swmansion`, bottom-left, sized relative to the frame) is stamped by the same ffmpeg invocation that encodes the capture, so there is no second encode and no temp intermediate. Its colour is chosen **per pixel** from the background luma underneath it — the white logo over dark areas, a runtime-derived near-black copy over light areas — so it keeps maximum contrast against whatever is behind it. It sits at 20 % opacity and is opt-out via `argent disable video-watermark` (read when the recording starts). Frame geometry comes from the first JPEG's SOF header, so no `ffprobe` is needed. ffmpeg is now the recorder rather than an optional finisher, so a missing ffmpeg fails `screen-recording-start` up front with an install hint instead of silently degrading the output at stop time. ## Reminder plumbing While a recording is running — or ended on its own but was never retrieved — **every** tool result carries a `note` telling the agent the recording is still going and how to stop it. It rides the same `note` field as the update reminder but is per-call and unsuppressed; both notes compose when they coincide. The note text switches to "already ended, retrieve the file" once the cap fires or the capture dies. ## Skill New `argent-screen-recording` skill (+ routing stanza in `rules/argent.md`): documents the start → interact → stop workflow and instructs the agent to schedule its own reminder/wakeup for the expected end of the recording immediately after starting one (harness reminder tool if available, otherwise a background `sleep` whose completion notification pulls it back). Scores 10/10 on `scripts/grade-skills.mjs`. ## Verification - Unit: 45 tests across `screen-recording.test.ts`, `mjpeg-stream.test.ts`, `watermark.test.ts`, `http-recording-note.test.ts`; full tool-server suite **2808 tests / 280 files green**; `tsc --noEmit -p tsconfig.test.json`, root `tsc --build`, eslint, prettier all clean. - E2E on a live tool-server (`POST /tools/...`), driving real interactions between start and stop: - **iOS sim** (iPhone 16 Pro Max, iOS 18.5): 1320×2868 h264, `r_frame_rate=30/1`, 338 frames, container duration 11.27 s vs reported `durationMs` 11314 — **within 50 ms of wall clock**. Extracted frames show real screen content (mid app-switcher animation) with the watermark rendered light-over-dark. - **Android emulator** (Pixel 3a, API 34): 1080×2220 h264, 30 fps, 353 frames, duration 11.77 s vs `durationMs` 11793 — within 30 ms. 512 KB for a mostly-static home screen. - **Static screen** (no interaction at all, 10 s): 300 frames, duration exactly 10.000 s — the case the previous `recordVideo` implementation reduced to ~0.03 s. - **Long silence then activity** (15 s untouched, then an app launch): stream survived, no warning, frames after the silence show the new screen. This caught a real bug during review — the connect timeout was also acting as a socket *inactivity* timeout and tore the stream down at 10 s of stillness; fixed and covered by a regression test that fails without the fix. - **Time-limit cap**: `timeLimitSeconds: 5` self-finalized at 4.97 s / 149 frames, the per-call note flipped to "already ended (it hit its 5s time limit)", and a stop issued 11 s later still returned the video with the matching warning. ## Not supported (capability-gated with clear errors) tvOS simulators, physical iPhones, Chromium apps, Vega, and remote (`remote:`-prefixed) simulators — none of them expose a readable frame stream. --------- Co-authored-by: Ignacy Łątka <claude-hubert.gancarczyk@swmansion.com>
115 lines
5.2 KiB
TypeScript
115 lines
5.2 KiB
TypeScript
import { describe, it, expect } from "vitest";
|
|
import { buildWatermarkGraph, computeWatermarkBox } from "../src/tools/screen-recording/watermark";
|
|
|
|
describe("computeWatermarkBox", () => {
|
|
it("sizes the box relative to frame width and pins it bottom-left", () => {
|
|
// 886x1920 portrait: 0.286*886~=254 wide, height keeps the 900:277 logo
|
|
// aspect (snapped even -> 78), side inset 0.03*886~=26, bottom inset
|
|
// 0.018*886~=16 (a touch lower), y = 1920 - h - bottomMargin.
|
|
const box = computeWatermarkBox({ width: 886, height: 1920 });
|
|
expect(box).toEqual({ w: 254, h: 78, x: 26, y: 1826 });
|
|
});
|
|
|
|
it("keeps every dimension even (yuv420p 4:2:0 crop/scale requirement)", () => {
|
|
for (const dims of [
|
|
{ width: 886, height: 1920 },
|
|
{ width: 1170, height: 2532 },
|
|
{ width: 2048, height: 2732 },
|
|
{ width: 1920, height: 886 },
|
|
]) {
|
|
const box = computeWatermarkBox(dims);
|
|
for (const v of [box.w, box.h, box.x, box.y]) expect(v % 2).toBe(0);
|
|
}
|
|
});
|
|
|
|
it("scales with resolution (wider frame -> wider watermark)", () => {
|
|
const phone = computeWatermarkBox({ width: 886, height: 1920 });
|
|
const tablet = computeWatermarkBox({ width: 2048, height: 2732 });
|
|
expect(tablet.w).toBeGreaterThan(phone.w);
|
|
// aspect of the box tracks the logo aspect on both
|
|
expect(phone.w / phone.h).toBeCloseTo(tablet.w / tablet.h, 1);
|
|
});
|
|
|
|
it("never places the box off the top of a tiny frame", () => {
|
|
const box = computeWatermarkBox({ width: 100, height: 20 });
|
|
expect(box.y).toBeGreaterThanOrEqual(0);
|
|
expect(box.w).toBeGreaterThanOrEqual(2);
|
|
expect(box.h).toBeGreaterThanOrEqual(2);
|
|
});
|
|
|
|
it("never lets the crop box exceed the frame, even for pathological aspect ratios", () => {
|
|
// A frame far wider than tall keeps the logo's 900:277 aspect, which would
|
|
// otherwise make the box taller than the frame. ffmpeg's crop aborts (-22)
|
|
// on a rectangle larger than the input and kills the whole recording, so
|
|
// the box must stay inside the frame on every axis regardless of shape.
|
|
for (const dims of [
|
|
{ width: 1000, height: 50 }, // 20:1, h would overflow the height
|
|
{ width: 2000, height: 30 }, // extreme wide-short
|
|
{ width: 40, height: 41 }, // sub-55px width, odd height rounding edge
|
|
]) {
|
|
const box = computeWatermarkBox(dims);
|
|
expect(box.x).toBeGreaterThanOrEqual(0);
|
|
expect(box.y).toBeGreaterThanOrEqual(0);
|
|
expect(box.x + box.w).toBeLessThanOrEqual(dims.width);
|
|
expect(box.y + box.h).toBeLessThanOrEqual(dims.height);
|
|
for (const v of [box.w, box.h, box.x, box.y]) expect(v % 2).toBe(0);
|
|
}
|
|
});
|
|
});
|
|
|
|
describe("buildWatermarkGraph", () => {
|
|
const graph = buildWatermarkGraph({ width: 886, height: 1920 });
|
|
|
|
it("normalizes to a constant 30fps before overlaying", () => {
|
|
expect(graph.startsWith("[0:v]fps=30,")).toBe(true);
|
|
});
|
|
|
|
it("fades the whole watermark to 20% opacity (80% transparent)", () => {
|
|
expect(graph).toContain("colorchannelmixer=aa=0.2");
|
|
});
|
|
|
|
it("derives a near-black twin of the white logo for light backgrounds", () => {
|
|
expect(graph).toContain("colorchannelmixer=rr=0.08:gg=0.08:bb=0.08");
|
|
});
|
|
|
|
it("chooses per pixel via a background-luma mask + maskedmerge", () => {
|
|
// ramp: white logo where bg luma <= 90, near-black where >= 165
|
|
expect(graph).toContain("lut=y='clip((165-val)/75*255,0,255)'");
|
|
expect(graph).toContain("maskedmerge");
|
|
});
|
|
|
|
it("samples the mask from exactly where the stamp lands (aligned crop+overlay)", () => {
|
|
const box = computeWatermarkBox({ width: 886, height: 1920 });
|
|
const crop = graph.match(/crop=(\d+):(\d+):(\d+):(\d+)/);
|
|
const overlay = graph.match(/overlay=(\d+):(\d+)/);
|
|
expect(crop).not.toBeNull();
|
|
expect(overlay).not.toBeNull();
|
|
// crop x/y must equal overlay x/y, else the mask reads the wrong region
|
|
expect(crop?.[3]).toBe(String(box.x));
|
|
expect(crop?.[4]).toBe(String(box.y));
|
|
expect(overlay?.[1]).toBe(String(box.x));
|
|
expect(overlay?.[2]).toBe(String(box.y));
|
|
});
|
|
|
|
it("evens an odd-resolution base before splitting so yuv420p can encode it", () => {
|
|
// iPhone 16 / 15 Pro / 15 / 14 Pro stream at 1179x2556 (odd width). Without
|
|
// an even base the overlayed output stays 1179 wide and libx264 rejects it
|
|
// ("width not divisible by 2"), killing the whole recording. Crop the base
|
|
// to 1178 up front, and derive the box from that so the mask stays inside.
|
|
const odd = buildWatermarkGraph({ width: 1179, height: 2556 });
|
|
expect(odd.startsWith("[0:v]fps=30,crop=1178:2556:0:0,split=2[base][under]")).toBe(true);
|
|
const box = computeWatermarkBox({ width: 1178, height: 2556 });
|
|
expect(box.x + box.w).toBeLessThanOrEqual(1178);
|
|
// the mask crop reads from within the evened base
|
|
const maskCrop = odd.match(/\[under\]crop=(\d+):(\d+):(\d+):(\d+)/);
|
|
expect(Number(maskCrop?.[3]) + Number(maskCrop?.[1])).toBeLessThanOrEqual(1178);
|
|
});
|
|
|
|
it("leaves an already-even frame's graph unchanged (no redundant base crop)", () => {
|
|
const even = buildWatermarkGraph({ width: 1320, height: 2868 });
|
|
expect(even.startsWith("[0:v]fps=30,split=2[base][under]")).toBe(true);
|
|
// the only crop is the mask crop; there is no base even-crop of the frame
|
|
expect(even).not.toMatch(/crop=1320:2868/);
|
|
});
|
|
});
|