Files
software-mansion__argent/packages/tool-server/test/watermark.test.ts
Ignacy Łątka 2251c7cf46 feat(screen-recording): screen-recording-start/stop tools with per-call stop reminders (#517)
## What

Adds a `screen-recording-start` / `screen-recording-stop` tool pair
(naming follows `native-profiler-start/stop`) that records the device
screen to an mp4 and returns it as a downloadable artifact.

Capture runs through **simulator-server** — the same backend
`screenshot` and every interaction tool already drive. simulator-server
publishes the device screen as an MJPEG stream (`/stream.mjpeg`), so
recording needs no platform-specific recorder: the tool-server
subscribes to that stream, paces the frames onto a fixed 30 fps
timeline, and pipes them into a single ffmpeg process that encodes — and
watermarks — straight to the final mp4. The stream is multi-subscriber,
so a preview window can stay open on the same device while recording.

- **One path for every device simulator-server drives** — iOS
simulators, Android emulators, and physical Android devices.
- **Native resolution, constant 30 fps** — h264, `+faststart`, no
post-processing pass.
- Session state lives in a per-device `ScreenRecordingSession` blueprint
(mirrors the native-profiler session): time-limit cap with
auto-finalize, unexpected-exit recovery (a capture that ended on its own
is still retrievable via stop), always-startable state reset on stop,
and dispose-on-shutdown cleanup through the registry.

## Why the frames are paced host-side

A device only emits a frame when the screen **changes**. Pointing ffmpeg
straight at the stream therefore produces a video whose length is the
sum of the moments something moved — a still screen collapses to a
fraction of a second — and an ffmpeg blocked on a silent socket does not
answer a stop signal promptly either (measured: 12 s late). Re-emitting
the last frame on a wall-clock schedule fixes both: **video duration
always equals real elapsed time**, and stop is a stdin EOF that
finalizes in ~60 ms. Duplicate frames cost almost nothing once encoded
(a 10 s recording of a static screen is ~1 MB).

## Finishing pass folded into capture

The Argent watermark (logo + `By @swmansion`, bottom-left, sized
relative to the frame) is stamped by the same ffmpeg invocation that
encodes the capture, so there is no second encode and no temp
intermediate. Its colour is chosen **per pixel** from the background
luma underneath it — the white logo over dark areas, a runtime-derived
near-black copy over light areas — so it keeps maximum contrast against
whatever is behind it. It sits at 20 % opacity and is opt-out via
`argent disable video-watermark` (read when the recording starts). Frame
geometry comes from the first JPEG's SOF header, so no `ffprobe` is
needed.

ffmpeg is now the recorder rather than an optional finisher, so a
missing ffmpeg fails `screen-recording-start` up front with an install
hint instead of silently degrading the output at stop time.

## Reminder plumbing

While a recording is running — or ended on its own but was never
retrieved — **every** tool result carries a `note` telling the agent the
recording is still going and how to stop it. It rides the same `note`
field as the update reminder but is per-call and unsuppressed; both
notes compose when they coincide. The note text switches to "already
ended, retrieve the file" once the cap fires or the capture dies.

## Skill

New `argent-screen-recording` skill (+ routing stanza in
`rules/argent.md`): documents the start → interact → stop workflow and
instructs the agent to schedule its own reminder/wakeup for the expected
end of the recording immediately after starting one (harness reminder
tool if available, otherwise a background `sleep` whose completion
notification pulls it back). Scores 10/10 on `scripts/grade-skills.mjs`.

## Verification

- Unit: 45 tests across `screen-recording.test.ts`,
`mjpeg-stream.test.ts`, `watermark.test.ts`,
`http-recording-note.test.ts`; full tool-server suite **2808 tests / 280
files green**; `tsc --noEmit -p tsconfig.test.json`, root `tsc --build`,
eslint, prettier all clean.
- E2E on a live tool-server (`POST /tools/...`), driving real
interactions between start and stop:
- **iOS sim** (iPhone 16 Pro Max, iOS 18.5): 1320×2868 h264,
`r_frame_rate=30/1`, 338 frames, container duration 11.27 s vs reported
`durationMs` 11314 — **within 50 ms of wall clock**. Extracted frames
show real screen content (mid app-switcher animation) with the watermark
rendered light-over-dark.
- **Android emulator** (Pixel 3a, API 34): 1080×2220 h264, 30 fps, 353
frames, duration 11.77 s vs `durationMs` 11793 — within 30 ms. 512 KB
for a mostly-static home screen.
- **Static screen** (no interaction at all, 10 s): 300 frames, duration
exactly 10.000 s — the case the previous `recordVideo` implementation
reduced to ~0.03 s.
- **Long silence then activity** (15 s untouched, then an app launch):
stream survived, no warning, frames after the silence show the new
screen. This caught a real bug during review — the connect timeout was
also acting as a socket *inactivity* timeout and tore the stream down at
10 s of stillness; fixed and covered by a regression test that fails
without the fix.
- **Time-limit cap**: `timeLimitSeconds: 5` self-finalized at 4.97 s /
149 frames, the per-call note flipped to "already ended (it hit its 5s
time limit)", and a stop issued 11 s later still returned the video with
the matching warning.

## Not supported (capability-gated with clear errors)

tvOS simulators, physical iPhones, Chromium apps, Vega, and remote
(`remote:`-prefixed) simulators — none of them expose a readable frame
stream.

---------

Co-authored-by: Ignacy Łątka <claude-hubert.gancarczyk@swmansion.com>
2026-07-23 21:14:37 +02:00

115 lines
5.2 KiB
TypeScript

import { describe, it, expect } from "vitest";
import { buildWatermarkGraph, computeWatermarkBox } from "../src/tools/screen-recording/watermark";
describe("computeWatermarkBox", () => {
it("sizes the box relative to frame width and pins it bottom-left", () => {
// 886x1920 portrait: 0.286*886~=254 wide, height keeps the 900:277 logo
// aspect (snapped even -> 78), side inset 0.03*886~=26, bottom inset
// 0.018*886~=16 (a touch lower), y = 1920 - h - bottomMargin.
const box = computeWatermarkBox({ width: 886, height: 1920 });
expect(box).toEqual({ w: 254, h: 78, x: 26, y: 1826 });
});
it("keeps every dimension even (yuv420p 4:2:0 crop/scale requirement)", () => {
for (const dims of [
{ width: 886, height: 1920 },
{ width: 1170, height: 2532 },
{ width: 2048, height: 2732 },
{ width: 1920, height: 886 },
]) {
const box = computeWatermarkBox(dims);
for (const v of [box.w, box.h, box.x, box.y]) expect(v % 2).toBe(0);
}
});
it("scales with resolution (wider frame -> wider watermark)", () => {
const phone = computeWatermarkBox({ width: 886, height: 1920 });
const tablet = computeWatermarkBox({ width: 2048, height: 2732 });
expect(tablet.w).toBeGreaterThan(phone.w);
// aspect of the box tracks the logo aspect on both
expect(phone.w / phone.h).toBeCloseTo(tablet.w / tablet.h, 1);
});
it("never places the box off the top of a tiny frame", () => {
const box = computeWatermarkBox({ width: 100, height: 20 });
expect(box.y).toBeGreaterThanOrEqual(0);
expect(box.w).toBeGreaterThanOrEqual(2);
expect(box.h).toBeGreaterThanOrEqual(2);
});
it("never lets the crop box exceed the frame, even for pathological aspect ratios", () => {
// A frame far wider than tall keeps the logo's 900:277 aspect, which would
// otherwise make the box taller than the frame. ffmpeg's crop aborts (-22)
// on a rectangle larger than the input and kills the whole recording, so
// the box must stay inside the frame on every axis regardless of shape.
for (const dims of [
{ width: 1000, height: 50 }, // 20:1, h would overflow the height
{ width: 2000, height: 30 }, // extreme wide-short
{ width: 40, height: 41 }, // sub-55px width, odd height rounding edge
]) {
const box = computeWatermarkBox(dims);
expect(box.x).toBeGreaterThanOrEqual(0);
expect(box.y).toBeGreaterThanOrEqual(0);
expect(box.x + box.w).toBeLessThanOrEqual(dims.width);
expect(box.y + box.h).toBeLessThanOrEqual(dims.height);
for (const v of [box.w, box.h, box.x, box.y]) expect(v % 2).toBe(0);
}
});
});
describe("buildWatermarkGraph", () => {
const graph = buildWatermarkGraph({ width: 886, height: 1920 });
it("normalizes to a constant 30fps before overlaying", () => {
expect(graph.startsWith("[0:v]fps=30,")).toBe(true);
});
it("fades the whole watermark to 20% opacity (80% transparent)", () => {
expect(graph).toContain("colorchannelmixer=aa=0.2");
});
it("derives a near-black twin of the white logo for light backgrounds", () => {
expect(graph).toContain("colorchannelmixer=rr=0.08:gg=0.08:bb=0.08");
});
it("chooses per pixel via a background-luma mask + maskedmerge", () => {
// ramp: white logo where bg luma <= 90, near-black where >= 165
expect(graph).toContain("lut=y='clip((165-val)/75*255,0,255)'");
expect(graph).toContain("maskedmerge");
});
it("samples the mask from exactly where the stamp lands (aligned crop+overlay)", () => {
const box = computeWatermarkBox({ width: 886, height: 1920 });
const crop = graph.match(/crop=(\d+):(\d+):(\d+):(\d+)/);
const overlay = graph.match(/overlay=(\d+):(\d+)/);
expect(crop).not.toBeNull();
expect(overlay).not.toBeNull();
// crop x/y must equal overlay x/y, else the mask reads the wrong region
expect(crop?.[3]).toBe(String(box.x));
expect(crop?.[4]).toBe(String(box.y));
expect(overlay?.[1]).toBe(String(box.x));
expect(overlay?.[2]).toBe(String(box.y));
});
it("evens an odd-resolution base before splitting so yuv420p can encode it", () => {
// iPhone 16 / 15 Pro / 15 / 14 Pro stream at 1179x2556 (odd width). Without
// an even base the overlayed output stays 1179 wide and libx264 rejects it
// ("width not divisible by 2"), killing the whole recording. Crop the base
// to 1178 up front, and derive the box from that so the mask stays inside.
const odd = buildWatermarkGraph({ width: 1179, height: 2556 });
expect(odd.startsWith("[0:v]fps=30,crop=1178:2556:0:0,split=2[base][under]")).toBe(true);
const box = computeWatermarkBox({ width: 1178, height: 2556 });
expect(box.x + box.w).toBeLessThanOrEqual(1178);
// the mask crop reads from within the evened base
const maskCrop = odd.match(/\[under\]crop=(\d+):(\d+):(\d+):(\d+)/);
expect(Number(maskCrop?.[3]) + Number(maskCrop?.[1])).toBeLessThanOrEqual(1178);
});
it("leaves an already-even frame's graph unchanged (no redundant base crop)", () => {
const even = buildWatermarkGraph({ width: 1320, height: 2868 });
expect(even.startsWith("[0:v]fps=30,split=2[base][under]")).toBe(true);
// the only crop is the mask crop; there is no base even-crop of the frame
expect(even).not.toMatch(/crop=1320:2868/);
});
});