Files
software-mansion__argent/packages/tool-server/test/http-recording-note.test.ts
Ignacy Łątka 2251c7cf46 feat(screen-recording): screen-recording-start/stop tools with per-call stop reminders (#517)
## What

Adds a `screen-recording-start` / `screen-recording-stop` tool pair
(naming follows `native-profiler-start/stop`) that records the device
screen to an mp4 and returns it as a downloadable artifact.

Capture runs through **simulator-server** — the same backend
`screenshot` and every interaction tool already drive. simulator-server
publishes the device screen as an MJPEG stream (`/stream.mjpeg`), so
recording needs no platform-specific recorder: the tool-server
subscribes to that stream, paces the frames onto a fixed 30 fps
timeline, and pipes them into a single ffmpeg process that encodes — and
watermarks — straight to the final mp4. The stream is multi-subscriber,
so a preview window can stay open on the same device while recording.

- **One path for every device simulator-server drives** — iOS
simulators, Android emulators, and physical Android devices.
- **Native resolution, constant 30 fps** — h264, `+faststart`, no
post-processing pass.
- Session state lives in a per-device `ScreenRecordingSession` blueprint
(mirrors the native-profiler session): time-limit cap with
auto-finalize, unexpected-exit recovery (a capture that ended on its own
is still retrievable via stop), always-startable state reset on stop,
and dispose-on-shutdown cleanup through the registry.

## Why the frames are paced host-side

A device only emits a frame when the screen **changes**. Pointing ffmpeg
straight at the stream therefore produces a video whose length is the
sum of the moments something moved — a still screen collapses to a
fraction of a second — and an ffmpeg blocked on a silent socket does not
answer a stop signal promptly either (measured: 12 s late). Re-emitting
the last frame on a wall-clock schedule fixes both: **video duration
always equals real elapsed time**, and stop is a stdin EOF that
finalizes in ~60 ms. Duplicate frames cost almost nothing once encoded
(a 10 s recording of a static screen is ~1 MB).

## Finishing pass folded into capture

The Argent watermark (logo + `By @swmansion`, bottom-left, sized
relative to the frame) is stamped by the same ffmpeg invocation that
encodes the capture, so there is no second encode and no temp
intermediate. Its colour is chosen **per pixel** from the background
luma underneath it — the white logo over dark areas, a runtime-derived
near-black copy over light areas — so it keeps maximum contrast against
whatever is behind it. It sits at 20 % opacity and is opt-out via
`argent disable video-watermark` (read when the recording starts). Frame
geometry comes from the first JPEG's SOF header, so no `ffprobe` is
needed.

ffmpeg is now the recorder rather than an optional finisher, so a
missing ffmpeg fails `screen-recording-start` up front with an install
hint instead of silently degrading the output at stop time.

## Reminder plumbing

While a recording is running — or ended on its own but was never
retrieved — **every** tool result carries a `note` telling the agent the
recording is still going and how to stop it. It rides the same `note`
field as the update reminder but is per-call and unsuppressed; both
notes compose when they coincide. The note text switches to "already
ended, retrieve the file" once the cap fires or the capture dies.

## Skill

New `argent-screen-recording` skill (+ routing stanza in
`rules/argent.md`): documents the start → interact → stop workflow and
instructs the agent to schedule its own reminder/wakeup for the expected
end of the recording immediately after starting one (harness reminder
tool if available, otherwise a background `sleep` whose completion
notification pulls it back). Scores 10/10 on `scripts/grade-skills.mjs`.

## Verification

- Unit: 45 tests across `screen-recording.test.ts`,
`mjpeg-stream.test.ts`, `watermark.test.ts`,
`http-recording-note.test.ts`; full tool-server suite **2808 tests / 280
files green**; `tsc --noEmit -p tsconfig.test.json`, root `tsc --build`,
eslint, prettier all clean.
- E2E on a live tool-server (`POST /tools/...`), driving real
interactions between start and stop:
- **iOS sim** (iPhone 16 Pro Max, iOS 18.5): 1320×2868 h264,
`r_frame_rate=30/1`, 338 frames, container duration 11.27 s vs reported
`durationMs` 11314 — **within 50 ms of wall clock**. Extracted frames
show real screen content (mid app-switcher animation) with the watermark
rendered light-over-dark.
- **Android emulator** (Pixel 3a, API 34): 1080×2220 h264, 30 fps, 353
frames, duration 11.77 s vs `durationMs` 11793 — within 30 ms. 512 KB
for a mostly-static home screen.
- **Static screen** (no interaction at all, 10 s): 300 frames, duration
exactly 10.000 s — the case the previous `recordVideo` implementation
reduced to ~0.03 s.
- **Long silence then activity** (15 s untouched, then an app launch):
stream survived, no warning, frames after the silence show the new
screen. This caught a real bug during review — the connect timeout was
also acting as a socket *inactivity* timeout and tore the stream down at
10 s of stillness; fixed and covered by a regression test that fails
without the fix.
- **Time-limit cap**: `timeLimitSeconds: 5` self-finalized at 4.97 s /
149 frames, the per-call note flipped to "already ended (it hit its 5s
time limit)", and a stop issued 11 s later still returned the video with
the matching warning.

## Not supported (capability-gated with clear errors)

tvOS simulators, physical iPhones, Chromium apps, Vega, and remote
(`remote:`-prefixed) simulators — none of them expose a readable frame
stream.

---------

Co-authored-by: Ignacy Łątka <claude-hubert.gancarczyk@swmansion.com>
2026-07-23 21:14:37 +02:00

231 lines
8.1 KiB
TypeScript

import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
import supertest from "supertest";
import { createHttpApp, type HttpAppHandle } from "../src/http";
import type { Registry } from "@argent/registry";
import {
__resetActiveScreenRecordingsForTesting,
buildScreenRecordingNote,
clearActiveScreenRecording,
getActiveScreenRecordings,
markScreenRecordingFinalized,
registerActiveScreenRecording,
} from "../src/utils/screen-recording-reminder";
// Keep the update note out of these assertions (same approach as the other
// http-*.test.ts files): no update available, never suppressed.
vi.mock("../src/utils/update-checker", () => ({
getUpdateState: vi.fn(() => ({
updateAvailable: false,
updateInstallable: false,
installableVersion: null,
latestVersion: null,
latestPublishedAt: null,
minReleaseAgeMs: 0,
currentVersion: "1.0.0",
})),
isUpdateNoteSuppressed: vi.fn(() => false),
suppressUpdateNote: vi.fn(),
}));
import { getUpdateState } from "../src/utils/update-checker";
const mockGetUpdateState = vi.mocked(getUpdateState);
function stubRegistry(toolResult: unknown = { ok: true }): Registry {
return {
getSnapshot: vi.fn(() => ({ services: new Map(), namespaces: [], tools: ["test-tool"] })),
getTool: vi.fn((name: string) => {
if (name === "test-tool") {
return {
id: "test-tool",
description: "A stub tool for testing",
inputSchema: { type: "object", properties: {} },
services: () => ({}),
execute: async () => toolResult,
};
}
return undefined;
}),
invokeTool: vi.fn(async () => toolResult),
} as unknown as Registry;
}
const UDID = "6DBF83B4-0000-0000-0000-000000000000";
describe("HTTP screen-recording reminder note", () => {
let handle: HttpAppHandle;
const request = supertest;
beforeEach(() => {
__resetActiveScreenRecordingsForTesting();
// Re-assert the no-update default: vi.clearAllMocks() clears call history
// but NOT a mockReturnValue installed by a previous test.
mockGetUpdateState.mockReturnValue({
updateAvailable: false,
updateInstallable: false,
installableVersion: null,
latestVersion: null,
latestPublishedAt: null,
minReleaseAgeMs: 0,
currentVersion: "1.0.0",
});
});
afterEach(() => {
handle?.dispose();
__resetActiveScreenRecordingsForTesting();
vi.clearAllMocks();
});
it("does NOT include a note when no recording is active", async () => {
handle = createHttpApp(stubRegistry());
const res = await request(handle.app).post("/tools/test-tool").send({}).expect(200);
expect(res.body).toHaveProperty("data");
expect(res.body).not.toHaveProperty("note");
});
it("attaches the reminder to every successful tool call while recording", async () => {
registerActiveScreenRecording(UDID, Date.now(), 180);
handle = createHttpApp(stubRegistry());
for (let call = 0; call < 3; call++) {
const res = await request(handle.app).post("/tools/test-tool").send({}).expect(200);
expect(res.body.note).toContain("screen recording is still running");
expect(res.body.note).toContain(UDID);
expect(res.body.note).toContain("screen-recording-stop");
}
});
it("tells the agent the exact stop call, including the udid parameter", async () => {
registerActiveScreenRecording(UDID, Date.now(), 60);
handle = createHttpApp(stubRegistry());
const res = await request(handle.app).post("/tools/test-tool").send({}).expect(200);
expect(res.body.note).toContain(`{ "udid": "${UDID}" }`);
expect(res.body.note).toContain("auto-stops after 60s");
// Shared-server caveat: other clients see this note too and must not stop
// a capture they don't own.
expect(res.body.note).toContain("another agent");
});
it("switches to the retrieval reminder once the capture finalized on its own", async () => {
registerActiveScreenRecording(UDID, Date.now(), 180);
markScreenRecordingFinalized(UDID, "it hit its 180s time limit");
handle = createHttpApp(stubRegistry());
const res = await request(handle.app).post("/tools/test-tool").send({}).expect(200);
expect(res.body.note).toContain("already ended");
expect(res.body.note).toContain("it hit its 180s time limit");
expect(res.body.note).toContain("screen-recording-stop");
});
it("drops the note once the recording is cleared", async () => {
registerActiveScreenRecording(UDID, Date.now(), 180);
handle = createHttpApp(stubRegistry());
await request(handle.app).post("/tools/test-tool").send({}).expect(200);
clearActiveScreenRecording(UDID);
const res = await request(handle.app).post("/tools/test-tool").send({}).expect(200);
expect(res.body).not.toHaveProperty("note");
});
it("lists every device when several recordings run concurrently", async () => {
registerActiveScreenRecording(UDID, Date.now(), 180);
registerActiveScreenRecording("emulator-5554", Date.now(), 120);
handle = createHttpApp(stubRegistry());
const res = await request(handle.app).post("/tools/test-tool").send({}).expect(200);
expect(res.body.note).toContain(UDID);
expect(res.body.note).toContain("emulator-5554");
});
it("joins the update note and the recording reminder when both apply", async () => {
mockGetUpdateState.mockReturnValue({
updateAvailable: true,
updateInstallable: true,
installableVersion: "1.2.3",
latestVersion: "1.2.3",
latestPublishedAt: null,
minReleaseAgeMs: 0,
currentVersion: "1.0.0",
});
registerActiveScreenRecording(UDID, Date.now(), 180);
handle = createHttpApp(stubRegistry());
const res = await request(handle.app).post("/tools/test-tool").send({}).expect(200);
expect(res.body.note).toContain("1.0.0 -> 1.2.3");
expect(res.body.note).toContain("screen-recording-stop");
});
it("does NOT include a note on tool error responses", async () => {
registerActiveScreenRecording(UDID, Date.now(), 180);
const errorRegistry = {
getSnapshot: vi.fn(() => ({ services: new Map(), namespaces: [], tools: ["test-tool"] })),
getTool: vi.fn(() => ({
id: "test-tool",
description: "A stub tool",
inputSchema: {},
services: () => ({}),
execute: async () => {
throw new Error("tool blew up");
},
})),
invokeTool: vi.fn(async () => {
throw new Error("tool blew up");
}),
} as unknown as Registry;
handle = createHttpApp(errorRegistry);
const res = await request(handle.app).post("/tools/test-tool").send({}).expect(500);
expect(res.body).toHaveProperty("error");
expect(res.body).not.toHaveProperty("note");
});
});
describe("screen-recording reminder state & note builder", () => {
beforeEach(() => {
__resetActiveScreenRecordingsForTesting();
});
it("register/clear round-trips and re-register resets a finalized entry", () => {
registerActiveScreenRecording(UDID, 1_000, 180);
markScreenRecordingFinalized(UDID, "it died");
expect(getActiveScreenRecordings()[0]?.status).toBe("finalized");
registerActiveScreenRecording(UDID, 2_000, 60);
const entry = getActiveScreenRecordings()[0]!;
expect(entry.status).toBe("recording");
expect(entry.startedAtMs).toBe(2_000);
clearActiveScreenRecording(UDID);
expect(getActiveScreenRecordings()).toHaveLength(0);
});
it("marking an unknown device as finalized is a no-op", () => {
markScreenRecordingFinalized("nope", "whatever");
expect(getActiveScreenRecordings()).toHaveLength(0);
});
it("formats elapsed time in minutes and seconds", () => {
const startedAt = 0;
registerActiveScreenRecording(UDID, startedAt, 300);
const note = buildScreenRecordingNote(getActiveScreenRecordings(), 125_000);
expect(note).toContain("started 2m 5s ago");
expect(note).toContain("auto-stops after 300s");
});
it("clamps a clock skew to zero elapsed instead of going negative", () => {
registerActiveScreenRecording(UDID, 10_000, 300);
const note = buildScreenRecordingNote(getActiveScreenRecordings(), 5_000);
expect(note).toContain("started 0s ago");
});
});