mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
2251c7cf46
## What Adds a `screen-recording-start` / `screen-recording-stop` tool pair (naming follows `native-profiler-start/stop`) that records the device screen to an mp4 and returns it as a downloadable artifact. Capture runs through **simulator-server** — the same backend `screenshot` and every interaction tool already drive. simulator-server publishes the device screen as an MJPEG stream (`/stream.mjpeg`), so recording needs no platform-specific recorder: the tool-server subscribes to that stream, paces the frames onto a fixed 30 fps timeline, and pipes them into a single ffmpeg process that encodes — and watermarks — straight to the final mp4. The stream is multi-subscriber, so a preview window can stay open on the same device while recording. - **One path for every device simulator-server drives** — iOS simulators, Android emulators, and physical Android devices. - **Native resolution, constant 30 fps** — h264, `+faststart`, no post-processing pass. - Session state lives in a per-device `ScreenRecordingSession` blueprint (mirrors the native-profiler session): time-limit cap with auto-finalize, unexpected-exit recovery (a capture that ended on its own is still retrievable via stop), always-startable state reset on stop, and dispose-on-shutdown cleanup through the registry. ## Why the frames are paced host-side A device only emits a frame when the screen **changes**. Pointing ffmpeg straight at the stream therefore produces a video whose length is the sum of the moments something moved — a still screen collapses to a fraction of a second — and an ffmpeg blocked on a silent socket does not answer a stop signal promptly either (measured: 12 s late). Re-emitting the last frame on a wall-clock schedule fixes both: **video duration always equals real elapsed time**, and stop is a stdin EOF that finalizes in ~60 ms. Duplicate frames cost almost nothing once encoded (a 10 s recording of a static screen is ~1 MB). ## Finishing pass folded into capture The Argent watermark (logo + `By @swmansion`, bottom-left, sized relative to the frame) is stamped by the same ffmpeg invocation that encodes the capture, so there is no second encode and no temp intermediate. Its colour is chosen **per pixel** from the background luma underneath it — the white logo over dark areas, a runtime-derived near-black copy over light areas — so it keeps maximum contrast against whatever is behind it. It sits at 20 % opacity and is opt-out via `argent disable video-watermark` (read when the recording starts). Frame geometry comes from the first JPEG's SOF header, so no `ffprobe` is needed. ffmpeg is now the recorder rather than an optional finisher, so a missing ffmpeg fails `screen-recording-start` up front with an install hint instead of silently degrading the output at stop time. ## Reminder plumbing While a recording is running — or ended on its own but was never retrieved — **every** tool result carries a `note` telling the agent the recording is still going and how to stop it. It rides the same `note` field as the update reminder but is per-call and unsuppressed; both notes compose when they coincide. The note text switches to "already ended, retrieve the file" once the cap fires or the capture dies. ## Skill New `argent-screen-recording` skill (+ routing stanza in `rules/argent.md`): documents the start → interact → stop workflow and instructs the agent to schedule its own reminder/wakeup for the expected end of the recording immediately after starting one (harness reminder tool if available, otherwise a background `sleep` whose completion notification pulls it back). Scores 10/10 on `scripts/grade-skills.mjs`. ## Verification - Unit: 45 tests across `screen-recording.test.ts`, `mjpeg-stream.test.ts`, `watermark.test.ts`, `http-recording-note.test.ts`; full tool-server suite **2808 tests / 280 files green**; `tsc --noEmit -p tsconfig.test.json`, root `tsc --build`, eslint, prettier all clean. - E2E on a live tool-server (`POST /tools/...`), driving real interactions between start and stop: - **iOS sim** (iPhone 16 Pro Max, iOS 18.5): 1320×2868 h264, `r_frame_rate=30/1`, 338 frames, container duration 11.27 s vs reported `durationMs` 11314 — **within 50 ms of wall clock**. Extracted frames show real screen content (mid app-switcher animation) with the watermark rendered light-over-dark. - **Android emulator** (Pixel 3a, API 34): 1080×2220 h264, 30 fps, 353 frames, duration 11.77 s vs `durationMs` 11793 — within 30 ms. 512 KB for a mostly-static home screen. - **Static screen** (no interaction at all, 10 s): 300 frames, duration exactly 10.000 s — the case the previous `recordVideo` implementation reduced to ~0.03 s. - **Long silence then activity** (15 s untouched, then an app launch): stream survived, no warning, frames after the silence show the new screen. This caught a real bug during review — the connect timeout was also acting as a socket *inactivity* timeout and tore the stream down at 10 s of stillness; fixed and covered by a regression test that fails without the fix. - **Time-limit cap**: `timeLimitSeconds: 5` self-finalized at 4.97 s / 149 frames, the per-call note flipped to "already ended (it hit its 5s time limit)", and a stop issued 11 s later still returned the video with the matching warning. ## Not supported (capability-gated with clear errors) tvOS simulators, physical iPhones, Chromium apps, Vega, and remote (`remote:`-prefixed) simulators — none of them expose a readable frame stream. --------- Co-authored-by: Ignacy Łątka <claude-hubert.gancarczyk@swmansion.com>
231 lines
8.1 KiB
TypeScript
231 lines
8.1 KiB
TypeScript
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
|
|
import supertest from "supertest";
|
|
import { createHttpApp, type HttpAppHandle } from "../src/http";
|
|
import type { Registry } from "@argent/registry";
|
|
import {
|
|
__resetActiveScreenRecordingsForTesting,
|
|
buildScreenRecordingNote,
|
|
clearActiveScreenRecording,
|
|
getActiveScreenRecordings,
|
|
markScreenRecordingFinalized,
|
|
registerActiveScreenRecording,
|
|
} from "../src/utils/screen-recording-reminder";
|
|
|
|
// Keep the update note out of these assertions (same approach as the other
|
|
// http-*.test.ts files): no update available, never suppressed.
|
|
vi.mock("../src/utils/update-checker", () => ({
|
|
getUpdateState: vi.fn(() => ({
|
|
updateAvailable: false,
|
|
updateInstallable: false,
|
|
installableVersion: null,
|
|
latestVersion: null,
|
|
latestPublishedAt: null,
|
|
minReleaseAgeMs: 0,
|
|
currentVersion: "1.0.0",
|
|
})),
|
|
isUpdateNoteSuppressed: vi.fn(() => false),
|
|
suppressUpdateNote: vi.fn(),
|
|
}));
|
|
|
|
import { getUpdateState } from "../src/utils/update-checker";
|
|
|
|
const mockGetUpdateState = vi.mocked(getUpdateState);
|
|
|
|
function stubRegistry(toolResult: unknown = { ok: true }): Registry {
|
|
return {
|
|
getSnapshot: vi.fn(() => ({ services: new Map(), namespaces: [], tools: ["test-tool"] })),
|
|
getTool: vi.fn((name: string) => {
|
|
if (name === "test-tool") {
|
|
return {
|
|
id: "test-tool",
|
|
description: "A stub tool for testing",
|
|
inputSchema: { type: "object", properties: {} },
|
|
services: () => ({}),
|
|
execute: async () => toolResult,
|
|
};
|
|
}
|
|
return undefined;
|
|
}),
|
|
invokeTool: vi.fn(async () => toolResult),
|
|
} as unknown as Registry;
|
|
}
|
|
|
|
const UDID = "6DBF83B4-0000-0000-0000-000000000000";
|
|
|
|
describe("HTTP screen-recording reminder note", () => {
|
|
let handle: HttpAppHandle;
|
|
const request = supertest;
|
|
|
|
beforeEach(() => {
|
|
__resetActiveScreenRecordingsForTesting();
|
|
// Re-assert the no-update default: vi.clearAllMocks() clears call history
|
|
// but NOT a mockReturnValue installed by a previous test.
|
|
mockGetUpdateState.mockReturnValue({
|
|
updateAvailable: false,
|
|
updateInstallable: false,
|
|
installableVersion: null,
|
|
latestVersion: null,
|
|
latestPublishedAt: null,
|
|
minReleaseAgeMs: 0,
|
|
currentVersion: "1.0.0",
|
|
});
|
|
});
|
|
|
|
afterEach(() => {
|
|
handle?.dispose();
|
|
__resetActiveScreenRecordingsForTesting();
|
|
vi.clearAllMocks();
|
|
});
|
|
|
|
it("does NOT include a note when no recording is active", async () => {
|
|
handle = createHttpApp(stubRegistry());
|
|
|
|
const res = await request(handle.app).post("/tools/test-tool").send({}).expect(200);
|
|
|
|
expect(res.body).toHaveProperty("data");
|
|
expect(res.body).not.toHaveProperty("note");
|
|
});
|
|
|
|
it("attaches the reminder to every successful tool call while recording", async () => {
|
|
registerActiveScreenRecording(UDID, Date.now(), 180);
|
|
handle = createHttpApp(stubRegistry());
|
|
|
|
for (let call = 0; call < 3; call++) {
|
|
const res = await request(handle.app).post("/tools/test-tool").send({}).expect(200);
|
|
expect(res.body.note).toContain("screen recording is still running");
|
|
expect(res.body.note).toContain(UDID);
|
|
expect(res.body.note).toContain("screen-recording-stop");
|
|
}
|
|
});
|
|
|
|
it("tells the agent the exact stop call, including the udid parameter", async () => {
|
|
registerActiveScreenRecording(UDID, Date.now(), 60);
|
|
handle = createHttpApp(stubRegistry());
|
|
|
|
const res = await request(handle.app).post("/tools/test-tool").send({}).expect(200);
|
|
|
|
expect(res.body.note).toContain(`{ "udid": "${UDID}" }`);
|
|
expect(res.body.note).toContain("auto-stops after 60s");
|
|
// Shared-server caveat: other clients see this note too and must not stop
|
|
// a capture they don't own.
|
|
expect(res.body.note).toContain("another agent");
|
|
});
|
|
|
|
it("switches to the retrieval reminder once the capture finalized on its own", async () => {
|
|
registerActiveScreenRecording(UDID, Date.now(), 180);
|
|
markScreenRecordingFinalized(UDID, "it hit its 180s time limit");
|
|
handle = createHttpApp(stubRegistry());
|
|
|
|
const res = await request(handle.app).post("/tools/test-tool").send({}).expect(200);
|
|
|
|
expect(res.body.note).toContain("already ended");
|
|
expect(res.body.note).toContain("it hit its 180s time limit");
|
|
expect(res.body.note).toContain("screen-recording-stop");
|
|
});
|
|
|
|
it("drops the note once the recording is cleared", async () => {
|
|
registerActiveScreenRecording(UDID, Date.now(), 180);
|
|
handle = createHttpApp(stubRegistry());
|
|
|
|
await request(handle.app).post("/tools/test-tool").send({}).expect(200);
|
|
clearActiveScreenRecording(UDID);
|
|
|
|
const res = await request(handle.app).post("/tools/test-tool").send({}).expect(200);
|
|
expect(res.body).not.toHaveProperty("note");
|
|
});
|
|
|
|
it("lists every device when several recordings run concurrently", async () => {
|
|
registerActiveScreenRecording(UDID, Date.now(), 180);
|
|
registerActiveScreenRecording("emulator-5554", Date.now(), 120);
|
|
handle = createHttpApp(stubRegistry());
|
|
|
|
const res = await request(handle.app).post("/tools/test-tool").send({}).expect(200);
|
|
|
|
expect(res.body.note).toContain(UDID);
|
|
expect(res.body.note).toContain("emulator-5554");
|
|
});
|
|
|
|
it("joins the update note and the recording reminder when both apply", async () => {
|
|
mockGetUpdateState.mockReturnValue({
|
|
updateAvailable: true,
|
|
updateInstallable: true,
|
|
installableVersion: "1.2.3",
|
|
latestVersion: "1.2.3",
|
|
latestPublishedAt: null,
|
|
minReleaseAgeMs: 0,
|
|
currentVersion: "1.0.0",
|
|
});
|
|
registerActiveScreenRecording(UDID, Date.now(), 180);
|
|
handle = createHttpApp(stubRegistry());
|
|
|
|
const res = await request(handle.app).post("/tools/test-tool").send({}).expect(200);
|
|
|
|
expect(res.body.note).toContain("1.0.0 -> 1.2.3");
|
|
expect(res.body.note).toContain("screen-recording-stop");
|
|
});
|
|
|
|
it("does NOT include a note on tool error responses", async () => {
|
|
registerActiveScreenRecording(UDID, Date.now(), 180);
|
|
const errorRegistry = {
|
|
getSnapshot: vi.fn(() => ({ services: new Map(), namespaces: [], tools: ["test-tool"] })),
|
|
getTool: vi.fn(() => ({
|
|
id: "test-tool",
|
|
description: "A stub tool",
|
|
inputSchema: {},
|
|
services: () => ({}),
|
|
execute: async () => {
|
|
throw new Error("tool blew up");
|
|
},
|
|
})),
|
|
invokeTool: vi.fn(async () => {
|
|
throw new Error("tool blew up");
|
|
}),
|
|
} as unknown as Registry;
|
|
handle = createHttpApp(errorRegistry);
|
|
|
|
const res = await request(handle.app).post("/tools/test-tool").send({}).expect(500);
|
|
|
|
expect(res.body).toHaveProperty("error");
|
|
expect(res.body).not.toHaveProperty("note");
|
|
});
|
|
});
|
|
|
|
describe("screen-recording reminder state & note builder", () => {
|
|
beforeEach(() => {
|
|
__resetActiveScreenRecordingsForTesting();
|
|
});
|
|
|
|
it("register/clear round-trips and re-register resets a finalized entry", () => {
|
|
registerActiveScreenRecording(UDID, 1_000, 180);
|
|
markScreenRecordingFinalized(UDID, "it died");
|
|
expect(getActiveScreenRecordings()[0]?.status).toBe("finalized");
|
|
|
|
registerActiveScreenRecording(UDID, 2_000, 60);
|
|
const entry = getActiveScreenRecordings()[0]!;
|
|
expect(entry.status).toBe("recording");
|
|
expect(entry.startedAtMs).toBe(2_000);
|
|
|
|
clearActiveScreenRecording(UDID);
|
|
expect(getActiveScreenRecordings()).toHaveLength(0);
|
|
});
|
|
|
|
it("marking an unknown device as finalized is a no-op", () => {
|
|
markScreenRecordingFinalized("nope", "whatever");
|
|
expect(getActiveScreenRecordings()).toHaveLength(0);
|
|
});
|
|
|
|
it("formats elapsed time in minutes and seconds", () => {
|
|
const startedAt = 0;
|
|
registerActiveScreenRecording(UDID, startedAt, 300);
|
|
const note = buildScreenRecordingNote(getActiveScreenRecordings(), 125_000);
|
|
expect(note).toContain("started 2m 5s ago");
|
|
expect(note).toContain("auto-stops after 300s");
|
|
});
|
|
|
|
it("clamps a clock skew to zero elapsed instead of going negative", () => {
|
|
registerActiveScreenRecording(UDID, 10_000, 300);
|
|
const note = buildScreenRecordingNote(getActiveScreenRecordings(), 5_000);
|
|
expect(note).toContain("started 0s ago");
|
|
});
|
|
});
|