Files
software-mansion__argent/packages/tool-server/test/screenshot-default-scale.test.ts
Ignacy Łątka 08f91fffd1 perf(screenshot): lower the default capture scale to 0.25 (#878)
Screenshots are billed to an agent's context by **pixel area**, not by
file size — so PNG level, JPEG quality and codec choice buy nothing
here, and the default scale is the only real lever. `0.3` had never been
measured against what a model can actually read, so I measured it.

## What was run

Bluesky (`bsky.app`) built from source and driven on an iOS 18.6
simulator, signed in against the repo's own seeded dev PDS (`yarn
e2e:mock-server` + `e2eSignInAlice`) — a live feed, profile,
notifications, Explore and Settings, not marketing screens. Every image
came out of argent's own `screenshot` tool over the tool-server HTTP
API, so the downscaler under test is the shipped one.

**160 API calls**: 5 screens × 5 scales × 2 models × 3 reps, plus a
blind control per screen per model. Questions were the things an agent
actually needs off a screen — an `@handle` two rows down, a relative
timestamp, a follower count, a badge number, which tab carries the
underline, the label of the row above another row.

Two controls decide whether the numbers mean anything:

- **Blind pass.** Each model answered every question with *no image at
all*. The 6 of 30 questions it still got right from priors alone (the
screen title, the tab selected by default) were struck from scoring. 24
survived.
- **Counting split out.** "How many rows between X and Y" was unreliable
at *every* scale including full resolution — it measures arithmetic, not
legibility — so it is excluded from the headline and reported
separately.

## Result

Text-reading accuracy, 66 graded answers per cell. Token counts are the
API's own reported `input_tokens`, image column net of the ~345-token
prompt.

| scale | image tokens | Haiku 4.5 | Opus 5 |
|---|---|---|---|
| 1.0 | 2692 | 100.0% (66/66) | 100.0% (66/66) |
| 0.3 *(old default)* | 451 | 98.5% (65/66) | 100.0% (66/66) |
| **0.25** *(new default)* | **298** | **100.0% (66/66)** | **100.0%
(66/66)** |
| 0.2 | 196 | 98.5% (65/66) | 100.0% (66/66) |
| 0.15 | 114 | 81.8% (54/66) | 97.0% (64/66) |

0.25 is the only scale that scored perfectly for both models on every
repetition, and it costs **34% less context** than 0.3.

## The cliff, and why the margin

Every wrong answer at 0.15:

| question | truth | answered | model | |
|---|---|---|---|---|
| 2nd author's handle | `@bob.test` | `@carla` | Haiku ×3 | read the row
above |
| selected feed tab | `Following` | `Feeds` | Haiku ×3 | couldn't
resolve the underline |
| 4th profile tab | `Videos` | `Likes` | Haiku ×3 | drifted two
positions |
| 3rd settings row | `Moderation and content filters` | `Privacy and
security` | Haiku ×3 | off by one row |
| bell badge number | `1` | `UNREADABLE` | Opus ×2 | declined rather
than guess |

**Opus fails safe; Haiku fails silent.** Opus refused the unreadable
badge. Haiku returned a confidently wrong *neighbouring* row — the same
wrong answer three times out of three. For an agent that turns a read
into a tap, a confident off-by-one row is worse than a refusal, because
nothing downstream catches it.

## Why 0.25 and not 0.2

0.2 measured no worse than the old default and would have saved 57%
instead of 34%. I didn't take it: the evidence is five screens of one
app at one device size, and Bluesky sets fairly generous type. A denser
interface — a table, a settings screen with subtitles, a smaller phone —
reaches the floor sooner. 0.25 keeps two steps of margin above the cliff
instead of one. `ARGENT_SCREENSHOT_SCALE=0.2` stays available for anyone
who wants to spend that margin deliberately.

## Scope

- **Chromium is untouched.** Its default is 1.0 and its `scale` is
silently dropped without the optional `sharp` dep — that's #673's
territory and I'm not duplicating it. Worth noting the two interact:
once Chromium can downscale (via the in-repo `resizeDecodedPng`, as #673
suggests as a follow-up), it should pick up this default too, and it is
currently by far the most expensive platform per capture.
- **`ARGENT_SCREENSHOT_SCALE` is documented as `0.3` in
`argent-private/docs/environment-variables.md`** — a submodule, so it
needs its own PR to stay in sync.
- Stale prose fixed in passing: `getScreenshotScale` carried a `//
default: halve the resolution` comment next to a `0.3`, and
`vega-screen.ts` documented the fallback as `0.3`.

## Checks

`npm run build` clean, `tsc --noEmit -p
packages/tool-server/tsconfig.test.json` clean, eslint `--max-warnings
0` clean, prettier clean. Shard 1/4 of the tool-server suite: 88 files /
1261 tests pass; the four files touching screenshot scale pass
individually (22 tests).

The default had **no test coverage at all** — the constant could be
edited to any value and the suite stayed green. This adds one.
Mutation-checked: reverting the constant to `0.3` fails 6 of the 8 new
cases.

### On the red Unit tests check

Inherited, not introduced. This branch's Unit tests job fails on exactly
one test — `describe-tool.test.ts > states the limitation without
ordering a reboot when the degraded read returned a tree` — which is the
same single test failing on `main` itself (main has been red since #583,
this branch's base). Already tracked as #871 and fixed by #877.
`flow-composition.test.ts` passes in CI on both.

Everything else is green: ESLint, Prettier, Knip, Static checks,
SpiderShield, lockfile sync, Linux + Chromium E2E, macOS + Chromium E2E,
macOS + iOS E2E.
2026-08-21 14:47:00 +02:00

36 lines
1.0 KiB
TypeScript

import { afterEach, describe, expect, it } from "vitest";
import { getScreenshotScale } from "../src/utils/simulator-client";
// Every iOS/Android/tvOS/Vega capture with no explicit `scale` lands on this
// value, and the agent pays for it in context on every screenshot.
const ENV = "ARGENT_SCREENSHOT_SCALE";
afterEach(() => {
delete process.env[ENV];
});
describe("getScreenshotScale", () => {
it("defaults to 0.25 when the env var is unset", () => {
delete process.env[ENV];
expect(getScreenshotScale()).toBe(0.25);
});
it("uses a valid env override verbatim", () => {
process.env[ENV] = "0.5";
expect(getScreenshotScale()).toBe(0.5);
});
it("accepts the 1.0 boundary", () => {
process.env[ENV] = "1";
expect(getScreenshotScale()).toBe(1);
});
// Rejected rather than producing a zero-pixel or upscaled capture.
it.each(["0", "-0.5", "1.5", "abc", ""])("falls back to 0.25 for %j", (value) => {
process.env[ENV] = value;
expect(getScreenshotScale()).toBe(0.25);
});
});