mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
236db0077f
## Problem
The `await-ui-element` and `await-screen-idle` tools refused a remote
simulator. The tool server sent this error:
```
Tool 'await-ui-element' is not supported on ios-remote simulator (no ios-remote support declared).
```
Both tools poll the same accessibility tree as `describe`. The
`describe` tool reads that tree correctly on a remote simulator.
Therefore the tree source was never the problem.
## Cause
Each tool declares its platform support in a capability record. The
`describe` tool declares `appleRemote`. The two wait tools did not
declare it. The HTTP layer rejects the request before the tool starts.
Three more branches test `platform === "ios"`. That test is false for a
remote device. These branches select the tree source, the dependency
preflight, and the tvOS probe.
Nobody excluded the platform deliberately. Git history has no commit
that adds `appleRemote` to either file. The `appleRemote` concept
arrived on 2026-07-01. The `await-screen-idle` tool arrived two days
later.
## Effect on an agent session
The MCP layer calls `await-screen-idle` before each automatic
screenshot. On a remote simulator that call failed. The fallback then
slept for the full budget:
- 1500 ms for each tap or gesture
- 2000 ms for `open-url`
- 3000 ms for `launch-app` and `restart-app`
- 15000 ms for `run-sequence`
The user saw no error. The fallback exists for an old tool server, not
for a current server that refuses a device.
## Change
This change adds three lines to each tool:
1. The capability record declares `appleRemote: { simulator: true }`.
2. The tree fetch sends a remote device to `describeIos`. Before, a
remote device fell through to the Chromium branch.
3. The dependency preflight requires `sim-remote`, and not `xcrun`.
The `describe` tool uses the same three settings. The ax-service
blueprint sends the accessibility read through the sim-remote tunnel.
## Test result
I ran the branch tool server against a leased Argent Cloud runner. The
device was a remote iPhone 17 Pro on iOS 26.5. The app was Bluesky. The
app read its bundle from a local Metro server through the sim-remote
tunnel.
| ID | Case | Result |
| --- | --- | --- |
| T1 | `visible` by text | `success: true` |
| T2 | `exists` by identifier | `success: true` |
| T3 | `exists` by a second identifier | `success: true` |
| T4 | `text` condition matches the label | `success: true` |
| T5 | compound role and text selector | `success: true` |
| T6 | absent text gives `unmet` | `success: false` |
| T7 | wrong expected text gives `unmet` | note quotes the actual text |
| T8 | 5000 ms budget honoured | `elapsed: 5003` |
| T9 | `hidden` on an absent element | `success: true` |
| T10 | screen settles | `settled: true, polls: 2` |
| T11 | screen honours `minStableMs: 2000` | `waitedMs: 2265, polls: 8`
|
| T12 | wait across an app restart | `success: true, elapsed: 27301` |
| T13 | `hidden` after the restart | `success: true` |
| T14 | flow record boundary | UNKNOWN verdict |
All 14 cases passed. Two results give the strongest evidence:
- Case T12 blocked for 27.3 seconds across an app restart. The app
rebuilt its bundle, the element appeared, and the tool returned success.
- Case T6 returned `cause: "unmet"`. A failed tree read returns `cause:
"unreadable"`. Therefore the tool read the remote tree and judged the
condition.
### Unit tests
Five new unit tests drive both tools with a remote udid. They use a mock
accessibility service. Each test fails against `main` and passes with
this change.
Local checks:
- `npm run build`: clean
- `npm test -w @argent/tool-server`: 431 files, 6153 tests, 0 failures
- `npm run lint`: clean
- `npx prettier --check`: clean
## Note on flow records
The flow tools declare no capability. Therefore they accept a remote
device today. This change lets the re-probe reach `fetchTree`, which has
no support for `ios-remote`. The recorder catches that error. The
recorder then keeps the step and marks the verdict UNKNOWN, not
known-bad.
An earlier test asserted the refusal. This change inverts that test and
records the new behavior in its comment.
## Documentation
This change needs no documentation update. The tool reference lists
these two tools with a description only. That page has no platform
column, and no page describes remote simulators.
---------
Co-authored-by: Hubert Gancarczyk <claude-hubert.gancarczyk@swmansion.com>
149 lines
5.5 KiB
TypeScript
149 lines
5.5 KiB
TypeScript
import { beforeEach, describe, expect, it, vi } from "vitest";
|
|
import type { AXServiceApi, AXDescribeResponse } from "../src/blueprints/ax-service";
|
|
import { createAwaitScreenIdleTool } from "../src/tools/await-screen-idle";
|
|
import { __primeDepCacheForTests, __resetDepCacheForTests } from "../src/utils/check-deps";
|
|
|
|
// execute() probes the target's form factor (isTvOsSimulator) before polling —
|
|
// a real `xcrun simctl list` that never caches for this fake UDID, so it re-runs
|
|
// on every test and takes seconds under the parallel suite load. The device here
|
|
// is a plain phone shape, so pin the probe to false and keep the rest real.
|
|
vi.mock("../src/utils/ios-devices", async () => {
|
|
const actual = await vi.importActual<typeof import("../src/utils/ios-devices")>(
|
|
"../src/utils/ios-devices"
|
|
);
|
|
return { ...actual, isTvOsSimulator: async () => false };
|
|
});
|
|
|
|
const IOS_UDID = "AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA";
|
|
const FRAME = { x: 0.1, y: 0.4, width: 0.8, height: 0.05 };
|
|
|
|
// AX service that walks `responses` one per call, repeating the last — lets a
|
|
// test simulate a screen that is blank, then renders, then holds still.
|
|
function makeSequencedAXService(responses: AXDescribeResponse[]): AXServiceApi {
|
|
let i = 0;
|
|
return {
|
|
degraded: false,
|
|
describe: async () => responses[Math.min(i++, responses.length - 1)],
|
|
alertCheck: async () => false,
|
|
ping: async () => true,
|
|
};
|
|
}
|
|
|
|
function axResponse(elements: AXDescribeResponse["elements"]): AXDescribeResponse {
|
|
return { alertVisible: false, screenFrame: { width: 440, height: 956 }, elements };
|
|
}
|
|
|
|
function iosRegistry(ax: AXServiceApi) {
|
|
return {
|
|
resolveService: vi.fn(async (urn: string) => {
|
|
if (urn.startsWith("AXService:")) return ax;
|
|
throw new Error(`unexpected service: ${urn}`);
|
|
}),
|
|
} as any;
|
|
}
|
|
|
|
const content = () => axResponse([{ label: "Settings", frame: FRAME, traits: ["button"] }]);
|
|
|
|
describe("await-screen-idle tool", () => {
|
|
beforeEach(() => {
|
|
__resetDepCacheForTests();
|
|
__primeDepCacheForTests(["xcrun", "adb", "sim-remote"]);
|
|
});
|
|
|
|
it("exposes the await-screen-idle id", () => {
|
|
expect(createAwaitScreenIdleTool(iosRegistry({} as AXServiceApi)).id).toBe("await-screen-idle");
|
|
});
|
|
|
|
it("settles once content renders and holds still", async () => {
|
|
// blank, then the same content on every later poll → stable
|
|
const tool = createAwaitScreenIdleTool(
|
|
iosRegistry(makeSequencedAXService([axResponse([]), content()]))
|
|
);
|
|
|
|
const result = await tool.execute(
|
|
{},
|
|
{ udid: IOS_UDID, timeoutMs: 2000, pollIntervalMs: 10, minStableMs: 30 }
|
|
);
|
|
|
|
expect(result.settled).toBe(true);
|
|
expect(result.waitedMs).toBeGreaterThanOrEqual(30);
|
|
expect(result.waitedMs).toBeLessThan(2000);
|
|
expect(result.polls).toBeGreaterThan(1);
|
|
});
|
|
|
|
it("does not settle while the screen stays blank (times out)", async () => {
|
|
const tool = createAwaitScreenIdleTool(iosRegistry(makeSequencedAXService([axResponse([])])));
|
|
|
|
const result = await tool.execute(
|
|
{},
|
|
{ udid: IOS_UDID, timeoutMs: 60, pollIntervalMs: 10, minStableMs: 30 }
|
|
);
|
|
|
|
expect(result.settled).toBe(false);
|
|
expect(result.waitedMs).toBeGreaterThanOrEqual(60);
|
|
});
|
|
|
|
it("does not settle while content keeps changing (times out)", async () => {
|
|
// a different label every poll never holds for minStableMs
|
|
const changing = Array.from({ length: 30 }, (_, i) =>
|
|
axResponse([{ label: `item-${i}`, frame: FRAME, traits: ["button"] }])
|
|
);
|
|
const tool = createAwaitScreenIdleTool(iosRegistry(makeSequencedAXService(changing)));
|
|
|
|
const result = await tool.execute(
|
|
{},
|
|
{ udid: IOS_UDID, timeoutMs: 80, pollIntervalMs: 5, minStableMs: 40 }
|
|
);
|
|
|
|
expect(result.settled).toBe(false);
|
|
expect(result.waitedMs).toBeGreaterThanOrEqual(80);
|
|
});
|
|
|
|
it("settles on the first non-empty read when minStableMs is 0", async () => {
|
|
const tool = createAwaitScreenIdleTool(iosRegistry(makeSequencedAXService([content()])));
|
|
|
|
const result = await tool.execute(
|
|
{},
|
|
{ udid: IOS_UDID, timeoutMs: 2000, pollIntervalMs: 50, minStableMs: 0 }
|
|
);
|
|
|
|
expect(result.settled).toBe(true);
|
|
expect(result.polls).toBe(1);
|
|
});
|
|
|
|
// ── ios-remote (a cloud sim reached through sim-remote) ──────────────────
|
|
// The AX service is the same one the local branch resolves; the blueprint
|
|
// puts it on a TCP transport across the tunnel. These pin that a remote udid
|
|
// actually EXECUTES, not merely that the capability gate lets it through.
|
|
|
|
it("settles on ios-remote through the same AX path as a local sim", async () => {
|
|
const tool = createAwaitScreenIdleTool(
|
|
iosRegistry(makeSequencedAXService([axResponse([]), content()]))
|
|
);
|
|
|
|
const result = await tool.execute(
|
|
{},
|
|
{ udid: `remote:${IOS_UDID}`, timeoutMs: 2000, pollIntervalMs: 10, minStableMs: 20 }
|
|
);
|
|
|
|
expect(result.settled).toBe(true);
|
|
expect(result.polls).toBeGreaterThan(1);
|
|
});
|
|
|
|
it("does not settle on ios-remote while the screen keeps changing", async () => {
|
|
// a different label every poll never holds for minStableMs
|
|
const changing = Array.from({ length: 30 }, (_, i) =>
|
|
axResponse([{ label: `item-${i}`, frame: FRAME, traits: ["button"] }])
|
|
);
|
|
const tool = createAwaitScreenIdleTool(iosRegistry(makeSequencedAXService(changing)));
|
|
|
|
const result = await tool.execute(
|
|
{},
|
|
{ udid: `remote:${IOS_UDID}`, timeoutMs: 80, pollIntervalMs: 5, minStableMs: 40 }
|
|
);
|
|
|
|
expect(result.settled).toBe(false);
|
|
expect(result.waitedMs).toBeGreaterThanOrEqual(80);
|
|
});
|
|
});
|