filip131311
23b9a07760
## Why
The rule's `<tapping_rule>` ("Before **every** tap, you MUST call a
discovery tool") makes agents loop `tap → describe → tap → describe`:
over 35 baseline runs, `describe` was called 90 times against 86 taps —
one round-trip per tap, each a full model turn. The screen after an
action is already captured by the auto-screenshot; the element tree is
the only thing the agent still had to fetch.
## What
- **MCP** (`packages/argent-mcp`): after the action settles, call
`describe` and append `--- Elements after action (describe) ---` + the
tree to the tool result. Follows the same tool allow-list as the
auto-screenshot, skips `describe` itself, and sits inside the non-secret
branch so a `{{secret:…}}` value never re-enters context as text.
Spy-logged as `auto_describe` (duration, chars). MCP `instructions`
reworded.
- **Flag** `disable-auto-describe` (opt-out, mirrors
`disable-auto-screenshot`). Auto-describe has its own tool list
(`AUTO_DESCRIBE_TOOLS`) and its own branch, so the two features are
independent; the secret-placeholder skip and the screen-idle wait cover
both.
- **Rule + skill**: `tapping_rule` now says take coordinates from the
returned tree, call a discovery tool only when there is no fresh tree
for the screen (first action on a screen, after a wait, target not
listed); the two-failed-taps rule and the "never from pixels"
prohibition stay. `argent-device-interact` best-practice #6 and the
general-rules line updated to match.
- **Docs**: `reference/configuration.mdx` flag row,
`features/interacting-with-apps.mdx` discovery section.
- **Tests**: `autoDescribeEnabled` (global / project scope) and
`shouldAutoDescribe`.
## Evidence
A/B with `claude -p --model sonnet`, one build (variant env-gated during
the experiment), two sandbox projects differing only in rule/skill text
and the toggle; 7 tasks × 2 arms × 5 reps = 70 runs on the iPhone 17 Pro
simulator (iOS Settings ×3, Expo `my-app` ×2, Contacts create verified
in sqlite) and the Electron testbed.
| | main | this PR | Δ |
|---|---|---|---|
| Task success | 35/35 | 35/35 | = |
| Turns / run | 10.4 | 8.2 | −21% |
| Wall / run | 36.5 s | 28.0 s | −23% (new won 28/34 pairs) |
| Cost / run | $0.28 | $0.23 | −17% (27/34) |
| `describe` calls / run | 2.2 | 0.24 | −89% |
| Cache-read tokens / run | 622k | 421k | −32% |
(34 clean pairs; one pair excluded for external interference on the
simulator — with it included: wall −5%, cost −10%, turns −16%.) Every
iOS task moved 15–30% in favour of the new arm; Electron was neutral
(both arms describe once after `boot-device`, then a single
`run-sequence`). The appended tree costs ≈2.4k chars (~700 tokens) and a
median 73 ms per action; per-turn latency was unchanged, the saving is
the removed model round-trip.
Full report:
https://claude.ai/code/artifact/279528cf-ea92-4f52-9276-4f0475ba3e3c
## Caveats
- Sonnet only, short tasks, small screens. On feed-heavy apps with
10–20k-char trees the per-action overhead grows; a follow-up could cap
or filter the appended tree.
- Flags are read once at MCP start (same as `disable-auto-screenshot`),
so toggling needs an MCP restart.
## Checks
`vitest` argent-mcp (85) + configuration-core (122) green; `eslint
--max-warnings 0` on changed packages; `prettier --check .`; `npx
docusaurus build` green.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Interaction tools now return an accessibility element tree alongside
the latest screenshot.
- Agents can use fresh element data for follow-up actions without an
additional discovery step.
- **Configuration**
- Added the `disable-auto-describe` feature flag to turn off automatic
element trees.
- **Documentation**
- Updated interaction guidance and configuration references to explain
the new behavior and flag.
- **Bug Fixes**
- Secret-placeholder handling now correctly accounts for both
screenshots and element trees.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Argent documentation
The Argent documentation site, published at docs.swmansion.com/argent.
Built with Docusaurus and the shared Software Mansion docs theme, @swmansion/t-rex-ui.
The site has no landing page of its own. That is argent.swmansion.com; the root route here redirects to the getting started page.
Development
This site lives at packages/docs but is a standalone npm project with its own lockfile, excluded from the root packages/* workspace glob so its dependencies stay out of the toolkit's lockfile. ESLint still runs from the repo root (npm run lint) with the shared eslint.config.mjs; install this package's dependencies first so the type-aware rules can load tsconfig.json.
cd packages/docs
npm install
npm start # dev server on http://localhost:3000/argent/
npm run build # production build into packages/docs/build
npm run serve # serve the production build
Layout
| Path | Contents |
|---|---|
docs/ |
Documentation pages (MDX), grouped into sidebar categories |
src/pages/ |
The root route, which redirects into the docs |
src/css/ |
Argent color palette, typography and theme overrides |
src/components/ |
Components used from MDX, registered in src/theme/MDXComponents.tsx |
static/img/ |
Logos, favicon and other static assets |
static/video/ |
Encoded screen recordings, see Videos |
Colors in src/css/colors.css are derived from the Argent brand palette used on
argent.swmansion.com.
Videos
Screen recordings go through scripts/encode-video.sh, which produces a
web-sized MP4 and a poster frame in static/video/:
npm run encode:video -- ~/Desktop/tap-flow.mov
Record in the simulator or with QuickTime, then encode. Never commit a GIF: the same ten second capture is around thirty times larger as a GIF than as H.264.
Embed the result with the <Video> component, which is global in MDX and needs
no import:
<Video src="/video/tap-flow.mp4" portrait caption="Tapping through the sign-in flow" />
| Prop | Meaning |
|---|---|
src |
Site-relative path to the MP4 |
portrait |
Caps the height at 480px, for simulator and emulator captures |
poster |
Poster frame; defaults to the sibling .jpg the encode script writes |
caption |
Optional caption rendered below the clip |
width / height |
Intrinsic pixel size, reserves space so the page does not reflow |
Clips are muted and loop on their own. Nothing downloads until the player nears the viewport, playback pauses while it is off screen, and readers who ask for reduced motion get a paused player with controls instead of a loop.
Once static/video/ grows past a few tens of megabytes, move the files behind a
CDN and pass absolute URLs to src rather than growing the git history.