## Summary
This adds raw normalized screen coordinates as first-class targets for
both `tap` and `long-press`. Gestures can now target either an element
selector or a point, while keeping the existing selector auto-wait and
off-screen diagnostics.
Supported forms include:
```yaml
- tap: { x: 0.5, y: 0.6 }
- tap: { on: { x: 0.5, y: 0.6 }, times: 2 }
- long-press: { x: 0.5, y: 0.6 }
- long-press: { on: { x: 0.5, y: 0.6 }, duration: 1200 }
```
## What changed
- Introduced a shared gesture-target parser for selector and point
targets.
- Added raw-point execution for `long-press` and shared point resolution
between tap and long-press.
- Validate coordinates as finite normalized `0–1` screen fractions,
rejecting pixels, malformed points, mixed selector/coordinate maps, and
unknown point keys.
- Canonicalize option-bearing coordinate gestures under `on`, keeping
option keys separate from target fields.
- Preserve coordinate targets in recording summaries and flow progress
reports instead of displaying an undefined selector.
- Added serialization validation so recording cannot persist a flow that
would fail when parsed for replay.
- Updated the flow authoring skill and tool description with the new
coordinate forms.
## Behavior notes
- Raw points act immediately and do not resolve or observe the UI tree.
- Selector targets retain settling, auto-wait, and the existing
`scroll-to` hint when an element cannot be found.
- Observation directives such as `type`, `await`, `assert`, and
`scroll-to` remain selector-only.
- The mixed spelling `tap: { x, y, times }` is now intentionally
rejected; use `tap: { on: { x, y }, times }` instead.
## Validation
- Full tool-server suite: 2,618 tests passed across 266 files.
- Added parser, serialization, recording, touch execution, Chromium
execution, reporting, and malformed-input coverage.
- Tool-server build, test typecheck, ESLint, Prettier, and skill
validation passed.
## The bug (found in the 0.15.0 full-surface QA, on both the Android
emulator and a physical S24)
Recording a `gesture-tap` on an icon-font tab-bar item wrote a flow step
that displays as an **empty selector**, with the coordinates gone:
```yaml
steps:
- tap:
text: # ← actually U+E163, an icon-font Private Use Area glyph
```
and `flow-execute` then **silently passed** against target `""`, tapping
an element no reader of the flow can predict. Root cause (hexdump of the
QA-recorded file: `ee 85 a3` = U+E163): icon fonts expose their glyph as
the element's accessibility label; `deriveSelector` accepted any label
that survived `trim()`, so a PUA code point counted as "stable text".
This violated the rewrite's own documented contract — *"otherwise
coordinates are kept with a warning"* — whose fallback path existed and
worked, but never ran for this case.
## The fix — one predicate, three layers
`hasVisibleText()` (strip Unicode `Cf` format / `Co` private-use / `Cc`
control, then trim; require something left):
1. **`deriveSelector`** — invisible-only label/value is not stable text;
fall through to role, then to the recorder's existing kept-coordinates
path.
2. **`selectorToYaml`** — the empty-text guard previously covered only
the bare-string spelling; it now also rejects empty/invisible text on
the strict `{text: …}` map form, so a hand-built selector fails loudly
instead of writing an unreadable flow file.
3. **`selectorSchema`** — parse-side refinement, so flow files already
recorded broken on 0.15.0 fail with an actionable message ("text must
contain at least one visible character … select by identifier or role
instead") instead of false-passing at replay. This also hardens the
`await-ui-element` tool surface, which shares the schema.
Text that merely *contains* an icon glyph alongside real text ("⟨U+E163⟩
Explore") stays valid throughout.
## Verification
- Unit: `deriveSelector` PUA/ZWSP/mixed cases, serialize + parse
rejection round-trips (explicit `\uXXXX` escapes, no invisible bytes in
the test source).
- Full tool-server suite: **2667 passed**.
- **Live on the Pixel_9 emulator against the original QA repro** (my-app
tab bar, same coordinates): recording now produces `tap: {x: 0.75, y:
0.947}` with the documented *"kept coordinates (brittle)"* warning — the
role fallback was correctly caught by the existing
resolves-to-a-different-element guard — replay passes 2/2 on the
coordinate targets, and executing the old broken YAML now fails parse
with the visible-character error.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Flows can now express double-taps declaratively:
```yaml
- tap: { on: Login, times: 2 } # selector double-tap
- tap: { x: 0.5, y: 0.57, times: 2 } # coordinate double-tap
```
The selector nests under `on` so option keys never share a map with
selector fields — zod would silently strip them; mixing errors with a
hint instead. `times: 1` normalizes to absent and the serializer emits
the minimal spelling, keeping parse/serialize exact inverses.
The taps dispatch as **one gesture** via `gesture-tap`'s new
`clickCount` (1–10): N Down/Up pairs ~100 ms apart on touch, escalating
CDP `clickCount` per click on Chromium (what makes `dblclick` actually
fire) — never N separate calls whose RPC gaps could miss the OS
double-tap window.
The recorder carries a recorded `clickCount` into `times`, so a recorded
double-tap can't silently replay as a single tap.
Jakub Piasecki