Commit Graph

9 Commits

Author SHA1 Message Date
Hubert Gancarczyk f002fc9540 feat(telemetry): track Argent Lens design-review funnel (#478)
## What

Adds three privacy-safe telemetry events for the Argent Lens
design-review flow, capturing the human-decision funnel that the generic
`tool:*` path can't see (submission is an HTTP POST, and the
`await_user_selection` tool is hidden in `argent lens` CLI sessions):

- **`lens:preview_opened`** — a human loaded the preview UI (`GET
/preview/`), deduped per round.
- **`lens:round_completed`** — the human submitted their picks (once per
round, never on a resubmit).
- **`lens:round_abandoned`** — a staged round was discarded before
submit (window closed, `argent lens` exited mid-review, or the round
superseded).

## Privacy

Payloads carry only aggregate counts, booleans, durations, and the
device `platform` enum — never element names, comment text, variant
code, file paths, or raw device ids. The store emits privacy-safe
aggregate structs that are decoupled from the telemetry package;
structural assignability to the event props is enforced at the `track()`
call site (drift is a compile error).

## Event fields

- **`preview_opened`**: `round, element_count, variant_count,
is_cli_session, platform`
- **`round_completed`**: `round, element_count, variant_count,
annotation_count, element_comment_count, skipped_comment_count,
has_global_comment, is_cli_session, had_parked_await, round_duration_ms,
platform`
- **`round_abandoned`**: `round, element_count, variant_count,
had_parked_await, is_cli_session, platform`

## Testing

- Unit tests for all three events: per-round dedup, exactly-once
completion (no resubmit re-emit), abandoned vs happy-path roll, CLI
up-front open, and content-agnostic privacy assertions.
- Verified end-to-end against a live tool-server: all three events fire
with correct aggregate values and zero content leakage through the real
sanitize pipeline.

---------

Co-authored-by: Ignacy Łątka <latkaignacy@gmail.com>
2026-07-13 16:56:49 +02:00
Ignacy Łątka a92c859d07 feat(argent-lens): argent lens — bind a claude session to the preview window (#416)
## What

Adds `argent lens` — open Argent Lens bound to a fresh `claude` session.

## Why

The Lens preview window and the agent communicate through a shared,
process-wide tool-server that carries **no per-agent identity** (only a
coarse telemetry header). So when the user requests changes in the
window, there's no reliable way to route them to the *specific* agent
that opened it.

`argent lens` sidesteps the problem by inverting ownership: the human
launches Lens, and Lens spawns (and therefore owns) exactly one `claude`
terminal. The binding is 1:1 by construction — no identity plumbing
required.

## How it works

1. `argent lens` ensures a tool-server and **begins a CLI Lens
session**, which opens the preview window up front (no
`await_user_selection` needed) and keeps it open across rounds.
2. It spawns a tracked `claude` terminal in the current directory — a
TypeScript port of the macOS applet's `osascript` spawn/track mechanism
— seeded to propose variants without blocking.
3. A watcher polls `GET /preview/outcome`; on each submitted round it
types a one-line feedback summary into that terminal, queuing it to the
agent as its next prompt. Requesting changes therefore "talks back" to
the bound agent instead of closing the window.

macOS only (drives Terminal/iTerm via `osascript`); off-darwin it exits
with a clear message.

## Changes

- **CLI** `lens-terminal.ts` — `osascript` spawn/track + a new
write-into-running-session path (the "outputting" channel; injects input
into the running TUI, which a tty write cannot do).
- **CLI** `lens.ts` — the command + feedback formatter.
- **tool-server** — a `cliSession` flag in the variant store (opens the
window, suppresses auto-close on submit); `POST /preview/cli-session`
and `GET /preview/outcome` routes.
- **UI** — relabels the submit action to "Request changes" while a CLI
session is active.

Behind the existing `argent-lens` feature flag.

## Verification

- Build + typecheck across the workspace; ESLint (`--max-warnings 0`) +
Prettier clean.
- 34 new unit tests; full `@argent/cli` suite **118 passing**,
tool-server variant/preview suites **46 passing**.
- **E2E (observed on this machine):**
- **tool-server HTTP** — `cli-session` toggles `cliSession` true/false;
submitted feedback round-trips through `/preview/outcome`.
- **terminal injection** — spawned a window running `cat`, confirmed
`writeToSession` lands text in the running program's **stdin** (both
iTerm and Terminal), with embedded newlines flattened.
- **full relay** — the real `watchAndRelay` loop against a real
tool-server typed two distinct submission rounds' feedback into a
stand-in agent terminal (dedup on `completedAt` verified).
- `argent lens --help` and the feature-flag gate (exit 1 with guidance,
no spawn).

Note: a real end-to-end with a live `claude` agent was not run (it would
drive a paid interactive session); the agent-facing channel is proven by
injecting into a stand-in process's stdin via the exact same code path.

---------

Co-authored-by: Filip131311 <contact@filipkaminski.com>
2026-07-01 19:57:56 +02:00
Ignacy Łątka ba7bb8b72e fix(argent-lens): native window controls, comment cursor, card layout polish (#409)
Part 1 out of N reworks for Argent Lens

Polish pass on the Argent Lens preview window.

## Changes
- **Native window controls.** `titleBarStyle: "hidden"` (+
`setWindowButtonVisibility`) shows the real macOS traffic lights on the
transparent window instead of the handcrafted close button.
`trafficLightPosition` insets them to share the corner panels' 14px
padding.
- **Comment-mode cursor.** Entering comment mode swaps the crosshair for
a Figma-style speech-bubble cursor over the stream, so "you are placing
a comment" is unmistakable. Defined as a `--cursor-comment` data-URI
token in `theme.css`.
- **Bottom toolbelt.** The "Add comment" button moved out of the
top-right cluster into a new bottom-centre floating toolbelt — the seed
of a Figma-style bottom tool tab. Same button look, placement only.
- **Cards show all variants.** Two fixes so short cards no longer clip a
variant: the left column is no longer height-limited by the bottom-right
footbar (per-column bound), and tighter variant-row padding lets a
2-variant card fit its column.
- **Cards repel / never overlap.** A deterministic per-poll de-overlap
of the spring **home** targets (a stable top-down sweep), not a
per-frame force. The damped springs glide to non-overlapping rests: no
micro-jitter, no rubber-banding, and a card wedged between two settles
instead of wiggling forever.
- **App named "Argent Lens".** The window title was already set, but the
macOS menu bar / Cmd-Tab / Dock showed "Electron" (the unpackaged
`Electron.app` bundle's `CFBundleName`, which `app.setName` can't
override). On macOS the tool-server now launches the window through a
thin **wrapper `.app`** whose `Info.plist` `CFBundleName` is "Argent
Lens" — it symlinks Electron's Frameworks/Resources (no ~270MB copy) and
is launched with `--no-sandbox` (the modified/symlinked bundle no longer
matches the signed Helper apps). Acceptable for this window: it only
loads the tool-server's localhost UI, and the renderer keeps
`contextIsolation` + `sandbox:true`. Falls back to plain Electron on
non-macOS or any failure.

## Verification
Driven live against the iOS simulator via a dev tool-server + the
Electron window over CDP:
- Native traffic lights render on the transparent window; handcrafted
close button removed.
- Comment-mode `.stage img` computed cursor is the speech-bubble
data-URI.
- "Add comment" is in `#float-toolbelt`, horizontally centred at the
window mid-line, 14px above the bottom.
- All demo cards report `bodyScrollHeight === clientHeight` (zero clip);
both variants visible.
- 5- and 9-card layouts: zero overlaps, positions identical across 2s
(no jitter), and pinning a card between two settles with no overlap and
no wiggle.
- Menu bar / Cmd-Tab read "Argent Lens"; `ps` confirms the window
launches via `…/argent-lens-app/Argent Lens.app/Contents/MacOS/Electron
--no-sandbox`.
- `tsc --build`, `eslint`, `prettier --check`, and the preview/variant
test files all pass.
2026-06-25 16:28:40 +02:00
Ignacy Łątka ccf488ff79 fix(argent-lens): don't mark a visible element off-screen when the variant frame is a larger crop (#359)
## The bug
A variant proposed for the Favourites header (matched by text
**"Export"**) was marked **off-screen / hidden** in Argent Lens even
though the "Export" button was clearly visible. Found during E2E testing
of the Pokémon app.

Repro proposal:
- `match: {by:"text", value:"Export"}`
- `variant.frame: {x:0, y:0, width:1, height:0.205}`

## Root cause
`vpMatchNode` (the on-screen matcher) seeds the match **anchor** from
the proposal's `variant.frame`, then a **size gate** (`MAX_SIZE_RATIO =
2.5`) rejects candidates whose size differs too much from that seed.

But `variant.frame` is a thumbnail **crop region** — frequently a whole
component — while `match` can anchor on a small **sub-element** inside
it. Here the agent cropped the full-width header band (`{w:1, h:0.205}`)
but anchored on the tiny "Export" button (live frame `{x:0.743, y:0.109,
w:0.181, h:0.037}` from `describe`). The ~5.5× width gap tripped the
gate, which eliminated the **only** on-screen candidate → `vpMatchNode`
returned `null` → reported off-screen.

## Fix
Make the size gate a **preference**, not an eliminator
(`packages/ui/index.html`, `vpMatchNode`): keep the nearest candidate
even when the gate rejects it and fall back to it, so a clearly
on-screen element is never dropped. The gate still disambiguates among
multiple same-text candidates (size-matched wins), and the
locked-identity re-match path is unchanged.

## Verification
Headless-Chrome repro against the real UI, injecting the live Favourites
AX tree + the exact proposal:
- **before:** `onScreen: false` (bug reproduced)
- **after:** `onScreen: true`, matched "Export"

Full UI regression sweep (theme seed, stream-reconnect, off-screen pill,
comment-Escape, reduced-motion) still green.

Follow-up to #271 (Argent Lens).
2026-06-25 13:28:03 +02:00
Ignacy Łątka 1e8e5fa309 fix(argent-lens): preview-UI serving, variant matcher, and off-screen reveal (#394)
Three independent reliability fixes for **Argent Lens** (the
`propose_variant` / `await_user_selection` variant-review flow and its
preview window). They touch disjoint files and were validated
separately; combined here into one PR.

---

### 1. Preview window 404s on dot-path installs (nvm/fnm/volta/asdf)

The Lens window showed a raw `NotFoundError` instead of the variant UI
whenever argent was installed under a path containing a dot-segment
(most commonly nvm, `~/.nvm/...`). `GET /preview/` and
`/preview/theme.css` returned 404 while JSON routes worked.

Cause: the express 4 → 5 bump (#354, shipped in v0.12.1). Express 5's
`send@1` defaults to `dotfiles: "ignore"`, which 404s any file path
containing a dot-segment, and the preview routes pass the absolute
install path to `res.sendFile`. Proven in isolation: express 5.2.1
`sendFile` of the file → 404; express 4.22.1 → 200; `sendFile(basename,
{ root: dir })` → 200.

Fix: serve the preview UI via `sendFile(basename, { root: dir })` (a
`serveUiFile` helper) so the request path carries no dot-segment.
Regression test serves the UI from a temp dir whose name starts with
`.`.

> Note: the underlying `send@1` dot-path behaviour is a property of the
express-5 migration, not of Lens — these two routes are simply the only
`sendFile` calls in the tool-server today.

### 2. Variant fails to anchor right after navigation

A `propose_variant` made immediately after navigating to a screen
(exactly the navigate → screenshot → propose workflow) often failed to
anchor: the card floated/hid ("didn't register"), or latched onto a
same-text distractor (e.g. a bottom-bar tab instead of a header).

Cause: `captureElementFrame` describes the device **once** at propose
time, but iOS' accessibility tree is empty for ~1.25 s after a screen
appears and then fills incrementally (nav chrome before screen content)
— measured on a real device.

Fix: `captureElementFrame` now retries across the warm-up window and
**holds out for an exact label/identifier hit** (the intended element),
falling back to a substring match only after the budget.
`findElementMatch` / `matchFrameInTree` rank exact matches above
substring ones. Adds `match-element-frame-cold-describe.test.ts`; two
existing tests updated for the new retry signature.

### 3. Off-screen choices were unreachable

In the preview window, the off-screen reveal prompt (the "Show them"
card and the "N hidden — show" pill) was hidden unless **every on-screen
choice was already staged** (`onScreenStaged`). So a proposal whose
element was off-screen showed a hidden-choice count with no affordance
to surface it.

Fix: show the prompt whenever any element is off-screen (`offScreenCount
> 0 && !revealOffscreen && !submitting`), dropping the `onScreenStaged`
gate.

---

Supersedes #390, #392, #393 (same commits, combined).
2026-06-24 11:22:57 +02:00
Ignacy Łątka f3d108a5a1 feat: Argent Lens (#271)
Adds a UI for proposing and choosing between multiple agent-generated
variants on a running app.

**MCP tools** (`packages/tool-server/src/tools/variants/`)
- `propose_variant` — non-blocking; agent stages a variant for an
on-screen element (text/label/identifier/role matcher).
- `await_user_selection` — one blocking call; parks until the human
submits, then returns the picks + free-form annotations + a global
comment.

**Preview UI** (`packages/ui`)
- Floating cards anchored to live element frames via the existing
`describe` tree, with connector lines.
- Comment-mode inspector: hover spotlight (blur+dim everything else),
right-click any element to pin a comment.
- Sticky padded hitbox + trajectory-aware dismissal so focus doesn't
drop on cursor jitter.
- Footbar [Complete] [Clear], confirm modal for destructive actions,
green toast on submit.
- Single source of feel constants in `theme.css`.

**Electron host** (`packages/preview-window`)
- New workspace package. Frameless transparent BrowserWindow at fixed
size; the squeeze open/close is a CSS `transform: scaleY` on `<html>`,
GPU-composited — no OS-level resize.
- Tool-server spawns it on `await_user_selection` park, reuses it across
re-awaits, and asks it to animate-close on submit (1s grace so the toast
is visible).
- Stdio-line IPC (no extra ports). Ephemeral throughout — the renderer
hits the tool-server's already-ephemeral `/preview/*` HTTP.

---------

Co-authored-by: Filip131311 <contact@filipkaminski.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-17 15:22:04 +02:00
filip131311 037c474956 Feat: Baseline web UI for the simulator (#158)
This is a very base line UI: 

- integrates into the existing simulator server on tools server 
- does not expose any endpoints via mcp
2026-04-24 15:27:53 +02:00
Paweł Fornagiel da707875b1 feat: deprecate UI package (#45)
* feat: delete ui package

* feat: delete left tsconfig.buildinfo
2026-03-31 14:19:10 +02:00
Filip131311 575d973510 Add React/Vite/Tailwind UI package (packages/ui)
Implements the webview frontend for the simulator control server:
- App.tsx state machine: waiting-for-url → picking-device → session-active → token-needed
- MjpegCanvas: hidden img + rAF canvas draw with 2s decode watchdog for reconnect
- TouchSurface: pointer events → normalized [0,1] coords → touch/scroll API calls
- TokenRequiredOverlay: shown on 403 responses, requests token from host
- VsCodeAdapter / StandaloneAdapter for VS Code webview and standalone browser use
- ConnectView, DevicePickerView, SessionView
- Vite build, Tailwind CSS, vitest + @testing-library/react test setup

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 09:59:45 +01:00