Files
actionbook__actionbook/packages/cli-v2/tests/e2e/iframe.rs
Alex f542a72f82 [packages/cli-v2] feat: iframe content expansion and frame-aware interaction (#406)
* [root]chore: add changeset for screenshot timeout fix

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [packages/cli-v2]feat: add iframe content expansion and frame-aware interaction support

Snapshot now automatically expands 1 level of iframe content, and all
interaction/observation commands route CDP operations to the correct
frame session for cross-origin iframe elements. This is fully transparent
to agents — no new commands or flags required.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [packages/cli-v2]fix: complete frame routing in screenshot annotate + clarify flag semantics

- get_rect_for_object() now uses execute_for_frame() for Runtime.callFunctionOn,
  fixing iframe element annotation boxes that would fail on cross-origin objectIds
- Design doc: replace overly broad "all flags work as before" with precise per-flag
  iframe semantics (--selector/--cursor main-frame only, --depth per-frame)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [packages/cli-v2]style: apply rustfmt formatting

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [packages/cli-v2]style: fix clippy collapsible_if warning

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [packages/cli-v2]test: add iframe E2E tests + RefCache unit tests + fix fill for iframe

- 7 E2E tests: snapshot expansion, interactive filter, html/click/fill/value
  on iframe refs, ref uniqueness across frames
- 4 RefCache unit tests: composite key collision, frame_id round-trip,
  cross-frame stability
- fix fill.rs: use Runtime.callFunctionOn(objectId) instead of
  document.activeElement to work correctly for iframe elements

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [packages/cli-v2]test: add cross-origin (OOPIF) iframe E2E tests

- Second HTTP server on different port for true cross-origin iframes
- 4 new tests: xo_iframe_snapshot_expands, xo_iframe_html, xo_iframe_fill,
  xo_iframe_click — all validate execute_for_frame() session routing
- Total iframe E2E coverage: 11 tests (7 same-origin + 4 cross-origin)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [packages/cli-v2]style: apply rustfmt to test files

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [packages/cli-v2]style: fix clippy useless_format in test harness

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [packages/cli-v2]refactor: rename execute_in_frame → execute_on_element

Rename frame-aware CDP execution methods to better reflect semantics:
- execute_in_frame → execute_on_element (TabContext method)
- execute_for_frame → execute_for_element (standalone helper)

The old names leaked the iframe implementation detail. The new names
express the caller's intent ("execute on the resolved element") without
implying frame awareness — consistent with the cli-v2 principle that
agents should not need to know about iframe boundaries.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [packages/cli-v2]refactor: restore execute_for_frame name for low-level helper

Layered naming convention:
- High-level (command files): ctx.execute_on_element() — named by intent
- Low-level (infrastructure): execute_for_frame() — named by mechanism

The low-level helper takes frame_id as input and routes by frame/session,
so execute_for_frame accurately describes what it does. Renaming it to
execute_for_element would obscure the mechanism it implements.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [packages/cli-v2]test: add screenshot --annotate and scroll into-view iframe E2E tests

Complete the iframe E2E test matrix:
- iframe_screenshot_annotate_includes_iframe_elements: verifies annotated
  screenshot succeeds with iframe refs in RefCache
- iframe_scroll_into_view_works_on_iframe_element: verifies scroll into-view
  routes callFunctionOn to correct frame session

Total: 13 iframe E2E tests (9 same-origin + 4 cross-origin)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 11:25:14 +08:00

632 lines
18 KiB
Rust

//! E2E tests for iframe content expansion and frame-aware interaction.
//! Covers both same-origin iframes and cross-origin (OOPIF) iframes.
use crate::harness::{
SessionGuard, assert_success, headless, headless_json, parse_json, skip, stdout_str,
unique_session, url_iframe_cross_origin_parent, url_iframe_parent, wait_page_ready,
};
fn start_iframe_session() -> (String, String, SessionGuard) {
let (sid, profile) = unique_session("iframe");
let out = headless_json(
&[
"browser",
"start",
"--mode",
"local",
"--headless",
"--set-session-id",
&sid,
"--profile",
&profile,
"--open-url",
&url_iframe_parent(),
],
30,
);
assert_success(&out, "start iframe session");
let v = parse_json(&out);
let sid = v["data"]["session"]["session_id"]
.as_str()
.unwrap()
.to_string();
let tid = v["data"]["tab"]["tab_id"].as_str().unwrap().to_string();
wait_page_ready(&sid, &tid);
// Give iframe a moment to load
std::thread::sleep(std::time::Duration::from_millis(500));
let guard = SessionGuard::new(&sid);
(sid, tid, guard)
}
// ── Snapshot expansion ────────────────────────────────────────────
#[test]
fn iframe_snapshot_expands_child_content() {
if skip() {
return;
}
let (sid, tid, _guard) = start_iframe_session();
let out = headless(
&["browser", "snapshot", "--session", &sid, "--tab", &tid],
15,
);
assert_success(&out, "snapshot");
let text = stdout_str(&out);
// Main page elements should be present
assert!(
text.contains("Main Page") || text.contains("heading"),
"main page heading should appear in snapshot"
);
// Iframe node should appear
assert!(
text.contains("Iframe"),
"Iframe node should appear in snapshot output"
);
// Child content should be expanded under the Iframe node
assert!(
text.contains("Child Content")
|| text.contains("Child Input")
|| text.contains("Child Button"),
"iframe child content should be expanded in snapshot.\nGot:\n{text}"
);
}
#[test]
fn iframe_snapshot_interactive_filter_includes_iframe_elements() {
if skip() {
return;
}
let (sid, tid, _guard) = start_iframe_session();
let out = headless(
&[
"browser",
"snapshot",
"-i",
"--session",
&sid,
"--tab",
&tid,
],
15,
);
assert_success(&out, "snapshot -i");
let text = stdout_str(&out);
// Interactive elements from both main page and iframe should have refs
assert!(
text.contains("[ref="),
"interactive snapshot should contain refs"
);
// The iframe's child button should appear with a ref
assert!(
text.contains("Child Button") || text.contains("Child Input"),
"iframe interactive elements should appear with -i flag.\nGot:\n{text}"
);
}
// ── Ref-based interaction with iframe elements ────────────────────
#[test]
fn iframe_html_reads_iframe_element() {
if skip() {
return;
}
let (sid, tid, _guard) = start_iframe_session();
// First snapshot to generate refs
let snap = headless_json(
&["browser", "snapshot", "--session", &sid, "--tab", &tid],
15,
);
assert_success(&snap, "snapshot for refs");
let snap_v = parse_json(&snap);
let content = snap_v["data"]["content"].as_str().unwrap_or("");
// Find the ref for "Child Button" in the snapshot content
let child_btn_ref = find_ref_for_name(content, "Child Button");
if child_btn_ref.is_empty() {
// If we can't find it by name, the iframe may not have expanded — skip gracefully
eprintln!(
"SKIP: could not find 'Child Button' ref in snapshot (iframe may not have loaded)"
);
return;
}
// Use html command with the iframe ref
let html_out = headless(
&[
"browser",
"html",
&format!("@{child_btn_ref}"),
"--session",
&sid,
"--tab",
&tid,
],
10,
);
assert_success(&html_out, "html on iframe element");
let html_text = stdout_str(&html_out);
assert!(
html_text.contains("child-btn") || html_text.contains("Click Me"),
"html should return iframe button content.\nGot:\n{html_text}"
);
}
#[test]
fn iframe_click_works_on_iframe_element() {
if skip() {
return;
}
let (sid, tid, _guard) = start_iframe_session();
// Snapshot to generate refs
let snap = headless_json(
&["browser", "snapshot", "--session", &sid, "--tab", &tid],
15,
);
assert_success(&snap, "snapshot");
let snap_v = parse_json(&snap);
let content = snap_v["data"]["content"].as_str().unwrap_or("");
let child_btn_ref = find_ref_for_name(content, "Child Button");
if child_btn_ref.is_empty() {
eprintln!("SKIP: could not find 'Child Button' ref");
return;
}
let click_out = headless(
&[
"browser",
"click",
&format!("@{child_btn_ref}"),
"--session",
&sid,
"--tab",
&tid,
],
10,
);
assert_success(&click_out, "click on iframe element");
}
#[test]
fn iframe_value_reads_iframe_input() {
if skip() {
return;
}
let (sid, tid, _guard) = start_iframe_session();
// Snapshot to generate refs
let snap = headless_json(
&["browser", "snapshot", "--session", &sid, "--tab", &tid],
15,
);
assert_success(&snap, "snapshot");
let snap_v = parse_json(&snap);
let content = snap_v["data"]["content"].as_str().unwrap_or("");
let child_input_ref = find_ref_for_name(content, "Child Input");
if child_input_ref.is_empty() {
eprintln!("SKIP: could not find 'Child Input' ref");
return;
}
let val_out = headless(
&[
"browser",
"value",
&format!("@{child_input_ref}"),
"--session",
&sid,
"--tab",
&tid,
],
10,
);
assert_success(&val_out, "value on iframe input");
let val_text = stdout_str(&val_out);
assert!(
val_text.contains("child-value"),
"value should return iframe input value.\nGot:\n{val_text}"
);
}
#[test]
fn iframe_fill_writes_to_iframe_input() {
if skip() {
return;
}
let (sid, tid, _guard) = start_iframe_session();
// Snapshot to generate refs
let snap = headless_json(
&["browser", "snapshot", "--session", &sid, "--tab", &tid],
15,
);
assert_success(&snap, "snapshot");
let snap_v = parse_json(&snap);
let content = snap_v["data"]["content"].as_str().unwrap_or("");
let child_input_ref = find_ref_for_name(content, "Child Input");
if child_input_ref.is_empty() {
eprintln!("SKIP: could not find 'Child Input' ref");
return;
}
// Fill the iframe input
let fill_out = headless(
&[
"browser",
"fill",
&format!("@{child_input_ref}"),
"new-iframe-value",
"--session",
&sid,
"--tab",
&tid,
],
10,
);
assert_success(&fill_out, "fill on iframe input");
// Read the value back to verify
let val_out = headless(
&[
"browser",
"value",
&format!("@{child_input_ref}"),
"--session",
&sid,
"--tab",
&tid,
],
10,
);
assert_success(&val_out, "value after fill");
let val_text = stdout_str(&val_out);
assert!(
val_text.contains("new-iframe-value"),
"value should reflect the filled text.\nGot:\n{val_text}"
);
}
// ── Ref isolation (no collision between frames) ───────────────────
#[test]
fn iframe_refs_do_not_collide_with_main_frame() {
if skip() {
return;
}
let (sid, tid, _guard) = start_iframe_session();
let snap = headless_json(
&["browser", "snapshot", "--session", &sid, "--tab", &tid],
15,
);
assert_success(&snap, "snapshot");
let snap_v = parse_json(&snap);
let nodes = snap_v["data"]["nodes"].as_array();
if let Some(nodes) = nodes {
// All refs should be unique
let refs: Vec<&str> = nodes.iter().filter_map(|n| n["ref"].as_str()).collect();
let unique: std::collections::HashSet<&str> = refs.iter().copied().collect();
assert_eq!(
refs.len(),
unique.len(),
"all refs must be unique (no collision between frames). Duplicates found."
);
}
}
// ── Screenshot --annotate with iframe refs ────────────────────────
#[test]
fn iframe_screenshot_annotate_includes_iframe_elements() {
if skip() {
return;
}
let (sid, tid, _guard) = start_iframe_session();
// Snapshot first to populate RefCache
let snap = headless_json(
&["browser", "snapshot", "--session", &sid, "--tab", &tid],
15,
);
assert_success(&snap, "snapshot");
// Take annotated screenshot
let tmp = tempfile::NamedTempFile::new().expect("create temp file");
let path = tmp.path().to_string_lossy().to_string() + ".png";
let out = headless_json(
&[
"browser",
"screenshot",
&path,
"--annotate",
"--session",
&sid,
"--tab",
&tid,
],
15,
);
assert_success(&out, "screenshot --annotate");
// Verify the file was created and has non-trivial size
let metadata = std::fs::metadata(&path);
assert!(
metadata.is_ok(),
"annotated screenshot file should exist at {path}"
);
assert!(
metadata.unwrap().len() > 1000,
"annotated screenshot should have non-trivial size"
);
let _ = std::fs::remove_file(&path);
}
// ── Scroll into-view on iframe element ────────────────────────────
#[test]
fn iframe_scroll_into_view_works_on_iframe_element() {
if skip() {
return;
}
let (sid, tid, _guard) = start_iframe_session();
// Snapshot to get refs
let snap = headless_json(
&["browser", "snapshot", "--session", &sid, "--tab", &tid],
15,
);
assert_success(&snap, "snapshot");
let snap_v = parse_json(&snap);
let content = snap_v["data"]["content"].as_str().unwrap_or("");
let child_btn_ref = find_ref_for_name(content, "Child Button");
if child_btn_ref.is_empty() {
eprintln!("SKIP: could not find 'Child Button' ref");
return;
}
let scroll_out = headless(
&[
"browser",
"scroll",
"into-view",
&format!("@{child_btn_ref}"),
"--session",
&sid,
"--tab",
&tid,
],
10,
);
assert_success(&scroll_out, "scroll into-view on iframe element");
}
// ══════════════════════════════════════════════════════════════════
// Cross-origin (OOPIF) iframe tests
// ══════════════════════════════════════════════════════════════════
fn start_xo_iframe_session() -> (String, String, SessionGuard) {
let (sid, profile) = unique_session("xo-iframe");
let out = headless_json(
&[
"browser",
"start",
"--mode",
"local",
"--headless",
"--set-session-id",
&sid,
"--profile",
&profile,
"--open-url",
&url_iframe_cross_origin_parent(),
],
30,
);
assert_success(&out, "start xo iframe session");
let v = parse_json(&out);
let sid = v["data"]["session"]["session_id"]
.as_str()
.unwrap()
.to_string();
let tid = v["data"]["tab"]["tab_id"].as_str().unwrap().to_string();
wait_page_ready(&sid, &tid);
// Give cross-origin iframe extra time to load and attach
std::thread::sleep(std::time::Duration::from_millis(1000));
let guard = SessionGuard::new(&sid);
(sid, tid, guard)
}
#[test]
fn xo_iframe_snapshot_expands_cross_origin_content() {
if skip() {
return;
}
let (sid, tid, _guard) = start_xo_iframe_session();
let out = headless(
&["browser", "snapshot", "--session", &sid, "--tab", &tid],
15,
);
assert_success(&out, "xo snapshot");
let text = stdout_str(&out);
// Iframe node should appear
assert!(
text.contains("Iframe"),
"Iframe node should appear in snapshot"
);
// Cross-origin child content should be expanded
assert!(
text.contains("Cross-Origin Content")
|| text.contains("XO Input")
|| text.contains("XO Button"),
"cross-origin iframe content should be expanded.\nGot:\n{text}"
);
}
#[test]
fn xo_iframe_html_reads_cross_origin_element() {
if skip() {
return;
}
let (sid, tid, _guard) = start_xo_iframe_session();
let snap = headless_json(
&["browser", "snapshot", "--session", &sid, "--tab", &tid],
15,
);
assert_success(&snap, "xo snapshot");
let snap_v = parse_json(&snap);
let content = snap_v["data"]["content"].as_str().unwrap_or("");
let xo_btn_ref = find_ref_for_name(content, "XO Button");
if xo_btn_ref.is_empty() {
eprintln!("SKIP: could not find 'XO Button' ref (cross-origin iframe may not have loaded)");
return;
}
let html_out = headless(
&[
"browser",
"html",
&format!("@{xo_btn_ref}"),
"--session",
&sid,
"--tab",
&tid,
],
10,
);
assert_success(&html_out, "html on xo iframe element");
let html_text = stdout_str(&html_out);
assert!(
html_text.contains("xo-btn") || html_text.contains("XO Click"),
"html should return cross-origin iframe button content.\nGot:\n{html_text}"
);
}
#[test]
fn xo_iframe_fill_writes_to_cross_origin_input() {
if skip() {
return;
}
let (sid, tid, _guard) = start_xo_iframe_session();
let snap = headless_json(
&["browser", "snapshot", "--session", &sid, "--tab", &tid],
15,
);
assert_success(&snap, "xo snapshot");
let snap_v = parse_json(&snap);
let content = snap_v["data"]["content"].as_str().unwrap_or("");
let xo_input_ref = find_ref_for_name(content, "XO Input");
if xo_input_ref.is_empty() {
eprintln!("SKIP: could not find 'XO Input' ref");
return;
}
// Fill
let fill_out = headless(
&[
"browser",
"fill",
&format!("@{xo_input_ref}"),
"xo-filled",
"--session",
&sid,
"--tab",
&tid,
],
10,
);
assert_success(&fill_out, "fill on xo iframe input");
// Read back
let val_out = headless(
&[
"browser",
"value",
&format!("@{xo_input_ref}"),
"--session",
&sid,
"--tab",
&tid,
],
10,
);
assert_success(&val_out, "value after xo fill");
let val_text = stdout_str(&val_out);
assert!(
val_text.contains("xo-filled"),
"value should reflect filled text in cross-origin iframe.\nGot:\n{val_text}"
);
}
#[test]
fn xo_iframe_click_works_on_cross_origin_element() {
if skip() {
return;
}
let (sid, tid, _guard) = start_xo_iframe_session();
let snap = headless_json(
&["browser", "snapshot", "--session", &sid, "--tab", &tid],
15,
);
assert_success(&snap, "xo snapshot");
let snap_v = parse_json(&snap);
let content = snap_v["data"]["content"].as_str().unwrap_or("");
let xo_btn_ref = find_ref_for_name(content, "XO Button");
if xo_btn_ref.is_empty() {
eprintln!("SKIP: could not find 'XO Button' ref");
return;
}
let click_out = headless(
&[
"browser",
"click",
&format!("@{xo_btn_ref}"),
"--session",
&sid,
"--tab",
&tid,
],
10,
);
assert_success(&click_out, "click on xo iframe element");
}
// ── Helper ────────────────────────────────────────────────────────
/// Find the ref ID (e.g. "e42") for an element with the given name in snapshot content.
fn find_ref_for_name(content: &str, name: &str) -> String {
for line in content.lines() {
if line.contains(name) {
// Look for [ref=eNN] pattern
if let Some(start) = line.find("[ref=") {
let after = &line[start + 5..];
if let Some(end) = after.find(']') {
return after[..end].to_string();
}
}
}
}
String::new()
}