mirror of
https://github.com/callstack/agent-device.git
synced 2026-09-14 20:06:34 +08:00
cba020de21
* fix: add iOS private AX snapshot fallback * fix: add public iOS snapshot query recovery * fix(ios): make private AX snapshot fallback recover deep React Native trees Four fixes that turn the #758 private AX fallback from works-on-one-tree-shape into reliable on Bluesky Home: - Depth ladder: the AX server rejects bulk snapshot requests outright (kAXErrorIllegalArgument) once requested depth crosses a tree-size-dependent limit that moves with live content. Retry at 56/40/24/12 instead of giving up after one attempt at 64. - Real attribute identifiers: the server silently ignored the raw keypath strings the bridge passed, so every node came back with a zero frame (breaking ref taps and the interactive/compact filters, which is why 'snapshot -i -c' stayed sparse). Map keypaths through XCElementSnapshot.axAttributesForElementSnapshotKeyPaths (it returns an NSSet) and drop the mapper's expensive extras (automation type, window display id, base type) that pushed deep requests past the 30s main-thread watchdog. - Viewport from the private root frame when the public windows query degrades to an infinite viewport, so off-screen drawer content stops passing the visibility filter. - Runner source fingerprint now includes .m/.h, so bridge edits stop reusing stale cached runner builds. Also hardens the bridge per review: UInt(exactly:) for untrusted element types, pid_t-sized objc_msgSend for process id matching, and objCType-checked NSValue frame decoding. * fix(ios): recover deadline-truncated near-empty compact snapshots The all-structural sparse detector misses the common large-RN-tree case where the typed-query sweep resolves one or two stray controls before its 1s deadline: the payload has 'content', so recovery never fires, yet 2 nodes is useless in practice. Treat deadline-truncated payloads with <= 8 nodes as needing recovery, and only replace the original payload when the recovered tree actually carries more nodes. Completed sweeps on legitimately minimal screens stay untouched (not truncated). * chore: fix CI for the AX snapshot fallback branch - Sync the setup metadata script's fingerprint extension list with the runtime (.m/.h were added for the ObjC bridge), fixing the cache metadata parity test. - Reduce find.ts complexity flagged by fallow: hoist the node fetcher into createFindNodeFetcher with a recoverSparseInteractiveSnapshot helper, split match disambiguation and resolution scoring into narrowMultipleMatches/resolvedTouchScore, extract rectsMatch. * feat(ios): make accessibility fallbacks and collapsed containers visible in snapshot output Two transparency gaps from #701's 'no silent fallback' requirement: - Runner-attached snapshot messages now surface as snapshot warnings (readAppleSnapshotResult previously dropped them), so every recovery through the fallback accessibility backend or query tier is announced, states what it usually means (the app publishes an unhealthy accessibility tree - fixing the app is the real cure), and points to screenshot as visual truth. - A leaf whose label merges many comma-joined segments is flagged as a collapsed accessible container: the app marks a container accessible, hiding every descendant from assistive tech and automation alike. Nothing can be recovered below it (VoiceOver sees the same merged element), so the warning names the node, estimates the merged label count, and gives the app-side fix plus the screenshot/coordinate-tap workaround. Validated live on the lab stress fixture (adlab://stress?accessible=1): the 6-node tree now carries '@e5 [Other] merges ~126 labels...'. * fix(ios): detect sparse trees with labeled roots and surface warnings through the daemon Validated against a real-world repro (a production React Native app's login screen, simulator build provided privately by the reporter): a full-screen accessibilityViewIsModal overlay leaves the public snapshot with just Application+Window. Two gaps kept recovery off: - The sparse detector counted the Application label (the app's display name) as content and the full-screen root as hittable, so the app name alone defeated recovery. Application/Window labels and root hittability say nothing about tree health and no longer count. - Interactor-level snapshot warnings were dropped by the daemon capture chain (only the runtime/commands layer kept them); they now thread through CaptureSnapshotResult into BackendSnapshotResult. With both fixes that login screen recovers through the public query tier: 16 nodes with every control addressable (fill @ref + read-back verified), and the output carries the recovery warning. Bluesky-class trees still ladder into the private fallback unchanged.
261 lines
7.8 KiB
Swift
261 lines
7.8 KiB
Swift
import XCTest
|
|
|
|
extension RunnerTests {
|
|
// MARK: - Blocking System Modal Snapshot
|
|
|
|
func blockingSystemAlertSnapshot() -> DataPayload? {
|
|
#if os(macOS)
|
|
return nil
|
|
#else
|
|
guard let modal = firstBlockingSystemModal(in: springboard) else {
|
|
return nil
|
|
}
|
|
let actions = actionableElements(in: modal)
|
|
guard !actions.isEmpty else {
|
|
return nil
|
|
}
|
|
|
|
let title = preferredSystemModalTitle(modal)
|
|
guard let modalNode = safeMakeSnapshotNode(
|
|
element: modal,
|
|
index: 0,
|
|
type: "Alert",
|
|
labelOverride: title,
|
|
identifierOverride: modal.identifier,
|
|
depth: 0,
|
|
hittableOverride: true
|
|
) else {
|
|
return nil
|
|
}
|
|
var nodes: [SnapshotNode] = [modalNode]
|
|
|
|
for content in informativeElements(in: modal, excluding: actions) {
|
|
guard let contentNode = safeMakeSnapshotNode(
|
|
element: content,
|
|
index: nodes.count,
|
|
type: elementTypeName(content.elementType),
|
|
depth: 1,
|
|
parentIndex: 0,
|
|
hittableOverride: false
|
|
) else {
|
|
continue
|
|
}
|
|
nodes.append(contentNode)
|
|
}
|
|
|
|
for action in actions {
|
|
guard let actionNode = safeMakeSnapshotNode(
|
|
element: action,
|
|
index: nodes.count,
|
|
type: elementTypeName(action.elementType),
|
|
depth: 1,
|
|
parentIndex: 0,
|
|
hittableOverride: true
|
|
) else {
|
|
continue
|
|
}
|
|
nodes.append(actionNode)
|
|
}
|
|
|
|
return DataPayload(nodes: nodes, truncated: false)
|
|
#endif
|
|
}
|
|
|
|
func firstBlockingSystemModal(in springboard: XCUIApplication) -> XCUIElement? {
|
|
let disableSafeProbe = RunnerEnv.isTruthy("AGENT_DEVICE_RUNNER_DISABLE_SAFE_MODAL_PROBE")
|
|
let queryElements: (() -> [XCUIElement]) -> [XCUIElement] = { fetch in
|
|
if disableSafeProbe {
|
|
return fetch()
|
|
}
|
|
return self.safeElementsQuery(fetch)
|
|
}
|
|
|
|
let alerts = queryElements {
|
|
springboard.alerts.allElementsBoundByIndex
|
|
}
|
|
for alert in alerts {
|
|
if safeIsBlockingSystemModal(alert, in: springboard) {
|
|
return alert
|
|
}
|
|
}
|
|
|
|
let sheets = queryElements {
|
|
springboard.sheets.allElementsBoundByIndex
|
|
}
|
|
for sheet in sheets {
|
|
if safeIsBlockingSystemModal(sheet, in: springboard) {
|
|
return sheet
|
|
}
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
func safeElementsQuery(_ fetch: () -> [XCUIElement]) -> [XCUIElement] {
|
|
safely("MODAL_QUERY", [], fetch)
|
|
}
|
|
|
|
private func safeIsBlockingSystemModal(_ element: XCUIElement, in springboard: XCUIApplication) -> Bool {
|
|
safely("MODAL_CHECK", false) { isBlockingSystemModal(element, in: springboard) }
|
|
}
|
|
|
|
private func isBlockingSystemModal(_ element: XCUIElement, in springboard: XCUIApplication) -> Bool {
|
|
guard element.exists else { return false }
|
|
let frame = element.frame
|
|
if frame.isNull || frame.isEmpty { return false }
|
|
|
|
let viewport = springboard.frame
|
|
if viewport.isNull || viewport.isEmpty { return false }
|
|
|
|
let center = CGPoint(x: frame.midX, y: frame.midY)
|
|
if !viewport.contains(center) { return false }
|
|
|
|
return true
|
|
}
|
|
|
|
func actionableElements(in element: XCUIElement) -> [XCUIElement] {
|
|
var seen = Set<String>()
|
|
var actions: [XCUIElement] = []
|
|
let descendants = actionableTypes.flatMap { modalDescendants(in: element, matching: $0) }
|
|
for candidate in descendants {
|
|
if !safeIsActionableCandidate(candidate, seen: &seen) { continue }
|
|
actions.append(candidate)
|
|
}
|
|
return actions
|
|
}
|
|
|
|
private func safeIsActionableCandidate(_ candidate: XCUIElement, seen: inout Set<String>) -> Bool {
|
|
safely("MODAL_ACTION", false) {
|
|
if !candidate.exists || !candidate.isHittable { return false }
|
|
if !actionableTypes.contains(candidate.elementType) { return false }
|
|
let frame = candidate.frame
|
|
if frame.isNull || frame.isEmpty { return false }
|
|
let key = "\(candidate.elementType.rawValue)-\(frame.origin.x)-\(frame.origin.y)-\(frame.size.width)-\(frame.size.height)-\(candidate.label)"
|
|
if seen.contains(key) { return false }
|
|
seen.insert(key)
|
|
return true
|
|
}
|
|
}
|
|
|
|
private func informativeElements(in element: XCUIElement, excluding actions: [XCUIElement]) -> [XCUIElement] {
|
|
let actionKeys = Set(actions.map(systemModalElementKey))
|
|
var seen = Set<String>()
|
|
var contents: [XCUIElement] = []
|
|
let descendants = readableSystemModalTypes.flatMap {
|
|
modalDescendants(in: element, matching: $0, limit: 2)
|
|
}
|
|
for candidate in descendants {
|
|
guard let key = safeInformativeElementKey(candidate, actionKeys: actionKeys) else {
|
|
continue
|
|
}
|
|
if seen.contains(key) { continue }
|
|
seen.insert(key)
|
|
contents.append(candidate)
|
|
}
|
|
return contents
|
|
}
|
|
|
|
private var readableSystemModalTypes: [XCUIElement.ElementType] {
|
|
[.staticText, .textView]
|
|
}
|
|
|
|
private func modalDescendants(
|
|
in element: XCUIElement,
|
|
matching type: XCUIElement.ElementType,
|
|
limit: Int? = nil
|
|
) -> [XCUIElement] {
|
|
let elements = safeElementsQuery {
|
|
element.descendants(matching: type).allElementsBoundByIndex
|
|
}
|
|
guard let limit else {
|
|
return elements
|
|
}
|
|
return Array(elements.prefix(limit))
|
|
}
|
|
|
|
private func safeInformativeElementKey(_ candidate: XCUIElement, actionKeys: Set<String>) -> String? {
|
|
safely("MODAL_CONTENT") { () -> String? in
|
|
let key = systemModalElementKey(candidate)
|
|
if actionKeys.contains(key) { return nil }
|
|
if actionableTypes.contains(candidate.elementType) { return nil }
|
|
if !candidate.exists { return nil }
|
|
let frame = candidate.frame
|
|
if frame.isNull || frame.isEmpty { return nil }
|
|
let label = candidate.label.trimmingCharacters(in: .whitespacesAndNewlines)
|
|
if label.isEmpty { return nil }
|
|
return key
|
|
}
|
|
}
|
|
|
|
private func systemModalElementKey(_ element: XCUIElement) -> String {
|
|
let frame = element.frame
|
|
return "\(element.elementType.rawValue)-\(frame.origin.x)-\(frame.origin.y)-\(frame.size.width)-\(frame.size.height)-\(element.label)-\(element.identifier)"
|
|
}
|
|
|
|
private func preferredSystemModalTitle(_ element: XCUIElement) -> String {
|
|
let label = element.label
|
|
if !label.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty {
|
|
return label
|
|
}
|
|
let identifier = element.identifier
|
|
if !identifier.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty {
|
|
return identifier
|
|
}
|
|
return "System Alert"
|
|
}
|
|
|
|
private func makeSnapshotNode(
|
|
element: XCUIElement,
|
|
index: Int,
|
|
type: String,
|
|
labelOverride: String? = nil,
|
|
identifierOverride: String? = nil,
|
|
depth: Int,
|
|
parentIndex: Int? = nil,
|
|
hittableOverride: Bool? = nil
|
|
) -> SnapshotNode {
|
|
let label = (labelOverride ?? element.label).trimmingCharacters(in: .whitespacesAndNewlines)
|
|
let identifier = (identifierOverride ?? element.identifier).trimmingCharacters(in: .whitespacesAndNewlines)
|
|
return SnapshotNode(
|
|
index: index,
|
|
type: type,
|
|
label: label.isEmpty ? nil : label,
|
|
identifier: identifier.isEmpty ? nil : identifier,
|
|
value: nil,
|
|
rect: snapshotRect(from: element.frame),
|
|
enabled: element.isEnabled,
|
|
focused: nil,
|
|
selected: nil,
|
|
hittable: hittableOverride ?? element.isHittable,
|
|
depth: depth,
|
|
parentIndex: parentIndex,
|
|
hiddenContentAbove: nil,
|
|
hiddenContentBelow: nil
|
|
)
|
|
}
|
|
|
|
private func safeMakeSnapshotNode(
|
|
element: XCUIElement,
|
|
index: Int,
|
|
type: String,
|
|
labelOverride: String? = nil,
|
|
identifierOverride: String? = nil,
|
|
depth: Int,
|
|
parentIndex: Int? = nil,
|
|
hittableOverride: Bool? = nil
|
|
) -> SnapshotNode? {
|
|
safely("MODAL_NODE") {
|
|
makeSnapshotNode(
|
|
element: element,
|
|
index: index,
|
|
type: type,
|
|
labelOverride: labelOverride,
|
|
identifierOverride: identifierOverride,
|
|
depth: depth,
|
|
parentIndex: parentIndex,
|
|
hittableOverride: hittableOverride
|
|
)
|
|
}
|
|
}
|
|
}
|