Get pixel location of each iframe element that's out of the viewport, convert from pixels to number of page lengths, and provide that to the LLM as context
Elements in iframes beyond the 1000px viewport threshold are filtered from selector_map, causing agents to scroll blindly to find them. Add hints to the LLM context indicating hidden content
uses CDP Runtime.evaluate with includeCommandLineAPI to access getEventListeners() and mark elements with click/mousedown/pointerdown listeners before DOM snapshot capture
- Added 'row', 'cell', 'gridcell' to interactive_roles in ClickableElementDetector
- Added 'row', 'cell', 'gridcell' to interactive_ax_roles in ClickableElementDetector
- Verified logic with isolated test case
- Changed the type of the all_frames parameter to allow for None, enabling lazy fetching of cross-origin iframes only when necessary.
- Updated comments to clarify the behavior of all_frames during DOM tree construction.
- Improved the `create_task_with_error_handling` function to allow for better exception logging and retrieval based on the `suppress_exceptions` parameter.
- Updated `SessionManager` to implement event-driven recovery for stale agent focus, replacing polling with efficient event handling.
- Refactored session retrieval methods to ensure focus validation and recovery are handled automatically.
- Enhanced logging for recovery processes and session management to provide clearer insights into state changes and errors.
- Adjusted various watchdogs to utilize the new session management methods, ensuring they correctly handle focus validation and session retrieval.
- Eliminated the caching of target types in SessionManager, simplifying the management of target information.
- Updated various components to directly access target types from the Target objects, enhancing clarity and reducing redundancy.
- Adjusted related methods in BrowserSession, CrashWatchdog, and DomService to reflect the changes in target type retrieval.
- Updated DOMWatchdog to provide a detailed hierarchical breakdown of timing metrics for various operations, improving clarity in performance logging.
- Enhanced DomService to calculate and log specific timing metrics for iframe scrolling, CDP calls, and snapshot processing, contributing to more accurate performance tracking.
- Updated DOMWatchdog to exclude specific sub-timings from total tracked time to avoid double-counting.
- Enhanced DomService to calculate and log overhead timings for both get_dom_tree and get_serialized_dom_tree operations, improving accuracy in performance metrics.
- Introduced a utility function `create_task_with_error_handling` to manage asyncio tasks with proper exception logging and suppression options.
- Updated multiple components, including `SessionManager`, `BrowserSession`, and various watchdogs, to utilize the new error handling mechanism for background tasks.
- Enhanced overall robustness of asynchronous operations by ensuring exceptions are logged and handled appropriately, improving maintainability and debugging capabilities.
- Updated DOM tree retrieval methods to lazily fetch frame hierarchy only when necessary, improving performance for pages without cross-origin iframes.
- Adjusted method signatures to reflect optional frame fetching, enhancing flexibility in DOM service operations.
- Streamlined timing metrics by setting placeholders for timing logs, preparing for future performance enhancements.
- Updated DOM tree retrieval methods to include pre-fetched frame hierarchy, reducing redundant CDP calls and improving performance.
- Adjusted timing logs to provide detailed breakdowns of DOM tree construction, including total time and individual component timings.
- Removed commented-out code related to Python-based highlighting, streamlining the DOMWatchdog class.
- Reduced wait time in DOMWatchdog for page stability, enhancing responsiveness during DOM builds.
- Replaced instances of `agent_focus` with `agent_focus_target_id` across various components to ensure consistent target management.
- Enhanced error handling and logging in `Watchdog` classes to reflect the new target ID approach.
- Improved session retrieval methods to streamline interactions with the `SessionManager`, ensuring better performance and reliability.
laithrw