Four small fixes for DOM capture performance. **4 files changed, +51
lines, -7 lines.**
## Fixes
### 1. `enhanced_snapshot.py` — Convert `isClickable` list to set before
loop
`_parse_rare_boolean_data` used `index in list` (O(n) per call), called
per node = O(n²).
```python
# Before: O(n) list scan per node
return index in rare_data['index'] # List[int]
# After: O(1) set lookup
is_clickable_set = set(nodes['isClickable']['index']) # once
return index in rare_data_set # per node
```
**20k elements: 14,160ms → 2,973ms. 100k elements: 356s → 9s.**
### 2. `paint_order.py` — Cap `RectUnionPure` at 5,000 rects
Each `add()` can split existing rects into up to 4 pieces. With
overlapping translucent layers this grows exponentially. `_MAX_RECTS =
5000` stops it. When hit, `contains()` returns False — less aggressive
filtering, same correctness.
**20k elements: 372s → 4.7s.**
### 3. `service.py` — Skip JS listener detection on pages with >10k
elements
Early bail-out inside the existing JS expression. Interactive elements
still detected via accessibility tree + `ClickableElementDetector`.
### 4. `dom_watchdog.py` — Add 2s timeout to pending network request
check
`_get_pending_network_requests()` had no timeout. On slow CI machines or
pages mid-navigation, it hangs for 15s+ silently eating into the 30s
`BrowserStateRequestEvent` budget. This causes the DOM capture to
timeout, producing 5 consecutive failures → agent termination.
Found by tracing eval failures on eBay search results where the
DOMWatchdog produced zero log output for 15 seconds between "STARTING"
and the timeout.
## Benchmarks
Profiled every pipeline stage on `main` vs this branch:
| Stage | main (20k) | This PR (20k) |
|---|---:|---:|
| `build_snapshot_lookup` | 14,160ms | **2,973ms** |
| JS listener detection | 2,326ms | **0ms** |
| `calculate_paint_order` | 372,303ms | **4,683ms** |
| Everything else | ~12s | ~8s |
| **Total** | **~400s** | **~16s** |
## Eval results
Ran WebBench_READ_v5 (198 tasks) on both branches, 4 runs total.
Task-by-task comparison on 35 common tasks:
- Always pass (both branches): 6
- Always fail (both branches): 13
- **Main always better: 0**
- **Ours always better: 1**
- Flaky (random): 15
No regressions. Normal pages (<2k nodes) unaffected — same timings, same
behavior.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Medium risk because it changes DOM capture heuristics (timeouts and
early-bail conditions) that can affect which elements are marked
clickable/visible or filtered by paint order on very large pages.
>
> **Overview**
> Speeds up DOM capture on very large pages by removing a key O(n²)
hotspot and adding protective caps/timeouts to avoid long stalls.
>
> `enhanced_snapshot.build_snapshot_lookup` now converts CDP
`isClickable.index` to a `set` once and uses O(1) membership checks per
node. Paint-order filtering adds a 5k-rectangle safety cap in
`RectUnionPure.add()` to prevent runaway rect fragmentation, and JS
click-listener detection skips entirely when the page has >10k elements.
>
> `DOMWatchdog` also wraps the pending-network-request precheck in a 2s
`asyncio.wait_for` and logs a timeout, preventing this step from
consuming most of the `BrowserStateRequestEvent` budget.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
3aa68384ad. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
On slow CI machines, _get_pending_network_requests() can hang for 15s+
when Chrome is busy loading/rendering after a navigation. This silently
eats into the 30s BrowserStateRequestEvent budget, leaving insufficient
time for the actual DOM capture — causing 5 consecutive timeouts and
agent termination.
Observed on eBay search results in eval runs: DOMWatchdog started but
produced zero log output for 15 seconds before the timeout killed it.
The pending network check was the first await after the URL log.
Three targeted fixes for pages with 5k+ elements:
1. build_snapshot_lookup: convert isClickable list to set before loop
_parse_rare_boolean_data used `index in list` (O(n) per call).
Called once per node = O(n²). Now O(1) via set lookup.
20k elements: 14,160ms → 2,973ms. 100k: 356s → 9s.
2. RectUnionPure: add _MAX_RECTS=5000 safety cap
Paint order rect union fragments exponentially with overlapping
layers. Uncapped, 20k elements took 372s. Capped: 4.7s.
Degrades gracefully (less aggressive filtering, same correctness).
3. Skip JS listener detection on pages with >10k elements
querySelectorAll('*') + per-element DOM.describeNode took 2.3s at 20k.
Elements still detected via accessibility tree + heuristics.
Combined effect at 20k elements: ~400s → ~16s (25x faster).
Normal pages (<5k elements) are completely unaffected.
_parse_rare_boolean_data used `index in list` (O(n) per call) on the
isClickable rare boolean data. Called once per node, this was O(n²)
total — the #1 bottleneck in the entire pipeline.
Fix: convert the list to a set once before the loop. O(1) per lookup.
Before → After:
5k elements: 1,788ms → 768ms (2.3x)
20k elements: 16,819ms → 2,911ms (5.8x)
100k elements: 356,629ms → 9,224ms (38.7x)
This single fix makes the full pipeline 2-3x faster at every scale:
5k: 8.0s → ~3.8s
20k: 29.7s → ~16s
100k: impossible → ~20s (Chrome-limited, not Python-limited)
Two additional performance fixes for heavy pages:
1. Hoist get_or_create_cdp_session() outside _construct_enhanced_node
Previously called once PER DOM NODE inside the recursive tree
construction. On a 100k-element page, this was 100k+ async
operations. Now resolved once before recursion starts.
2. Add _MAX_RECTS=5000 safety cap to RectUnionPure
The paint order rect union can fragment exponentially with many
overlapping translucent layers (each add() splits up to 4 rects).
Cap prevents memory/CPU explosion on complex pages.
Also: expanded stress test suite to 15 pages (up to 132k elements)
including shadow DOM + iframe combos, overlapping layers, cross-origin
iframes, and a 100k flat element test. All 15 pass.
10 progressively heavier test pages (1k to 50k+ elements):
- Flat divs, nested tables, shadow DOM, iframes, deep nesting
- Mega forms, SVG heavy, event listeners, cross-origin iframes
- Ultimate stress test combining everything
Two test modes:
- --dom-only: tests DOM capture without LLM (fast, no API key needed)
- --agent: tests full agent loop with real LLM on subset of pages
Pages with very large DOMs (e.g. Stimulsoft designer with 20,000+
elements) cause the browser state capture to time out, making the
agent unable to interact with the browser.
Three targeted fixes:
1. Skip JS listener detection on heavy pages (>10k elements)
The querySelectorAll('*') + getEventListeners() loop followed by
individual DOM.describeNode CDP calls for each listener element is
O(n) and can take 10s+ alone on heavy pages.
2. Batch DOM.describeNode calls (chunks of 50)
Previously all calls fired at once via asyncio.gather, flooding the
CDP WebSocket and causing timeouts on concurrent operations.
3. Adaptive CDP timeouts based on page complexity
- >15k elements: 25s initial / 10s retry (was 10s/2s)
- >5k elements: 15s initial / 5s retry
- Normal pages: unchanged 10s/2s
## Summary
- Close `CloudBrowserClient`'s httpx connection pool in
`on_BrowserStopEvent` after `stop_browser()` completes
- Make `CloudBrowserClient.close()` idempotent (safe to call multiple
times)
## Problem
`BrowserSession.on_BrowserStopEvent` calls `stop_browser()` to end the
cloud session but never calls `_cloud_browser_client.close()`. The
underlying `httpx.AsyncClient` connection pool stays alive in memory.
On AWS Lambda with provisioned concurrency, the same container handles
many invocations sequentially. Each invocation creates a new
`BrowserSession` → new `CloudBrowserClient` → new `httpx.AsyncClient`,
but the old ones are never cleaned up. Memory climbs from ~1.3GB to the
3GB ceiling over hours, triggering OOM kills.
**Observed in production**: 21 `Runtime.ExitError` crashes in 6 hours,
`aws.lambda.enhanced.max_memory_used` saturated at 3,009 MB (the Lambda
limit), 3 confirmed `out_of_memory` events.
## Test plan
- [ ] Run `tests/ci` suite to verify no regressions
- [ ] Verify cloud browser sessions still clean up properly
(`stop_browser` + `close`)
- [ ] Verify calling `close()` twice doesn't raise
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Close the `httpx` client pool when a cloud browser session stops to
prevent memory leaks and OOM on AWS Lambda. Also makes
`CloudBrowserClient.close()` safe to call multiple times.
- **Bug Fixes**
- Always call `_cloud_browser_client.close()` in
`BrowserSession.on_BrowserStopEvent` after `stop_browser()` (in finally)
to free the `httpx.AsyncClient` pool.
- Make `CloudBrowserClient.close()` idempotent by checking
`client.is_closed` before `aclose()`.
<sup>Written for commit 5e644981e8.
Summary will update on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
BrowserSession.on_BrowserStopEvent calls stop_browser() but never calls
_cloud_browser_client.close(), leaving the httpx connection pool alive.
On Lambda provisioned concurrency, these pools accumulate across
invocations — memory climbs from ~1.3GB to the 3GB ceiling over hours,
triggering OOM kills (21 Runtime.ExitError crashes in 6 hours observed
in production).
Changes:
- Call _cloud_browser_client.close() in on_BrowserStopEvent after
stop_browser completes (in a finally block so it runs even if
stop_browser fails)
- Make CloudBrowserClient.close() idempotent (check is_closed before
calling aclose) so it's safe to call multiple times
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
<!-- This is an auto-generated description by cubic. -->
## Summary by cubic
Adds a lightweight installer (`install_lite.sh`) that sets up the
`browser-use` CLI with minimal deps and Chromium, avoiding the full
library. Also extracts CLI deps into `requirements-cli.txt`, adds a CI
test, and hardens the installer with a `curl` pre-check and non-fatal
validate.
- New Features
- `install_lite.sh` installs Python 3.11+, `uv`, the CLI (`--no-deps`)
with minimal deps from `requirements-cli.txt`, and Chromium via `uvx
playwright` in `~/.browser-use-env`, and configures PATH (Linux, macOS,
Windows).
- Installs the `profile-use` helper into `~/.browser-use/bin` for
profile management.
- CI test (`tests/ci/test_cli_lite_deps.py`) boots the CLI in a clean
venv with only minimal deps and checks key imports.
- Minimal CLI deps pinned in
`browser_use/skill_cli/requirements-cli.txt`.
- Bug Fixes
- Validation is non-fatal; next steps are shown even if checks warn.
- Checks for `curl` before installing `uv` to prevent confusing
failures.
<sup>Written for commit 7e92031593.
Summary will update on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
## Problem
Closes#4211
`browser_click` in the MCP server uses `oneOf` at the top level of
`inputSchema` to express "provide either `index` OR
`coordinate_x`+`coordinate_y`":
```json
{
"type": "object",
"properties": { ... },
"oneOf": [
{"required": ["index"]},
{"required": ["coordinate_x", "coordinate_y"]}
]
}
```
Claude's API (and other strict MCP clients) rejects
`oneOf`/`allOf`/`anyOf` at the top level of a tool input schema:
```
400 tools.N.custom.input_schema: input_schema does not support oneOf, allOf, or anyOf at the top level
```
This cascades — it breaks **all** MCP tools in the session, not just
`browser_click`.
## Fix
Remove the `oneOf` block. The constraint is now expressed in the
property descriptions instead. No behaviour change: the runtime guard in
`_click()` already returns a clear error string when neither `index` nor
coordinates are provided.
## Diff summary
- `-4 lines`: remove the `oneOf: [...]` block entirely
- `+3 lines`: update the three property descriptions to say "Provide
this OR ..."
The `_click()` handler is untouched.
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Remove the top-level oneOf from the `browser_click` tool input schema so
Claude and other strict MCP clients accept it and the session doesn’t
break. No behavior change; `_click()` still validates, and field
descriptions now clarify using `index` or `coordinate_x`+`coordinate_y`.
<sup>Written for commit 6094cddd90.
Summary will update on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
## Summary
- derive cloud browser session UUID from Browser Use CDP host (e.g.
`<uuid>.cdp0.browser-use.com`)
- use that UUID as fallback in `BrowserSession.on_BrowserStopEvent` when
`_cloud_browser_client.current_session_id` is empty
- ensure reconnected sessions created via `BrowserSession(cdp_url=...)`
are properly stopped in cloud, not just reset locally
## Why
Reconnected cloud sessions (common MFA resume pattern) can leak browser
instances because stop logic only used `current_session_id`, which is
populated when the same process created the cloud browser. In reconnect
flows, that field is often unset.
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Ensure cloud browser sessions are stopped when reconnecting via cdp_url,
preventing leaked instances during MFA/resume flows. We derive the
session UUID from the CDP URL host and pass it to `stop_browser` with
the explicit ID.
- **Bug Fixes**
- Added a helper to extract the session UUID from the CDP host (e.g.,
<uuid>.cdpN.browser-use.com).
- Updated stop logic to use `current_session_id` or the derived UUID and
call `stop_browser(id)`; logs now include the session ID.
<sup>Written for commit 4c883feb7c.
Summary will update on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
Resolves#4352
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes Windows tunnel lifecycle bugs by spawning `cloudflared` as a
detached daemon and reliably terminating it to prevent orphaned
processes. Resolves#4352.
- **Bug Fixes**
- Windows spawn uses `CREATE_NEW_PROCESS_GROUP | CREATE_NO_WINDOW`; Unix
uses `start_new_session=True`.
- Windows kill uses Win32 `OpenProcess/TerminateProcess` with brief
polling; Unix uses `SIGTERM` with `SIGKILL` fallback.
<sup>Written for commit 4950c3baa1.
Summary will update on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
## Summary
- Clear `last_model_output` and `last_result` at the start of `step()`
to prevent stale data from previous steps being recorded in history on
timeout
- Increment `n_steps` in `_execute_step`'s timeout handler to prevent
the main loop from retrying the same step number
Fixes#4480
## What changed
**`step()` — clear stale state at entry:**
```diff
self.step_start_time = time.time()
+
+ # Clear previous step state to prevent stale data from being recorded
+ self.state.last_model_output = None
+ self.state.last_result = None
+
browser_state_summary = None
```
**`_execute_step()` — ensure counter advances on timeout:**
```diff
self.state.last_result = [ActionResult(error=error_msg)]
+ # Ensure step counter advances on timeout
+ if self.state.n_steps == step + 1:
+ self.state.n_steps += 1
```
The guard `if self.state.n_steps == step + 1` prevents double-increment
when `_finalize()` has already incremented on the normal path.
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes stale history entries and a stuck step counter when a step times
out. Clears per-step state after `_prepare_context` and ensures
`n_steps` advances on timeout.
- **Bug Fixes**
- Clear `last_model_output` and `last_result` right after
`_prepare_context` (before LLM/action calls) so prompts keep previous
output and timeouts don't record stale data.
- In `_execute_step()`, increment `n_steps` after a timeout (with a
guard) so the loop moves forward.
<sup>Written for commit c0a11dc61e.
Summary will update on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
Resolves#4510
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes double-serialized action fields in Anthropic tool responses to
prevent schema validation errors and failed tool calls. Preserves
original tracebacks in the fallback path.
- **Bug Fixes**
- Parse nested JSON strings in tool inputs; normalize newline/carriage
return/tab escapes when needed.
- Applied in `browser_use/llm/anthropic/chat.py` and
`browser_use/llm/aws/chat_anthropic.py`.
<sup>Written for commit f74a4435b1.
Summary will update on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
Slightly modified #4401
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Allow overriding the model pricing source for token cost calculations,
with URL-aware caching to keep caches from different sources separate.
Defaults remain the LiteLLM pricing JSON.
- **New Features**
- `TokenCost` accepts an optional `pricing_url`; `Agent(...,
pricing_url=...)` passes it through.
- New env var BROWSER_USE_MODEL_PRICING_URL to set a global pricing
source.
- Cache files now include `source_url` so multiple sources can coexist.
- **Bug Fixes**
- Cache validation and cleanup are source-aware and only remove expired
or same-source files.
<sup>Written for commit 4f268e4390.
Summary will update on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
Resolves#4491
<!-- This is an auto-generated description by cubic. -->
## Summary by cubic
Make CDP navigation timeout configurable and apply it to
`Page.navigate`, fixing mismatched timeouts. This aligns
`NavigateToUrlEvent` (and `TIMEOUT_NavigateToUrlEvent`) with the actual
CDP call and prevents early failures on heavy pages.
- **Bug Fixes**
- Add `nav_timeout` to `_navigate_and_wait` and use it for
`Page.navigate(...)`.
- Pass `event.event_timeout`; default remains 20s when not provided.
Outer `timeout_ms` behavior is unchanged.
<sup>Written for commit 801b134f90.
Summary will update on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
Resolves#4492
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes daemon orphaning when the external browser or CDP connection dies
(with `--connect`, `--cdp-url`, or cloud). Adds safer connection
watching, bounded cleanup, and a hard exit that preserves the exit code.
Resolves#4492.
- **Bug Fixes**
- Skip shutdown while the browser is reconnecting; shut down only when
CDP stays disconnected.
- Prevent duplicate shutdowns by scheduling a single shutdown task.
- Add 10s timeouts around `browser_session.stop()`/`kill()` to avoid
hangs; log on timeout.
- Force process exit in `finally` via `os._exit(exit_code)` to preserve
error status and prevent orphaned daemons.
<sup>Written for commit a31809b83d.
Summary will update on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
<!-- This is an auto-generated description by cubic. -->
## Summary by cubic
Removed a redundant pass in the exception handler of
`SecurityWatchdog.on_NavigationCompleteEvent`, simplifying the code
while keeping error logging intact. No behavior change.
<sup>Written for commit 393d2c5b0f.
Summary will update on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
MagMueller