# Fix: `output_model_schema` not working with `@sandbox`
## Problem
When using `output_model_schema` with an Agent inside a `@sandbox`
decorated function, the `history.structured_output` property always
returned `None`.
```python
@sandbox()
async def my_task(browser: Browser) -> AgentHistoryList[MyModel]:
agent = Agent(task="...", browser=browser, llm=llm, output_model_schema=MyModel)
return await agent.run()
result = await my_task()
result.structured_output # Always None ❌
```
## Root Cause
`AgentHistoryList` stores the output schema in `_output_model_schema` —
a **Pydantic private attribute** (underscore prefix). Private attributes
are **not serialized** when converting to JSON.
When sandbox executes remotely:
1. Agent runs and sets `history._output_model_schema = MyOutputModel` ✅
2. Result is serialized to JSON for transport → **private attribute
dropped** ❌
3. Client reconstructs `AgentHistoryList` but `_output_model_schema` is
`None`
4. `structured_output` property returns `None` (schema missing)
## Solution
### 1. Added `get_structured_output(output_model)` method
Users can explicitly pass their schema to parse the result:
```python
result.get_structured_output(MyModel) # Works even without type annotation
```
### 2. Enhanced sandbox parsing for Pydantic generics
When return type is `AgentHistoryList[MyModel]`, the sandbox now:
- Detects Pydantic generic metadata via `__pydantic_generic_metadata__`
- Extracts the type parameter (`MyModel`)
- Auto-sets `_output_model_schema` on the reconstructed object
### 3. Fixed import extraction for Pydantic generics
The sandbox's import extraction wasn't detecting `AgentHistoryList` from
return types like `AgentHistoryList[MyModel]`, causing `NameError` on
remote execution.
## Usage
### Option 1: Generic return type (recommended)
```python
@sandbox()
async def task(browser: Browser) -> AgentHistoryList[MyModel]:
agent = Agent(task="...", browser=browser, llm=llm, output_model_schema=MyModel)
return await agent.run()
result = await task()
result.structured_output # ✅ Works automatically
```
### Option 2: Non-generic return type with explicit method
```python
@sandbox()
async def task(browser: Browser) -> AgentHistoryList:
agent = Agent(task="...", browser=browser, llm=llm, output_model_schema=MyModel)
return await agent.run()
result = await task()
result.structured_output # None (expected)
result.get_structured_output(MyModel) # ✅ Works
```
## Files Changed
- `browser_use/agent/views.py` - Added `get_structured_output()` method
- `browser_use/sandbox/sandbox.py` - Enhanced
`_parse_with_type_annotation()` and `_get_imports_used_in_function()`
for Pydantic generics
- `tests/ci/test_sandbox_structured_output.py` - Unit tests
- `examples/sandbox/structured_output.py` - Example demonstrating both
approaches
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes structured output when agents run inside @sandbox.
structured_output now works with AgentHistoryList[Model], and a new
helper lets you parse results without type annotations.
- **Bug Fixes**
- Preserve output model schema in sandbox results by detecting Pydantic
generics and setting it on AgentHistoryList.
- Improve import extraction to include generic types like
AgentHistoryList[MyModel] and prevent NameError.
- Enhance _parse_with_type_annotation to reconstruct AgentHistoryList[T]
and set _output_model_schema correctly.
- **New Features**
- Added AgentHistoryList.get_structured_output(schema) to parse the
final result explicitly when no generic return type is used.
<sup>Written for commit 9b894150d5.
Summary will update automatically on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
<!-- This is an auto-generated description by cubic. -->
## Summary by cubic
Enable coordinate-based clicking only for supported models, keeping
clicks index-only by default. This reduces misclicks and keeps the API
simpler.
- **New Features**
- Auto-enable coordinate clicking for models matching: claude-sonnet-4,
claude-opus-4, gemini-3-pro, and browser-use/*.
- Added set_coordinate_clicking(enabled) to toggle coordinate support
and re-register the click action accordingly.
- New ClickElementActionIndexOnly model for index-only clicks;
coordinate and index handlers are retained and shared.
<sup>Written for commit 8c9e3b272c.
Summary will update automatically on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
Re-enable sensitive data filtering for state messages to prevent real
sensitive values from action results leaking into LLM messages.
State messages, specifically the `agent_history_description`, include
action results which contain actual sensitive values (e.g., passwords)
after they've been replaced from placeholders during action execution.
The `_filter_sensitive_data` method was previously not called for state
messages, leading to these real values being sent to the LLM. This PR
ensures that state messages are properly filtered, replacing real
sensitive data with placeholder tags before being sent to the LLM. New
integration tests confirm the end-to-end filtering.
---
[Slack
Thread](https://browser-use.slack.com/archives/D092QUQDC56/p1765410795078139?thread_ts=1765410795.078139&cid=D092QUQDC56)
<a
href="https://cursor.com/background-agent?bcId=bc-97cd8546-fd39-4e4f-84af-8882815d705b"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/open-in-cursor-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/open-in-cursor-light.svg"><img alt="Open in
Cursor"
src="https://cursor.com/open-in-cursor.svg"></picture></a> <a
href="https://cursor.com/agents?id=bc-97cd8546-fd39-4e4f-84af-8882815d705b"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/open-in-web-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/open-in-web-light.svg"><img alt="Open in Web"
src="https://cursor.com/open-in-web.svg"></picture></a>
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Restored sensitive data filtering for state messages so real values from
action results never reach the LLM. Sensitive values like passwords and
API keys are replaced with placeholder tags in
agent_history_description.
- **Bug Fixes**
- Filter state messages in MessageManager when sensitive_data is
present; system/context messages remain unfiltered.
- Replace real values in action results with <secret>...</secret>
placeholders before sending to the LLM.
- Added integration tests covering global and domain-specific formats
(e.g., password, api_key) to prevent leaks.
<sup>Written for commit 201b990591.
Summary will update automatically on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
This change ensures that sensitive data in action results is filtered before being included in state messages sent to the LLM. This prevents accidental leakage of private information. New tests are added to verify this functionality.
Co-authored-by: mailmertunsal <mailmertunsal@gmail.com>
<!-- This is an auto-generated description by cubic. -->
## Summary by cubic
Fixed ChatOpenAI import path in actor docs to
browser_use.llm.openai.chat so code examples import correctly.
<sup>Written for commit ab48cc51df.
Summary will update automatically on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
<!-- This is an auto-generated description by cubic. -->
## Summary by cubic
Add support for browser-use fine-tuned models with simplified,
JSON-first system prompts, and reorganize prompt templates into a
dedicated package for reliable loading and packaging.
- **New Features**
- Auto-detect browser-use models (model name contains "browser-use/")
and select simplified prompts for thinking, no-thinking, and flash
modes.
- Plumb is_browser_use_model through MessageManager to load the correct
template.
- **Refactors**
- Moved all prompt templates to browser_use/agent/system_prompts and
load via importlib.resources for package-safe access.
- Updated pyproject to include the new prompt directory and remove old
file paths.
<sup>Written for commit 110b96dc8d.
Summary will update automatically on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
<!-- This is an auto-generated description by cubic. -->
## Summary by cubic
Corrected the documented default for max_actions_per_step to 3 to match
current behavior. Cleaned minor formatting in AGENTS.md (removed
trailing spaces and fixed a tips blockquote).
<sup>Written for commit 2e887d0076.
Summary will update automatically on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
<!-- This is an auto-generated description by cubic. -->
## Summary by cubic
Integrates Browser Use API skills into the agent, auto-registering them
as actions with Pydantic-validated parameters and structured outputs.
Adds cookie-aware execution and surfaces unavailable skills (missing
cookies) in the agent prompt; requires BROWSER_USE_API_KEY and passing
skills (or skill_ids) to enable skills.
- **New Features**
- Adds browser_use/skills package with SkillService (fetch/execute),
schema converters, and SDK view wrappers.
- Agent accepts skills or skill_ids (supports '*'); 'skills' takes
precedence and they cannot be set together; lazily loads skills,
registers them as actions with clean slugs, rebuilds action models, and
closes the service on shutdown.
- Validates parameters against skill schemas; fills cookie values from
browser cookies; returns clear errors for missing cookies; supports
output schema conversion.
- Shows "Unavailable Skills" info in the state message when required
cookies are missing.
- Updates BrowserSession.cookies to return all cookies via CDP
Storage.getCookies; removes the urls parameter.
- **Dependencies**
- Adds browser-use-sdk>=2.0.12.
<sup>Written for commit 22b37f3c11.
Summary will update automatically on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
<!-- This is an auto-generated description by cubic. -->
## Summary by cubic
Fixes structured output when tools are provided by initializing tools
first and auto-detecting the output model from tools. Ensures the agent
uses the correct schema, with a warning if the agent and tools models
differ.
- **Bug Fixes**
- Initialize tools before task/LLM setup to allow schema detection.
- Use the tools’ output model when output_model_schema isn’t set; warn
on mismatch (agent schema takes precedence).
- ToolsService now stores and exposes the output model to keep done
action registration consistent.
<sup>Written for commit 1489d55210.
Summary will update automatically on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
<!-- This is an auto-generated description by cubic. -->
## Summary by cubic
Added an analysis step for reruns that replaces extract replay,
generating fresh results from filtered page markdown and optional
screenshot. Rerun now routes extract actions through this step and
supports a dedicated LLM.
- **New Features**
- Added _execute_ai_step with system/user prompts, clean markdown
extraction, optional screenshot, and LLM invocation with timeout.
- rerun_history accepts ai_step_llm; extract actions are re-evaluated
via the analysis step, while other actions run in batch with order
preserved.
- New prompts: get_ai_step_system_prompt and get_ai_step_user_prompt.
- Updated example to show custom LLM usage for the analysis step and
summary.
- Added tests (basic extraction, screenshot handling, error path) in
tests/ci/test_ai_step.py.
<sup>Written for commit 00866bd885.
Summary will update automatically on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
<!-- This is an auto-generated description by cubic. -->
## Summary by cubic
Adds fallback LLM support to keep the agent running when the primary
model hits rate limits or provider errors. Also standardizes HTTP error
handling to custom exceptions used by the agent.
- **New Features**
- Agent accepts a fallback_llm and switches on 401, 402, 429, 500, 502,
503, 504 after primary retries are exhausted.
- Fallback stays active for the rest of the run and is registered with
token cost tracking.
- New properties: is_using_fallback_llm and current_llm_model.
- Added example and docs for configuring and observing fallback
behavior.
- **Refactors**
- ChatBrowserUse now raises ModelProviderError and ModelRateLimitError
(instead of ValueError) for HTTP failures.
- Updated tests to cover fallback switching and new exception types.
<sup>Written for commit cad2216410.
Summary will update automatically on new commits.</sup>
<!-- End of auto-generated description by cubic. -->
Mert Unsal
