mirror of
https://github.com/google/adk-docs.git
synced 2026-09-14 16:16:59 +08:00
George Weale
1bd29540df
* docs(evaluate): fix conformance CLI, defaults, optimizer imports * docs(evaluate): fix App construction, agent path type, user-sim coverage * docs: drop parenthetical asides and future tense from the evaluate corrections * Update environment_simulation.md * Update user-sim.md * Update user-sim.md * Update index.md * Update index.md removing repetitive (and soon out of date) default model declarations. --------- Co-authored-by: Joe Fernandez <931947+joefernandez@users.noreply.github.com>
447 lines
16 KiB
Markdown
447 lines
16 KiB
Markdown
# Environment simulation for evaluations
|
|
|
|
<div class="language-support-tag">
|
|
<span class="lst-supported">Supported in ADK</span><span class="lst-python">Python v1.24.0</span>
|
|
</div>
|
|
|
|
When evaluating agents that rely on external dependencies — such as APIs,
|
|
databases, or third-party services — running those tools live during testing can
|
|
be slow, costly, or unreliable. The **Environment Simulator** lets you safely
|
|
intercept these tool calls during agent execution and replace them with
|
|
controlled, deterministic responses, without modifying the agent itself. This
|
|
approach can fill a critical gap in the agent improvement loop, allowing you to
|
|
create hermetic, offline test runs that isolate your agent logic for reliable
|
|
scoring.
|
|
|
|
Overall, this feature lets you:
|
|
|
|
* Test how an agent handles API errors or edge-case responses.
|
|
* Run evaluations offline, without access to live backends.
|
|
* Generate realistic mock responses automatically using an LLM.
|
|
* Produce reproducible test runs by seeding probabilistic injections.
|
|
|
|
The Environment Simulation integrates with ADK's tool execution pipeline via the
|
|
[`before_tool_callback`](/callbacks/types-of-callbacks/#tool-execution-callbacks)
|
|
hook or the [plugin system](/plugins/), so no
|
|
changes to your agent code are required.
|
|
|
|
```
|
|
The Environment Simulation is an experimental feature. Its API may change in future
|
|
releases.
|
|
```
|
|
|
|
## How it works
|
|
|
|
While [User Simulation](/evaluate/user-sim/)
|
|
drives the conversation forward, Environment Simulation provides the stable
|
|
backend. At a high level, the Environment Simulator sits between your agent and
|
|
its tools. When the agent calls a tool, the simulator intercepts the call and
|
|
decides whether to return a synthetic response — either a predefined injection
|
|
or an LLM-generated mock — or to let the real tool execute.
|
|
|
|
The decision logic follows this order for each configured tool:
|
|
|
|
1. **Injection configs** are checked first, in order. If a matching injection
|
|
is found (based on argument matching and probability), its error or response
|
|
is returned immediately.
|
|
2. **Mock strategy** is used as a fallback if no injection config applies. The
|
|
simulator calls an LLM to generate a realistic response based on the tool's
|
|
schema and any stateful context.
|
|
3. **No-op** is returned (`None`) if the tool is not in the simulator config,
|
|
allowing the real tool to execute normally.
|
|
|
|
## Integration
|
|
|
|
The `EnvironmentSimulationFactory` class provides two integration points:
|
|
|
|
* `create_callback()` — Returns an async callable suitable for use as a
|
|
`before_tool_callback` on any `LlmAgent`.
|
|
* `create_plugin()` — Returns an `EnvironmentSimulationPlugin` instance that
|
|
integrates with the ADK plugin system.
|
|
|
|
### Using as a callback
|
|
|
|
The following example shows how to create an environment simulation as one of the adk agent callbacks.
|
|
|
|
|
|
```python
|
|
from google.adk.agents import LlmAgent
|
|
from google.adk.tools.environment_simulation import EnvironmentSimulationFactory
|
|
from google.adk.tools.environment_simulation.environment_simulation_config import (
|
|
EnvironmentSimulationConfig,
|
|
InjectedError,
|
|
InjectionConfig,
|
|
ToolSimulationConfig,
|
|
)
|
|
|
|
config = EnvironmentSimulationConfig(
|
|
tool_simulation_configs=[
|
|
ToolSimulationConfig(
|
|
tool_name="get_user_profile",
|
|
injection_configs=[
|
|
InjectionConfig(
|
|
injected_error=InjectedError(
|
|
injected_http_error_code=503,
|
|
error_message="Service temporarily unavailable.",
|
|
)
|
|
)
|
|
],
|
|
)
|
|
]
|
|
)
|
|
|
|
agent = LlmAgent(
|
|
name="my_agent",
|
|
model="gemini-flash-latest",
|
|
tools=[get_user_profile],
|
|
before_tool_callback=EnvironmentSimulationFactory.create_callback(config),
|
|
)
|
|
```
|
|
|
|
### Using as a plugin
|
|
|
|
The following example shows how to create environment simulation as an ADK agent plugin.
|
|
|
|
```python
|
|
from google.adk.apps import App
|
|
from google.adk.tools.environment_simulation import EnvironmentSimulationFactory
|
|
from google.adk.tools.environment_simulation.environment_simulation_config import (
|
|
EnvironmentSimulationConfig,
|
|
MockStrategy,
|
|
ToolSimulationConfig,
|
|
)
|
|
|
|
config = EnvironmentSimulationConfig(
|
|
tool_simulation_configs=[
|
|
ToolSimulationConfig(
|
|
tool_name="search_products",
|
|
mock_strategy_type=MockStrategy.MOCK_STRATEGY_TOOL_SPEC,
|
|
)
|
|
]
|
|
)
|
|
|
|
app = App(
|
|
name="my_app",
|
|
root_agent=my_agent,
|
|
plugins=[EnvironmentSimulationFactory.create_plugin(config)],
|
|
)
|
|
```
|
|
|
|
## Configuration reference
|
|
|
|
You can configure the Environment Simulator with a set of dataclasses. The
|
|
following sections provide a detailed reference for each configuration object.
|
|
|
|
### `EnvironmentSimulationConfig`
|
|
|
|
The top-level configuration object.
|
|
|
|
Field | Type | Default | Description
|
|
:------------------------------- | :--------------------------- | :------------------- | :----------
|
|
`tool_simulation_configs` | `List[ToolSimulationConfig]` | required | One entry per tool to simulate. Must not be empty, and tool names must be unique.
|
|
`simulation_model` | `str` | `"gemini-flash-latest"` | The LLM used for tool connection analysis and mock response generation.
|
|
`simulation_model_configuration` | `GenerateContentConfig` | thinking enabled | LLM generation config for internal simulator calls.
|
|
`environment_data` | `str \| None` | `None` | Optional environment context (e.g., a JSON database snapshot) passed to mock strategies to generate more realistic responses.
|
|
`tracing` | `str \| None` | `None` | Tracing data (e.g., a prior agent run trace in JSON string format) to provide historical context.
|
|
|
|
### `ToolSimulationConfig`
|
|
|
|
Defines how a single named tool should be simulated.
|
|
|
|
Field | Type | Default | Description
|
|
:------------------- | :---------------------- | :-------------------------- | :----------
|
|
`tool_name` | `str` | required | Must match the tool's registered name exactly.
|
|
`injection_configs` | `List[InjectionConfig]` | `[]` | Zero or more injection configs, checked in order before the mock strategy.
|
|
`mock_strategy_type` | `MockStrategy` | `MOCK_STRATEGY_UNSPECIFIED` | Fallback strategy when no injection is triggered.
|
|
|
|
### `InjectionConfig`
|
|
|
|
Controls a single synthetic response that can be injected into a tool call.
|
|
Exactly one of `injected_error` or `injected_response` must be set.
|
|
|
|
Field | Type | Default | Description
|
|
:------------------------- | :----------------------- | :------ | :----------
|
|
`injected_error` | `InjectedError \| None` | `None` | Error to return (mutually exclusive with `injected_response`).
|
|
`injected_response` | `Dict[str, Any] \| None` | `None` | Fixed response dict to return (mutually exclusive with `injected_error`).
|
|
`injection_probability` | `float` | `1.0` | Probability `[0.0, 1.0]` that this injection fires.
|
|
`match_args` | `Dict[str, Any] \| None` | `None` | If set, the injection only fires when the tool's arguments contain all key-value pairs in `match_args`.
|
|
`injected_latency_seconds` | `float` | `0.0` | Artificial delay (≤ 120 s) added before returning the injection result.
|
|
`random_seed` | `int \| None` | `None` | Seed for the probability check, enabling deterministic injection behavior.
|
|
|
|
### `InjectedError`
|
|
|
|
Defines an HTTP-style error response.
|
|
|
|
| Field | Type | Description |
|
|
| :------------------------- | :---- | :-------------------------------------- |
|
|
| `injected_http_error_code` | `int` | HTTP status code to surface as |
|
|
: : : `"error_code"` in the tool response. :
|
|
| `error_message` | `str` | Human-readable message surfaced as |
|
|
: : : `"error_message"` in the tool response. :
|
|
|
|
### `MockStrategy`
|
|
|
|
Enum controlling how the simulator generates responses when no injection fires.
|
|
|
|
| Value | Description |
|
|
| :------------------------ | :---------------------------------------------- |
|
|
| `MOCK_STRATEGY_TOOL_SPEC` | Uses the tool's schema and stateful context to |
|
|
: : prompt an LLM to generate a realistic response. :
|
|
| `MOCK_STRATEGY_TRACING` | *(Deprecated)* Please use |
|
|
: : `MOCK_STRATEGY_TOOL_SPEC` with tracing input. :
|
|
|
|
## Injection mode
|
|
|
|
Use injection configs to test specific failure or edge-case scenarios.
|
|
Injections are evaluated in list order; the first one whose `match_args`
|
|
criteria are met (and whose probability check passes) is applied.
|
|
|
|
### Injecting errors
|
|
|
|
The following example shows how to inject errors with specific error code and error message to the agent.
|
|
|
|
```python
|
|
from google.adk.tools.environment_simulation.environment_simulation_config import (
|
|
InjectedError,
|
|
InjectionConfig,
|
|
ToolSimulationConfig,
|
|
)
|
|
|
|
ToolSimulationConfig(
|
|
tool_name="charge_payment",
|
|
injection_configs=[
|
|
InjectionConfig(
|
|
injected_error=InjectedError(
|
|
injected_http_error_code=402,
|
|
error_message="Payment declined.",
|
|
)
|
|
)
|
|
],
|
|
)
|
|
```
|
|
|
|
The agent will receive `{"error_code": 402, "error_message": "Payment
|
|
declined."}` instead of a real tool result, allowing you to evaluate how the
|
|
agent handles payment failures.
|
|
|
|
### Injecting fixed responses
|
|
|
|
Use the following InjectionConfig to specify a success response with fixed response payload.
|
|
|
|
```python
|
|
InjectionConfig(
|
|
injected_response={"status": "ok", "order_id": "ORD-9999"}
|
|
)
|
|
```
|
|
|
|
### Conditional injection with argument matching
|
|
|
|
Use `match_args` to inject only when specific arguments are passed.
|
|
|
|
```python
|
|
InjectionConfig(
|
|
match_args={"item_id": "ITEM-404"},
|
|
injected_error=InjectedError(
|
|
injected_http_error_code=404,
|
|
error_message="Item not found.",
|
|
),
|
|
)
|
|
```
|
|
|
|
Here, the error is injected only when the tool is called with
|
|
`item_id="ITEM-404"`. All other calls pass through to the next injection config
|
|
or to the mock strategy.
|
|
|
|
### Probabilistic injection
|
|
|
|
Set `injection_probability` to a value between `0.0` and `1.0` to simulate flaky
|
|
behavior. For reproducible test runs, pin the random outcome with `random_seed`.
|
|
|
|
```python
|
|
InjectionConfig(
|
|
injection_probability=0.3,
|
|
random_seed=42,
|
|
injected_error=InjectedError(
|
|
injected_http_error_code=500,
|
|
error_message="Internal server error.",
|
|
),
|
|
)
|
|
```
|
|
|
|
### Injecting latency
|
|
|
|
Use `injected_latency_seconds` to simulate slow backend responses, useful for
|
|
testing timeout handling or user experience under degraded conditions.
|
|
|
|
```python
|
|
InjectionConfig(
|
|
injected_latency_seconds=5.0,
|
|
injected_response={"result": "slow but successful"},
|
|
)
|
|
```
|
|
|
|
### Combining multiple injection configs
|
|
|
|
Multiple injection configs on a single tool are checked in order. You can
|
|
combine them to test multiple scenarios:
|
|
|
|
```python
|
|
ToolSimulationConfig(
|
|
tool_name="get_inventory",
|
|
injection_configs=[
|
|
# Always fail for a specific out-of-stock item
|
|
InjectionConfig(
|
|
match_args={"sku": "OOS-001"},
|
|
injected_response={"quantity": 0, "available": False},
|
|
),
|
|
# Randomly fail 20% of the time for all other items
|
|
InjectionConfig(
|
|
injection_probability=0.2,
|
|
random_seed=7,
|
|
injected_error=InjectedError(
|
|
injected_http_error_code=503,
|
|
error_message="Inventory service unavailable.",
|
|
),
|
|
),
|
|
],
|
|
)
|
|
```
|
|
|
|
## Mock strategy mode
|
|
|
|
When you want the simulator to generate plausible responses automatically —
|
|
rather than returning hand-crafted values — use `MOCK_STRATEGY_TOOL_SPEC`.
|
|
|
|
The simulator uses an LLM to:
|
|
|
|
1. Analyze the schemas of all tools the agent has access to, and identify
|
|
*stateful dependencies* between them (e.g., a `create_order` tool produces
|
|
an `order_id` that `get_order` consumes).
|
|
2. Track a **state store** of IDs and resources created during the session.
|
|
3. Generate a response that is consistent with the tool's schema and the
|
|
current state — returning a 404-style error if a consuming tool requests a
|
|
resource that was never created.
|
|
|
|
```python
|
|
from google.adk.tools.environment_simulation.environment_simulation_config import (
|
|
EnvironmentSimulationConfig,
|
|
MockStrategy,
|
|
ToolSimulationConfig,
|
|
)
|
|
|
|
config = EnvironmentSimulationConfig(
|
|
tool_simulation_configs=[
|
|
ToolSimulationConfig(
|
|
tool_name="create_order",
|
|
mock_strategy_type=MockStrategy.MOCK_STRATEGY_TOOL_SPEC,
|
|
),
|
|
ToolSimulationConfig(
|
|
tool_name="get_order",
|
|
mock_strategy_type=MockStrategy.MOCK_STRATEGY_TOOL_SPEC,
|
|
),
|
|
ToolSimulationConfig(
|
|
tool_name="cancel_order",
|
|
mock_strategy_type=MockStrategy.MOCK_STRATEGY_TOOL_SPEC,
|
|
),
|
|
]
|
|
)
|
|
```
|
|
|
|
With this config, the simulator will automatically generate an `order_id` when
|
|
`create_order` is mocked, and use it to return consistent results (or a
|
|
not-found error) when `get_order` or `cancel_order` are subsequently called.
|
|
|
|
### Providing environment data
|
|
|
|
Pass domain-specific context through `environment_data` to make mock responses
|
|
more realistic. This can be a JSON string representing a snapshot of your
|
|
database or any structured context the LLM should use when generating responses.
|
|
|
|
```python
|
|
import json
|
|
|
|
db_snapshot = {
|
|
"products": [
|
|
{"id": "P-001", "name": "Wireless Headphones", "price": 79.99, "stock": 12},
|
|
{"id": "P-002", "name": "USB-C Hub", "price": 34.99, "stock": 0},
|
|
],
|
|
"warehouse_location": "US-WEST-2",
|
|
}
|
|
|
|
config = EnvironmentSimulationConfig(
|
|
tool_simulation_configs=[
|
|
ToolSimulationConfig(
|
|
tool_name="search_products",
|
|
mock_strategy_type=MockStrategy.MOCK_STRATEGY_TOOL_SPEC,
|
|
),
|
|
],
|
|
environment_data=json.dumps(db_snapshot),
|
|
)
|
|
```
|
|
|
|
The LLM will use this data to return product names, prices, and stock levels
|
|
that match your domain, rather than generating arbitrary placeholder values.
|
|
|
|
### Providing tracing data
|
|
|
|
Feed traces generated in the agent to be mocked through `tracing` to make mock
|
|
responses more realistic.
|
|
|
|
```python
|
|
import json
|
|
|
|
agent_traces = [
|
|
{
|
|
"invocation_id": "inv-001",
|
|
"user_content": {"role": "user", "parts": [{"text": "Search for high-end headphones"}]},
|
|
"intermediate_data": {
|
|
"tool_uses": [
|
|
{
|
|
"name": "search_products",
|
|
"args": {"query": "high-end headphones"},
|
|
"response": {"products": [{"id": "P-123", "name": "Premium Wireless ANC Headphones"}]}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
]
|
|
|
|
config = EnvironmentSimulationConfig(
|
|
tool_simulation_configs=[
|
|
ToolSimulationConfig(
|
|
tool_name="search_products",
|
|
mock_strategy_type=MockStrategy.MOCK_STRATEGY_TOOL_SPEC,
|
|
),
|
|
],
|
|
tracing=json.dumps(agent_traces),
|
|
)
|
|
```
|
|
|
|
The LLM will use this data to return product names, prices, and stock levels
|
|
that match your domain, rather than generating arbitrary placeholder values.
|
|
|
|
## Mixing injections and mock strategy
|
|
|
|
Injection configs and a mock strategy can be combined on the same tool.
|
|
Injections are always checked first; the mock strategy fires only when no
|
|
injection applies.
|
|
|
|
```python
|
|
ToolSimulationConfig(
|
|
tool_name="send_notification",
|
|
injection_configs=[
|
|
# Always fail for a known-bad recipient
|
|
InjectionConfig(
|
|
match_args={"recipient_id": "INVALID"},
|
|
injected_error=InjectedError(
|
|
injected_http_error_code=400,
|
|
error_message="Invalid recipient.",
|
|
),
|
|
),
|
|
],
|
|
# For all other recipients, generate a plausible success response
|
|
mock_strategy_type=MockStrategy.MOCK_STRATEGY_TOOL_SPEC,
|
|
)
|
|
```
|
|
|