Kristopher Overholt
f6b1259cdc
* Add agent routing and model routing docs for RoutedAgent and RoutedLlm in ADK JS * Update example code snippets * Add renamed code sample files * Update lychee link checker config * Shorten docs and snippets paths and filenames * Update titles * Rework intros * Rework intros * Add notes about experimental status * Move InMemoryRunner outside of code snippet regions since it's unused in rendered snippet
2.5 KiB
Route between models
!!! example "Experimental"
Model routing is experimental and may change in future releases. We welcome
your
[feedback](https://github.com/google/adk-js/issues/new?template=feature_request.md)!
An LlmAgent uses a single model by default. When you need to dynamically
select between different models for each request, you can define a routing
function that chooses which model to use. RoutedLlm provides this capability,
enabling model fallback on error, A/B testing between models, and auto-routing
by input complexity. If the selected model fails before producing any output,
the routing function is called again with error context so it can select a
different model.
Pass a RoutedLlm as an LlmAgent's model parameter. Use RoutedLlm when
only the model varies between routes. If you also need to switch instructions,
tools, or sub-agents, use RoutedAgent instead.
How routing works
The LlmRouter function receives the map of available models and the current
LlmRequest, and returns the key of the model to use:
=== "TypeScript"
```typescript
type LlmRouter = (
models: Readonly<Record<string, BaseLlm>>,
request: LlmRequest,
errorContext?: { failedKeys: ReadonlySet<string>; lastError: unknown },
) => Promise<string | undefined> | string | undefined;
```
The models parameter accepts either a Record<string, BaseLlm> with explicit
keys, or an array of BaseLlm instances. If an array is provided, each model's
name is used as its key.
Failover follows the same rules as
RoutedAgent: the router is re-called with
errorContext only if the selected model fails before yielding any response.
After yielding, errors propagate without retry. The router can return
undefined to stop retrying and propagate the last error.
Live connections: RoutedLlm.connect() selects the model at connection
time. Once a live connection is established, the model cannot be switched
mid-stream.
Basic usage
The following example creates a RoutedLlm that tries a primary model first and
falls back to a secondary model if the primary fails. The router checks
errorContext.failedKeys to avoid re-selecting the failed model:
=== "TypeScript"
```typescript
--8<-- "examples/typescript/snippets/agents/models/routing/basic-usage.ts:full"
```