google__adk-docs/docs/integrations/cloud-trace.md

catalog_title, catalog_description, catalog_icon, catalog_tags

catalog_title	catalog_description	catalog_icon	catalog_tags
Google Cloud Trace	Monitor, debug, and trace ADK agent interactions	/integrations/assets/cloud-trace.svg	observability google

Google Cloud Trace observability for ADK

Supported in ADKPythonTypeScriptGo

During local development, you can inspect agent behavior with the Trace view in the ADK web UI. Once your agent is deployed, you need a way to observe traces from real traffic in one place.

Cloud Trace is the distributed tracing component of Google Cloud Observability. It collects and visualizes trace data so you can monitor latency, debug errors, and improve performance across your applications. For ADK agents, Cloud Trace captures how each request flows through model calls, tool executions, and agent steps, so you can pinpoint bottlenecks and errors in production.

Overview

Cloud Trace is built on OpenTelemetry, an open-source standard that supports many languages and ingestion methods for generating trace data. This aligns with observability practices for ADK applications, which also leverage OpenTelemetry-compatible instrumentation, allowing you to:

• Trace agent interactions: Cloud Trace continuously gathers and analyzes trace data from your project, enabling you to rapidly diagnose latency issues and errors within your ADK applications. This automatic data collection simplifies the process of identifying problems in complex agent workflows.
• Debug issues: Quickly diagnose latency issues and errors by analyzing detailed traces. These traces are crucial for understanding issues that manifest as increased communication latency across different services or during specific agent actions like tool calls.
• In-depth analysis and visualization: Trace Explorer is the primary tool for analyzing traces, offering visual aids like heatmaps for span duration and line charts for span rates. It also provides a spans table, groupable by service and operation, which gives one-click access to representative traces and a waterfall view to easily identify bottlenecks and sources of errors within your agent's execution path.

The following example will assume the following agent directory structure:

working_dir/
├── weather_agent/
│   ├── agent.py
│   └── __init__.py
└── deploy_agent_engine.py
└── deploy_fast_api_app.py
└── agent_runner.py

=== "Python" ```python # weather_agent/agent.py

import os
from google.adk.agents import Agent

os.environ.setdefault("GOOGLE_CLOUD_PROJECT", "{your-project-id}")
os.environ.setdefault("GOOGLE_CLOUD_LOCATION", "global")
os.environ.setdefault("GOOGLE_GENAI_USE_ENTERPRISE", "True")


# Define a tool function
def get_weather(city: str) -> dict:
    """Retrieves the current weather report for a specified city.

    Args:
        city (str): The name of the city for which to retrieve the weather report.

    Returns:
        dict: status and result or error msg.
    """
    if city.lower() == "new york":
        return {
            "status": "success",
            "report": (
                "The weather in New York is sunny with a temperature of 25 degrees"
                " Celsius (77 degrees Fahrenheit)."
            ),
        }
    else:
        return {
            "status": "error",
            "error_message": f"Weather information for '{city}' is not available.",
        }


# Create an agent with tools
root_agent = Agent(
    name="weather_agent",
    model="gemini-flash-latest",
    description="Agent to answer questions using weather tools.",
    instruction="You must use the available tools to find an answer.",
    tools=[get_weather],
)
```

Cloud Trace setup

Use the ADK CLI

You can enable cloud tracing by adding a flag when deploying or running your agent using the ADK CLI.

=== "Python"

When deploying your agent using the `adk deploy` command:

```bash
adk deploy agent_engine \
    --project=$GOOGLE_CLOUD_PROJECT \
    --region=$GOOGLE_CLOUD_LOCATION \
    --otel_to_cloud \
    $AGENT_PATH
```

=== "Go"

When running your agent built with the ADK Go launcher:

```bash
adkgo web -otel_to_cloud
```

Programmatic setup

Use ADK app abstractions

=== "Python"

If you are using the Agent Platform SDK `AdkApp` abstraction, you can enable cloud tracing by adding `enable_tracing=True`:

```python
from vertexai.agent_engines import AdkApp

adk_app = AdkApp(
    agent=root_agent,
    enable_tracing=True,
)
```

Use telemetry modules

For fully customized agent runtimes, you can enable cloud tracing by using the built-in telemetry modules.

=== "Python"

```python
from google.adk.telemetry import google_cloud
from google.adk.telemetry.setup import maybe_set_otel_providers

# Get GCP exporters configuration
hooks = google_cloud.get_gcp_exporters(enable_cloud_tracing=True)

# Initialize and set global OTel providers
maybe_set_otel_providers(otel_hooks_to_setup=[hooks])
```

=== "TypeScript"

```typescript
import { getGcpExporters, maybeSetOtelProviders } from '@google/adk';

// Get GCP exporters configuration
const gcpExporters = await getGcpExporters({
  enableTracing: true,
});

// Initialize and set global OTel providers
maybeSetOtelProviders([gcpExporters]);

// ... your agent code ...
```

=== "Go"

```go
import (
	"context"
	"log"
	"time"

	"google.golang.org/adk/v2/telemetry"
)

func main() {
	ctx := context.Background()

	// Initialize telemetry with cloud export enabled.
	// By default, the GCP project ID is read from the GOOGLE_CLOUD_PROJECT environment variable.
	// You can also specify it explicitly using telemetry.WithGcpResourceProject("my-project").
	telemetryProviders, err := telemetry.New(ctx,
		telemetry.WithOtelToCloud(true),
		// telemetry.WithGcpResourceProject("your-project-id"),
	)
	if err != nil {
		log.Fatalf("failed to initialize telemetry: %v", err)
	}
	defer func() {
		shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
		defer cancel()
		if err := telemetryProviders.Shutdown(shutdownCtx); err != nil {
			log.Printf("failed to shutdown telemetry: %v", err)
		}
	}()

	// Register as global OTel providers
	telemetryProviders.SetGlobalOtelProviders()

	// ... your agent code ...
}
```

Inspect Cloud Trace data

After the setup is complete, whenever you interact with the agent, it will automatically send trace data to Cloud Trace. You can inspect the traces by visiting the Trace Explorer in the Google Cloud Console.

You will see all available traces produced by the ADK agent, with span names such as invoke_agent, generate_content, call_llm, and execute_tool.

If you click on one of the traces, you will see a waterfall view of the detailed process, similar to the trace view in the local ADK web UI.

Captured attributes

The Agent Development Kit (ADK) enriches traces with telemetry attributes to help you filter, monitor, and analyze agent behavior.

Attribute	Description
gen_ai.agent.name	The name of the agent being executed.
gcp.vertex.agent.invocation_id	Unique ID of the invocation.
gcp.vertex.agent.event_id	ID of the specific event.
gen_ai.conversation.id	The session or conversation ID.
gcp.vertex.agent.session_id	The session ID associated with the agent invocation context.
gcp.vertex.agent.llm_request	Serialized LLM request containing prompt text and configuration.
gcp.vertex.agent.llm_response	Serialized LLM response containing model output.
gcp.vertex.agent.tool_call_args	Serialized arguments passed to tool calls.
gcp.vertex.agent.tool_response	Serialized result returned by the tool.
gcp.vertex.agent.data	Serialized data payloads sent to the agent.

Data privacy and payload redaction

To prevent exposing sensitive data and Personally Identifiable Information (PII) in production:

• Deployment default: When deploying with adk deploy agent_engine --otel_to_cloud, ADK automatically sets ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS='false' unless it is already defined in .env settings. For other targets like Cloud Run or GKE, set this variable explicitly.
• Redacted payloads: When ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS is 'false' or '0', payload attributes (gcp.vertex.agent.llm_request, gcp.vertex.agent.llm_response, gcp.vertex.agent.tool_call_args, gcp.vertex.agent.tool_response, and gcp.vertex.agent.data) are replaced with placeholder values, such as "{}" or "N/A".
• Enabling capture: To capture full payloads for local testing or debugging, explicitly set ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS='true' in your .env file or environment variables.
• OpenTelemetry message capturing: Set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT='true' (or '1') to enable logging of prompt and response content in OpenTelemetry events.

Resources

To learn more about tracing, OpenTelemetry, and Google Cloud integrations, explore the following documentation:

• Google Cloud Trace Documentation
• OpenTelemetry Documentation
• Connect to Google Cloud and Agent Platform