google__adk-docs/docs/observability/traces.md

Agent activity traces

Supported in ADKPython v1.17.0Go v1.0.0Kotlin v0.1.0

Agent Development Kit (ADK) provides distributed tracing capabilities to help you visualize the end-to-end journey of a request as it travels through your agent's architecture. While metrics tell you how long a process took and logs tell you what happened, traces connect these events, showing you exactly where the time was spent and the hierarchical relationship between LLM reasoning, tool calls, and external APIs.

Traces philosophy

ADK's approach to tracing is built on standard protocols to ensure seamless integration with your existing observability stack.

• OpenTelemetry Semantic Conventions: ADK implements the OpenTelemetry (OTel) Semantic Conventions for GenAI. This ensures that trace spans and attributes are recorded under standard, predictable names.
• OTLP Wire Format: ADK emits data using the standard OTLP format, ensuring that your traces will seamlessly integrate into any OTel-compatible backend (e.g., Google Cloud Trace, Jaeger, Grafana Tempo, Datadog).
• Hierarchical Visualization: Traces are organized into "Spans." An agent run is a root span, which contains child spans for LLM operations, which may in turn contain child spans for tool executions. This creates a clear "waterfall" view of the agent's reasoning loop.
• Context Propagation: ADK automatically passes trace context across process boundaries, ensuring that if your agent calls an external microservice via a tool, that service's spans are linked to the agent's root trace.

---

Traces schema

When tracing is enabled, ADK automatically instruments key operations following the OpenTelemetry GenAI Semantic Conventions for Agents. A typical trace waterfall includes the following spans:

Span Name	Type	Description	Key Attributes
invoke_agent	Client / Internal Span	Describes GenAI agent invocation over a remote service or locally. Represents the lifecycle of an agent interaction.	gen_ai.agent.name, gen_ai.system
invoke_workflow	Child Span	Describes the invocation of a multi-step agentic workflow.	gen_ai.workflow.name, gen_ai.system
execute_tool	Child Span	Represents the execution of a specific tool or function call requested by the GenAI system.	gen_ai.tool.name, gen_ai.system
generate_content {model.name}	Internal Span	Represents the invocation of the underlying language model (via the GenAI SDK) to generate content. It tracks the request parameters, response details, and usage metrics.	gen_ai.operation.name, gen_ai.system, gen_ai.request.model, gen_ai.agent.name, gen_ai.conversation.id, user.id, gen_ai.request.top_p, gen_ai.request.max_tokens, gen_ai.response.finish_reasons, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens

---

Traces export setup

Traces export in ADK Web

If you are running your agent using the adk web or adk api_server CLI commands, you can configure trace exports.

OTLP export

To export traces to an OTLP-compatible backend, set the standard OTel environment variables:

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://your-collector:4318/v1/traces"
adk web path/to/your/agents_dir

Note: You can also set the general OTEL_EXPORTER_OTLP_ENDPOINT environment variable if you would like to send metrics and logs to the same endpoint in addition to traces.

GCP export

To enable trace export to Google Cloud Trace, use the --otel_to_cloud flag:

adk web --otel_to_cloud path/to/your/agents_dir

Programmatic traces export

You can also configure trace export programmatically in your application code.

OTLP export setup

To enable tracing and export spans to an OpenTelemetry Collector programmatically:

from google.adk.telemetry.setup import maybe_set_otel_providers
import os

os.environ["OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"] = "http://your-collector:4318/v1/traces"
os.environ["OTEL_SERVICE_NAME"] = "your-adk-agent"
os.environ["OTEL_RESOURCE_ATTRIBUTES"] = "key1=value1,key2=value2"
maybe_set_otel_providers()

GCP export setup

To export traces to Google Cloud Trace programmatically, use the OpenTelemetry Google Cloud exporter. Here is an example in Python:

from google.adk.telemetry.google_cloud import get_gcp_exporters
from google.adk.telemetry.setup import maybe_set_otel_providers
import os

gcp_exporters = get_gcp_exporters(
  enable_cloud_tracing = True,
)
os.environ["OTEL_SERVICE_NAME"] = "your-adk-agent"
os.environ["OTEL_RESOURCE_ATTRIBUTES"] = "key1=value1,key2=value2"
maybe_set_otel_providers([gcp_exporters])

Kotlin programmatic setup

In Kotlin, ADK automatically uses the GlobalOpenTelemetry instance to export traces. You should configure your OpenTelemetry SDK before starting the agent.

OTLP export setup

To enable tracing and export spans to an OpenTelemetry Collector, configure the OpenTelemetry SDK and register it globally:

--8<-- "examples/kotlin/snippets/observability/SetupExample.kt:full_example"