proffesor-for-testing__agentic-qe/docs/telemetry-cli-commands.md

Telemetry CLI Commands

Overview

The telemetry CLI commands provide comprehensive monitoring and querying capabilities for the Agentic QE Fleet's OpenTelemetry instrumentation, including metrics, traces, token usage, and cost tracking.

Commands

aqe telemetry status

Show the current telemetry configuration and initialization status.

aqe telemetry status
aqe telemetry status --json

Output includes:

• Initialization status
• Service name and version
• Environment configuration
• OTLP endpoint configuration
• Active exporters (console, gRPC, auto-instrumentation)
• Trace sample rate and metric export interval

Example:

📊 Telemetry Status

✅ Initialized: Yes

🔧 Configuration:
  Service:     agentic-qe-fleet v1.8.4
  Environment: development
  OTLP:        http://localhost:4317
  Metrics:     http://localhost:4317

📤 Exporters:
  Console:     ❌ Disabled
  gRPC:        ✅ Enabled
  Auto-Instr:  ✅ Enabled

⚙️  Settings:
  Trace Sample Rate: 100%
  Metric Interval:   60s

---

aqe telemetry metrics [metric-name]

Query and display various metrics. If no metric name is provided, shows a summary of all metrics.

# Show all metrics
aqe telemetry metrics

# Show specific metric type
aqe telemetry metrics tokens
aqe telemetry metrics cost
aqe telemetry metrics system

# Filter by agent
aqe telemetry metrics --agent test-gen-001

# Output as JSON
aqe telemetry metrics --json

Metric Types:

tokens - Token Usage

Shows LLM token consumption including:

• Input tokens
• Output tokens
• Cache write tokens (Anthropic)
• Cache read tokens (Anthropic)
• Total tokens

cost - Cost Breakdown

Shows cost analysis including:

• Input cost ($)
• Output cost ($)
• Cache write cost ($)
• Cache read cost ($)
• Total cost ($)
• Cache savings ($ and %)

system - System Metrics

Shows system resource usage including:

• Memory usage (heap, RSS)
• CPU usage (user, system)
• Process information (PID, Node version, platform)
• Uptime

Example:

📊 Fleet Metrics Summary

🌐 Fleet-Wide:
┌──────────────┬──────────────┬──────────────┬──────────────┐
│ (index)      │ Input        │ Output       │ Total        │
├──────────────┼──────────────┼──────────────┼──────────────┤
│ Total Tokens │ '245,892'    │ '52,134'     │ '298,026'    │
│ Total Cost   │ '$0.7377'    │ '$0.7820'    │ '$1.5197'    │
└──────────────┴──────────────┴──────────────┴──────────────┘
💰 Cache Savings: $0.2341

🤖 Per-Agent Breakdown:
┌───────────────┬──────────┬────────────┬────────────┬─────────────────────────────┐
│ (index)       │ Tokens   │ Cost       │ Provider   │ Model                       │
├───────────────┼──────────┼────────────┼────────────┼─────────────────────────────┤
│ test-gen-001  │ '145,892'│ '$0.8234'  │ 'anthropic'│ 'claude-sonnet-4-5-2025...' │
│ coverage-002  │ '98,134' │ '$0.4963'  │ 'anthropic'│ 'claude-sonnet-4-5-2025...' │
│ security-003  │ '54,000' │ '$0.2000'  │ 'anthropic'│ 'claude-3-5-haiku-20241...' │
└───────────────┴──────────┴────────────┴────────────┴─────────────────────────────┘

---

aqe telemetry trace [trace-id]

View distributed traces. Requires OTLP backend configuration (Jaeger, Zipkin, Grafana Tempo).

# View specific trace
aqe telemetry trace abc123def456

# View recent traces by agent
aqe telemetry trace --agent test-generator --limit 10

# Output as JSON
aqe telemetry trace --json

Note: Trace viewing requires:

1. An OTLP collector or backend (Jaeger, Zipkin, Grafana Tempo)
2. OTEL_EXPORTER_OTLP_ENDPOINT environment variable configured
3. Fleet restarted with telemetry enabled

Example backends:

• Jaeger: https://www.jaegertracing.io/
• Zipkin: https://zipkin.io/
• Grafana Tempo: https://grafana.com/oss/tempo/

---

aqe telemetry export-prometheus

Export metrics in Prometheus format for integration with Prometheus monitoring.

aqe telemetry export-prometheus > metrics.txt

Output format:

# HELP aqe_fleet_tokens_total Total tokens consumed by fleet
# TYPE aqe_fleet_tokens_total counter
aqe_fleet_tokens_total{type="input"} 245892
aqe_fleet_tokens_total{type="output"} 52134

# HELP aqe_fleet_cost_total Total cost incurred by fleet
# TYPE aqe_fleet_cost_total counter
aqe_fleet_cost_total 1.5197

# HELP aqe_fleet_cache_savings_total Total cache savings
# TYPE aqe_fleet_cache_savings_total counter
aqe_fleet_cache_savings_total 0.2341

aqe_agent_tokens_total{agent_id="test-gen-001",provider="anthropic",model="claude-sonnet-4",type="input"} 145892
aqe_agent_tokens_total{agent_id="test-gen-001",provider="anthropic",model="claude-sonnet-4",type="output"} 24567
aqe_agent_cost_total{agent_id="test-gen-001",provider="anthropic",model="claude-sonnet-4"} 0.8234

---

Integration with OpenTelemetry

Environment Variables

Configure telemetry behavior with these environment variables:

# OTLP endpoint for traces and metrics
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

# Separate metrics endpoint (optional)
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://localhost:4318/v1/metrics

# Protocol: http/protobuf or grpc (default: grpc)
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

# Enable console export for debugging
export OTEL_CONSOLE_EXPORT=true

# Trace sample rate (0.0 to 1.0)
export OTEL_TRACES_SAMPLER_ARG=1.0

# Metric export interval (milliseconds)
export OTEL_METRIC_EXPORT_INTERVAL=60000

# Enable debug logging
export OTEL_LOG_LEVEL=debug

Setting up OTLP Collector

Option 1: Jaeger All-in-One

docker run -d --name jaeger \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 16686:16686 \
  jaegertracing/all-in-one:latest

Then access Jaeger UI at http://localhost:16686

Option 2: OpenTelemetry Collector + Jaeger

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true
  prometheus:
    endpoint: "0.0.0.0:8889"

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [jaeger]
    metrics:
      receivers: [otlp]
      exporters: [prometheus]

docker run -d -p 4317:4317 -p 4318:4318 -p 8889:8889 \
  -v $(pwd)/otel-collector-config.yaml:/etc/otel-collector-config.yaml \
  otel/opentelemetry-collector:latest \
  --config=/etc/otel-collector-config.yaml

---

Usage Examples

Monitor Token Usage During Test Generation

# Run test generation
aqe generate tests src/services/user.ts

# Check token usage
aqe telemetry metrics tokens

# Check costs
aqe telemetry metrics cost

Export Metrics to Prometheus

# Start Prometheus scraping
aqe telemetry export-prometheus | curl --data-binary @- http://localhost:9091/metrics/job/aqe-fleet

# Or save to file for manual inspection
aqe telemetry export-prometheus > /tmp/aqe-metrics.txt

Monitor System Resources

# Watch system metrics in real-time
watch -n 2 'aqe telemetry metrics system'

Query Specific Agent Performance

# Get metrics for specific agent
aqe telemetry metrics --agent test-gen-001 --json | jq '.cost'

# Export to CSV
aqe telemetry metrics --json | jq -r '.agents[] | [.id, .cost.totalCost] | @csv'

---

Performance Metrics

The telemetry system tracks:

1. Token Metrics

   • Input/output token counts
   • Cache utilization (Anthropic models)
   • Per-agent and fleet-wide totals

2. Cost Metrics

   • Real-time cost tracking per API call
   • Cache savings calculations
   • Multi-provider cost comparison

3. System Metrics

   • Memory usage (heap, RSS)
   • CPU utilization
   • Process information

4. Quality Metrics (via OpenTelemetry)

   • Test execution duration
   • Coverage percentages
   • Defect density
   • Quality gate pass rates

---

Troubleshooting

No Metrics Available

If metrics show zeros:

# Initialize telemetry
aqe init

# Run some agents
aqe generate tests src/

# Check metrics again
aqe telemetry metrics

Trace Viewing Not Working

# Check if OTLP endpoint is configured
aqe telemetry status

# Set OTLP endpoint
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

# Restart fleet
aqe start

High Token Costs

# Check cost breakdown
aqe telemetry metrics cost

# Identify expensive agents
aqe telemetry metrics --json | jq '.agents | sort_by(.cost.totalCost) | reverse | .[0:5]'

# Consider using cheaper models via routing
aqe routing enable

---

Related Documentation

• Telemetry Architecture
• OpenTelemetry Configuration
• Cost Tracking
• System Metrics