skills/gastownhall__beads
1
0
Fork 0
mirror of https://github.com/gastownhall/beads.git synced 2026-09-14 20:17:24 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
main
gastownhall__beads/docs/reference/observability.md
T
Graeme Foster 1cf7490c71 refactor(telemetry): adopt standard OTel SDK env vars with BD_OTEL_* back-compat (#3859) 
* refactor(telemetry): adopt standard OpenTelemetry SDK env vars with explicit BD_OTEL_ENABLED gate

Telemetry configuration moves from BD_OTEL_* data variables to the
standard OTEL_* environment variables. Activation now requires
BD_OTEL_ENABLED=true so a machine-global OTEL_* setting (for some other
instrumented tool) cannot silently turn bd telemetry on.

The resource picks up resource.WithFromEnv() so OTEL_SERVICE_NAME and
OTEL_RESOURCE_ATTRIBUTES take effect — users can override service.name
or stamp deployment.environment without forking.

Backwards compatibility: BD_OTEL_METRICS_URL, BD_OTEL_LOGS_URL, and
BD_OTEL_STDOUT continue to work and activate telemetry on their own.
Each translates to its OTEL_* equivalent unconditionally — a legacy
value wins over a pre-existing OTEL_* value so cross-tool environments
don't redirect bd traffic. Every invocation that sees a BD_OTEL_*
variable logs a one-line deprecation warning to stderr.

OTLP endpoint discovery is delegated to the SDK; the explicit URL
helper in internal/telemetry/otlp.go is removed.

OTEL_SDK_DISABLED=true forces telemetry off even when an opt-in is in
effect, matching SDK conventions.

Tests cover: legacy var translation + override semantics, master-switch
activation, OTEL_*-alone-doesn't-activate, OTEL_SDK_DISABLED override,
WithFromEnv resource attribute merge, deprecation warning emission, and
otlpMetricsEndpointSet detection (specific endpoint, generic endpoint,
negative).

Part of the gastownhall/beads PR #3475 split: isolates the OTEL_* env
var standardisation from the WrapStorage wiring (#3858) and the
bd.prefix attribute work (separate follow-up).

* refactor(cmd/bd): extract command-telemetry wiring into testable helpers

cmd/bd/main.go's PreRunE block contained two pieces of OTel wiring —
telemetry.Init and the bd.command.<name> root span construction — that
silently decay if a future refactor drops them. Same regression class
as the WrapStorage bug (telemetry implemented but never called) —
wiring buried in cobra PreRunE escapes review.

Extract two helpers in cmd/bd/command_telemetry.go:

  - commandSpanAttrs(cmdName, version, args): pure function returning
    the initial attribute set for the bd.command.<name> span. The
    bd.actor attribute is stamped later via SetAttributes once
    getActorWithGit() resolves.
  - startCommandTelemetry(...): composes Init + Tracer.Start. Single
    callable line in main.go.

Three tests cover the commandSpanAttrs output (full attrs and the
empty-args edge case) and the startCommandTelemetry wiring contract:
it must always return a non-nil context and span, since the
bd.command.<name> span is the parent for every storage and AI span
downstream — silently returning nil would break trace nesting across
the whole invocation. Whether the span actually records is governed
by telemetry.Init which is covered in internal/telemetry/telemetry_test.go.

main.go shrinks by 10 lines.

* fixup(cmd/bd): stage command_telemetry.go split omitted from merge commit

The previous merge commit (def9f97a4) resolved main.go's PersistentPreRunE
conflict to call initTelemetry()/startCommandSpan() but the working-tree edit
to cmd/bd/command_telemetry.go that defines those two helpers was never
staged, so the merge commit as recorded does not compile:

  cmd/bd/main.go:680:3: undefined: initTelemetry
  cmd/bd/main.go:706:26: undefined: startCommandSpan

This stages that omitted edit: command_telemetry.go now defines
initTelemetry (the Init half) and startCommandSpan (the span-start half,
using commandSpanAttrs) as separate functions, with startCommandTelemetry
kept as their combined form for callers/tests that don't need to interleave
other setup between the two steps. See the merge commit message for why the
split is necessary (main's usage-metrics bootstrap block sits between the
two steps for one internal subcommand).

Verified: CGO_ENABLED=0 go build -tags gms_pure_go ./... now succeeds.

Agent-Signature: claude-fable-5-high on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(telemetry): gate the OTLP metric exporter on OTEL_METRICS_EXPORTER

A console-only or none selection — including the legacy BD_OTEL_STDOUT=true
translation — previously still installed an OTLP reader whenever a
machine-global OTEL_EXPORTER_OTLP_ENDPOINT existed, silently exporting bd
metrics to an endpoint configured for some other tool. The selector is now
parsed per the SDK spec (comma-separated; unset defaults to otlp; none
disables) and the OTLP reader requires the selection to include otlp.

The BD_OTEL_STDOUT translation keeps both destinations when
BD_OTEL_METRICS_URL is also set (console,otlp), preserving the existing
both-exporters behavior for that combination.

Also bound the test-cleanup Shutdown with a 2s context so suites configured
with unreachable endpoints (example.invalid) don't spend ~30s in OTLP
retries.

Both issues found by a cross-vendor (Codex) review of the rebased branch.

Agent-Signature: claude-fable-5-high on behalf of matt wilkie

---------

Co-authored-by: matt wilkie <maphew@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-03 00:05:06 -07:00

218 lines
8.4 KiB
Markdown
Raw Permalink Blame History

---
title: Observability (OpenTelemetry)
description: Exporting bd metrics and traces over OpenTelemetry (OTLP), with a local VictoriaMetrics and Grafana stack, env vars, and a metric reference.
---
Beads exports metrics via OTLP HTTP. Telemetry is **disabled by default** — zero overhead when no variable is set.
## Recommended local stack
| Service | Port | Role |
|---------|------|------|
| VictoriaMetrics | 8428 | OTLP metrics storage |
| VictoriaLogs | 9428 | Reserved for future OTLP log storage |
| Grafana | 9429 | Dashboards |
```bash
# From your personal stack's opentelemetry/ folder
docker compose up -d
```
## Configuration
Telemetry is **explicit opt-in**. Set `BD_OTEL_ENABLED=true` and configure the
exporter via standard OpenTelemetry SDK environment variables — both go in
your shell profile or workspace `.env`:
```bash
export BD_OTEL_ENABLED=true
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://localhost:8428/opentelemetry/api/v1/push
```
A standard `OTEL_*` variable alone will not turn bd telemetry on — `bd` won't
auto-activate from a machine-global `OTEL_*` setting that was set for some
other instrumented tool.
Log export is not implemented yet. `BD_OTEL_LOGS_URL` is reserved for a future
VictoriaLogs exporter and does not activate telemetry today.
### Shell profile (recommended)
```bash
# ~/.zshrc or ~/.bashrc
export BD_OTEL_ENABLED=true
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://localhost:8428/opentelemetry/api/v1/push
```
### Environment variables
| Variable | Example | Description |
|----------|---------|-------------|
| `BD_OTEL_ENABLED` | `true` | **Master switch.** Activates telemetry. Without it, the variables below are ignored. |
| `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` | `http://localhost:8428/opentelemetry/api/v1/push` | Push metrics to an OTLP HTTP receiver. |
| `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | `http://localhost:9428/insert/opentelemetry/v1/logs` | Push logs to an OTLP HTTP receiver (reserved for future log export). |
| `OTEL_TRACES_EXPORTER` | `console` | Write spans to stderr (dev/debug). |
| `OTEL_METRICS_EXPORTER` | `console` | Metric exporter selection, comma-separated (`otlp`, `console`, `none`). Unset defaults to `otlp`; `console` writes to stderr (dev/debug); a selection without `otlp` never exports to an OTLP endpoint, even if one is configured machine-globally. |
| `OTEL_SERVICE_NAME` | `bd` | Override the `service.name` resource attribute. |
| `OTEL_RESOURCE_ATTRIBUTES` | `deployment.environment=workstation,team=infra` | Extend or override resource attributes (comma-separated `key=value`). |
| `OTEL_SDK_DISABLED` | `true` | Force telemetry off even when `BD_OTEL_ENABLED=true` is set. |
### Resource attributes
Every metric and span carries the OTel resource describing the bd process:
| Attribute | Value | Notes |
|-----------|-------|-------|
| `service.name` | `bd` | Override with `OTEL_SERVICE_NAME`. |
| `service.version` | bd version | |
Add anything else via `OTEL_RESOURCE_ATTRIBUTES`.
### Local debug mode
```bash
BD_OTEL_ENABLED=true OTEL_TRACES_EXPORTER=console OTEL_METRICS_EXPORTER=console bd list
```
### Legacy environment variables (deprecated)
The earlier `BD_OTEL_*` data variables are honored for backwards
compatibility. Setting any of them activates telemetry on its own (no
`BD_OTEL_ENABLED=true` required) and translates to the standard OTLP equivalent
— a legacy value wins over a pre-existing `OTEL_*` value so a machine-global
`OTEL_*` setting cannot silently redirect bd telemetry. Each `bd` invocation
that sees one logs a one-line deprecation warning to stderr:
| Legacy | Standard equivalent |
|--------|---------------------|
| `BD_OTEL_METRICS_URL` | `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` |
| `BD_OTEL_LOGS_URL` | `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` |
| `BD_OTEL_STDOUT=true` | `OTEL_TRACES_EXPORTER=console` + `OTEL_METRICS_EXPORTER=console` |
## Verification
```bash
bd list # triggers metrics → visible in VictoriaMetrics
```
Verification queries in Grafana (VictoriaMetrics datasource):
```promql
bd_storage_operations_total # raw counter
sum(rate(bd_storage_operations_total[5m])) # operation rate
```
### Confirm storage instrumentation locally
To verify the storage decorator chain is wired up without standing up a
collector, run `bd` with stdout exporters and look for `bd.storage.*`
records on stderr:
```bash
BD_OTEL_STDOUT=true bd list 2>&1 | grep -F bd.storage.operations
```
Expect at least one line per storage call (`GetReadyWork`, `GetIssue`, …).
If `bd.storage.*` and `bd.issue.count` are absent but `bd.db.pool_*` is
present, the storage decorator is not in the chain — check
`wireStorageDecorators` in `cmd/bd/storage_chain.go`.
---
## Metrics
### Storage (`bd_storage_*`)
| Metric | Type | Attributes | Description |
|--------|------|------------|-------------|
| `bd_storage_operations_total` | Counter | `db.operation` | Storage operations executed |
| `bd_storage_operation_duration_ms` | Histogram | `db.operation` | Operation duration (ms) |
| `bd_storage_errors_total` | Counter | `db.operation` | Storage errors |
> These metrics are emitted by `InstrumentedStorage`, the beads SDK wrapper.
### Dolt database (`bd_db_*`)
| Metric | Type | Attributes | Description |
|--------|------|------------|-------------|
| `bd_db_retry_count_total` | Counter | — | SQL retries in server mode |
| `bd_db_lock_wait_ms` | Histogram | `dolt_lock_exclusive` | Wait time to acquire database locks |
### Issues (`bd_issue_*`)
| Metric | Type | Attributes | Description |
|--------|------|------------|-------------|
| `bd_issue_count` | Gauge | `status` | Number of issues by status |
`status` values: `open`, `in_progress`, `closed`, `deferred`.
### AI (`bd_ai_*`)
| Metric | Type | Attributes | Description |
|--------|------|------------|-------------|
| `bd_ai_input_tokens_total` | Counter | `bd_ai_model` | Anthropic input tokens |
| `bd_ai_output_tokens_total` | Counter | `bd_ai_model` | Anthropic output tokens |
| `bd_ai_request_duration_ms` | Histogram | `bd_ai_model` | API call latency |
---
## Traces (spans)
Spans are only exported when `OTEL_TRACES_EXPORTER=console` — there is no trace backend in the recommended local stack.
| Span | Source | Description |
|------|--------|-------------|
| `bd.command.<name>` | CLI | Total duration of the command |
| `dolt.exec` / `dolt.query` / `dolt.query_row` | SQL | Each SQL operation |
| `dolt.commit` / `dolt.push` / `dolt.pull` / `dolt.merge` | Dolt VC | Version control procedures |
| `ephemeral.count` / `ephemeral.nuke` | SQLite | Ephemeral store operations |
| `hook.exec` | Hooks | Hook execution (root span, fire-and-forget) |
| `tracker.sync` / `tracker.pull` / `tracker.push` | Sync | Tracker sync phases |
| `anthropic.messages.new` | AI | Claude API calls |
### Notable attributes
**`bd.command.<name>`**
| Attribute | Description |
|-----------|-------------|
| `bd.command` | Subcommand name (`list`, `create`, ...) |
| `bd.version` | bd version |
| `bd.args` | Raw arguments passed to the command (e.g. "create 'title' -p 2") |
| `bd.actor` | Actor (resolved from git config / env) |
**`hook.exec`**
| Attribute / Event | Description |
|-------------------|-------------|
| `hook.event` | Event type (`create`, `update`, `close`) |
| `hook.path` | Absolute path to the script |
| `bd.issue_id` | ID of the triggering issue |
| event `hook.stdout` | Script standard output (truncated to 1 024 bytes) |
| event `hook.stderr` | Script error output (truncated to 1 024 bytes) |
The `hook.stdout` / `hook.stderr` events carry two attributes: `output` (the text) and `bytes` (original size before truncation).
---
## Architecture
```
cmd/bd/main.go
└─ telemetry.Init()
├─ OTEL_TRACES_EXPORTER=console → TracerProvider stdout
├─ OTEL_METRICS_EXPORTER=console → MeterProvider stdout
└─ OTEL_EXPORTER_OTLP_METRICS_ENDPOINT → MeterProvider HTTP → VictoriaMetrics
internal/storage/dolt/ → bd_db_* metrics + dolt.* spans
internal/storage/ephemeral/ → ephemeral.* spans
internal/hooks/ → hook.exec span
internal/tracker/ → tracker.* spans
internal/compact/ → bd_ai_* metrics + anthropic.* spans
internal/telemetry/storage.go → bd_storage_* metrics (SDK wrapper)
```
When no OpenTelemetry SDK environment variable selects an exporter,
`telemetry.Init()` installs **no-op** providers: hot paths execute only no-op
calls with no memory allocation.
Reference in New Issue View Git Blame Copy Permalink