mirror of
https://github.com/joelhooks/joelclaw.git
synced 2026-09-19 01:24:04 +08:00
Joel Hooks
58ee00c49d
- summarizeRollupCost: OpenRouter-benchmark cost for rows without provider cost, wired into joelclaw usage and the daily token report - clickhouse-usage-query: source option (router|agents|all) so agent_usage.turn tailer events roll up alongside model_router.result; coverage % is router-only - claude tailer parser: dedupe by message.id (Claude Code repeats usage per content-block line; counts inflated ~2x) — inflated rows purged, offsets reset - inference: non-zero pi exit with empty output now fails for json callers too (was recorded as circuit success); --mode json event-stream soup no longer leaks into result.text; caller metadata can't clobber usage fields - daily report: rollup widened to 5000 groups; agent-sessions line added - k8s worker manifest published with SLOG_SYSTEM_ID (pod blocked by pre-existing flannel failure, see brain note follow-ups) - brain notes + skills updated: langfuse decommission receipts, usage receipts Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
65 lines
3.5 KiB
Plaintext
65 lines
3.5 KiB
Plaintext
---
|
|
title: "Observability Stack Map"
|
|
type: "resource"
|
|
status: "draft"
|
|
created_at: "2026-06-06"
|
|
tags:
|
|
- observability
|
|
- telemetry
|
|
- grafana
|
|
- otel
|
|
---
|
|
|
|
# Observability Stack Map
|
|
|
|
Source spark: Marc Köhlbrugge's 2026-06-06 X post mapping the common observability stack:
|
|
|
|
> Grafana → dashboards
|
|
> Prometheus → metrics
|
|
> Loki → logs
|
|
> Tempo → traces
|
|
> Alloy → collector/forwarder
|
|
> OpenTelemetry → app traces
|
|
> node-exporter → host metrics
|
|
> cAdvisor → container metrics
|
|
|
|
Source URL: https://x.com/marckohlbrugge/status/2063292480594891057
|
|
|
|
## Fit for joelclaw
|
|
|
|
JoelClaw already has structured telemetry, but it is mostly agent-readable and domain-specific:
|
|
|
|
- `@joelclaw/telemetry` emits OTEL-shaped events.
|
|
- `system-bus-worker` accepts `/observability/emit`.
|
|
- ClickHouse is the canonical raw OTEL event sink.
|
|
- Typesense must not be used as an unbounded raw OTEL/log warehouse; only bounded/searchable projections are acceptable after retention and query cutoffs are proven.
|
|
- Panda/old Central OTEL endpoints are relay-only and forward to Flagg's `/observability/emit` path.
|
|
- Langfuse is decommissioned (2026-07-09): all code paths removed — `traceLlmGeneration`, the `joelclaw langfuse` CLI command, sdk/inference-router/gateway tracing shims, and `@langfuse/*` deps are gone (`grep -ri langfuse packages/ apps/` returns zero). The live pi-output parsing it hosted moved to `packages/system-bus/src/lib/pi-output.ts`. Usage/cost accounting now reads straight from ClickHouse `model_router.result` + `agent_usage.turn` events via `packages/system-bus/src/lib/clickhouse-usage-query.ts` and `joelclaw usage`. Background: never migrated to Flagg, its legacy `traces` API 429s/422s under any real query load (hit during the 2026-07 backup/self-healing token-drain incident post-mortem). See `.brain/projects/agent-usage-observability-2026-07-09.svx`.
|
|
|
|
The Grafana stack is useful as an operator lens, not a replacement for JoelClaw's event memory.
|
|
|
|
## Recommended split
|
|
|
|
- **Keep ClickHouse** for raw agent-readable event history, search, and correlation.
|
|
- **Use Typesense only for derived/bounded projections** that need search UX; do not write raw OTEL there.
|
|
- **Use Prometheus** for numeric health: queue depths, request latency, error counters, worker restarts, k8s/container metrics.
|
|
- **Use Loki** for raw log streams when CLI `joelclaw logs` is not enough.
|
|
- **Use Tempo** only after services emit real distributed traces; otherwise it is ceremony.
|
|
- **Use Alloy** as the boring collector/forwarder that wires node, container, OTEL, logs, and metrics into the right sinks.
|
|
- **Use Grafana** as the human glanceable surface over Prometheus/Loki/Tempo/ClickHouse.
|
|
|
|
## Small first slice
|
|
|
|
Do not deploy the whole stack in one heroic blob. First slice:
|
|
|
|
1. Add Prometheus scrape targets for k8s, Dkron's existing Prometheus endpoint, and app-level `/metrics` where present.
|
|
2. Add Grafana with one dashboard: Central health, queue depth, recent failures, worker/runtime status.
|
|
3. Add Alloy once there are at least two sources to forward and normalize.
|
|
4. Defer Loki/Tempo until logs/traces are actually needed to answer questions the CLI cannot answer quickly.
|
|
|
|
## Open questions
|
|
|
|
- Should ClickHouse be queried directly from Grafana as the canonical OTEL event store once ADR-0224 lands?
|
|
- Which services should expose `/metrics` first: `system-bus-worker`, `restate-worker`, gateway daemon, or queue drainer?
|
|
- What is the alert boundary: Grafana alerting, existing gateway/operator alerting, or both with clear ownership?
|