mirror of
https://github.com/vectorize-io/hindsight.git
synced 2026-09-14 19:31:49 +08:00
MΞTΔ
a6f99c995c
* feat(helm): add Prometheus operator ServiceMonitor support The api (port 8888) and worker (port 8889) containers expose Prometheus format metrics at /metrics (verified against app source); the chart had no wiring for them — the worker's scrape annotations are gated behind the unrelated podAnnotations value and the api service had nothing. Add a gated metrics.serviceMonitor block that emits per-component ServiceMonitor resources (api always when enabled, worker only when worker.enabled). Selection labels are configurable for the Prometheus operator's serviceMonitorSelector (e.g. release: kube-prometheus-stack). Also scrape the dedicated worker in the dev LGTM compose stack, which previously only scraped the api on :8888. Verified end-to-end on k3d + kube-prometheus-stack: all three targets (api + 2 worker pods via headless endpoints) discovered and up=1. * fix(helm): address ServiceMonitor review
Hindsight Monitoring Stack
Docker-based monitoring stack using Grafana LGTM (Loki, Grafana, Tempo, Mimir) for complete observability.
Quick Start
# Start the monitoring stack
./scripts/dev/start-monitoring.sh
# Or manually with docker-compose
cd scripts/dev/monitoring && docker-compose up -d
Access
- Grafana UI: http://localhost:3000
- No login required (anonymous admin enabled for dev)
Features
- Traces: OpenTelemetry traces with GenAI semantic conventions (Tempo)
- Metrics: Prometheus scraping of Hindsight API
/metricsendpoint - Logs: Loki log aggregation (future)
- Dashboards: Pre-configured dashboards from
monitoring/grafana/dashboards/:- Hindsight Operations
- Hindsight LLM Metrics
- Hindsight API Service
Configure Hindsight API
Set these environment variables in your .env:
# Enable tracing
HINDSIGHT_API_OTEL_TRACES_ENABLED=true
# Grafana Tempo OTLP endpoint (HTTP)
HINDSIGHT_API_OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
# Optional: Custom service name
HINDSIGHT_API_OTEL_SERVICE_NAME=hindsight-api
# Optional: Deployment environment
HINDSIGHT_API_OTEL_DEPLOYMENT_ENVIRONMENT=development
View Data
Traces
- Open http://localhost:3000
- Go to Explore (compass icon)
- Select Tempo as data source
- Click "Search" to see recent traces
Metrics & Dashboards
- Open http://localhost:3000
- Go to Dashboards (dashboard icon)
- Browse the Hindsight folder
Raw Metrics
- Prometheus metrics: http://localhost:8888/metrics
- PromQL queries: Explore → Prometheus
Ports
| Port | Service |
|---|---|
| 3000 | Grafana UI |
| 4317 | OTLP gRPC endpoint |
| 4318 | OTLP HTTP endpoint |
Stop
cd scripts/dev/monitoring && docker-compose down
Architecture
- Single Container: Grafana LGTM (~515MB) provides all observability components
- Auto-provisioned Dashboards: Dashboards from
monitoring/grafana/dashboards/are automatically loaded - Prometheus Scraping: Configured to scrape Hindsight API at
host.docker.internal:8888/metricsevery 5 seconds - Network: Uses
hindsight-network(shared with API for future service-to-service tracing)