mirror of
https://github.com/vectorize-io/hindsight.git
synced 2026-09-14 19:31:49 +08:00
perf/profiling-env
10 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Nicolò Boschi
|
05c775c415 |
fix(consolidation): teach the prompt to name delete targets, and count discarded batch responses (#4151, #4152) (#4183)
A consolidation response that fails `_ConsolidationBatchResponse` validation is classified FAIL_FAST, fails the batch, and gets bisected. When the halves then validate, every fact consolidates and nothing is left carrying `consolidation_failed_at` -- so `failed_consolidation` reads 0 and the run is indistinguishable from a clean one, even though everything those responses asked for was thrown away. #4152 is the concrete instance. `deletes[].observation_id` is required, but the prompt never showed the shape of a delete entry: both worked examples ended in `"deletes": []` and the `deletes` field rule said only *when* to delete. A model that answered with a reason-only delete had its whole response rejected -- the good creates and updates alongside it included -- and the supersession-cleanup path quietly did nothing for a whole backlog drain. Three changes: - Prompt: a worked example with a populated `deletes` array, and a field rule stating that every entry must carry the exact `observation_id`, that prose in `reason` is not enough, and that a bad entry costs the whole response. - `_DeleteAction` accepts `id` as an alias for `observation_id` -- the name a model copying the observation's own field emits, unambiguous because a delete entry has exactly one identifier. The field stays required (a delete naming no target has no defensible fallback), and the generated JSON schema still advertises `observation_id` alone, so grammar-constrained providers see no change. This only stops a near-miss from discarding the batch. - Visibility: a `hindsight.consolidation.batch_failures` counter labelled by failure class and exception type, an `llm_batch_failures` count in the consolidation job's stats, and a warning line in the run summary when it is non-zero. `failed_consolidation` is a gauge over stuck rows and structurally cannot report a failure that bisection recovered from. Recording happens on the exception path only, so a successful batch costs nothing. Tests: `test_consolidation_delete_schema.py` covers the alias, that a delete naming nothing still fails closed, that the JSON schema is unchanged, the prompt rules, and both end-to-end delete paths. `test_consolidation_batch_failure_visibility.py` covers the counter -- including that retried attempts each count, and that a clean run still reports zero. `test_consolidation_delete_prompt_llm.py` is the `hs_llm_core` prompt-following check against a real model. |
||
|
Nicolò Boschi
|
08a119eeb6 |
fix(observability): trace the worker process and join the caller's trace (#3625)
Two independent gaps meant a deployment that looked fully instrumented was instrumented on one half, and its traces never linked up with the caller's. Worker emits no spans (#3614). initialize_tracing() was called from exactly one place — the FastAPI lifespan — and the standalone `hindsight-worker` entrypoint never goes through it. Since both tracing chokepoints degrade to deliberate no-ops, consolidation, batch retain and mental-model refresh — most of the long-running work and token spend — produced nothing, with no error or warning to say so. The bootstrap moves into a shared tracing.initialize_tracing_from_config() that both entrypoints call. Workers default to the service name "hindsight-worker", matching the name they already report for metrics, while an explicit HINDSIGHT_API_OTEL_SERVICE_NAME still wins. No trace-context propagation (#3604). Every operation opened a new root span, so a caller's request and the Hindsight work it triggered were two unrelated traces. The ASGI instrumentation — already a declared dependency, previously unused — now extracts W3C traceparent and opens a SERVER span, which the engine's existing spans nest under through the ambient context, with no changes at those call sites. Requests without a traceparent still start their own root trace. Health and metrics URLs are excluded so probe traffic doesn't drown out real work, and per-ASGI-message spans are excluded both to cut span volume and because that leaves the raw receive callable untouched for ClientDisconnectCancellationMiddleware (#2122). Also adds a shutdown flush, so spans still queued in the BatchSpanProcessor survive SIGTERM — a consolidation span can be minutes long — and reports the real package version in service.version instead of a hardcoded 0.4.8. |
||
|
Nicolò Boschi
|
00bad17110 |
fix(api): add a DB-free liveness probe so a slow database stops restarting pods (#3337)
Adds /health/live (no DB access) and /health/ready alongside the existing /health, on the API server and the worker. Helm liveness probes now use /health/live; readiness stays on /health. Worker liveness reports seconds_since_last_poll for alerting without gating on it. Fixes #3329 |
||
|
Nicolò Boschi
|
9452ac29da |
feat(retain): report zero-fact documents at write time (#3040) (#3044)
* feat(retain): report zero-fact documents at write time (#3040) A document whose fact extraction legitimately returns zero facts is stored but unreachable: only memory_units carry embeddings, so recall and reflect cannot reach a document that owns none. The retain still succeeds, the operation reports completed, and nothing in the response, the webhook or the metrics says the document produced no memories — the operator has no way to know it needs a reprocess. FAIL_ON_EXTRACTION_ERRORS (#2721) cannot help by construction: there is no error to fail on. #2861 made retain.completed fire for zero-fact batches, but the payload is byte-identical to a successful one, so it still carries no signal. Add the count to all three write-time surfaces: - retain.completed gains data.memory_unit_count, filled inside the outbox callback on the retain's own connection so units written by the enclosing transaction are visible. - The synchronous retain response gains memory_units_created. - New counter hindsight.retain.documents.total{outcome=facts|no_facts}, emitted per document at both extraction exits. The webhook and the metric report the document's total *after* the retain, not what the call created: the delta path skips unchanged chunks, so an idempotent re-retain creates zero units while the document keeps every memory it had. Reporting units created would raise a false alarm on every re-submit. The count query only runs when the call created nothing, which is the path where no work was done anyway. Docs: how a retain mission trades away retrieval of the raw source, the three signals, the non-determinism caveat, and reprocess as the way back. * fix(retain): drop memory_units_created from the retain response The synchronous response field reported units created by that call, which is a different number from the one the webhook and the metric report (the document's total after the retain) and only ever populated on the sync path. The async path is the one that matters, and it is already covered by retain.completed carrying data.memory_unit_count. Removing it also takes the API surface back to identical with main — the webhook payload is now the only public shape change — so the regenerated Python/TypeScript/Go clients and the OpenAPI spec carry no delta. Also renames the metric's parameter to memory_unit_count to match what it is actually handed: the document total, not units created. |
||
|
Evo
|
955b0c523c | docs(monitoring): document worker operation metrics (#2296) | ||
|
Ben
|
a908ade6d6 |
docs: add Pydantic Logfire guide for Hindsight tracing (#1339)
* docs: add Pydantic Logfire as an OTel backend for Hindsight Hindsight already emits OpenTelemetry spans for retain / recall / reflect (plus their LLM sub-spans) via the existing OTLP HTTP exporter. Logfire is an OTel-native receiver, so wiring it up is three env vars — no code changes, no new dependency. - New /developer/logfire guide page: env-var config, what the trace tree looks like, pairing with logfire.instrument_pydantic_ai(), useful Logfire queries, and troubleshooting - Cross-link from the existing Distributed Tracing section in monitoring.md so Logfire sits next to Langfuse / DataDog / Honeycomb in the supported backends list * docs: drop dedicated Logfire page per review feedback Per Nicolò's review on this PR — the dedicated /developer/logfire page was mostly Logfire setup, not Hindsight. Keeping only the one-line mention in the existing OTLP-backends list in monitoring.md, with the link pointing to logfire.pydantic.dev directly. The setup walkthrough, query examples, and troubleshooting moved into the companion blog post (hindsight-marketing-content#113). |
||
|
Nicolò Boschi
|
db70fdbe5e |
feat: add LiteLLM LLM provider for Bedrock and 100+ providers (#679)
* feat: add LiteLLM LLM provider for Bedrock and 100+ providers Add a new `litellm` LLM provider that uses the LiteLLM SDK for chat completions and tool calling, enabling AWS Bedrock and 100+ other providers for Hindsight's core engine (retain, recall, reflect). - New LiteLLMLLM provider in engine/providers/litellm_llm.py - Registered in factory, valid providers list, and no-api-key set - Refactored API key validation to use requires_api_key() helper - Added boto3 dependency for Bedrock auth - Updated docs: configuration, models, monitoring, providers grid * feat: add bedrock as first-class LLM provider alias Add `bedrock` as a dedicated provider name that auto-prepends the `bedrock/` prefix to model names and delegates to LiteLLMLLM under the hood. This makes Bedrock support more discoverable — users set `HINDSIGHT_API_LLM_PROVIDER=bedrock` with plain Bedrock model IDs. * test: add Bedrock to CI provider tests - Add bedrock/us.amazon.nova-lite-v1:0 to MODEL_MATRIX in test_llm_provider.py - Add AWS credential check in should_skip_provider() - Pass AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION_NAME secrets to test-api job - Update default bedrock model to amazon.nova-2-lite-v1:0 * fix: regenerate docs skill files and bump memory test timeout - Regenerate skills/hindsight-docs references after docs changes - Bump test_llm_provider_memory_operations timeout to 600s for slower providers like Bedrock via LiteLLM * test: skip bedrock lite models in memory operations test Nova Lite has a 10K output token limit which is too low for fact extraction (requires 64K). The api_methods test (completion, tools, structured output) already validates the provider works correctly. * test: use Nova Pro for bedrock CI tests to cover full memory pipeline Nova Lite only supports 10K output tokens, too low for fact extraction. Switch to Nova Pro which supports the full 64K output needed for retain/reflect operations. This ensures bedrock is tested on all Hindsight functionalities, not just basic API methods. * test: switch bedrock CI to Nova 2 Lite (supports 64K output tokens) Nova v1 models (Pro, Lite) have a 10K output token limit which is too low for fact extraction. Nova 2 Lite supports 64K+ output tokens, enabling full memory pipeline testing (retain + reflect). |
||
|
Nicolò Boschi
|
69dec8ec34 |
feat: add otel traceability (#330)
* feat: add comprehensive OpenTelemetry tracing - Add tool execution spans for reflect operations - Add tool call information (names, params) to spans - Change verification scope from 'test' to 'verification' - Add hindsight.reflect_generation span for done() processing - Implement no-op tracer for improved code readability - Update documentation for OTEL configuration - Resolve merge conflicts from rebase * fix: properly serialize Pydantic models in span recording - Add _serialize_for_span() helper to handle Pydantic models - Update all providers to use the helper function - Fixes test failures with 'Object of type X is not JSON serializable' * feat: add Grafana LGTM stack for unified local observability Add Grafana LGTM (Loki, Grafana, Tempo, Mimir) as the recommended local development observability stack. This provides traces, metrics, and logs in a single Docker container instead of separate tools. Changes: - Add scripts/dev/grafana/ with docker-compose and README - Add scripts/dev/start-grafana.sh startup script - Update .env.example to reference Grafana LGTM - Update configuration docs to emphasize Grafana LGTM as primary option - Reorder OTLP backend list to show Grafana LGTM first Benefits: - Single container vs multiple separate tools (Jaeger, SigNoz, etc.) - ~515MB image with full observability stack - Compatible with existing OTLP configuration - Simpler local development setup * chore: remove SigNoz scripts and references Remove SigNoz observability stack in favor of Grafana LGTM as the sole recommended local development tracing solution. Changes: - Delete scripts/dev/signoz/ directory and all SigNoz configurations - Delete scripts/dev/start-signoz.sh startup script - Remove SigNoz references from .env.example - Remove SigNoz from OTLP backends list in configuration docs Grafana LGTM provides the same capabilities (traces, metrics, logs) in a simpler single-container setup. * feat: add consolidation span hierarchy for tracing Add parent-child span structure for consolidation operations: - hindsight.consolidation: Parent span for each memory being processed - hindsight.consolidation_recall: Child span for finding related observations - LLM call span: Automatically created by LLM provider (scope="consolidation") This enables detailed timing breakdown in Grafana Tempo: - Total consolidation time per memory - Time spent in recall - Time spent in LLM call - Time spent executing actions (create/update) All consolidation tests pass (31/31). * feat: add Prometheus metrics and GenAI dashboard to Grafana stack Add comprehensive metrics and dashboarding to the Grafana LGTM stack: Metrics Collection: - Configure Prometheus to scrape Hindsight API /metrics endpoint - Scrape interval: 10 seconds - Targets hindsight-api on host.docker.internal:8888 GenAI Dashboard: - Pre-configured dashboard with 6 panels: - LLM call rate (by provider/model) - LLM call duration (p50/p95 by scope) - Token usage - input tokens/sec by scope - Token usage - output tokens/sec by scope - Operations rate (retain/recall/reflect/consolidation) - Operation duration p95 by operation type Configuration: - Mount prometheus.yml for metrics scraping - Mount dashboards directory for auto-provisioning - Add host.docker.internal mapping for container->host access - Dashboard provisioning with auto-reload every 10s Documentation: - Updated README with metrics viewing instructions - Added PromQL query examples - Documented dashboard access and navigation This provides full observability: traces (Tempo) + metrics (Prometheus/Mimir) + dashboards (Grafana) * refactor: merge Grafana setup into existing monitoring stack Consolidate the separate scripts/dev/grafana/ setup into the existing scripts/dev/monitoring/ stack, using Grafana LGTM (Loki, Grafana, Tempo, Mimir). Changes: - Remove separate scripts/dev/grafana/ directory and start-grafana.sh - Rewrite scripts/dev/monitoring/start.sh to use Docker + Grafana LGTM (was: download native Prometheus/Grafana binaries) - Add docker-compose.yaml for Grafana LGTM container - Add prometheus.yml for scraping Hindsight API metrics - Mount existing dashboards from monitoring/grafana/dashboards/ - Add comprehensive README.md Benefits: - Single unified monitoring command: ./scripts/dev/start-monitoring.sh - Uses existing dashboard files (hindsight-operations, hindsight-llm, hindsight-api-service) - Simpler setup: Docker-based vs downloading/running native binaries - Full observability: traces + metrics + logs + dashboards in one container - Standard ports: Grafana on 3000, OTLP on 4317/4318 Architecture: - Grafana LGTM container (~515MB) provides all components - Dashboards auto-provisioned from monitoring/grafana/dashboards/ - Prometheus scrapes host.docker.internal:8888/metrics - Shared hindsight-network for future service-to-service tracing * fix: run monitoring stack in foreground for easy Ctrl+C stop Change docker-compose from detached (-d) to foreground mode. Users can now stop the stack with Ctrl+C instead of needing to run docker-compose down separately. * fix: remove invalid home dashboard path and obsolete version field - Remove GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH environment variable (was pointing to wrong path causing 'Failed to load home dashboard' error) - Remove obsolete 'version' field from docker-compose.yaml (docker-compose v2+ doesn't require version field) * fix: load Hindsight dashboards in Grafana LGTM Mount Hindsight dashboard JSON files and custom provisioning config to make dashboards visible in Grafana. Changes: - Mount hindsight-operations.json, hindsight-llm.json, hindsight-api-service.json to /otel-lgtm/ - Create grafana-dashboards.yaml with all dashboard providers (default + Hindsight) - Mount custom provisioning config to override LGTM default All 3 Hindsight dashboards now appear in Grafana UI with metrics from Prometheus scraping the Hindsight API /metrics endpoint. * fix: configure Prometheus to scrape Hindsight API metrics Update prometheus.yml to include both OTLP receiver config (from LGTM) and scrape_configs for pulling metrics from Hindsight API. Changes: - Mount prometheus.yml to /otel-lgtm/prometheus.yaml (where LGTM reads it) - Add scrape_configs section to pull from host.docker.internal:8888/metrics - Keep OTLP receiver configuration for trace metrics - Set scrape_interval to 5s Verified: Prometheus now successfully scrapes hindsight_llm_calls_total and other Hindsight metrics. Dashboards now show live data! * feat: add comprehensive tracing for recall and improve reflect/mental_model_refresh spans - Add recall operation tracing with parent-child span hierarchy - Parent: hindsight.recall with attributes (bank_id, query, fact_types, etc.) - Children: recall_embedding, recall_retrieval, recall_fusion, recall_rerank - Fixed context propagation using start_as_current_span() - Improve reflect tracing spans - Remove reflect_generation spans, use reflect instead - Change done() tool processing to hindsight.reflect_tool_call - Fix mental_model_refresh span nesting - Add _skip_span parameter to reflect_async to avoid duplicate hindsight.reflect spans - Mental model refresh now has clean span hierarchy without nested reflect parent - Add comprehensive tracing verification tests - Test span hierarchy and attributes for all operations - Verify parent-child relationships - 5 passing tests covering recall, reflect, consolidation, and mental_model_refresh * refactor: remove redundant is_tracing_enabled() checks - Remove all is_tracing_enabled() conditional checks before tracing calls - NoOpTracer/NoOpSpan handle disabled tracing automatically - Simplify code by always calling tracer methods directly - Fix NoOpTracer.start_as_current_span() to yield NoOpSpan instead of None Changes: - memory_engine.py: Remove 5 is_tracing_enabled checks in recall spans - agent.py: Remove 2 is_tracing_enabled checks in reflect tool spans - tracing.py: Fix NoOpTracer context manager to yield proper NoOpSpan This eliminates ~50 lines of redundant conditional code while maintaining identical behavior. * docs: simplify distributed tracing section in monitoring.md - Make tracing documentation more concise - Focus on span hierarchy and attributes - Remove verbose troubleshooting and performance sections - Keep configuration.md for env vars only |
||
|
Nicolò Boschi
|
522b71aab8 |
doc: mental models (#199)
* doc: mental models * doc: mental models |
||
|
Nicolò Boschi
|
eb2702bcba |
misc: performance improvements (#140)
* misc: performance improvements * misc: performance improvements * misc: performance improvements |