The API and worker run /health and all task work on a single event loop, and
/health acquires a DB connection. A failing liveness probe therefore has two
very different causes that today are indistinguishable: the event loop is
blocked by synchronous work (a restart helps), or the connection pool is
exhausted and /health can't get a connection while the loop is idle (a restart
just thrashes). Add two always-on, cheap signals so the failure is
self-diagnosing instead of an opaque restart.
LoopWatchdog (hindsight_api/loop_watchdog.py): runs in a separate OS thread —
deliberately, since a coroutine-based monitor would be frozen by the very stall
it's watching — pings the loop, and on a stall past a threshold logs the loop
thread's stack (naming the blocking frame) and emits
hindsight.event_loop.stalls / stall_duration. Works with uvloop. Wired into the
worker CLI and the API lifespan; enabled by default.
DB pool acquire instrumentation (engine/db/pool_instrumentation.py): tracks
callers currently queued for a connection (hindsight.db.pool.waiting gauge, the
signal that actually distinguishes exhaustion from a busy-but-healthy pool),
records an acquire-wait histogram, and logs a warning with pool stats when an
acquire waits too long. Wired into both the PostgreSQL and Oracle backends.
health_check() now reports db_acquire_ms and pool utilization in its payload.
Static config: HINDSIGHT_API_LOOP_WATCHDOG_ENABLED / _STALL_THRESHOLD_MS /
_POLL_INTERVAL_MS, HINDSIGHT_API_DB_ACQUIRE_WARN_THRESHOLD_MS.
Tests: test_loop_watchdog.py (detects on-loop blocks, ignores off-loop work,
quiet when responsive) and test_pool_instrumentation.py (waiter counting through
success/mid-acquire/failure, slow-acquire logging).
Per-(bank, fact_type) partial vector indexes are created only at fresh-bank
creation. A bank populated outside that path (logical restore, cross-version
upgrade, extension switch) never gets them, so its recall silently falls back
to the global index + post-filter — slower and under-returning (~0.63-0.72
recall@10 measured by the reporter).
Two fixes:
- import-bank: create the per-bank indexes explicitly after restoring the
banks row. The prior get_or_create_bank_profile call was a no-op here (the
row already exists, so it takes the SELECT branch), leaving every restored
bank uncovered.
- hindsight-admin repair-bank (--bank ID | --all): re-runnable operator escape
hatch for the out-of-app routes (raw pg_dump restore, extension switch) that
a one-time migration can't cover (a restore carries alembic_version at head,
so the migration is already stamped). Detects missing OR invalid coverage
(INVALID leftovers / drifted access method count as missing, unlike a
name-only check) and rebuilds with CREATE INDEX CONCURRENTLY off any txn.
Idempotent; concurrency handled by idempotency, not advisory locks.
Deliberately excludes the boot/periodic background reconcile and retain-path
self-heal: a bank restored and only ever read stays degraded until an operator
runs repair-bank. That background layer can be a follow-up.
* feat(llm): opt-in 4xx request-dump for diagnosing rejected calls (all providers)
Generalizes the Gemini-only diagnostic from #2475 into a provider-agnostic
helper (engine/providers/llm_debug.py) wired into every remote LLM provider:
Gemini/Vertex, OpenAI-compatible (+ Fireworks/Nous subclasses), Anthropic,
LiteLLM (+ router subclass), and Codex — on both call() and call_with_tools().
Gated by HINDSIGHT_API_LLM_DEBUG_DUMP_4XX (off by default). On any 4xx it logs
[LLM_4XX_DUMP] with the serialized request config (message bodies stripped) and
per-message role/size + a length-capped preview. Self-gates on the env flag and
a 4xx status, extracts the status across SDK error shapes (status_code / code /
response.status_code), and never raises.
* style: ruff format single-line dump_request_on_4xx calls
* refactor(llm): source 4xx-dump flag from HindsightConfig, not raw env
Adds llm_debug_dump_4xx as a static (server-level) config field; the helper
reads get_config().llm_debug_dump_4xx instead of os.getenv directly. Documents
the flag in configuration.md and .env.example (+ bundled embed copy). Replaces
the tuple return in the message-preview helper with a dataclass per project
standards.
* feat(mcp): let create_mental_model configure tags_match (#2808)
A tagged mental model with no explicit tags_match in its trigger JSON
refreshes under all_strict (a memory must carry every one of the model's
tags), while the staleness check and every recall/reflect path default to
any. Broadly-tagged models reading narrowly-tagged memories therefore get
marked stale and then refresh to empty content.
The HTTP API, generated SDK clients, and Control Plane UI already let users
set trigger.tags_match; the MCP create_mental_model tool did not. Add a
tags_match argument (validated against TagsMatch) to both MCP variants. It
is only written into the trigger when explicitly passed, so the resolved
all_strict default is preserved for existing callers.
Document the all_strict footgun and the tags_match override in the MCP and
mental-models API docs (regen skills/hindsight-docs mirror).
* fix(ts-client): expose tags_match/tag_groups on createMentalModel
The ergonomic TypeScript wrapper's createMentalModel accepted only
{ refreshAfterConsolidation } in its trigger option and dropped every other
trigger field, so a wrapper user could not set tags_match — the exact knob
needed to avoid the empty-refresh footgun in #2808. The low-level generated
sdk already accepts the full MentalModelTriggerInput; thread tagsMatch and
tagGroups through, mirroring how recall/reflect already expose them.
The Python client needs no change: its wrapper takes a pass-through
trigger dict and the generated MentalModelTriggerInput already validates
tags_match.
* test(ts-client): cover createMentalModel trigger mapping
Mock the generated sdk layer (no server needed) and assert the ergonomic
camelCase trigger options map onto the snake_case body: tagsMatch ->
tags_match, tagGroups -> tag_groups, refreshAfterConsolidation still maps,
and omitting trigger sends none (preserving the all_strict default). Locks
in the #2808 wrapper fix.
* docs(mental-models): add tags_match code snippet
Replace the static JSON block in the tags_match override section with a
live CodeSnippet pulled from the Python example, showing how to create a
model with trigger.tags_match="any" so a broadly-tagged model reads
narrowly-tagged memories on refresh (#2808).
* feat(cli): add --tags-match to mental-model create + all-language docs
The Rust CLI's `mental-model create` was the last creation surface with no
way to set tags_match, so a tagged model created via the CLI hit the same
empty-refresh footgun (#2808). Add a `--tags-match` flag (any/all/any_strict/
all_strict/exact) that is only sent when passed, preserving the server's
all_strict default; invalid values are rejected before the request.
Expand the mental-models docs "tags_match override" example from a single
Python snippet to a full Tabs block (Python / Node.js / CLI / Go), each
pulled from the runnable example files, and regen the skills mirror.
Add HINDSIGHT_API_LLM_STRICT_SCHEMA_{RETAIN,REFLECT,CONSOLIDATION}, each resolved per-operation env -> global env -> default (mirroring the per-operation temperature knobs). All five structured-output call sites thread their operation's resolved flag.
Also fixes a latent resolution bug in LLMConfig.call: 'strict_schema or get_config().llm_strict_schema' made a per-call False indistinguishable from unset, silently ignoring any scope opting out while the global flag was on. The arg is now bool|None: None inherits the global flag, explicit True/False wins in both directions.
Supersedes #2669.
* feat(audit): make audit_log_enabled overridable per bank
Auditing was all-or-nothing per deployment. This makes the existing
audit_log_enabled switch hierarchical (env -> tenant -> bank) so a bank
can opt in while the server default is off, or opt out while it is on,
rather than introducing a second near-identically-named field.
Making the flag per-bank forces three call sites to change:
- AuditLogger: the enabled check can no longer be a synchronous
pre-filter, since a bank may enable auditing the global value has off.
Split into action_allowed() (bank-independent allowlist, still a cheap
sync pre-filter) and should_log() (awaits the per-bank resolution).
Resolution failure falls back to the deployment default rather than
failing closed, so a transient DB blip cannot silently create an audit
gap for a bank that is meant to be audited.
- Retention sweep: previously gated on audit_log_enabled, which is now
per-bank while the sweep is a global cross-tenant job with no bank in
scope. A bank opting in under a default-off deployment would have had
its rows accumulate forever. Retention now keys off the (still
server-level) retention window alone.
- _audit_memory_defense: was sync and reached log_fire_and_forget
directly, bypassing the per-bank decision entirely. Made async so the
memory_defense action honours the bank's setting like every other path.
The actions allowlist and retention window stay server-level: both are
global sweeps with no bank scope. The /version audit_log flag keeps
reporting the deployment default and now says so.
Adds the Audit Logging toggle to the bank Configuration tab.
The hindsight-docs skill regen also picks up pre-existing drift from
#2694 (retain.md), which the pre-commit generator syncs unconditionally.
* fix(control-plane): make the audit toggle tri-state
A Switch cannot express "inherit the server default". It rendered the
resolved value, so a bank inheriting `true` looked identical to one
explicitly set to `true`, and touching it always wrote an explicit
boolean with no way back to inherit.
Replaced with a Select: Server Default / Enabled / Disabled. The slice
now reads the bank's `overrides` rather than the resolved config, since
the resolved value cannot distinguish inherited from explicitly-set.
Choosing "Server Default" sends null, the tombstone the config resolver
already treats as "clear this override".
The option label shows which way the server default currently points,
read from the existing /version features flag.
Uses INHERIT_SENTINEL rather than "" for the inherit option: Radix
rejects an empty SelectItem value at runtime.
* chore(clients): regenerate for audit_log description change
The audit_log field description in openapi.json changed; regenerate the
Go/Python/TypeScript clients that embed it (they were skipped earlier
because the generator needs Docker). Verify-generated-files was failing
on the drift.
* fix(audit): resolve gating config internally, bypassing permission filter
_resolve_bank_audit_enabled used get_bank_config, the API-facing resolver
that runs the tenant permission filter (get_allowed_config_fields). A
deployment that makes audit_log_enabled read-only for a user — exactly
the intended way to lock the field via an extension — would have that
field stripped from the resolved config, so gating silently reverted to
the deployment default and ignored the bank's stored override.
Switch to resolve_full_config (the internal, unfiltered resolver every
other internal config consumer uses). Gating is a system decision and
must see the bank's true value regardless of who is asking.
Adds a regression test with a restrictive tenant extension: the API read
strips the field, but gating still audits the opted-in bank.
Also: document the fail-open opt-out edge in should_log's comment, and
refresh a stale "static, server-level switch" comment in the memory
defense test.
Follow-up to #2708, which bounded terminal `async_operations` history. Two
issues with what landed:
1. The cleanup worker did not use a cross-tenant routine. It opened a connection
and a prune transaction against *every* tenant schema on every cleanup cycle,
paying the full per-tenant cost even when nothing was prunable — the query
storm the server-side maintenance routines exist to avoid.
2. It shipped as a breaking change, silently switching deployments from
unbounded operation history to a 30-day TTL on upgrade.
Adds `schemas_with_expired_operations(p_days int) RETURNS SETOF text` — the
`async_operations` counterpart to `schemas_with_expired_rows`. One round-trip
returns just the schemas holding expired terminal rows; the worker then acquires
a connection and prunes only there. It needs its own routine rather than reusing
`schemas_with_expired_rows` because eligibility here isn't "row older than N
days" — pending and processing rows are never prunable, so the status filter has
to be part of the predicate.
Install policy follows b6d2f8a4c1e7 (#2638/#2824): the routine is database-global
(it enumerates pg_class across every schema), so exactly one copy is installed —
into the schema this deployment is configured to use, which is the one the worker
calls via fq_routine. Gating on the literal "public" instead of the configured
schema is what left non-public deployments without the sibling routines (#2638);
installing into every schema would leave a dead duplicate per tenant.
Exactly one run satisfies that predicate, so concurrent per-schema runs never
issue competing CREATE OR REPLACE against the same pg_proc row and cannot hit
`tuple concurrently updated`. No cross-process coordination, and in particular no
advisory lock, which is unusable behind connection poolers and managed PG
(#2817). Runs targeting any other schema drop the routine there instead.
The worker calls the routine through schema.fq_routine() (added in #2824) rather
than a hardcoded public. qualifier — duplicating that qualifier across callers is
precisely how #2638 recurs.
Vanishing schemas are skipped rather than fatal (c7e9f1a3b5d2), and an absent
routine degrades cost, not correctness — Oracle and un-migrated PostgreSQL fall
back to the previous full sweep with a warning.
DEFAULT_OPERATION_RETENTION_DAYS 30 -> 0. Operation history is a user-visible
audit trail, so bounding it is an opt-in policy decision rather than something an
upgrade applies silently. Set HINDSIGHT_API_OPERATION_RETENTION_DAYS to a
positive number of days to enable pruning. Docs, .env.example and the bundled
embed template updated to match.
- test_schemas_with_expired_operations — drives the real routine against pg0 in a
throwaway schema: old pending/processing rows alone don't make a schema
eligible, a terminal row does, a too-old cutoff doesn't, p_days <= 0 is empty.
- test_expired_operations_routine_installs_in_the_configured_schema —
parametrized over base / default public / non-public single-tenant; guards
against reintroducing the #2638 literal gate or an advisory lock.
- test_expired_operations_tenant_runs_install_nothing — tenant runs emit no
CREATE and drop any copy in their own schema.
- test_discovery_targets_the_configured_non_public_schema — the worker calls the
copy in its configured schema, not a hardcoded public one.
- TestWorkerOperationCleanupSchemaNarrowing — only reported schemas are pruned,
nothing expired means no pruning, unclaimed schemas are skipped, a missing
routine falls back to the full sweep, Oracle never calls the routine.
* feat(api): attribute remote reranker calls by bank
* fix(api): omit empty reranker bank attribution
* fix(reflect): bind bank attribution for tool calls
Entity resolution is fuzzy name matching (SequenceMatcher) reinforced by
co-occurrence and temporal proximity — there is no nickname/alias logic in
the resolver. Dissimilar names like 'Bob' and 'Robert Chen' do not unify on
the name alone, so the 'nickname resolution' example was inaccurate. Verified
against hindsight-api-slim/hindsight_api/engine/retain/entity_resolver.py.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds HINDSIGHT_API_DB_MAX_PARALLEL_WORKERS_PER_GATHER — when set, every
pool connection of the process runs SET max_parallel_workers_per_gather
at init time (alongside the existing statement_timeout / ANN tuning
session setup). Unset (the default) leaves the server setting untouched,
so existing deployments see no behavior change.
Motivation: in multi-tenant deployments where background workers share a
database with latency-sensitive foreground traffic, bulk maintenance
queries (consolidation, graph upkeep) can fan out across parallel
workers and occupy several cores each. Parallelism buys latency — which
background work doesn't need — at the cost of concurrent CPU footprint,
which a shared primary does care about. Setting the cap to 0 on worker
processes makes those queries run serially: measured on a representative
multi-million-row aggregate, serial execution cost ~29% more wall-clock
but used 67% fewer concurrent cores (and less total CPU, since parallel
coordination isn't free).
0 is a meaningful value (disable parallelism), so the env parse
distinguishes unset (None, no opinion) from 0 via a new
_parse_optional_non_negative_int helper; negative or non-integer values
fail fast at startup.
The field is static (process-level infrastructure tuning), deliberately
not in _CONFIGURABLE_FIELDS.
Roll a CachedContent forward through the reflect tool loop so each auto turn reuses the entire prior conversation at the cached-input rate and sends only its new tool results. Measured on gemini-2.5-flash-lite: ~29% cached on short loops, ~74-81% on deep loops (deepest turns ~99%), vs ~9% for the old static prefix and 0% for implicit caching.
Cache creates overlap tool execution to hide their latency, and the ephemeral per-reflect caches are torn down detached so the response path never waits on deletes. New HINDSIGHT_API_REFLECT_PROMPT_CACHE_ENABLED flag (default true) disables it independently of the global prompt cache.
Mental-model content in delta-refresh mode can grow past an embedding
model's fixed input-token limit (e.g. Bedrock Titan V2's hard 8192 cap),
after which every refresh fails permanently with ContextWindowExceededError
and no recovery path.
Add an opt-in `HINDSIGHT_API_EMBEDDINGS_LITELLM_SDK_MAX_INPUT_TOKENS` cap.
When set, `LiteLLMSDKEmbeddings.encode()` truncates each input to that many
cl100k_base tokens before calling litellm.embedding(), mirroring the existing
reranker `max_tokens_per_doc` pattern. Truncation emits a log.warning naming
the model and largest original token count so it isn't silent.
Off by default (no behavior change / data loss for large-context models);
Titan users set it to the model's real limit with a little headroom.
Add opt-in HINDSIGHT_API_FAIL_ON_EXTRACTION_ERRORS (default False, preserves behavior). When enabled and a retain accumulated extraction errors (extraction_errors_count > 0), the operation is marked failed instead of silently completed. Self-contained: config flag + _mark_operation_completed decision + docs + .env.example. Deferred follow-ups (status API field, completed_with_errors status, webhook field, metric) noted in the PR.
Own change verified (config + status tests pass, verify-generated-files green, ruff/tsc clean). Remaining CI reds are unrelated: Core LLM + pg0 test-api flakes, a transient ts-client-oracle, and test-embed-windows (the #2676 regression already fixed on main by #2723; #2721 does not touch the embed daemon).
Fixes#2700
Add configurable TTL (default 30 days, 0=keep-forever) for terminal async_operations rows. Expired completed/failed/cancelled rows are pruned in bounded batches (1000/cycle) by a background task that never touches pending/processing work. Batch children are protected until their parent is pruned; cancelled-child cleanup atomically cancels a pending parent first. PG uses FOR UPDATE SKIP LOCKED; both PG and Oracle re-check eligibility under the row lock before deleting. Includes indexes, docs, and regenerated SDKs.
184 retention/worker/operation-status tests pass locally. All CI green.
Fixes#2705
* fix(ollama): make native num_ctx opt-in
* docs(ollama): add HINDSIGHT_API_LLM_OLLAMA_NUM_CTX to .env.example
* fix(config): keep Ollama num_ctx optional for direct config construction
---------
Co-authored-by: r266-tech <r266-tech@users.noreply.github.com>
Co-authored-by: Ben <ben.bartholomew@vectorize.io>
Verifying the Oracle guide end-to-end surfaced three setup steps that weren't
documented and that block a first-time deployment:
- HINDSIGHT_API_DATABASE_SCHEMA must be set to the Oracle schema user. The
default `public` is a PostgreSQL notion and makes migrations fail with
ORA-01435. Added it to the configure step (with a warning), the quick start,
the config reference table, and troubleshooting.
- Migrations must run with the same embedding dimension as the serving model,
or retain fails with ORA-51803. Added a warning to the migrate step and a
troubleshooting row (including the --embedding-dimension resize path).
- The dev quick-start container can report a provisioning error on a cold
start's first run; noted that re-running the idempotent script succeeds.
Mirrored into versioned_docs/version-0.8 and regenerated the docs skill.
Claude-Session: https://claude.ai/code/session_01PginSDrapXsoDd6gN5Pszo
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: add Oracle Database setup guide
Hindsight supports Oracle Database 23ai as a storage backend, but the docs
only mentioned it in passing — a one-paragraph note on the Storage page and a
couple of Configuration reference rows, with no `oracle+oracledb://` example
anywhere. This adds a dedicated Oracle Database page under Hosting.
The guide covers requirements (Oracle 23ai, the ASSM-tablespace requirement
for VECTOR columns, Oracle Text / CTXAPP), installing the python-oracledb
driver, a local quick start via scripts/dev/start-oracle.sh, production
provisioning SQL + connection URL + env vars + migrations, a config reference,
the differences from PostgreSQL, and troubleshooting. Content is grounded in
the CI Oracle job, the dev script, and the backend code.
Registered in the sidebar and cross-linked from Storage and Configuration.
Regenerated the docs agent-skill and mirrored the change into
versioned_docs/version-0.8 so it ships on the currently-served version.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PginSDrapXsoDd6gN5Pszo
* docs(oracle): correct managed-service note, add connection caveat
The connection layer builds the Oracle DSN from the URL as a plain
host:port/service_name descriptor — wallet-based mTLS, TLS/TCPS, and TNS
aliases / full connect descriptors are not wired up. The previous "Least
privilege" note implied Oracle Autonomous Database works via an
ADMIN-provisioned user, which is misleading since ADB defaults to wallet/mTLS.
- Reworded the managed-service note to drop the specific ADB claim while
keeping the accurate requirement (ASSM tablespace + CTXAPP).
- Added an "Easy Connect only" warning documenting that wallet/mTLS/TLS and
TNS descriptors are unsupported, and that transport encryption must be
handled at the network layer.
Applied to the current and version-0.8 copies; regenerated the docs skill.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PginSDrapXsoDd6gN5Pszo
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The list and get memory-unit paths selected tags and timing fields
but skipped the memory_units metadata column, so metadata retained
on facts was invisible outside recall.
Select and serialize metadata for live and invalidated memory units,
add a curation regression test for both paths, and update docs plus
OpenAPI examples.
Hindsight already honors CODEX_HOME (openai-codex LLM + embeddings), but it
was only mentioned in the 0.8.3 changelog. Long-running services sharing
~/.codex/auth.json with another Codex process can get their refresh token
rotated out, leaving /reflect broken while /health stays green.
Add a 'Isolating Codex auth for long-running services' section to the Models
docs and a pointer next to the openai-codex snippet in configuration.
Refs #2476
Lazy reranker init was the only mode in which CrossEncoderReranker.ensure_initialized()
could double-load the model: its check-then-act over the `await` is a real race, but
in the default (eager) path init_cross_encoder() runs at startup — single-threaded,
before any request — so the per-request guard always short-circuits and the window
never opens (see PR #2445 discussion).
Rather than guard the lazy path with a lock, drop the flag entirely. The cross-encoder
is now always initialized eagerly at startup, which removes the race by construction and
the first-recall latency cliff. The only thing the flag bought was skipping an ~80MB
model load for retain-only deployments — not worth the extra config surface and the
concurrency footgun.
- Remove ENV_LAZY_RERANKER, the config field, and from_env() wiring
- Remove the lazy_reranker constructor param; always append init_cross_encoder()
- Drop the now-dead kwarg/env from tests; rename the ensure_initialized timeout tests
- Update docs + regenerate the docs skill mirror
ensure_initialized() is kept as a cheap idempotent guard on the recall path.
* fix(llm): make per-operation temperature configurable (#2459)
Internal LLM calls used hardcoded temperatures (verification 0.0, fact
extraction 0.1, reflect thinking 0.9, consolidation 0.0, bank mission 0.3).
Models like Azure gpt-5.5 reject any explicit temperature other than their
default, breaking retain/reflect/verification.
Expose each as an env knob with a global override:
- HINDSIGHT_API_LLM_TEMPERATURE (global) + _VERIFICATION/_RETAIN/_REFLECT/
_CONSOLIDATION/_MISSION (per-operation override).
- Resolution: per-operation env -> global env -> historical default.
- A value of none/default/off/empty omits the temperature parameter entirely,
so HINDSIGHT_API_LLM_TEMPERATURE=none fixes gpt-5.5 in one variable.
call() already drops temperature=None across providers, so the None config
value naturally omits the param. Defaults preserve prior behavior exactly
(fully backwards compatible). Server-level/static config.
* test(llm): verify per-operation temperature reaches the LLM call
MockLLM now records the temperature it receives, and a new pipeline test
drives the real engine: retain forwards 0.1 to fact extraction, the reflect
thinking path forwards 0.9, and HINDSIGHT_API_LLM_TEMPERATURE=none omits the
parameter (None) on a live call.
* test(llm): set llm_temperature_retain on the fact-extraction retry mock config
The retry tests build a MagicMock(spec=HindsightConfig); dataclass
annotation-only fields aren't in the spec, so the new llm_temperature_retain
field (now read at the extraction call site) must be set explicitly.
The per-operation LLM request settings were resolved into HindsightConfig but
never reached the provider that uses them, so configuring them was a silent
no-op:
- *_llm_timeout (retain/reflect/consolidation) and the global llm_timeout never
reached the provider impl; it fell back to HINDSIGHT_API_LLM_TIMEOUT/120s, so
HINDSIGHT_API_RETAIN_LLM_TIMEOUT=300 did nothing ("LiteLLM call exceeded
timeout=120.0s").
- reflect_llm_max_retries/initial_backoff/max_backoff and
consolidation_llm_initial_backoff/max_backoff were never consumed; reflect and
consolidation used the hardcoded call()/call_with_tools() defaults (10/5),
ignoring the documented "falls back to llm_max_retries" contract.
Fix: resolve each operation's effective request defaults (per-op override else
global) in MemoryEngine and carry them on the LLMProvider:
- timeout is threaded config -> LLMProvider -> create_llm_provider -> provider
impl for the providers that honour a configurable request timeout (LiteLLM,
LiteLLM Router, OpenAI-compatible, Nous). None preserves each provider's own
default, so Anthropic/Gemini keep their bespoke timeouts and the no-config
path is byte-identical.
- max_retries/initial_backoff/max_backoff become LLMProvider instance defaults
that call()/call_with_tools() use when the per-call arg is omitted. Explicit
per-call args (retain's resolved values, reflect's fast structured-extraction
path) still win; providers built without config (from_env, tests) keep the
10/5 method fallback.
The four operation scopes (default/retain/reflect/consolidation) and multi-LLM
chain members all share their operation's resolved values via a small
_LLMCallDefaults bundle.
max_concurrent is intentionally left as-is (process-global semaphores read from
env at startup, server-level only); the docs are clarified to call out that
distinction.
Also fixes a pre-existing breakage in test_llm_router_provider's __new__-based
helper (missing _default_headers after #2466) so the suite is green.
Tests: tests/test_llm_timeout_propagation.py covers provider-impl timeout
threading, the call() retry-policy fallback/override, and per-op
resolution/fallback in MemoryEngine.
Replace the recall result's single `score` with a `scores` object exposing the
scores from each pipeline stage, and replace the `min_score` request param with
`min_scores`, a per-stage filter that operates at two levels.
Response — each result carries `scores`:
- final : the value results are ranked by
- reranker : cross-encoder normalized relevance (null for passthrough rerankers)
- semantic : raw vector cosine similarity (null if not surfaced semantically)
- text : raw keyword/BM25 score (null if not surfaced by keyword search)
Per-arm semantic/text scores are aggregated across retrieval arms during RRF /
interleave fusion (ArmScores on MergedCandidate), since fusion otherwise keeps
only the first-seen arm's score per doc.
Request — `min_scores` floors (inclusive, AND-ed, opt-in; default no filtering):
- semantic / text : retrieval-level cutoffs pushed into the SQL arms, overriding
the global similarity / BM25 minimums for the request (prune before fusion)
- reranker / final: post-query filters on the scored results
There is deliberately no default threshold: the cross-encoder's absolute scores
are reliable for ordering but not calibrated across queries (a clearly-relevant
match can score ~0.001 on one query and ~1.0 on another), so a fixed cutoff would
silently drop good results.
Also surfaces proof_norm in the search trace and reworks the control-plane trace
view to render scores at full precision (no rounding) and show the per-stage
`scores` breakdown; relabels the trace's "CE" column to "reranker score".
Threaded through engine, HTTP, MCP (both recall tools), and the control-plane
proxy; OpenAPI spec, Python/TS/Go/Rust clients, and the docs-skill mirror
regenerated; docs updated.
Adds a third, independent way to refresh a mental model — on a cron schedule —
alongside the existing auto (refresh_after_consolidation) and manual paths,
driven by the background MaintenanceLoop ticker.
API/engine:
- trigger.refresh_cron (UTC 5-field cron, croniter-validated); mutually
exclusive with refresh_after_consolidation.
- PG-only discovery routine public.mental_models_with_cron() (migration
f4d1c2b3a5e6); cron due-ness evaluated in Python, refresh only when stale.
- HINDSIGHT_API_MENTAL_MODEL_REFRESH_TICK_SECONDS check cadence.
- One timing line logged per maintenance sweep.
Control plane:
- Single "Refresh trigger" choice (Manual / On new memories / On a schedule)
with per-option sub-labels; cron input shown only when scheduled.
- Live cron schedule preview (human-readable + next/upcoming runs, UTC+local).
- "Next refresh" shown next to "last refreshed" in list, dashboard, and dialog.
- Fixed an app-wide off-by-one in formatRelativeTime.
Regenerated OpenAPI + clients + bank-template schema; i18n across all locales.
Follow-up to #2384. That PR let an indexed multi-LLM member carry its own
Vertex AI project/region, but two parity gaps remained vs the primary provider:
- A `litellmrouter` member had no per-member router config, so it silently fell
back to the global `HINDSIGHT_API_LLM_LITELLMROUTER_CONFIG` — a chain could not
fail over between differently-routed LiteLLM routers (same bug class #2384 fixed
for Vertex).
- A `vertexai` member used only the global service-account key, so cross-project
failover with distinct credentials was impossible (project/region alone weren't
enough).
Adds `litellmrouter_config` and `vertexai_service_account_key` to
`LLMMemberConfig`, reads `{prefix}LLM_{n}_LITELLMROUTER_CONFIG` /
`_VERTEXAI_SERVICE_ACCOUNT_KEY` in `_parse_llm_members`, threads both through
`_member_to_llm`, and lets `LLMProvider.__init__` take a per-instance Vertex SA
key (explicit wins, else global fallback). Single-LLM/global behavior unchanged.
Tests: parse (incl. per-op prefix + invalid-JSON), and build-path proving the
member's own values reach `litellm.Router` and the Vertex SDK client. Docs table
updated with the new per-member keys.
* feat(llm): multi-LLM failover & round-robin via indexed config
Configure extra LLMs by index (HINDSIGHT_API_LLM_<n>_*) alongside the
unindexed primary, then route across them with HINDSIGHT_API_LLM_STRATEGY
(JSON): {"mode":"failover"} or {"mode":"round-robin"} with optional
per-member "weights" for unbalanced rotation. Each operation can override
the global chain with a RETAIN_/REFLECT_/CONSOLIDATION_ prefix.
A general, provider-agnostic alternative to the LiteLLM Router and a more
extensible replacement for the single-secondary failover approach.
- config.py: LLMMemberConfig/LLMStrategyConfig dataclasses, indexed-member
+ strategy parsing, new HindsightConfig fields (credential, server-level).
- engine/multi_llm.py: MultiLLMProvider mirrors the LLMProvider surface so it
drops into with_config()/ConfiguredLLMProvider and _provider_impl passthrough;
smooth weighted round-robin; failover passes through OutputTooLongError and
cancellation; strict-primary/soft-secondary verify_connection.
- memory_engine.py: _build_llm wraps each of the 4 LLM slots; no-config path
returns the plain LLMProvider unchanged.
Batch retain runs on the primary member only (documented).
* docs: regenerate hindsight-docs skill reference for multi-LLM config
Atlas Cloud (https://www.atlascloud.ai) exposes an OpenAI-compatible
chat/completions endpoint, so it slots into the existing
OpenAICompatibleLLM path exactly like deepseek / zai / opencode-go.
Set `HINDSIGHT_API_LLM_PROVIDER=atlas` to route fact extraction,
reflection and consolidation through Atlas Cloud. The base URL defaults
to https://api.atlascloud.ai/v1 and the default model is
deepseek-ai/deepseek-v4-pro (a reasoning model — give it enough
max_tokens, >= 512).
Changes:
- engine/llm_wrapper.py: register "atlas" in create_llm_provider(),
LLMProvider.valid_providers, and the default base_url map
- engine/providers/openai_compatible_llm.py: register "atlas" in
valid_providers, default base_url, and the API-key-required check
- config.py: PROVIDER_DEFAULT_MODELS["atlas"] = deepseek-ai/deepseek-v4-pro
- hindsight-embed control center: add Atlas Cloud to the provider wizard
- docs: add Atlas Cloud to llmProviders.json (drives the providers grid
and table) and a config example in developer/models.mdx
- README + .env.example: document the new provider
Verified end-to-end: instantiated the atlas provider through Hindsight's
own create_llm_provider() and made a live call() to
deepseek-ai/deepseek-v4-pro (HTTP 200, valid content + token usage).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Recalling `observation` alongside `world`/`experience` can return the same
information twice — once as a raw fact and once folded into an observation
consolidated from it. The opt-in `prefer_observations` flag drops any raw fact
that a returned observation lists in its `source_memory_ids`, so the observation
supersedes it. Dedup is by provenance (exact id membership), not semantics, and
runs before recall truncation so freed slots backfill — keeping the result count
at the requested budget.
Disabled by default (opt-in). Internal callers — notably consolidation, which
needs the raw facts it folds into observations — leave it off.
Exposed on the full client surface: the maintained Python (`recall`/`arecall`)
and TypeScript (`recall`) wrappers, the Rust CLI (`--prefer-observations`), the
regenerated OpenAPI + low-level Python/TS/Go/Rust SDKs, the control-plane proxy +
types, and the generated docs skill. Includes docs and deterministic
provenance-based tests (engine + both wrappers).
* fix(recall): allow exact filtering of untagged/global observations (#2295)
An empty tag set with tags_match="exact" now selects only untagged
(global-scope) observations — the scope that observation_scopes="shared"
consolidation writes to. Previously empty/absent tags meant "no filter"
in every mode, so there was no way to recall only global observations
when mixing shared and tagged scopes.
- tags.py: in exact mode, empty/absent tags emit an untagged-only clause
(tags IS NULL OR tags = '{}') with no bind param, across the flat SQL
builders, Python post-filter, and compound tag-group leaves. All other
modes keep treating empty/absent tags as "no filtering".
- link_expansion_retrieval.py: always run filter_results_by_tags so the
exact-empty/global scope is applied (it's a no-op otherwise).
- http.py + regenerated clients/docs: document the exact-empty scope.
- Tests: SQL builders (flat + compound, param-offset preserved), Python
post-filter, and a recall API test asserting only untagged memories
return for tags=[] + tags_match="exact".
* chore(docs-skill): regenerate references for untagged exact-scope recall
Regenerated skills/hindsight-docs/references via generate-docs-skill.sh so the
docs-skill mirror matches the updated recall/observations docs (and the canonical
configuration table). Unblocks verify-generated-files.
* feat(recall): configurable recency decay function (linear/exponential/none)
The recency boost in apply_combined_scoring hard-coded a linear decay over an
arbitrary 365-day window. Make the age->freshness curve configurable:
- linear (default, unchanged): straight decay to a 0.1 floor over a window now
exposed as HINDSIGHT_API_RECENCY_DECAY_LINEAR_WINDOW_DAYS (365).
- exponential: 0.5 ** (days_ago / halflife); half-life is the age at which the
signal is neutral. Smooth, no hard cutoff.
HINDSIGHT_API_RECENCY_DECAY_HALFLIFE_DAYS (90).
- none: disables the recency boost entirely.
Selected via HINDSIGHT_API_RECENCY_DECAY_FUNCTION. Static config (read via
get_config() at the recall call site, mirroring recall_strategy_boosts).
* fix(test): accept new recency-decay kwargs in scoring stub; regen docs skill
The runtime default for the anthropic provider is the self-updating alias
`claude-haiku-4-5` (config.py PROVIDER_DEFAULT_MODELS, enforced by
tests/test_provider_default_models.py), but the Models docs advertised the
date-pinned snapshot `claude-haiku-4-5-20251001`. A pinned snapshot and a
self-updating alias differ for pricing/retirement, and the page contradicted
hermes.md (which already says `claude-haiku-4-5`).
Sync the canonical sources (llmProviders.json default-model table +
models.mdx examples) to the alias and regenerate the docs skill mirror.
The run-db-migration Options table listed only `--schema`, but the command
also exposes two operator-facing flags (hindsight_api/admin/cli.py):
- `--embedding-dimension` — enforce an expected embedding dimension after
migrations (omit to skip the dimension sync).
- `--skip-extension-reconcile` — added in #2309; skip the post-migration
vector/text-search index reconcile to speed up no-change re-migrations across
many tenant schemas when the backend is unchanged.
Add both rows to the canonical Options table and regenerate the docs skill
mirror.
The Next.js auth middleware buffers proxied request bodies and truncates
anything over its default 10MB limit before /api/files/retain can parse
the multipart form, so single uploads >10MB silently fail with
"Failed to parse body as FormData".
Set experimental.proxyClientMaxBodySize, defaulting to 100MB to match the
dataplane's HINDSIGHT_API_FILE_CONVERSION_MAX_BATCH_SIZE_MB default and
overridable via the new HINDSIGHT_CP_MAX_UPLOAD_SIZE env var (size string
or byte count).
The "computed freshness trend (stable/strengthening/weakening/new/stale)"
described across the developer docs maps to code in
reflect/observations.py that is unreferenced — not wired into recall,
reflect, or the API, and absent from the OpenAPI schema. It is not a
surfaced feature, so the docs overstated it.
Replace those claims with the freshness behavior that IS shipped: when
newer memories haven't been consolidated yet, reflect treats the affected
observations as stale and verifies them against raw facts. Touches
developer/index, observations, configuration, api/recall, and
best-practices, plus the regenerated skills/hindsight-docs mirror.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(metrics): expose async-operation queue + consolidation backlog as gauges
The bank-stats endpoint already computes operations_by_status,
pending_consolidation and failed_consolidation, but only as a point-in-time
HTTP response per bank. There's no way to trend or alert on "is the worker
keeping up?" / "is the knowledge base caught up?" from Prometheus.
This adds three observable gauges, fed by a 30s background-refresh cache (the
same pattern as the existing db-pool gauges, so the /metrics scrape path stays
synchronous):
- hindsight_async_operations{operation_type,status} -- worker queue depth for
non-terminal states. pending = queued backlog (e.g. retain / consolidation),
processing = in-flight, failed = stranded. Terminal states (completed,
cancelled) are deliberately excluded: a gauge of finished work grows without
bound and says nothing about current load. The processing series is the only
signal that surfaces a hung operation holding a worker slot.
- hindsight_consolidation_backlog -- source memories (experience/world) not yet
consolidated into observations (pending_consolidation).
- hindsight_consolidation_failed -- source memories whose consolidation
permanently failed, recoverable via the consolidation recovery endpoint
(failed_consolidation).
The SQL is lifted from the bank-stats endpoint and is index-backed
(idx_async_operations_status, idx_memory_units_unconsolidated). Per-bank labels
are gated behind the existing metrics_include_bank_id flag (off by default);
when off, counts aggregate per tenant/schema, bounding cardinality to a handful
of series. All queries are PostgreSQL-specific (FILTER, information_schema),
consistent with this collector already being bound to an asyncpg pool.
* review: address feedback on backlog metrics
- Split the consolidation backlog into two separate COUNT(*) queries, each with
a WHERE matching a partial-index predicate exactly (idx_memory_units_
unconsolidated / idx_memory_units_consolidation_failed), instead of one
aggregate with two FILTERs that seq-scans the whole memory_units table on
every 30s refresh across every schema. GROUP BY bank_id still composes
(bank_id is each index's lead column).
- Type the gauge cache keys as NamedTuples (_AsyncOpKey, _BacklogKey) instead of
raw tuples.
- Hoist `import asyncio` to module scope (was imported inside two methods).
- Document that _backlog_task is process-lifetime and intentionally not
cancelled (no teardown hook to hang it on).
- Add tests for the per_bank=True path (bank_id in the cache key + GROUP BY
bank_id in the SQL + bank_id gauge attribute) and assert the backlog queries
are index-matched, not FILTER scans.
* fix(metrics): force index scan for the consolidation backlog count
Splitting the consolidation count into two index-predicate-matched COUNT(*)
queries fixed the failed count (index-only scan) but NOT the backlog count.
Verified on a 114k-row memory_units via EXPLAIN ANALYZE: the backlog query still
seq-scans (~92 ms) because `consolidated_at IS NULL` is true for ~40% of the
table (every observation has a null consolidated_at), so the planner misjudges
selectivity and won't use idx_memory_units_unconsolidated even though the
predicate matches it exactly. ANALYZE doesn't change the plan (structural, not
stale stats); `enable_seqscan=off` confirms the index is usable (~0.1 ms).
Run the backlog count in a scoped transaction with SET LOCAL enable_seqscan=off
to force the partial-index scan (verified ~0.07 ms, transaction-scoped, no
leak). The failed count needs no nudge — consolidation_failed_at IS NOT NULL is
rare, so its index is chosen on cost.
* feat(metrics): gate consolidation backlog gauges behind config flag (off by default)
Add HINDSIGHT_API_METRICS_BACKLOG_ENABLED (default false). The
async-operation queue + consolidation backlog gauges run periodic
per-schema COUNT queries on a background task, so they are now opt-in
rather than always-on when a db pool is set.
* chore: sync embed env template + prettify paperclip README after main merge
---------
Co-authored-by: Nicolò Boschi <boschi1997@gmail.com>
MarkItDown advertises image extensions, but without OCR config it can
fail screenshots or scanned images with low-level no-content errors.
Add server-level MarkItDown OCR config that is off by default and
independent from HINDSIGHT_API_LLM_*. When OCR is enabled, the OCR API
key, base URL, and model are required explicitly.
Wire those settings into MarkItDown's llm_client support with a built-in
OCR prompt. Image uploads now fail fast with actionable errors when OCR
is disabled or required settings are missing.
Docs and front-end copy explain that image OCR depends on server config
and requires an OpenAI-compatible OCR/vision endpoint.
Closes#927
The `exact` set-equality match mode landed in the API + generated clients
in #2149 but was never exposed in the control plane, documented, or added
to the hand-maintained SDK wrappers. This completes the feature.
Control plane: add `exact` to the TagsMatch type/unions and to the
tags_match dropdowns in think-view, search-debug-view, and both
mental-model trigger forms; add translated labels to all 10 locales.
Docs: document `exact` in the recall tags_match table + tag_groups,
the reflect tags value list, and the observations scope-listing guide;
regenerate the docs skill mirror.
Clients: add `exact` to the hand-maintained Python and TypeScript wrapper
Literals/unions and docstrings (generated clients already had it; Rust is
generated from openapi.json at build time).
Supersedes #2159.
Add POST /v1/default/banks/{bank_id}/memories/dry-run-extract — a
read-only tool that previews what the retain step would extract from
text WITHOUT changing the bank: extraction only, no entity resolution,
links, embeddings, or persistence. The "dry-run-extract" path makes the
non-mutating nature explicit.
Returns a dedicated DryRunExtractionResult: the candidate facts plus the
aggregated LLM token usage. Each fact (ExtractedFact) is a subset of the
memory-unit shape — only what a fresh extraction produces: text,
fact_type, occurred_start/end, entities[] (raw, unresolved names).
Every prompt-affecting setting is overridable per call (retain_mission,
extraction_mode, custom_instructions, chunk_size, entity_labels,
entities_allow_free_form, llm_output_language) plus the narrator
(agent_name), so a candidate config can be A/B'd against the bank's
current one. The reference date field is named `timestamp` to match the
retain item payload. The engine authenticates the tenant before reading
any bank-scoped config.
Gated by a static server-level flag HINDSIGHT_API_ENABLE_DRY_RUN_EXTRACT
(default true). Since extraction makes a real LLM call, set it to false
to remove the endpoint (returns 404) on cost/abuse-sensitive deployments.
Control plane: a "Dry-run extraction" dialog opened from the Memory Bank
actions menu (text input + raw JSON output, side-by-side).
Regenerated OpenAPI spec + Python/TypeScript/Go client SDKs.
Add retain_structured_chunk_size as an explicit retain chunking knob
for structured inputs. When unset, structured inputs follow the
effective retain_chunk_size instead of the hidden 1.5x overflow factor.
Thread the setting through retain extraction, append/prepend chunking,
bank config resolution, templates, MCP docs, maintained clients,
generated OpenAPI artifacts, the control-plane retain strategy UI, and
the Rust CLI set-config command.
Validate retain_chunk_size and retain_structured_chunk_size as positive
integers while allowing either value to be smaller. Keep the existing
retain_max_completion_tokens check scoped to retain_chunk_size.
Preserve upstream validation details for client errors through the
control-plane proxy so UI alerts and toasts can show concrete
configuration errors without exposing server-side failure details.
Update chunking, config, hierarchical config, template, MCP, client
payload, control-plane serialization, SDK-response, API-client, and
retain UI validation tests for the new behavior.
Nicolò Boschi