- Switch the minimax provider default from MiniMax-M2.7 to MiniMax-M3
in PROVIDER_DEFAULT_MODELS (hindsight-api-slim/hindsight_api/config.py).
- Update the LiteLLM router test fixture to exercise MiniMax-M3.
- Update provider docstrings and example .env entries to mention MiniMax-M3
while keeping MiniMax-M2.7 noted as a previous-generation option.
- Refresh hindsight-docs (developer/models, integrations/hermes,
llmProviders.json) and the docs-skill reference table to list
MiniMax-M3 as the documented default.
The deprecated MiniMax-M2.5 / M2.1 / M2 / M1 IDs are not referenced
anywhere in the active codebase, so no removals are required.
Co-authored-by: octo-patch <octo-patch@github.com>
A single child segfault under load propagates through start-all.sh and
exits the whole container; with the documented --rm run there was no
recovery. Replace --rm with --name hindsight --restart unless-stopped in
the documented server-run commands so a transient crash self-heals.
Leaves the throwaway --rm --entrypoint sh model-inspection command in
custom-models/README.md untouched. Refs #1918.
Local embeddings/reranking pull in numpy (OpenBLAS), torch, and ONNX
Runtime, each of which sizes a native worker pool to the host CPU count.
Hindsight already parallelizes across requests via its own thread-pool
executors, so these native intra-op pools oversubscribe the CPU: on a
many-core host the process accumulates well over 100 native threads,
inflating memory and, under contention, degrading throughput.
Add hindsight_api/_thread_limits.py and apply it as the first statement
in __init__.py (before numpy is imported), bounding OMP/OPENBLAS/MKL/
NUMEXPR to min(16, available CPUs) via setdefault. 'Available' is the
budget actually granted to the process — the smallest of the CPU-affinity
set, the cgroup CPU quota (--cpus / cpuset), and os.cpu_count(). This
matters in containers: os.cpu_count() reports the host's cores even when
the container is limited, so a --cpus=4 container on a 64-core host would
otherwise size BLAS pools to far more threads than it can run.
The 16 ceiling caps runaway growth on large hosts while leaving
within-call parallelism intact; setdefault means any operator-set value
is honored. These are read once at library load time, so they are
process-level (not per-tenant/bank) — documented in configuration.md.
A subprocess regression test reproduces the oversubscription on Linux
hosts with more cores than the ceiling, guarding the before-numpy import
ordering that makes the cap effective. Unit tests cover the cgroup quota
parsing and the available-CPU computation.
This bounds native-thread pressure, which a user reported building up
until the container stopped responding (v0.5.3-v0.5.6). It is a
mitigation; pinning the exact event-loop stall requires a thread dump
from a wedged container and is tracked separately.
* feat(api): add Fireworks AI batch inference provider
Adds a `fireworks` LLM provider with native batch-retain support. Fireworks' batch API isn't OpenAI /v1/batches-compatible, so FireworksLLM subclasses OpenAICompatibleLLM (reusing the OAI-compatible online path) and overrides only the four batch members, adapting Fireworks' dataset->job->download REST workflow back to the OpenAI-batch shapes fact_extraction consumes. No changes to the retain driver/consumer.
* test(api): add live Fireworks batch integration test
Creds-gated end-to-end test that runs the real Fireworks batch workflow through extract_facts_from_contents_batch_api. Validates the live output-JSONL shape against the normalizer (the one thing MockTransport unit tests can't). Skips without HINDSIGHT_API_FIREWORKS_API_KEY + _ACCOUNT_ID; registers the integration/slow markers.
* fix(api): surface Fireworks API error bodies + fix dataset-create payload
The integration test hit a 400 on dataset create. Two fixes: (1) _request now includes the API response body in the raised error instead of discarding it via raise_for_status, so failures are debuggable; (2) drop the invalid 'userUploaded' field from the create-dataset body (it's an output-only source marker) in favor of {format: CHAT}.
* fix(api): include exampleCount in Fireworks dataset-create body
Live API rejected the create with 'example_count is required for uploaded datasets'. Send exampleCount = len(requests) (the JSONL line count) as a string (int64 proto field). Unit test now asserts the dataset body shape.
* test(api): raise Fireworks integration-test timeout to 3600s
A real batch job queues/runs past the suite-wide --timeout 300. The 300s failure was the pytest cap, not a code issue — the workflow got through dataset create, upload, and job create into the poll loop.
* test(api): revert Fireworks integration-test timeout override
Confirmed working end-to-end against live Fireworks (real batch returned facts), so the default suite timeout is fine.
* docs(configuration): add HINDSIGHT_CP_DATAPLANE_API_KEY to Control Plane table + example
* docs(env): add HINDSIGHT_CP_DATAPLANE_API_KEY to Control Plane section
* feat(consolidation): scope-locked parallel LLM dispatch
Adds opt-in HINDSIGHT_API_CONSOLIDATION_LLM_PARALLELISM (default 1, sequential).
Parallel groups acquire per-scope asyncio.Locks computed from each memory's
observation_scopes setting, so two tag groups whose write-scope sets overlap
serialise on the overlapping scope rather than racing on the same observation
row. Locks acquired in tuple(sorted(scope)) order across all groups for
deadlock-freedom. Covers combined / per_tag / all_combinations / explicit-list
scopes uniformly with no operator opt-in.
Refactor extracts the per-memory observation_scopes resolver into module-level
helpers (_resolve_obs_tags_list, _resolve_write_scopes, _parse_observation_scopes,
_scope_sort_key) so the dispatcher and the lock layer share one source of truth.
Per-batch stats deltas now return as _BatchDeltas and merge serially after
dispatch — no lost-update race on shared counters/tag set.
* feat(consolidation): per-batch perf log + default parallelism=4
- Per-batch log uses a batch-local ConsolidationPerfLog so timings,
llm_calls, and input_tokens reflect only that batch's work — no
delta-from-shared-snapshot bleed under parallelism > 1. Local perf
merges into the job-level perf at end-of-batch so the final flush
still totals everything.
- Restore the cumulative processed=N/total progress indicator. The
counter increments + snapshots atomically between awaits in
single-threaded asyncio, no lock needed.
- Bump DEFAULT_CONSOLIDATION_LLM_PARALLELISM from 1 to 4 to match
retain_max_concurrent and let combined-mode banks pick up the
throughput win out of the box. Lock-on-overlap makes this safe by
construction; per_tag / all_combinations banks degrade to serial
automatically.
- New regression test test_per_batch_log_line_attributes_only_own_work
asserts per-batch log fields are isolated (llm calls / memories /
created / timing) and cumulative processed indicator is monotonic.
* chore: regenerate docs-skill + merge two alembic heads to unblock CI
- skills/hindsight-docs/references/developer/configuration.md: regenerated via
./scripts/generate-docs-skill.sh to pick up the new
HINDSIGHT_API_CONSOLIDATION_LLM_PARALLELISM entry from the source
configuration.md edited in the previous commit.
- alembic/versions/mrgvchgraf01_*: empty merge revision unifying main's two
open heads (b5a4c3e2f1d8 add_graph_maintenance_queue and b8c9d0e1f2a3
vchord_cosine_opclass). test_alembic_dag.py::test_single_head catches the
divergence and recommends `alembic merge heads`; this is that. Pre-existing
on main — only surfaced because this PR touches API code and trips the
path-filtered test-api job.
* ci: cap every job in test.yml at 30 minutes
Adds timeout-minutes: 30 to all 60 jobs. Without it each job inherits
GitHub Actions' 6-hour default, so a hung worker or a flaky LLM call can
keep the whole suite "running" for hours before someone notices.
30 min is ~2x headroom over the slowest current job (test-api shards
~13 min, test-doc-examples ~14 min, test-python-client-oracle ~13 min).
If a specific job legitimately needs more later, bump just that one.
* chore: drop redundant alembic merge migration
Main shipped its own merge revision c1d2e3f4a5b6 for the same two heads
(b5a4c3e2f1d8 and b8c9d0e1f2a3) in #1854/#1857's neighbourhood, so my
mrgvchgraf01 became redundant after rebase. Keeping only main's version
to avoid a fresh divergent-heads situation.
* test: bump pool_max_size from 5 to 30 in memory fixtures
The 4 MemoryEngine fixtures in conftest were sized for sequential
consolidation; with consolidation_llm_parallelism now defaulting to 4
(and other parallel knobs like retain_max_concurrent=4 already active),
a pool of 5 connections can be exhausted when an HTTP integration test
triggers multiple async retains that each fan consolidation across
several concurrent tag groups.
CI surfaced this as test_async_retain_parallel hanging on test-api
shard 2 — 5 parallel retains × 4-way intra-op consolidation parallelism
+ the test's own polling HTTP calls all competed for 5 connections
under xdist's worker concurrency. Bumping to 30 keeps tests bounded
but matches a more realistic deployment pool size (default prod cap
is 100) and removes the head-of-line stall.
* test: bump pg0 max_connections to 300, pool to 15, fix configurable counter
CI surfaced two real failures from the previous bump:
- shard 2: tests/test_hierarchical_config.py::test_hierarchical_fields_categorization
hardcoded `assert len(configurable) == 36`. Adding consolidation_llm_parallelism
to _CONFIGURABLE_FIELDS made it 37. Bumped and added an explicit
membership assertion so a future drop of the flag fails loudly.
- shard 3: asyncpg.TooManyConnectionsError. With pool_max_size=30 and
8 xdist workers, peak demand was ~240 connections against postgres's
default cap of 100. Two related changes:
* EmbeddedPostgres now accepts a ``config: dict[str, str]`` and
forwards it to Pg0 (which has been a documented Pg0 kwarg). The
pg0_db_url fixture passes ``{"max_connections": "300"}`` so 8
workers × pool=15 fits comfortably.
* Pool back to 15 (from 30 in the previous commit). 15 still
accommodates default consolidation_llm_parallelism=4 +
retain_max_concurrent=4 + the test's own queries without
head-of-line stalls, but caps total connections at a sane
fraction of the 300 max.
* fix(retain): apply batching to Oracle entity resolution + guarantee pg_trgm RESET
Follow-up to #1841.
- Batch the Oracle UTL_MATCH fuzzy candidate query with the same
retain_entity_resolution_batch_size knob as PG. The Oracle path had the
identical single JSON_TABLE-join risk on banks with many entities.
- Convert the PG trigram `try/except…else + raise` to `try/finally` so
RESET pg_trgm.similarity_threshold is unconditionally issued. Without
RESET, the lowered threshold leaks back to the pooled connection for
whoever borrows it next.
- Add a test that exercises the RESET path when conn.fetch raises mid-batch.
- Add a test for Oracle batching that mirrors the PG batching test.
- Document HINDSIGHT_API_RETAIN_ENTITY_RESOLUTION_BATCH_SIZE in
configuration.md (the table next to HINDSIGHT_API_RETAIN_ENTITY_LOOKUP).
* chore: regenerate hindsight-docs skill after configuration.md edit
The generate-docs-skill.sh mirror under skills/hindsight-docs/references/
needed to be rebuilt after the new env var was added to the developer
configuration table. Caught by the verify-generated-files CI job.
* chore(alembic): merge graph_maintenance_queue and vchord_cosine_opclass heads
PRs #1668 (vchord cosine opclass) and #1772 (async link recompute) both
branched off the same parent and were merged onto main without rebasing,
leaving two parallel Alembic heads:
b5a4c3e2f1d8 (graph_maintenance_queue, parent: e9b2c7d1f3a4)
b8c9d0e1f2a3 (vchord_cosine_opclass, parent: 86f7a033d372)
tests/test_alembic_dag::test_single_head fails on every PR until they're
unified. This is a structural merge revision with no schema changes —
its only job is to make `alembic upgrade head` unambiguous again.
Bundled into this follow-up PR rather than split out because the same CI
job blocks both and the merge is a one-line topology fix.
* docs(multilingual): add pg_search backend to selector and comparison table
* docs(multilingual): add pg_search backend to selector and comparison table
* docs(models): list openai-codex and openrouter in embeddings Supported Providers table
* docs(models): list openai-codex and openrouter in embeddings Supported Providers table
* feat(worker): add priority-based consolidation bank scheduling (#1715)
Add HINDSIGHT_API_WORKER_CONSOLIDATION_BANK_PRIORITY env var to control
which banks' consolidation tasks are claimed first when a slot opens.
This prevents large banks from being starved by many small banks cycling
through limited global consolidation slots.
Format: comma-separated bank-pattern:priority pairs (higher = claimed first).
Patterns support * wildcards; bare * is the catch-all default.
Example: "shadow-*:10,staging-*:5,*:1"
Implementation uses tiered claiming — each priority level is a separate
index-friendly query, no JOINs or computed ORDER BY. Bank serialization
(max 1 concurrent consolidation per bank) is preserved.
* fix: suppress chained exception in _parse_bank_priority
Hindsight's published image deliberately omits llama-cpp-python to keep
the image small, so setting HINDSIGHT_API_LLM_PROVIDER=llamacpp directly
against ghcr.io/vectorize-io/hindsight fails with ModuleNotFoundError.
Adds a docker-compose recipe that runs the official llama.cpp server
container as a sidecar and points Hindsight's openai provider at it via
HINDSIGHT_API_LLM_BASE_URL. Verified end-to-end against
ghcr.io/ggml-org/llama.cpp:server pulling Gemma 4 E2B from HuggingFace.
The named volume is mounted at /root/.cache/huggingface (where
llama-server actually caches downloads) so the GGUF survives stack
recreation. README documents the CPU perf reality and how to flip the
relevant blocks for NVIDIA GPU acceleration.
Also links the recipe from the "Built-in llama.cpp" tip in the models
docs so users following the docs find the Docker setup.
Closes#1807. The HTTP-based rerankers (cohere, openrouter, zeroentropy,
siliconflow, alibaba, litellm proxy/SDK, google) all hardcoded a 60s
timeout, forcing users with slower self-hosted models or large batches
to patch the source. Each provider now reads its own
HINDSIGHT_API_RERANKER_<PROVIDER>_TIMEOUT env var (default 60.0s, so
unset envs keep current behavior). TEI already had its own knob.
Use recall question_date/query_timestamp as the reference time for combined
scoring instead of always using server utcnow(). This keeps historical replay
and offline evaluations from penalizing memories that were recent at query
time.
Normalize naive query timestamps to UTC before scoring, update
API/client/OpenAPI/docs/MCP descriptions, and add recall-level coverage proving
combined scoring receives the query-time anchor.
* feat(api): async link recompute to fix outgoing-link staleness after deletes
When a memory_unit is deleted (via delete_document, delete_memory_unit, or
document re-ingest via handle_document_tracking), the FK cascade removes its
incoming temporal/semantic links. Other units that had this unit in their
top-K neighbours therefore lose links and stay permanently under-capped —
retain only generates links for newly-inserted units, never re-evaluates
surviving ones.
This adds a reactive top-up:
* Inside the delete transaction, capture from_unit_ids that pointed at the
doomed units and write them to a new link_recompute_queue table (PG: ON
CONFLICT DO NOTHING, Oracle: IGNORE_ROW_ON_DUPKEY_INDEX hint for dedup).
* After commit, submit_async_link_recompute schedules a new task type
("link_recompute"), deduplicating per bank.
* Worker drains the queue in batches of 50; for each victim it counts
current outgoing temporal/semantic links and, if below cap, runs the
same probes used at retain time (fetch_temporal_neighbours,
compute_semantic_links_ann) to find replacements. bulk_insert_links has
ON CONFLICT DO NOTHING, so re-probing freely is safe.
submit_async_link_recompute is also called after every retain, where it
short-circuits with no_work=True when the queue is empty — that lets the
upsert path (handle_document_tracking) enqueue victims inline without
needing a return-value plumbing change.
Worker slot is opt-in (default 0) via HINDSIGHT_API_WORKER_LINK_RECOMPUTE_MAX_SLOTS.
Tests cover enqueue correctness (cross-doc, self-exclude, entity-link
skip, dedup), worker behaviour (empty drain, missing-victim no-op,
top-up to cap, no-op at cap), and a cap-parity guard against retain-side
constants drifting.
* docs: revamp /developer/api/operations with all 6 operation types
The page previously listed only batch_retain + consolidate. Rewritten to
cover every async task type Hindsight runs: retain, file_convert_retain,
consolidation, refresh_mental_model, link_recompute (new), and
webhook_delivery — with triggers, lifecycle states, bank-dedup notes, and
the full list/status/cancel/retry endpoint surface.
Also adds HINDSIGHT_API_WORKER_LINK_RECOMPUTE_MAX_SLOTS to the worker
configuration table.
* refactor(api): rename link_recompute → graph_maintenance + kind discriminator
Generalize the queue and worker so future post-mutation cleanups (orphan
entity pruning, stale cooccurrence removal, etc.) can ride on the same
async surface without spawning their own task types.
Schema (alembic b5a4c3e2f1d8): table renamed to graph_maintenance_queue
with shape (bank_id, kind, target_id, enqueued_at) and PK on
(bank_id, kind, target_id). Today the only kind is 'relink_unit', which
holds the same payload as the previous link_recompute_queue.
Renames (mechanical):
* task_type and operation_type: link_recompute → graph_maintenance
* env var: HINDSIGHT_API_WORKER_LINK_RECOMPUTE_MAX_SLOTS →
HINDSIGHT_API_WORKER_GRAPH_MAINTENANCE_MAX_SLOTS
* module hindsight_api/engine/link_recompute.py →
hindsight_api/engine/graph_maintenance.py
* engine helpers: enqueue_link_recompute_victims → enqueue_relink_victims;
run_link_recompute_job → run_graph_maintenance_job;
submit_async_link_recompute → submit_async_graph_maintenance;
_handle_link_recompute → _handle_graph_maintenance
* ops methods: enqueue_link_recompute_victims → enqueue_graph_maintenance
(now takes kind + target_ids);
claim_link_recompute_batch → claim_graph_maintenance_batch
(now returns (kind, target_id) tuples)
* worker job result keys: victims_processed → targets_processed,
links_added → relink_links_added
Worker now groups each claimed batch by kind and dispatches to a per-kind
handler; unknown kinds are dequeued and logged without crashing (added
test_skips_unknown_kind_without_failing). The 'relink_unit' handler is
the same code that previously lived inline in run_link_recompute_job.
Docs updated: operations.md reframes the section around graph_maintenance
as a framework with kinds, with relink_unit documented as the first one;
configuration.md gets the new env var name.
Revision ID bumped from d8f1e2c3a4b5 to b5a4c3e2f1d8 since the table
schema changed shape — dev/staging DBs that already applied the previous
revision get a fresh migration instead of a silent no-op.
* docs(operations): rework per review — trim, link out, multi-language tabs
- Drop the unsupported Kafka note and the type-summary table; the
per-section headings carry the same info without duplication.
- Add a parent-op section for retain_batch explaining how Hindsight splits
large submissions into a parent + N children and how exclude_parents
hides the parent rows.
- file_convert_retain: point at Configuration → File Processing for which
converter runs (markitdown / Docling / LlamaParse).
- consolidation: shorten to a one-liner pointing at the Observations page
instead of restating it.
- refresh_mental_model: mention the auto-refresh trigger and drop the
LLM-provider gate caveat (the model-level check covers it).
- graph_maintenance: shorter why/what framing without the algorithm walk,
drop the PG/Oracle asymmetry note (matches retain-time semantic behaviour
and isn't operations-doc material).
- Convert curl examples to <Tabs>/<CodeSnippet> with Python, Node.js, CLI,
and Go variants, matching the pattern used by recall/retain/documents.
Added examples/api/operations.{py,mjs,sh,go} with sections wired into
the Tabs blocks.
Page renamed .md → .mdx so the Tabs/CodeSnippet imports work.
* docs(operations): correct file-parser list
Hindsight ships three parsers: markitdown (default), iris (Vectorize Iris
cloud), and llama_parse. Docling was never wired up — drop it from the
file_convert_retain note and name the actual options + the
HINDSIGHT_API_FILE_PARSER env var that selects between them.
* refactor(api): drop kind discriminator; add entity + cooccurrence prune passes
graph_maintenance is one job now, not a dispatcher of subtypes. Every
invocation runs three passes:
1. Link top-up — drains graph_maintenance_queue (the only queued work) and
tops up each victim unit's outgoing temporal/semantic links via the same
probes retain uses.
2. Orphan entity prune (NEW) — deletes entities in the bank that no longer
have any unit_entities references. FK ON DELETE CASCADE on
entity_cooccurrences cleans up cooccurrences pointing at pruned entities
automatically.
3. Stale cooccurrence prune (NEW) — defensive sweep for cooccurrence rows
where both endpoints still exist but no current memory_unit references
both of them (the cooccurrence was real when recorded, but every unit
witnessing it has since been deleted).
Schema change: graph_maintenance_queue loses the kind column. It's now just
(bank_id, unit_id, enqueued_at) with PK (bank_id, unit_id). Renamed
target_id → unit_id to make intent obvious. The bank-wide sweeps in passes
2 and 3 don't need per-target queueing — they're backed by entities(bank_id)
and unit_entities(entity_id) indexes.
Ops surface: enqueue_graph_maintenance / claim_graph_maintenance_batch lose
the kind parameter and return unit-id-only payloads. Added
prune_orphan_entities and prune_stale_cooccurrences as ops methods with PG
and Oracle implementations.
Triggers: delete_document and delete_memory_unit now submit
graph_maintenance whenever any unit is removed (not gated on whether relink
victims were enqueued), so the entity/cooccurrence sweeps fire even when a
deleted unit had no incoming links.
Test surface: dropped the unknown-kind test and the cross-kind enqueue
test. Added TestOrphanEntityPrune (scoped sweep, doesn't cross banks) and
TestStaleCooccurrencePrune (prunes when no shared unit, keeps when shared).
All 14 tests in tests/test_graph_maintenance.py pass.
Docs: operations.mdx graph_maintenance section drops the kinds framing and
describes the three passes directly.
* docs(ops_oracle): correct misleading rowcount comment
The Oracle DatabaseConnection wrapper reshapes cursor.rowcount into a
PG-compatible "DELETE N" status string before returning, so the shared
parsing in prune_orphan_entities works on both dialects. The previous
comment claimed the opposite.
* fix(ci): test/example bugs surfaced by CI run
* test_graph_maintenance: _insert_cooccurrence now sorts the two entity
IDs before insert. entity_cooccurrences has a CHECK constraint
entity_id_1 < entity_id_2 (canonical ordering to dedupe (A,B) vs (B,A))
which my helper ignored. asyncpg surfaced this as a CheckViolationError
in test_keeps_cooccurrence_with_shared_unit.
* examples/api/operations.py: collapsed two top-level asyncio.run() calls
into a single asyncio.run(main()). Multiple event loops on the same
Hindsight client broke the SDK's async HTTP context ("Timeout context
manager should be used inside a task"). The doc snippets also use a
real operation_id pulled from list_operations rather than a hardcoded
one that doesn't exist.
* examples/api/operations.sh: was using a hardcoded UUID, so cancel/retry
returned 404 against the live API. Now creates a real pending op via
--async retain, exercises get/cancel on it, then creates a second op
and cancels it so retry has something to re-queue.
* operations.mdx: added the CLI tab to the async-retain Tabs block —
code-parity check requires all four language tabs and was rejecting
the build.
* fix(ci): cooccurrence assertions + python example loop reuse
* tests/test_graph_maintenance.py: both stale-cooccurrence assertions
now query (entity_id_1, entity_id_2) with the same canonical sort the
insert helper applies. The test_keeps_cooccurrence_with_shared_unit
failure ("None == 5") was caused by inserting (sorted_a, sorted_b)
but reading (ent_a, ent_b) — the SELECT just missed the row.
* examples/api/operations.py: dropped the sync client.retain() seed call
in favour of aretain_batch inside the async main(). Mixing sync
(client.retain → _run_async → its own event loop) with the async
operations API (asyncio.run(main) → fresh loop) left the underlying
HTTP client bound to a dead loop, surfacing as
"Timeout context manager should be used inside a task".
* skills/hindsight-docs/references/developer/api/operations.md: regenerated
to match the .mdx — verify-generated-files caught the drift from the
previous CLI-tab edit.
Allow ParadeDB pg_search BM25 indexes to be created with a configured
tokenizer via HINDSIGHT_API_TEXT_SEARCH_EXTENSION_PG_SEARCH_TOKENIZER.
Validate supported tokenizer values and thread the setting through
startup reconciliation, Alembic index creation paths, Docker examples,
docs, generated docs, and tests.
The default remains unset so existing pg_search deployments continue to
use ParadeDB's default tokenizer unless explicitly configured. Changing
the value for an existing database still requires rebuilding the
pg_search indexes or recreating the database.
* docs(config): note litellm-sdk embeddings api key is optional for ambient credentials
* docs(config): note litellm-sdk embeddings api key is optional for ambient credentials
Allow enabling uvicorn access log via environment variable, so Docker/k8s
users can turn it on declaratively without modifying start-all.sh.
Closes#1752
* feat(api): add ParadeDB pg_search as Citus-compatible BM25 backend
Adds a fourth value (`pg_search`) for `HINDSIGHT_API_TEXT_SEARCH_EXTENSION`
alongside the existing `native`, `vchord`, and `pg_textsearch`. ParadeDB
pg_search is the only true-BM25 backend that works on a Citus distributed
Postgres cluster, so this unblocks horizontally scaled deployments.
The retrieval arm builds the @@@ predicate via paradedb.boolean(should =>
ARRAY[paradedb.match('text', $4), ...]) since @@@ on the key_field requires
field-qualified terms; this preserves multi-field coverage (text + context
+ text_signals) without needing query string interpolation.
Includes a docker-compose example under docker/docker-compose/pg_search/
based on the official paradedb/paradedb:latest-pg17 image.
Closes#1754
* fix: accept pgroonga in n9i0 migration; clarify consolidator search_vector comment
- n9i0 (learnings + pinned_reflections) validation now permits 'pgroonga',
treating it as native at this migration stage. ensure_text_search_extension()
at startup converts the reflections table (renamed from pinned_reflections in
p1k2l3m4n5o6) to pgroonga structures; the learnings table is dropped in the
same later migration so its transient native column never reaches steady state.
Without this, pgroonga users hit ValueError on a fresh install.
- consolidator.py single-observation INSERT: the previous comment claimed
search_vector was GENERATED ALWAYS, but migration p4q5r6s7t8u9 dropped that
expression. Updated to reflect current behavior and flag the resulting gap
for native (observations land with NULL search_vector and are not BM25-
searchable until reflected/re-ingested) so a follow-up can address it.
* chore: regenerate hindsight-docs skill after rebase
Rebasing onto main pulled in hindsight-docs/ changes from #1704
(Codex OAuth embeddings) and #1538 (pgroonga). Re-run the
generate-docs-skill.sh generator so the cached
skills/hindsight-docs/references/developer/configuration.md mirror
matches the current developer docs and verify-generated-files passes.
Add openai-codex embeddings provider using the existing Codex OAuth token, support OpenAI output dimension overrides, and document the 384-dimension configuration path. Also redacts the example Telegram bot token in docs.\n\nTests:\n- uv run pytest tests/test_embeddings_openai_batch_size.py -q\n- uv run pytest tests/test_embeddings_openai_batch_size.py tests/test_custom_embedding_dimension.py tests/test_gemini_embeddings.py tests/test_litellm_sdk_embeddings.py -q\n- HINDSIGHT_API_LLM_PROVIDER=mock HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai-codex HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL=text-embedding-3-small HINDSIGHT_API_EMBEDDINGS_OPENAI_DIMENSIONS=384 HINDSIGHT_API_EMBEDDINGS_OPENAI_BATCH_SIZE=2 uv run python - <<'PY' ... create_embeddings_from_env/encode smoke
Co-authored-by: Irgendwer <irgendwer@MacBook-Pro-von-Irgendwer.local>
* feat(bm25): make native language configurable + opt-in pgroonga backend
Adds two new env-level config knobs and a new opt-in BM25 backend so users
can serve non-English banks (especially CJK) out of the box.
- HINDSIGHT_API_BM25_LANGUAGE drives the PostgreSQL text search dictionary
used by the native tsvector backend (default: english). Validated as a
PG identifier so it can be safely embedded in to_tsvector('<lang>', ...).
- HINDSIGHT_API_RETAIN_OUTPUT_LANGUAGE forces the fact extractor to emit
facts in the specified language regardless of source content's language.
Independent from bm25_language so users can mix indexing/extraction
languages deliberately.
- New 'pgroonga' option for HINDSIGHT_API_TEXT_SEARCH_EXTENSION. Uses
TokenBigram + NormalizerNFKC150 — single polyglot index handles English,
CJK, etc. simultaneously. Ships with a docker-compose recipe.
To support a per-deployment language, the GENERATED ALWAYS expression on
memory_units.search_vector (and reflections.search_vector) is dropped via
new alembic migration p4q5r6s7t8u9. The application now populates these
columns at INSERT time using the configured bm25_language.
* docs(bm25): rename env var to scope it to native; move multilingual content to dedicated page
- Rename HINDSIGHT_API_BM25_LANGUAGE → HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE.
The setting only applies to the "native" backend (vchord/pg_textsearch/pgroonga
use their own tokenizers), so the env var name now reflects that scope. Field
renamed to text_search_extension_native_language.
- Trim configuration.md back to a brief env-var table + link. The expanded
multilingual / CJK / pgroonga content moves to the dedicated multilingual.md
page, alongside the existing LLM / embedding / reranker multilingual guidance.
* feat(llm-output-language): rename and broaden to cover retain + consolidation + reflect
Renames HINDSIGHT_API_RETAIN_OUTPUT_LANGUAGE → HINDSIGHT_API_LLM_OUTPUT_LANGUAGE
(field llm_output_language) and applies the same "respond exclusively in {lang}"
directive across every LLM-generated artifact:
- retain (fact extraction) — already wired, just renamed.
- consolidation (observations / mental models) — appended to the batch
consolidation prompt via a new llm_output_language parameter.
- reflect (response synthesis) — appended to the final-system prompt via a
new parameter threaded through run_reflect_agent and memory_engine.
The shared directive lives in engine/prompt_utils.output_language_directive
so all three pipelines build the same instruction from a single source.
* docs(multilingual): drop the backfill-after-language-change section
* feat(api): add targeted consolidation by observation scopes (#1625)
Add `observation_scopes` parameter to the consolidate endpoint to run
consolidation only on memories matching specific tag scopes, and add
`enable_auto_consolidation` config flag to disable automatic
post-retain consolidation.
* docs: add targeted consolidation and auto-consolidation config docs
Update observations docs with targeted consolidation section,
trigger consolidation endpoint reference, and auto-consolidation
disable flag. Regenerate OpenAPI spec and client SDKs.
* docs: add enable_auto_consolidation to banks API docs
The Ollama provider's native API path (_call_ollama_native) used raw httpx
without passing authentication headers, causing 401 errors when connecting
to Ollama Cloud endpoints. The verify_connection call succeeded because it
uses the OpenAI-compatible path (AsyncOpenAI client) which includes the
API key, but structured output calls failed.
- Pass Authorization Bearer header in native Ollama httpx calls when a
real API key is provided (not the "local" dummy fallback)
- Add ollama-cloud as a first-class provider that uses the OpenAI-compatible
path exclusively (no native /api/chat fallback), requires an API key,
and defaults to https://ollama.com/v1Closes#1559
* feat(api): add clear endpoint for mental model content (#1706)
Add POST /mental-models/{id}/clear that resets content to empty so the
next refresh performs a full re-synthesis regardless of trigger mode.
Useful for periodic compaction of delta-mode models that accumulate
drift over many incremental refreshes.
* docs: add SDK code examples for clear_mental_model
Add clear_mental_model to Python and TypeScript wrapper clients, and
add code snippets (Python, Node.js, CLI, Go) to the mental models
docs page using the same CodeSnippet pattern as other operations.
* ci: add clear_mental_model to CLI coverage skip list
* fix: update MCP tool count assertion for clear_mental_model
* fix(api): wire up per-operation LLM concurrency caps
HINDSIGHT_API_RETAIN_LLM_MAX_CONCURRENT,
HINDSIGHT_API_REFLECT_LLM_MAX_CONCURRENT, and
HINDSIGHT_API_CONSOLIDATION_LLM_MAX_CONCURRENT were parsed into config but
never read — every LLM call shared the single global semaphore. Users on
rate-limited providers who set these to reserve per-operation capacity
silently got the global cap instead.
Add per-operation semaphores in llm_wrapper, dispatched by call scope
prefix (retain*/reflect*/consolidation*). Each per-op cap composes with
the global cap rather than replacing it: a retain call must acquire both
the retain semaphore and the global semaphore. Scopes without a tracked
operation (bank_mission, memory_think, mental_model_delta_ops,
verification) keep the global-only behavior.
Fixes#1574.
* chore: apply ruff format to cross_encoder.py
CI's verify-generated-files job fails on main because this line drifted
out of the ruff-format style. Folding the auto-format into this PR so the
job goes green.
Storage page referenced `DATABASE_URL` but the actual env var is
`HINDSIGHT_API_DATABASE_URL` (matches configuration.md and admin-cli.md).
The admonition heading uses a gradient via `-webkit-text-fill-color: transparent`,
which inline `<code>` children inherited — making backtick content in titles
like `:::tip Set a stable HINDSIGHT_API_WORKER_ID in production` invisible.
Reset the fill color on code inside admonition headings.
Closes#1722
* fix(mental-models): cap history array length to prevent jsonb overflow
Each content-changing update to a mental model appends a full snapshot
(previous_content + previous_reflect_response + changed_at) to the
`mental_models.history` jsonb array. Without a cap the array grows
unboundedly. Postgres has a hard 256MB limit on the total size of jsonb
array elements; once a row crosses it, every subsequent UPDATE to that
row fails with SQLSTATE 54000 ("total size of jsonb array elements
exceeds the maximum of 268435455 bytes") — the mental model becomes
permanently un-writable until the history is manually trimmed at the DB
level.
This is reachable in normal use: with reflect responses on the order of
hundreds of KB (common when the bank has many memories) and a workload
that refreshes a small set of mental models repeatedly, the limit is
hit in a few hundred refreshes.
Fix
---
Trim history to the most recent N entries at write time. The append
becomes a single subquery that takes the last N elements of
`COALESCE(history, '[]'::jsonb) || $new::jsonb` ordered by their array
index. New env var `HINDSIGHT_API_MENTAL_MODEL_HISTORY_MAX_ENTRIES`
controls N; default 50 (well under the 256MB ceiling even with large
reflect responses, while preserving enough recent history for audit /
rollback).
Rows already over the limit pre-fix need a one-shot manual trim of
their `history` column — the SQL-side append in this PR cannot heal a
row whose existing `history` is already too large to materialize in
the jsonb engine, because evaluating `history || $new` itself raises
54000. After the manual trim, this fix prevents recurrence.
Tests
-----
New `test_history_capped_to_max_entries`: with max_entries=3, six
content updates produce a 3-element history (most recent first: v5,
v4, v3 — v1 and v2 dropped). Existing history tests cover the unchanged
ordering, snapshot, and gating behaviors.
Docs
----
New row in `configuration.md`.
* fix(mental-models): slim history snapshot to based_on only
Each history entry previously stored the full reflect_response payload
(~400-500 KB), pushing per-row size to ~22 MB at the cap. That exceeds
heap-page fit, so every UPDATE writes a full TOAST row and skips HOT,
leaving a dead tuple that must be vacuumed.
The control-plane history view only reads previous_reflect_response.based_on;
everything else in the payload is unused. Store just that slice — per-entry
size drops ~100x, rows fit on a heap page, HOT updates re-enable, dead
tuples self-clean.
Existing bulky rows rotate out naturally via the cap=50 ring buffer.
* fix: pass max_entries as SQL parameter and fix history test assertion
- Pass mental_model_history_max_entries as a query parameter ($N) instead
of f-string interpolation to harden against future config source changes
- Fix test_history_snapshots_omit_reflect_response_when_based_on_missing:
the test was asserting against the *current* reflect_response rather than
the *previous* one captured in the history entry. Added an extra update
so the based_on={} reflect_response actually becomes a "previous" state.
---------
Co-authored-by: Nicolò Boschi <boschi1997@gmail.com>
* docs: add HINDSIGHT_API_WORKER_ID tip to API quickstart
Mirrors the tip already present in installation.md so users who follow
the API quickstart's Docker tab see the same guidance about pinning a
stable worker ID. Closes#1616.
* docs: mirror WORKER_ID tip to versioned_docs v0.6 (from #1648)
Folding in xmh1011's strict-improvement hunk from #1648: the
versioned snapshot for v0.6 should carry the same production tip
as the live doc. Same prose, same `:::tip` block. Includes the
auto-regenerated skills/ reference.
* feat(api): add litellmrouter provider for LLM fallback chains
Closes#1464.
New "litellmrouter" provider wraps LiteLLM Router with ordered fallback
across a configurable chain of deployments. On transient errors
(rate-limit, timeout, 5xx) the Router falls back to the next deployment
in declared order; auth errors (401/403) are not retried so a
misconfigured key cannot silently cascade through the chain.
Configuration is provider-scoped (one-word LITELLMROUTER namespace to
avoid clashing with the existing LITELLM_* settings used by the
embeddings/reranker layers):
HINDSIGHT_API_LLM_PROVIDER=litellmrouter
HINDSIGHT_API_LLM_LITELLMROUTER_CHAIN=<json list of deployments>
Per-operation chains are supported via the same pattern that already
exists for retain/reflect/consolidation:
HINDSIGHT_API_RETAIN_LLM_LITELLMROUTER_CHAIN=...
HINDSIGHT_API_REFLECT_LLM_LITELLMROUTER_CHAIN=...
HINDSIGHT_API_CONSOLIDATION_LLM_LITELLMROUTER_CHAIN=...
Each per-op chain falls back to the default chain when unset, mirroring
the existing per-op provider/model overrides.
Chain entries are tagged as credential fields and are never exposed via
the bank-config API. Batch APIs are intentionally unsupported in router
mode; users that need batch retain should configure a single provider.
* refactor(api): dedup litellmrouter on top of LiteLLMLLM, accept arbitrary chain keys, add CI matrix entry
The retry/parse/metrics loop in LiteLLMRouterLLM was a near-verbatim copy of
LiteLLMLLM. Extract three small hooks on the base class
(_acompletion, _resolve_completion_model, _stage_label) and have the Router
provider inherit + override only what differs.
Drop strict validation of chain entries. The parser now requires only
'provider' and 'model'; everything else passes through to LiteLLM Router
unchanged. Top-level keys (rpm, tpm, weight, model_info, ...) flow to the
deployment record; an optional 'litellm_params' sub-object merges into the
inner params dict. Documented and tested.
Add a litellmrouter row to the LLM acceptance matrix using a single OpenAI
deployment in the chain. The chain JSON is built from secrets in a
dedicated step and masked in logs before being written to GITHUB_ENV.
* refactor(api): pure pass-through to litellm.Router, drop translation layer
Replace the chain-with-Hindsight-shape API with a thin pass-through to
litellm.Router. The HINDSIGHT_API_LLM_LITELLMROUTER_CONFIG env var is now
a JSON object forwarded verbatim to Router(**config). Hindsight's only
imposed rules: model_list is non-empty, each entry has a model_name, and
requests route against the first entry's model_name.
This removes _LITELLM_PROVIDER_PREFIX (provider→prefix translation),
_build_model_list (flat→nested rewrite), and _build_fallbacks (auto-wired
ordered fallback). Users now write LiteLLM-native configs and pick their
own routing strategy — ordered fallback via 'fallbacks', load-balancing
via shared model_name + 'routing_strategy', rate-limit awareness via rpm/
tpm, and so on. The docs link to LiteLLM's reference rather than
recapitulating it.
Renames:
ENV_LLM_LITELLMROUTER_CHAIN -> ENV_LLM_LITELLMROUTER_CONFIG
llm_litellmrouter_chain -> llm_litellmrouter_config
_parse_llm_router_chain -> _parse_llm_router_config
LLMProvider(litellmrouter_chain=) -> LLMProvider(litellmrouter_config=)
The dataclass fields change shape from list[dict] to dict (JSON object).
Net reduction across the touched files: ~165 lines.
* docs: regenerate hindsight-docs skill from updated configuration.md
* refactor(api): drop all shape validation on litellmrouter config, use fixed 'default' entrypoint
The previous version still inspected the user's config in two places:
the parser checked model_list/model_name shape, and __init__ pulled
primary_model_name out of model_list[0]. Both are gone.
The parser now only verifies the env var is parseable JSON. Whatever the
user supplies — dict, list, missing keys, weird shapes — flows through.
LiteLLM Router is authoritative about the shape and raises its own
errors at construction time if something's wrong.
The provider no longer extracts a 'primary' name from the input. Instead
it always issues completions against model_name='default' — the single
Hindsight-imposed convention. Users put one entry with that name in
their model_list as the entrypoint and use any names they want for
fallback/load-balance/weighted-pool members. This avoids both pre-
validation footguns and any dependence on Router's internal API
(model_names, model_list attributes) that could shift between versions.
Docs and tests updated to match. The CI matrix already used 'default'.
* docs: regenerate hindsight-docs skill
* ci(test): cap retain max_completion_tokens for litellmrouter matrix row
gpt-4.1-nano caps OpenAI completion at 32768 tokens, but Hindsight's
default DEFAULT_RETAIN_MAX_COMPLETION_TOKENS is 64000. The 'openai'
matrix row passes because OpenAICompatibleLLM has model-specific token
capping; LiteLLMLLM (and the new LiteLLMRouterLLM by inheritance) don't.
That's a pre-existing limitation orthogonal to this PR — the cap-aware
behaviour lives in OpenAICompatibleLLM and intentionally doesn't apply
to LiteLLM-routed calls.
Lower retain max_completion_tokens via env in the litellmrouter job so
CI exercises the Router path end-to-end instead of dying on a
provider-side BadRequestError that's not the thing we're testing.
* fix(api): cap LiteLLM-routed max_completion_tokens to model registry limit
Hindsight defaults retain_max_completion_tokens to 64000 — fine for
high-capacity models, but breaks against models with smaller caps
(gpt-4.1-nano: 32768; gpt-4o-mini: 16384). OpenAICompatibleLLM already
caps via a hardcoded string-match table; LiteLLMLLM and the new Router
provider didn't, so a default Hindsight install pointed at a small
model would fail with provider BadRequestError.
Cap pre-emptively using LiteLLM's own per-model registry
(litellm.get_max_tokens). For LiteLLMLLM the cap is self.model. For
LiteLLMRouterLLM the cap is the min across all configured deployments,
computed once at __init__ — this way a single max_completion_tokens
value works no matter which deployment Router picks (primary,
fallback, weighted-pool member). Unknown models contribute no cap.
Reverts the temporary CI workaround that lowered HINDSIGHT_API_RETAIN_
MAX_COMPLETION_TOKENS=32000 for the litellmrouter row — Hindsight
should work out of the box.
* docs: shorten litellmrouter config section, add models.mdx pointer
Move the discoverability pointer into models.mdx alongside the existing
LiteLLM tip, where users browsing for model options will find it. Strip
the configuration page entry to its essentials: env-var table, one
ordered-fallback example, and the three short caveats. Defer routing
details to LiteLLM's docs rather than recapitulating them.
* docs(zai): document z.ai provider and add default model
Follow-up to #1529. Adds z.ai (Zhipu GLM series) to the provider list,
example blocks, default-model table, and `.env.example`. Also wires
`zai` into `PROVIDER_DEFAULT_MODELS` so the new docs entry actually
matches what the engine resolves when only the provider is set.
* docs(zai): use glm-4.5-flash as default (free tier)
glm-4.5-air requires a paid balance on z.ai; flash is on the free
tier and works as a sensible default. Air is still listed in the
example as the paid-tier upgrade.
* fix(cp): improve access-key auth UX and harden middleware
- Move logout button from sidebar to header bar (next to GitHub icon),
shown only when access-key auth is configured
- Remove redundant status bar from dashboard page
- Return 401 JSON for unauthenticated API requests instead of HTML redirect
- Redirect to /login on 401 in the API client (skip if already on /login)
- Allow /logo.png through middleware for the login page
- Replace brain emoji with Hindsight logo on login page
- Fix error message visibility in dark mode
- Add loading spinner for bank selector while banks are fetching
- Expose access_key_auth as a feature flag via version endpoint
- Document HINDSIGHT_CP_ACCESS_KEY in configuration and installation docs
* fix(cp): spread default features to handle unknown fields from API
* fix(cp): wrap login page in Suspense for useSearchParams
* docs: surface stable worker_id guidance and zombie-operation recovery
Worker identity defaults to the container hostname, which Docker rotates
on every restart. That stranded several real deployments' consolidation
queues (issue #1470 and the related closed tickets #991 / #696 / #624).
Move the guidance from the configuration reference table — where it
only gets read after the bug bites — into the install path and add a
recovery section next to the decommission commands.
* docs(faq): add zombie-operations entry
* docs: document AlloyDB ScaNN vector extension
Follow-up to #1459. Adds `scann` to the supported vector-extension
list in installation.md and configuration.md, with installation
hints, the 10k-row deferred-build caveat, AlloyDB Omni compose
pointer, and the relaxed switching rules (switching *to* scann is
allowed with existing data).
* refactor(_vector_index): address review nits from #1459
- Lift `from sqlalchemy import text` (and add `Connection`) to module
top in `_vector_index.py`; both helpers now have proper type hints.
- Make `pg_diskann` a first-class entry in a new `RESOLVED_EXTENSIONS`
tuple via `_normalize_resolved`. The configurable boundary stays
strict (`validate_extension` rejects `pg_diskann`); the resolved
helpers (`index_using_clause`, `index_type_keyword`,
`minimum_rows_for_index`, `uses_per_bank_vector_indexes`) accept it
without per-call special-case branches. Behavior is identical.
- Harden `test_alembic_vector_migrations_freeze_vector_sql_locally`
to resolve the migrations dir from `__file__` so the test no longer
depends on cwd.
- Add a one-liner explaining why `_drop_per_bank_vector_indexes`
inlines identifiers instead of using bound parameters (DDL).
Tests: tests/test_vector_index.py (10), tests/test_migration_shape.py
+ tests/test_migrations_thread_safety.py (64). Lint and ty clean.
Octopus