134 Commits

Author	SHA1	Message	Date
Nicolò Boschi	565303d913	docs(documents): say the tags PATCH replaces the array, and test clearing it (#4272) ... * test(documents): cover clearing a document's tags with an empty array The tags PATCH replaces the array rather than merging it, so `tags: []` is how a caller drops every tag. Every guard on that path is written `is not None` rather than a truthiness check so the empty list survives it, but nothing exercised it: a regression to `if tags:` would have turned a clear into a silent no-op and a 200. Adds an engine test (clears the document's tags and its units', runs the same observation-invalidation cascade, and is a no-op when repeated) and an HTTP test (PATCH `{"tags": []}` is 200, an omitted `tags` is still 422). * docs(documents): say that the tags PATCH replaces the array The endpoint description and the docs page both said only that tags are "propagated to all associated memory units", which leaves the question a caller actually has — does sending a tag ADD it, and how do I drop one — unanswered. The replace semantics were documented in exactly one place: two comments in the CLI tab of the docs page, which an API or SDK user never reads. States it where they will see it: the array replaces rather than merges, an omitted tag is dropped, `[]` clears them all, and only an omitted FIELD is the 422. Regenerates the spec, the clients and the docs skill.	2026-09-09 17:13:50 +02:00
Nicolò Boschi	f5b3f76a8d	fix(reflect): say when the structured-output extraction failed (#4230) (#4248) ... Reflect with a `response_schema` runs a second LLM call that reshapes the prose answer into the caller's schema. When that call errored or returned something unparseable, the bare `except` swallowed it and the caller got 200 with `structured_output: null` — indistinguishable from an answer that genuinely held nothing matching the schema. The machine-readable half, which is the reason a caller supplied a schema at all, failed invisibly: no retry signal, no alert. Returning 200 with the text answer is still right; the missing piece was saying the structured half did not happen. `StructuredOutputResult` now carries an `error`, and reflect surfaces it as a nullable `structured_output_error` on the response. Present => the extraction broke (retryable); absent with a null `structured_output` => nothing to extract. The mental-model refresh path uses the same helper and now records the reason in its `structured_output_failed` failure detail instead of only "extraction failed".	2026-09-09 12:59:00 +02:00
Chris Bartholomew	fb94ce0341	feat(mental-models): default list to metadata; MCP list returns metadata only (#4225) ... * feat(mental-models): default list to metadata; MCP list returns metadata only Listing mental models defaulted to returning every model's full synthesized content (and reflect_response). That bloats a caller's context and lets a single list call pull an entire bank's synthesized knowledge in bulk, when the intended way to read a model's content is the single-model read. - MCP list_mental_models tool: returns metadata only (id, name, tags, staleness); the `detail` parameter is removed. An agent discovers models here and reads a specific model's content with get_mental_model. - HTTP GET .../mental-models: `detail` now defaults to `metadata` instead of `full`. Content stays available opt-in via `detail=content`/`full`, and when requested it is delivered and metered the same as a single-model read. - Engine list_mental_models is unchanged and still honors `detail` for internal callers (bank-template export/import need full content). - Regenerated OpenAPI + clients (Python/TypeScript/Go). Tests: the MCP tool is metadata-only with no `detail` param; the HTTP list defaults to metadata and returns content only when detail=content is passed; is_stale is still reported per model on the list. * fix(mental-models): follow through on the list default flip in every caller Flipping the list endpoint's `detail` default from `full` to `metadata` left the callers that were relying on the old default reading nulls. - Control plane: `MentalModelsView` now asks for `detail=content` — it renders the content preview, source query and trigger chips, and seeds the update dialog from the listed row, so metadata alone crashed the search filter (`m.source_query.toLowerCase()` on null) and would have clobbered every trigger setting on save. The search filter is null-guarded too. - CLI: `hindsight mental-model list` asks for `content` (`--verbose` → `full`), restoring the per-row preview and keeping `--output json` useful to scripts. - Docs: the detail-levels table said `full (default)` for both endpoints and showed a `detail` argument on the `list_mental_models` MCP tool that no longer exists; the three SDK list examples printed `source_query` off a default list. Added an upgrade note. - Wrapper clients: the Python docstring still promised a server-side `full` default; the TS one said nothing. - Dropped the "metered the same as a single-model read" claim from the endpoint docstring — a `detail=content` list still validates as one `LIST_MENTAL_MODELS` bank read, not one read per model. * fix(hindsight-all): let the facade ask for mental-model content `mental_models.list()` in both facade paths (the client wrapper and the embedded namespaces) forwarded no `detail`, so after the list default flipped to metadata a hindsight-all caller got content-free rows with no way to ask for more — the one wrapper where the capability was not just defaulted away but unreachable. Forwards `detail` like the TypeScript and Python wrappers do. --------- Co-authored-by: Nicolò Boschi <boschi1997@gmail.com>	2026-09-08 18:47:54 +02:00
Nicolò Boschi	9e6d9e76cc	feat(api,control-plane): a prompt tester for retain, and prompt preview for every operation (#4140) ... * feat(api,control-plane): a prompt tester for retain, and prompt preview for every operation Closes the "render prompts without calling an LLM" half of #3774. A bank's missions only mean something once you can see the prompt they land in, and today that means tracing Python constants and format calls. `POST /banks/{id}/prompts/preview` returns the messages retain, consolidation or reflect would send — in send order, no LLM call, no writes. The operation is the whole request: everything comes from the bank, and the runtime data an operation would be given is a fixed placeholder. A message arrives as `blocks`. The active ones concatenate back to the exact text sent — enforced by a test against what the extraction path itself builds. Each is identified by machine values only (`field`, a `section` slug, or the `heading` the prompt text carries); the response ships no display copy, so names and explanations live in the UI that localises them. An inactive block has no text and marks a setting switched off at the point it would land, so an unset mission is still visible where it would go. Both messages always come back: retain and consolidation keep their system prompt bank-agnostic so one provider-side cache serves every bank, and carry the mission in the user message instead. Only reflect puts its mission in the system prompt. **Two bugs found on the way, both pre-existing in dry-run extraction:** - Neither dry-run nor the preview applied retain strategies. Both resolved config directly instead of through `_resolve_retain_config`, so they ignored the bank's `retain_default_strategy` too — silently extracting and previewing under settings a real retain would never use. Both now resolve the way retain does and take an optional `strategy`. - The preview read the bank's config without running `validate_bank_read`, so a tenant extension denying `GET_BANK_CONFIG` was bypassed by an endpoint that renders that config as prompt text. Both paths now share `_authorize_bank_config_read`, which also carries the bank-existence check. **Control plane.** The dry-run dialog is gone; its work moved into the prompt tester on the bank Configuration tab, because changing a mission and seeing what it extracts is one loop that was split across two dialogs. Blocks re-render for free as settings change; a sample-text box and a Run button spend the LLM call on demand. A strategy picker renders any of the bank's named strategies. Editing a block saves that setting to the bank; `editable` comes from the config layer's own allowlist, so server-level fields say so rather than offering an edit that would collect a 400. `PromptBlockModel.kind` is required with no default: progenitor rejects a default on an inline enum with TypeError(InvalidValue), which breaks the Rust client. * chore(cli): skip preview_prompt in the OpenAPI coverage manifest	2026-09-07 17:27:51 +02:00
Nicolò Boschi	66992496f5	fix(api): 404 bank-scoped reads for a bank that does not exist (#4175) (#4186) ... * fix(api): 404 bank-scoped reads for a bank that does not exist (#4175) GET /stats and GET /memories/list answered 200 with zeroed counters and an empty page for a bank nobody ever created — byte-identical to a healthy, empty bank. A monitor built on either kept passing after the bank it watched was renamed, deleted or recreated under another id, and a typo in bank_id was never surfaced. The same held for every other bank-scoped aggregate/list read: /graph, /stats/memories-timeseries, /entities, /entities/graph, /mental-models, /knowledge-base/{tree,export,search}, /directives, /documents, /tags, /operations, /observations/scopes, /config and /webhooks. (Sub-resource GETs already 404 on the missing child.) Each of those engine reads now calls _require_bank_exists after its own authentication and read authorization, so the check neither widens what a request may see nor creates the bank; the profile row is cached per process, so an existing bank costs no extra query. The 404 is declared in the OpenAPI spec on those operations, so generated clients have a documented missing-bank case. The Rust client's build script drops schema-less error responses first: progenitor models at most one error type per operation and the typed 422 is the one worth keeping. * fix(api): derive the 404 endpoint list from the routing table, not by hand Two follow-ups to the same fix. The regression test enumerated the 17 bank-scoped reads by hand, so a bank-scoped collection GET added later would be covered the day someone remembered to extend the file — exactly the sibling-parity trap the reviewed change is about. It now walks the app's own routes and asserts the contract over every one of them, with two commented exemptions (/document-transfer and /profile, both withdrawn endpoints that answer 410 Gone for every bank). The scan immediately found both, which the hand-written list had missed. The rebase onto main also landed the /profile removal underneath the earlier commit, leaving a declared 404 on an endpoint that can now only ever return 410. Dropped it, and regenerated the spec and clients. * fix(tests): create the bank in tests that read a bank they never created CI found three suites that reached an engine read with no bank row, which the new 404 turns from an empty result into an error. All three are test setup gaps, not behaviour the fix gets wrong — a real deployment always has the row, because every write path (retain included) creates it before anything else exists. - test_memories_extension: a store owns the facts, never the bank row itself, so the three seam reads now create the bank the way a retain would. - test_schema_isolation: the bank row goes into each tenant schema alongside the memory_units row it inserts directly — "created" is per schema, which is part of what the test is about. - test_knowledge_search_text_search_disabled: this engine is stubbed down to the one method under test and has no real pool, so the existence read is stubbed alongside _authenticate_tenant. Also carries the docs-skill copy of the OpenAPI spec, which verify-generated-files caught: it mirrors hindsight-docs/static/openapi.json and was left behind when the /profile 404 was dropped.	2026-09-07 13:32:10 +02:00
Nicolò Boschi	b1de1b9418	fix(operations): allow cancelling in-flight operations (#4131) ... `DELETE /v1/default/banks/{bank}/operations/{id}` only accepted `pending` operations, so an operation stranded in `processing` — orphaned when a worker was killed before it could write a terminal status — could only be cleared by hand-editing `async_operations` and restarting the container. Cancel now accepts `processing` too. It stays cooperative and is never immediate: the row is flipped to `cancelled` and the worker running it stops at its next `_check_op_alive` checkpoint (between retain sub-batches/documents, between consolidation LLM batches). For the orphaned case nothing is running, so the flip is the whole fix. No heartbeat and no per-batch bookkeeping is added. Making the flip stick required guarding the worker writes that had none, and would otherwise overwrite it: - `_schedule_retry` — the one that actually resurrected cancelled work: a task failing after cancellation went back to `pending` and was re-claimed. - `_mark_failed` (poller and engine), `_defer_operation`. `_mark_completed` already guarded on `status='processing'`; tests now pin it. The sibling rollup counted only `completed`/`failed` as done, so a cancelled child stranded its `batch_retain` parent in `processing` forever — the same wedge one level up. Both rollup copies now treat `cancelled` as done and settle the parent on `cancelled` (a real failure still outranks it), cancel performs the rollup itself so cancelling the last outstanding child terminalizes the parent, and a cancelled parent is never flipped back by a child finishing later. Control plane: the Cancel button was gated on `pending`, hiding the fix from the UI an operator would reach for. It now shows for `processing` rows too.	2026-09-07 11:21:06 +02:00
Nicolò Boschi	f747d96c38	feat(recall): fuzzy tag matching on tag_groups leaves (#4026) (#4028) ... * feat(recall): fuzzy tag matching on tag_groups leaves (#4026) Tags increasingly hold user-facing names, and tag filtering is exact array containment. A caller filtering by what a query mentioned passes `typsecript`, and the memory tagged `typescript` is dropped before ranking runs — so the recall returns empty even though ranking would have found it. Better ranking cannot fix that; the match itself has to tolerate the misspelling. A `tag_groups` leaf gains one optional field, `resolve`, defaulting to `exact` (today's behaviour). Set to `fuzzy`, its tags are matched against the bank's tags by similarity instead of literally. That is the whole API change: no new config, no new response field, no new TagsMatch values. Matching is trigram similarity at 0.45 via `entity_resolver._trigram_similarity`, already verified byte-identical to Postgres `similarity()` (#3107), so Postgres, Oracle and store-owned backends behave the same. Resolves: typescropt/typescript 0.57, kubernets/kubernetes 0.62, user:alcie/user:alice 0.47. Does not: mango/mongo 0.33, k9s/k8s 0.14. Known limit, pinned by a test: similarity is length-sensitive. A short tag has few trigrams and one edit destroys three of them, so kakfa/kafka scores 0.20 and does not resolve. Fuzzy matching is effective on descriptive tags and close to inert on very short ones — and that same property is what keeps different short words apart. Resolution runs above the SQL layer, rewriting the leaf into ordinary exact leaves so only those reach the query builders. The ~20 SQL call sites, the Python mirrors used on the graph path, the GIN(tags) index, the store protocol and the Oracle dialect are untouched. Per mode, for tokens t1..tn resolving to E1..En: any/any_strict becomes one leaf over the union; all/all_strict becomes an AND of one OR-leaf per token, so a memory must carry some spelling of each; exact becomes an OR over the cross product, one tag per token, bounded at 32 branches and checked before enumeration, with combinations carrying fewer distinct tags than tokens dropped. Failing closed: a tag that resolves to nothing stays in the filter as itself, leaving the leaf unsatisfiable. Returning an empty list would read as "no tag filtering" in the builders and hand back the whole bank. The vocabulary comes from the existing `list_tags` store method, so there is no schema change. A bank holding more than 5000 distinct tags is rejected with a 422 rather than resolved against a truncated vocabulary. Claude-Session: https://claude.ai/code/session_011KDT484YujNcBxHzbfzNbk * fix(clients): make tag_groups reachable through the wrapper SDKs `Hindsight.recall(tag_groups=...)` and `.reflect(tag_groups=...)` raised ModuleNotFoundError for every caller. The wrapper imported `hindsight_client_api.models.recall_request_tag_groups_inner`, which the generator does not emit: it produces one union model per tag_groups shape and names it after the first schema that used it, so the class is `MentalModelTriggerInputTagGroupsInner`. Nothing caught it because the wrapper's tests never passed tag_groups and the import sits inside the `if tag_groups is not None` branch, so it only fires when the feature is used. Fixed at both call sites, with mirrored regression tests on the Python and TypeScript wrappers asserting a tag group reaches the request body with the leaf's `resolve` intact — the pair the review checklist asks for, since a capability that exists in one wrapper and not the other is invisible to client-coverage-check (it validates request-body fields, not wrapper surface). Also thread tag_groups through the control-plane recall and reflect proxy routes and their client types. Both accepted every other tag filter and silently dropped this one, so no control-plane caller could use compound tag filtering at all — fuzzy or exact. Two follow-ups from reviewing #4026: - Reject `resolve="fuzzy"` in a mental-model trigger's tag_groups. A trigger's scope is read by two paths that resolve differently: the refresh runs through reflect, which resolves fuzzy leaves, while the staleness check and the scope watermark build SQL straight from the stored groups and do not. A stored fuzzy leaf would build content from the resolved tags while never being marked stale by them, and would drift as the bank's tag vocabulary changes. - Promote `entity_resolver._trigram_similarity` to `trigram_similarity`. Two subsystems now share it — entity resolution and fuzzy tag matching — so the leading underscore misrepresented a real contract between them. Claude-Session: https://claude.ai/code/session_011KDT484YujNcBxHzbfzNbk	2026-09-02 16:50:53 +02:00
Nicolò Boschi	21c2160046	feat(entity-labels): add an open-vocabulary multi-valued label type (multi-text) (#4027) ... Entity labels could classify a fact against a fixed vocabulary ("value", "multi-values") or capture one free string ("text"). There was no way to extract *several* values that cannot be enumerated when the bank is configured — the names a thing is known by (canonical name plus abbreviations, acronyms and alternative spellings), ticket references a fact cites, product codes. "multi-text" is a list[str] field with no Literal constraint, so the extractor writes as many values as the content warrants and nothing has to be declared up front. Each value becomes its own key:value entity and, with tag: true, a tag — so a bank can derive a classification from content and then filter on it at recall without the caller supplying the vocabulary. Added to MapField as well as LabelGroup: the control plane renders top-level label groups through the map-field editor, so the two type unions have to stay in sync or the UI can emit a shape the server rejects. Client wrappers could not express this. The TypeScript wrapper's updateBankConfig had no entityLabels option at all, so TS consumers could not configure a controlled vocabulary of any type; the Python wrapper typed entity_labels as list[str], which passes through at runtime but no typed caller can satisfy. Both fixed, with the mirrored mapping regression tests the wrappers' parity rule asks for. Closes #4025	2026-09-02 12:51:29 +02:00
Nicolò Boschi	921ae824a4	fix(reflect): fail the run when a retrieval tool fails, and record refused refreshes (#4003) ... A mental-model refresh could replace a document built over months with "I don't have information about that" (#2894). When a reflect tool raised, reflect handed the exception back to the model as a tool result and let the loop continue; the model answered from whatever it had — usually nothing — and that answer was indistinguishable from a run over a bank that genuinely holds nothing on the topic. Non-empty prose, so every emptiness guard on the write path let it through, and the operation was recorded as completed. Reflect now fails the run instead: - A tool that RAISES is an infrastructure failure (the database, the embedder, the reranker), not something the model can retry its way out of, so it raises ReflectToolExecutionError. One failure in a parallel batch fails the run. A tool that RETURNS {"error": ...} for a malformed or unavailable call is the model's own mistake and is still fed back to it, unchanged. - A non-context-overflow LLM error that survives one retry re-raises rather than falling through to a forced final synthesis built on evidence the failed turn never finished gathering. Context overflow keeps its deliberate degradation: that is a prompt-budgeting problem with the gathered evidence intact. - OperationCancelledError passes through untouched on both paths, so a client disconnect stays a 499 (#2122) instead of becoming a 500. The line this draws is between "we could not look" and "we looked and there is nothing there". A retrieval that succeeds and returns nothing is an answer, and still rewrites the document. The refresh re-raises both reflect failures as a typed MentalModelRefreshError (refresh_failed_reflect_error, with reasons retrieval_failed / no_answer), so they reach the operation's typed `details` through the same failure-metadata hook every other refusal already used. Content, structured document and watermark are all left untouched, so the retry re-reads the same window. Every refused refresh — the new reflect-side ones and the existing _preserve_and_fail paths — now also appends a failure record to mental_model_history carrying the reason and the exception. Before this a failed refresh left no trace on the model at all: the History tab kept rendering the last SUCCESSFUL trace as though it were current, and the only record was prose on an async-operation row no mental-model view reads. Retention is applied per kind so a run of failures cannot evict the version history. Control plane: a new Errors tab on the mental-model modal shows those records as a timeline of events — reason, attempt count, time, exception — with a red dot on the tab when there are any. History goes back to being a version browser. Consecutive identical failures collapse into one event (the worker retries each refresh), but a successful refresh between two of them breaks the chain, so separate outages stay separate. Also fixes a metadata bug found while verifying this live: a refresh is retried on the same operation row, and the success path wrote no failure_reason to overwrite the failed attempt's — producing outcome=content_written alongside failure_reason=no_answer. The success now writes it as null explicitly. Claude-Session: https://claude.ai/code/session_01F3i5UVdZRK16AZ9oFQqsg4	2026-09-01 17:07:15 +02:00
Nicolò Boschi	d936d4931c	feat(webhooks): emit X-Hub-Signature-256 and a timestamped signature (#3986) ... Webhook deliveries signed only `X-Hindsight-Signature`, a vendor name for a construction that is byte-for-byte the one GitHub popularised: `sha256=<hex>` HMAC-SHA256 over the raw body. Every receiver therefore needed a Hindsight-specific shim to verify a signature it already knew how to check. Emit `X-Hub-Signature-256` alongside it, carrying the identical value. Same secret, same algorithm, same bytes, so duplicating it grants no new capability to an attacker, and existing consumers of `X-Hindsight-Signature` keep working. Preferred over a per-webhook configurable header name: that would add a config field (plus migration, API, control plane, clients, CLI coverage, docs) to let users type the one string this already sends, and would leave every SDK verifier asking which header the sender was configured for. Two adjacent gaps found while in here: - The body-only signature has no notion of freshness, so a delivery captured off the wire stays verifiable forever. Add `X-Hindsight-Signature-V2` (`t=<unix>,v1=<hex>` over `<t>.<raw body>`, Stripe-style), signed at attempt time so retries re-sign. The timestamp is inside the MAC, so receivers can trust it and reject anything outside a tolerance window. The existing headers keep their body-only meaning — `X-Hub-Signature-256` is body-only by convention and must not be redefined. - `http_config.headers` was spread *after* `X-Hindsight-Event`, so a webhook's custom headers could overwrite the event type a receiver keys off. Spread user headers first and set the Hindsight-controlled headers after, so neither the event type nor any signature can be clobbered. Content-Type stays overridable (some receivers insist on a vendor media type; the body is JSON regardless). Document the whole header set with a verification example, which the webhooks page previously did not cover at all. The two unrelated `skills/hindsight-docs/` hunks are pre-existing generated drift from #3896, picked up by re-running generate-docs-skill.sh. Closes #3207	2026-09-01 12:44:36 +02:00
Nicolò Boschi	b75e941916	fix(worker): rotate slots across banks so bulk ingest stops starving them (#3861) (#3980) ... * fix(worker): rotate slots across banks so bulk ingest stops starving them (#3861) Claiming was a strict global FIFO on created_at, so a bank under sustained bulk ingest held every worker slot for as long as its queue lasted. Measured on a six-bank instance: one bulk bank owned 17 of 17 retain slots in 90.6% of daily samples, and a write to a bank with an empty queue timed out at 300s behind the backlog. The queue drains correctly once ingest stops — this is fairness, not correctness. Deficit round robin, with a quantum of one slot. Every operation costs exactly one slot, so DRR's deficit counter is always zero and drops out; what is left is the round-robin walk. The poller keeps a cursor over the bank id space per schema — one level below the tenant rotation it already had — and the claim takes one row for the first bank sorting after it. Both tiers are one statement: a `rot` CTE (bounded index seek past the cursor) unioned with the `fifo` claim that was always there, joined back and locked with FOR UPDATE OF ... SKIP LOCKED. Same query count as before. 0.10ms against 0.02ms for the bare FIFO claim, at 50k pending rows. The cursor is a *range*, not a set of known banks: the starved bank is by definition one this worker has never claimed for, so only a range can discover it. And `fifo` is what makes the rotation safe at both ends — it is the wrap (once the cursor passes the last bank, `rot` matches nothing and `fifo` claims the whole pool, so the end of a round costs neither a query nor an empty claim, and a single-bank deployment sits in that state permanently), and it is what keeps this work-conserving: a bank alone with work still takes every slot, so nothing is throttled and no slot is held open for an idle bank. Rejected along the way, both measured: an ordering predicate that re-scanned the bank's queue per candidate row (11s at 10k pending), and a separate seek before the claim (correct, but a round trip on every claim, for every schema, on every poll). No new index — `idx_async_operations_bank_status` and `idx_async_operations_bank_created_desc` already serve both branches. Oracle keeps the plain FIFO claim, deliberately: its ROWNUM rewrite for FOR UPDATE + LIMIT applies before ORDER BY, so a rotation there would land on whichever bank is scanned first — under bulk ingest, the one it exists to rotate away from. claim_tasks now returns ClaimedOperations (rows + the rotation's next cursor) rather than a bare row list, so learning where the rotation got to costs no second statement; callers updated. Claude-Session: https://claude.ai/code/session_017ufCz6qrNxn36Stug7ek8A * chore(docs): regenerate the docs skill for the worker-tuning paragraph hindsight-docs/docs/developer/api/operations.mdx is the source; the skill reference is generated from it by scripts/generate-docs-skill.sh, and verify-generated-files fails on the drift. Claude-Session: https://claude.ai/code/session_017ufCz6qrNxn36Stug7ek8A	2026-09-01 12:08:09 +02:00
Ashley Unitt	88f1472e52	docs(docker): reserve shared memory for embedded PostgreSQL (#3896) ... * docs(docker): reserve shared memory for embedded postgres * docs(docker): keep shared memory guidance concise	2026-09-01 12:03:16 +02:00
Nicolò Boschi	c507e70e34	fix(mental-models): skip the reflect loop when the scope holds nothing to read (#3875) (#3943) ... A refresh with nothing in scope is the reflect agent's worst case, not a cheap one. The forced retrieval turns all come back empty, and the evidence guardrail then refuses every `done` call — evidence is exactly what cannot be gathered — so the loop runs to its iteration limit and pays a forced synthesis on top. Creating a knowledge page enqueues its refresh immediately, so a bank created with its default pages spent its whole LLM budget on five worst-case reflects over an empty graph: the budget it needed to ingest the content those pages were waiting for. Ask first whether the model's own flags leave anything to retrieve. The check reads the resolved scope, not the bank: tags/tags_match, tag_groups and fact_types bound which memory units the agent's tools can return, and the window bounds both retrieval tools — open in full mode, the watermark window in delta mode, using the same `updated_at` predicates as the recall arms. With `exclude_mental_models` off, a sibling document with real content is a source (`search_mental_models` applies no time bound, so that holds in delta mode too); one still holding the `Generating content...` placeholder is not. It costs nothing on a refresh that goes on to reflect: the check decides on the `MAX(updated_at)` the refresh already runs for its watermark. That reading now travels out unclamped (`_MentalModelScopeWatermark`), because the clamp that stops a watermark regressing destroys precisely what the check needs. Only a scope with no readable memory pays a query, and only while sibling documents are in reach. The reflect call is gated rather than the function returning early, so the delta legs still run — a retraction is a reason to edit the document by itself. Full mode returns `content_preserved_no_new_facts` before the empty-candidate guard, which raises and would make the worker retry identical inputs; reusing the outcome the delta leg already reports for an empty window keeps this off the API surface.	2026-09-01 11:51:57 +02:00
Nicolò Boschi	78d46a7181	fix(recall): honour min_scores.keyword on every text-search backend (#3882) (#3938) ... * fix(recall): honour min_scores.keyword on every text-search backend (#3882) `min_scores.keyword` was a no-op on four of the six text-search backends, including `native`, the default. A caller asking for `keyword >= 0.30` got rows scoring 0.2 back. `bm25_min_score` was added in #1947 as a VectorChord-specific gate: vchord's `<&>` operator ranks *every* document, so it needed the analogue of native tsvector's boolean `@@` match gate. Default 0, Oracle got it for symmetry, behaviour unchanged everywhere else — correct and complete for that purpose. #2422 then built the public `min_scores.keyword` floor on top of that same parameter and touched no file under `engine/sql/`. From `retrieval.py` the wiring looked finished, but only the vchord and Oracle branches ever read the value; `native`, `pg_textsearch`, `pgroonga` and `pg_search` accepted it and silently dropped it. An internal gate that defaults to off had been promoted to a public per-request floor without the backends being re-audited. - Push the floor into all six backends. pgroonga's `pgroonga_score()` and pg_search's `<schema>.score()` are only valid in the target list, and re-evaluating native's `ts_rank_cd` or pg_textsearch's `<@>` in WHERE would compute the score twice per row (and, for `<@>`, forfeit the index scan the ORDER BY relies on), so those four apply it by filtering the ordered LIMIT slice from the outside. Every arm orders by score DESC, so that keeps exactly the rows an inner predicate would. - Make the floor inclusive. `min_scores` is documented as inclusive and the semantic arm uses `>= min_similarity`, but vchord/Oracle used `>`. The new `bm25_score_gate()` helper resolves the overload: `> 0` at the 0.0 default (the structural match gate #1947 needed), `>=` once a caller sets a floor, which subsumes it. Default behaviour is byte-identical. The second half of #3882 is a documentation bug. "All inclusive, AND-ed" reads as a predicate over each returned result, but `semantic` and `keyword` prune only the arm they name: recall fuses four arms and returns what any of them surfaced, so a result may carry `null` for a stage that did not surface it, and graph/temporal results carry neither. That is deliberate — an intersection would discard the strong single-arm matches hybrid retrieval exists to find. Only `reranker` and `final` are per-result predicates, and they are what a caller wanting abstention should use. Note the two halves interact: once the pushdown is fixed, the union behaviour can only ever surface as a `null`, never as a below-floor number, because fusion copies `semantic` only from the semantic arm and `keyword` only from the BM25 arm. The reporter's `{"keyword": 0.2}` under a 0.30 floor was purely the pushdown bug. `MinScores`, the `RecallRequest` field, both MCP tool descriptions and the recall docs now say exactly that. Tests: `test_bm25_min_score_pushdown.py` asserts the floor reaches the SQL on all six backends, that it is inclusive, and that the 0.0 default is unchanged — SQL-shape assertions, so a new backend branch cannot repeat the omission on a machine with no vchord/pgroonga/pg_search/Oracle available. (`pg_search` gained a configurable function schema on main while this bug was open and would have inherited the same gap.) The backend list is hoisted out of `HindsightConfig.validate()` into `VALID_TEXT_SEARCH_EXTENSIONS` and the test parametrizes over it, so a sixth backend is covered the moment it becomes selectable rather than when someone remembers to update a second copy. Plus DB-level regression tests for the keyword floor and for the per-arm contract. Closes #3882 Claude-Session: https://claude.ai/code/session_011KDT484YujNcBxHzbfzNbk * fix(recall): inclusive keyword floor — update the vchord contract test, harden the regression test CI caught two things the local run could not. 1. `test_db_abstraction.py::test_build_bm25_arm_vchord_honors_custom_min_score` asserted `> 2.5`. That is the old exclusive gate this PR deliberately replaces, so the assertion is now `>= 2.5` with a comment recording why it changed. The test was doing its job; it encoded the behaviour the parameter had while it was vchord's internal match gate. 2. `test_keyword_floor_prunes_in_retrieval` asserted `len(kws) >= 2` so the floor would discriminate. On the three-fact corpus the keyword arm surfaces only one row for "animals", so the guard failed on its own precondition. Reworked to assert the contract without depending on corpus rank spread: a floor above every observed score must leave nothing keyword-scored (before the fix, native returned those rows with their real below-floor scores), and a floor at exactly the top score must keep that row (inclusivity, end to end). Neither a row count nor a score spread is something to assert on here — ranks can tie and the arm may surface a single row. Also: format the floor with `!r` rather than `:g`. `:g` truncates to six significant digits, so a caller echoing a `scores.keyword` value back as a floor could get a literal that rounds up past its own row and silently drops it — the exact round-trip the new inclusivity assertion exercises. Claude-Session: https://claude.ai/code/session_011KDT484YujNcBxHzbfzNbk * test(recall): drop the DB-level keyword-floor test; it asserts an environment property `test_keyword_floor_prunes_in_retrieval` failed in CI twice, on two different preconditions, for the same underlying reason: the BM25 arm surfaces nothing for this module's seeded fixture, so `scores.keyword` is `null` on every result and there is no floor to exercise. No other test in the repo asserts a non-null `scores.keyword`, so nothing else depends on that arm surfacing rows here. `search_vector` is a GENERATED ALWAYS column, so the fixture's raw INSERT does populate it — the cause is somewhere else in the test configuration and is worth a separate look, but it is not this fix. (The arm demonstrably works in a real deployment: the #3882 reporter's own responses carry keyword scores.) Deleted rather than skipped. The guard for this bug is `test_bm25_min_score_pushdown.py`, which asserts the floor reaches the SQL on all six backends deterministically and is what would have caught the original defect; a DB test that cannot observe a keyword score adds no coverage over it. Also fixes a vacuous assertion in `test_retrieval_floors_are_per_arm_not_per_result`: `any(keyword is None)` holds trivially when every keyword is None. It now asserts that not every result carries a score for both floored arms, which is the union property the test is named for. Claude-Session: https://claude.ai/code/session_011KDT484YujNcBxHzbfzNbk	2026-08-31 17:54:20 +02:00
Nicolò Boschi	43f545e9b8	fix(documents): skip the retag cascade when a tags PATCH changes nothing (#3912) (#3931) ... * fix(documents): skip the retag cascade when a tags PATCH changes nothing (#3912) `update_document` never compared the incoming tags to the ones the document already carries — the path was `if tags is not None:`. So a PATCH re-sending an identical tags array, which is exactly what an idempotent tag-normalisation sweep does on every run after its first, paid the full retag cascade for a write that changes nothing. That cascade is not cheap and is not meant to be: it deletes every observation built over the document's memories and then resets `consolidated_at` on each of those observations' OTHER sources. Both halves are required for correctness — consolidation scopes a memory by its tag set, so an observation formed under the old tags is no longer valid, and deleting it strands the co-sourced memories it carried unless they are requeued. The consequence is that the blast radius of one PATCH is the co-source degree of the affected observations, not the document's own memory count, and a sweep multiplies that by its document count. Read the current tags before overwriting them and compare as SETS — consolidation scopes by tag set, so a reordered array changes nothing it can observe. When the set is unchanged, skip the memory-unit retag, the observation deletion, the source requeue and the consolidation submit. A tag set that differs at all still runs the cascade unchanged; tags that could not be read (document absent, or a store record not carrying them) are never treated as unchanged, so a real retag is never silently skipped. Both the SQL and store-owned branches are covered. Also drops a redundant `get_document_record` round-trip on the store-owned path, which the new pre-read already fetched. Tests: a repeat PATCH leaves the observation and every co-source consolidated; a reordered array is not a change; a superset still invalidates; and a real retag stamps `memory_units.updated_at` while a no-op leaves it alone. * chore(docs-skill): regenerate for the documents.mdx re-consolidation note The bundled hindsight-docs skill is generated from hindsight-docs/docs/**; verify-generated-files caught references/developer/api/documents.md drifting from the callout edited in the previous commit. Claude-Session: https://claude.ai/code/session_018HDqrzHgqZqsGc7EDqoTEu	2026-08-31 16:16:24 +02:00
Nicolò Boschi	4b01d02f42	docs: update retain extraction mode documentation ... Document all supported retain extraction modes and regenerate the docs skill references.	2026-08-28 10:37:35 +02:00
Nicolò Boschi	99a319513f	fix(knowledge-pages): move a page's scope onto its mental model, and stop trigger patches resetting it (#3687) (#3755) ... * docs(knowledge-pages): say that a page's tags are a filter, and let the dialog widen it (#3687) A page's `tags` are the scope it is synthesized from, not labels on it, and a tagged page matches with `all_strict` by default: a memory must carry EVERY tag and untagged memories are excluded outright. Nothing said so. The create-page API documented `tags` as "tags that scope which memories the page is built from" with no match mode, and `type:<x>` is documented as setting the page's rendered *type* — so a page created with `["type:runbook", "homelab", "infrastructure"]` silently required all three on every memory, matched nothing, and generated as "I don't have information about this" while a plain recall for the same query returned 81 results scoring 1.087. The server default stays. What changes is that you can now see it and move it: - The control-plane create/edit dialog states the rule under the tags field and, once there is a tag to widen, offers "Also build from untagged memories" -> `trigger.tags_match: "all"`. The edit dialog prefills from the page's stored mode and only sends `tags_match` when the checkbox actually moved, so a page on `any`/`any_strict`/`exact` survives a rename untouched. - `CreatePageRequest.tags`, `UpdateNodeRequest.tags` and `MentalModelTrigger.tags_match` carry the rule in the OpenAPI spec, so it reaches Swagger UI and the generated SDKs; both hand-written wrappers (TS `HindsightClient`, Python `Hindsight`) get it in parity. - A "Tags Are a Filter" section in the API docs works the reported page through, with the three ways to scope one and the `PATCH {"tags": []}` repair. Also corrects a stale claim in the same three places: a supplied `trigger` has merged over the page defaults since #3506, but the docs and both wrapper docstrings still said it REPLACES them and told you to repeat the fields you wanted to keep. Under that text the fix above reads as destructive. * fix(knowledge-pages): move a page's scope onto its mental model, and stop trigger patches resetting it (#3687) Two changes, one subject: where a knowledge page's retrieval scope is edited, and whether editing it destroys the rest of the page's configuration. 1. The page dialogs no longer take tags. `tags` on a page are the scope it is synthesized FROM, and a tagged page matches with `all_strict`: every tag required, untagged memories excluded. The create dialog offered a bare text input for them next to a hint that a `type:<x>` tag "sets the page's type" — so tags typed there to describe a topic silently became a hard filter that matched nothing, and the page generated as "I don't have information about this". A partial copy of the scope controls was the problem, so the copy is gone rather than extended: the page dialogs now take a name and a source query, and the page links to its backing mental model, which already owns the whole scope (tags, tags_match, tag_groups, fact types, schedule). Pages are created reading the whole bank; narrowing one is now a deliberate step taken where the match mode is visible next to the tags it governs. The edit dialog no longer sends `tags` at all. That is load-bearing: the PATCH applies only the keys present, so a dialog that no longer shows the field must not send it, or every rename would clear the page's scope. 2. `PATCH /mental-models/{id}` patches its trigger instead of replacing it. Which matters much more now that it is where pages send people. The route dumped the whole request model and the engine wrote that dict wholesale, so setting one field stamped `MentalModelTrigger`'s own defaults over every field left unset: a page edited there lost `mode: delta`, its observation-only `fact_types`, and `exclude_mental_models`, quietly becoming a from-scratch rebuild that also reflected over its sibling pages. #3506 fixed exactly this, but only on the two page routes. The MCP `update_mental_model` tool drives the same endpoint with a one-key dict (`{"refresh_after_consolidation": ...}`), so it was wiping triggers too. `_merge_page_trigger` is renamed `_merge_trigger` and is now the shared merge for any mental model; a full trigger still replaces, which keeps bank-template import declarative. Verified against a live API: a page created through the UI, opened via its new "Advanced options" link and saved from the mental-model editor, keeps `mode: delta`, `fact_types: [observation]`, `exclude_mental_models` and `refresh_after_consolidation`. Four of the five new regression tests fail without the engine change; the fifth is the guard that a complete trigger still replaces. * fix(knowledge-pages): open the backing mental model in place, and name it in the link (#3687) The link opened a page's scope editor by navigating to the Mental Models tab. Changing a page's scope is part of working on the page, so sending the reader to another tab — losing the page they were reading, and their place in the tree — was the wrong trade for a shareable URL. It now opens the same detail modal in place, on its configuration, and hands off to the very same `UpdateMentalModelDialog` the Mental Models tab uses (exported for it, not copied). Saving re-pulls the tree and the open page, since tags and trigger drive the chips and the freshness line. The label says "Mental model options" rather than "Advanced options": what opens is that model, and the page's scope living on a mental model is exactly the thing a reader needs told. The ?mentalModel= deep link added for the navigation is removed with it — nothing links there now, and a query param nothing produces is a trap for the next reader. * refactor(knowledge-pages): name the options i18n key for what it opens (#3687) The key was still `advancedOptions` after the label became "Mental model options", and the refresh callback asserted `currentBank!` where guarding is free.	2026-08-24 14:06:12 +02:00
Nicolò Boschi	1f5006583c	fix(recall): skip over-budget facts instead of stopping, and never answer a match with nothing (#3688) (#3704) ... The max_tokens filter stopped at the first fact that did not fit the remaining budget, so one long fact evicted every shorter fact ranked behind it, and a budget no fact fits at all (CJK facts of 100-200 tokens against max_tokens=80) returned an empty list — which to an agent reads as "this bank has no such memory". This is the defect #3221 fixed for the source_facts budget in #3419; that fix never reached the fact budget one function over. select_facts_within_budget applies the same rule there — skip, don't stop — and adds the floor that case needs: if not even the top fact fits, it comes back whole and over budget rather than clipped, since MemoryFact carries no truncation flag. max_tokens=0 still means "no facts" (#364). The response surface is unchanged. A "the budget dropped facts" flag was considered and dropped: recall's candidate set carries no relevance cutoff of its own, so on a bank of any size it would be true on nearly every call and tell a caller nothing. The recall log line and trace still report it.	2026-08-24 13:09:04 +02:00
Nicolò Boschi	3de41af867	feat(recall): let callers supply the temporal window instead of parsing it (#3678) ... * feat(recall): let callers supply the temporal window instead of parsing it Recall derives the temporal arm's window by parsing dates out of the query text. A caller that already knows the range it means — a date picker, an agent that resolved "last quarter" itself — had no way to say so, and had to phrase it in English and hope dateparser agreed. Add `temporal_window: {start, end}` to RecallRequest. When set it is used verbatim and the extraction is skipped entirely, which is also the point: that work is pure CPU serialised through a single worker and costs up to ~1.3s on document-sized query text, which is exactly what consolidation and reflect recall with. Naming and wording carry weight here, because the obvious reading of a date range on a search API is "restrict results to this period" and that is not what this does. The temporal arm is one of four retrieval arms: it surfaces memories whose own dates fall in the window so fusion ranks them higher, and the other three arms are untouched, so memories outside the window are still returned. Every description — model docstring, OpenAPI field, MCP tool, both wrappers, control plane, docs — says so explicitly. It does not override `enable_temporal_retrieval`. That per-bank flag gates the arm itself and stays the single switch for it, so a supplied window cannot re-enable an arm a bank turned off. Bounds are inclusive and naive datetimes are read as UTC at parse time, so both ends are unambiguous before they reach a query; a reversed window is rejected at the boundary rather than silently returning nothing. The Rust struct literals in the CLI and the client's doctest have to name the new field or progenitor's generated RecallRequest stops compiling. * feat(control-plane): add the temporal window to the Recall Analyzer The recall UI could not reach the window it now proxies. Adds a Time window row to the Recall Analyzer: two datetime inputs, a Clear button, and a hint that states what the window actually does — ranks memories dated in the range higher, does not hide the ones outside it — since "date range on a search form" reads as a filter otherwise. The two rules live in lib/temporal-window.ts rather than the component so they are testable: a window needs both ends (one alone is an incomplete range, not a half-open filter), and a reversed range is withheld and flagged inline with the Recall button disabled, instead of being sent for the API to reject with a 422. Comparing the raw `datetime-local` strings is exact — they are already YYYY-MM-DDTHH:mm, which sorts chronologically — so there is no Date parsing and no local-timezone reinterpretation between the input and the request. The value is sent with no offset, which the API reads as UTC, matching what query_timestamp already does from this same form; the hint says so. Strings added to all ten locales. * fix(control-plane): reject a reversed window on Enter, not just on the button Disabling the Recall button left the Enter-key handler on the query input calling runSearch() directly. With a reversed range that ran the search anyway and silently dropped the window, which is the failure the inline warning exists to prevent. Guard in runSearch so every entry point agrees, and toast the same message rather than doing nothing visible. * feat(cli): expose the recall temporal window as --window-start/--window-end check-cli-coverage caught that recall_memories gained a request-body field the CLI neither exposes nor exempts. The exemption list is for genuinely complex nested bodies — min_scores' four calibrated floats, tag_groups' boolean tree — and two datetimes is not that, so expose it rather than write it off. Flattened into two flags and recorded as such in the coverage manifest, the same shape `include` already uses. Both ends are required: one alone is an incomplete range, not a half-open filter, and running a recall without the window the caller asked for is worse than refusing. A reversed range is rejected before the request rather than sent for the API to 422, and a datetime with no offset is read as UTC, which is how the API reads one.	2026-08-21 13:02:46 +02:00
Nicolò Boschi	fe5c25d64c	fix(mental-models): store delta documents as verbatim markdown blocks (#3361, #3273) (#3622) ... * fix(mental-models): store delta documents as verbatim markdown blocks (#3361, #3273) A knowledge page could come back with a whole table welded onto one line, in sections no delta operation had named, and never recover. The delta refresh was blamed, but the damage was done one refresh earlier. `structured_content` was only the source of truth on the delta leg. The full leg stored the LLM candidate markdown verbatim in `content` while deriving the structure from it with `parse_markdown`, a typed-block parser that flattened anything its union could not express -- nested lists, list continuation lines, blockquotes, hard line breaks, horizontal rules, HTML, indented code, table alignment, a table row missing an outer pipe. 15 of 16 common constructs lost information, and every loss was a fixed point, so no later refresh could undo it. The two columns disagreed by construction, and the next delta refresh published the degraded one over the whole document. Schema v2 stores each block as a verbatim markdown fragment plus an id. Nothing parses a table, so nothing can flatten one. `parse_markdown` is deleted; `split_markdown` replaces it and recognises only ATX headings and blank lines, both fence-aware, which makes it lossless -- asserted as a property over a 26-case corpus of exactly the constructs v1 destroyed. Blocks are now addressed by id rather than by index (#3273). An index has to be counted by the model, and an off-by-one lands in range, silently overwrites an unrelated block, and is recorded as a success. An id is copied, not derived; one that does not resolve -- or that names a block in a different section -- is skipped and reported. Operation payloads are plain markdown strings, so the model no longer has to emit a typed block union either. `content` is now always the render of `structured_content`, on both legs, so the two can no longer drift apart. Also here: - `parse_llm_json` escapes `\n \r \t \b \f` inside JSON string values instead of blanking them. A model writing a markdown table into a string often forgets to escape its line breaks, and replacing them with spaces delivered the table already collapsed. Other control characters keep the previous treatment. - A model that adds a table row as its own block would render a broken table, so the prompt asks for `replace_block` and bare rows landing directly after a table are folded into it. - Migration `d1e2f3a4b5c6` clears v1 blobs. They are a lossy projection of the row's own `content`, so there is nothing to convert: the next refresh re-imports the structure from the markdown, losslessly. `content` is untouched. - The Gemini eval fixture pinned `gemini-2.0-flash`, which the provider has retired (404), so the whole eval class was dead. Verified against a real model: `test_document_survives_many_delta_rounds_intact` runs five delta rounds feeding one new fact each, asserting after every round that content is the render of the structure, that no line welds a table separator to other cells, that sections no operation named are byte-identical, and that a section never named across the whole run is unchanged at the end. Run four times, 20 real rounds, green. It is what caught the orphan table row. * fix(mental-models): write the structure whenever content is written `create_mental_model(content=...)` and `create_knowledge_page` inserted the markdown and left `structured_content` NULL, so a model authored as markdown had no structure until its first delta refresh -- and that refresh was then the one to derive it, silently reshaping a document nobody had asked it to touch. `update_mental_model(content=...)` was worse: it could set the markdown while leaving the *previous* document's structure in place, so the two columns described different documents until the next refresh papered over it. Nothing enforced the pairing; the refresh path just happened to pass both. Both writes now go through `canonical_document()`, which splits the authored markdown and hands back the structure together with its render. The insert stores both, and the update derives the structure whenever a caller supplies content without one. A refresh still passes both explicitly -- there the structure is authoritative and the markdown is already its render -- and is untouched. The derivation is hoisted above the embedding computation so the embedding, the history snapshot and the UPDATE all see one text. `content` is therefore the render of `structured_content` from the first byte rather than from the first refresh, which is what the assertion churn in this commit is: authored markdown now comes back canonicalised, so a document that was stored as "v1" reads back as "v1\n". Also makes the migration test rerunnable: a pg0 instance survives between runs and alembic will not replay a migration on a DB already stamped past it, so seeding into it would have left the rows untouched and the test asserting nothing. * feat(reflect): answer with a document, render the markdown from it The mental-model refresh asked the agent for markdown and worked out the document's structure by reading that markdown back. Reading LLM markdown back is where #3361 destroyed tables, and it is unnecessary here: the refresh knows it is producing a document, so it can ask for one. `done()` gains a document mode. Instead of an `answer` string it takes a `document` -- an ordered list of sections, each with a heading, a level and its blocks -- and the markdown that gets stored and shown is rendered from it. The model no longer writes the markdown that gets persisted, and nothing parses markdown to find out what the model meant. `answer` is not merely discouraged in that mode, it is absent from the schema, so there is no escape hatch back to prose. The shape is deliberately flat -- an array of sections holding arrays of block strings, no unions. A tool schema goes to the provider verbatim and not every provider accepts `oneOf` (Gemini rejects it), and a shape the model can fill without thinking is one it fills correctly. `document_from_sections` is tolerant, because a tool call is still model output: a missing heading, a `##` the model prefixed anyway, an out-of-range level or a non-string block is coerced rather than rejected. A block holding several blank-line-separated fragments is split into one block each, so the document keeps the granularity delta operations address even when the model packs a whole section into one string. Downstream is unchanged: the rendered markdown still flows on as `text`, so structured-output extraction, the length rewrite and the HTTP response all behave as before. The one place the two could drift is the length rewrite, which edits the text after the fact -- there the structure is re-derived from the rewritten markdown, which is lossless and keeps the invariant that the stored text is exactly what the stored structure renders. Splitting markdown is now only an import path: a model created from authored markdown, a restored export, or a run that produced plain text anyway (a provider that dropped the tool call, the iteration-limit answer). Verified against a real model: all five `hs_llm_core` refresh evals pass with the agent emitting structure, including the five-round stability run. The ordered list that a previous run had rewritten now survives untouched, and the new table row lands inside the table rather than beside it. * test(benchmarks): compare two builds on how a document survives being edited Neither half of "is the new pipeline better" was measurable before this. The unit tests prove the mechanics in isolation and the refresh evals prove one build behaves, but nothing compared a build against another one on the thing that actually broke: a document rewritten by an LLM over and over. The harness talks HTTP only, so the same code drives a server built from any revision. That is what makes an A/B possible without a feature flag inside the code under test: run it once per build, compare the two artifacts. Every document from every round is stored, and metrics are recomputed at comparison time, so sharpening a metric costs nothing instead of another few hundred LLM calls. Two things it measures, deliberately separately. Structural, no LLM: collapsed tables (the detector from #3361), rows, nesting, hard breaks, fences and quotes lost, sections that drifted with no operation naming them, plus pipeline health and latency. Damage is counted only in sections no operation named -- a refresh that rewrites a section it targeted may legitimately restructure it, and scoring that as corruption would punish the model for doing its job. Content, judged: each round declares what must be true afterwards and what must no longer be stated, checked one claim at a time so a miss points at a fact rather than at a score; plus a blind pairwise preference between the two builds' final documents, judged in both orderings so position bias cannot decide it. Three cases, and the third is the point. `api-reference` carries every fragile construct; `onboarding-playbook` carries none, because a change that fixes tables while degrading ordinary prose is not an improvement and that is where it would show; `release-runbook` puts a table with a missing outer pipe in a section the fact stream never touches. Both details are load-bearing: a well-formed table never triggered the bug, and a model asked to edit a malformed table tends to rewrite it correctly, repairing the damage before it can be measured. The reported failure was in sections the operations never named, where nothing could repair it. Token usage is not reported. The stored reflect_response does not carry it, and a column that is always zero reads as "this is free" rather than "this is not measured here". * test(benchmarks): harden the judging, and say what the A/B measured Running the benchmark against main and the branch turned up three problems in the benchmark itself, all of which would have made its verdict untrustworthy. The runner slept a fixed interval instead of awaiting the async operations it submitted, so its first results were five rounds of "Generating content..." scored as though they were documents. Retain, create and refresh are all submit-and-poll; it now polls, and refuses outright if the seed it is about to measure is still a placeholder. Damage was attributed to the whole document rather than to the sections nobody asked to change, which scored a model deliberately restructuring a section it targeted as if the machinery had corrupted it. Damage is now measured only in untouched sections, and the metrics are recomputed from the stored documents at comparison time, so this sharper reading could be applied to results already collected instead of paying to re-run them. A single judge call decided each claim, and one pedantic reading moved a build's score: a document describing an operation as "synthesises stored memories" was scored as not supporting "answers questions over stored memories" — the same operation, in the wording the source fact itself used. Claims are now decided by majority of three, and that claim was rewritten to test the fact rather than one phrasing of it. A claim a correct document can fail measures the corpus, not the pipeline. The report prints mean document length beside the preference column, because judges favour longer documents and a preference that tracks that column should be read sceptically rather than counted. The prompt change is the finding that landed back in the product: stating a document's structure is more clerical than writing prose, and the model got terser at it — measurably shorter documents that the judge liked less. Document mode now says plainly that the structure is the shape of the answer, not a budget for it. Baseline recorded in baseline_report.json: 45 refresh rounds per build from identical seeds. main lost 3 tables to collapse, 9 table rows, 6 levels of list nesting and 5 hard line breaks across 6 damaged rounds, and drifted 14 sections nobody had named. The branch lost nothing and drifted nothing. Content came out level — 100% recall and zero stale claims on both sides. * fix(mental-models): give the delta leg the document's own token budget `max_tokens` was enforced in exactly one place: a rewrite of the *synthesis* answer when it came back longer than the budget. In delta mode that answer is only context for the operations call and never becomes the document, so the document that actually gets stored was never measured against the budget at all. A delta refresh only adds. The document-evolution benchmark measured ~20 tokens of growth per round across 45 rounds, monotonic, which crosses the 4096-token knowledge-page default after a couple of hundred refreshes — and knowledge pages refresh after every consolidation. The configured budget was quietly ignored for the entire life of a page after its first full build. Truncating the document here would delete knowledge nobody asked to delete, so the budget is stated instead: the delta call is told the document's current size against its budget, and when it is over, asked to make room with the same operations it uses for everything else — on content that is superseded or duplicated, never by dropping the facts it is integrating and never by summarising a section that is still current. Below 80% of the budget nothing is said at all. Every refresh records document_tokens and document_budget, and going over adds a warning, so a page that keeps growing is visible rather than merely large. Also closes the one path where the model still wrote markdown that got stored: the over-budget trim. In document mode it is now asked for a document, so the structure survives the trim instead of being re-derived from prose the model wrote. A response that is not JSON falls back to the previous split, which is lossless — the worst case is the old behaviour, not a lost answer. The real-LLM eval is the part that could not be mocked: told that a document is over budget, does a model reclaim space or append anyway? It drops the twelve archived sections and keeps the current process and the checklist — 434 tokens to 39 against a 200-token budget, stable across four runs. The assertions check where the space came from, because getting under budget by deleting current content would pass a naive shrink check and be worse than going over. * test(mental-models): audit every trigger flag on the delta leg `max_tokens` looked wired up — read from the model, passed to reflect, enforced by a rewrite — and was still ignored for the document that actually got stored, because in delta mode the thing it capped never becomes the document. Reading the code is how that was missed; nothing asserted the flag at its destination. So every flag is now exercised through a real delta refresh and asserted where it lands: retrieval options at the reflect call, document options in the delta prompt or the persisted row. Full mode is covered by the surrounding modules; this is the leg where a flag goes to die. Fifteen flags checked. Fourteen were already honoured. The audit pins them so a future change to either leg cannot quietly drop one: - retrieval: fact_types, exclude_mental_models, exclude_mental_model_ids, the model's own id (a model must not feed on its previous version), include_chunks, recall_max_tokens, recall_chunks_max_tokens, the model's tags, tags_match - document: max_tokens, response_schema (extracted from the merged document, not from reflect's delta-only answer), keep_trace, mode - and the whole trigger surviving a create/read round trip, since a flag that does not persist is not honoured either The fifteenth is documented behaviour that reads as a bug from outside, so it is pinned as intended rather than "fixed": `tag_groups` overrides flat tags entirely, dropping the model's own tags and forcing `tags_match` to `any`, because each group carries its own match mode. A `tags_match` set alongside groups is deliberately not forwarded. The default for a tagged model with no `tags_match` is `all_strict`, not `any` — a model scoped to tags must not widen its own scope by default. Scheduling flags (`refresh_cron`, `refresh_after_consolidation`) are honoured outside the refresh executor — the maintenance loop and the consolidation hook — and keep their existing coverage there. * test(benchmarks): type the structural summary, and cover the flag main just added Two findings from reviewing the branch against a fresh main. `_structural_summary` returned a raw dict of known keys, which the project standards forbid for exactly the reason it bit here: the report read it with `summary["rounds"]` and nothing would have caught a renamed metric until the table rendered wrong. It is a `StructuralSummary` model now, and the side-by-side table renders `model_dump()` so a metric added later appears without being listed twice. Main added a sixteenth trigger flag while this branch was in flight (`min_refresh_interval_seconds`, #3621). It gates automatic refreshes rather than shaping one, so it is honoured in the submit path and covered there — but the round-trip test enumerates the whole trigger on purpose, because a field that round-trips as None looks like the flag being ignored rather than like a storage bug. Adding it keeps that list exhaustive. * fix(mental-models): teach the retraction prompt the schema it emits into CI caught what the rebase brought: main added an unsay pass (#3618) whose prompt documents the operation vocabulary a second time, in prose, and it still told the model to say `{"op": "remove_block", "section_id": "...", "index": N}` with typed `block` payloads. Under the id-addressed schema those ops fail validation and are dropped, so a retracted fact would keep being stated and nothing would say why — the unsay feature silently doing nothing. The prompt now describes the schema it actually emits into: blocks addressed by `block_id`, block payloads as markdown strings, and the note about emitting removals in descending index order deleted, because ids do not shift when a sibling is removed and telling a model to order by position invites it to think in positions again. Guarded structurally rather than by one more test. The op vocabulary is written down twice — Pydantic models the applier validates against, and prose in each system prompt — and a test for the prompt that drifted does not exist by construction, since it is the one nobody wrote. `test_delta_prompt_schema_parity` asserts over *every* prompt carrying an operations vocabulary that each op exists, that no shape names a field the schema rejects, that no v1 typed block survives, and that blocks are addressed by id; plus a check that a prompt asking for `{"operations": [...]}` cannot be left off the list. Reverting the prompt fails three of them. The rest is the same schema change reaching tests the rebase brought in: canned ops in the outcome matrix moved to `text`, the retraction tests resolve a real `block_id` out of the document the prompt shows them (which is what a model does, and what a hardcoded index cannot express), the stale `parse_markdown` monkeypatch points at `structured_document_from_stored`, and four assertions on authored content now expect the canonical render.	2026-08-20 11:35:32 +02:00
Nicolò Boschi	2b54a1d3cd	feat(mental-models): a minimum interval between automatic refreshes (#3480) (#3621) ... * feat(mental-models): a minimum interval between automatic refreshes (#3480) A bank with several models on refresh_after_consolidation paid for a full agentic rebuild of every stale model on every retain, however small: a three-fact retain cost ~250k tokens, and ordinary conversational traffic reached ~11.6M tokens a day, ~90% of it refresh work. min_refresh_interval_seconds puts a floor on how often that may happen, configurable at all three levels and defaulting to 0 everywhere, so nothing changes for anyone who does not set it: HINDSIGHT_API_MENTAL_MODEL_MIN_REFRESH_INTERVAL_SECONDS server mental_model_min_refresh_interval_seconds tenant / bank trigger.min_refresh_interval_seconds one model The per-model value wins, including an explicit 0 — that is how one model that does need to stay current opts out of a floor its bank imposes. The refresh is parked, not skipped. The handler raises DeferOperation before any recall or LLM work, which leaves the operation queued with next_retry_at set to the end of the window; every trigger that fires while it waits folds into it (#3487), and the eventual run reads everything that accumulated. Skipping the submit instead would drop the work: nothing re-checks the model afterwards, so memories that arrived during a burst would stay unsynthesised until some later, unrelated trigger happened to land outside the window. Only the triggers nobody asked for individually are rate-limited — the after-consolidation flush and the cron scan. An explicit refresh asked for one now and gets one; when it folds into a parked refresh it also releases the park, since otherwise "refresh now" would silently inherit somebody else's wait. Releasing means dropping the automatic marker as well as next_retry_at: left in place, the next claim would park the operation all over again. Built on existing machinery rather than new scheduling: next_retry_at is already indexed and honoured by every claim query on both dialects, and DeferOperation already means "requeue later, not a failure". No new table, no new sweep, and nothing dialect-specific — the payload edit is read-modify-write rather than jsonb `-`, which Oracle's SQL translation does not rewrite. The control-plane trigger form carries the field because update_mental_model replaces the trigger wholesale: without it, editing a model in the UI would silently wipe the interval (the #3549 bug class). The TypeScript wrapper client needed it explicitly too — unlike the Python wrapper, which splats the whole trigger dict, it maps a hand-picked list of trigger fields. * feat(control-plane): a Mental Models section, and show when an operation is parked Two gaps found while testing the refresh floor on a real bank. The bank-level setting had no home in the UI at all — it was reachable only through the config API. It now has its own "Mental Models & Knowledge Pages" section after Reflect: it governs the models and the knowledge pages backed by them rather than consolidation, and the section is where the rest of that surface will go. The operations list showed a parked refresh as plain "pending", which reads as stuck. The dataplane has always returned next_retry_at, but no control-plane type declared it and nothing rendered it, so the one piece of information that explains the wait was invisible. A pending operation held until a known time now renders as "Waiting" with that time, in the list and the detail panel; the badge keys off status AND a future timestamp, because next_retry_at is not cleared when the task finally runs. That last point also corrects the API docs, which claimed next_retry_at is "always null for completed tasks" — a completed operation that was deferred still carries the timestamp of its last wait.	2026-08-19 16:24:38 +02:00
Nicolò Boschi	a5d00660b5	fix(knowledge-base): answer page staleness per scope, not from the bank watermark (#3291) (#3589) ... The knowledge tree, the mental-model list and the control-plane stats card derived staleness from one bank-wide write watermark: any write flagged every page that had not read it. Only an in-scope write can move a page's own watermark, so a page whose scope went quiet stayed flagged indefinitely while the refresh gate — which does check each page's own scope — correctly refused to refresh it. The reporter measured 24 of 29 pages flagged for hours against 0 of 29 by the exact check. The approximation existed because the exact check cost a scan of the bank's memories per page: memory_units had no index involving updated_at, so `WHERE bank_id = $1 AND updated_at > $2` could not stop early on the common negative answer. Add (bank_id, updated_at DESC) and that cost goes from 10.7ms (tagged page, fresh) and 29.8ms (untagged model) to 0.046ms and 0.021ms on a 400k-row table. With the index the exact answer is affordable, so those surfaces now ask it, batched: any_memory_updated_since_batch answers a whole bank in one query per scope shape (30 pages in 0.2-0.5ms). Reflect's search_mental_models, which ran the scoped check per model inside every reflect call, uses the same batch. The watermark shortcut is kept for callers holding a live watermark, but the polling surfaces no longer pass the 60s-cached one — it can be older than a model's own last read and would prove a freshness that is not there. _bank_write_watermark had no callers left and is removed. Staleness still asks only what has been *written*; deletions leave no write behind and remain invisible, which the API and docs now state outright.	2026-08-19 14:15:02 +02:00
Nicolò Boschi	6582e26ef9	feat(mcp): expose knowledge-base CRUD as native MCP tools (#3486) (#3611) ... The knowledge base was reachable only over HTTP, so an MCP client had to fall back to a second integration path to browse or maintain it. Register the seven agent-facing operations as native MCP tools, with the same bank scoping, tenant auth and operation-validator behaviour as the existing tools: get_knowledge_base_tree, search_knowledge_base, get_knowledge_page, create_knowledge_folder, create_knowledge_page, update_knowledge_node, delete_knowledge_node export_knowledge_base stays HTTP/CLI-only — it returns the whole bank as one markdown bundle, which does not belong in an agent's context window. Two places where the MCP surface cannot mirror the HTTP one, both commented at the call site: - MCP arguments cannot express an explicit null, so update_knowledge_node reads parent_id="root" as "move to the top level". Node ids are prefixed kf-/kp-, so the literal cannot collide with a real folder id. - The page refresh trigger is flattened to a single refresh_after_consolidation flag, matching how create/update_mental_model already expose it. It is sent as a patch, so an unstated flag leaves the knowledge-page defaults (delta mode, observation-only) intact — the regression #3506 fixed. get_knowledge_page returns the rendered markdown document once instead of the HTTP body+markdown pair, which would double the tokens for no new information. search_knowledge_base clamps limit instead of rejecting it: an agent that asked for 500 pages wants results, not a 422. Also adds a structural guard that the three hand-maintained tool allowlists (_ALL_TOOLS, register_mcp_tools()'s default set, and the single-bank set in create_mcp_server) agree with what is actually registered — a name added to one but not the others silently drops the tool from the endpoint.	2026-08-19 11:58:18 +02:00
Nicolò Boschi	df8ac42b52	fix(memory): add resolve_entities flag to update_memory and retain (#3576) ... * fix(curation): resolve edited entity names exactly, not fuzzily (#3479) update_memory ran the caller's entity names through the same fuzzy resolver retain uses, so an entity name was a *guess* to be reconciled against the graph rather than an instruction. Name identity is worth at most 0.5 of the 0.6 match threshold, while co-occurrence (0.3) and recency (0.2) make up the rest, so a similar-but-wrong entity that is well connected to the other names in the same edit outscores the one the caller actually named — with a 200 and no warning. Curation now resolves exactly: an existing entity is reused only when its canonical name matches case-insensitively, any other name creates its own entity, and same-batch names are never merged with each other. Retain keeps fuzzy resolution, which is right for names that came out of extraction. The exact path skips the trigram/UTL_MATCH probe and the co-occurrence fetch entirely and reuses the existing find-or-create pass, so it is dialect-agnostic and strictly less work than the fuzzy one. * fix(curation): add entity_resolution_mode to update_memory, default fuzzy Make the exact/fuzzy choice the caller's, rather than changing what an edit does. `entity_resolution_mode` defaults to "fuzzy" — retain's behaviour, so every existing caller is unaffected — and "exact" opts into literal matching for hand-authored corrections. Plumbed through the HTTP request model, the MCP tool, the control-plane proxy route and its client, with the engine validating the value for direct callers. The control-plane memory editor sends "exact": a person typing an entity list into the admin UI is naming the entity they mean. * refactor(curation): make the flag a boolean, resolve_entities Replaces the entity_resolution_mode enum with a plain boolean on update_memory. `resolve_entities` defaults to True — retain's behaviour, so existing callers are unaffected — and False takes the submitted names literally. Carries the same change through the resolver, which now takes `fuzzy_matching` rather than a mode string. The engine's value guard goes away with the enum: a bool needs no validation, so the invalid-value test goes too (the HTTP boundary still 422s a non-boolean, which the HTTP test covers). * feat(retain): honour resolve_entities for caller-supplied entities too Same flag, same default, on the retain item — a caller passing explicit entity names there has the same exposure as one correcting a memory: a name close to an existing entity can be matched onto it and quietly replaced. Retain resolves caller-supplied and LLM-extracted names in ONE batch, so a per-batch flag would have turned resolution off for the extractor's names too and filled the bank with near-duplicate entities. The flag is therefore carried per mention: extracted names always resolve, supplied names follow the item's flag, and a supplied name the extractor also produced keeps the caller's intent. The in-batch dedup pass (#3107) skips the literal names for the same reason. Also renames the resolver's `fuzzy_matching` parameter to the per-mention `resolve` key, so one name is used end to end. * fix(clients): carry resolve_entities through the maintained wrappers Code review found two gaps the generated SDKs hide. The TypeScript and Python convenience wrappers rebuild each retain item field by field, so `resolve_entities` was silently dropped for every wrapper caller — the same class of gap #2975/#3042 closed for the mental-model methods, and here it would have quietly restored the substitution the flag prevents. Both wrappers now forward it, with mapping tests on each side. Intake also lost the flag when normalization collapsed two spellings into one: entity_processing dedups caller-supplied against extracted names on the RAW text, so a caller's literal "Acme Corp" and the extractor's "Acme\nCorp" both reach _prepare_entities_for_resolution and only merge there. Keeping the first entry verbatim dropped the caller's resolve=False with it; the merge now keeps the stricter flag. * fix: build the Rust CLI, and keep pg_trgm detection on empty batches Two CI breaks from the retain change. MemoryItem gained a field, and the CLI builds it with a struct literal, so every Rust job failed to compile. The CLI supplies no entities, so `true` (the server default) is the right value there. The skip-the-probe guard also fired on an *empty* batch — `any([])` is False — so _resolve_entities_batch_impl returned before the pg_trgm auto-detection that hangs off the strategy dispatch. Only shortcut when there is data and none of it resolves. * fix(rust): add resolve_entities to the remaining MemoryItem literals The first pass only fixed the CLI's src/ literal — `cargo build` does not compile test targets, so the ones in hindsight-cli/tests/integration_test.rs and hindsight-clients/rust/src/lib.rs went unnoticed until CI. Verified with `cargo check --all-targets` in both crates this time.	2026-08-18 16:29:33 +02:00
Nova Lux	19318ac088	fix(engine): drop null metadata values at retain and recall (#3209) (#3531) ... * fix(engine): drop null metadata values at retain and recall Retain accepts arbitrary JSON metadata; a null value (e.g. {"ocr_engine": null}) stored verbatim made every read path that returns MemoryFact fail dict[str, str] validation — recall, consolidation, and mental-model refresh errored for the affected bank. Normalize on both ends: RetainContent drops null-valued keys at construction (canonical storage) and MemoryFact.parse_metadata drops them for legacy rows, preserving the existing string coercion for other non-string values. Closes #3209 * fix(engine): normalize null metadata on the delta and export paths too (#3209) Follow-up to the retain/recall normalization on this branch, which left two gaps. Delta retain never went through RetainContent: all three call sites of update_memory_units_metadata_and_tags pass the raw retain_params bag straight to the UPDATE, so a re-retain left the units it preserved carrying null-valued keys while the units re-extracted beside them did not. Drop the nulls inside that storage function — the one chokepoint every delta path goes through — leaving documents.retain_params holding the caller's input verbatim. Bank export was the other read path that validates stored metadata as dict[str, str]: TransferFact is built directly from the row, so a bank already holding a null (or a raw integer, e.g. {"original_id": 348}) failed export with the same ValidationError the issue reports — locking an operator out of the one operation that gets them off the bad data. It now applies the same read contract as recall. Both rules now live in engine/metadata_utils.py rather than being spelled out at each site, and RetainContent normalizes an explicit "metadata": null to {} so the field always matches its declared type. Regression coverage on a real database: test_delta_retain_drops_null_metadata_values walks a full retain, a metadata-only delta and a partial delta with surviving units; test_export_tolerates_legacy_null_and_numeric_fact_metadata exports a bank whose rows were poisoned before the fix existed. Both fail without their respective fix. --------- Co-authored-by: Nova Lux <NovaLux12@users.noreply.github.com> Co-authored-by: Nicolò Boschi <boschi1997@gmail.com>	2026-08-18 15:19:11 +02:00
Nicolò Boschi	d69c53739c	fix(mental-models): keep at most one queued refresh per model (#3487) (#3550) ... * fix(mental-models): keep at most one queued refresh per model (#3487) A bank whose refresh queue drains slower than it fills accumulated one refresh_mental_model operation per model per consolidation round — 12k pending operations covering 259 models, ~45 identical copies each, every copy a full recall + LLM refresh when it eventually ran. The in-flight guard existed but was opt-in per call site, so any enqueue path that did not ask for it (and every path before #3411) piled up copies. Make the floor structural instead: a submit for a model that already has a *queued* refresh always folds into it and returns that operation's id. Nothing is lost — a refresh carries no per-request options and the queued one has not started, so it still reads whatever the caller just changed. skip_if_in_flight now only widens the guard to an already-*running* refresh, which an explicit refresh must not fold into: it may have read the model before the caller's edit. The check moves out of the INSERT and back in front of it, where the bank-row FOR NO KEY UPDATE lock (held for the rest of the transaction) already serialises submits for the bank and makes check-and-insert atomic. That also makes it work on Oracle: the previous INSERT ... SELECT ... WHERE NOT EXISTS is a FROM-less SELECT there, and its bind-parameter JSON key was never rewritten to JSON_VALUE, so since #3411 every after-consolidation refresh submit raised on Oracle and was swallowed as a warning. * test(mental-models): force the submit race in the dedupe concurrency test (#3487) The eight-way concurrent submit test passed with the bank-row lock removed — asyncio happened to run each short transaction to completion before the next, so it never actually raced. Stall every in-flight lookup before it returns, so all eight submits would sit between their lookup and their INSERT at once. With the lock it still queues one operation; without it the same test inserts eight.	2026-08-18 10:46:54 +02:00
Nicolò Boschi	8fbdc6bf79	fix(mental-models): last_refreshed_at records the refresh, not the source watermark (#3538) ... A refresh persisted its source-data watermark into last_refreshed_at, and that watermark is clamped so it never regresses. On a model whose scope gained no new memories the max IS the stored value, so the refresh wrote it back over itself: the document was rewritten, the timestamp never moved, and a client asking "have I already refreshed this?" refreshed it again on every tick — one reporter drove ~6,000 refreshes/day against an intended ~350 for four days. Split the two meanings the column carried: - last_memory_seen_at (new) takes over the watermark. Staleness, the delta window, the knowledge-tree flag and is_stale all key off it, so refresh behaviour is unchanged. - last_refreshed_at reverts to wall-clock, stamped on every refresh that completes — including one that found nothing new and preserved the content. A failed refresh stamps neither, so a retry re-reads the same window. The migration backfills the new column from last_refreshed_at, which today holds the watermark, so the copy is lossless and no bank changes staleness on deploy. BREAKING (semantics, not schema): a client following the v0.9.0-documented rule of comparing last_refreshed_at against last_memory_write_at must switch to last_memory_seen_at, or it will read models as up to date that are not. This reverts semantics that #2866/#2878 changed four weeks ago; the field was wall-clock from inception until v0.8.5. Also surfaces mental_model_id on the operations list — refresh operations return document_id: null and the list carries no result_metadata, so it could not say which model an operation refreshed.	2026-08-17 12:47:10 +02:00
Nicolò Boschi	681a79e6b4	fix(graph-maintenance): drive the entity prune off a queue, not a bank-wide sweep (#3222) (#3409) ... * fix(graph-maintenance): drive the entity prune off a queue, not a bank-wide sweep (#3222) The graph_maintenance job's Pass 2/3 were two bank-wide statements re-evaluated on every invocation, whether or not anything had changed: the orphan-entity prune probed once per entity in the bank, and the stale-cooccurrence prune evaluated an INTERSECT per cooccurrence row in the bank. Their cost tracked the size of the bank rather than the size of the delete, so past a few million rows neither could finish inside asyncpg's 60s command timeout. The job then failed on every run with a bare TimeoutError, forever, on exactly the banks that most needed it — and, because db_utils treats a timeout as transient, re-ran the doomed statement nine times per attempt, holding a worker slot for ~10 minutes each time. Both prunes are now driven by `entity_maintenance_queue`, filled inside the deleting transaction the way `graph_maintenance_queue` already is for the relink pass. A run claims a bounded batch of candidate entities, prunes what is genuinely dead, and commits — so the cost is O(delta), and the work already done survives whatever stops the run. Measured on a dense fixture (100k entities, 1.5M unit_entities, 2.86M cooccurrences, hub entities holding 150-400 postings): bank-wide orphan prune 9.6s → batch of 50: 15ms bank-wide cooccurrence prune >11 min → batch of 50: 2.0s (cancelled; ~1.7ms per pair over 2.86M pairs) Also: * A wall-clock budget for the whole job. Both passes commit per batch, so exhausting it is not a failure — the run reports `queues_drained: false`, logs it, and chains a follow-up (under a real queue; a synchronous backend would recurse instead of schedule). Large backlogs converge over runs. * The scoping predicate is a UNION of the two endpoint columns, not `entity_id_1 = ANY(...) OR entity_id_2 = ANY(...)` — that OR is the #3387 shape and cannot be driven from either index. * Every site that removes units or replaces entity postings now enqueues candidates: document delete, single and bulk memory delete, curation edit/invalidate, document re-ingest, and the delta-retain chunk cascade. * The migration seeds the queue with every existing entity, so garbage a bank accumulated while its sweep was failing is still reclaimed — incrementally, a bounded batch per run, instead of in one statement that cannot finish. * fix(graph-maintenance): compose the queue-scoped prune with the set-based staleness check Rebase reconciliation with #3408, which landed the same statement while this was in review. staleness against a set of live pairs built once, instead of a correlated INTERSECT re-scanned per row — removing the (rows judged) x (hub degree) product. That is the better predicate, and it composes with the queue scoping rather than competing with it: `live` is now seeded from the *claimed candidates'* units instead of the whole bank's. Correctness holds because every pair being judged has a candidate as an endpoint, so any unit still grounding one of those pairs references a candidate and is in the seeded set. Measured on the dense fixture (100k entities, 1.5M unit_entities, 2.86M cooccurrences): bank-wide, set-based (#3408 as merged) did not finish in 10 min batch of 50, per-pair INTERSECT (mine) 2.0 s batch of 50, composed 16-65 ms The batch-size rationale is updated to the new numbers; 50 still holds, now with three orders of magnitude of margin instead of one. #3367's hub/bank- scoping regression test is kept, adapted to seed candidates. Also re-chains the migration onto d9c1a7b4e2f6, which took the same parent on main and would otherwise leave two alembic heads. * fix(graph-maintenance): restore the review fixes without the birth-time enqueue Drops the "queue every entity at creation" change and keeps the rest of the review round (dataclass pass results, the Oracle IN-list chunking on the by-unit enqueue, the per-site enqueue tests, the migration-seed test, the budget's follow-up-chain test, the stale-comment sweep). The birth-time enqueue existed to reclaim an entity created in retain's Phase 1 whose Phase-2 link never landed. It is not worth what it costs: such a row is a single entry in the registry with no postings and no cooccurrences, and #2662 exists because the retry is *supposed* to adopt it — Phase 2 reasserts resolved parents under FOR KEY SHARE precisely so a pruner cannot delete one out from under it. Pointing the pruner at every freshly created entity leans on that race for a leak that is one row wide. #3408 landing the set-based predicate is what made the trade obviously bad: the expensive half of this job was never those rows. Entities created but never linked are therefore no longer proactively reclaimed. The migration's one-time seed still clears the population a bank has already accumulated. * fix(graph-maintenance): don't backfill the entity queue on upgrade The migration seeded one queue row per existing entity so a bank could reclaim what it stranded while its bank-wide sweep was failing. That is the wrong trade: the INSERT runs inside a migration at API startup, so a large deployment pays a slow upgrade writing a row per entity, and then a prune check for every one of them — a self-inflicted backlog to collect rows that cost the bank nothing. The queue now starts empty and fills from real deletes. Historical strays stay until something touches them; they are single registry rows with no postings and no cooccurrences. The migration test pins the two properties that are easy to lose later: the upgrade enqueues nothing, and the composite key collapses overlapping deletes into one row (which is also what the #3034 locking upsert conflicts on).	2026-08-12 16:14:19 +02:00
Nicolò Boschi	8391296af5	fix(recall): fill source_facts budget in rank order and flag truncation (#3221) (#3419)	2026-08-12 10:08:02 +02:00
Sanderhoff-alt	82859af02b	docs(reflect): clarify tag-scoped directive behavior (#3038) ... Explain how tags, tags_match, tag_groups, and directive isolation interact across REST, MCP, versioned docs, and agent skills. Clarify the different defaults used by reflect and directive listing, then regenerate OpenAPI and supported client artifacts.	2026-08-12 07:41:47 +02:00
Nicolò Boschi	00b520e592	feat(transfer): async document export (#3321) (#3340) ... * feat(transfer): async document export (issue #3321) The synchronous GET /banks/{id}/document-transfer loaded the whole bank into memory, held a DB connection for the full request, and blocked the event loop building the ZIP — enough to take down the shared API on a large bank. Make export asynchronous, mirroring the already-async import path: - new document_export operation: submit_export_documents_async enqueues it; the worker builds the archive, stores it in file storage, and records download_url/storage_key/byte_size in result_metadata - POST /banks/{id}/document-transfer/export (202 + operation_id) - the sync GET is removed -> 410, pointing at the async endpoint - GET /v1/default/files/download/{key} streams the archive; retrieval + bank authorization live in MemoryEngine.retrieve_bank_file (IDOR guard) Harden export_documents: batch the entity/causal attach ANY() queries instead of passing hundreds of thousands of UUIDs at once, and move ZIP assembly off the event loop with anyio.to_thread. Regenerate all SDKs; add blocking export_documents convenience helpers to the Python + TS wrappers (submit -> poll -> download), fetching the server-provided download_url to avoid %2F path-encoding. Update the control-plane proxy to orchestrate the async flow and the docs. * refactor(transfer): name the export op export_documents; surface it in the CP - rename the async operation/task type document_export -> export_documents (and _handle_document_export -> _handle_export_documents) so it mirrors the import_documents operation - control plane: add export_documents + import_documents to the operations type filter and localize both (operationType.exportDocuments/importDocuments across all 10 locales) — previously neither appeared in the filter and both rendered as the raw task_type string * feat(transfer): clean up export archives with their operation + add download button Export archives were stored in file storage but never deleted, so they outlived their operation: the retention sweep prunes the async_operations row but left the blob orphaned, and a user delete didn't remove it either. Tie the archive to its operation record: - delete_operation now deletes the export archive along with the row - the retention sweep purges export archives (matching prune's terminal + updated_at < cutoff predicate) before pruning the rows So an export is retained exactly as long as its operation — indefinitely by default, or until HINDSIGHT_API_OPERATION_RETENTION_DAYS prunes it. Control plane: - add a Download button to the export operation's detail dialog (streams the archive through a new /api/files/download proxy, SSRF-guarded to the file-download path) + localize the label across all 10 locales * chore(docs-skill): regenerate references for export retention note * chore(cli): skip new export/download ops in CLI OpenAPI coverage export_documents_sync_removed (the 410 stub) and download_file are served via the API/control plane, not the end-user Rust CLI. * fix(transfer): register export_documents slot config + fix cleanup-sweep tests - add export_documents to WORKER_SLOT_TYPE_DEFAULTS (every operation_type used in memory_engine must be listed there — enforced by test_worker) - stub engine.purge_expired_export_archives in the operation-cleanup test mocks (the sweep now calls it before pruning each schema) * test(transfer): make export-archive purge test xdist-safe purge_expired_export_archives is schema-wide, and CI shares the schema across xdist workers, so a future cutoff purged other concurrent tests' fresh archives (flaky count + cross-test interference). Backdate this op and use a past cutoff so it targets only itself; assert purged >= 1.	2026-08-10 17:54:55 +02:00
Nicolò Boschi	3e3e3f372c	feat(bank-template): make every bank config field export+importable (#3332) ... Seven per-bank configurable fields were not declared on BankTemplateConfig, so export/import silently dropped them: consolidation_llm_parallelism, consolidation_max_memories_per_round, enable_auto_consolidation, memory_defense, recall_chunks_max_tokens, recall_include_chunks, recall_max_tokens. Cloning a bank produced a clone that looked correctly configured while quietly running on the server defaults for those seven — and memory_defense being one of them means a bank's defense policy did not travel with it. All seven are now part of the template engine, so an exported bank reproduces its full configuration on import. The rest of the change is about not needing to notice this again. Adding a per-bank config field is a multi-step flow, and each step now fails until the previous one is done: 1. add it to _CONFIGURABLE_FIELDS -> test_every_configurable_field_is_exportable fails until it is declared on BankTemplateConfig (both directions: a template field that is not configurable fails too, since the engine would reject it); 2. declare it there -> test_sample_values_cover_every_exportable_field fails until it has a value in _SAMPLE_VALUES; 3. give it a value -> the existing round-trip test exercises it end to end; 4. touching BankTemplateConfig moves the OpenAPI spec, the generated clients and bank-template-schema.json, so verify-generated-files fails until those are regenerated. An intentional exclusion is now a decision to record in the guard with a reason, not an omission that no one sees. The docs listed 15 of the 45 fields in a hand-maintained table that was already stale and would contradict "every field is supported" the moment it drifted again. It now states the guarantee and points at the generated schema as the authoritative list, keeping the common fields as examples. Verified by mutation: adding a configurable field without a template field, adding a template field that is not configurable, and adding a template field with no sample value each fail the suite.	2026-08-10 13:49:21 +02:00
Nicolò Boschi	f9fb3e934a	perf(entity-resolution): exclude label entities from the trigram fuzzy-match index (#3208) (#3214) ... Label entities resolve by exact match only, yet their rows were still covered by the shared trigram index — every fuzzy probe for a regular entity pulled them into its candidate set only to discard them in the bitmap recheck. On banks where a free-text label group accumulated tens of thousands of mutually-similar values this dominated database CPU under ingest bursts. - Add entities.entity_kind ('regular'/'label', CHECK-constrained) on both dialects, set at insert time by the resolver; the Phase-2 reassert carries the kind so a pruned label parent resurrects as a label. - Rebuild the PG trigram index as a partial index excluding label rows (CONCURRENTLY, create-before-drop) and add the matching entity_kind != 'label' predicate to the trigram candidate query and the Oracle UTL_MATCH fuzzy scan. - Migration backfills existing rows per bank by classifying canonical_name against the bank's entity_labels config with the same is_label_entity() the resolver uses. - Fix the label classification gating on the enum lookup set: a config with only text/map groups builds an empty set, so its labels were never recognised — neither by the #3187 exact-lookup split nor by the new insert-time kind. Bank import needs no changes: transfer archives treat entities as derived data and re-resolve them through the standard retain Phase 1, which now stamps the kind.	2026-08-06 19:41:58 +02:00
Nicolò Boschi	ba5a4813b3	fix(mental-models): never overwrite a document with a delta-window candidate (#3182) ... * fix(mental-models): never overwrite a document with a delta-window candidate A delta refresh runs reflect with created_after = last_refreshed_at, so its candidate only covers memories newer than the last refresh. When the delta operations failed to reach the document, that candidate was written as the whole document and the watermark advanced past it — everything grounded in older memories was gone for good, while the log said "falling back to full synthesis" and the operation completed successfully (#3112). Refuse it instead, keyed on the window rather than on each failure branch so future ones inherit the guard: when delta was requested, was not applied, and the reflect window was narrowed, preserve the document and raise MentalModelRefreshError. The watermark stays put, so the retry reads the same facts again. Also: - Treat "the model emitted operations but every one was rejected" as a delta failure. The document is unchanged, so persisting it looked like a clean refresh while dropping that run's facts outside every future delta window. - Recover from an unusable structured_content by re-parsing the stored markdown instead of giving up — nothing else rewrites that column, so failing there wedged the model permanently. - Record skipped operations even when the delta did not land, count them in the operation's result_metadata, and warn when a partial skip means some of this run's evidence never reached the document. - Route every failure through one preserve-and-fail helper, so the structured-output failure now leaves the same reflect_response audit trail the other two already did. * docs(mental-models): describe what a failed delta refresh does to the document The delta section promised the opposite of what the code now does — "zero valid operations means an identical document … never corrupt it" read as a guarantee while a failed delta was in fact replacing the document with a partial one. Say plainly that the document is kept and the refresh fails, and list the two new diagnostic values.	2026-08-05 14:47:13 +02:00
Nicolò Boschi	0ca0e87a08	fix(control-plane): report mental-model freshness from the bank write watermark (#3156) ... * fix(control-plane): report mental-model freshness from the bank write watermark The mental-models card compared each model's last_refreshed_at against the bank's last_consolidated_at, so any consolidation after a refresh — nearly always — reported every model as stale, and a bank that had never consolidated reported the opposite. Computing the real per-model answer on a list is the expensive fix: compute_mental_model_is_stale has no index to use (there is none on memory_units.updated_at), so it scans the bank's memories in full, per model — 10ms per model at 100k memories, 101ms at 500k, on a view that polls every 5s. Report a bank-wide watermark instead. MAX(updated_at) rides along on the aggregate _compute_bank_stats already runs and is served from the same cached payload as last_memory_write_at. A model refreshed at or after it is up to date, exactly; older only means something was written, possibly outside its tags, so the card says "may need refresh" rather than asserting stale. The exact answer stays on the single mental-model read, behind the dialog. The knowledge-base tree ran that same scan once per page and polls every 12s — already a full scan per page per tick in production. It now shares the watermark: one cached lookup for the whole tree. Fixes #3139 * perf(reflect): skip the per-model staleness scan below the bank watermark search_mental_models computed staleness with the exact scoped query for every model it returned — up to 5 full scans of the bank's memories per tool call, serially, on a held connection, and the agent can call the tool several times per reflect. get_bank_freshness already computes the bank's write watermark in the same scan it runs once per reflect, and was discarding it. Thread it through: a model refreshed at or after the newest write in the bank cannot be stale whatever its scope, so it skips the query entirely. Everything above the watermark still gets the exact tag-aware answer — the agent only trusts a model without a verifying recall() when is_stale is False, so guessing conservatively here would buy LLM turns to save a query. * chore(docs-skill): re-sync the generated reference copies generate-docs-skill.sh output drifted from the docs pages that landed on main (retain narrator guidance, configuration, coding-agents). Regenerated so verify-generated-files has nothing to report.	2026-08-04 16:53:19 +02:00
Nicolò Boschi	4278f0989d	feat(reflect,mental-models): surface structured output in the control plane (#3113) ... * feat(reflect,mental-models): surface structured output in control plane Reflect's response_schema -> structured_output was already implemented and tested in the engine but never exposed in the UI. Surface it in the reflect (think) view, and extend the same structured-output extraction to mental models via a per-model response_schema stored in the trigger config. - engine: refresh_mental_model reads trigger.response_schema, forwards it to the internal reflect call, and persists the parsed structured_output onto the stored reflect_response payload; fix stale 'not yet supported' docstrings - api: add response_schema to MentalModelTrigger - control-plane: reflect route + api.ts forward response_schema; think-view gets a JSON-schema input and renders structured_output; create/update mental model dialogs get a schema editor; detail modal renders structured_output - tests: mental model structured-output plumbing (schema forwarded + persisted) - regenerate OpenAPI spec + client SDKs; add i18n keys for all locales * feat(control-plane): show configured response_schema in mental model config tab Adds a read-only JSON card for the mental model's trigger.response_schema in the detail modal's Configuration tab (mirrors the tag_groups card), plus the regenerated go openapi.yaml. * style: ruff format test_mental_model_structured_output * fix(control-plane): don't route the JSON schema example through next-intl The response_schema placeholder was a t() message whose value is literal JSON. next-intl parses messages as ICU, so the '{' in the example was read as an argument placeholder, the parse failed, and the field rendered the raw message key instead of the example. Inline the JSON example directly on the placeholder prop (i18n:check skips JSON-shaped placeholders) and drop the now-unused *Placeholder message keys. Caught by running the control plane. * feat(structured-output): validate response_schema + add a no-code schema builder Validation (both reflect and mental models): a schema that is valid JSON but not a usable object-with-properties silently produced empty structured_output or blew up inside the LLM extraction call later. Now: - engine: validate_response_schema() enforces the usable-shape contract (object schema, non-empty properties, well-formed required); wired as Pydantic field_validators on ReflectRequest.response_schema and MentalModelTrigger.response_schema (invalid -> HTTP 422). - control-plane: the reflect and mental-model forms validate the schema shape on submit (not just JSON.parse) and surface the specific error. No-code schema builder: a 'Build schema' button on both the reflect view and the mental-model dialogs opens a dialog with Visual and Code modes. Visual mode edits a flat field list (name, type, array item-type, description, required); Code mode edits raw JSON. The two stay in sync and Apply is gated on a usable schema. Shared frontend lib (response-schema.ts) mirrors the backend contract. tests: test_response_schema_validation.py (16 cases: validator + model integration). * refactor(control-plane): schema only via the builder, show set/unset status Removes the inline response_schema JSON textarea from the reflect view and the mental-model dialogs. Editing now happens exclusively in the schema builder; the page shows only whether a schema is set (field count + names, with Edit/Remove) or a Build schema button when none. Extracts the shared ResponseSchemaField component used identically by reflect and both mental-model dialogs. * fix(mental-models): derive structured_output from final content, not reflect's answer In delta mode reflect only sees facts created since the last refresh, so its answer (and any structured_output it derived) reflects just the delta — while the stored content is the delta-merged document. Persisting the reflect-derived value made structured_output inconsistent with the markdown. Now the mental-model refresh no longer passes response_schema to reflect; instead it extracts structured_output from the FINAL stored content (correct for both full and delta), and carries the previous value forward untouched when a delta refresh preserves content (no new facts). Adds a delta test asserting extraction runs against the merged document, not reflect's partial answer. * fix(schema-builder): allow switching an empty schema from Code back to Visual An empty schema serialises to properties:{}, which schemaToFields mapped to an empty array — and the Code->Visual guard treated 'empty' the same as 'not representable', blocking the switch. schemaToFields now returns [] (representable) for a missing/empty properties map and null only for genuinely unrepresentable schemas; the switch seeds a blank field when empty. * docs(reflect): document structured output (response_schema) + schema builder Adds a Structured Output section to the reflect docs: how response_schema returns both text and a structured_output projection of the same answer, the schema rules, mental-model structured output (extracted from the final/merged document), and the no-code Build schema editor. Regenerates the docs-skill mirror. * feat(schema-builder): recursive visual editor for nested objects & arrays The visual editor was flat — object/array fields had no way to define their inner shape. Reworks the field model into a recursive tree (each field has a node; an object node nests fields, an array node nests an item node) so you can build nested objects and arrays-of-objects entirely in the visual editor. Code<->Visual round-trips losslessly; schemas using features the editor can't represent (enum, oneOf, $ref, tuple items, …) stay in code mode rather than being silently flattened. * fix(structured-output): recursive model for nested schemas + fail refresh loudly Two problems surfaced by nested schemas on Gemini: 1. _generate_structured_output mapped object/array properties to bare dict/list, which serialize with additionalProperties — rejected by the Gemini API. So any schema with a nested object/array silently failed extraction. Now it builds a proper recursive Pydantic model (nested objects -> nested models, arrays -> typed lists), matching how retain's structured output already works on Gemini. 2. On extraction failure the mental-model refresh silently persisted content with no structured_output, clobbering the previously-stored value. Now, when a response_schema is configured and extraction yields nothing, the refresh raises MentalModelRefreshError — prior content and structured_output are preserved and the refresh can be retried. Verified live on Gemini: a nested {location:object, people:array} schema now extracts (structured_output present) instead of failing on additionalProperties. Adds a fail-loud regression test. * fix(schema-builder): readable error text in dark mode text-destructive resolves to a dark red (#C0183A) in dark mode, which is low-contrast on the dark dialog background. Use the codebase's standard readable pattern (text-red-600 dark:text-red-400) for the builder's validation error and the invalid-schema notice. * fix(cli): set response_schema on MentalModelTriggerInput literals Adding response_schema to MentalModelTrigger regenerated the Rust MentalModelTriggerInput struct with a new field; the hand-written CLI struct literals must initialize it (E0063). Sets response_schema: None in the three construction sites (create/update mental model, knowledge-base pin). * docs(api): document mental-model response_schema; fix stale reflect text-empty claim; test schema lib - api/mental-models: document the trigger.response_schema flag + a Structured Output section (extraction from final content, fail-loud, validation). - api/reflect: correct the stale claim that text is empty with response_schema — reflect returns both text and structured_output. - control-plane: vitest unit tests for the response-schema lib (validation + recursive fields<->schema round-trip). - regenerate docs-skill mirror. * chore: regenerate bank-template-schema for MentalModelTrigger.response_schema The bank template schema embeds MentalModelTrigger; adding response_schema to the trigger changed the generated schema. Regenerated so verify-generated-files passes.	2026-08-03 16:51:38 +02:00
Nicolò Boschi	06e9c7054e	feat(mental-models): dry-run refresh and keep_trace for troubleshooting (#3119) ... When a refresh produced an unexpected document, nothing said why. The mode decision, resolved scope, snapshot window, retrieved-versus-used fact counts and dropped delta operations only ever reached a log line — and cron- or consolidation-driven refreshes run with nobody watching. Two ways to see that reasoning, from opposite directions. POST /mental-models/{id}/dry-run-refresh runs the production refresh pipeline and reports what it would do, skipping exactly two writes: the content (with its structured document and history entry) and the watermark that moves last_refreshed_at. It takes no parameters, on purpose — a dry run you can configure stops predicting the refresh it exists to predict. Because nothing is persisted, a delta dry run reads exactly the window the next real refresh will. trigger.keep_trace records the same reasoning on every refresh of a model, scheduled ones included, under reflect_response.trace. It is written even when a refresh fails, which is when it matters most. The trace is shaped like reflect's — the calls the agent made plus the refresh decision — and holds nothing derivable from elsewhere: evidence stays in based_on, and the resolved scope and window are reported by the dry run. Each tool call records the window bound it was given, named `updated_at` for what the predicate actually filters; null means the tool applies no time bound at all, which is what explains results older than the window would suggest. refresh_mental_model is split into a shared _execute_mental_model_refresh that computes a result and writes nothing, plus a thin persistence step, so the preview and the real refresh run the same body. Existing refresh behaviour is unchanged. In the control plane the dry run is an action on the mental model, and its result opens in a dialog built from the History tab's own diff components. History shows each version's own trace: the history snapshot now carries `trace` alongside `based_on` so it survives being superseded. Surfaced but deliberately not fixed here: when delta operations fail, the fallback writes a candidate built from a delta-scoped recall over the whole document, dropping content grounded in older memories (#3112).	2026-08-03 15:56:28 +02:00
Nicolò Boschi	4d22a882f5	docs(knowledge-pages): document Knowledge Pages and Mental Models, and manage them from the CLI (#3151) ... * docs(knowledge-pages): document Knowledge Pages and Mental Models Knowledge Pages shipped in #2455 with no documentation at all — no architecture page, no API page, no mention in the sidebar. Mental models had an API page but nothing explaining what they are or why they are fast. Add both, as top-level entries under Architecture and API. - Architecture: how pages are mental models with a simplified, document-shaped configuration; the folder hierarchy; the `hindsight fs` filesystem projection; page-level search; and why a projected view over reconciled memory is not the same thing as a folder of raw files. - Architecture: mental models as standing answers built in the background, so an application reads the current version instead of paying for synthesis on the request path. - API: the full knowledge-base endpoint surface, the page defaults and what each one buys, staleness gating, what a refresh reads, and how delta mode edits a structured document instead of regenerating prose. The mental-model trigger table gains the seven settings it was missing. - FAQ: mental model vs knowledge page. Also corrects the neighbouring answer, which described mental models as built automatically during retain — that is observations. The API examples use the maintained clients like every other API page, so this adds the knowledge-base surface to the Python and TypeScript wrappers (kept at parity, with request-mapping tests on both sides) and runnable Python/Node/Go examples. * feat(cli): manage knowledge pages from the CLI The knowledge base was reachable from every client except the CLI, where the eight endpoints were listed as deliberate coverage skips ("managed in the control plane UI"). That left `hindsight fs` able to mirror pages read-only but nothing able to create, edit, search, or delete them — and it meant the API docs could not show a CLI tab alongside Python/Node/Go. Adds `hindsight knowledge-base` with tree, create-folder, create-page, get-page, search, update, delete, and export, removing the skips so cli-coverage-check enforces the surface from here on. `create-page` sends no trigger unless --mode or --fact-types is passed, so the server's page defaults stand; when either is given the whole trigger has to be restated, because a supplied trigger replaces the defaults rather than merging with them. Also adds the CLI tab to the Knowledge Pages API page and a Knowledge Base section to the CLI reference.	2026-08-03 14:53:57 +02:00
Nicolò Boschi	6a460d2c9a	feat(reflect): add apply_all_directives to bypass directive tag scoping (#3031) (#3046) ... * feat(reflect): add apply_all_directives to bypass directive tag scoping (#3031) Directives are tag-scoped like memories: a reflect with no tags loads only untagged directives, and tagged directives apply only when the request's tags match. This is deliberate (isolation_mode), but it means an operator's tag-organized directives silently never reach an untagged reflect — 45% of standing rules in the deployment reported in #3031. Add an opt-in `apply_all_directives` flag on the reflect request (default false, preserving current behavior). When true, every active directive is loaded regardless of tags, ignoring tag scope. Wired through the HTTP API, both MCP reflect variants, and the engine. Also correct the docs, which claimed directives are "always" enforced without mentioning tag scoping. Regenerated OpenAPI, clients (Go/Python/TS/Rust), and the docs skill mirror; updated the control-plane reflect proxy + api.ts types. * chore(cli): record apply_all_directives CLI-coverage exemption The reflect field is intentionally not exposed as a CLI flag (available via the REST API, SDKs, and control plane). Record the exemption so cli-coverage-check passes, matching the existing tag_groups entry.	2026-07-29 15:39:50 +02:00
Nicolò Boschi	9452ac29da	feat(retain): report zero-fact documents at write time (#3040) (#3044) ... * feat(retain): report zero-fact documents at write time (#3040) A document whose fact extraction legitimately returns zero facts is stored but unreachable: only memory_units carry embeddings, so recall and reflect cannot reach a document that owns none. The retain still succeeds, the operation reports completed, and nothing in the response, the webhook or the metrics says the document produced no memories — the operator has no way to know it needs a reprocess. FAIL_ON_EXTRACTION_ERRORS (#2721) cannot help by construction: there is no error to fail on. #2861 made retain.completed fire for zero-fact batches, but the payload is byte-identical to a successful one, so it still carries no signal. Add the count to all three write-time surfaces: - retain.completed gains data.memory_unit_count, filled inside the outbox callback on the retain's own connection so units written by the enclosing transaction are visible. - The synchronous retain response gains memory_units_created. - New counter hindsight.retain.documents.total{outcome=facts|no_facts}, emitted per document at both extraction exits. The webhook and the metric report the document's total *after* the retain, not what the call created: the delta path skips unchanged chunks, so an idempotent re-retain creates zero units while the document keeps every memory it had. Reporting units created would raise a false alarm on every re-submit. The count query only runs when the call created nothing, which is the path where no work was done anyway. Docs: how a retain mission trades away retrieval of the raw source, the three signals, the non-determinism caveat, and reprocess as the way back. * fix(retain): drop memory_units_created from the retain response The synchronous response field reported units created by that call, which is a different number from the one the webhook and the metric report (the document's total after the retain) and only ever populated on the sync path. The async path is the one that matters, and it is already covered by retain.completed carrying data.memory_unit_count. Removing it also takes the API surface back to identical with main — the webhook payload is now the only public shape change — so the regenerated Python/TypeScript/Go clients and the OpenAPI spec carry no delta. Also renames the metric's parameter to memory_unit_count to match what it is actually handed: the document total, not units created.	2026-07-29 15:04:58 +02:00
Nicolò Boschi	0e5aa8896e	fix(curation): keep causal links across edit and invalidate/restore (#2951) ... Causal edges (`caused_by` plus the historical `causes`/`enables`/`prevents`) are retain-time extraction output. Nothing recreates them: graph maintenance only rebuilds temporal/semantic links and consolidation regenerates observations, not raw-fact edges. Curation destroyed them anyway (#2864): * every edit — including a context-only one — deleted all incident `memory_links` rows, and * invalidation moves the row out of `memory_units`, so the FK cascade took its causal edges with it and restore had nothing to bring back. Edits now delete only the derived link types, so a corrected fact keeps the causality the extractor asserted for it (preserving the assertion is the reversible choice; deleting it is not). Invalidation snapshots the incident causal edges into a new `causal_links` JSONB column on the archive row, and restore rematerializes the ones whose peer endpoint is live again. The snapshot also picks up descriptors parked on archived peers that name the unit, so an edge whose both endpoints are invalidated survives on both archive rows and is recreated by whichever endpoint is restored last — restore order doesn't matter. Rematerialization goes through the existing bulk-insert path, which drops links whose endpoints aren't live and is `ON CONFLICT DO NOTHING`, so repeated invalidate/restore cycles never duplicate an edge or resurrect one pointing at a permanently deleted memory.	2026-07-24 16:38:06 +02:00
Nicolò Boschi	c41ad9bd75	feat(api): filter memory list by linked entity + entity timeline UI (#2945) ... * feat(api): filter memory list by linked entity + entity timeline UI Add an `entity_id` query param to `GET /memories/list` — an exact reverse lookup over stored entity links (not text/semantic match), backed by the existing idx_unit_entities_entity_unit index. Because entity links reference live memory units only, combining `entity_id` with `state=invalidated` returns nothing. Wire it through the control-plane list route + clients, and use it in the entity detail panel to render an observation timeline (reuses the memories TimelineView) — click an entity, see its linked observations over time. Closes #2936. * fix(control-plane): entity timeline shows all linked memories, not just observations Verified against real data: observations are derived/consolidated summaries and carry no entity links — entity links live on the source world/experience facts, which are also the ones with occurred dates. Filtering the entity timeline to type=observation therefore always rendered an empty panel. Drop the type filter so the panel shows every memory linked to the entity (the actual dated timeline), and relabel the section "Timeline" with dedicated i18n keys. * chore(control-plane): drop now-unused observation i18n keys from entitiesView * style(reflect): wrap over-length _generate_structured_output call Ruff format wraps this >120-char call; committing the formatter output so the verify-generated-files CI check (which runs the formatter and diffs) is clean.	2026-07-24 14:28:44 +02:00
Nicolò Boschi	31218127e0	fix(retain): make async retries idempotent via caller-supplied operation_id (#2937) (#2947) ... * fix(retain): make async retries idempotent via caller-supplied operation_id An async retain whose HTTP acknowledgement is lost or times out leaves the caller unable to tell whether the operation was created; retrying enqueues a second parent operation and repeats extraction, embeddings, and provider spend. Add an optional caller-supplied operation_id (UUID) used directly as the parent async_operations primary key. Re-submitting with the same id returns the original operation and creates no new work; the existing primary key is the concurrency authority, so no new columns, constraints, or migration are needed. Reusing an id owned by a different bank or operation type returns HTTP 409. Omitting operation_id keeps the current create-each-time behavior. Fixes #2937 * docs(retain): explain why the idempotency read is not in the create txn * fix(retain): sync generated docs-skill + Rust clients for operation_id - Regenerate the two docs-skill artifacts derived from the retain doc / OpenAPI change (verify-generated-files). - Add operation_id: None to the Rust client test and CLI RetainRequest literals so both crates compile against the regenerated struct.	2026-07-24 13:43:57 +02:00
Nicolò Boschi	41d71a9818	fix(#2808): make mental model tags_match configurable on all creation surfaces (MCP, TS client, CLI) (#2858) ... * feat(mcp): let create_mental_model configure tags_match (#2808) A tagged mental model with no explicit tags_match in its trigger JSON refreshes under all_strict (a memory must carry every one of the model's tags), while the staleness check and every recall/reflect path default to any. Broadly-tagged models reading narrowly-tagged memories therefore get marked stale and then refresh to empty content. The HTTP API, generated SDK clients, and Control Plane UI already let users set trigger.tags_match; the MCP create_mental_model tool did not. Add a tags_match argument (validated against TagsMatch) to both MCP variants. It is only written into the trigger when explicitly passed, so the resolved all_strict default is preserved for existing callers. Document the all_strict footgun and the tags_match override in the MCP and mental-models API docs (regen skills/hindsight-docs mirror). * fix(ts-client): expose tags_match/tag_groups on createMentalModel The ergonomic TypeScript wrapper's createMentalModel accepted only { refreshAfterConsolidation } in its trigger option and dropped every other trigger field, so a wrapper user could not set tags_match — the exact knob needed to avoid the empty-refresh footgun in #2808. The low-level generated sdk already accepts the full MentalModelTriggerInput; thread tagsMatch and tagGroups through, mirroring how recall/reflect already expose them. The Python client needs no change: its wrapper takes a pass-through trigger dict and the generated MentalModelTriggerInput already validates tags_match. * test(ts-client): cover createMentalModel trigger mapping Mock the generated sdk layer (no server needed) and assert the ergonomic camelCase trigger options map onto the snake_case body: tagsMatch -> tags_match, tagGroups -> tag_groups, refreshAfterConsolidation still maps, and omitting trigger sends none (preserving the all_strict default). Locks in the #2808 wrapper fix. * docs(mental-models): add tags_match code snippet Replace the static JSON block in the tags_match override section with a live CodeSnippet pulled from the Python example, showing how to create a model with trigger.tags_match="any" so a broadly-tagged model reads narrowly-tagged memories on refresh (#2808). * feat(cli): add --tags-match to mental-model create + all-language docs The Rust CLI's `mental-model create` was the last creation surface with no way to set tags_match, so a tagged model created via the CLI hit the same empty-refresh footgun (#2808). Add a `--tags-match` flag (any/all/any_strict/ all_strict/exact) that is only sent when passed, preserving the server's all_strict default; invalid values are rejected before the request. Expand the mental-models docs "tags_match override" example from a single Python snippet to a full Tabs block (Python / Node.js / CLI / Go), each pulled from the runnable example files, and regen the skills mirror.	2026-07-21 12:04:23 +02:00
Nicolò Boschi	d61e8ffff6	fix(worker): discover expired-operation schemas in one query, default retention off (#2819) ... Follow-up to #2708, which bounded terminal `async_operations` history. Two issues with what landed: 1. The cleanup worker did not use a cross-tenant routine. It opened a connection and a prune transaction against *every* tenant schema on every cleanup cycle, paying the full per-tenant cost even when nothing was prunable — the query storm the server-side maintenance routines exist to avoid. 2. It shipped as a breaking change, silently switching deployments from unbounded operation history to a 30-day TTL on upgrade. Adds `schemas_with_expired_operations(p_days int) RETURNS SETOF text` — the `async_operations` counterpart to `schemas_with_expired_rows`. One round-trip returns just the schemas holding expired terminal rows; the worker then acquires a connection and prunes only there. It needs its own routine rather than reusing `schemas_with_expired_rows` because eligibility here isn't "row older than N days" — pending and processing rows are never prunable, so the status filter has to be part of the predicate. Install policy follows b6d2f8a4c1e7 (#2638/#2824): the routine is database-global (it enumerates pg_class across every schema), so exactly one copy is installed — into the schema this deployment is configured to use, which is the one the worker calls via fq_routine. Gating on the literal "public" instead of the configured schema is what left non-public deployments without the sibling routines (#2638); installing into every schema would leave a dead duplicate per tenant. Exactly one run satisfies that predicate, so concurrent per-schema runs never issue competing CREATE OR REPLACE against the same pg_proc row and cannot hit `tuple concurrently updated`. No cross-process coordination, and in particular no advisory lock, which is unusable behind connection poolers and managed PG (#2817). Runs targeting any other schema drop the routine there instead. The worker calls the routine through schema.fq_routine() (added in #2824) rather than a hardcoded public. qualifier — duplicating that qualifier across callers is precisely how #2638 recurs. Vanishing schemas are skipped rather than fatal (c7e9f1a3b5d2), and an absent routine degrades cost, not correctness — Oracle and un-migrated PostgreSQL fall back to the previous full sweep with a warning. DEFAULT_OPERATION_RETENTION_DAYS 30 -> 0. Operation history is a user-visible audit trail, so bounding it is an opt-in policy decision rather than something an upgrade applies silently. Set HINDSIGHT_API_OPERATION_RETENTION_DAYS to a positive number of days to enable pruning. Docs, .env.example and the bundled embed template updated to match. - test_schemas_with_expired_operations — drives the real routine against pg0 in a throwaway schema: old pending/processing rows alone don't make a schema eligible, a terminal row does, a too-old cutoff doesn't, p_days <= 0 is empty. - test_expired_operations_routine_installs_in_the_configured_schema — parametrized over base / default public / non-public single-tenant; guards against reintroducing the #2638 literal gate or an advisory lock. - test_expired_operations_tenant_runs_install_nothing — tenant runs emit no CREATE and drop any copy in their own schema. - test_discovery_targets_the_configured_non_public_schema — the worker calls the copy in its configured schema, not a hardcoded public one. - TestWorkerOperationCleanupSchemaNarrowing — only reported schemas are pruned, nothing expired means no pruning, unclaimed schemas are skipped, a missing routine falls back to the full sweep, Oracle never calls the routine.	2026-07-20 18:35:54 +02:00
Voscko	86ff344c93	fix(worker): bound terminal operation history (#2708) ... Add configurable TTL (default 30 days, 0=keep-forever) for terminal async_operations rows. Expired completed/failed/cancelled rows are pruned in bounded batches (1000/cycle) by a background task that never touches pending/processing work. Batch children are protected until their parent is pruned; cancelled-child cleanup atomically cancels a pending parent first. PG uses FOR UPDATE SKIP LOCKED; both PG and Oracle re-check eligibility under the row lock before deleting. Includes indexes, docs, and regenerated SDKs. 184 retention/worker/operation-status tests pass locally. All CI green. Fixes #2705	2026-07-14 10:04:26 -04:00
Sanderhoff-alt	5cc1482a72	fix(memory): return metadata from memory browse endpoints (#2583) ... The list and get memory-unit paths selected tags and timing fields but skipped the memory_units metadata column, so metadata retained on facts was invisible outside recall. Select and serialize metadata for live and invalidated memory units, add a curation regression test for both paths, and update docs plus OpenAPI examples.	2026-07-07 11:11:04 -04:00
Evo	dd83bffeef	docs(recall): align min_scores score field names (#2432) ... * docs(recall): align min_scores score field names * test: apply generated formatting	2026-06-29 10:41:59 +02:00
Nicolò Boschi	758f346d30	feat(recall): structured per-stage scores and two-level min_scores filtering (#2422) ... Replace the recall result's single `score` with a `scores` object exposing the scores from each pipeline stage, and replace the `min_score` request param with `min_scores`, a per-stage filter that operates at two levels. Response — each result carries `scores`: - final : the value results are ranked by - reranker : cross-encoder normalized relevance (null for passthrough rerankers) - semantic : raw vector cosine similarity (null if not surfaced semantically) - text : raw keyword/BM25 score (null if not surfaced by keyword search) Per-arm semantic/text scores are aggregated across retrieval arms during RRF / interleave fusion (ArmScores on MergedCandidate), since fusion otherwise keeps only the first-seen arm's score per doc. Request — `min_scores` floors (inclusive, AND-ed, opt-in; default no filtering): - semantic / text : retrieval-level cutoffs pushed into the SQL arms, overriding the global similarity / BM25 minimums for the request (prune before fusion) - reranker / final: post-query filters on the scored results There is deliberately no default threshold: the cross-encoder's absolute scores are reliable for ordering but not calibrated across queries (a clearly-relevant match can score ~0.001 on one query and ~1.0 on another), so a fixed cutoff would silently drop good results. Also surfaces proof_norm in the search trace and reworks the control-plane trace view to render scores at full precision (no rounding) and show the per-stage `scores` breakdown; relabels the trace's "CE" column to "reranker score". Threaded through engine, HTTP, MCP (both recall tools), and the control-plane proxy; OpenAPI spec, Python/TS/Go/Rust clients, and the docs-skill mirror regenerated; docs updated.	2026-06-26 17:12:27 +02:00
Evo	1621e5d261	docs(mental-models): document scheduled refresh triggers (#2421)	2026-06-26 14:54:03 +02:00