* fix(curation): resolve edited entity names exactly, not fuzzily (#3479)
update_memory ran the caller's entity names through the same fuzzy resolver
retain uses, so an entity name was a *guess* to be reconciled against the graph
rather than an instruction. Name identity is worth at most 0.5 of the 0.6 match
threshold, while co-occurrence (0.3) and recency (0.2) make up the rest, so a
similar-but-wrong entity that is well connected to the other names in the same
edit outscores the one the caller actually named — with a 200 and no warning.
Curation now resolves exactly: an existing entity is reused only when its
canonical name matches case-insensitively, any other name creates its own
entity, and same-batch names are never merged with each other. Retain keeps
fuzzy resolution, which is right for names that came out of extraction.
The exact path skips the trigram/UTL_MATCH probe and the co-occurrence fetch
entirely and reuses the existing find-or-create pass, so it is dialect-agnostic
and strictly less work than the fuzzy one.
* fix(curation): add entity_resolution_mode to update_memory, default fuzzy
Make the exact/fuzzy choice the caller's, rather than changing what an edit
does. `entity_resolution_mode` defaults to "fuzzy" — retain's behaviour, so
every existing caller is unaffected — and "exact" opts into literal matching for
hand-authored corrections.
Plumbed through the HTTP request model, the MCP tool, the control-plane proxy
route and its client, with the engine validating the value for direct callers.
The control-plane memory editor sends "exact": a person typing an entity list
into the admin UI is naming the entity they mean.
* refactor(curation): make the flag a boolean, resolve_entities
Replaces the entity_resolution_mode enum with a plain boolean on update_memory.
`resolve_entities` defaults to True — retain's behaviour, so existing callers are
unaffected — and False takes the submitted names literally.
Carries the same change through the resolver, which now takes `fuzzy_matching`
rather than a mode string. The engine's value guard goes away with the enum: a
bool needs no validation, so the invalid-value test goes too (the HTTP boundary
still 422s a non-boolean, which the HTTP test covers).
* feat(retain): honour resolve_entities for caller-supplied entities too
Same flag, same default, on the retain item — a caller passing explicit entity
names there has the same exposure as one correcting a memory: a name close to an
existing entity can be matched onto it and quietly replaced.
Retain resolves caller-supplied and LLM-extracted names in ONE batch, so a
per-batch flag would have turned resolution off for the extractor's names too and
filled the bank with near-duplicate entities. The flag is therefore carried per
mention: extracted names always resolve, supplied names follow the item's flag,
and a supplied name the extractor also produced keeps the caller's intent. The
in-batch dedup pass (#3107) skips the literal names for the same reason.
Also renames the resolver's `fuzzy_matching` parameter to the per-mention
`resolve` key, so one name is used end to end.
* fix(clients): carry resolve_entities through the maintained wrappers
Code review found two gaps the generated SDKs hide.
The TypeScript and Python convenience wrappers rebuild each retain item field by
field, so `resolve_entities` was silently dropped for every wrapper caller — the
same class of gap #2975/#3042 closed for the mental-model methods, and here it
would have quietly restored the substitution the flag prevents. Both wrappers now
forward it, with mapping tests on each side.
Intake also lost the flag when normalization collapsed two spellings into one:
entity_processing dedups caller-supplied against extracted names on the RAW text,
so a caller's literal "Acme Corp" and the extractor's "Acme\nCorp" both reach
_prepare_entities_for_resolution and only merge there. Keeping the first entry
verbatim dropped the caller's resolve=False with it; the merge now keeps the
stricter flag.
* fix: build the Rust CLI, and keep pg_trgm detection on empty batches
Two CI breaks from the retain change.
MemoryItem gained a field, and the CLI builds it with a struct literal, so every
Rust job failed to compile. The CLI supplies no entities, so `true` (the server
default) is the right value there.
The skip-the-probe guard also fired on an *empty* batch — `any([])` is False — so
_resolve_entities_batch_impl returned before the pg_trgm auto-detection that
hangs off the strategy dispatch. Only shortcut when there is data and none of it
resolves.
* fix(rust): add resolve_entities to the remaining MemoryItem literals
The first pass only fixed the CLI's src/ literal — `cargo build` does not compile
test targets, so the ones in hindsight-cli/tests/integration_test.rs and
hindsight-clients/rust/src/lib.rs went unnoticed until CI. Verified with
`cargo check --all-targets` in both crates this time.
* fix(engine): drop null metadata values at retain and recall
Retain accepts arbitrary JSON metadata; a null value (e.g. {"ocr_engine":
null}) stored verbatim made every read path that returns MemoryFact fail
dict[str, str] validation — recall, consolidation, and mental-model refresh
errored for the affected bank. Normalize on both ends: RetainContent drops
null-valued keys at construction (canonical storage) and
MemoryFact.parse_metadata drops them for legacy rows, preserving the
existing string coercion for other non-string values.
Closes#3209
* fix(engine): normalize null metadata on the delta and export paths too (#3209)
Follow-up to the retain/recall normalization on this branch, which left two
gaps.
Delta retain never went through RetainContent: all three call sites of
update_memory_units_metadata_and_tags pass the raw retain_params bag straight
to the UPDATE, so a re-retain left the units it preserved carrying null-valued
keys while the units re-extracted beside them did not. Drop the nulls inside
that storage function — the one chokepoint every delta path goes through —
leaving documents.retain_params holding the caller's input verbatim.
Bank export was the other read path that validates stored metadata as
dict[str, str]: TransferFact is built directly from the row, so a bank already
holding a null (or a raw integer, e.g. {"original_id": 348}) failed export with
the same ValidationError the issue reports — locking an operator out of the one
operation that gets them off the bad data. It now applies the same read
contract as recall.
Both rules now live in engine/metadata_utils.py rather than being spelled out
at each site, and RetainContent normalizes an explicit "metadata": null to {}
so the field always matches its declared type.
Regression coverage on a real database: test_delta_retain_drops_null_metadata_values
walks a full retain, a metadata-only delta and a partial delta with surviving
units; test_export_tolerates_legacy_null_and_numeric_fact_metadata exports a
bank whose rows were poisoned before the fix existed. Both fail without their
respective fix.
---------
Co-authored-by: Nova Lux <NovaLux12@users.noreply.github.com>
Co-authored-by: Nicolò Boschi <boschi1997@gmail.com>
* fix(retain): make async retries idempotent via caller-supplied operation_id
An async retain whose HTTP acknowledgement is lost or times out leaves the
caller unable to tell whether the operation was created; retrying enqueues a
second parent operation and repeats extraction, embeddings, and provider spend.
Add an optional caller-supplied operation_id (UUID) used directly as the parent
async_operations primary key. Re-submitting with the same id returns the
original operation and creates no new work; the existing primary key is the
concurrency authority, so no new columns, constraints, or migration are needed.
Reusing an id owned by a different bank or operation type returns HTTP 409.
Omitting operation_id keeps the current create-each-time behavior.
Fixes#2937
* docs(retain): explain why the idempotency read is not in the create txn
* fix(retain): sync generated docs-skill + Rust clients for operation_id
- Regenerate the two docs-skill artifacts derived from the retain doc /
OpenAPI change (verify-generated-files).
- Add operation_id: None to the Rust client test and CLI RetainRequest
literals so both crates compile against the regenerated struct.
MarkItDown advertises image extensions, but without OCR config it can
fail screenshots or scanned images with low-level no-content errors.
Add server-level MarkItDown OCR config that is off by default and
independent from HINDSIGHT_API_LLM_*. When OCR is enabled, the OCR API
key, base URL, and model are required explicitly.
Wire those settings into MarkItDown's llm_client support with a built-in
OCR prompt. Image uploads now fail fast with actionable errors when OCR
is disabled or required settings are missing.
Docs and front-end copy explain that image OCR depends on server config
and requires an OpenAI-compatible OCR/vision endpoint.
Closes#927
* feat(consolidation): add "shared" observation_scopes keyword
Add a "shared" value for observation_scopes that resolves to a single
global, untagged scope ([[]]). Memories consolidate into one observation
regardless of their tags, while the tags stay on the source facts for
recall filtering.
This is the supported way to deduplicate observations across volatile
per-call provenance tags (e.g. per-session ids): with combined/per_tag,
a unique session tag puts every retain in its own scope, so near-identical
facts never dedup and accumulate one observation per session. "shared"
keeps recall and the dedup probe on the same (empty) scope, fixing
consolidation quality rather than only the duplicate count.
- consolidator: _resolve_obs_tags_list -> [[]], _resolve_write_scopes -> [frozenset()]
- API/engine type literals + OpenAPI + regenerated Python/TS/Go/Rust clients
- CP client type kept in sync
- docs: retain.mdx 'shared' section (+ shared vs [[]] vs [] caveat),
observations.mdx dedup pointer; regenerated hindsight-docs skill mirror
- tests: unit scope-resolution + e2e parallel-consolidation scope correctness
* chore(opencode): apply prettier formatting to plugin.test.ts
Pre-existing lint drift unrelated to this PR — CI's verify-generated-files
job reformats all integrations (LINT_ALL_INTEGRATIONS) and flagged this file.
Folding the one-line reflow in here to get the gate green.
Adds Gemini Batch API support for retain fact extraction (50% discount, 24h SLA) via the existing HINDSIGHT_API_RETAIN_BATCH_ENABLED flag. GeminiLLM overrides the 4 LLMInterface batch methods, adapting Gemini's upload->create->poll->download flow to the OpenAI-batch shapes the consumer expects (same pattern as FireworksLLM). Gemini-only (Vertex unsupported). Threads usageMetadata into the result body. Live-verified end-to-end. Gemini portion of #1144; consolidation batch is a follow-up.
PR #932 added update_mode (replace/append) to retain items but
did not update the docs. Add a section explaining the parameter,
when to use append mode, and a JSON example.
Closes#957
* feat: independent versioning for integrations
- Add per-integration changelog pages at /changelog/integrations/<name>
- Move main changelog to changelog/index.md (URL unchanged)
- Add --integration flag to generate-changelog for LLM-based per-integration changelog generation
- Add scripts/release-integration.sh <name> <version> for cutting integration releases
- Add .github/workflows/release-integration.yml to publish on integrations/** tags
- Remove integrations from main release.sh and release.yml cycle
* fix: add agno and hermes integration docs to version-0.4 for production build
* chore: apply ruff formatting to generate_changelog.py
* feat: add 4-tab code parity across all documentation examples
Every code snippet Tabs block now has Python, Node.js, CLI, and Go variants.
Raw HTTP/curl tabs replaced with proper SDK calls.
New example files:
- Go: retain.go, recall.go, reflect.go, memory-banks.go, directives.go,
mental-models.go, documents.go, main-methods.go
- Shell: memory-banks.sh, directives.sh, mental-models.sh
- Node.js: mental-models.mjs
Extended example files with missing sections:
- recall.mjs/sh: world/experience/observation types, token-budget, all tag modes
- reflect.sh: reflect-with-params, reflect-disposition, reflect-sources, reflect-with-tags
- reflect.mjs: reflect-with-tags, fixed reflect-sources API usage
- retain.mjs/sh: retain-conversation, retain-batch, retain-files-batch
SDK/CLI additions:
- TypeScript: getMentalModelHistory method
- CLI recall: --tags, --tags-match flags
- CLI reflect: --tags, --tags-match, --include-facts flags
- CLI directive update: --is-active flag
- CLI bank set-config: --retain-mission, --retain-extraction-mode,
--observations-mission, --reflect-mission, --disposition-* flags
Build validation:
- scripts/check-code-parity.mjs validates 4-tab parity across all MDX files
- Integrated into npm run build — fails if any Tabs block is missing a variant
* fix: fix doc examples for Go, Node.js, CLI + add mental model with-id examples
- Fix Go Budget constants: BUDGET_HIGH/LOW/MID → HIGH/LOW/MID
- Fix Go documents.go: ListDocuments returns []map[string]interface{}, use map access
- Fix Go retain.go: use correct relative path for sample.pdf
- Fix Node.js createMentalModel: use positional args (name, sourceQuery) not object
- Add CLI 'history' subcommand for mental models (api.rs, main.rs, mental_model.rs)
- Rebuild TypeScript/Python clients to support id param in createMentalModel
- Add create-mental-model-with-id examples across all 4 languages and docs
* fix: move id param to end of create_mental_model signature for backwards compat
* feat: support timestamp="unset" to retain content without a date
When callers retain timeless content (e.g. fictional documents, static
reference material), passing timestamp="unset" now skips the utcnow()
default so mentioned_at is stored as NULL instead of an artificial date.
- HTTP: validate_timestamp recognises "unset" sentinel and threads it
through api_retain as event_date=None (key present, value None), which
the orchestrator distinguishes from key-absent (still defaults to now)
- Orchestrator: new branching logic separates "key absent" → utcnow()
from "key present but None" → no date
- types.py: RetainContent.event_date and ProcessedFact.mentioned_at are
now datetime | None; removed the unused _now_utc factory
- fact_extraction.py: all event_date params accept datetime | None;
_build_user_message emits "Event Date: Unknown" when None; removed
mentioned_at from the Fact LLM response model (LLM never sets it)
- embedding_processing: skip date suffix when fact_date is None
- entity_resolver: COALESCE(event_date, now()) for first_seen/last_seen
so entities table NOT NULL constraint is preserved
- link_utils: skip temporal linking for units without event_date
- Migration aa2b3c4d5e6f: DROP NOT NULL on memory_units.event_date
- Tests: test_retain_no_timestamp and test_retain_omit_timestamp_defaults_to_now
- Docs + OpenAPI + TypeScript client updated
* refactor: replace _TIMESTAMP_UNKNOWN sentinel with plain string comparison
The sentinel object() was only needed to distinguish "unset" from None
at the boundary — but since the field type is datetime | str | None,
"unset" can pass through the validator unchanged and be compared directly.
* chore: regenerate OpenAPI spec and clients after timestamp type change
timestamp field is now datetime | str | None to accept the "unset" sentinel value.
* feat: observation_scopes field to drive observations granularity
* fix(migration): make a2b3c4d5e6f7 a no-op to fix CI on fresh DB
The z1u2v3w4x5y6 migration already creates observation_scopes directly,
so the rename migration fails on fresh installs where observation_tags
never existed.
* chore: remove no-op migration a2b3c4d5e6f7
* feat: regenerate clients with observation_scopes field
- Add observation_scopes to OpenAPI spec and all generated clients
- Fix Rust build.rs to handle anyOf with >2 variants containing null
(previously only handled 2-item anyOf, causing progenitor to panic
on the observation_scopes union type)
* fix(rust): add observation_scopes: None to MemoryItem struct literals
* fix(api): add title to observation_scopes Field for deterministic client generation
Adding title="ObservationScopes" makes the inline anyOf schema use
the explicit name instead of deriving it from the field name, which
was non-deterministic between arm64 (macOS) and amd64 (CI) Docker.
Also fixes description: "each entity" -> "each tag".
* fix(scripts): use linux/amd64 Docker for client generation to ensure reproducibility
Both Python and Go client generation now use --platform linux/amd64
Docker, ensuring identical output on macOS arm64 (local) and Linux
amd64 (CI). Also switches Go from JAR+Java to Docker to eliminate
Java version variability.
* chore: update generated clients to API v0.4.14
* fix(test): add retry logic to test_retain_chinese_content to handle non-deterministic LLM output
* fix(test): mark test_retain_chinese_content as xfail due to non-deterministic LLM translation
* feat: accept pdf, images and office files
* refactor: rename FileConverter to FileParser, simplify file retain API
- Rename engine/converters/ → engine/parsers/, FileConverter → FileParser,
ConverterRegistry → FileParserRegistry, MarkitdownConverter → MarkitdownParser
- Rename env var HINDSIGHT_API_FILE_CONVERTER → HINDSIGHT_API_FILE_PARSER
- Remove async/document_tags params from FileRetainRequest (always async now)
- Add retain_files() to Python Hindsight client and retainFiles() to TypeScript client
- Add sample.pdf to doc examples for working file upload demonstrations
- Update test_file_retain.py to use new parser names and always-async behavior
- Fix Go client missing os import in api_files.go
- Simplify postgresql.py storage to minimal schema
* fix: update rust CLI tests to use is_supported_file instead of is_text_file
* fix: patch Go api_files.go to add missing 'os' import after generation
* fix: insert 'os' import after 'net/url' in api_files.go patch for correct position
* chore: regenerate OpenAPI spec and clients (converter→parser description update)
* feat: Record LLM token metrics via Prometheus
Wire up the existing token metrics infrastructure to actually record
token usage from LLM calls. The MetricsCollector already had
record_tokens() method and Prometheus counters (hindsight.tokens.input,
hindsight.tokens.output), but they were never being populated.
Changes:
- Import get_metrics_collector in llm_wrapper.py
- Call record_tokens() after successful LLM calls for:
- OpenAI/Groq (using response.usage.prompt_tokens, completion_tokens)
- Anthropic (using response.usage.input_tokens, output_tokens)
- Gemini (using response.usage_metadata.prompt_token_count, candidates_token_count)
- Add test file to verify token metrics are recorded
Note: Ollama's native API doesn't return token usage, so metrics
are not recorded for that provider.
The token metrics will now be available via /metrics endpoint:
- hindsight_tokens_input_total
- hindsight_tokens_output_total
* feat: add per-request token usage tracking to retain and reflect endpoints
- Add TokenUsage model with input_tokens, output_tokens, total_tokens
- Return usage metrics in retain response (sync operations only)
- Return usage metrics in reflect response
- Update Python, TypeScript, and Rust clients
- Add API documentation for usage fields
- Add changelog entry
* Fix main-methods.py: entities is a dict, use .items() and .canonical_name
* Migrate docs to use CodeSnippet components
- Convert quickstart.md, retain.md, recall.md, reflect.md, memory-banks.md to .mdx
- Use CodeSnippet to pull code from validated example scripts
- Add missing 'name' parameter to create_bank calls
- Fix main-methods.py entities iteration (dict not list)
- Remove retain-new.mdx demo file
* Migrate existing docs to match testing pattern with code snippet and add CLI tests to the CI
* Fix doc-id issue + add main-method tests
* CLI fixes
* Update openAPI json
* Fix rust build issues
* increase sleep time for Hindsight to process the document
* Added a polling sleep instead of fixed
* Delete immediately fails, so create the doc a earlier in the test to get the doc ready
* Add debug logs
* Remove debug logs
Nicolò Boschi