Makes images and files first-class raw content in `retain`. `content` accepts an
ordered list of text/image/file blocks, the extractor reads each attachment in
the position it occupies, and every read surface hands back the attachments
behind what it returns. A plain string behaves exactly as before — text-only
retain is byte-identical, because everything new sits behind an ATTACHMENTS
block that is empty when a chunk carries none.
Blocks are flattened at the API boundary into one canonical body with atomic
placeholders, so `documents.original_text` stays plain text and content_hash
idempotency, `update_mode=append`, chunk-delta re-extraction and
`reprocess_document` keep working untouched. Bytes live in the existing
FileStorage abstraction, content-addressed by sha256.
Schema (one migration, both dialects): `attachments` for the blob,
`document_attachments` for which documents reference it, and
`memory_units.attachment_ids` for which attachments a *fact* came from — a
column rather than a third table, because those ids behave exactly like `tags`.
Provenance is per fact, not per chunk. Extraction runs one call per chunk, and a
chunk holding a screenshot also holds the prose around it, so a chunk-level edge
cited the diagram as evidence for the paragraph that never mentioned it. The
extractor is asked instead, and a fact stated in the prose carries nothing.
Extraction quality was measured against a real image-QA dataset with a raw-VLM
ceiling arm before merging: transcribing structured attachments rather than
summarizing them, and recording how each value is drawn, took the gap between
"the model can read this off the image" and "memory can answer it" from 31.3% to
10.0% on the same 40 charts. The prose-article benchmark went 75% -> 100% over
the same change, so it is not chart-specific tuning.
Also here:
* A vision slot (`HINDSIGHT_API_VLM_*`) so attachment-bearing chunks alone use a
vision model and text-only chunks stay on a cheaper retain LLM. A vision call
deliberately does not fail over to the retain chain's text models — that would
reintroduce the silent omission the 422 gate exists to prevent.
* The extension retain hook can now see each attachment (media type, size, kind,
filename) and refusing a retain reclaims its bytes, which previously stayed
fetchable forever.
* A filename lives on the document edge, not the blob: the same PDF can be
attached under a different name elsewhere, and content-addressing made the
first name win for both.
Known limitations, documented rather than hidden: store-owned memory backends
get nothing (that retain path is Postgres-free and pre-dates this work), very
dense pages are sampled rather than exhausted, and the Python client's
ContentBlock is a plain dict where TypeScript gets the real union.
Breaking for Go and Rust callers: `content` is now a union, so a bare string no
longer satisfies it. Go gains a `TextContent()` helper; Rust uses
`Content::Variant0(...)`.
* docs: explain how entity resolution decides to merge a name
Entity resolution's thresholds were documented one env var at a time, with
nothing describing how the pieces fit together — so the obvious reading is
that pg_trgm similarity is what decides a match. It isn't: the trigram
threshold only admits candidates, and the merge decision is a separate
weighted score (sequence-ratio name similarity 0.5 + co-occurrence 0.3 +
recency 0.2, merged above 0.6) with no minimum name similarity of its own.
The two measures disagree most on short names, which is how a new person's
facts can land on an unrelated existing entity.
- configuration.md: new "How entity resolution decides" section covering both
stages, the weights, and what to change in either direction.
- configuration.md: RETAIN_ENTITY_LOOKUP said `full` was "exact match". It is
exact-or-substring, and both strategies feed the same scoring pass, so the
choice changes which names can merge — not just how fast the lookup is.
- retain.md: short note that resolution can absorb a new name into an existing
entity, pointing at the reference section.
* fix(entities): don't merge a name onto an existing entity on history alone (#3751)
A retain that correctly extracted a new person, Tigran, stored the fact on a
pre-existing country entity, Iran. Three defects combined:
1. The metric that admits candidates and the metric that decides the merge were
different, and they rank differently. pg_trgm scores tigran/iran at 0.20 —
above the 0.15 that admits it, below anything that should merge. difflib's
SequenceMatcher, which actually decides, scores it 0.80: higher than
alice/alice chen (0.67), a merge the resolver exists to make. With the
non-name signals worth 0.5 of the 0.6 threshold and no floor of their own,
history alone could carry it.
Gate candidates on trigram similarity before scoring
(HINDSIGHT_API_ENTITY_MERGE_MIN_SIMILARITY, 0.3). A gate, not a replacement:
SequenceMatcher stays load-bearing for typo variants arriving with no
co-occurrence context ("Dr Waler" -> "Dr Wall"), which is why the metric is
not simply swapped. This gives the existing-entity side the same three-tier
story the in-batch side already had — 0.15 to be considered, 0.3 to be merged
onto, 0.5 to be folded in-batch.
2. The co-occurrence term counted every shared entity equally, so `user` — which
co-occurs with nearly everything in a mature bank — was worth as much as a
selective partner, and was what pushed Iran over. Weight each shared partner
by 1/sqrt(degree). Degrees come from the co-occurrence rows already fetched,
so no extra query; with no degrees known the formula is arithmetically the old
overlap fraction, so nothing changes for callers that pass none.
3. A score *of* 0.6 landed on either side of the strict `>` depending only on
which signals produced it: 0.4 + 0.2 is 0.6000000000000001 and merged, 0.3 +
0.3 is 0.6 and did not. Round, and compare inclusively — which is the verdict
the common composition already got, so no merge that happens today stops.
Also folds the three byte-identical co-occurrence-map builders into one indexer
(it now computes degrees too), and shares the trigram Jaccard so the mention's
trigram set is built once per mention rather than once per candidate — the
scoring loop runs up to max_candidates times per mention (GH-3211).
* test(entities): dataset-driven eval of the whole resolution pipeline
The unit tests around entity resolution pin individual pieces — a similarity
number, one scoring branch, one strategy's SQL. #3751 was not any of those: each
piece behaved as designed and the *combination* attributed a new person's facts
to an unrelated country. Nothing was positioned to catch that.
25 cases, each seeding real entities and entity_cooccurrences into a bank and
running the real resolve_entities_batch over them — pg_trgm probe, the partial
index that excludes labels, the candidate cap, scoring, insert. Covers identity
(case, Turkish İ, emoji), variants that must merge, coincidental names that must
not, co-occurrence quality, the recency window, labels, caller-authored names,
in-batch clustering and containment.
Every case runs under both lookup strategies and must agree, unless it declares
otherwise — which turns "full vs trigram is not just a performance choice" from
a claim in the docs into six cases that say exactly where they diverge.
Time is fixed (last_seen is seeded relative to a constant EVENT_DATE), and the
resolver is built from the shipped config rather than constructor defaults, so
the dataset measures a real deployment and moving a threshold shows up here.
Verified it bites: with HINDSIGHT_API_ENTITY_MERGE_MIN_SIMILARITY=0.01 the three
collision cases fail exactly as reported, and only under trigram.
Two cases are xfailed as known limitations rather than quietly given the answer
the code produces:
- Iran/Iraq clears the floor at 0.43, so a genuinely selective shared partner
still merges two distinct entities. Telling "same entity, different spelling"
from "different entity, similar spelling" needs the extractor's type or an
adjudication step, not another threshold.
- A decorated form of a stored name ("Wren 🎵" vs "Wren") has *identical*
trigram sets, but the score reads the sequence ratio (0.80), so it forks into
a second entity once the stored one is a day stale — while the same two forms
DO unify when both are new, since the in-batch pass merges at 0.5 trigram.
#3107 fixed that half only.
* test(entities): move the resolution eval to JSONL and grow it to 61 cases
The dataset was Python literals inside the test module, which put the cases and
the harness in one file and made adding a case a code change. It now lives in
tests/entity_resolution_cases.jsonl, one object per line, and the module is only
a loader plus the runner.
61 passing cases (up from 23), 3 recorded as known limitations. New coverage:
diacritics across four scripts, punctuation as a pg_trgm separator (hyphen,
apostrophe, trailing period), spelling variants, typos, abbreviations, middle
initials, corporate suffixes, near-miss place and person names, the recency
window from both sides, the floor's inclusive boundary (Nick/Nicolas is exactly
0.30), label keys and unseen values, three-way in-batch clustering, containment
in both directions, a crowded candidate field, and bank isolation.
Two things the dataset now guards that it did not before:
- A well-formedness test. A case that co-occurs with a name it never seeded, or
expects the wrong number of answers, would pass while measuring nothing — how
the first draft of this eval nearly shipped blind. It also fails if the file
drops below 50 passing cases.
- The bank-isolation case seeds the other bank with exactly the entity the
mention would merge onto, so a leak changes the answer instead of being masked
by an identical name.
Verified the dataset discriminates in both directions: with
ENTITY_MERGE_MIN_SIMILARITY=0.01 five cases fail (the collisions, one of them
under `full` too, since "Hannah" contains "anna"), and at 0.6 twenty fail as
legitimate variants stop merging. So it brackets the setting rather than only
pinning today's value.
The third known limitation is unchanged in kind but worse than recorded: with
the stored entity seen the same day, Iran/Iraq merges on an *indiscriminate*
partner too, not only a selective one — 0.75 by sequence ratio is 0.375 and
recency alone adds 0.2. The damping case now holds the stored entity outside the
recency window so it pins the damping rather than the clock.
* fix(entities): close the two defects the resolution eval surfaced
Both came out of the dataset in the previous commit, where they were recorded as
known limitations rather than given whatever answer the code produced.
**Different given names sharing a surname merged.** "John Smith" and "Jane
Smith" are 0.47 by trigram and 0.80 by sequence ratio, so two people who share a
surname and a workplace scored as one entity — the shared long word drowns out a
completely different short one. A candidate must now agree word by word: every
word of the shorter name has to find a counterpart in the longer one, by
equality, by abbreviation (corp/corporation, via prefix) or by spelling
(são/sao 0.67, waler/wall 0.67, arbor/arbour 0.91). The cutoff sits at 0.6,
between john/jane at 0.50 and every legitimate word difference above it.
Single-word names are exempt on purpose: with one token the whole-name floor
already IS the token check, and applying this on top would reject real variants
that have no long shared word to hide behind ("Nick"/"Nicolas" is 0.55).
**A decorated form of a stored name forked off it.** "Wren 🎵" against a stored
"Wren" has identical trigram sets — pg_trgm builds trigrams per word, so only the
separators differ — yet the score reads the sequence ratio (0.80 -> 0.40) and
needed history to reach 0.6. The in-batch pass already unifies such names on the
name alone and at a *lower* bar (0.5), so whether two forms were one entity
depended only on whether they arrived in the same retain; #3107 fixed that half
only. Identical trigram sets now reuse the candidate outright.
The candidate-cap test counted SequenceMatcher calls as a proxy for "candidates
scored"; the word-level check calls it too, and a candidate the trigram gate
rejects never reaches the name score, so it now counts the word-level check —
once per scored candidate, used nowhere else. The cap itself is unchanged.
Left as the one known limitation, now with the proof: Iran/Iraq is a single
character apart, and so are Alice/Alise (0.33 trigram) and Sara/Sarah (0.57),
which are real merges scoring below and around it. No character-level threshold
can order "different entity, similar spelling" above "same entity, different
spelling", and the word rule cannot help — both names are one word. That one
needs the extractor's type or an adjudication step.
* docs(entities): correct three comments the resolution change made stale
From code review of the branch, no behaviour change:
- _tokens_are_compatible lowercases its own input now. _trigram_set does, so a
mixed-case caller would have silently disagreed with the gate beside it.
- The resolve=True contract said similar entities "are scored"; there is a floor
in front of the score now, so say which.
- The max_candidates note said scoring is one SequenceMatcher call per
candidate. It is a trigram set per candidate, plus a word-level check and a
SequenceMatcher pass for those clearing the floor — which is also why the
cap's own test no longer counts SequenceMatcher.
Label entities resolve by exact match only, yet their rows were still
covered by the shared trigram index — every fuzzy probe for a regular
entity pulled them into its candidate set only to discard them in the
bitmap recheck. On banks where a free-text label group accumulated tens
of thousands of mutually-similar values this dominated database CPU
under ingest bursts.
- Add entities.entity_kind ('regular'/'label', CHECK-constrained) on
both dialects, set at insert time by the resolver; the Phase-2
reassert carries the kind so a pruned label parent resurrects as a
label.
- Rebuild the PG trigram index as a partial index excluding label rows
(CONCURRENTLY, create-before-drop) and add the matching
entity_kind != 'label' predicate to the trigram candidate query and
the Oracle UTL_MATCH fuzzy scan.
- Migration backfills existing rows per bank by classifying
canonical_name against the bank's entity_labels config with the same
is_label_entity() the resolver uses.
- Fix the label classification gating on the enum lookup set: a config
with only text/map groups builds an empty set, so its labels were
never recognised — neither by the #3187 exact-lookup split nor by the
new insert-time kind.
Bank import needs no changes: transfer archives treat entities as
derived data and re-resolve them through the standard retain Phase 1,
which now stamps the kind.
The bank profile `name` field is documented as a display label only, but at
retain time `_resolve_narrator` silently uses it as the narrator (memory owner)
whenever it differs from `bank_id` — the undocumented coupling reported in #3138.
Stop advertising that path without changing any runtime behavior (100% backward
compatible):
- retain.md now steers speaker attribution solely through each item's `context`,
dropping the advice to set a bank `name` as the agent's name.
- The dry-run extract `agent_name` override is marked `deprecated` in the schema
(still honored) and repointed to `context`.
`_resolve_narrator` and the prompt injection are unchanged, so existing banks
behave exactly as before.
* feat(retain): report zero-fact documents at write time (#3040)
A document whose fact extraction legitimately returns zero facts is stored
but unreachable: only memory_units carry embeddings, so recall and reflect
cannot reach a document that owns none. The retain still succeeds, the
operation reports completed, and nothing in the response, the webhook or
the metrics says the document produced no memories — the operator has no
way to know it needs a reprocess. FAIL_ON_EXTRACTION_ERRORS (#2721) cannot
help by construction: there is no error to fail on.
#2861 made retain.completed fire for zero-fact batches, but the payload is
byte-identical to a successful one, so it still carries no signal.
Add the count to all three write-time surfaces:
- retain.completed gains data.memory_unit_count, filled inside the outbox
callback on the retain's own connection so units written by the enclosing
transaction are visible.
- The synchronous retain response gains memory_units_created.
- New counter hindsight.retain.documents.total{outcome=facts|no_facts},
emitted per document at both extraction exits.
The webhook and the metric report the document's total *after* the retain,
not what the call created: the delta path skips unchanged chunks, so an
idempotent re-retain creates zero units while the document keeps every
memory it had. Reporting units created would raise a false alarm on every
re-submit. The count query only runs when the call created nothing, which
is the path where no work was done anyway.
Docs: how a retain mission trades away retrieval of the raw source, the
three signals, the non-determinism caveat, and reprocess as the way back.
* fix(retain): drop memory_units_created from the retain response
The synchronous response field reported units created by that call, which is
a different number from the one the webhook and the metric report (the
document's total after the retain) and only ever populated on the sync path.
The async path is the one that matters, and it is already covered by
retain.completed carrying data.memory_unit_count.
Removing it also takes the API surface back to identical with main — the
webhook payload is now the only public shape change — so the regenerated
Python/TypeScript/Go clients and the OpenAPI spec carry no delta.
Also renames the metric's parameter to memory_unit_count to match what it is
actually handed: the document total, not units created.
Entity resolution is fuzzy name matching (SequenceMatcher) reinforced by
co-occurrence and temporal proximity — there is no nickname/alias logic in
the resolver. Dissimilar names like 'Bob' and 'Robert Chen' do not unify on
the name alone, so the 'nickname resolution' example was inaccurate. Verified
against hindsight-api-slim/hindsight_api/engine/retain/entity_resolver.py.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Implement Memory Guard Lite for OSS
Allow users to prevent token and secret leakage in agent memory.
feat(memory-defense): reject quarantine action in policy parser
refactor(retain): drop quarantine branch from orchestrator
test(retain): remove quarantine-path tests
refactor(memory-defense): remove DefenseAction.QUARANTINE enum value
refactor(api): remove include_quarantined query parameter
refactor(recall): drop include_quarantined parameter from memory engine
test(memory-defense): replace stale parser-reject test with full-union accept test
The previous parametrized test asserted parse_policy() should 422 on any
detector name other than sensitive_data. That contract was deliberately
widened on 2026-06-07 so cloud-style policies pass through api-slim's
parser unchanged. The test was stale; the runtime is correct.
Replaced with test_parse_policy_accepts_full_detector_union, which proves
the actual contract: all 7 detector names are valid in the parser, with
dispatch and entitlement enforcement deferred to the loaded extension.
Memory defense UI
* i18n labels
* Fix tests
* Client changes to fix breaking tests
* Test fixes
* chore: regenerate API clients via generate-clients.sh
The clients were previously hand-generated in a way that diverged from the
project's tooling — including a non-standard hindsight-clients/typescript/client/
directory the generator never produces (the standard output is typescript/generated/),
plus ~150 spurious files.
Revert the entire hindsight-clients/ tree to main and regenerate from the
OpenAPI spec using ./scripts/generate-clients.sh (Rust via progenitor build.rs,
Python/Go via openapi-generator, TypeScript via @hey-api/openapi-ts). The spec
itself is unchanged (a code-regenerated spec is byte-identical to what was
already committed).
Net result is the real API delta only: the new nullable MemoryItem.receipt_uri
field propagated to the Python, TypeScript and Go models.
* refactor(memory-defense): per-bank regex defense, webhooks, drop dead surface
Review cleanup of the memory-defense feature:
- Rename the OSS extension Lite -> Regex (MemoryDefenseRegexExtension,
memory_defense_regex.py). It is pure regex redaction now.
- Drop the agent_memory_guard (OWASP) dependency entirely — the
SensitiveDataDetector fallback and to_owasp_policy are gone; nothing
cloud-tier remains in api-slim.
- Trim the policy to what OSS enforces: { enabled, rules:[{on:sensitive_data,
action}] }. Removed default_action, protected/immutable namespaces,
detector_overrides, min_severity, and the unused
memory_defense_enabled_default server default. Per-bank override stays
(memory_defense is a configurable field) and the UI writes the trimmed shape.
- Fire a memory_defense.triggered webhook on every non-allow decision (redact
and block) when one is configured, via the retain orchestrator. Adds
WebhookEventType.MEMORY_DEFENSE_TRIGGERED + MemoryDefenseEventData. Replaces
the no-op record_violation hook.
- Block is now actually enforced (drop item / 422 when all blocked) instead of
being silently downgraded to redact.
- Remove the unused 'status' lifecycle: the add_status migration + its two
merge migrations, the recall quarantine filter, the status column reads in
search, and MemoryFact.status. Branch now adds zero migrations (single head).
- Remove receipt_uri from the API (MemoryItem) and clients — it was always
None and carried no value.
Tests updated/renamed accordingly; OWASP smoke + enabled-default tests removed.
* fix(memory-defense): address code-review findings
- Delete test_migration_status.py (asserted the removed status column/constraint).
- Remove receipt_uri from the Rust CLI (memory.rs + integration_test.rs) — the
generated client struct no longer has the field, so it wouldn't compile.
- Type the blocked-violations as a BlockedViolation dataclass instead of raw
dicts (serialized via asdict() in the 422 body); type the webhook helper's
decision param as DefenseDecision.
- Add an end-to-end test asserting a redact decision queues a
memory_defense.triggered webhook delivery.
- Drop the unrelated docs/ entry from .gitignore (local scratch, not this PR).
* test(memory-defense): consolidate into a single test_memory_defense.py
Merge the 10 scattered memory-defense test modules (policy parser, regex
engine/screen, redaction benchmark, extension loader, extension-context
wiring, bank-config validation, and the three retain e2e files) into one
test_memory_defense.py, deduping the overlapping unit screen tests and the
duplicated retain redact e2e. 36 tests, same coverage.
* docs(memory-defense): document memory_defense.triggered webhook + block action
- Add memory_defense.triggered to the control-plane webhook event-type selector
(it was firing but wasn't selectable in the UI).
- Document the memory_defense.triggered event (payload + data fields) on the
webhooks API page, and link it from the Memory Defense page.
- Document the block action (the page only described redact) and add a
Notifications section. Regenerate the docs skill copies.
* docs: remove Memory Defense page from version-0.7 (unreleased feature)
The feature was snapshotted into the 0.7 versioned docs by mistake — 0.7 never
shipped Memory Defense. Remove the page and its (sole) Security sidebar category.
* test(memory-defense): assert webhook payload fields + cover block path
- test_retain_fires_webhook_on_redact now parses the queued delivery and
asserts the MemoryDefenseEventData payload (action/detector/matched_types/
message + event status), not just that the event type was queued.
- Add test_retain_fires_webhook_on_block: a block decision fires the webhook
(before the 422 is raised) with action=block. Confirms the delivery persists
despite the blocked retain returning 422.
- Factor out _memory_defense_webhook_events() helper.
* fix(control-plane): render structured API error details as a string
A blocked retain returns 422 with detail {violations: [{message, ...}]}; the
proxy forwards it as `details` and the client passed that object straight into
the sonner toast, crashing with "Objects are not valid as a React child".
Add describeErrorDetails() to reduce details to a string — joining violation
messages when present (so a Memory Defense block shows e.g. "Sensitive data
pattern matched: aws_access_key"), else JSON-stringifying.
* docs(webhooks): clarify WebhookEvent.status covers the memory_defense action
* feat(memory-defense): record redact/block actions in the audit log
Emit a fire-and-forget 'memory_defense' audit entry for each non-allow decision
(alongside the webhook), with the action/detector/document_id/matched_types in
metadata. Threads the engine's AuditLogger into retain_batch like the webhook
manager; gated by the existing audit_log_enabled switch (off by default).
- Add the memory_defense option to the audit-logs UI action filter + the
actionMemoryDefense i18n key across all locales.
- Document it on the Memory Defense page and the audit-logging config section.
- Test: a redact retain writes a memory_defense audit row with the expected
metadata (audit enabled on the test engine).
---------
Co-authored-by: Chris Latimer <chris.latimer@vectorize.io>
* fix(retain): stop bank_id routing key polluting fact attribution (#1680)
The fact extractor injects a 'Narrator: {banks.name}' line that is stamped
into the who-dimension of every first-person fact (and the observations
consolidated from them). On auto-create banks.name defaults to bank_id, which
is typically a routing key (e.g. my-agent::channel-456::user-789), not a
speaker — so the routing key ends up embedded in stored fact text.
- Suppress the narrator when name == bank_id (_resolve_narrator).
- Make the Context take precedence over the narrator for speaker attribution:
when the Context names a different first-person speaker (a user/customer in a
transcript), those statements are classified 'world' and attributed to that
speaker, not the agent.
Tests: pure unit tests for the suppression + injection logic, and a real-LLM
test (llm_judge) verifying user first-person statements are attributed to the
user as 'world'. The agent-self-log behaviour is unchanged.
* fix(retain): only add Context-precedence clause when context is set
The narrator's 'Context above takes precedence' clause referenced a
'Context: none' line when no context was provided. Gate it on context.
* test+docs: judge fact_type classification; document LLM-judge tests and world/experience facts
- test_narrator_context_override: assert fact_type via LLM judge (not a hard
enum assert), matching the codebase's hs_llm_core pattern.
- CLAUDE.md + code-review skill: document real-LLM + llm_judge tests for any
change to model-interpreted behaviour (classification, attribution, prompts).
- docs/developer/retain.md: clarify world vs experience facts — the split is
by speaker; set the bank name and describe the speaker in context.
* docs: move entity labels detail to memory-banks, simplify retain overview
* docs: move entity labels blurb under entity-recognition section in retain
* docs: update metadata filtering FAQ to cover entity graph retrieval and entity labels tag option
* docs: enable TOC and fix missing separators in FAQ
* docs: add benchmarks leaderboard screenshot and link to models page
* docs: add 'Which model should I use?' FAQ entry with leaderboard screenshot
* docs: fix leaderboard description to cover retain, reflect, and observations
* feat: entity labels
* feat: entity labels — optional, free_values, multi_value, UI polish
Completes the entity labels system:
**Schema & extraction**
- Dynamic Pydantic Labels model per fact: each group becomes a typed
field (Literal | None, list[Literal], str | None, or list[str])
- `optional: bool` flag per group — non-optional enum fields appear in
JSON schema required array so structured-output providers enforce them
- `free_values: bool` flag per group — accepts any LLM-generated string
instead of a predefined enum; example values shown as hints in prompt
- New `is_label_entity()` helper for labels-only mode filtering that
handles both enum lookup and free_values key-prefix matching
- Sentinel rejection: "None"/"null"/"n/a" strings dropped in post-processing
**BM25 / dense retrieval**
- `text_signals` column on memory_units: entity names + date tokens for
enriched BM25 indexing without polluting stored fact text
- Dense embedding includes occurred_end when it differs from occurred_start
- Alembic migration z1u2v3w4x5y6 (merge revision fixing two heads)
**UI (bank-config-view)**
- Shadcn Switch replaces custom Toggle for both entity-labels and observations
- Shadcn Checkbox for multi/optional/free_values per group
- Input heights bumped to h-8 throughout the editor
- "Label Groups" → "Entity Labels", "Free-form entities" → "Entities"
- Free-text groups show "Example hints" banner in values section
**Tests (45 unit + 3 LLM integration)**
- build_labels_model: single, multi, mixed, free_values optional/required/multi
- is_label_entity: enum match, free_values prefix match, no false positives
- Post-processing: null/absent/string-None/free_values/sentinels/multi-value
- Schema: labels in required, structured object, no labels when unconfigured
- LLM integration: single-value enum, multi-value enum, free_values retain
**Docs**
- retain.md: new Entity Labels section covering groups, flags, examples
- configuration.md: retain_free_form_entities env var + entity_labels note
* fix(tests): update hierarchical fields count for entity_labels additions
entity_labels and retain_free_form_entities are hierarchical fields,
bumping the expected count from 11 to 13.
* fix(migration): rename text_signals revision to avoid collision with main
Main branch claimed z1u2v3w4x5y6 for observation_scopes. Rename our
text_signals migration to a2b3c4d5e6f7, chaining after z1u2v3w4x5y6.
* refactor(entity-labels): simplify free_values — always str|None, no multi
- free_values groups always produce str | None (multi_value and optional
flags are ignored for free text groups — always optional, never multi)
- Prompt section for free_values groups shows only key + description,
no values list (users put examples in the description instead)
- UI: section title "Entities", toggle "Free Form Entities", replace
per-group checkboxes with a type dropdown (Enum / Free text); only
show multi checkbox and values list when type is Enum
- Update tests to reflect new behaviour
* refactor(entity-labels): replace free_values/multi_value booleans with type field
- LabelGroup now uses type: "value" | "multi-values" | "text" instead of
free_values/multi_value boolean pair
- Backward-compat migration converts legacy dicts automatically
- Rename retain_free_form_entities → entities_allow_free_form throughout
- Update UI dropdown to show Single value / Multi-values / Free text
- Remove separate multi checkbox (captured by type selection)
- Update docs examples and configuration.md
- Update all tests to use new field names
* fix(migration): backfill observation_scopes column for DBs with swapped z1u2v3w4x5y6
Local DBs that had z1u2v3w4x5y6 applied when it referred to the old
text_signals migration (before it was renamed to a2b3c4d5e6f7) won't have
observation_scopes in their memory_units table. This migration adds the
column with IF NOT EXISTS so it's a no-op on clean installs.
* feat(entity-labels): add tag field to auto-populate memory unit tags from labels
When a LabelGroup has tag=True, extracted key:value entities for that group
are automatically written to the memory unit's tags array. This lets entity
labels double as tags, enabling immediate filtering via the existing
tags/tags_match API params with no extra infrastructure.
- Add tag: bool = False to LabelGroup
- _inject_label_tags() helper called in both sync and batch extraction paths
- UI: add Tag checkbox per label group row
- Docs: document the new tag field
- Tests: 4 new unit tests covering all tag injection paths
* style: ruff format migration file
* fix(migration): fix multiple alembic heads after rebase — point text_signals after nullable_event_date
* fix(clients): update timestamp field to use Timestamp wrapper type after timestamp=unset feature
* style: ruff format agent.py
* fix(docs): update Go quickstart example to use NullableTimestamp for timestamp field
* bump pg0 0.11.x and improve documentation
* bump pg0 0.11.x and improve documentation
* bump pg0 0.11.x and improve documentation
* ci: test notebooks on ci
* ci: test notebooks on ci
* rm llms-full from repo
* formatting
* formatting
Nicolò Boschi