vectorize-io__hindsight/hindsight-docs/docs/developer/retain.md

---
sidebar_position: 2
---

# Retain: How Hindsight Stores Memories

When you call `retain()`, Hindsight transforms conversations and documents into structured, searchable memories that preserve meaning and context.

## What Retain Does

```mermaid
graph LR
    A[Your Content] --> B[Extract Facts]
    B --> C[Identify Entities]
    C --> D[Build Connections]
    D --> E[Memory Bank]
```

---

## Rich Fact Extraction

Hindsight doesn't just store what was said — it captures **why**, **how**, and **what it means**.

### What Gets Captured

When you retain "Alice joined Google last spring and was thrilled about the research opportunities", Hindsight extracts:

**The core facts:**
- Alice joined Google
- This happened last spring

**The emotions and meaning:**
- She was thrilled
- It represented an important opportunity

**The reasoning:**
- She chose it for the research opportunities

This rich extraction means you can later ask "Why did Alice join Google?" and get a meaningful answer, not just "she joined Google."

### Preserving Context

Traditional systems fragment information:
- "Bob suggested Summer Vibes"
- "Alice wanted something unique"
- "They chose Beach Beats"

Hindsight preserves the full narrative:
- "Alice and Bob discussed naming their summer party playlist. Bob suggested 'Summer Vibes' because it's catchy, but Alice wanted something unique. They ultimately decided on 'Beach Beats' for its playful tone."

This means search results include the full context, not disconnected fragments.

---

## Two Types of Facts

Every fact is classified by **whose perspective it captures** — the agent that owns the bank, or the outside world:

| Type           | What it captures                                                              | Example |
|----------------|------------------------------------------------------------------------------|---------|
| **experience** | The bank's own agent acting, observing, or interacting — its first-person history | "I recommended Python to Alice" |
| **world**      | Facts about other people, places, things, and events                          | "Alice works at Google" |

The split is decided by **who is speaking**, not by grammar. A first-person statement is an `experience` only when the speaker *is* the bank's agent. The same words said by someone else are a `world` fact about that person:

- Agent's own log — "I patched the auth bug" → **experience** (the agent did it).
- A user talking to the agent — "I bought a Tesla" → **world** (a fact about the *user*, not the agent).

**Describe the speaker in each item's `context`** to steer this correctly. When retaining transcripts or third-party content, a context like *"Customer Maria is speaking"* ensures her first-person statements are stored as `world` facts about Maria rather than mistaken for the agent's own experiences. For the agent's own logs, a context like *"The assistant is speaking"* attributes its first-person statements to the agent as `experience` facts.

**Note:** Observations are consolidated automatically in the background after `retain()` operations complete. This consolidation process synthesizes patterns from new facts into the bank's knowledge base.

---

## Entity Recognition

Hindsight automatically identifies and tracks **entities** — the people, organizations, and concepts that matter.

### What Gets Recognized

- **People:** "Alice", "Dr. Smith", "Bob Chen"
- **Organizations:** "Google", "MIT", "OpenAI"
- **Places:** "Paris", "Central Park", "California"
- **Products & Concepts:** "Python", "TensorFlow", "machine learning"

### Entity Resolution

The same entity mentioned different ways gets unified through **fuzzy name matching**, reinforced by co-occurrence and temporal proximity:
- "Alice" + "Alice Chen" + "Alice C." → one person

Because resolution keys off name similarity, close variants merge automatically. Names that do not resemble each other (a nickname and an unrelated formal name, for example) are not unified on the name alone, though shared co-occurring entities can still link them.

**Why it matters:** You can ask "What do I know about Alice?" and get everything, even if she was mentioned as "Alice Chen" in some conversations.

Resolution is a judgement call, so it can go the other way too: on a bank with a lot of history, a short new name that resembles an existing entity — and that turns up alongside entities the existing one is already associated with — can be absorbed into it instead of becoming its own entity. If you see facts about a new person attached to an unrelated entity, [How entity resolution decides](/developer/configuration#how-entity-resolution-decides) explains what is being compared and which setting makes matching stricter.

### Context-Aware Disambiguation

If "Alice" appears with "Google" and "Stanford" multiple times, a new "Alice" mentioning those is likely the same person. Hindsight uses co-occurrence patterns to disambiguate common names.

### Entity Labels

You can define a controlled vocabulary of `key:value` classification labels (e.g. `pedagogy:scaffolding`, `engagement:active`) that are extracted at retain time and stored as entities. Because labels become entities, they automatically link related memories in the knowledge graph and improve both semantic and keyword retrieval. Labels can optionally also write to the memory unit's tags, enabling standard tag-based filtering during recall and reflect.

Unlike regular entities, label entities never merge by name similarity — distinct label values must stay distinct, so they resolve by exact match only and are excluded from fuzzy name matching altogether.

See [entity_labels in the bank config](/developer/api/memory-banks#entity-labels) for full configuration details.

---

## Building Connections

Memories aren't isolated — Hindsight creates a **knowledge graph** with four types of connections:

### Entity Connections

All facts mentioning the same entity are linked together.

**Enables:** "Tell me everything about Alice" → retrieves all Alice-related facts

### Time-Based Connections

Facts close in time are connected, with stronger links for closer dates.

**Enables:** "What else happened around then?" → finds contextually related events

### Meaning-Based Connections

Semantically similar facts are linked, even if they use different words.

**Enables:** "Tell me about similar topics" → finds thematically related information

### Causal Connections

Cause-effect relationships are explicitly tracked.

**Enables:** "Why did this happen?" → trace reasoning chains
**Example:** "Alice felt burned out" ← caused by ← "She worked 80-hour weeks"

---

## Understanding Time

Hindsight tracks **two temporal dimensions**:

### When It Happened

For events (meetings, trips, milestones), Hindsight records when they occurred.
- "Alice got married in June 2024" → occurred in June 2024

For general facts (preferences, characteristics), there's no specific occurrence time.
- "Alice prefers Python" → ongoing preference

### When You Learned It

Hindsight also tracks when you told it each fact.

**Why both?**

Imagine in January 2025, someone tells you "Alice got married in June 2024":
- **Historical queries** work: "What did Alice do in 2024?" → finds the marriage
- **Recency ranking** works: Recent mentions get priority in search
- **Temporal reasoning** works: "What happened before her marriage?" → finds earlier events

Without this distinction, old information would either be unsearchable by date or treated as irrelevant.

---

## Tagging Memories

Tags enable visibility scoping—useful when one memory bank serves multiple users but each should only see relevant memories.

- **Item tags**: Tag individual memories with specific scopes
- **Document tags**: Apply tags to all items in a batch
- **Tag filtering**: Filter during recall/reflect by tags

See [Retain API](./api/retain) for code examples and [Recall API](./api/recall) for filtering options.

---

## Images and Files in Your Content

A lot of what a document actually says lives in its pictures: the screenshot of
the button an instruction refers to, the diagram holding an escalation path, the
chart that *is* the data. `content` accepts an ordered list of blocks so those sit
where they belong, instead of being summarised into a caption beforehand:

```json
{
  "content": [
    {"type": "text",  "text": "To reset the VPN, click the button shown:"},
    {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": "..."}},
    {"type": "text",  "text": "...then reconnect."}
  ]
}
```

A plain string still works exactly as before — nothing about text-only retain
changes.

The point is **position**. Extraction sends the text and the pictures together,
in order, so the model reads the screenshot beside the sentence that introduces
it. "Click the button shown below" is meaningless on its own; next to the
picture it becomes a fact that names the button.

### What comes back

Facts read naturally — `[image: image/png]` marks where an attachment was, never
a content hash — and each memory carries the attachments **it was drawn from**,
with a URL you can fetch:

```json
{
  "text": "The escalation path for a stuck sync begins by contacting Tier 3 Platform.",
  "attachments": [
    {"id": "c414cd0e204d", "kind": "image", "media_type": "image/png",
     "url": "/v1/default/banks/my-bank/attachments/c414cd0e204d"}
  ]
}
```

Facts that came from the prose have no `attachments`, even when the same document
is full of pictures. So an attachment shown next to a memory means the model
looked at it to produce that memory — it is evidence, not decoration.

### What to expect from charts and tables

An attachment carrying structured data is transcribed rather than summarised:
each row, bar or labelled value becomes its own fact, carrying its label, its
figure, and how it is drawn. A chart therefore produces many more memories than a
screenshot does — that is deliberate, since a summary of a twenty-bar chart keeps
three values and silently drops seventeen, and later questions ask about the ones
it dropped.

Very dense pages are still sampled rather than exhausted: a single extraction pass
over an infographic listing a hundred entries will capture a large share of them,
not all of them.

### Requirements

- A model that can read images. By default that is the bank's retain model, but
  it does not have to be: `HINDSIGHT_API_VLM_MODEL` names a **vision slot** used
  only for the chunks that actually carry an attachment, leaving every text-only
  chunk on the retain LLM. So a bank whose documents are mostly prose can keep a
  cheap text model and pay for vision only where there is something to look at.
- If that model cannot read images — or if Hindsight cannot tell, which is the
  case for gateway backends serving mixed catalogues — the retain is refused
  with `422` rather than dropping the attachment silently. See
  [`HINDSIGHT_API_LLM_VISION`](/developer/configuration#llm-configuration).
- Batch retain (`HINDSIGHT_API_RETAIN_BATCH_ENABLED`) cannot carry attachments,
  and also refuses with `422`.

This is different from [`POST /files/retain`](/developer/configuration#file-conversion),
which converts a whole file to markdown as its **own** document. That is still the
right tool for a scanned report you want parsed — but it separates the file from
the prose that referred to it, which is exactly what inline attachments avoid.

For storage, size limits and accepted media types, see
[Inline attachments in retain](/developer/configuration#inline-attachments-in-retain).

---

## What You Get

After `retain()` completes:

- **Structured facts** that preserve meaning, emotions, and reasoning
- **Unified entities** that resolve different name variations
- **Knowledge graph** with entity, temporal, semantic, and causal links
- **Temporal grounding** for both historical and recency-based queries
- **Optional tags** for filtering during recall

All stored in your isolated **memory bank**, ready for `recall()` and `reflect()`.

---

## Steering Extraction with a Mission

By default, `retain()` extracts all significant facts from the content. You can narrow this focus with a **retain mission** (`retain_mission`) — a plain-language description of what this bank should pay attention to.

```
e.g. Always include technical decisions, API design choices, and architectural trade-offs.
     Ignore meeting logistics, greetings, and social exchanges.
```

The mission is injected into the extraction prompt alongside the built-in rules — it steers the LLM without replacing the extraction logic. It works with the LLM-based extraction modes (`concise`, `verbose`, `verbatim`, `custom`) and is ignored in `chunks` mode.

For finer control, you can also change the **extraction mode**:

| Mode | When to use |
|------|-------------|
| `concise` *(default)* | General-purpose — selective, fast |
| `verbose` | When you need richer facts with full context and relationships |
| `custom` | When you want to write your own extraction rules entirely |
| `verbatim` | When you need the original chunk text preserved, with LLM-extracted metadata such as entities and dates |
| `chunks` | When you want to store chunks as-is with no LLM call or extracted metadata |

Set `retain_mission` and `retain_extraction_mode` via the [bank config API](/developer/api/memory-banks#retain-configuration), or with the [`HINDSIGHT_API_RETAIN_MISSION`](/developer/configuration#retain) and [`HINDSIGHT_API_RETAIN_EXTRACTION_MODE`](/developer/configuration#retain) environment variables.

### When a mission excludes everything in a document

A mission narrows what becomes a memory — and content that produces no facts produces no memories at all. The document itself is still stored, but `recall` and `reflect` search memories, so a document with zero memories cannot be found by either. Tightening a mission therefore trades away retrieval of the raw source, not just fact creation.

This is a normal outcome, not an error: the retain succeeds and the operation is reported as completed. Two signals tell you it happened:

| Where | What to look for |
|-------|------------------|
| [`retain.completed` webhook](/developer/api/webhooks#retaincompleted) | `data.memory_unit_count: 0` |
| [Metrics](/developer/monitoring#retain-metrics) | `hindsight.retain.documents.total{outcome="no_facts"}` |

You can also audit after the fact: `GET /documents` returns `memory_unit_count` per document, so filtering for `0` lists everything currently unreachable.

Extraction is not fully deterministic — a borderline document can yield facts on one run and none on the next. Treat a zero as "this document needs another pass" rather than as a permanent verdict.

To recover a document, widen the mission and reprocess it — the stored text is re-extracted, and no re-upload is needed:

```
POST /v1/default/banks/{bank_id}/documents/{document_id}/reprocess
```

---

## Observation Consolidation

After `retain()` completes, Hindsight automatically triggers **observation consolidation** in the background. This process:

1. Analyzes new facts against existing observations
2. Creates new observations when patterns emerge
3. Refines existing observations with new evidence
4. Tracks which facts support each observation

This happens asynchronously — your `retain()` call returns immediately while consolidation runs in the background.

See [Observations](./observations) for details on how consolidation works.

---

## Memory Defense and Source Provenance

### receipt_uri (optional)

Type: `string`.

Optional pointer into an external receipt or co-signature system. Stored as-is and surfaced in `security_events.receipt_uri` for any Memory Defense decision on this item.

### 422 — Memory Defense violation

When Memory Defense is enabled on the target bank and **every** item in the batch is blocked by policy, the request returns 422 with a violation list:

```json
{
  "detail": {
    "violations": [
      { "index": 0, "detector": "prompt_injection", "severity": "high", "message": "..." }
    ]
  }
}
```

Partial-block batches return 200 with the un-blocked items processed; blocked items are silently dropped from the result with their decisions recorded in `security_events`.

See [Memory Defense](./memory-defense/index.md) for the full guide.

---

## Next Steps

- [**Observations**](./observations) — How knowledge is consolidated after retain
- [**Recall**](./retrieval) — How multi-strategy search retrieves relevant memories
- [**Reflect**](./reflect) — How the agentic loop uses observations
- [**Retain API**](./api/retain) — Code examples and parameters