vectorize-io__hindsight/hindsight-docs/docs/developer/api/operations.mdx

---
sidebar_position: 9
---

# Operations

Hindsight runs several maintenance and ingestion tasks asynchronously instead of blocking the API call that triggers them. These tasks share a single queue (`async_operations`) and a single worker pool, and the same REST endpoints — list, status, cancel, retry — work across every type.

This page explains each operation type, when it fires, and how to inspect or manage it.

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import CodeSnippet from '@site/src/components/CodeSnippet';

{/* Import raw source files */}
import operationsPy from '!!raw-loader!@site/examples/api/operations.py';
import operationsMjs from '!!raw-loader!@site/examples/api/operations.mjs';
import operationsSh from '!!raw-loader!@site/examples/api/operations.sh';
import operationsGo from '!!raw-loader!@site/examples/api/operations.go';

:::tip Prerequisites
Make sure you've completed the [Quick Start](./quickstart) and understand [how retain works](./retain).
:::

## How operations work

When an API call needs background work, the request handler writes a row to the `async_operations` table with `status=pending` and returns immediately. A worker (running either in-process inside the API by default, or as a dedicated service — see [Services - Worker Service](../services#worker-service)) polls the table, claims pending rows, executes the corresponding handler, and marks the row `completed` or `failed`.

By default, every operation runs in-process: no external queue, no extra process to deploy. The same code paths support scaling out to dedicated worker processes when throughput demands it.

### Lifecycle

| Status | Meaning |
|--------|---------|
| `pending` | The row is queued. Either no worker has picked it up yet, or an extension has parked it via `next_retry_at` in the future (e.g., for backpressure). |
| `processing` | A worker has claimed the row and is actively running the handler. |
| `completed` | The handler returned successfully. |
| `failed` | The handler raised. `error_message` carries the reason; you can re-queue with `POST /…/retry`. |
| `cancelled` | The operation was cancelled via `DELETE /…/operations/{id}`. Works on `pending` and `processing` operations alike. |

The worker retries failed operations up to `HINDSIGHT_API_WORKER_MAX_RETRIES` times before settling on `failed`. Deterministic failures (e.g., invalid embedding dimensions, integrity violations) skip retries — they won't succeed by re-running.

Completed, failed, and cancelled operations are kept indefinitely by default. Set `HINDSIGHT_API_OPERATION_RETENTION_DAYS` to a positive number of days to bound that history: the background maintenance loop then prunes expired terminal rows in bounded batches, on its own schedule rather than as a side effect of task processing. PostgreSQL only — the maintenance loop does not run on Oracle, so operation history is unbounded there. The full row shares that TTL, so while an operation is retained its payload stays available — failed and cancelled operations can be retried, and completed ones inspected with `include_payload=true`. Pending and processing operations are never removed by retention cleanup.

## Operation types

Every operation has an `operation_type` in the database and a `task_type` in the payload. They're usually the same.

### `retain`

Submitted by `POST /v1/default/banks/{bank_id}/memories` with `async=true`, or by the multi-item `retain_batch` call. The handler runs the same pipeline as a synchronous retain: fact extraction (LLM), embedding generation, entity resolution, and link creation (temporal, semantic).

Use async retain when you're ingesting thousands of items and don't want the HTTP call to hold for minutes. The `operation_id` in the response lets you poll for completion.

#### Parent op: `retain_batch`

For large submissions, Hindsight automatically splits the input into sub-batches and creates a single `retain_batch` parent operation that tracks the children. The parent's status reflects the aggregate — `pending` until at least one child is running, `processing` while children execute, `completed` once every child has finished, `failed` if any child failed. Each child is itself a `retain` operation linked to the parent, so you can drill in for per-batch error messages.

When you list operations, the parent and its children all appear by default. Pass `exclude_parents=true` to hide the aggregate rows and show only individual `retain` jobs.

### `file_convert_retain`

Submitted by file upload endpoints. The handler runs MIME-specific conversion (PDF → text, DOCX → text, etc.) and then passes the extracted text into the retain pipeline. Failures here are **non-retryable** by default — a corrupted PDF or missing OCR won't improve on rerun, so the operation goes straight to `failed`.

Which parser runs (`markitdown`, `iris`, or `llama_parse`) is selected per deployment via `HINDSIGHT_API_FILE_PARSER`, and clients can override it per request — see [Configuration → File Processing](../configuration#file-processing).

### `consolidation`

Produces **observations** from new world/experience memories. See [Observations](../observations) for what they are and how they're synthesized.

Triggered automatically:

- After every retain that added world/experience facts (gated by per-bank `enable_auto_consolidation` and `enable_observations`).
- After deletes that invalidated existing observations (the source memory disappeared → derived observations are stale → re-run with the surviving co-source memories).
- Manually via `POST /v1/default/banks/{bank_id}/consolidate`. Pass `observation_scopes` to consolidate only memories matching specific tag combinations.

**Bank-deduped**: while one `consolidation` job is pending for a bank, repeat submits return the existing `operation_id` instead of stacking. Once the job starts processing, the next submit becomes the next pending slot.

### `refresh_mental_model`

A mental model has a `source_query` that defines which memories it summarizes. The handler re-runs that query, re-summarizes the result, and updates the model's content in place.

Triggered either manually via `POST /v1/default/banks/{bank_id}/mental-models/{id}/refresh`, or automatically by the auto-refresh schedule for mental models that have one configured.

### `graph_maintenance`

Reconciles derived state that goes stale after a delete. Every invocation drains two queues, both filled by the delete itself, so a run only ever looks at what that delete touched:

1. **Link top-up.** Drains the units whose outgoing temporal/semantic links lost a neighbour. For each, if the unit is under its cap (20 temporal, 50 semantic), Hindsight re-runs the same probes retain uses and inserts the missing links. Without this, the retain pipeline's top-K capping would leave surviving units permanently under-capped after every delete — degrading graph-expansion recall.
2. **Entity prune.** Drains the entities the delete may have stranded. Those with no remaining `unit_entities` reference are deleted (FK `ON DELETE CASCADE` removes their `entity_cooccurrences` rows with them); for the survivors, cooccurrence rows where both endpoints still exist but no current memory_unit references both are cleaned up — the cooccurrence was real when recorded, but every unit that witnessed it has since been deleted.

Bank-deduped at submit time, so concurrent triggers against the same bank coalesce into one drain.

Each run works in committed batches under a wall-clock budget. A backlog too large for one run — a bulk delete, say — is not an error: the run reports what it finished and the next one resumes where it stopped.

**Triggers:** any delete that removes memory_units — `DELETE /documents/{id}`, `DELETE /memories/{id}`, and re-retaining an existing `document_id` (the upsert path). A full bank wipe (`delete_bank`) is a no-op: there's nothing left in the bank to maintain.

### `webhook_delivery`

After certain operations complete (e.g., consolidation finishing on a bank with a registered webhook), Hindsight enqueues a `webhook_delivery` task. The handler POSTs the payload to the configured URL and retries on transient failures.

## Endpoints

All paths below are scoped by `bank_id`.

### List operations

```bash
GET /v1/default/banks/{bank_id}/operations
```

Query parameters:

| Param | Description |
|-------|-------------|
| `status` | Filter by `pending`, `processing`, `completed`, `failed`, `cancelled`. |
| `type` | Filter by `retain`, `file_convert_retain`, `consolidation`, `refresh_mental_model`, `graph_maintenance`, `webhook_delivery`. |
| `limit` | 1–100, default 20. |
| `offset` | Pagination offset. |
| `exclude_parents` | Exclude parent batch operations from results (large `retain_batch` calls create one parent + N children). |

<Tabs>
<TabItem value="python" label="Python">
<CodeSnippet code={operationsPy} section="operations-list" language="python" />
</TabItem>
<TabItem value="node" label="Node.js">
<CodeSnippet code={operationsMjs} section="operations-list" language="javascript" />
</TabItem>
<TabItem value="cli" label="CLI">
<CodeSnippet code={operationsSh} section="operations-list" language="bash" />
</TabItem>
<TabItem value="go" label="Go">
<CodeSnippet code={operationsGo} section="operations-list" language="go" />
</TabItem>
</Tabs>

`items_count` is operation-specific — non-zero only for retain-shaped operations (it counts content items in the submission).

### Get operation status

<Tabs>
<TabItem value="python" label="Python">
<CodeSnippet code={operationsPy} section="operations-get" language="python" />
</TabItem>
<TabItem value="node" label="Node.js">
<CodeSnippet code={operationsMjs} section="operations-get" language="javascript" />
</TabItem>
<TabItem value="cli" label="CLI">
<CodeSnippet code={operationsSh} section="operations-get" language="bash" />
</TabItem>
<TabItem value="go" label="Go">
<CodeSnippet code={operationsGo} section="operations-get" language="go" />
</TabItem>
</Tabs>

Query parameters:

| Param | Description |
|-------|-------------|
| `include_payload` | Include the raw task payload (the submission params) in the response as `task_payload`. Default `false`; may be large. |

A few response fields are worth calling out:

| Field | Description |
|-------|-------------|
| `updated_at` | When the operation's row last changed — claim, progress heartbeat, or completion. |
| `progress` | Last-known progress snapshot for a running operation, or `null` if none was recorded (completed-instantly or pre-feature rows). |
| `task_payload` | The raw submission params; only populated when `include_payload=true`. |

`progress` is written at coarse phase/batch boundaries (consolidation, batch retain) and lets you tell a healthy long-running job from a frozen one: if `processed` keeps advancing across polls the job is alive; identical numbers with no movement in `at` mean it's stuck. Its shape:

| Field | Description |
|-------|-------------|
| `stage` | Coarse phase the operation last reported (e.g. `processing_batch`). |
| `at` | ISO-8601 timestamp when this snapshot was written. |
| `processed` | Units of work finished so far (sub-batches, memories), when known. |
| `total` | Total units of work for the operation, when known. |
| `detail` | Operation-specific counters (e.g. `observations_created`, `round`, `items_in_sub_batch`). |

### Cancel an operation

Cancels a `pending` or `processing` operation. Returns `409` if it has already reached a
terminal state (`completed`, `failed`, `cancelled`).

The row is marked `cancelled` straight away, but cancelling running work is **cooperative
and not immediate**. A worker executing the operation notices at its next checkpoint — the
boundary between sub-batches or documents for retain, between LLM batches for consolidation —
and stops there, so whatever it had already committed stays committed and the batch in
progress may still finish. Operation types without checkpoints run to the end; the row stays
`cancelled` either way, because no worker write may overwrite that status.

This is also how you clear an operation stranded in `processing` by a worker that was killed
before it could finish: nothing is running, so the cancel takes effect immediately. Use
`POST /…/operations/{id}/retry` to re-queue the work afterwards.

<Tabs>
<TabItem value="python" label="Python">
<CodeSnippet code={operationsPy} section="operations-cancel" language="python" />
</TabItem>
<TabItem value="node" label="Node.js">
<CodeSnippet code={operationsMjs} section="operations-cancel" language="javascript" />
</TabItem>
<TabItem value="cli" label="CLI">
<CodeSnippet code={operationsSh} section="operations-cancel" language="bash" />
</TabItem>
<TabItem value="go" label="Go">
<CodeSnippet code={operationsGo} section="operations-cancel" language="go" />
</TabItem>
</Tabs>

### Retry a failed operation

The row's status resets to `pending` and the worker picks it up again. Returns `409` if the operation isn't in `failed` or `cancelled` state.

<Tabs>
<TabItem value="python" label="Python">
<CodeSnippet code={operationsPy} section="operations-retry" language="python" />
</TabItem>
<TabItem value="node" label="Node.js">
<CodeSnippet code={operationsMjs} section="operations-retry" language="javascript" />
</TabItem>
<TabItem value="cli" label="CLI">
<CodeSnippet code={operationsSh} section="operations-retry" language="bash" />
</TabItem>
<TabItem value="go" label="Go">
<CodeSnippet code={operationsGo} section="operations-retry" language="go" />
</TabItem>
</Tabs>

## Async retain example

Submit a batch asynchronously and poll until the operation completes:

<Tabs>
<TabItem value="python" label="Python">
<CodeSnippet code={operationsPy} section="operations-async-retain" language="python" />
</TabItem>
<TabItem value="node" label="Node.js">
<CodeSnippet code={operationsMjs} section="operations-async-retain" language="javascript" />
</TabItem>
<TabItem value="cli" label="CLI">
<CodeSnippet code={operationsSh} section="operations-async-retain" language="bash" />
</TabItem>
<TabItem value="go" label="Go">
<CodeSnippet code={operationsGo} section="operations-async-retain" language="go" />
</TabItem>
</Tabs>

## Worker tuning

Each worker has a single concurrency budget (`HINDSIGHT_API_WORKER_MAX_SLOTS`, default 10) shared across all operation types. Per-type slot reservations (`HINDSIGHT_API_WORKER_<TYPE>_MAX_SLOTS`) carve out guaranteed capacity within that budget; remaining slots form a shared pool any type can use. See [Configuration → Worker Configuration](../configuration#distributed-workers) for the full table.

For most deployments the defaults are fine. Reserve slots for an operation type if you've seen it starved by a flood of another type (e.g., a long file_convert_retain blocking graph_maintenance on a deletion-heavy workload).

Slots are also rotated across banks. Each claim serves the next bank in turn — one operation — then fills the rest of the pool oldest-first from anywhere. So a bank ingesting in bulk cannot own the whole pool while another bank's single write waits behind its backlog, and it is not throttled either: when no one else is waiting it still takes every slot.

## Next Steps

- [**Documents**](./documents) — Track document sources
- [**Memory Banks**](./memory-banks) — Configure bank settings