skills/vectorize-io__hindsight
1
0
Fork 0
mirror of https://github.com/vectorize-io/hindsight.git synced 2026-09-14 19:31:49 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
main
vectorize-io__hindsight/hindsight-docs/docs/developer/api/knowledge-pages.mdx
T
Nicolò Boschi 99a319513f fix(knowledge-pages): move a page's scope onto its mental model, and stop trigger patches resetting it (#3687) (#3755) 
* docs(knowledge-pages): say that a page's tags are a filter, and let the dialog widen it (#3687)

A page's `tags` are the scope it is synthesized from, not labels on it, and a
tagged page matches with `all_strict` by default: a memory must carry EVERY tag
and untagged memories are excluded outright. Nothing said so. The create-page
API documented `tags` as "tags that scope which memories the page is built
from" with no match mode, and `type:<x>` is documented as setting the page's
rendered *type* — so a page created with `["type:runbook", "homelab",
"infrastructure"]` silently required all three on every memory, matched nothing,
and generated as "I don't have information about this" while a plain recall for
the same query returned 81 results scoring 1.087.

The server default stays. What changes is that you can now see it and move it:

- The control-plane create/edit dialog states the rule under the tags field and,
  once there is a tag to widen, offers "Also build from untagged memories" ->
  `trigger.tags_match: "all"`. The edit dialog prefills from the page's stored
  mode and only sends `tags_match` when the checkbox actually moved, so a page on
  `any`/`any_strict`/`exact` survives a rename untouched.
- `CreatePageRequest.tags`, `UpdateNodeRequest.tags` and
  `MentalModelTrigger.tags_match` carry the rule in the OpenAPI spec, so it
  reaches Swagger UI and the generated SDKs; both hand-written wrappers (TS
  `HindsightClient`, Python `Hindsight`) get it in parity.
- A "Tags Are a Filter" section in the API docs works the reported page through,
  with the three ways to scope one and the `PATCH {"tags": []}` repair.

Also corrects a stale claim in the same three places: a supplied `trigger` has
merged over the page defaults since #3506, but the docs and both wrapper
docstrings still said it REPLACES them and told you to repeat the fields you
wanted to keep. Under that text the fix above reads as destructive.

* fix(knowledge-pages): move a page's scope onto its mental model, and stop trigger patches resetting it (#3687)

Two changes, one subject: where a knowledge page's retrieval scope is edited,
and whether editing it destroys the rest of the page's configuration.

1. The page dialogs no longer take tags.

   `tags` on a page are the scope it is synthesized FROM, and a tagged page
   matches with `all_strict`: every tag required, untagged memories excluded.
   The create dialog offered a bare text input for them next to a hint that a
   `type:<x>` tag "sets the page's type" — so tags typed there to describe a
   topic silently became a hard filter that matched nothing, and the page
   generated as "I don't have information about this".

   A partial copy of the scope controls was the problem, so the copy is gone
   rather than extended: the page dialogs now take a name and a source query,
   and the page links to its backing mental model, which already owns the whole
   scope (tags, tags_match, tag_groups, fact types, schedule). Pages are created
   reading the whole bank; narrowing one is now a deliberate step taken where the
   match mode is visible next to the tags it governs.

   The edit dialog no longer sends `tags` at all. That is load-bearing: the PATCH
   applies only the keys present, so a dialog that no longer shows the field must
   not send it, or every rename would clear the page's scope.

2. `PATCH /mental-models/{id}` patches its trigger instead of replacing it.

   Which matters much more now that it is where pages send people. The route
   dumped the whole request model and the engine wrote that dict wholesale, so
   setting one field stamped `MentalModelTrigger`'s own defaults over every field
   left unset: a page edited there lost `mode: delta`, its observation-only
   `fact_types`, and `exclude_mental_models`, quietly becoming a from-scratch
   rebuild that also reflected over its sibling pages.

   #3506 fixed exactly this, but only on the two page routes. The MCP
   `update_mental_model` tool drives the same endpoint with a one-key dict
   (`{"refresh_after_consolidation": ...}`), so it was wiping triggers too.
   `_merge_page_trigger` is renamed `_merge_trigger` and is now the shared merge
   for any mental model; a full trigger still replaces, which keeps bank-template
   import declarative.

Verified against a live API: a page created through the UI, opened via its new
"Advanced options" link and saved from the mental-model editor, keeps
`mode: delta`, `fact_types: [observation]`, `exclude_mental_models` and
`refresh_after_consolidation`. Four of the five new regression tests fail without
the engine change; the fifth is the guard that a complete trigger still replaces.

* fix(knowledge-pages): open the backing mental model in place, and name it in the link (#3687)

The link opened a page's scope editor by navigating to the Mental Models tab.
Changing a page's scope is part of working on the page, so sending the reader to
another tab — losing the page they were reading, and their place in the tree —
was the wrong trade for a shareable URL.

It now opens the same detail modal in place, on its configuration, and hands off
to the very same `UpdateMentalModelDialog` the Mental Models tab uses (exported
for it, not copied). Saving re-pulls the tree and the open page, since tags and
trigger drive the chips and the freshness line.

The label says "Mental model options" rather than "Advanced options": what opens
is that model, and the page's scope living on a mental model is exactly the thing
a reader needs told.

The ?mentalModel= deep link added for the navigation is removed with it — nothing
links there now, and a query param nothing produces is a trap for the next reader.

* refactor(knowledge-pages): name the options i18n key for what it opens (#3687)

The key was still `advancedOptions` after the label became "Mental model
options", and the refresh callback asserted `currentBank!` where guarding is
free.
2026-08-24 14:06:12 +02:00

432 lines
18 KiB
Plaintext
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
sidebar_position: 5
---
# Knowledge Pages
Living markdown documents, organized in a folder tree, that rewrite themselves as the bank learns.
A page is backed by a [mental model](./mental-models) but is configured as a document: it is built from the bank's [observations](../observations) only, it never reads other pages, and it refreshes incrementally after each consolidation. See [Knowledge Pages](../knowledge-pages) for the concepts behind the API.
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import CodeSnippet from '@site/src/components/CodeSnippet';
{/* Import raw source files */}
import knowledgePagesPy from '!!raw-loader!@site/examples/api/knowledge-pages.py';
import knowledgePagesMjs from '!!raw-loader!@site/examples/api/knowledge-pages.mjs';
import knowledgePagesSh from '!!raw-loader!@site/examples/api/knowledge-pages.sh';
import knowledgePagesGo from '!!raw-loader!@site/examples/api/knowledge-pages.go';
All endpoints below are relative to a bank:
```
/v1/default/banks/{bank_id}/knowledge-base
```
---
## Get the Tree
Returns the whole knowledge base as a nested tree of folders and pages. Page bodies are **not** included — fetch a page to read its content.
<Tabs>
<TabItem value="python" label="Python">
<CodeSnippet code={knowledgePagesPy} section="get-tree" language="python" />
</TabItem>
<TabItem value="node" label="Node.js">
<CodeSnippet code={knowledgePagesMjs} section="get-tree" language="javascript" />
</TabItem>
<TabItem value="cli" label="CLI">
<CodeSnippet code={knowledgePagesSh} section="get-tree" language="bash" />
</TabItem>
<TabItem value="go" label="Go">
<CodeSnippet code={knowledgePagesGo} section="get-tree" language="go" />
</TabItem>
</Tabs>
```json
{
"roots": [
{
"id": "kf-9f2c...",
"kind": "folder",
"name": "Operations",
"parent_id": null,
"mental_model_id": null,
"managed": false,
"description": null,
"tags": [],
"timestamp": "2026-08-01T11:04:02+00:00",
"is_stale": null,
"children": [
{
"id": "kp-2e85...",
"kind": "page",
"name": "Deploying the API",
"parent_id": "kf-9f2c...",
"mental_model_id": "mm-77ab...",
"managed": false,
"description": "How is the API deployed?",
"tags": ["ops"],
"timestamp": "2026-08-03T09:12:44+00:00",
"is_stale": true,
"children": []
}
]
}
]
}
```
| Field | Description |
|---|---|
| `kind` | `folder` or `page` |
| `mental_model_id` | The backing mental model (pages only) |
| `description` | The page's source query — the question that rebuilds it |
| `timestamp` | Last refresh for a page, last update for a folder |
| `is_stale` | Pages only: `true` when a memory in *this page's* scope has been written since the page last read the memories (see below) |
| `managed` | `true` when the node is flagged as system-owned rather than hand-authored |
### How `is_stale` is decided
Each page is answered against its own scope — its tags and its `fact_types` — using the same [staleness check](./mental-models#staleness-gating) that decides whether a scheduled refresh does any work. A flagged page is one a refresh would actually rewrite; an unflagged one is a page a refresh would leave alone. Activity elsewhere in the bank does not flag a page whose own scope is quiet.
The whole tree is answered in one query, so the flag costs the same whether the bank has three pages or three hundred, and [`GET /mental-models/{id}`](./mental-models) returns the identical value for the page's backing model.
One thing it does not see: **deletions**. The check asks what has been *written* since the page last read the memories, and deleting an in-scope memory leaves no write behind — a page that cites a deleted fact keeps reporting itself up to date.
---
## Create a Page
Creating a page stores it with placeholder content and schedules the first build in the background. Poll the returned `operation_id` via the [operations API](./operations) to know when the content is ready.
<Tabs>
<TabItem value="python" label="Python">
<CodeSnippet code={knowledgePagesPy} section="create-page" language="python" />
</TabItem>
<TabItem value="node" label="Node.js">
<CodeSnippet code={knowledgePagesMjs} section="create-page" language="javascript" />
</TabItem>
<TabItem value="cli" label="CLI">
<CodeSnippet code={knowledgePagesSh} section="create-page" language="bash" />
</TabItem>
<TabItem value="go" label="Go">
<CodeSnippet code={knowledgePagesGo} section="create-page" language="go" />
</TabItem>
</Tabs>
```json
{
"page_id": "kp-2e85...",
"mental_model_id": "mm-77ab...",
"operation_id": "op-1d0f..."
}
```
### Parameters
| Parameter | Type | Required | Description |
|---|---|---|---|
| `name` | string | Yes | Page name. Must be unique within its folder, case-insensitively — a duplicate returns `409`. (Enforced on PostgreSQL only.) |
| `source_query` | string | Yes | The question the page answers, re-asked on every refresh. |
| `parent_id` | string | No | Folder to create the page in. `null` (or omitted) creates it at the root. |
| `tags` | list | No | Tags that scope which memories the page is built from — see [Tags Are a Filter](#tags-are-a-filter). A `type:<x>` tag also sets the page's rendered type, and still counts as part of the filter. |
| `max_tokens` | int | No | Content budget. Defaults to `4096` (a plain mental model defaults to `2048`). |
| `trigger` | object | No | Refresh configuration — see below. |
### Tags Are a Filter
`tags` are not labels on the page. They are the scope the page is built from, and a tagged page matches memories with **`all_strict`** by default: a memory must carry **every** one of the page's tags, and untagged memories are **excluded entirely**.
So a page created like this:
```json
{
"name": "Homelab Infrastructure",
"source_query": "NAS, ThinkPad, docker containers, jellyfin",
"tags": ["type:runbook", "homelab", "infrastructure"]
}
```
is built only from memories tagged `type:runbook` **and** `homelab` **and** `infrastructure`. If your memories were retained without those exact tags — which is the usual case when the tags are invented at page-creation time to describe the topic — the page matches nothing and generates as *"I don't have information about this."* A direct [recall](./recall) for the same query still returns everything, because recall was not given the same filter.
The `type:<x>` tag makes this easy to trip over: it is documented as setting the page's rendered type, but it narrows retrieval like any other tag.
Three ways to get this right:
| You want | Do this |
|---|---|
| The page built from the whole bank | Omit `tags` (or pass `[]`) |
| The tags to scope the page, but untagged memories still included | Keep `tags`, add `"trigger": {"tags_match": "all"}` |
| Only memories carrying every tag (strict isolation, e.g. per-user pages) | Keep `tags` and the `all_strict` default |
To repair a page that already generated empty, `PATCH` it with `{"tags": []}` or with the widened `tags_match`, then refresh it — the tags are stored on the backing mental model, not baked into the content.
See [tag matching modes](./recall#tags) for the full semantics of `any`, `all`, `any_strict`, `all_strict`, and `exact`.
### Default Trigger
When `trigger` is omitted, the page is created with a document-oriented configuration:
```json
{
"mode": "delta",
"fact_types": ["observation"],
"exclude_mental_models": true,
"refresh_after_consolidation": true
}
```
This makes the page a living document built from consolidated observations only, refreshed incrementally whenever consolidation produces new knowledge in its scope, and never influenced by other pages.
| Setting | Why |
|---|---|
| `fact_types: ["observation"]` | The page reads consolidated beliefs, not the raw conversational noise underneath them. Observations are already deduplicated and evidence-backed, so a page reads as a settled document instead of a transcript. Enforced structurally — with only `observation` in scope, the refresh agent isn't given the raw-memory recall tool at all. |
| `exclude_mental_models: true` | A page never reflects on sibling pages. Without this, pages would cite each other and drift into a feedback loop where one wrong claim propagates across the knowledge base. |
| `mode: "delta"` | Each refresh edits the existing document with what is new since the last refresh instead of regenerating it, so hand-tuned structure and wording survive. See [Refresh Mode](./mental-models#refresh-mode). |
| `refresh_after_consolidation: true` | The page rewrites itself whenever consolidation produces new knowledge in its scope — gated by the same [staleness check](./mental-models#staleness-gating) as any mental model, so unrelated bank activity doesn't trigger rebuilds. |
### Page Lifecycle
1. **Create** — the page is stored with placeholder content and a background refresh is submitted; the call returns immediately with an `operation_id`.
2. **First build** — a full generation, since there is no prior document to edit.
3. **Consolidation** — new memories are retained and consolidated into observations.
4. **Staleness check** — pages whose trigger asks for it are checked against their own scope (tags and `fact_types` both apply).
5. **Delta refresh** — stale pages are rewritten by editing the existing document with the new observations only.
Observations are what a page is *built from*, but it can still inspect the evidence underneath them: the refresh agent can expand a memory to its original chunk or document (unless `store_document_text` is disabled for the bank), and if `HINDSIGHT_API_REFLECT_SOURCE_FACTS_MAX_TOKENS` is enabled — off by default — observation search also returns each observation's grounding facts.
:::info
A supplied `trigger` is a **patch**: only the fields you actually send are applied, and the rest keep the defaults above. Sending `{"trigger": {"tags_match": "all"}}` widens the tag filter and leaves `mode`, `fact_types`, `exclude_mental_models`, and `refresh_after_consolidation` as they are. The one exception is the two refresh triggers, which stay mutually exclusive: setting `refresh_cron` clears `refresh_after_consolidation`, and vice versa.
:::
Every [mental model trigger setting](./mental-models#trigger-settings) is accepted here — including `refresh_cron` for scheduled rebuilds instead of consolidation-driven ones, and `tag_groups` for compound tag scoping.
---
## Create a Folder
<Tabs>
<TabItem value="python" label="Python">
<CodeSnippet code={knowledgePagesPy} section="create-folder" language="python" />
</TabItem>
<TabItem value="node" label="Node.js">
<CodeSnippet code={knowledgePagesMjs} section="create-folder" language="javascript" />
</TabItem>
<TabItem value="cli" label="CLI">
<CodeSnippet code={knowledgePagesSh} section="create-folder" language="bash" />
</TabItem>
<TabItem value="go" label="Go">
<CodeSnippet code={knowledgePagesGo} section="create-folder" language="go" />
</TabItem>
</Tabs>
| Parameter | Type | Required | Description |
|---|---|---|---|
| `name` | string | Yes | Folder name |
| `parent_id` | string | No | Parent folder. Omit or pass `null` for the root. |
A `parent_id` that does not exist, or that points at a page rather than a folder, returns `400`.
---
## Read a Page
Returns the page rendered as a markdown document.
<Tabs>
<TabItem value="python" label="Python">
<CodeSnippet code={knowledgePagesPy} section="get-page" language="python" />
</TabItem>
<TabItem value="node" label="Node.js">
<CodeSnippet code={knowledgePagesMjs} section="get-page" language="javascript" />
</TabItem>
<TabItem value="cli" label="CLI">
<CodeSnippet code={knowledgePagesSh} section="get-page" language="bash" />
</TabItem>
<TabItem value="go" label="Go">
<CodeSnippet code={knowledgePagesGo} section="get-page" language="go" />
</TabItem>
</Tabs>
```json
{
"id": "kp-2e85...",
"name": "Deploying the API",
"type": "runbook",
"description": "How is the API deployed?",
"tags": ["ops"],
"timestamp": "2026-08-03T09:12:44+00:00",
"body": "# Deploying the API\n\n...",
"markdown": "---\nid: \"kp-2e85...\"\ntype: \"runbook\"\n...\n---\n\n# Deploying the API\n\n..."
}
```
- `body` is the synthesized markdown body on its own.
- `markdown` is the full document: a YAML frontmatter block (`id`, `type`, `title`, `description`, `tags`, `timestamp`) followed by the body.
- `type` comes from a `type:<x>` tag and defaults to `knowledge-page`. The `type:` tag is removed from the returned `tags`.
---
## Search Pages
Document-level hybrid search: a full-text (BM25) match and a vector-similarity match, fused with Reciprocal Rank Fusion. There is no reranking step, which keeps it fast enough to be an agent's first call.
<Tabs>
<TabItem value="python" label="Python">
<CodeSnippet code={knowledgePagesPy} section="search-pages" language="python" />
</TabItem>
<TabItem value="node" label="Node.js">
<CodeSnippet code={knowledgePagesMjs} section="search-pages" language="javascript" />
</TabItem>
<TabItem value="cli" label="CLI">
<CodeSnippet code={knowledgePagesSh} section="search-pages" language="bash" />
</TabItem>
<TabItem value="go" label="Go">
<CodeSnippet code={knowledgePagesGo} section="search-pages" language="go" />
</TabItem>
</Tabs>
```json
{
"results": [
{
"id": "kp-2e85...",
"name": "Deploying the API",
"mental_model_id": "mm-77ab...",
"snippet": "The API is deployed via ...",
"score": 0.032,
"updated_at": "2026-08-03T09:12:44+00:00"
}
],
"total": 1
}
```
| Parameter | Type | Default | Description |
|---|---|---|---|
| `q` | string | — | Required. Search query (min length 1). |
| `limit` | int | `10` | Maximum results, 1–50. |
This searches whole pages. To search individual memories, use [recall](./recall).
---
## Update or Move a Node
One `PATCH` renames a node, moves it, and/or updates a page's options. Each field applies only when present in the body.
<Tabs>
<TabItem value="python" label="Python">
<CodeSnippet code={knowledgePagesPy} section="update-node" language="python" />
</TabItem>
<TabItem value="node" label="Node.js">
<CodeSnippet code={knowledgePagesMjs} section="update-node" language="javascript" />
</TabItem>
<TabItem value="cli" label="CLI">
<CodeSnippet code={knowledgePagesSh} section="update-node" language="bash" />
</TabItem>
<TabItem value="go" label="Go">
<CodeSnippet code={knowledgePagesGo} section="update-node" language="go" />
</TabItem>
</Tabs>
| Parameter | Type | Applies to | Description |
|---|---|---|---|
| `name` | string | Both | New name |
| `parent_id` | string \| null | Both | New parent folder. Pass `null` explicitly to move to the root. |
| `source_query` | string | Pages | New question. Changing it schedules an async refresh so the page rebuilds against the new question. |
| `tags` | list | Pages | Replaces the page's tags, and so [the scope it is built from](#tags-are-a-filter). Pass `[]` to clear them and rebuild the page from the whole bank. |
| `max_tokens` | int | Pages | New content budget |
| `trigger` | object | Pages | Refresh settings to change, applied as a patch. `{"tags_match": "all"}` keeps the page's tags but stops excluding untagged memories. |
Sending an empty body returns `400`; an unknown node returns `404`.
---
## Delete a Node
Deletes a folder or page **and its entire subtree**. The mental models backing the deleted pages are removed too.
<Tabs>
<TabItem value="python" label="Python">
<CodeSnippet code={knowledgePagesPy} section="delete-node" language="python" />
</TabItem>
<TabItem value="node" label="Node.js">
<CodeSnippet code={knowledgePagesMjs} section="delete-node" language="javascript" />
</TabItem>
<TabItem value="cli" label="CLI">
<CodeSnippet code={knowledgePagesSh} section="delete-node" language="bash" />
</TabItem>
<TabItem value="go" label="Go">
<CodeSnippet code={knowledgePagesGo} section="delete-node" language="go" />
</TabItem>
</Tabs>
```json
{"status": "deleted"}
```
---
## Export as a Markdown Bundle
Returns the whole knowledge base as a flat set of portable markdown files: a nested `index.md`, one `<page-id>.md` per page, and a `<page-id>.log.md` refresh history for pages that have been rebuilt.
<Tabs>
<TabItem value="python" label="Python">
<CodeSnippet code={knowledgePagesPy} section="export" language="python" />
</TabItem>
<TabItem value="node" label="Node.js">
<CodeSnippet code={knowledgePagesMjs} section="export" language="javascript" />
</TabItem>
<TabItem value="cli" label="CLI">
<CodeSnippet code={knowledgePagesSh} section="export" language="bash" />
</TabItem>
<TabItem value="go" label="Go">
<CodeSnippet code={knowledgePagesGo} section="export" language="go" />
</TabItem>
</Tabs>
```json
{
"files": [
{"path": "index.md", "content": "---\ntype: \"index\"\n..."},
{"path": "kp-2e85....md", "content": "---\nid: \"kp-2e85...\"\n..."},
{"path": "kp-2e85....log.md", "content": "---\ntype: \"log\"\n..."}
]
}
```
:::tip Mirror it to disk
`hindsight fs mount --bank my-bank` keeps a local folder in sync with this bundle via a background refresh loop, so `ls`, `grep`, `rg`, and your editor work against real files.
:::
---
## Storage
| Table | Holds |
|---|---|
| `knowledge_pages` | The tree: folders and pages, their names, parents, and ordering. A page row references its backing mental model; a folder row has none. |
| `mental_models` | The content: the document body, its source query, tags, token budget, trigger, and refresh history. |
The page layer owns only tree structure — everything about the content lives on the backing mental model, which is why every [mental model](./mental-models) capability applies to pages unchanged.
---
## Endpoint Summary
| Method | Path | Description |
|---|---|---|
| `GET` | `/knowledge-base/tree` | Nested folder/page tree with staleness |
| `POST` | `/knowledge-base/folders` | Create a folder |
| `POST` | `/knowledge-base/pages` | Create a page (async first build) |
| `GET` | `/knowledge-base/pages/{page_id}` | Read a page as markdown |
| `GET` | `/knowledge-base/search` | Hybrid search over pages |
| `PATCH` | `/knowledge-base/nodes/{node_id}` | Rename, move, or reconfigure a node |
| `DELETE` | `/knowledge-base/nodes/{node_id}` | Delete a node and its subtree |
| `GET` | `/knowledge-base/export` | Export the whole base as markdown files |
Reference in New Issue View Git Blame Copy Permalink