Per-(bank, fact_type) partial vector indexes are created only at fresh-bank
creation. A bank populated outside that path (logical restore, cross-version
upgrade, extension switch) never gets them, so its recall silently falls back
to the global index + post-filter — slower and under-returning (~0.63-0.72
recall@10 measured by the reporter).
Two fixes:
- import-bank: create the per-bank indexes explicitly after restoring the
banks row. The prior get_or_create_bank_profile call was a no-op here (the
row already exists, so it takes the SELECT branch), leaving every restored
bank uncovered.
- hindsight-admin repair-bank (--bank ID | --all): re-runnable operator escape
hatch for the out-of-app routes (raw pg_dump restore, extension switch) that
a one-time migration can't cover (a restore carries alembic_version at head,
so the migration is already stamped). Detects missing OR invalid coverage
(INVALID leftovers / drifted access method count as missing, unlike a
name-only check) and rebuilds with CREATE INDEX CONCURRENTLY off any txn.
Idempotent; concurrency handled by idempotency, not advisory locks.
Deliberately excludes the boot/periodic background reconcile and retain-path
self-heal: a bank restored and only ever read stays degraded until an operator
runs repair-bank. That background layer can be a follow-up.
The run-db-migration Options table listed only `--schema`, but the command
also exposes two operator-facing flags (hindsight_api/admin/cli.py):
- `--embedding-dimension` — enforce an expected embedding dimension after
migrations (omit to skip the dimension sync).
- `--skip-extension-reconcile` — added in #2309; skip the post-migration
vector/text-search index reconcile to speed up no-change re-migrations across
many tenant schemas when the backend is unchanged.
Add both rows to the canonical Options table and regenerate the docs skill
mirror.
#1903 expanded BACKUP_TABLES to all 15 tables (the 7 previously-missing ones that could be silently dropped on restore), but the "backup includes" list still reflected the old ~8-table coverage. Update it to match: mental models, directives, webhooks, file storage, plus internal operational tables for a faithful full-database snapshot. Oracle-only observation_sources stays excluded (PostgreSQL-only backup). Regenerated skills/hindsight-docs mirror.
* feat(transfer): admin export-bank command (whole-bank portable archive)
Add 'hindsight admin export-bank --bank <id> [--schema] [--include-history]'
that exports an entire bank to a portable ZIP for migrating it to a new
instance configured with a different embedding model / vector / text-search
backend. No embeddings are written — they are regenerated on import.
The archive is a superset of the documents archive:
* logical document/fact/observation export (replayed + re-embedded on import);
* bank config, mental models (vector stripped → re-embed), directives, webhooks
carried as JSON rows;
* audit_log / llm_requests only with --include-history.
Every bank-scoped table (BACKUP_TABLES) is classified logical / carried /
history / skipped; test_export_bank_covers_schema fails if a future migration
adds a table without classifying it. Import of the new sections is a follow-up.
Tests: schema-coverage guard + a contents test (archive_type, carried bank
config + webhook, no embeddings, history gated by the flag).
* feat(transfer): import-bank — restore a whole-bank archive (cross-instance migration)
Add the import half of bank migration:
* transfer.import_bank: restores bank config, then docs/facts/observations
(re-embedded with the TARGET instance's model via import_documents), then
mental models, directives, webhooks as verbatim rows. Restores exact state —
fires no webhooks and triggers no consolidation (observations/mental models
are restored, not regenerated). _restore_rows coerces JSON values back to
column types (timestamps/uuids/jsonb) and is idempotent (ON CONFLICT DO NOTHING).
* MemoryEngine.import_bank_async / export_bank_async wrappers.
* admin 'import-bank' command (boots a MemoryEngine for the target model);
plus engine-backed export.
Tests: exact round-trip (export -> delete -> import) asserts every section —
bank config, documents, facts, observations, entities, temporal links, webhooks,
directives, mental models — matches exactly, with facts re-embedded (no NULL
vectors). Semantic links compared loosely (ANN index regenerated). Also a guard
that import-bank rejects a documents-only archive.
* docs(transfer): bank migration runbook (export-bank / import-bank)
Document the admin export-bank/import-bank commands and the blue-green runbook
for moving a bank to a new instance with a different embedding model / vector /
text-search backend, re-embedding on import without LLM re-extraction.
* refactor(transfer): drop unused export_bank_async engine method
Code-review: the engine wrapper had no caller but the test — the export-bank CLI
reads rows directly via transfer.export_bank (no engine/embeddings boot needed
for a read-only export). Call transfer.export_bank directly in the test instead.
* docs(transfer): document export-bank/import-bank + migration playbook on the Admin CLI page
Use the installed 'hindsight-admin <cmd>' convention (not 'uv run'). Add the full
export-bank/import-bank command reference and blue-green migration runbook to the
Admin CLI page; reduce the memory-banks section to a short summary that links there.
* refactor(transfer): _admin_connect helper + clearer _REPLAYED_TABLES naming
- Extract _admin_connect(db_url); resolve_database_url already handles pg0:// vs
postgres://, so export-bank no longer re-implements the connect dance inline.
- Rename _LOGICAL_TABLES -> _REPLAYED_TABLES + clarify: entities/unit_entities/
memory_links/entity_cooccurrences are NOT exported (rebuilt by the import
pipeline); the bucket only exists for the coverage guard.
* fix(transfer): import-bank requires a non-existent target bank (no merge)
Importing into an existing bank silently merged: bank config kept (ON CONFLICT
DO NOTHING), docs per on_conflict, and mental_models/directives/webhooks added
alongside existing rows. import-bank restores a WHOLE bank, so refuse when the
target already exists — delete it or pass a fresh --target-bank.
Since a fresh target has no document conflicts, drop the now-meaningless
on_conflict knob from import_bank / import_bank_async / the import-bank CLI.
Test: importing an archive whose bank still exists raises.
* test(transfer): add manual two-instance bank-migration e2e script
scripts/dev/e2e-bank-migration.sh spins instance A (bge-small/384) and B
(bge-base/768), retains into A, runs export-bank -> import-bank, and asserts
recall on B returns the migrated fact ranked first with both instances on
different embedding dims. Self-asserting (exits non-zero on failure); not run in
CI (needs two cached models + an LLM key). Verified passing locally.
* test(transfer): drop manual e2e-bank-migration.sh script
Remove the two-instance migration e2e script from the repo (kept as a local-only
dev tool). Engine-level integration tests in test_document_transfer.py cover the
export/import round-trip.
* docs(admin-cli): add 'Running the CLI' intro (how to run, what it points to)
Explain that hindsight-admin connects directly to PostgreSQL (not the HTTP API),
uses the same config/.env as the API (HINDSIGHT_API_DATABASE_URL), is PostgreSQL-only,
and is typically run inside the API host/container (docker exec / kubectl exec).
* docs: surface stable worker_id guidance and zombie-operation recovery
Worker identity defaults to the container hostname, which Docker rotates
on every restart. That stranded several real deployments' consolidation
queues (issue #1470 and the related closed tickets #991 / #696 / #624).
Move the guidance from the configuration reference table — where it
only gets read after the bug bites — into the install path and add a
recovery section next to the decommission commands.
* docs(faq): add zombie-operations entry
* docs(admin-cli): document decommission-workers and worker-status
PR #1165 added two new admin CLI commands (decommission-workers,
worker-status) but admin-cli.md was not updated. Readers scanning the
Commands section could only find the singular decommission-worker.
Added dedicated sections for each new command following the existing
style (Arguments/Options/Examples/When to Use). Pure docs, mirrors
behavior documented in typer command help strings.
* docs(admin-cli skill): sync decommission-workers and worker-status
Mirror change from hindsight-docs/docs/developer/admin-cli.md so the
docs skill reference stays in sync (matches the pattern set by #1137).
Nicolò Boschi