mirror of
https://github.com/vectorize-io/hindsight.git
synced 2026-09-14 19:31:49 +08:00
Nicolò Boschi
280f098202
Makes images and files first-class raw content in `retain`. `content` accepts an ordered list of text/image/file blocks, the extractor reads each attachment in the position it occupies, and every read surface hands back the attachments behind what it returns. A plain string behaves exactly as before — text-only retain is byte-identical, because everything new sits behind an ATTACHMENTS block that is empty when a chunk carries none. Blocks are flattened at the API boundary into one canonical body with atomic placeholders, so `documents.original_text` stays plain text and content_hash idempotency, `update_mode=append`, chunk-delta re-extraction and `reprocess_document` keep working untouched. Bytes live in the existing FileStorage abstraction, content-addressed by sha256. Schema (one migration, both dialects): `attachments` for the blob, `document_attachments` for which documents reference it, and `memory_units.attachment_ids` for which attachments a *fact* came from — a column rather than a third table, because those ids behave exactly like `tags`. Provenance is per fact, not per chunk. Extraction runs one call per chunk, and a chunk holding a screenshot also holds the prose around it, so a chunk-level edge cited the diagram as evidence for the paragraph that never mentioned it. The extractor is asked instead, and a fact stated in the prose carries nothing. Extraction quality was measured against a real image-QA dataset with a raw-VLM ceiling arm before merging: transcribing structured attachments rather than summarizing them, and recording how each value is drawn, took the gap between "the model can read this off the image" and "memory can answer it" from 31.3% to 10.0% on the same 40 charts. The prose-article benchmark went 75% -> 100% over the same change, so it is not chart-specific tuning. Also here: * A vision slot (`HINDSIGHT_API_VLM_*`) so attachment-bearing chunks alone use a vision model and text-only chunks stay on a cheaper retain LLM. A vision call deliberately does not fail over to the retain chain's text models — that would reintroduce the silent omission the 422 gate exists to prevent. * The extension retain hook can now see each attachment (media type, size, kind, filename) and refusing a retain reclaims its bytes, which previously stayed fetchable forever. * A filename lives on the document edge, not the blob: the same PDF can be attached under a different name elsewhere, and content-addressing made the first name win for both. Known limitations, documented rather than hidden: store-owned memory backends get nothing (that retain path is Postgres-free and pre-dates this work), very dense pages are sampled rather than exhausted, and the Python client's ContentBlock is a plain dict where TypeScript gets the real union. Breaking for Go and Rust callers: `content` is now a union, so a bare string no longer satisfies it. Go gains a `TextContent()` helper; Rust uses `Content::Variant0(...)`.
Hindsight Benchmarks
This directory contains benchmark suites for evaluating Hindsight's memory capabilities.
Prerequisites
-
Set up your environment variables in
.envat the project root:cp .env.example .env # Edit .env with your API keys -
Make sure you have
uvinstalled.
Available Benchmarks
LoComo
Tests conversational memory with multi-turn dialogues.
# Run from project root
./scripts/benchmarks/run-locomo.sh
# With options
./scripts/benchmarks/run-locomo.sh --max-conversations 10
./scripts/benchmarks/run-locomo.sh --skip-ingestion # Reuse existing data
./scripts/benchmarks/run-locomo.sh --use-think # Use think API
./scripts/benchmarks/run-locomo.sh --conversation conv-26 # Single conversation
Options:
--max-conversations N- Limit number of conversations--max-questions N- Limit questions per conversation--skip-ingestion- Skip data ingestion, use existing--use-think- Use think API instead of search + LLM--conversation NAME- Run specific conversation only--api-url URL- Custom API URL (default: local memory)--only-failed- Retry only failed questions--only-invalid- Retry only invalid questions
LongMemEval
Tests long-term memory across different categories.
# Run from project root
./scripts/benchmarks/run-longmemeval.sh
# With options
./scripts/benchmarks/run-longmemeval.sh --max-instances 50
./scripts/benchmarks/run-longmemeval.sh --category single-session-user
./scripts/benchmarks/run-longmemeval.sh --parallel 4 # Faster evaluation
Options:
--max-instances N- Limit total questions--max-instances-per-category N- Limit per category--skip-ingestion- Skip data ingestion--category NAME- Filter by category:single-session-usermulti-sessionsingle-session-preferencetemporal-reasoningknowledge-updatesingle-session-assistant
--parallel N- Parallel instances (default: 1)--only-failed- Retry failed questions--fill- Resume interrupted runs
Consolidation Performance
Tests consolidation throughput and identifies bottlenecks.
./scripts/benchmarks/run-consolidation.sh
# With custom memory count
NUM_MEMORIES=200 ./scripts/benchmarks/run-consolidation.sh
System Performance Test
Runs retain throughput and recall latency benchmarks using mock LLM + pg0. No external dependencies needed.
# Run all suites at default (small) scale
./scripts/benchmarks/run-perf-test.sh
# Quick smoke test
./scripts/benchmarks/run-perf-test.sh --scale tiny
# Single suite
./scripts/benchmarks/run-perf-test.sh --suite retain
# Save results
./scripts/benchmarks/run-perf-test.sh --output results.json
Options:
--scale {tiny,small,medium,large}- Test scale (default: small)--suite {retain,recall}- Run specific suite (default: all)--output PATH- Save JSON results to file
See perf/README.md for detailed documentation.
Multimodal Retain
Is the information in the pictures reaching memory? Retains the same KB articles
twice — once with their images inline, once with the images absent — and asks
questions that only the images can answer. See
multimodal_retain/README.md for the corpus, the
metrics and the recorded baseline.
Needs a server with a vision-capable retain LLM; without one the multimodal
arm is refused with 422 and the report says so rather than reporting zeros.
uv run python -m benchmarks.multimodal_retain run --api-url http://localhost:8917 --build my-branch
# re-render a saved artifact, no LLM calls
uv run python -m benchmarks.multimodal_retain report benchmarks/results/multimodal_retain/<artifact>.json
Options:
--article NAME- Run one article (repeatable)--build LABEL- Label recorded in the artifact--out PATH- Where to write the artifact--keep-banks- Leave the benchmark banks behind for inspection
Token Counting (micro)
Measures the token counting recall does per candidate fact, chunk and reranker document — wall time, CPU time (all threads) and peak traced allocation, against a set of cheaper spellings of the same count. No DB, no LLM, no network.
# All workloads
./scripts/benchmarks/run-token-count-bench.sh
# One call site, more repeats, raw results
./scripts/benchmarks/run-token-count-bench.sh --workload facts_200 --repeats 10 --json tok.json
# Against real text rather than the synthetic generator (a small vocabulary
# flatters every BPE implementation)
./scripts/benchmarks/run-token-count-bench.sh --corpus /path/to/text
# On another vocabulary instead of production's o200k_base
./scripts/benchmarks/run-token-count-bench.sh --encoding cl100k_base
# Include the tiktoken baseline (deliberately not a project dependency)
cd hindsight-dev && uv run --with tiktoken token-count-bench
Options:
--workload NAME- Run one workload (repeatable); shaped after a real call site--encoding NAME- Vocabulary to measure (defaulto200k_base, what production uses). A 200k vocabulary is a different amount of work per byte, so a ranking measured on one does not transfer to another for free.--repeats N- Timed repeats per variant, best-of (default: 5)--threads N-num_threadsfor the parallel batch variant--corpus PATH- Slice workload texts out of a real text file--no-conformance- Skip the count-agreement check on adversarial inputs--json PATH- Save raw results
Every variant is checked against the production count first, on inputs that have broken token counting before (special-token literals, unicode, empty). A variant that counts differently — or raises — is reported as such, never as a speedup.
Visualizer
View benchmark results in a web UI:
./scripts/benchmarks/start-visualizer.sh
# Opens at http://localhost:8001
Results
Results are saved in JSON format in each benchmark's results/ directory.