Token counting is on the hot path of both retain and recall. Recall counts once
per candidate fact, per candidate chunk, per source fact and per reranker
document; retain counts whole documents. All of it went through
`len(encoding.encode(text))` — which builds a full Python list of ids only to
take its length.
Measured on this repo's own text with the microbenchmark added here, against
tiktoken 0.12.0 on a 14-core M-series, both on o200k_base:
workload tiktoken quicktok speedup peak alloc
200 ranked facts 4.39 ms 0.75 ms 5.8x 3 KiB -> 1 KiB
500 source facts 6.92 ms 1.12 ms 6.2x 2 KiB -> 1 KiB
50 candidate chunks 12.02 ms 1.57 ms 7.7x 21 KiB -> 1 KiB
100 reranker documents 10.84 ms 1.55 ms 7.0x 12 KiB -> 1 KiB
one 77k-token document 36.08 ms 3.00 ms 12.0x 2.8 MB -> 1 KiB
Summed across the four counting stages one recall runs: 34.2 ms -> 5.0 ms.
Three things make this worth a dependency change rather than a micro-opt:
* `count()` returns an int without materialising the ids, so counting a large
document allocates nothing. tiktoken has no count-only API — `encode_to_numpy`
reaches the same 1 KiB but none of the speed, and is measured here too.
* ids are byte-identical to tiktoken's; the benchmark asserts that on adversarial
inputs before it times anything.
* the vocabularies ship inside the wheel, so nothing is downloaded at runtime.
That removes the tiktoken pre-download from the Docker build (both stages) and
from scripts/dev/setup.sh — air-gapped deployments no longer need it baked in.
The dependency risk is maintenance, not correctness, so it is contained:
engine/token_encoding.py is the only module that imports quicktok, every call
site routes through get_token_encoding() / count_tokens(), and its one
dependency (numpy) was already in the tree. Replacing it means rewriting that
file and nothing else. This also removes the last direct tokenizer import that
had escaped the seam (`__import__("tiktoken")` in reflect/prompts.py).
Removes #3756's workaround. count_tokens_windowed existed to bound the memory of
counting a large retain body, encoding a megabyte at a time and accepting an
approximate answer because a fixed character cut can split a token. count()
allocates nothing at any size AND is exact, so the windowing, its helpers and its
six call sites are gone — those callers now get an exact count. The test file
keeps the property that made #3756 worth fixing (allocation does not track the
input), now asserted against count_tokens itself.
Default encoding moves to o200k_base, selectable with
HINDSIGHT_API_TOKENIZER_ENCODING (server-level: budgets are only comparable
between banks if they are all counted the same way). o200k_base is what current
OpenAI models tokenize with. On English and code it counts within a fraction of
a percent of cl100k_base, but on non-Latin scripts it is far closer to what a
model actually charges — a mixed-script line with emoji is 19 tokens under
cl100k_base and 13 under o200k_base. Since these counts back budgets that stand
in for a context window, the closer vocabulary is the more honest one. Set
cl100k_base to reproduce the previous counts exactly.
Call sites that only need a number now call count(); the ones that need ids
(query truncation, chunk truncation, reranker truncation, prompt fitting) still
encode, but only after a count shows the text does not fit. The chunk-budget loop
also stopped encoding each oversized chunk twice.
* fix(llm): send an explicitly configured reasoning_effort (#3449)
`reasoning_effort` was gated on a substring match against the model *name*
(`gpt-5`, `o1`, `o3`), which can only ever match OpenAI's own products. Every
`HINDSIGHT_API_*_REASONING_EFFORT` variable was therefore a silent no-op on
self-hosted reasoning models served through `provider=openai` + a custom
base_url — vLLM, Ollama, llama.cpp, TGI — where controlling thinking-token
volume matters most, and `none` (the only value that removes the thinking
block) was unreachable through any documented configuration.
A name proves nothing on an endpoint that can serve anything under any name, so
a configured effort now outranks the heuristic and is sent on both the plain and
the tool-calling path. With nothing configured the heuristic still decides, so
endpoints that never received the parameter don't start getting it. The one
exception is a model whose name identifies an OpenAI product that rejects the
parameter outright (gpt-4o, gpt-4.1, gpt-4-*, gpt-3.5), where honouring the
setting would trade a silently ignored value for a hard 400 — and that drop is
logged at WARNING rather than being silent.
Carrying explicitness that far down meant `llm_reasoning_effort` had to stop
baking in its default: it is None when unset, and `LLMInterface` resolves the
effective level to DEFAULT_LLM_REASONING_EFFORT, so the effective default is
unchanged. The startup line now reports the reasoning mode in force.
* docs(llm): freeze _supports_reasoning_model as a capability check
Name matching only ever recognised OpenAI's own products; a new reasoning
model is now a configuration question, not a new substring.
* fix(llm): never send a reasoning effort nobody configured
Unset resolved to "low" in the config layer, so four lanes shipped a level the
operator never chose: openai-compatible for recognised reasoning models,
openai-responses, codex (unconditionally) and xai-oauth (unconditionally).
Everything else already sent nothing. That asymmetry is what made the setting so
hard to reason about — a configured value could be silently dropped while an
unconfigured one was transmitted.
Now None means None the whole way down: no provider sends a reasoning parameter
unless HINDSIGHT_API_*_REASONING_EFFORT is set, and each model runs at its own
default effort instead. `configured_reasoning_effort` collapses back into
`reasoning_effort`, and DEFAULT_LLM_REASONING_EFFORT is gone — nothing resolves
unset to a level any more.
Behaviour change for deployments that never set the variable on those four
lanes: they move from Hindsight's "low" to the model's own default effort.
* fix(llm): honour or report reasoning_effort in the remaining providers
#3449 was filed against the OpenAI-compatible lane, but three more lanes made
the same setting dead weight for a different reason: litellm, litellm-router,
gemini/vertexai, anthropic and claude-code accepted reasoning_effort into the
constructor and never looked at it again. Same symptom from the operator's
seat — the variable is set, documented, visible in the environment, and nothing
happens.
litellm and litellm-router can honour it: litellm.completion takes
reasoning_effort natively and translates it per target provider (Anthropic
thinking budgets, Gemini thinking config, OpenAI's flat parameter), and
litellm.drop_params=True discards it for models with no reasoning knob instead
of raising. Both lanes now forward it when configured — the Router builds its
own kwargs, so it needed the same line rather than inheriting one.
gemini/vertexai, anthropic and claude-code have no reasoning-effort control at
all, so they log a WARNING naming the ignored value instead of swallowing it.
Mapping effort onto Gemini's thinking_config and Anthropic's extended-thinking
budget is a real feature with cost and temperature implications; it deserves its
own change, not a guess buried in this one.
* docs(performance): add Tuning for Local & Small Environments section
Supersedes #1721. Keeps the local-LLM concurrency guidance from that PR
(HINDSIGHT_API_LLM_MAX_CONCURRENT, saturation symptom + diagnostics) and
expands it into a dedicated section covering the other knobs that matter
on laptops, single-GPU boxes, and local LLM servers:
- per-operation concurrency caps to reserve reflect headroom
- timeouts/retries for slow local generation
- smaller per-operation models + low reasoning effort + LLM=none
- built-in llama.cpp tuning (gpu layers, context size, threads, grammar)
- CPU reranker knobs (fp16, bucket batching, max concurrent, flashrank)
- CPU embeddings (force_cpu)
* docs(performance): drop saturation symptom + diagnostics block
* docs(performance): drop LLM_PROVIDER=none chunk-mode note
* docs(performance): add reranker candidate-set + consolidation batch-size levers; drop CPU embeddings note
* fix: improve async batch retain with large payloads
* fix: improve async batch retain with large payloads
* api
* api
* api
* api
* api
* Clean up perf benchmark: keep only Python files
- Remove README.md and PERFORMANCE_FINDINGS.md
- Remove results/ JSON files (gitignored)
- Remove test_data/ directory
- Keep only __init__.py and retain_perf.py
* docs: explain automatic batch optimization for async retain
- Add section explaining Hindsight automatically handles batch sizing
- Users don't need to manually tune batch sizes with async mode
- Hindsight splits large batches (>10k tokens) into optimized sub-batches
- Include example showing best practices
* docs: remove emojis and code example from performance page
* fix: correct OperationDetails type to match API response
- Change optional fields to use | null instead of ?
- Fixes TypeScript compilation error in control plane build
* fix: use discriminated union for OperationDetails type
- Support both success and error states properly
- Fixes TypeScript error when setting error state
* fix: use unique document_ids in batch retain examples
- Each item in a batch must have unique document_id
- Update both Python and JavaScript examples
- Fixes test-doc-examples CI failure
* chore: trigger CI
* fix: test mocking and duplicate document_ids in examples
- Mock _get_pool() in test_async_retain_tags.py to avoid _initialized error
- Set _initialized = True on mocked MemoryEngine instances
- Fix duplicate document_ids in retain.py and retain.mjs examples
* fix: properly mock async pool/connection and fix more duplicate document_ids
- Use AsyncMock for pool.acquire() to fix 'can't be used in await' error
- Fix duplicate document_ids in retain-async examples (retain.py and retain.mjs)
- Remove batch-level document_id parameter that caused duplicates
* ci: collect all doc example failures and show summary
- Run all Python/Node.js/CLI examples regardless of individual failures
- Collect failure list and display summary at the end
- Show pass/fail count and list of failed files
- Exit with failure only after running all examples
* refactor: extract doc example testing to standalone script
- Create scripts/test-doc-examples.sh to run all examples
- Collects logs of failed examples separately
- Shows full error logs only for failures at the end
- Clean summary with pass/fail counts
- Proper exit codes
- Replaces inline bash in CI workflow
* fix: doc examples - duplicate document_ids and error handling
- retain.py: move document_id to item level to avoid duplicates
- documents.mjs: add error handling for getDocument to show clear error message
* fix: update tests for duplicate document_id validation
- test_async_retain_tags: verify operation structure instead of exact UUID
- test_delete_bank: use unique document_ids (team-doc-1, team-doc-2)
* fix: misc fixes for observations and mental models
* feat: improve graph retrieval for observations
- Update LinkExpansionRetriever to traverse through source_memory_ids
for observation entity connections (avoiding data duplication)
- Remove entity link copy from world facts to observations in consolidator
- Add tests for link expansion graph retrieval
- Add directives_applied field to ReflectResult
- Include user's other changes (CLI, docs, client updates)
* fix: CI test failures
- Add mental_model_id parameter to create_mental_model function
- Fix ToolCallTrace not including reason field from ToolCall
- Improve test_link_expansion_observation_graph_retrieval to wait for consolidation with retry
* chore: reduce link expansion log verbosity
* Revert "chore: reduce link expansion log verbosity"
This reverts commit 3ce759391cead1012157785fa78fef16ef9bfe3b.
* feat: add semantic/temporal/entity links as fallback in graph retrieval
- Add fallback query for semantic, temporal, and entity links from memory_links
- Check both directions (outgoing and incoming links)
- Weight fallback results at 0.5x to prioritize entity links via unit_entities
- Fixes graph retrieval returning 0 when data has cross-cluster temporal connections
* fix: enable observations fixture for link expansion test
- Add enable_observations fixture to ensure observations are created
- Increase wait time from 10 to 30 seconds for CI reliability
* bump pg0 0.11.x and improve documentation
* bump pg0 0.11.x and improve documentation
* bump pg0 0.11.x and improve documentation
* ci: test notebooks on ci
* ci: test notebooks on ci
* rm llms-full from repo
* formatting
* formatting
Nicolò Boschi