Swaps `quicktok-v1` for `toktok-rs` (vectorize-io/toktok), a Rust BPE tokenizer
whose ids are byte-identical to tiktoken's, then collapses the module's
interface onto the two operations the engine actually performs.
The swap itself is behaviour-neutral: compared side by side over 258 texts
(~250 real files from `hindsight_api/` plus the special-token and mixed-script
edge cases), quicktok and toktok produce identical counts AND identical ids on
all three shared encodings. No token budget, chunk boundary or truncation point
moves.
Interface. `_SafeEncoding` existed to force `disallowed_special=()` onto
`encode()`. Every caller of the object it returned was doing either a plain
`count` or `decode(encode(x)[:n])` open-coded — which is `truncate_to_tokens`.
So the module's whole public surface is now `count_tokens`,
`truncate_to_tokens` / `truncate_many_to_tokens`, and `BUNDLED_ENCODINGS`; the
tokenizer itself is private. That keeps #1883 fixed by construction rather than
by convention: every route to the raising `encode()` went through the accessor
that is now `_load_encoding`.
Character-boundary truncation (toktok 0.1.3). Truncation was
`decode(encode(text)[:n])`, cutting on a *token* boundary. Byte-level BPE
splits one character across several tokens (under o200k_base "🧠" is three), so
a cut could land mid-character and decode to U+FFFD:
truncate_to_tokens("hello 🧠", 2).text -> 'hello �' (before)
-> 'hello ' (now)
The native call also never builds ids, never decodes, and returns the original
string object untouched when nothing needs cutting. `batch_truncate` replaces
the Python loop in the two callers that truncate a whole list: every reranker
document (both LiteLLM cross-encoders) and every embedding input.
Two user-visible consequences:
* `llama3` and `qwen3` are gone — quicktok bundled five vocabularies, toktok
bundles three. `HINDSIGHT_API_TOKENIZER_ENCODING=llama3` now fails at the
first token count with the existing "Unknown tokenizer encoding" ValueError.
Docs and both env templates updated, and `BUNDLED_ENCODINGS` (which had been
lying about those two) now has a test that loads every name it advertises.
* A negative budget used to slice a list with a negative index, silently
dropping tokens off the end; the native call would raise. It clamps to 0.
Wheels: cp311-abi3 covers 3.11-3.14, so 3.14 no longer compiles from source the
way quicktok did. No musllinux wheels, which is irrelevant to the shipped
images (all Python stages are glibc python:3.11-slim). numpy drops to an
optional extra, so the tokenizer pulls in no dependency of its own.
Measured on this repo's text: 2-7x faster than tiktoken on cl100k_base, 10-16x
on o200k_base; counting an 81k-token document peaks at 1 KiB vs ~3 MB.
Token counting is on the hot path of both retain and recall. Recall counts once
per candidate fact, per candidate chunk, per source fact and per reranker
document; retain counts whole documents. All of it went through
`len(encoding.encode(text))` — which builds a full Python list of ids only to
take its length.
Measured on this repo's own text with the microbenchmark added here, against
tiktoken 0.12.0 on a 14-core M-series, both on o200k_base:
workload tiktoken quicktok speedup peak alloc
200 ranked facts 4.39 ms 0.75 ms 5.8x 3 KiB -> 1 KiB
500 source facts 6.92 ms 1.12 ms 6.2x 2 KiB -> 1 KiB
50 candidate chunks 12.02 ms 1.57 ms 7.7x 21 KiB -> 1 KiB
100 reranker documents 10.84 ms 1.55 ms 7.0x 12 KiB -> 1 KiB
one 77k-token document 36.08 ms 3.00 ms 12.0x 2.8 MB -> 1 KiB
Summed across the four counting stages one recall runs: 34.2 ms -> 5.0 ms.
Three things make this worth a dependency change rather than a micro-opt:
* `count()` returns an int without materialising the ids, so counting a large
document allocates nothing. tiktoken has no count-only API — `encode_to_numpy`
reaches the same 1 KiB but none of the speed, and is measured here too.
* ids are byte-identical to tiktoken's; the benchmark asserts that on adversarial
inputs before it times anything.
* the vocabularies ship inside the wheel, so nothing is downloaded at runtime.
That removes the tiktoken pre-download from the Docker build (both stages) and
from scripts/dev/setup.sh — air-gapped deployments no longer need it baked in.
The dependency risk is maintenance, not correctness, so it is contained:
engine/token_encoding.py is the only module that imports quicktok, every call
site routes through get_token_encoding() / count_tokens(), and its one
dependency (numpy) was already in the tree. Replacing it means rewriting that
file and nothing else. This also removes the last direct tokenizer import that
had escaped the seam (`__import__("tiktoken")` in reflect/prompts.py).
Removes #3756's workaround. count_tokens_windowed existed to bound the memory of
counting a large retain body, encoding a megabyte at a time and accepting an
approximate answer because a fixed character cut can split a token. count()
allocates nothing at any size AND is exact, so the windowing, its helpers and its
six call sites are gone — those callers now get an exact count. The test file
keeps the property that made #3756 worth fixing (allocation does not track the
input), now asserted against count_tokens itself.
Default encoding moves to o200k_base, selectable with
HINDSIGHT_API_TOKENIZER_ENCODING (server-level: budgets are only comparable
between banks if they are all counted the same way). o200k_base is what current
OpenAI models tokenize with. On English and code it counts within a fraction of
a percent of cl100k_base, but on non-Latin scripts it is far closer to what a
model actually charges — a mixed-script line with emoji is 19 tokens under
cl100k_base and 13 under o200k_base. Since these counts back budgets that stand
in for a context window, the closer vocabulary is the more honest one. Set
cl100k_base to reproduce the previous counts exactly.
Call sites that only need a number now call count(); the ones that need ids
(query truncation, chunk truncation, reranker truncation, prompt fitting) still
encode, but only after a count shows the text does not fit. The chunk-budget loop
also stopped encoding each oversized chunk twice.
* fix(control-plane): force NODE_ENV=production for production build
A globally-exported NODE_ENV=development (common in dev shells) overrides
Next.js's production default during `next build`, bundling React's development
build under the production server renderer. Static prerendering then crashes
with "Cannot read properties of null (reading 'useContext')" — even on the
built-in _global-error page.
Pin NODE_ENV=production for the build step so it is robust regardless of the
caller's shell. Docker is unaffected (it invokes next build directly in a clean
env).
* chore(dev): add one-shot dev environment setup script
Add scripts/dev/setup.sh: an idempotent bootstrap that installs the required
toolchains (uv/Python, Node/npm, Rust/cargo) when missing, creates .env,
configures git hooks, installs all Python + Node workspace deps, pre-downloads
the local ML models + tokenizer for offline use, and builds the TypeScript SDK
and Rust CLI. Flags: --skip-build, --skip-models, --with-docs, --force.
Document it in CONTRIBUTING.md as the recommended setup, keeping the manual
steps as a fallback.
Nicolò Boschi