mirror of
https://github.com/vectorize-io/hindsight.git
synced 2026-09-14 19:31:49 +08:00
Nicolò Boschi
bce43b8e14
Swaps `quicktok-v1` for `toktok-rs` (vectorize-io/toktok), a Rust BPE tokenizer whose ids are byte-identical to tiktoken's, then collapses the module's interface onto the two operations the engine actually performs. The swap itself is behaviour-neutral: compared side by side over 258 texts (~250 real files from `hindsight_api/` plus the special-token and mixed-script edge cases), quicktok and toktok produce identical counts AND identical ids on all three shared encodings. No token budget, chunk boundary or truncation point moves. Interface. `_SafeEncoding` existed to force `disallowed_special=()` onto `encode()`. Every caller of the object it returned was doing either a plain `count` or `decode(encode(x)[:n])` open-coded — which is `truncate_to_tokens`. So the module's whole public surface is now `count_tokens`, `truncate_to_tokens` / `truncate_many_to_tokens`, and `BUNDLED_ENCODINGS`; the tokenizer itself is private. That keeps #1883 fixed by construction rather than by convention: every route to the raising `encode()` went through the accessor that is now `_load_encoding`. Character-boundary truncation (toktok 0.1.3). Truncation was `decode(encode(text)[:n])`, cutting on a *token* boundary. Byte-level BPE splits one character across several tokens (under o200k_base "🧠" is three), so a cut could land mid-character and decode to U+FFFD: truncate_to_tokens("hello 🧠", 2).text -> 'hello �' (before) -> 'hello ' (now) The native call also never builds ids, never decodes, and returns the original string object untouched when nothing needs cutting. `batch_truncate` replaces the Python loop in the two callers that truncate a whole list: every reranker document (both LiteLLM cross-encoders) and every embedding input. Two user-visible consequences: * `llama3` and `qwen3` are gone — quicktok bundled five vocabularies, toktok bundles three. `HINDSIGHT_API_TOKENIZER_ENCODING=llama3` now fails at the first token count with the existing "Unknown tokenizer encoding" ValueError. Docs and both env templates updated, and `BUNDLED_ENCODINGS` (which had been lying about those two) now has a test that loads every name it advertises. * A negative budget used to slice a list with a negative index, silently dropping tokens off the end; the native call would raise. It clamps to 0. Wheels: cp311-abi3 covers 3.11-3.14, so 3.14 no longer compiles from source the way quicktok did. No musllinux wheels, which is irrelevant to the shipped images (all Python stages are glibc python:3.11-slim). numpy drops to an optional extra, so the tokenizer pulls in no dependency of its own. Measured on this repo's text: 2-7x faster than tiktoken on cl100k_base, 10-16x on o200k_base; counting an 81k-token document peaks at 1 KiB vs ~3 MB.