mirror of
https://github.com/vectorize-io/hindsight.git
synced 2026-09-14 19:31:49 +08:00
Nicolò Boschi
d9df6d2a4d
* perf(api): serialize a recall's audit row once, and decode embedding batches with orjson Two costs on every recall that a CPU profile of a recall-heavy API put at ~5% of its busy CPU (450 recalls/s, 2 vCPU): - The HTTP audit wrapper built a Python dict of the whole response with model_dump(mode="json") on the request path, and the writer then re-encoded that dict with json.dumps (3.5% alone). A pydantic response now goes straight to JSON with model_dump_json(), one pass in Rust, carried on AuditEntry.response_json. Anything else still goes through _safe_json, which uses orjson when it is installed. The stored document is the same; the column is JSON, so key order and escaping are not observable. - A TEI embedding batch is a large JSON array of floats, parsed by the stdlib decoder behind response.json() (1.4%). orjson.loads(response.content) when available. orjson is optional in both places: without it the previous code path runs unchanged. * perf(metrics): opt-in sampling for recall-phase observations Every recall records ~10 phase histograms, and OTel's aggregation behind them was ~4.6% of a recall-heavy API's busy CPU. HINDSIGHT_API_RECALL_PHASE_SAMPLE_EVERY=N records 1 in N calls, sampled independently per call, so each phase's distribution -- and its percentiles -- stay unbiased; only absolute counts scale by 1/N. The default of 1 records everything, as before. * perf(embeddings): embed a recall query on the event loop instead of a worker thread A recall embeds one short string. On the thread path that costs an executor hop plus httpx's pure-Python sync stack: 1.02 ms of CPU per query, against 0.39 ms for an aiohttp request made on the loop (measured in-process against the same TEI server; identical vectors). RemoteTEIEmbeddings.aencode_query makes one attempt and returns None on any failure, so the existing thread path and its retry policy still handle every error. It also declines when uninitialized or when a test injected a client. generate_embeddings_batch sends both paths through the same alignment and vector validation. * fix(recall-perf): declare orjson, move phase sampling into config orjson was imported behind an ImportError guard but never declared, so it was not installed and both speedups (audit rows, TEI decode) were dead code; ty also failed on the unresolved import. Declare it and drop the fallbacks. HINDSIGHT_API_RECALL_PHASE_SAMPLE_EVERY was read straight from the environment in metrics.py; it is now a HindsightConfig field with docs and env-template entries, plus a sampling unit test. The TEI embedder's aiohttp session attributes are initialised in __init__ instead of via getattr. * docs: regenerate docs skill for the recall-phase sampling flag * test(tei): hold per-thread clients so a freed client's reused id can't read as shared