18 Commits

Author	SHA1	Message	Date
Nicolò Boschi	630c3a63e7	fix(reflect): grounding defects in reflect and knowledge-page delta refresh, plus system-evals (#4304) ... * test(dev): add a deterministic retrieval eval for reflect's forced prelude We had no baseline to judge a retrieval change against, so a change to how reflect's opening hierarchy (mental models -> observations -> recall) picks its queries could not be told apart from a regression. The corpus is authored, never extracted: every stored row's text is byte-identical to the YAML, which is what makes gold labelling possible at all. Facts and observations are retained with the mock provider scripted through set_response_callback; mental models are created with explicit content; staleness is produced by ORDERING (stale models created before the facts, fresh ones after) because it is derived, not stored. Questions are grouped by which layer should answer them -- mm_only, mm_stale, observations_only, raw_facts_only, multi_layer, near_miss, absent -- so the eval exercises the descent decision and not just one search. The facts carry deliberate near misses; a corpus of unrelated facts scores 1.0 for any query and measures nothing. Scored by rank WITHIN each layer. Set membership saturates once the corpus is smaller than one recall page, and a flat ranking scores every raw fact behind every mental model however good the query was. So recall@3 and MRR per layer, plus the short-circuit fire rate reported separately, since the mental-model query is what decides it. First deliverable is the floor/ceiling experiment, which needs no LLM in either arm: the question verbatim (the real fallback when planning fails) against a hand-written ideal query. First run: recall@3 floor=0.929 ceiling=1.000 gap=+0.071 MRR floor=0.762 ceiling=1.000 gap=+0.238 Wording moves retrieval, mostly through rank rather than presence, with the mm_stale question the largest mover (0.50 -> 1.00) -- the case where a wrong mental model has to be superseded by raw facts. Two behaviours it already surfaced, neither introduced here: - A created-but-never-refreshed mental model is always stale: staleness resolves from the refresh stamps and an unstamped model is reported stale unconditionally, so it can never short-circuit the descent until something refreshes it once. - One stale model suppresses the short-circuit for all of them, since the rule requires every returned model to be fresh. An unrelated stale model in the top-5 keeps the descent going even when a fresh model answers the question outright. * test(dev): grade reflect's actual answer, not just what it retrieved The retrieval eval scores the evidence set, never the prose reflect returns. That is a necessary condition -- reflect is grounded, so an unretrieved fact cannot be answered -- and nowhere near a sufficient one. The blind spot is the category the whole exercise is about. For the mm_stale question, retrieval can return the stale "Stripe" mental model AND the Adyen facts, score recall@k = 1.0, and the answer can still say Stripe because the model trusted the summary over the raw facts. The retrieval tier calls that a pass. So answer_eval.py runs reflect_async end to end on a real model and grades the text with an independent judge, scoring two things separately because they fail differently: - correct: meets answer_criteria. Missing it can just mean incomplete. - trap: asserts must_not_claim, the specific wrong answer the question baits. That is a grounding failure, and it is the number that matters -- a confidently wrong answer is worse than a hedged one. Every question runs N times and the output is a rate, not a verdict. Not for CI. The judge mirrors tests/llm_judge.py (independent model, majority confirmation on a "not met") but is reimplemented here because that module lives under tests/ and is not importable from this package; the eval warns when the judge and reflect resolve to the same model, since the local .env makes that the default and a model grading its own output agrees with itself. First run -- gemini-2.5-flash-lite reflecting, gemini-2.5-flash judging, 2 runs, budget=low, on this branch: overall correct 93.8%, trap rate 0% on every baited question multi_layer 50%, everything else 100% The multi_layer miss is real run-to-run variance rather than a judge artifact: one run named the platform-team handover, the other dropped it. This is one arm only. Comparing main against the branch on the same corpus, model and N is what actually settles the cold-query question, and is not done here. * test(dev): hard corpus, failure attribution, and runner env-precedence fixes WIP checkpoint before A/B testing the thought_signature failure. * test(dev): stop the outage corpus generating two "April 2026 outage" rows The numeric_precision cluster cycled months with `i % 12` and years with `i // 12`, which produced a second row claiming to be THE April 2026 outage with different values (850 connections / 87 minutes vs the gold's 200 / 47). The question then had two contradictory answers, and reflect reporting "conflicting information" -- exactly what its Conflicts and Ambiguity rules prescribe -- was scored as a failure. The corpus was wrong, not the answer. I reported it as a reflect defect before checking; it was mine. Near-misses must differ in what they ASSERT, never in what they claim to BE. Each outage now owns a distinct (month, year) slot with April 2026 reserved for the gold row, and `_assert_subjects_are_unique` fails the build when two rows in a cluster name the same subject -- verified by re-introducing the collision, which the guard catches. * fix(reflect): don't manufacture a value for a period the memories don't cover Asked for an engineering headcount in a year the bank held no data for, reflect extrapolated backwards from the following year's monthly figures and answered with a specific number -- calling it "reliably inferred" and "reliably deduced". Three runs out of three, on gemini-3.7-flash. That is not a hedge: it is a fabricated data point wearing the language of certainty, and it is worse than "not recorded" because a reader cannot tell the difference. The prompts asked for it. Every path that writes an answer said some version of "if the exact answer isn't stated, use what IS stated to give the best possible answer", with "only say you don't have information if the retrieved data is truly unrelated" closing the escape hatch. A neighbouring year IS related, so declining was effectively disallowed. The missing distinction: inference may CHARACTERISE what the data covers; it may not MANUFACTURE a value for something the data does not cover. _GROUNDING_BOUNDARY states that, and is shared by all three answer paths (the tool-loop system prompt, the forced-synthesis system prompt, and the final-synthesis instructions) so they cannot drift apart. It explicitly preserves qualitative inference, because a rule read as "never infer" would break the synthesis that makes reflect worth having. Tests split per the convention: the wiring is deterministic and asserted directly, including that the rule keeps its teeth and its carve-out. The behavioural pair is marked hs_llm_core and its docstring says plainly what it does NOT do -- it does not reproduce the incident (verified: it passes against the pre-fix prompt on two models), it guards the contract, and its more valuable half is the check that the rule has not become a refusal reflex. Reproduction lives in hindsight-dev/benchmarks/prelude (hq-absent-2024). 488 reflect/prompt tests pass; the golden prompt fixture is updated. * test(dev): accept reflect's accurate UK qualification on the scoped_truth question The criteria said 2FA is "mandatory for EU (and UK) accounts", which reads as unconditional. The memory actually says UK accounts are mandatory FROM 2026, and reflect answered "UK Accounts: Mandatory starting in 2026" -- more precise than the criteria, and marked wrong for it (1 run in 6). Scoring a correct answer as a failure is the worse error for a benchmark: it manufactures a defect to chase. The criteria now accepts any accurate treatment of the UK and keeps the real assertion, which is that the answer must be regionally qualified rather than a flat yes or no. * fix(mental-models): stop delta ops treating the batch-only synthesis as authoritative The delta refresh writes a synthesis from the new batch alone, then asks a second call to merge it into the stored page. That call read the synthesis as evidence: its "a total of 4" counted only the batch and replaced a page's 3 customers with 4; its "no release was deployed" described only the batch and overwrote a production release recorded one wave earlier. Label the synthesis UNTRUSTED (only the supporting facts justify an operation) and add the combine-not-swap, absence-is-not-contradiction and refutation threshold rules. Replayed against both captured failures: 0/5 -> 5/5. * test: add hindsight-system-evals, published as a quality metric by the perf monitor Blackbox quality evals over a real hindsight-api and a real model, through the published Python client only — the system-tests shape without the stub, since stubbing the model would score the stub. First suite: knowledge-page convergence, the eval that found the delta-ops regressions. Each page is graded twice: correct, and whether it stores the specific baited falsehood. Runs in perf-test.yml (daily, not on PRs — needs secrets, and one red run is as likely noise as regression) and publishes correct rate and trap count to the continuous performance monitor. Seeding uses chunks retain with consolidation off, so the only model calls are the ones under test: minimum acceptance ~90s, full ~5 min. Also: the hindsight-dev knowledge-page tools used to find and replay those failures (kp_eval, kp_diagnose, replay_gemini, iterate_delta_prompt). * refactor: move the reflect evals into hindsight-system-evals and trim the new prompt text Everything from hindsight-dev/benchmarks/prelude now lives in the blackbox package, driven only through the public client: - test_02_reflect_answers: the one-shot reflect eval (minimum acceptance is the 2024-headcount incident), with retrieval-vs-reasoning blame from the tool trace; - debug/diagnose_page: dry-runs a page's second refresh and dumps every traced prompt; debug/replay_delta_ops: replays one captured delta-ops request per prompt variant and interrogates the model. The retrieval floor/ceiling tier is dropped: it measured the planned-prelude change (#4066), which was closed, and needed engine internals to author layers. Prompt size, measured against main: - grounding boundary rewritten compactly and no longer repeated in the final instructions (the final and reduce calls already carry it in the system prompt, so it was sent twice): tool loop +131 tokens, final +112; - delta ops: the combine-not-swap rule removed. Ablation by replaying the two captured failures: every other piece is load-bearing (dropping the absence or refutation rules, or the long synthesis paragraph, falls to 2-3/5), this one is not (5/5 on both without it). +491 tokens. Full run on the result: 14/14 correct, 0 traps (7 pages, 7 reflect answers). * test(system-evals): drop an unused property and type the page eval tests	2026-09-11 11:41:34 +02:00
Sanderhoff-alt	b045794817	perf(retain): accelerate within-batch semantic link calculation (#3977) ... Cuts the within-batch semantic link pass to float32 and one reused buffer. The batch was widened to float64, but PackedEmbedding is array("f") and pgvector's vector column stores float32, so the extra 32 bits were padding nothing downstream could read. Dropping to float32 halves the working set and puts BLAS on SGEMM; normalising in place, deriving validity from the row norms instead of an (n, dim) isfinite mask, and reusing one similarity buffer across blocks remove three further copies. Peak transient falls 74-86% (at 5,000 facts, 235 MB -> 48 MB). argpartition replaces a full sort that existed only to discard all but top_k, and the self-link mask and score unboxing move out of the Python loop: 1.7-2.2x on a realistic clustered batch, up to 4.8x when nearly every pair clears the threshold. Norms are accumulated in float64 via einsum, since a float32 sum of 1536 squares overflows above ~1e19 and flushes to zero below ~1e-22. The batch is copied with np.array rather than aliased with asarray, as it is now normalised in place. Verified against the float64 implementation across 120 randomised batches plus NaN/inf/zero embeddings, degenerate magnitudes and all-ties: identical link pairs, scores within 1e-6.	2026-09-08 11:38:19 +02:00
Sanderhoff-alt	bc06bd051f	perf(entity-resolver): optimize in-batch dedup via prefix filtering (#3991) ... Replaces the O(N^2) pairwise loop in _find_intrabatch_similar_pairs with a prefix-filtering set-similarity join, returning exactly the pairs the double loop returned. Each name's trigrams are sorted rarest-first and only the leading |A| - ceil(t*|A|) + 1 are indexed; a pair sharing none of those tokens cannot clear the cutoff. Cutting each prefix from its own size is what makes the pruning lossless — ascending set-size order is for the size filter, not for correctness. 3.9x faster at the 250-name cap and 8x at 1000 on distinct names; 2.5x over 9,525 real per-document batches harvested from LoCoMo and LongMemEval. Two shapes are slower and are now benchmark workloads rather than footnotes: batches under ~50 names (tens of microseconds) and batches of mutually similar names, where the filter prunes nothing and costs ~1.2x. _INTRABATCH_MAX_NAMES stays at 250 — that second shape costs ~156ms at 500 with no await in the join. Equivalence is pinned against the loop it replaced over randomised batches at 14 cutoffs, including exactly-achievable Jaccard ratios where a float t*|A| a hair above a whole number would shorten the prefix and drop a pair on the cutoff. Verified further on 60,000 set-level trials and the 9,525 real batches at six cutoffs: zero mismatches. Also memoises candidate trigram sets in _resolve_from_candidates, filled lazily so candidates the scoring loop never reaches cost nothing and no work happens ahead of the _SCORING_YIELD_EVERY yield points (#3211).	2026-09-08 09:51:16 +02:00
Sanderhoff-alt	0478c09c41	perf(retain): optimize embedding_to_pgvector via zero-copy orjson (#3815) ... Retain and import paths convert float embeddings to pgvector vector literals for asyncpg binding (insert_facts_batch, compute_semantic_links_within_batch, update_memory_unit_embedding). The baseline implementation used a Python generator: "[" + ",".join(repr(float(value)) for value in embedding) + "]" For 500 facts (768,000 floats at 1536d), this allocated 768,000 PyFloat objects and 768,000 PyUnicode strings, taking ~220 ms CPU time and ~15 MB heap memory. The optimized implementation leverages np.frombuffer on PackedEmbedding (array('f')) for zero-copy buffer views, and orjson.OPT_SERIALIZE_NUMPY to format floats directly into the output byte buffer using Rust Ryu SIMD: * Promotes numpy to explicit direct dependency across hindsight-api and dev; * Formats shortest float32 representation (byte-identical Postgres storage); * Isolates _repr_literal fallback helper for non-finite and non-float inputs; * Unifies _dumps_or_repr_fallback with single payload parameter and no option branching; * Streamlines embedding_to_pgvector into a concise polymorphic dispatcher; * Seamlessly supports array('f'), list[float], tuple, ndarray, and str. Measured on Apple Silicon via vector-serialization-bench (best of 5 repeats): workload baseline prod speedup peak alloc single_bge_384 (1x 384d) 0.136 ms 0.040 ms 3.4x 36K -> 13K single_openai_1536 (1x 1536d) 0.489 ms 0.085 ms 5.8x 144K -> 50K batch_20_gemini_768 (20x 768d) 4.580 ms 0.514 ms 8.9x 356K -> 185K batch_200_openai_1536 (200x 1536d) 92.64 ms 9.45 ms 9.8x 6.1M -> 3.4M batch_500_large_doc (500x 1536d) 221.78 ms 23.84 ms 9.3x 15.0M -> 8.4M batch_200_raw_list (200x 1536d) 87.70 ms 11.48 ms 7.6x 6.1M -> 6.0M Throughput increased from 3.3 Mfloat/s to 32.5 Mfloat/s (~9.8x speedup on typical retain batches), with ~44% peak memory reduction on 500-fact batches. Includes unit tests in test_packed_embeddings.py covering bit-identical float32 roundtrips, custom non-serializable objects, non-f array fallthrough, tuples, ndarrays, and non-finites.	2026-08-31 12:50:04 +02:00
Nicolò Boschi	9fcb7ca7ac	perf(tokenizer): replace tiktoken with quicktok and default to o200k_base (#3788) ... Token counting is on the hot path of both retain and recall. Recall counts once per candidate fact, per candidate chunk, per source fact and per reranker document; retain counts whole documents. All of it went through `len(encoding.encode(text))` — which builds a full Python list of ids only to take its length. Measured on this repo's own text with the microbenchmark added here, against tiktoken 0.12.0 on a 14-core M-series, both on o200k_base: workload tiktoken quicktok speedup peak alloc 200 ranked facts 4.39 ms 0.75 ms 5.8x 3 KiB -> 1 KiB 500 source facts 6.92 ms 1.12 ms 6.2x 2 KiB -> 1 KiB 50 candidate chunks 12.02 ms 1.57 ms 7.7x 21 KiB -> 1 KiB 100 reranker documents 10.84 ms 1.55 ms 7.0x 12 KiB -> 1 KiB one 77k-token document 36.08 ms 3.00 ms 12.0x 2.8 MB -> 1 KiB Summed across the four counting stages one recall runs: 34.2 ms -> 5.0 ms. Three things make this worth a dependency change rather than a micro-opt: * `count()` returns an int without materialising the ids, so counting a large document allocates nothing. tiktoken has no count-only API — `encode_to_numpy` reaches the same 1 KiB but none of the speed, and is measured here too. * ids are byte-identical to tiktoken's; the benchmark asserts that on adversarial inputs before it times anything. * the vocabularies ship inside the wheel, so nothing is downloaded at runtime. That removes the tiktoken pre-download from the Docker build (both stages) and from scripts/dev/setup.sh — air-gapped deployments no longer need it baked in. The dependency risk is maintenance, not correctness, so it is contained: engine/token_encoding.py is the only module that imports quicktok, every call site routes through get_token_encoding() / count_tokens(), and its one dependency (numpy) was already in the tree. Replacing it means rewriting that file and nothing else. This also removes the last direct tokenizer import that had escaped the seam (`__import__("tiktoken")` in reflect/prompts.py). Removes #3756's workaround. count_tokens_windowed existed to bound the memory of counting a large retain body, encoding a megabyte at a time and accepting an approximate answer because a fixed character cut can split a token. count() allocates nothing at any size AND is exact, so the windowing, its helpers and its six call sites are gone — those callers now get an exact count. The test file keeps the property that made #3756 worth fixing (allocation does not track the input), now asserted against count_tokens itself. Default encoding moves to o200k_base, selectable with HINDSIGHT_API_TOKENIZER_ENCODING (server-level: budgets are only comparable between banks if they are all counted the same way). o200k_base is what current OpenAI models tokenize with. On English and code it counts within a fraction of a percent of cl100k_base, but on non-Latin scripts it is far closer to what a model actually charges — a mixed-script line with emoji is 19 tokens under cl100k_base and 13 under o200k_base. Since these counts back budgets that stand in for a context window, the closer vocabulary is the more honest one. Set cl100k_base to reproduce the previous counts exactly. Call sites that only need a number now call count(); the ones that need ids (query truncation, chunk truncation, reranker truncation, prompt fitting) still encode, but only after a count shows the text does not fit. The chunk-budget loop also stopped encoding each oversized chunk twice.	2026-08-25 15:02:26 +02:00
Nicolò Boschi	fe5c25d64c	fix(mental-models): store delta documents as verbatim markdown blocks (#3361, #3273) (#3622) ... * fix(mental-models): store delta documents as verbatim markdown blocks (#3361, #3273) A knowledge page could come back with a whole table welded onto one line, in sections no delta operation had named, and never recover. The delta refresh was blamed, but the damage was done one refresh earlier. `structured_content` was only the source of truth on the delta leg. The full leg stored the LLM candidate markdown verbatim in `content` while deriving the structure from it with `parse_markdown`, a typed-block parser that flattened anything its union could not express -- nested lists, list continuation lines, blockquotes, hard line breaks, horizontal rules, HTML, indented code, table alignment, a table row missing an outer pipe. 15 of 16 common constructs lost information, and every loss was a fixed point, so no later refresh could undo it. The two columns disagreed by construction, and the next delta refresh published the degraded one over the whole document. Schema v2 stores each block as a verbatim markdown fragment plus an id. Nothing parses a table, so nothing can flatten one. `parse_markdown` is deleted; `split_markdown` replaces it and recognises only ATX headings and blank lines, both fence-aware, which makes it lossless -- asserted as a property over a 26-case corpus of exactly the constructs v1 destroyed. Blocks are now addressed by id rather than by index (#3273). An index has to be counted by the model, and an off-by-one lands in range, silently overwrites an unrelated block, and is recorded as a success. An id is copied, not derived; one that does not resolve -- or that names a block in a different section -- is skipped and reported. Operation payloads are plain markdown strings, so the model no longer has to emit a typed block union either. `content` is now always the render of `structured_content`, on both legs, so the two can no longer drift apart. Also here: - `parse_llm_json` escapes `\n \r \t \b \f` inside JSON string values instead of blanking them. A model writing a markdown table into a string often forgets to escape its line breaks, and replacing them with spaces delivered the table already collapsed. Other control characters keep the previous treatment. - A model that adds a table row as its own block would render a broken table, so the prompt asks for `replace_block` and bare rows landing directly after a table are folded into it. - Migration `d1e2f3a4b5c6` clears v1 blobs. They are a lossy projection of the row's own `content`, so there is nothing to convert: the next refresh re-imports the structure from the markdown, losslessly. `content` is untouched. - The Gemini eval fixture pinned `gemini-2.0-flash`, which the provider has retired (404), so the whole eval class was dead. Verified against a real model: `test_document_survives_many_delta_rounds_intact` runs five delta rounds feeding one new fact each, asserting after every round that content is the render of the structure, that no line welds a table separator to other cells, that sections no operation named are byte-identical, and that a section never named across the whole run is unchanged at the end. Run four times, 20 real rounds, green. It is what caught the orphan table row. * fix(mental-models): write the structure whenever content is written `create_mental_model(content=...)` and `create_knowledge_page` inserted the markdown and left `structured_content` NULL, so a model authored as markdown had no structure until its first delta refresh -- and that refresh was then the one to derive it, silently reshaping a document nobody had asked it to touch. `update_mental_model(content=...)` was worse: it could set the markdown while leaving the *previous* document's structure in place, so the two columns described different documents until the next refresh papered over it. Nothing enforced the pairing; the refresh path just happened to pass both. Both writes now go through `canonical_document()`, which splits the authored markdown and hands back the structure together with its render. The insert stores both, and the update derives the structure whenever a caller supplies content without one. A refresh still passes both explicitly -- there the structure is authoritative and the markdown is already its render -- and is untouched. The derivation is hoisted above the embedding computation so the embedding, the history snapshot and the UPDATE all see one text. `content` is therefore the render of `structured_content` from the first byte rather than from the first refresh, which is what the assertion churn in this commit is: authored markdown now comes back canonicalised, so a document that was stored as "v1" reads back as "v1\n". Also makes the migration test rerunnable: a pg0 instance survives between runs and alembic will not replay a migration on a DB already stamped past it, so seeding into it would have left the rows untouched and the test asserting nothing. * feat(reflect): answer with a document, render the markdown from it The mental-model refresh asked the agent for markdown and worked out the document's structure by reading that markdown back. Reading LLM markdown back is where #3361 destroyed tables, and it is unnecessary here: the refresh knows it is producing a document, so it can ask for one. `done()` gains a document mode. Instead of an `answer` string it takes a `document` -- an ordered list of sections, each with a heading, a level and its blocks -- and the markdown that gets stored and shown is rendered from it. The model no longer writes the markdown that gets persisted, and nothing parses markdown to find out what the model meant. `answer` is not merely discouraged in that mode, it is absent from the schema, so there is no escape hatch back to prose. The shape is deliberately flat -- an array of sections holding arrays of block strings, no unions. A tool schema goes to the provider verbatim and not every provider accepts `oneOf` (Gemini rejects it), and a shape the model can fill without thinking is one it fills correctly. `document_from_sections` is tolerant, because a tool call is still model output: a missing heading, a `##` the model prefixed anyway, an out-of-range level or a non-string block is coerced rather than rejected. A block holding several blank-line-separated fragments is split into one block each, so the document keeps the granularity delta operations address even when the model packs a whole section into one string. Downstream is unchanged: the rendered markdown still flows on as `text`, so structured-output extraction, the length rewrite and the HTTP response all behave as before. The one place the two could drift is the length rewrite, which edits the text after the fact -- there the structure is re-derived from the rewritten markdown, which is lossless and keeps the invariant that the stored text is exactly what the stored structure renders. Splitting markdown is now only an import path: a model created from authored markdown, a restored export, or a run that produced plain text anyway (a provider that dropped the tool call, the iteration-limit answer). Verified against a real model: all five `hs_llm_core` refresh evals pass with the agent emitting structure, including the five-round stability run. The ordered list that a previous run had rewritten now survives untouched, and the new table row lands inside the table rather than beside it. * test(benchmarks): compare two builds on how a document survives being edited Neither half of "is the new pipeline better" was measurable before this. The unit tests prove the mechanics in isolation and the refresh evals prove one build behaves, but nothing compared a build against another one on the thing that actually broke: a document rewritten by an LLM over and over. The harness talks HTTP only, so the same code drives a server built from any revision. That is what makes an A/B possible without a feature flag inside the code under test: run it once per build, compare the two artifacts. Every document from every round is stored, and metrics are recomputed at comparison time, so sharpening a metric costs nothing instead of another few hundred LLM calls. Two things it measures, deliberately separately. Structural, no LLM: collapsed tables (the detector from #3361), rows, nesting, hard breaks, fences and quotes lost, sections that drifted with no operation naming them, plus pipeline health and latency. Damage is counted only in sections no operation named -- a refresh that rewrites a section it targeted may legitimately restructure it, and scoring that as corruption would punish the model for doing its job. Content, judged: each round declares what must be true afterwards and what must no longer be stated, checked one claim at a time so a miss points at a fact rather than at a score; plus a blind pairwise preference between the two builds' final documents, judged in both orderings so position bias cannot decide it. Three cases, and the third is the point. `api-reference` carries every fragile construct; `onboarding-playbook` carries none, because a change that fixes tables while degrading ordinary prose is not an improvement and that is where it would show; `release-runbook` puts a table with a missing outer pipe in a section the fact stream never touches. Both details are load-bearing: a well-formed table never triggered the bug, and a model asked to edit a malformed table tends to rewrite it correctly, repairing the damage before it can be measured. The reported failure was in sections the operations never named, where nothing could repair it. Token usage is not reported. The stored reflect_response does not carry it, and a column that is always zero reads as "this is free" rather than "this is not measured here". * test(benchmarks): harden the judging, and say what the A/B measured Running the benchmark against main and the branch turned up three problems in the benchmark itself, all of which would have made its verdict untrustworthy. The runner slept a fixed interval instead of awaiting the async operations it submitted, so its first results were five rounds of "Generating content..." scored as though they were documents. Retain, create and refresh are all submit-and-poll; it now polls, and refuses outright if the seed it is about to measure is still a placeholder. Damage was attributed to the whole document rather than to the sections nobody asked to change, which scored a model deliberately restructuring a section it targeted as if the machinery had corrupted it. Damage is now measured only in untouched sections, and the metrics are recomputed from the stored documents at comparison time, so this sharper reading could be applied to results already collected instead of paying to re-run them. A single judge call decided each claim, and one pedantic reading moved a build's score: a document describing an operation as "synthesises stored memories" was scored as not supporting "answers questions over stored memories" — the same operation, in the wording the source fact itself used. Claims are now decided by majority of three, and that claim was rewritten to test the fact rather than one phrasing of it. A claim a correct document can fail measures the corpus, not the pipeline. The report prints mean document length beside the preference column, because judges favour longer documents and a preference that tracks that column should be read sceptically rather than counted. The prompt change is the finding that landed back in the product: stating a document's structure is more clerical than writing prose, and the model got terser at it — measurably shorter documents that the judge liked less. Document mode now says plainly that the structure is the shape of the answer, not a budget for it. Baseline recorded in baseline_report.json: 45 refresh rounds per build from identical seeds. main lost 3 tables to collapse, 9 table rows, 6 levels of list nesting and 5 hard line breaks across 6 damaged rounds, and drifted 14 sections nobody had named. The branch lost nothing and drifted nothing. Content came out level — 100% recall and zero stale claims on both sides. * fix(mental-models): give the delta leg the document's own token budget `max_tokens` was enforced in exactly one place: a rewrite of the *synthesis* answer when it came back longer than the budget. In delta mode that answer is only context for the operations call and never becomes the document, so the document that actually gets stored was never measured against the budget at all. A delta refresh only adds. The document-evolution benchmark measured ~20 tokens of growth per round across 45 rounds, monotonic, which crosses the 4096-token knowledge-page default after a couple of hundred refreshes — and knowledge pages refresh after every consolidation. The configured budget was quietly ignored for the entire life of a page after its first full build. Truncating the document here would delete knowledge nobody asked to delete, so the budget is stated instead: the delta call is told the document's current size against its budget, and when it is over, asked to make room with the same operations it uses for everything else — on content that is superseded or duplicated, never by dropping the facts it is integrating and never by summarising a section that is still current. Below 80% of the budget nothing is said at all. Every refresh records document_tokens and document_budget, and going over adds a warning, so a page that keeps growing is visible rather than merely large. Also closes the one path where the model still wrote markdown that got stored: the over-budget trim. In document mode it is now asked for a document, so the structure survives the trim instead of being re-derived from prose the model wrote. A response that is not JSON falls back to the previous split, which is lossless — the worst case is the old behaviour, not a lost answer. The real-LLM eval is the part that could not be mocked: told that a document is over budget, does a model reclaim space or append anyway? It drops the twelve archived sections and keeps the current process and the checklist — 434 tokens to 39 against a 200-token budget, stable across four runs. The assertions check where the space came from, because getting under budget by deleting current content would pass a naive shrink check and be worse than going over. * test(mental-models): audit every trigger flag on the delta leg `max_tokens` looked wired up — read from the model, passed to reflect, enforced by a rewrite — and was still ignored for the document that actually got stored, because in delta mode the thing it capped never becomes the document. Reading the code is how that was missed; nothing asserted the flag at its destination. So every flag is now exercised through a real delta refresh and asserted where it lands: retrieval options at the reflect call, document options in the delta prompt or the persisted row. Full mode is covered by the surrounding modules; this is the leg where a flag goes to die. Fifteen flags checked. Fourteen were already honoured. The audit pins them so a future change to either leg cannot quietly drop one: - retrieval: fact_types, exclude_mental_models, exclude_mental_model_ids, the model's own id (a model must not feed on its previous version), include_chunks, recall_max_tokens, recall_chunks_max_tokens, the model's tags, tags_match - document: max_tokens, response_schema (extracted from the merged document, not from reflect's delta-only answer), keep_trace, mode - and the whole trigger surviving a create/read round trip, since a flag that does not persist is not honoured either The fifteenth is documented behaviour that reads as a bug from outside, so it is pinned as intended rather than "fixed": `tag_groups` overrides flat tags entirely, dropping the model's own tags and forcing `tags_match` to `any`, because each group carries its own match mode. A `tags_match` set alongside groups is deliberately not forwarded. The default for a tagged model with no `tags_match` is `all_strict`, not `any` — a model scoped to tags must not widen its own scope by default. Scheduling flags (`refresh_cron`, `refresh_after_consolidation`) are honoured outside the refresh executor — the maintenance loop and the consolidation hook — and keep their existing coverage there. * test(benchmarks): type the structural summary, and cover the flag main just added Two findings from reviewing the branch against a fresh main. `_structural_summary` returned a raw dict of known keys, which the project standards forbid for exactly the reason it bit here: the report read it with `summary["rounds"]` and nothing would have caught a renamed metric until the table rendered wrong. It is a `StructuralSummary` model now, and the side-by-side table renders `model_dump()` so a metric added later appears without being listed twice. Main added a sixteenth trigger flag while this branch was in flight (`min_refresh_interval_seconds`, #3621). It gates automatic refreshes rather than shaping one, so it is honoured in the submit path and covered there — but the round-trip test enumerates the whole trigger on purpose, because a field that round-trips as None looks like the flag being ignored rather than like a storage bug. Adding it keeps that list exhaustive. * fix(mental-models): teach the retraction prompt the schema it emits into CI caught what the rebase brought: main added an unsay pass (#3618) whose prompt documents the operation vocabulary a second time, in prose, and it still told the model to say `{"op": "remove_block", "section_id": "...", "index": N}` with typed `block` payloads. Under the id-addressed schema those ops fail validation and are dropped, so a retracted fact would keep being stated and nothing would say why — the unsay feature silently doing nothing. The prompt now describes the schema it actually emits into: blocks addressed by `block_id`, block payloads as markdown strings, and the note about emitting removals in descending index order deleted, because ids do not shift when a sibling is removed and telling a model to order by position invites it to think in positions again. Guarded structurally rather than by one more test. The op vocabulary is written down twice — Pydantic models the applier validates against, and prose in each system prompt — and a test for the prompt that drifted does not exist by construction, since it is the one nobody wrote. `test_delta_prompt_schema_parity` asserts over *every* prompt carrying an operations vocabulary that each op exists, that no shape names a field the schema rejects, that no v1 typed block survives, and that blocks are addressed by id; plus a check that a prompt asking for `{"operations": [...]}` cannot be left off the list. Reverting the prompt fails three of them. The rest is the same schema change reaching tests the rebase brought in: canned ops in the outcome matrix moved to `text`, the retraction tests resolve a real `block_id` out of the document the prompt shows them (which is what a model does, and what a hardcoded index cannot express), the stale `parse_markdown` monkeypatch points at `structured_document_from_stored`, and four assertions on authored content now expect the canonical render.	2026-08-20 11:35:32 +02:00
Nicolò Boschi	613a699e9f	fix(consolidation): eliminate duplicate observations via interleave dedup recall (#1907) ... Round-robin interleave fusion for consolidation dedup recall (guarantees the semantic-#1 'twin' a slot so the LLM updates instead of duplicating), unified 'reranking' strategy param (cross_encoder/rrf/interleave), case-sensitive exact-dup guard, obs-dedup tool + benchmark wired into the perf dashboard (English dataset). Near-dup observation rate 4% -> 0% on the English hermes transcript (1/10 and 1/4), coverage 89% -> 94%, no false merges.	2026-06-03 17:37:57 +02:00
Nicolò Boschi	efe5ff8494	feat(perf): publish perf-test results to external dashboard (#1474) ... * feat(perf): publish perf-test results to external dashboard repo Adds `--benchmark-output-dir` to perf-test, which emits two JSON files in github-action-benchmark format: latency.json (smaller-is-better: durations + recall p50/p95/p99/mean) and throughput.json (bigger-is- better: items/queries/memories per sec). The Performance Tests workflow now publishes both to vectorize-io/hindsight-continuous-performance- monitor's gh-pages branch on each scheduled run. Iteration mode (TEMP — search "TEMP" to revert before merge): push trigger on this branch, default scale=small, locomo skipped unless manually dispatched. Setup needed (one-time): - PAT with Contents:write on the dashboard repo, stored as secret PERF_DASHBOARD_TOKEN. - After the first run creates gh-pages there, enable Pages on that repo (Settings → Pages → gh-pages branch). * fix(perf): wipe benchmark working dir between latency and throughput publishes github-action-benchmark clones the dashboard repo into a fixed ./benchmark-data-repository directory and doesn't clean up, so the second invocation in the same job fails with 'destination path already exists'. * feat(perf): replace github-action-benchmark with custom dashboard publisher Drops the two benchmark-action steps (and the dead `--benchmark-output-dir` flag + `_to_benchmark_entries` helper in system_perf.py) in favour of a single `scripts/benchmarks/publish-perf-results.sh` step. The script: 1. Reads the perf-test JSON output. 2. Enriches it with commit metadata (subject, author, author_date, commit URL, PR URL via `gh api commits/<sha>/pulls`). 3. Clones the dashboard repo's gh-pages branch using PERF_DASHBOARD_TOKEN. 4. Writes data/<timestamp>-<short_sha>.json and prepends the run to data/index.json (newest first). 5. Commits and pushes (with one rebase-retry on push rejection). The matching custom static site lives on gh-pages of vectorize-io/hindsight-continuous-performance-monitor (separate commit in that repo). * perf(workflow): publish dashboard on workflow_dispatch too * feat(perf): publish workflow run URL and LoComo results to dashboard Perf script now embeds workflow_run.{id,url} in each enriched run JSON and the manifest entry, sourced from default GitHub Actions env vars (GITHUB_RUN_ID + GITHUB_REPOSITORY). LoComo gets its own publish script (publish-locomo-results.sh) and a new step in the locomo job. The script strips per-question detailed_results (kept in the workflow artifact) before pushing — keeps each run small enough for git. Output lands at: data/locomo/<timestamp>-<short_sha>.json data/locomo-index.json The matching dashboard page (locomo.html) is in the dashboard repo. * perf(workflow): revert iteration-mode TEMP markers Restores the production defaults that were temporarily flipped while iterating on the dashboard: - drop the push trigger on feat/perf-dashboard - default scale: small → large - default locomo_skip: true → false - locomo job condition: workflow_dispatch-only → inputs.locomo_skip != true Scheduled cron now runs the full suite + LoComo daily and publishes to the dashboard.	2026-05-06 15:12:11 +02:00
Nicolò Boschi	9c33a7c730	feat(perf): add system performance test runner (#1201) ... * feat(perf): add system performance test runner and CI workflow Add `uv run perf-test` command that orchestrates retain throughput and recall latency benchmarks using mock LLM + pg0 for deterministic, LLM-independent baselines. Wraps existing recall_perf/retain_perf building blocks without duplicating benchmark logic. Also fixes _RRFReranker in recall_perf.py to include the cross_encoder attribute now required by the engine's combined scoring path. * feat(perf): add run-perf-test.sh script * feat(perf): use run-perf-test.sh in CI, remove run-retain-perf.sh Replace ad-hoc retain perf wrapper with the new system perf test script in CI workflow and docs. The standalone retain_perf.py is still available for ad-hoc document benchmarking. * feat(perf): add suite input to workflow dispatch	2026-04-22 17:37:58 +02:00
Nicolò Boschi	aefb3fcf4d	fix: improve async batch retain with large payloads (#366) ... * fix: improve async batch retain with large payloads * fix: improve async batch retain with large payloads * api * api * api * api * api * Clean up perf benchmark: keep only Python files - Remove README.md and PERFORMANCE_FINDINGS.md - Remove results/ JSON files (gitignored) - Remove test_data/ directory - Keep only __init__.py and retain_perf.py * docs: explain automatic batch optimization for async retain - Add section explaining Hindsight automatically handles batch sizing - Users don't need to manually tune batch sizes with async mode - Hindsight splits large batches (>10k tokens) into optimized sub-batches - Include example showing best practices * docs: remove emojis and code example from performance page * fix: correct OperationDetails type to match API response - Change optional fields to use | null instead of ? - Fixes TypeScript compilation error in control plane build * fix: use discriminated union for OperationDetails type - Support both success and error states properly - Fixes TypeScript error when setting error state * fix: use unique document_ids in batch retain examples - Each item in a batch must have unique document_id - Update both Python and JavaScript examples - Fixes test-doc-examples CI failure * chore: trigger CI * fix: test mocking and duplicate document_ids in examples - Mock _get_pool() in test_async_retain_tags.py to avoid _initialized error - Set _initialized = True on mocked MemoryEngine instances - Fix duplicate document_ids in retain.py and retain.mjs examples * fix: properly mock async pool/connection and fix more duplicate document_ids - Use AsyncMock for pool.acquire() to fix 'can't be used in await' error - Fix duplicate document_ids in retain-async examples (retain.py and retain.mjs) - Remove batch-level document_id parameter that caused duplicates * ci: collect all doc example failures and show summary - Run all Python/Node.js/CLI examples regardless of individual failures - Collect failure list and display summary at the end - Show pass/fail count and list of failed files - Exit with failure only after running all examples * refactor: extract doc example testing to standalone script - Create scripts/test-doc-examples.sh to run all examples - Collects logs of failed examples separately - Shows full error logs only for failures at the end - Clean summary with pass/fail counts - Proper exit codes - Replaces inline bash in CI workflow * fix: doc examples - duplicate document_ids and error handling - retain.py: move document_id to item level to avoid duplicates - documents.mjs: add error handling for getDocument to show clear error message * fix: update tests for duplicate document_id validation - test_async_retain_tags: verify operation structure instead of exact UUID - test_delete_bank: use unique document_ids (team-doc-1, team-doc-2)	2026-02-16 12:51:42 +01:00
Nicolò Boschi	b43ef98686	feat: consolidation performance benchmark and optimization (#227)	2026-01-29 11:24:15 +01:00
Nicolò Boschi	a3ad76d165	many improvements	2025-11-26 18:22:25 +01:00
Nicolò Boschi	3e618cb001	papers and fixes	2025-11-14 14:07:41 +01:00
Nicolò Boschi	588065182a	polish + cli + helm + standalone	2025-11-11 12:35:20 +01:00
Nicolò Boschi	ada9562cc2	lot of improvements	2025-11-10 15:02:51 +01:00
Nicolò Boschi	5d32e79955	fixes	2025-11-07 14:14:50 +01:00
Nicolò Boschi	fcc5250656	bm25 and re-rankers	2025-11-07 10:22:59 +01:00
Nicolò Boschi	39801d9f8b	local setup and speed	2025-11-05 15:35:07 +01:00