mirror of
https://github.com/vectorize-io/hindsight.git
synced 2026-09-14 19:31:49 +08:00
Nicolò Boschi
630c3a63e7
* test(dev): add a deterministic retrieval eval for reflect's forced prelude
We had no baseline to judge a retrieval change against, so a change to how
reflect's opening hierarchy (mental models -> observations -> recall) picks
its queries could not be told apart from a regression.
The corpus is authored, never extracted: every stored row's text is
byte-identical to the YAML, which is what makes gold labelling possible at
all. Facts and observations are retained with the mock provider scripted
through set_response_callback; mental models are created with explicit
content; staleness is produced by ORDERING (stale models created before the
facts, fresh ones after) because it is derived, not stored.
Questions are grouped by which layer should answer them -- mm_only,
mm_stale, observations_only, raw_facts_only, multi_layer, near_miss, absent
-- so the eval exercises the descent decision and not just one search. The
facts carry deliberate near misses; a corpus of unrelated facts scores 1.0
for any query and measures nothing.
Scored by rank WITHIN each layer. Set membership saturates once the corpus
is smaller than one recall page, and a flat ranking scores every raw fact
behind every mental model however good the query was. So recall@3 and MRR
per layer, plus the short-circuit fire rate reported separately, since the
mental-model query is what decides it.
First deliverable is the floor/ceiling experiment, which needs no LLM in
either arm: the question verbatim (the real fallback when planning fails)
against a hand-written ideal query. First run:
recall@3 floor=0.929 ceiling=1.000 gap=+0.071
MRR floor=0.762 ceiling=1.000 gap=+0.238
Wording moves retrieval, mostly through rank rather than presence, with the
mm_stale question the largest mover (0.50 -> 1.00) -- the case where a wrong
mental model has to be superseded by raw facts.
Two behaviours it already surfaced, neither introduced here:
- A created-but-never-refreshed mental model is always stale: staleness
resolves from the refresh stamps and an unstamped model is reported stale
unconditionally, so it can never short-circuit the descent until something
refreshes it once.
- One stale model suppresses the short-circuit for all of them, since the
rule requires every returned model to be fresh. An unrelated stale model
in the top-5 keeps the descent going even when a fresh model answers the
question outright.
* test(dev): grade reflect's actual answer, not just what it retrieved
The retrieval eval scores the evidence set, never the prose reflect returns.
That is a necessary condition -- reflect is grounded, so an unretrieved fact
cannot be answered -- and nowhere near a sufficient one.
The blind spot is the category the whole exercise is about. For the mm_stale
question, retrieval can return the stale "Stripe" mental model AND the Adyen
facts, score recall@k = 1.0, and the answer can still say Stripe because the
model trusted the summary over the raw facts. The retrieval tier calls that a
pass.
So answer_eval.py runs reflect_async end to end on a real model and grades
the text with an independent judge, scoring two things separately because
they fail differently:
- correct: meets answer_criteria. Missing it can just mean incomplete.
- trap: asserts must_not_claim, the specific wrong answer the question baits.
That is a grounding failure, and it is the number that matters -- a
confidently wrong answer is worse than a hedged one.
Every question runs N times and the output is a rate, not a verdict. Not for
CI. The judge mirrors tests/llm_judge.py (independent model, majority
confirmation on a "not met") but is reimplemented here because that module
lives under tests/ and is not importable from this package; the eval warns
when the judge and reflect resolve to the same model, since the local .env
makes that the default and a model grading its own output agrees with itself.
First run -- gemini-2.5-flash-lite reflecting, gemini-2.5-flash judging, 2
runs, budget=low, on this branch:
overall correct 93.8%, trap rate 0% on every baited question
multi_layer 50%, everything else 100%
The multi_layer miss is real run-to-run variance rather than a judge
artifact: one run named the platform-team handover, the other dropped it.
This is one arm only. Comparing main against the branch on the same corpus,
model and N is what actually settles the cold-query question, and is not done
here.
* test(dev): hard corpus, failure attribution, and runner env-precedence fixes
WIP checkpoint before A/B testing the thought_signature failure.
* test(dev): stop the outage corpus generating two "April 2026 outage" rows
The numeric_precision cluster cycled months with `i % 12` and years with
`i // 12`, which produced a second row claiming to be THE April 2026 outage
with different values (850 connections / 87 minutes vs the gold's 200 / 47).
The question then had two contradictory answers, and reflect reporting
"conflicting information" -- exactly what its Conflicts and Ambiguity rules
prescribe -- was scored as a failure. The corpus was wrong, not the answer.
I reported it as a reflect defect before checking; it was mine.
Near-misses must differ in what they ASSERT, never in what they claim to BE.
Each outage now owns a distinct (month, year) slot with April 2026 reserved
for the gold row, and `_assert_subjects_are_unique` fails the build when two
rows in a cluster name the same subject -- verified by re-introducing the
collision, which the guard catches.
* fix(reflect): don't manufacture a value for a period the memories don't cover
Asked for an engineering headcount in a year the bank held no data for,
reflect extrapolated backwards from the following year's monthly figures and
answered with a specific number -- calling it "reliably inferred" and
"reliably deduced". Three runs out of three, on gemini-3.7-flash. That is not
a hedge: it is a fabricated data point wearing the language of certainty, and
it is worse than "not recorded" because a reader cannot tell the difference.
The prompts asked for it. Every path that writes an answer said some version
of "if the exact answer isn't stated, use what IS stated to give the best
possible answer", with "only say you don't have information if the retrieved
data is truly unrelated" closing the escape hatch. A neighbouring year IS
related, so declining was effectively disallowed.
The missing distinction: inference may CHARACTERISE what the data covers; it
may not MANUFACTURE a value for something the data does not cover.
_GROUNDING_BOUNDARY states that, and is shared by all three answer paths (the
tool-loop system prompt, the forced-synthesis system prompt, and the
final-synthesis instructions) so they cannot drift apart. It explicitly
preserves qualitative inference, because a rule read as "never infer" would
break the synthesis that makes reflect worth having.
Tests split per the convention: the wiring is deterministic and asserted
directly, including that the rule keeps its teeth and its carve-out. The
behavioural pair is marked hs_llm_core and its docstring says plainly what it
does NOT do -- it does not reproduce the incident (verified: it passes against
the pre-fix prompt on two models), it guards the contract, and its more
valuable half is the check that the rule has not become a refusal reflex.
Reproduction lives in hindsight-dev/benchmarks/prelude (hq-absent-2024).
488 reflect/prompt tests pass; the golden prompt fixture is updated.
* test(dev): accept reflect's accurate UK qualification on the scoped_truth question
The criteria said 2FA is "mandatory for EU (and UK) accounts", which reads as
unconditional. The memory actually says UK accounts are mandatory FROM 2026,
and reflect answered "UK Accounts: Mandatory starting in 2026" -- more precise
than the criteria, and marked wrong for it (1 run in 6).
Scoring a correct answer as a failure is the worse error for a benchmark: it
manufactures a defect to chase. The criteria now accepts any accurate
treatment of the UK and keeps the real assertion, which is that the answer
must be regionally qualified rather than a flat yes or no.
* fix(mental-models): stop delta ops treating the batch-only synthesis as authoritative
The delta refresh writes a synthesis from the new batch alone, then asks a
second call to merge it into the stored page. That call read the synthesis as
evidence: its "a total of 4" counted only the batch and replaced a page's 3
customers with 4; its "no release was deployed" described only the batch and
overwrote a production release recorded one wave earlier.
Label the synthesis UNTRUSTED (only the supporting facts justify an operation)
and add the combine-not-swap, absence-is-not-contradiction and refutation
threshold rules. Replayed against both captured failures: 0/5 -> 5/5.
* test: add hindsight-system-evals, published as a quality metric by the perf monitor
Blackbox quality evals over a real hindsight-api and a real model, through the
published Python client only — the system-tests shape without the stub, since
stubbing the model would score the stub. First suite: knowledge-page
convergence, the eval that found the delta-ops regressions. Each page is
graded twice: correct, and whether it stores the specific baited falsehood.
Runs in perf-test.yml (daily, not on PRs — needs secrets, and one red run is as
likely noise as regression) and publishes correct rate and trap count to the
continuous performance monitor. Seeding uses chunks retain with consolidation
off, so the only model calls are the ones under test: minimum acceptance ~90s,
full ~5 min.
Also: the hindsight-dev knowledge-page tools used to find and replay those
failures (kp_eval, kp_diagnose, replay_gemini, iterate_delta_prompt).
* refactor: move the reflect evals into hindsight-system-evals and trim the new prompt text
Everything from hindsight-dev/benchmarks/prelude now lives in the blackbox
package, driven only through the public client:
- test_02_reflect_answers: the one-shot reflect eval (minimum acceptance is the
2024-headcount incident), with retrieval-vs-reasoning blame from the tool trace;
- debug/diagnose_page: dry-runs a page's second refresh and dumps every traced
prompt; debug/replay_delta_ops: replays one captured delta-ops request per
prompt variant and interrogates the model.
The retrieval floor/ceiling tier is dropped: it measured the planned-prelude
change (#4066), which was closed, and needed engine internals to author layers.
Prompt size, measured against main:
- grounding boundary rewritten compactly and no longer repeated in the final
instructions (the final and reduce calls already carry it in the system
prompt, so it was sent twice): tool loop +131 tokens, final +112;
- delta ops: the combine-not-swap rule removed. Ablation by replaying the two
captured failures: every other piece is load-bearing (dropping the absence or
refutation rules, or the long synthesis paragraph, falls to 2-3/5), this one
is not (5/5 on both without it). +491 tokens.
Full run on the result: 14/14 correct, 0 traps (7 pages, 7 reflect answers).
* test(system-evals): drop an unused property and type the page eval tests