mirror of
https://github.com/Dicklesworthstone/coding_agent_session_search.git
synced 2026-09-22 11:59:25 +08:00
Dicklesworthstone
4198cc1cd4
Add a repeatable search-quality evaluation harness so relevance/trust
regressions are visible before release: cass can pass every unit test
while search quality silently regresses, and relevance is not something a
type checker can prove.
- src/search_quality_eval.rs: pure, deterministic, metadata-only core.
Qrel + ObservedHit + QueryRun -> recall@k / precision@k / MRR, latency,
realized/fallback mode, a trust-tier distribution, and a regression/drift
diff (diff_reports). build_report + render_markdown + render_diff_markdown.
The report holds only refs + numeric metrics (never body text), so it
cannot leak raw private text. 17 unit tests.
- tests/e2e_search_quality_gate.rs: real-binary gate. Seeds a checked-in
corpus as Codex sessions, indexes once, runs each checked-in qrel via
`cass search --json --robot-meta`, reduces hits to metadata-only
ObservedHits, and asserts observed==expected on a real index, the live
trust verdict feeds the distribution (the aged session scores stale),
JSON+markdown artifacts carry id/expected/observed/mode/latency/diff, and
the privacy fixture's body email never reaches an artifact.
- tests/fixtures/search_quality/{corpus.json,qrels.json}: small checked-in
corpus + judgments (the CI suite; larger corpora stay separate/manual).
Proof: unit 17/0, e2e 1/1 (5 qrels pass, mean recall/precision/mrr=1.0,
trust_dist={stale:1, unverified:5}), clippy clean on the new files,
rustfmt clean, UBS 0 critical. Unblocks 5u82n.13.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>