Builds out the hybrid search stack described in the semantic-search plan.
- Embedders behind an `embedder_registry`: `hash_embedder` (FNV-1a feature
hashing as an explicit degraded mode), `fastembed_embedder` (pure-Rust native
MiniLM), with `model_download` / `model_manager` handling the local model
lifecycle and `pack_planner` sizing the work.
- Reranking via `reranker_registry` (pure-Rust cross-encoder) and RRF-style
fusion in search/query; `trust_scoring` + `trust_correlation` weight results
and `storage_integrity` / `contention_diagnostics` keep the vector path safe.
- indexer/semantic drives semantic indexing; citation-path comparisons annotated
`ubs:ignore` (public paths, never secret material).
- Planning + reference docs and bakeoff scripts capture the design and the
trust-meta e2e gate.
Co-Authored-By: Claude <noreply@anthropic.com>
This lands the long-running "doctor v2" effort (tracked in
docs/planning/DOCTOR_V2_RELEASE_CHECKLIST.md). It reshapes `cass doctor`
from a flat "check + auto-fix" command into a typed surface of bounded
operations whose recovery model is **archive-first**: cass-owned evidence
(archives, raw-session mirrors, backups, receipts, source ledgers) is
preserved before any repair runs, and every mutating action goes through a
plan-fingerprint candidate flow that the operator must inspect and approve.
==== New / changed doctor surfaces (`src/doctor.rs`) ====
DoctorCommandSurface now enumerates: Diagnose, Check, Repair, Cleanup,
**ArchiveScan**, **ArchiveNormalize**, Backups, Reconstruct, Restore,
SafeAutoFix, Init.
DoctorExecutionMode adds the corresponding `*DryRun` / `*Apply` pairs for
ArchiveNormalize and tightens the existing Cleanup/Repair modes around
plan-fingerprint matching. DoctorCommandRequest grows `archive_scan` and
`archive_normalize` flags and the plumbing to keep the typed surface
exhaustive (compile-time match coverage on every variant).
Repair, restore, archive-normalize, and cleanup all emit
`operation_outcome` envelopes and machine-readable receipts; the SafeAutoFix
path is preserved as the legacy escape hatch for low-risk derived-only
repairs (Tantivy index rebuild, stale lock removal) and now fails closed if
it detects archive- or source-affecting risk.
==== Source / indexer / storage hardening (`src/sources/*`, `src/indexer/*`, `src/storage/sqlite.rs`) ====
- `src/sources/sync.rs` — `remote_spec_for_scp`, `parse_remote_home_stdout`,
and friends so the SSH/SCP source-mirror path quotes remote operands
consistently and rejects ambiguous remote-home outputs (more than one
candidate path → refuse instead of guessing). Closes the #raw-mirror
scan-root capture-before-parse hole that 5c1dba9e / 5be8a6e9 first surfaced.
- `src/sources/config.rs`, `index.rs`, `install.rs`, `probe.rs`, `setup.rs`
— typed source-provider lifecycle: discovery, probe, install, setup, sync
now share a single error vocabulary the doctor surface understands so its
candidate plans can describe both what they will do and which source the
action is rooted at.
- `src/indexer/mod.rs`, `src/indexer/refresh_ledger.rs` — small fixes to
bind refresh ledgers to source-authority decisions.
- `src/storage/sqlite.rs` — extra integrity probes called by `doctor check`
before any cleanup or archive operation runs.
==== UI / HUD / CLI surface ====
- `src/ui/app.rs` — new `DoctorHudSummary` (parsed from the unified
doctor runtime summary) drives a HUD lane that surfaces archive coverage,
source-mirror state, fallback mode, sole-copy warnings, active repair and
index-maintenance, and the recommended next action. Operators no longer
have to leave the TUI to know whether running a repair is safe.
- `src/lib.rs` — large surface change: every CLI subcommand that touched
doctor surfaces is rewired through the bounded request/response types,
the new archive-scan / archive-normalize subcommands are exposed, the
receipts and plan-fingerprint payloads are emitted on stdout (json/jsonl)
with stderr reserved for fatal envelopes, and the `--robot` mode learns
the new doctor JSON shape (see golden updates).
- `src/update_check.rs` — version-check now feeds into the doctor HUD as
one of the recommended-action sources (an outdated cass that knows about
archive issues should still nudge the operator to upgrade first).
==== Tests + golden snapshots ====
- `tests/cli_doctor.rs` (+2086 lines) — behavioral coverage for every new
subcommand, plan-fingerprint mismatch handling, dry-run/apply parity,
archive-coverage gating of mutating ops, and the legacy `cass doctor`
alias mapping into the v2 surface.
- `tests/doctor_e2e_runner.rs`, `tests/util/doctor_e2e_runner.rs`,
`tests/util/doctor_fixture.rs`, `tests/doctor_fixture_factory.rs` —
end-to-end doctor harness updates: synthesizes archive trees, raw-session
mirrors, partial backups, and corrupt SQLite headers; asserts that
archive-normalize and repair never escape into source / archive paths.
- `tests/doctor_release_checklist.rs` (new) — release-gate test that
cross-checks the runbook against the implemented surfaces (every entry in
DOCTOR_V2_RELEASE_CHECKLIST.md must map to a known DoctorCommandSurface
variant; new variants must add a checklist entry).
- `tests/cli_robot.rs`, `tests/cli_status.rs`, `tests/cli_dispatch_coverage.rs`,
`tests/e2e_health.rs` — smaller dispatch-coverage / status / robot
surface-area follow-ups required by the new doctor shape.
- `tests/golden_robot_docs.rs`, `tests/golden_robot_json.rs` — assertion
helpers for the regenerated robot JSON / docs goldens.
- `tests/golden/robot/*.json.golden` and
`tests/golden/robot_docs/*.txt.golden` — regenerated for the new shapes;
`introspect.json.golden` and `introspect_shape.json.golden` carry the
bulk of the diff because they enumerate the full doctor surface +
argument schema.
==== Docs + scripts ====
- `docs/planning/RECOVERY_RUNBOOK.md` — operator runbook (referenced from
the README) describing the archive-first model, plan-fingerprint flow,
source-pruning warnings, sole-copy prompts, and the support-bundle
handoff.
- `docs/planning/DOCTOR_V2_RELEASE_CHECKLIST.md` (new) — release gate; the
new checklist test consumes this file.
- `README.md` — replaces the old single-doctor table with the v2 surface
table and updated examples (`cass doctor check --json`,
`cass doctor repair --plan-fingerprint`, etc.).
- `CHANGELOG.md` — Unreleased entry covering the archive-first
documentation and the doctor migration guidance for existing data dirs.
- `scripts/e2e/doctor_v2.sh`, `scripts/bakeoff/cass_validation_e2e.sh` —
scripted runners for the doctor v2 surface; consumed by the e2e harness
and the cass bakeoff campaign.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/bakeoff/cass_validation_e2e.sh:
- Add EMBEDDER env var (default: fastembed, options: hash, fastembed)
- Pass --embedder flag to cass index when in semantic/hybrid mode
Hash embedder results vs lexical baseline:
- Lexical: NDCG@10 = 0.0412, p50=224ms, p95=270ms
- Semantic (hash): NDCG@10 = 0.1177, p50=255ms, p95=454ms
(2.9x quality improvement with similar latency)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
scripts/bakeoff/cass_validation_e2e.sh:
- Extract topic keyword (first word) from synthetic queries like
"metrics search query 0" → "metrics" to match corpus doc format
- Pass CASS_DATA_DIR env to subprocess.run to ensure cass finds the
correct isolated index
This fixes zero-hits issue where full query strings didn't match
the synthetic corpus text patterns.
Before: NDCG@10 = 0.0000, 0 hits across all queries
After: NDCG@10 = 0.0412, 50/50 queries with hits, p50=224ms, p95=270ms
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Introduces comprehensive documentation for the bakeoff validation process
and adds integration testing for the daemon client. Includes shell scripts
for automated E2E validation of the cass bakeoff system.
- Add cass_bakeoff_validation.md documenting validation methodology
- Create bakeoff E2E validation shell script
- Add daemon directory with startup/management scripts
- Implement daemon client integration tests
Co-Authored-By: Claude <noreply@anthropic.com>
Dicklesworthstone