41 Commits

Author SHA1 Message Date
Dicklesworthstone b8fc786ae4 fix(index): preserve incremental work and complete pending analytics safely
Retain Devin commits from the native watermark second, preserve source
completion across bounded and signal-driven stops, and repair stale initial
analytics cursors without double-counting. Expose pending OMP analytics in
status/doctor/validation and complete all projections under the existing
maintenance lock while preserving published search and canonical identities.

Keep statistics errors and ordering truthful; combine unfiltered conversation
scans without claiming a measured speedup. Extend real CLI coverage for
deferred analytics, WAL appends, replay, interrupted resume and timeout output.

RCH v8c: STAGE=fmt EXIT=0; STAGE=clippy EXIT=0;
STAGE=lib-tests EXIT=0 (225 passed); selected integrations86 passed;
STAGE=goldens EXIT=0 (68 passed). Original OMP helper failed by reading the
wrong error stream; its retained test-only follow-up passed1/1 with
STAGE=fmt EXIT=0, STAGE=clippy EXIT=0 and STAGE=test-connector_omp EXIT=0.
All remaining runtime inputs and application ELF were unchanged.

Full gate is RED: STAGE=ubs EXIT=1 (Rust300s timeout; original Python warning).
The separately reviewed Python line pragma passed its pinned strict scan and
16 controls. This does not clear Rust UBS or authorize publication.
Large-copy stats10.25s/1498792KiB showed no clear improvement.
All26 original issue acceptances remain open; no release or native proof.

Broad receipt SHA256: d9552cf8bd1bf7b31766b141594513c19212c7fd575cc8ebc1a9c149117014ef
OMP receipt SHA256: f51bc7bb92563b1f574779024377f939be13f0e4ee9c4592ffe33a0c758f2135
Application SHA256: 1182c93d1ba643a1988cb628796d6e33a29c2fed6fd81ca16d5f9af4f75d4241
2026-09-11 04:14:37 -04:00
Dicklesworthstone 8ec967ae32 Parse live-fleet JSON without echoing private inventory on decode failure.
json_document raises ValueError with line/column only, so a malformed
inventory or remote command receipt stays in its existing file. Sync,
discovery, Tailscale baseline, and mirror-reingest all use it. UBS
acceptance asserts that require JSON booleans and exact identity sets
are marked ubs:ignore so the scanner does not rewrite them.
2026-09-11 01:45:18 -04:00
Dicklesworthstone 5d187f8760 Unreleased: Shelley connector, reingest --source, schema-only opens, backfill skip cache.
Enable FAD 0.2.4 `shelley` in Cargo.toml/build.rs and document CASS_SHELLEY_DB
defaults, live WAL/SHM watch, CASS_SKIP_SUBAGENTS parent-ID skip, and the
sensitive-container policy. Raw mirroring and remote DB ingest stay disabled
for Shelley; doctor will not promote a Shelley file into a repairable mirror
(GH#415 / 3kde4).

sources reingest --source now selects remote roots instead of only narrowing
the report, preserves local watermarks so later ordinary indexing still finds
local history, fails mixed valid/unknown names before archive creation, hashes
configured paths including a bare `~`, and no longer opens the database before
the index lock (av59c). Live-fleet harness resolves TMPDIR before the
containment check so a symlink cannot plant the private inventory in-tree.

Current-schema archive opens use schema-only admission and skip whole-database
hydration while keeping required migration/repair checks (#443/#450). Semantic
backfill rebuilds the known legacy hash-vector format; on Unix an unchanged
completed backfill can skip canonical replay, bound to archive/WAL/vector and
producer identity. That cache is a maintenance skip hint, never search-serving
authority (#458). Combined remote validation and remaining issue closure are
not claimed.
2026-09-10 22:46:14 -04:00
Dicklesworthstone 28f65e9fd6 Treat null TailscaleIPs as empty and compare live-search content plus provenance (av59c).
merge_tailscale_hosts deserializes TailscaleIPs as Option<Vec<IpAddr>> so JSON
null or a missing field skips that peer instead of rejecting the whole status
document. The mixed-peer unit test now includes an online peer with null
addresses.

The live harness compares (source_id, path, line, origin_host, content)
across global, source-scoped, and default-hybrid hits, not just identity
triples. CHANGELOG/CHANGELOG_RESEARCH record the 02:45 UTC review: 91 tests,
nine authenticated machines, tenth still needs auth, UBS MODULE_TIMEOUT.
Final ELF 7241298dc93353fe3aef6456c092bef5ac87c36831e43a84b9c6f990551268d4.

Does not close av59c. Skip .release-format/.
2026-09-10 03:12:44 -04:00
Dicklesworthstone 04f0163fdf Bound SSH/Tailscale child stdout/stderr and keep draining overflow (av59c).
wait_for_child_output_with_timeout now wraps wait_for_child_output_with_limit
(unlimited). When max_bytes is set, each pipe reads at most limit+1 then
drains the rest to sink so a chatty child cannot deadlock on a full pipe;
overflow is InvalidData. Tailscale status already calls the limited helper.

Unix tests: 4-byte limit keeps 1234/abcd, rejects 5-byte stdout or stderr,
and still times out after overflow plus sleep. The live harness catches
Exception (not a short list) so unexpected failures write failure.json
without printing private paths.

Skip .release-format/.
2026-09-09 22:07:15 -04:00
Dicklesworthstone da54a759f7 Treat Match as a new SSH block and skip already-configured hosts by address (av59c).
parse_ssh_config now closes the current Host set on Match so conditional
options cannot attach to the preceding literal alias. Host tokens are split
with shell_words (quoted aliases), stop at comments, and reject glob/negate/
unsafe/leading-dash names. Hostname/User/Port/IdentityFile keep the first
value so a later HostName cannot overwrite a quoted address.

sources discover --skip-existing treats a host as configured when the source
name matches or the SSH target host (after user@) matches the discovered
name/hostname, so a Tailscale IPv4 peer that is already operator@example
is not offered again. JSON already_configured uses the same predicate.
will_reindex is true only when indexing will actually run (files > 0).
The discover next-step hint uses cass sources add <user>@<host> --name --path.

Tests: Match+quoted HostName first-wins still surfaces the extra tailnet
peer; --skip-existing --json reports all_existing for a configured target.
The live harness requires source-scoped identities to equal the global hit
set, refuses PYTHONOPTIMIZE, and maps invalid inventory/config to a parser
error without printing private paths.

Skip .release-format/. Does not close av59c.
2026-09-09 22:07:15 -04:00
Dicklesworthstone a8655107b3 Document --tailscale in CHANGELOG and require a Tailscale-only inventory alias (av59c).
CHANGELOG notes optional sources discover/setup --tailscale IPv4 merge with
SSH-config fallback. CHANGELOG_RESEARCH records that the new capability is
not covered by the earlier nine-host SSH proof; a frozen remote gate and
private live tailnet run are still pending.

The live harness --tailscale path now compares against SSH-only discover and
requires at least one inventory target that is absent from configured SSH
aliases, so a run cannot pass on SSH-config discovery alone.

Does not close av59c. Skip .release-format/.
2026-09-09 18:42:05 -04:00
Dicklesworthstone cc32c30df5 Prove --tailscale discover fallback and opt-in live Tailscale fleet discovery (av59c).
sources discover --tailscale --json with an empty PATH keeps the SSH Include
hosts and reports discovery_warning containing "could not start", using a
missing executable directory rather than a fake tailscale binary.

The live-fleet harness gains --tailscale: discovery then requires
--tailscale and an empty discovery_warning. Inventory SSH targets are
expected to be tailnet IPs when that flag is set. Default runs stay
SSH-config-only.

Does not close av59c. Skip .release-format/.
2026-09-09 18:36:07 -04:00
Dicklesworthstone 182c197b1c Honor CASS_SSH_CONFIG Includes and emit one JSON document for sync/reingest indexing (av59c).
Discovery previously always read ~/.ssh/config and ignored Include files, so a
private override that transport already used produced an empty host list while
direct SSH through the same file worked. Discovery now follows the same
CASS_SSH_CONFIG / ssh_config_override path, concatenates Include fragments with
bounded recursion (depth 16, 256 files, 1 MiB), skips duplicate aliases, roots
relative Includes at ~/.ssh per ssh_config(5), and never executes Match exec.

sources sync --json and sources reingest --json used to print a premature sync
document and then a second indexing document. run_index_with_data now accepts
an optional captured_result so robot sync/reingest emit one nested payload.
Indexing lock refusal reports status index_failed with indexing.code 7 and
keeps the nonzero exit, rather than advertising completed indexing.

The live-fleet harness wraps the operator SSH config with Host
cass-live-unreachable, records discovery-pending ordinals, holds
index-run.lock via fcntl, asserts expected_exit 7/8, numbers append events,
resolves ssh -G users instead of substituting the runner, and checks default
hybrid identity against lexical hits. README documents the opt-in inventory
outside git (example hosts laptop/workstation, not css/csd/yto). CHANGELOG
and CHANGELOG_RESEARCH record the nine-reachable-host proof (18/18/18/27/36
hits; busy exit 7; refused-SSH exit 8) and the 9lz4y rustfmt-only doctor
collector already on main.

Tests: sources_discover_uses_private_ssh_config_override_and_includes and
sources_reingest_json_preserves_index_result_and_busy_exit.

Does not close av59c: UBS MODULE_TIMEOUT, auth-blocked tenth host, and
setup-timeout host selection remain open. Skip .release-format/ scratch.
2026-09-09 18:36:07 -04:00
Dicklesworthstone 761e4e5374 Add an opt-in real-SSH fleet search harness that never stores host inventory in git (av59c).
scripts/e2e/live_fleet_search.py takes --inventory and --cass-bin. The
inventory file must live outside the repo with mode 0600. Each host gets
a fresh cass-live-fleet temp dir and a synthetic Codex rollout session;
only those sessions are transferred. Strict host-key checks, no archive
edits, no file deletes. Console output uses ordinal host labels, never
identities. Unreachable hosts fail the run rather than skip. Skip
.release-format/.
2026-09-09 17:24:51 -04:00
Dicklesworthstone 78ea1ecb43 fix(e2e): bind strict-RCH evidence to immutable runs (coding_agent_session_search-k13gt) 2026-07-28 16:27:50 -04:00
Dicklesworthstone aefd6357b9 fix(e2e): bound semantic trace artifacts (coding_agent_session_search-jyfuq.1) 2026-07-28 10:02:31 -04:00
Dicklesworthstone 916c59471f chore(v0.4.3): align CI / e2e / validation shell scripts with the new CLI surface
The v0.4.3 CLI surface added aliases (cass index, robot docs topics, current
session shorthands, html export, query, search hit source, line_number
drill-down) and the new robot-trace-ingest flag. The supporting shell
scripts get aligned with the canonical names so the e2e and CI scripts
keep matching what the binary actually accepts.

Touches:
- scripts/e2e/connector_stress.sh, query_parser_e2e.sh,
  security_paths_e2e.sh, e2e_logging_acceptance_test.sh,
  full_coverage_validation.sh: invoke the new alias-free canonical CLI
  forms and consume the v0.4.3 robot envelope fields.
- scripts/test-all.sh, scripts/test-pages-e2e.sh, scripts/tests/run_all.sh:
  pick up the new test files (tests/cli_index.rs, doctor_e2e_runner.rs,
  e2e_search_index.rs, e2e_tui_smoke_flows.rs, metamorphic_introspect_schema.rs)
  in their shard plans.
- scripts/validate-e2e-jsonl.sh: handles the new ingest-trace NDJSON shape.
- scripts/validate_ci.sh: minor flag-set update to match the v0.4.3 CLI.

No production behaviour change here — these are CI/test plumbing edits.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 15:38:02 -04:00
Dicklesworthstone 2d58a3efbf fix(e2e): isolate duplicate acceptance output 2026-05-08 05:34:56 -04:00
Dicklesworthstone aa219639ac fix(e2e): archive duplicate logging acceptance artifacts 2026-05-08 05:27:55 -04:00
Dicklesworthstone 74e6401e46 fix(e2e): prefer rch binary in harnesses 2026-05-08 05:07:00 -04:00
Dicklesworthstone f74c681f7d fix(e2e): prefer rch sources sync binary 2026-05-08 05:05:35 -04:00
Dicklesworthstone 489b788ae3 fix(e2e): prefer rch cli flow binary 2026-05-08 05:04:20 -04:00
Dicklesworthstone ea8d7cd21b fix(e2e): rch-wrap doctor v2 harness 2026-05-08 04:53:22 -04:00
Dicklesworthstone 5588dd7604 fix(e2e): rch-wrap multi-machine sync bootstrap 2026-05-08 04:51:34 -04:00
Dicklesworthstone 25b67d2cd9 fix(e2e): rch-wrap semantic index bootstrap 2026-05-08 04:43:48 -04:00
Dicklesworthstone 5a97520d36 fix(e2e): rch-wrap logging acceptance tests 2026-05-08 04:42:00 -04:00
Dicklesworthstone 8eb9597158 fix(e2e): rch-wrap cli flow bootstrap 2026-05-08 04:34:10 -04:00
Dicklesworthstone e9fc8139e6 fix(e2e): rch-wrap full coverage cargo gates 2026-05-08 04:34:08 -04:00
Dicklesworthstone d551ade22f fix(e2e): rch-wrap sources sync bootstrap 2026-05-08 04:31:15 -04:00
Dicklesworthstone fad576e5cb fix(e2e): rch-wrap connector stress bootstrap 2026-05-08 04:28:24 -04:00
Dicklesworthstone 7ef18f6bbd fix(e2e): rch-wrap security paths bootstrap 2026-05-08 04:25:47 -04:00
Dicklesworthstone 2640642805 fix(e2e): rch-wrap query parser bootstrap 2026-05-08 04:22:02 -04:00
Dicklesworthstone 2aa2cc92a7 feat(doctor): doctor v2 — archive-first, candidate-based recovery model
This lands the long-running "doctor v2" effort (tracked in
docs/planning/DOCTOR_V2_RELEASE_CHECKLIST.md). It reshapes `cass doctor`
from a flat "check + auto-fix" command into a typed surface of bounded
operations whose recovery model is **archive-first**: cass-owned evidence
(archives, raw-session mirrors, backups, receipts, source ledgers) is
preserved before any repair runs, and every mutating action goes through a
plan-fingerprint candidate flow that the operator must inspect and approve.

==== New / changed doctor surfaces (`src/doctor.rs`) ====

DoctorCommandSurface now enumerates: Diagnose, Check, Repair, Cleanup,
**ArchiveScan**, **ArchiveNormalize**, Backups, Reconstruct, Restore,
SafeAutoFix, Init.

DoctorExecutionMode adds the corresponding `*DryRun` / `*Apply` pairs for
ArchiveNormalize and tightens the existing Cleanup/Repair modes around
plan-fingerprint matching. DoctorCommandRequest grows `archive_scan` and
`archive_normalize` flags and the plumbing to keep the typed surface
exhaustive (compile-time match coverage on every variant).

Repair, restore, archive-normalize, and cleanup all emit
`operation_outcome` envelopes and machine-readable receipts; the SafeAutoFix
path is preserved as the legacy escape hatch for low-risk derived-only
repairs (Tantivy index rebuild, stale lock removal) and now fails closed if
it detects archive- or source-affecting risk.

==== Source / indexer / storage hardening (`src/sources/*`, `src/indexer/*`, `src/storage/sqlite.rs`) ====

- `src/sources/sync.rs` — `remote_spec_for_scp`, `parse_remote_home_stdout`,
  and friends so the SSH/SCP source-mirror path quotes remote operands
  consistently and rejects ambiguous remote-home outputs (more than one
  candidate path → refuse instead of guessing). Closes the #raw-mirror
  scan-root capture-before-parse hole that 5c1dba9e / 5be8a6e9 first surfaced.
- `src/sources/config.rs`, `index.rs`, `install.rs`, `probe.rs`, `setup.rs`
  — typed source-provider lifecycle: discovery, probe, install, setup, sync
  now share a single error vocabulary the doctor surface understands so its
  candidate plans can describe both what they will do and which source the
  action is rooted at.
- `src/indexer/mod.rs`, `src/indexer/refresh_ledger.rs` — small fixes to
  bind refresh ledgers to source-authority decisions.
- `src/storage/sqlite.rs` — extra integrity probes called by `doctor check`
  before any cleanup or archive operation runs.

==== UI / HUD / CLI surface ====

- `src/ui/app.rs` — new `DoctorHudSummary` (parsed from the unified
  doctor runtime summary) drives a HUD lane that surfaces archive coverage,
  source-mirror state, fallback mode, sole-copy warnings, active repair and
  index-maintenance, and the recommended next action. Operators no longer
  have to leave the TUI to know whether running a repair is safe.
- `src/lib.rs` — large surface change: every CLI subcommand that touched
  doctor surfaces is rewired through the bounded request/response types,
  the new archive-scan / archive-normalize subcommands are exposed, the
  receipts and plan-fingerprint payloads are emitted on stdout (json/jsonl)
  with stderr reserved for fatal envelopes, and the `--robot` mode learns
  the new doctor JSON shape (see golden updates).
- `src/update_check.rs` — version-check now feeds into the doctor HUD as
  one of the recommended-action sources (an outdated cass that knows about
  archive issues should still nudge the operator to upgrade first).

==== Tests + golden snapshots ====

- `tests/cli_doctor.rs` (+2086 lines) — behavioral coverage for every new
  subcommand, plan-fingerprint mismatch handling, dry-run/apply parity,
  archive-coverage gating of mutating ops, and the legacy `cass doctor`
  alias mapping into the v2 surface.
- `tests/doctor_e2e_runner.rs`, `tests/util/doctor_e2e_runner.rs`,
  `tests/util/doctor_fixture.rs`, `tests/doctor_fixture_factory.rs` —
  end-to-end doctor harness updates: synthesizes archive trees, raw-session
  mirrors, partial backups, and corrupt SQLite headers; asserts that
  archive-normalize and repair never escape into source / archive paths.
- `tests/doctor_release_checklist.rs` (new) — release-gate test that
  cross-checks the runbook against the implemented surfaces (every entry in
  DOCTOR_V2_RELEASE_CHECKLIST.md must map to a known DoctorCommandSurface
  variant; new variants must add a checklist entry).
- `tests/cli_robot.rs`, `tests/cli_status.rs`, `tests/cli_dispatch_coverage.rs`,
  `tests/e2e_health.rs` — smaller dispatch-coverage / status / robot
  surface-area follow-ups required by the new doctor shape.
- `tests/golden_robot_docs.rs`, `tests/golden_robot_json.rs` — assertion
  helpers for the regenerated robot JSON / docs goldens.
- `tests/golden/robot/*.json.golden` and
  `tests/golden/robot_docs/*.txt.golden` — regenerated for the new shapes;
  `introspect.json.golden` and `introspect_shape.json.golden` carry the
  bulk of the diff because they enumerate the full doctor surface +
  argument schema.

==== Docs + scripts ====

- `docs/planning/RECOVERY_RUNBOOK.md` — operator runbook (referenced from
  the README) describing the archive-first model, plan-fingerprint flow,
  source-pruning warnings, sole-copy prompts, and the support-bundle
  handoff.
- `docs/planning/DOCTOR_V2_RELEASE_CHECKLIST.md` (new) — release gate; the
  new checklist test consumes this file.
- `README.md` — replaces the old single-doctor table with the v2 surface
  table and updated examples (`cass doctor check --json`,
  `cass doctor repair --plan-fingerprint`, etc.).
- `CHANGELOG.md` — Unreleased entry covering the archive-first
  documentation and the doctor migration guidance for existing data dirs.
- `scripts/e2e/doctor_v2.sh`, `scripts/bakeoff/cass_validation_e2e.sh` —
  scripted runners for the doctor v2 surface; consumed by the e2e harness
  and the cass bakeoff campaign.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 17:17:14 -04:00
Dicklesworthstone 9b061f12be feat(doctor): emit structured root-cause incidents and harden V15 migration
Two related advances on top of the doctor command surface landed in
15f6bba3:

1) Structured root-cause incidents
2) V14 → V15 migration safe re-run

src/lib.rs (+1736 LOC):
- New DoctorIncidentRootCauseKind enum with kebab-case Serialize:
  derived-index-stale, derived-index-stale-after-db-promotion,
  semantic-model-missing-lexical-fallback,
  source-pruned-with-mirror-intact, mirror-missing-with-db-sole-copy,
  archive-db-unreadable-with-valid-candidate, archive-db-unreadable,
  active-lock-blocking-repair, interrupted-repair-state,
  storage-pressure-derived-cleanup-available, backup-exclusion-risk,
  backup-unverified, previous-repair-failed, unknown
- New DoctorRootCauseIncident struct surfaced through cass doctor
  check --json: schema_version, incident_id, root_cause_kind,
  severity, affected_asset_classes, archive_risk_level,
  derived_risk_level, confidence, evidence_check_ids,
  blocked_actions, safe_next_actions
- Replaces the prior "string-typed advisory" output so downstream
  tooling (br/bv triage, ops dashboards) can pivot on the typed kind
  rather than scraping wording out of free-form text
- Doctor check evaluator now classifies each evidence row into one
  of the kinds above, attaches blocked_actions (e.g. cannot run
  reconstruct while archive db unreadable) and safe_next_actions
  (e.g. run cass doctor cleanup --derived first), and includes the
  evidence_check_ids that produced the classification so the report
  is auditable end-to-end

src/storage/sqlite.rs (+109 / -33):
- MIGRATION_V15 split: MIGRATION_V15_TAIL_STATE_TABLE keeps the
  CREATE TABLE IF NOT EXISTS conversation_tail_state DDL idempotent.
  The two ALTER TABLE conversations ADD COLUMN statements now run
  conditionally via franken_table_column_names() so retrying the
  migration on a partially-promoted database (where last_message_idx
  was added but last_message_created_at was not, or both were added
  but conversation_tail_state was not) does not error with
  "duplicate column name"
- discover_historical_database_bundles comment block compressed:
  the V13 / V14 / lazy-FTS reasoning is preserved but the redundant
  "MIGRATION_FRESH_SCHEMA (V13) creates fts_messages eagerly..."
  preamble is gone now that the gating logic itself is the
  authoritative source of truth

src/doctor.rs (+12 LOC) routes the new incident report through the
typed surface so check / repair / cleanup variants all consume the
same structured payload.

src/indexer/mod.rs (+11) and src/search/asset_state.rs (+13) wire
the new incident types into the existing asset-state classifier so
"derived index stale" vs "derived index stale after db promotion"
flow into the right cleanup recommendation.

Test coverage:
- tests/cli_doctor.rs (+397/-): asserts the JSON shape, kebab-case
  variant emission, and that blocked_actions/safe_next_actions are
  populated for every kind
- tests/doctor_e2e_runner.rs (+154 LOC) and tests/util/doctor_e2e_runner.rs
  (+378/-): drive the full check → repair plan → apply cycle
  end-to-end with realistic incident states
- tests/util/doctor_fixture.rs (+117/-): adds fixtures for the
  V14→V15 partial-promotion scenarios so the migration safe-rerun
  is exercised in CI
- Goldens regenerated for tests/golden/robot/{doctor, doctor_quarantine,
  doctor_shape, introspect, introspect_shape}.json.golden and
  tests/golden/robot_docs/schemas.txt.golden — these are deterministic
  outputs of the new incident schema

Cargo.toml / Cargo.lock / build.rs / README.md: bump franken-agent-
detection pinned rev to 029253c450702a... so the doctor evidence
collector can call into the new connector source-bundle inspection
helper that was added upstream.

scripts/e2e/doctor_v2.sh (+4/-): drive the new
--repair --plan-fingerprint --yes path so the existing E2E shell
script exercises the typed apply boundary, not the legacy --fix flag.

Beads update (.beads/issues.jsonl + .beads/last-touched): mark the
incident-classifier sub-bead in_progress to match the work landing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 20:14:39 -04:00
Dicklesworthstone fe299bb09b feat(doctor): hard-timeout DB-open helper + intentional-failure self-test scenario
Two coordinated additions to the doctor v2 e2e + CLI surface
that together make the doctor's failure paths first-class
testable instead of "should-never-happen" branches.

src/lib.rs (+552 net inserts)

  `open_franken_cli_read_db_with_hard_timeout(path, reason,
                                              timeout)`:
    A hard-timeout wrapper around the existing
    `open_franken_cli_read_db` helper. The naked open path
    is dependent on frankensqlite's internal lock-contention
    backoff, which can stall a CLI invocation indefinitely
    if the DB file is corrupt OR if a runaway process holds
    the lock with no SIGINT path. The wrapper:
      - spawns a worker thread that runs the open
      - sets up a `mpsc::channel`
      - blocks on `rx.recv_timeout(timeout)`
      - on timeout: returns a typed
        `CliError { code: 9, kind: DbOpen, retryable: true }`
        with a "open timed out after Ns" message — operator
        can then run `cass doctor` to investigate the cause
      - on Ok: returns the open conn
      - on disconnect (worker thread panic): returns a typed
        error rather than a panic-aborted CLI

    Pulled out into
    `receive_franken_cli_read_db_open_result_with_hard_timeout`
    so the worker-side and recv-side can be tested
    independently — the timeout / disconnect / Ok arms each
    get their own unit test.

  Wired into the doctor + status + history command paths
  that read from the archive DB so a single corrupt-DB
  scenario never wedges the entire CLI.

scripts/e2e/doctor_v2.sh (+10 lines)
  `--include-failure-self-test` flag. When set, exports
  `CASS_DOCTOR_E2E_INCLUDE_FAILURE_SELF_TEST=1` so the test
  binary picks up the intentional-failure scenario.
  Default behaviour (flag unset) is unchanged.

tests/util/doctor_e2e_runner.rs (~29 net inserts)
  - DoctorE2eCliArgs gains `include_failure_self_test`
    boolean.
  - `selects(scenario)` widens to include scenarios labeled
    `self-test` when the flag is on.
  - `DoctorE2eScenarioSpec::expected_runner_status()` ->
    `"pass"` for normal scenarios, `"fail"` for self-test
    scenarios — pins the contract that the self-test
    intentionally fails.
  - New `failure_self_test_doctor_e2e_scenario()` returns a
    DoctorE2eScenarioSpec named `intentional-failure-self-test`
    with label `self-test` and required JSON pointers that
    deliberately won't be satisfied.

tests/doctor_e2e_runner.rs (~42 net inserts)
  `doctor_e2e_include_failure_self_test_selects_intentional_failure`
    test: parses CLI args including the flag, builds the
    scenario set with the flag-driven extension, asserts the
    intentional-failure scenario is selected, and asserts
    the runner records a `"fail"` status for it.

Pins the contract that the doctor v2 e2e harness can prove
its own failure-detection — running `--include-failure-self-test`
on a healthy build produces a deliberately-failing scenario
that the runner correctly reports as failed (not silently
masked).

`.beads/issues.jsonl` + `.beads/last-touched` advance one
paired touch as the metadata mirror.
2026-05-05 01:05:23 -04:00
Dicklesworthstone 423b67ea95 test(doctor): add scripted e2e artifact runner 2026-05-05 00:50:27 -04:00
Dicklesworthstone 9ece89c3d6 search/index: complete FSVI + frankensearch migration for semantic pipeline and perf tests
Summary:
- Migrates cass semantic indexing/search internals from legacy CVVI/HNSW wrappers to frankensearch-owned FSVI and ANN primitives.
- Replaces remaining in-repo lexical helper implementations with frankensearch lexical APIs where applicable.
- Aligns daemon/indexer/perf suites with on-disk FSVI behavior, normalized vectors, and doc-id metadata parsing.

Details:
- search/query integration:
  - switches query sanitization/boolean parsing/wildcard handling and tantivy query construction to cass_* helpers from frankensearch::lexical.
  - removes legacy local bridge/index conversion paths and local parser helpers superseded by frankensearch.
  - keeps cass-specific orchestration and filter mapping while delegating lexical mechanics.
- vector index facade:
  - retires CVVI-specific structures in favor of frankensearch::index VectorIndex/Writer/SearchParams exports.
  - standardizes index filenames to index-<embedder>.fsvi.
  - preserves cass role-code/filter helper APIs around the new backend.
- ANN module:
  - shrinks local ann_index module to diagnostics/stat payloads and path helpers.
  - removes in-tree HNSW graph build/search logic now provided by frankensearch.
- semantic indexer/worker/model manager:
  - builds FSVI directly via create_with_revision + write_record + finish.
  - uses SemanticDocId serialization and VectorIndex::open for runtime loading and dedup hash recovery.
  - updates rebuild checks to read embedder metadata from FSVI headers.
- indexer/connectors/source robustness:
  - tantivy preflight now uses cass-compatible reader open helper.
  - local root skip behavior only triggers when root is actually missing.
  - franken-agent-detection integration now does targeted connector probing with UnknownConnectors fallback handling.
  - OpenCode timestamp normalization now handles plausible seconds vs milliseconds safely.
  - source install disk probe avoids HOME tilde expansion races under concurrent tests.
- benches/tests:
  - regex cache bench now calls frankensearch lexical cache functions directly.
  - search/memory/perf benches and property tests now generate on-disk FSVI fixtures with normalized vectors.
  - semantic e2e checks updated from .cvvi expectations to .fsvi.
  - semantic e2e script now verifies .fsvi artifacts.

Why:
- Consolidates core search behavior in frankensearch to reduce duplicated implementations and divergence risk.
- Moves cass onto a single canonical vector artifact format and runtime path.
- Keeps benchmark/test coverage aligned with production code paths to catch regressions in the new backend.
2026-02-18 20:50:18 -05:00
Dicklesworthstone b10009f0f7 coding_agent_session_search-3koo: e2e logging + fixture updates 2026-01-27 16:37:28 -06:00
Dicklesworthstone 351430b49d feat(e2e): Add full coverage validation script and expand connector tests
New Master Coverage Script (scripts/e2e/full_coverage_validation.sh):
- Orchestrates unit tests, E2E tests, JSONL validation, and coverage artifacts
- Sources e2e_log.sh library for structured logging
- Tracks pass/fail/skip counters across all test phases
- Generates comprehensive summary.md report
- Creates organized output in test-results/ directory

Connector Test Expansions:
- tests/connector_claude.rs: Expanded Claude connector test coverage
- tests/connector_cline.rs: Expanded Cline connector test coverage
- tests/connector_pi_agent.rs: Expanded Pi Agent connector test coverage

Infrastructure Tests:
- tests/cli_robot.rs: CLI robot mode test improvements
- tests/storage.rs: Storage layer test updates

Co-Authored-By: Claude <noreply@anthropic.com>
2026-01-27 15:30:13 -06:00
Dicklesworthstone 422647177c coding_agent_session_search-ai4a: harden encoded path checks 2026-01-27 14:34:17 -06:00
Dicklesworthstone 550a81f22a Add standard E2E logging to shell test scripts
Implement sources_sync.sh from scratch using e2e_log.sh (was empty).
Add e2e_log.sh sourcing and standard event emission to cli_flow.sh,
semantic_index.sh, and cass_daemon_e2e.sh while preserving their
existing custom JSONL logging (Option A compatibility layer).

All scripts now emit structured JSONL to test-results/e2e/ in addition
to their custom output locations. Remove sources_sync.sh from .gitignore
since it is no longer a stub.

Closes br-20bz.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-27 14:23:55 -06:00
Dicklesworthstone a8df258b53 fix(e2e): Fix daemon E2E test to pass all phases
Two bugs were preventing the daemon E2E test from working:

1. Semantic indexing needed --full flag - incremental scan found no
   messages because the test data timestamps were considered "old"
2. Test data filename pattern was wrong - Codex connector expects
   rollout-*.jsonl, not daemon-e2e.jsonl

Also add scripts/e2e/daemon_fallback.sh wrapper so the daemon test
is included in the orchestrated E2E runner (run_all.sh picks up
scripts/e2e/*.sh).

Part of T7.3: E2E daemon fallback + health script

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-27 01:23:54 -06:00
Dicklesworthstone a1cc4cf797 feat: Add new E2E scripts and Playwright test suites
New E2E infrastructure and test suites:

scripts/e2e/multi_machine_sync.sh:
- Multi-machine sync E2E test script for T7.2
- Tests rsync over SSH workflow
- Validates provenance tracking across hosts

scripts/validate-e2e-jsonl.sh:
- JSONL log file validation script
- Checks schema compliance
- Validates event ordering and completeness

tests/e2e/accessibility/:
- aria-live.spec.ts: Live region announcements
- axe-core.spec.ts: Automated accessibility testing
- visual-preferences.spec.ts: Reduced motion, contrast

tests/e2e/mobile/:
- Touch navigation tests
- Responsive layout verification
- Virtual keyboard interaction
- Mobile performance benchmarks

tests/e2e/offline/:
- network-transitions.spec.ts: Online/offline transitions
- service-worker-cache.spec.ts: Cache validation

tests/reproduction_sync_oscillation.rs:
- Regression test for sync oscillation bug
- Validates stable remote source handling

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-27 00:46:47 -06:00
Dicklesworthstone 4aa5085312 Enhance semantic search with reranker support and model management
Major improvements to the semantic search subsystem:

- Add fastembed-based reranker for improved result relevance
- Implement model manager for dynamic embedding model selection
- Expand indexer semantic capabilities with batch processing
- Enhance TUI with reranker-aware result presentation
- Add e2e test script for semantic indexing validation

The reranker module provides a secondary ranking pass that
significantly improves search result quality for ambiguous queries.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-01-25 01:01:16 -05:00
Dicklesworthstone 46a3cbf077 test(pages): Add comprehensive E2E and performance testing infrastructure
End-to-end testing:
- scripts/e2e/cli_flow.sh: Full CLI workflow test coverage
- tests/pages_pipeline_e2e.rs: Rust integration tests for pages pipeline

Performance testing:
- cass-pages-perf-bundle binary for generating test bundles
- tests/performance/: Node.js-based perf harness with:
  - Lighthouse integration for bundle audits
  - Memory profiler tests
  - Decrypt timing measurements
  - Search latency benchmarks
- scripts/perf/run_pages_perf.sh: Unified perf orchestration

CI integration:
- .github/workflows/perf.yml: Manual dispatch workflow for perf regression
- lighthouse-budget.json: Performance budgets for bundle size/timing

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 00:17:48 -05:00