mirror of
https://github.com/Dicklesworthstone/coding_agent_session_search.git
synced 2026-09-22 11:59:25 +08:00
5f05938484
Correction to 1f642a24's message: the golden leak was not caused by the golden harness's own stale-on-read refresh — verify7 reproduced it with that opt-out in place. The writer is tests/cli_robot.rs: 52 call sites use tests/fixtures/search_demo_data IN PLACE through a base_cmd() that never disabled auto-refresh, and on a fleet worker the checkout is not under the OS temp dir, so a robot-CLI search spawned `cass index --background` into the committed fixture and ingested that worker's real sessions before the golden tests copied it. base_cmd() now sets CASS_AUTO_REFRESH=0. Proof: verify9 ran the in-place robot tests before golden regeneration in one job and search_robot/stats_full_payload came back byte-identical to HEAD. pages key (ctigq): `cass pages key list|add-password|add-recovery|revoke| rotate --archive <bundle>` over the complete key_management engine (3,915 lines, zero callers until now). Passwords via prompt or --password-stdin (current on line 1, new on line 2), never argv; --json; exit 1 engine refused, 2 usage, 3 not a bundle, 6 password required. Four unit tests on a real encrypted bundle plus a CLI suite (tests/pages_key_cli.rs). docs/RECOVERY.md rewritten to the real surface; README row. WAL hygiene (z2uon): forget --apply, dedup --apply and analytics validate --fix close through the index run's checkpointing close; doctor `archive_wal` (pass ≤ 64 MiB; warn with size and remedy; warn-and-defer under an index run; --fix checkpoints and reports pass / blocked / failed truthfully) with a sparse-WAL e2e in tests/cli_doctor.rs. status/health e2e for db_bytes / wal_bytes / shm_present, now on the `status` surface too (q2vn9). Segment pressure (wpe6s, GH #441): quill_bridge::segment_file_count (metadata-only upper bound), status/health `index.segment_files`, doctor `index_segments` (warn above 8× the merge threshold, `cass index --full` remedy), CASS_TEST_SKIP_POST_RUN_LEXICAL_MAINTENANCE hook and an e2e that builds a 40-segment generation, sees the warning, and proves a plain `cass index` consolidates it. GH #439 (vg2e1) and GH #440 (tcrxi) end-to-end proofs: FTS-repair hooks (sleep per page after the heartbeat; park before the first page) — a slow repair survives a 20 s abort window, a parked one exits 70 with the `index-stalled` envelope; a hook parks a force-rebuild between the staged engine commit and the checkpoint write, records the gap, and a plain `cass index` resumes with exit 0 and no duplicate identities. sources (vdyxd, G.1): setup no longer marks the final sync complete without running it — interactive setup runs `cass sources sync` afterwards and records it only on success; --json setup defers it and reports sync.status=pending. Remote install walks candidate_methods() (binstall → verified prebuilt → cargo install → bootstrap) and falls through on failure instead of stopping at the first choice. `cass sources sync --all` exists (jwox6): doctor, robot-docs and the fleet rehearsal recommended it while clap rejected it. installer (apriu, G.2a): install.sh probes host glibc and falls back to build-from-source below 2.38; proven on ubuntu:22.04 (2.35 → source route) and ubuntu:24.04 (2.39 → prebuilt). docs validator (xwuon, A.9): scripts/validate_docs.sh --keys/--flags/--env/ --truth check README key bindings against the TUI key map (123/123, planted fake key fails), every `cass … --flag` usage against `cass … --help`, and every README env var against the code; found and fixed `sources sync --all`, `timeline --days`, the `<mark>` highlight claim and a dead CASS_UI_METRICS row. gate.sh gains an opt-in docs-truth stage. gate (c2xxq): scripts/gate.sh had a false green — verify8 was cut by the fleet's 30-minute SSH ceiling after fmt+clippy and the script still said GREEN. Now a non-zero rch exit is RED, every expected stage (plus a terminal job-complete marker) must report, the receipt captures both streams and survives a RED run, --lib-only/--no-goldens split runs under the ceiling, the lib stage runs under `timeout`, and the fleet target dir no longer inherits the shell's CARGO_TARGET_DIR. contract (hr9rn): introspect/capabilities has_json_output looks through the subcommand tree (11 commands flip to true and gain the robot-format arg). TUI: the unreachable `Alt+W => swarm cockpit` arm removed. CHANGELOG [Unreleased], AGENTS swarm/issue rules, plan §7b progress log. Verification (fleet, scripts/gate.sh STAGE receipts, RUST_MIN_STACK=16777216): - verify9 (2026-09-02 00:35-00:58): fmt 0, clippy 0, goldens-regen 0, goldens 0 with search_robot/stats_full_payload byte-identical to HEAD after the in-place cli_robot tests ran first (leak fix proven); cli_index/cli_robot/cli_status 101 (Codex fixture names, thresholds, health nesting) fixed in tree and re-run below. - verify10 (01:04-01:19): `cargo test --lib` to completion, 6,847 passed / 6 failed / 38 ignored, no wedge; the six (bookmarks missing from CANONICAL_TOP_LEVEL_COMMANDS, four sqlite3-CLI tests on a worker without sqlite3, daemon socket path over sun_path on a long TMPDIR) fixed in tree and re-run below. - verify11 (05:08-05:23): fmt 0, clippy 0, lib-tests 0 (canonical commands, daemon, sqlite, sources install/setup, pages key, quill_bridge, stall diagnostics, fingerprint cache), pages_key_cli 0, cli_robot 0, cli_doctor 0, tui_flows 0, bookmarks_cli 0, goldens-regen 0, goldens 0; cli_index 101 (gh439 reads the last stdout JSON line now) and cli_status 101 (health nests state.database) fixed in tree. - verify12 (05:41-05:47, full lib suite on this tree): fmt 0, clippy 0, lib-tests 0 — 6,855 passed / 0 failed / 38 ignored in 144 s. - verify13 (05:47-05:55, --regen-goldens): fmt 0, clippy 0, lib-tests 0 (quill_bridge, stall diagnostics), cli_status 0, cli_doctor 0, pages_key_cli 0, goldens-regen 0, goldens 0; cli_index 101 because the gh439 slow-repair test parsed the last LINE of a pretty-printed summary (harness bug) - it now stream-parses the last JSON document. - verify14 (05:57-06:00, cli_index gh439 after that fix): fmt 0, clippy 0, test-cli_index 0 (gate: GREEN). Goldens: 12 reviewed diffs (segment_files / db_bytes / wal_bytes / shm_present fields, has_json_output flips, pages key and sources sync --all in the command trees, doctor archive_wal / index_segments checks); the four that differ only by the regeneration writer's dropped final newline were left at HEAD. Measurement (owner's archive: 10.3 GB db, 200 MB WAL, 267 Quill segment files; `search "borrow checker lifetimes" --robot --robot-meta --limit 5`, five warm runs, CASS_AUTO_REFRESH=0): v0.7.1 median wall 15.3 s (search_ms 6.3-7.5 s, other_ms 1.0-1.9 s); this tree (release build) median wall 1.9 s (search_ms 337-406 ms, other_ms 754-810 ms after a first-run 2.6 s sidecar miss). hits=1 on both. Not yet isolated: the engine-time drop happened with the segment count unchanged, so the per-open seal in the old Quill config is the likelier cause than fragmentation on this archive. Consolidation of the owner's 267 segments was NOT achieved: the documented path (`cass index`, and `cass doctor --fix` for the WAL) never leaves the frankensqlite writable open on that archive - one core, 6.7 GB RSS, ~30k 24-byte header reads/s on the 200 MB WAL, no progress in 7 min - while read-only opens take 1.5 s. On a COPY whose WAL C SQLite checkpointed losslessly (1,012 conversations / 538,807 messages before and after), the same `cass index` gets through the open and indexes, so the WAL is the trigger; C SQLite's quick_check on that copy also reports a doubly- referenced freelist page and free-space corruption on ~35 pages that `cass doctor` calls healthy. The archive's last completed index is 2026-08-14. Recorded as a new P0 in the plan (section 7b); the live archive was not modified. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2616 lines
100 KiB
Rust
2616 lines
100 KiB
Rust
//! Golden-file regression tests for cass robot-mode JSON outputs.
|
|
//!
|
|
//! Bead `u9osp`: cass ships a robot/LLM discovery surface via
|
|
//! `cass capabilities --json`, `cass robot-docs --json`, `cass selftest --json`,
|
|
//! `cass health --json`, and `cass models status --json`. These payloads are the contract every
|
|
//! downstream agent consumes — a single renamed field or moved key silently
|
|
//! breaks every consumer without failing any existing test.
|
|
//!
|
|
//! This file freezes the **shape** of those payloads against scrubbed golden
|
|
//! files under `tests/golden/robot/`. Scrubbing rules live in
|
|
//! [`scrub_robot_json`] below; see `tests/golden/PROVENANCE.md` for
|
|
//! regeneration procedure.
|
|
//!
|
|
//! ## Regenerating a golden
|
|
//!
|
|
//! ```bash
|
|
//! UPDATE_GOLDENS=1 rch exec -- env CARGO_TARGET_DIR=/data/tmp/cass-golden-target cargo test --test golden_robot_json
|
|
//! git diff tests/golden/ # review EVERY change
|
|
//! git add tests/golden/
|
|
//! git commit -m "Update robot-mode goldens: <reason>"
|
|
//! ```
|
|
//!
|
|
//! Any diff between `actual` and golden is either a bug or an intentional
|
|
//! schema change that requires human review before it ships.
|
|
|
|
use assert_cmd::Command;
|
|
use coding_agent_search::franken_sync::Connection;
|
|
use coding_agent_search::franken_sync::compat::ConnectionExt;
|
|
use coding_agent_search::franken_sync::params;
|
|
use coding_agent_search::search::tantivy::expected_index_dir;
|
|
use coding_agent_search::subsystem_coverage_matrix::matrix_report;
|
|
use serde_json::{Value, json};
|
|
use std::error::Error;
|
|
use std::fs;
|
|
use std::io;
|
|
use std::path::{Component, Path, PathBuf};
|
|
use std::time::Duration;
|
|
use walkdir::WalkDir;
|
|
|
|
/// Build a `cass` binary invocation with the env knobs required for
|
|
/// deterministic test output (no update check, no ambient data-dir or CWD
|
|
/// connector-discovery surprise).
|
|
fn cass_cmd(test_home: &std::path::Path) -> Command {
|
|
let mut cmd = Command::new(assert_cmd::cargo::cargo_bin!("cass"));
|
|
let codex_home = test_home.join(".codex");
|
|
let claude_home = test_home.join(".claude");
|
|
let gemini_home = test_home.join(".gemini");
|
|
let opencode_root = test_home.join(".opencode");
|
|
let aider_root = test_home.join(".aider-missing");
|
|
let xdg_config_home = test_home.join(".config");
|
|
cmd.env("CODING_AGENT_SEARCH_NO_UPDATE_PROMPT", "1")
|
|
.current_dir(test_home)
|
|
// Pin data dir so the test never touches the user's real cache.
|
|
.env("XDG_DATA_HOME", test_home)
|
|
.env("XDG_CONFIG_HOME", xdg_config_home)
|
|
.env("HOME", test_home)
|
|
.env("CODEX_HOME", codex_home)
|
|
.env("CLAUDE_HOME", claude_home)
|
|
.env("GEMINI_HOME", gemini_home)
|
|
.env("OPENCODE_STORAGE_ROOT", opencode_root)
|
|
.env("CASS_AIDER_DATA_ROOT", aider_root)
|
|
// WS-A.6: the Pi family connectors honor these overrides ahead of
|
|
// the synthetic HOME. On 2026-09-01 a fleet worker's real
|
|
// `~/.pi/agent/sessions` leaked into `search_robot` and
|
|
// `stats_full_payload` through exactly this gap. Pin every override
|
|
// to a path inside the test HOME so the goldens observe only the
|
|
// fixture, whatever the host's environment says.
|
|
.env("PI_SESSIONS_DIR", test_home.join(".pi-sessions-missing"))
|
|
.env("PI_CODING_AGENT_DIR", test_home.join(".pi-agent-missing"))
|
|
.env(
|
|
"PI_CODING_AGENT_SESSION_DIR",
|
|
test_home.join(".pi-coding-agent-sessions-missing"),
|
|
)
|
|
.env_remove("PI_CONFIG_DIR")
|
|
.env_remove("PI_PROFILE")
|
|
// WS-A.6: stale-on-read auto-refresh is suppressed only for data dirs
|
|
// under the OS temp dir, and rch workers place `tempfile` dirs beneath
|
|
// the checkout, so a golden `search`/`stats` could spawn a detached
|
|
// `cass index --background` into the fixture copy. (The 2026-09-01
|
|
// leak itself came from `tests/cli_robot.rs` refreshing the committed
|
|
// fixture in place before these tests copied it; both writers are now
|
|
// off.) Goldens observe a fixture, never a live refresh.
|
|
.env("CASS_AUTO_REFRESH", "0")
|
|
// Never probe or connect to a daemon owned by the host running the
|
|
// golden suite. The default socket is user-global, while each golden
|
|
// must observe only its synthetic HOME.
|
|
.env("CASS_DAEMON_SOCKET", test_home.join("cass-daemon.sock"))
|
|
.env("CASS_IGNORE_SOURCES_CONFIG", "1")
|
|
// Keep resource-policy goldens stable across hosts; dynamic default
|
|
// scaling is covered by responsiveness unit tests.
|
|
.env("CASS_RESPONSIVENESS_MAX_INFLIGHT_BYTES", "536870912");
|
|
cmd
|
|
}
|
|
|
|
/// Keep doctor fixtures independent of a repository that happens to contain
|
|
/// the platform temp directory. RCH workers may place `tempfile` directories
|
|
/// beneath the checkout; without this inner repository boundary, doctor sees
|
|
/// the checkout's `.gitignore` as policy for the synthetic archive.
|
|
fn isolate_doctor_fixture_from_parent_repo(test_home: &Path) -> io::Result<()> {
|
|
fs::create_dir(test_home.join(".git"))
|
|
}
|
|
|
|
fn seed_analytics_incidents_fixture(test_home: &Path) -> PathBuf {
|
|
let db_path = test_home.join("analytics-incidents.db");
|
|
let conn = Connection::open(db_path.to_string_lossy().into_owned())
|
|
.expect("open analytics incidents fixture db");
|
|
conn.execute_batch(
|
|
"CREATE TABLE agents (
|
|
id INTEGER PRIMARY KEY,
|
|
slug TEXT NOT NULL
|
|
);
|
|
CREATE TABLE conversations (
|
|
id INTEGER PRIMARY KEY,
|
|
agent_id INTEGER NOT NULL,
|
|
workspace_id INTEGER,
|
|
source_id TEXT,
|
|
external_id TEXT,
|
|
source_path TEXT NOT NULL,
|
|
started_at INTEGER,
|
|
ended_at INTEGER,
|
|
origin_host TEXT
|
|
);
|
|
CREATE TABLE messages (
|
|
id INTEGER PRIMARY KEY,
|
|
conversation_id INTEGER NOT NULL,
|
|
idx INTEGER NOT NULL,
|
|
content TEXT NOT NULL
|
|
);
|
|
INSERT INTO agents(id, slug) VALUES (1, 'codex'), (2, 'claude_code');",
|
|
)
|
|
.expect("create analytics incidents fixture schema");
|
|
|
|
let primary_path = test_home
|
|
.join("sessions/codex-primary.jsonl")
|
|
.to_string_lossy()
|
|
.into_owned();
|
|
let ordinary_path = test_home
|
|
.join("sessions/codex-ordinary.jsonl")
|
|
.to_string_lossy()
|
|
.into_owned();
|
|
conn.execute_compat(
|
|
"INSERT INTO conversations(
|
|
id, agent_id, workspace_id, source_id, external_id,
|
|
source_path, started_at, origin_host
|
|
) VALUES (1, 1, 10, 'local', 'primary-session', ?1, 200, NULL)",
|
|
params![primary_path.as_str()],
|
|
)
|
|
.expect("insert primary incident conversation");
|
|
conn.execute_compat(
|
|
"INSERT INTO conversations(
|
|
id, agent_id, workspace_id, source_id, external_id,
|
|
source_path, started_at, origin_host
|
|
) VALUES (
|
|
2, 2, 20, 'remote-fixture', 'remote-session',
|
|
'/remote/archive/remote-session.jsonl', 300, 'fixture-host'
|
|
)",
|
|
params![],
|
|
)
|
|
.expect("insert remote incident conversation");
|
|
conn.execute_compat(
|
|
"INSERT INTO conversations(
|
|
id, agent_id, workspace_id, source_id, external_id,
|
|
source_path, started_at, origin_host
|
|
) VALUES (3, 1, 10, 'local', 'ordinary-session', ?1, 100, NULL)",
|
|
params![ordinary_path.as_str()],
|
|
)
|
|
.expect("insert ordinary conversation");
|
|
conn.execute_batch(
|
|
"INSERT INTO messages(id, conversation_id, idx, content) VALUES
|
|
(1, 1, 0,
|
|
'cass health_class degraded recommended_action inspect semantic_fallback_lexical model missing database is locked busy'),
|
|
(2, 1, 1,
|
|
'cass index-ingest-out-of-memory quarantined_conversations=1'),
|
|
(3, 2, 0,
|
|
'cass source sync ssh permission denied auth timeout'),
|
|
(4, 3, 0,
|
|
'ordinary application note with no operational markers');",
|
|
)
|
|
.expect("insert analytics incidents fixture messages");
|
|
drop(conn);
|
|
db_path
|
|
}
|
|
|
|
fn write_quarantined_manifest(generation_dir: &std::path::Path) {
|
|
std::fs::create_dir_all(generation_dir).expect("create generation dir");
|
|
std::fs::write(
|
|
generation_dir.join("lexical-generation-manifest.json"),
|
|
serde_json::to_vec_pretty(&json!({
|
|
"manifest_version": 1,
|
|
"generation_id": "gen-quarantined",
|
|
"attempt_id": "attempt-1",
|
|
"created_at_ms": 1_733_000_000_000_i64,
|
|
"updated_at_ms": 1_733_000_000_321_i64,
|
|
"source_db_fingerprint": "fp-test",
|
|
"conversation_count": 3,
|
|
"message_count": 9,
|
|
"indexed_doc_count": 9,
|
|
"equivalence_manifest_fingerprint": null,
|
|
"shard_plan": null,
|
|
"build_budget": null,
|
|
"shards": [{
|
|
"shard_id": "shard-a",
|
|
"shard_ordinal": 0,
|
|
"state": "quarantined",
|
|
"updated_at_ms": 1_733_000_000_222_i64,
|
|
"indexed_doc_count": 9,
|
|
"message_count": 9,
|
|
"artifact_bytes": 512,
|
|
"stable_hash": "stable-hash-a",
|
|
"reclaimable": false,
|
|
"pinned": false,
|
|
"recovery_reason": null,
|
|
"quarantine_reason": "validation_failed"
|
|
}],
|
|
"merge_debt": {
|
|
"state": "none",
|
|
"updated_at_ms": null,
|
|
"pending_shard_count": 0,
|
|
"pending_artifact_bytes": 0,
|
|
"reason": null,
|
|
"controller_reason": null
|
|
},
|
|
"build_state": "failed",
|
|
"publish_state": "quarantined",
|
|
"failure_history": []
|
|
}))
|
|
.expect("serialize manifest"),
|
|
)
|
|
.expect("write manifest");
|
|
}
|
|
|
|
fn seed_diag_quarantine_fixture(test_home: &std::path::Path) -> PathBuf {
|
|
let data_dir = test_home.join("cass-data");
|
|
let backups_dir = data_dir.join("backups");
|
|
std::fs::create_dir_all(&backups_dir).expect("create backups dir");
|
|
|
|
let failed_seed_root =
|
|
backups_dir.join("agent_search.db.20260423T120000.12345.deadbeef.failed-baseline-seed.bak");
|
|
std::fs::write(&failed_seed_root, b"seed-backup").expect("write failed seed bundle");
|
|
std::fs::write(
|
|
failed_seed_root.with_file_name(format!(
|
|
"{}-wal",
|
|
failed_seed_root
|
|
.file_name()
|
|
.and_then(|name| name.to_str())
|
|
.expect("file name")
|
|
)),
|
|
b"seed-wal",
|
|
)
|
|
.expect("write failed seed wal");
|
|
|
|
let index_path = expected_index_dir(&data_dir);
|
|
std::fs::create_dir_all(&index_path).expect("create expected index dir");
|
|
let retained_publish_dir = index_path
|
|
.parent()
|
|
.expect("index parent")
|
|
.join(".lexical-publish-backups");
|
|
std::fs::create_dir_all(&retained_publish_dir).expect("create retained publish dir");
|
|
|
|
let older_backup = retained_publish_dir.join("prior-live-older");
|
|
std::fs::create_dir_all(&older_backup).expect("create older retained backup");
|
|
std::fs::write(older_backup.join("segment-a"), b"retained-live-segment-old")
|
|
.expect("write older retained publish backup");
|
|
std::thread::sleep(Duration::from_millis(20));
|
|
let newer_backup = retained_publish_dir.join("prior-live-newer");
|
|
std::fs::create_dir_all(&newer_backup).expect("create newer retained backup");
|
|
std::fs::write(newer_backup.join("segment-b"), b"retained-live-segment-new")
|
|
.expect("write newer retained publish backup");
|
|
|
|
let generation_dir = index_path
|
|
.parent()
|
|
.expect("index parent")
|
|
.join("generation-quarantined");
|
|
write_quarantined_manifest(&generation_dir);
|
|
std::fs::write(
|
|
generation_dir.join("segment-a"),
|
|
b"quarantined-generation-bytes",
|
|
)
|
|
.expect("write quarantined generation artifact");
|
|
|
|
data_dir
|
|
}
|
|
|
|
fn safe_fixture_destination(dst_root: &Path, rel: &Path) -> io::Result<PathBuf> {
|
|
let mut dst = dst_root.to_path_buf();
|
|
for component in rel.components() {
|
|
match component {
|
|
Component::CurDir => {}
|
|
Component::Normal(part) => dst.push(part),
|
|
_ => {
|
|
return Err(io::Error::new(
|
|
io::ErrorKind::InvalidInput,
|
|
"fixture path escaped source root",
|
|
));
|
|
}
|
|
}
|
|
}
|
|
Ok(dst)
|
|
}
|
|
|
|
fn isolated_search_demo_data(test_home: &Path) -> Result<PathBuf, Box<dyn Error>> {
|
|
let src = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
|
|
.join("tests")
|
|
.join("fixtures")
|
|
.join("search_demo_data");
|
|
let dst_root = test_home.join("search_demo_data");
|
|
for entry in WalkDir::new(&src) {
|
|
let entry = entry?;
|
|
// Machine-local frankensqlite namespace lock sidecars (created by
|
|
// any local run that opens the fixture DB) must not reach the
|
|
// clone: their foreign lock state fails the copied DB's open.
|
|
if entry
|
|
.path()
|
|
.file_name()
|
|
.and_then(|name| name.to_str())
|
|
.is_some_and(|name| {
|
|
name.ends_with("-fsqlite-ns-gate") || name.ends_with("-fsqlite-ns-use")
|
|
})
|
|
{
|
|
continue;
|
|
}
|
|
let rel = entry.path().strip_prefix(&src)?;
|
|
let dst = safe_fixture_destination(&dst_root, rel)?;
|
|
if entry.file_type().is_dir() {
|
|
fs::create_dir_all(&dst)?;
|
|
} else {
|
|
if let Some(parent) = dst.parent() {
|
|
fs::create_dir_all(parent)?;
|
|
}
|
|
fs::copy(entry.path(), &dst)?;
|
|
}
|
|
}
|
|
Ok(dst_root)
|
|
}
|
|
|
|
fn json_value_schema(value: &Value) -> Value {
|
|
match value {
|
|
Value::Null => json!({ "type": "null" }),
|
|
Value::Bool(_) => json!({ "type": "boolean" }),
|
|
Value::Number(number) => {
|
|
if number.is_f64() {
|
|
json!({ "type": "number" })
|
|
} else {
|
|
json!({ "type": "integer" })
|
|
}
|
|
}
|
|
Value::String(_) => json!({ "type": "string" }),
|
|
Value::Array(values) => {
|
|
let items = values
|
|
.first()
|
|
.map(json_value_schema)
|
|
.unwrap_or_else(|| json!({ "type": "unknown" }));
|
|
json!({
|
|
"type": "array",
|
|
"items": items
|
|
})
|
|
}
|
|
Value::Object(map) => {
|
|
let properties = map
|
|
.iter()
|
|
.map(|(key, value)| (key.clone(), json_value_schema(value)))
|
|
.collect::<serde_json::Map<String, Value>>();
|
|
json!({
|
|
"type": "object",
|
|
"properties": properties
|
|
})
|
|
}
|
|
}
|
|
}
|
|
|
|
fn sort_example_paths(value: &mut Value) {
|
|
match value {
|
|
Value::Object(map) => {
|
|
if let Some(Value::Array(paths)) = map.get_mut("example_paths") {
|
|
paths.sort_by(|left, right| left.as_str().cmp(&right.as_str()));
|
|
}
|
|
for child in map.values_mut() {
|
|
sort_example_paths(child);
|
|
}
|
|
}
|
|
Value::Array(values) => {
|
|
for child in values {
|
|
sort_example_paths(child);
|
|
}
|
|
}
|
|
_ => {}
|
|
}
|
|
}
|
|
|
|
fn is_json_schema_type_value(value: &Value) -> bool {
|
|
fn is_schema_type_name(value: &str) -> bool {
|
|
matches!(
|
|
value,
|
|
"array" | "boolean" | "integer" | "null" | "number" | "object" | "string"
|
|
)
|
|
}
|
|
|
|
match value {
|
|
Value::String(value) => is_schema_type_name(value),
|
|
Value::Array(values) => {
|
|
!values.is_empty()
|
|
&& values
|
|
.iter()
|
|
.all(|value| value.as_str().is_some_and(is_schema_type_name))
|
|
}
|
|
_ => false,
|
|
}
|
|
}
|
|
|
|
fn looks_like_json_schema_object(value: &Value) -> bool {
|
|
let Value::Object(map) = value else {
|
|
return false;
|
|
};
|
|
map.get("type").is_some_and(is_json_schema_type_value)
|
|
&& map.keys().all(|key| {
|
|
matches!(
|
|
key.as_str(),
|
|
"$defs"
|
|
| "$id"
|
|
| "$ref"
|
|
| "$schema"
|
|
| "additionalProperties"
|
|
| "allOf"
|
|
| "anyOf"
|
|
| "const"
|
|
| "contains"
|
|
| "default"
|
|
| "dependentRequired"
|
|
| "dependentSchemas"
|
|
| "definitions"
|
|
| "description"
|
|
| "else"
|
|
| "enum"
|
|
| "examples"
|
|
| "exclusiveMaximum"
|
|
| "exclusiveMinimum"
|
|
| "format"
|
|
| "if"
|
|
| "items"
|
|
| "maxItems"
|
|
| "maxLength"
|
|
| "maximum"
|
|
| "minItems"
|
|
| "minLength"
|
|
| "minimum"
|
|
| "multipleOf"
|
|
| "not"
|
|
| "oneOf"
|
|
| "pattern"
|
|
| "patternProperties"
|
|
| "properties"
|
|
| "required"
|
|
| "then"
|
|
| "title"
|
|
| "uniqueItems"
|
|
| "type"
|
|
)
|
|
})
|
|
}
|
|
|
|
fn normalize_live_robot_values(value: &mut Value) {
|
|
match value {
|
|
Value::Object(map) => {
|
|
let is_topology_budget = map.contains_key("topology")
|
|
&& map.contains_key("reserved_core_policy")
|
|
&& map.contains_key("advisory_budgets")
|
|
&& map.contains_key("fallback_active")
|
|
&& map.contains_key("decision_reason")
|
|
&& map.contains_key("proof_notes");
|
|
if is_topology_budget {
|
|
if let Some(topology) = map.get_mut("topology").and_then(Value::as_object_mut) {
|
|
topology.insert("source".to_string(), json!("fallback"));
|
|
topology.insert("memory_total_bytes".to_string(), Value::Null);
|
|
topology.insert("memory_available_bytes".to_string(), Value::Null);
|
|
}
|
|
if let Some(policy) = map
|
|
.get_mut("reserved_core_policy")
|
|
.and_then(Value::as_object_mut)
|
|
{
|
|
policy.insert("policy".to_string(), json!("current conservative default"));
|
|
policy.insert(
|
|
"reason".to_string(),
|
|
json!("topology could not be derived, so cass preserves existing worker and RAM defaults"),
|
|
);
|
|
}
|
|
map.insert("fallback_active".to_string(), json!(true));
|
|
map.insert(
|
|
"decision_reason".to_string(),
|
|
json!("using conservative defaults: linux sysfs topology is unavailable on this platform"),
|
|
);
|
|
map.insert(
|
|
"proof_notes".to_string(),
|
|
json!([
|
|
"fallback is intentionally isomorphic to current defaults for live rebuild budgets",
|
|
"no /sys-derived CPU locality assumptions are made in fallback mode"
|
|
]),
|
|
);
|
|
}
|
|
let redact_result_content = map.contains_key("source_path")
|
|
&& map.contains_key("line_number")
|
|
&& map.contains_key("agent");
|
|
for (key, child) in map.iter_mut() {
|
|
if key == "response_schemas" || looks_like_json_schema_object(child) {
|
|
continue;
|
|
}
|
|
|
|
if redact_result_content && key == "content" && child.is_string() {
|
|
*child = json!("[RESULT_CONTENT]");
|
|
continue;
|
|
}
|
|
if redact_result_content && key == "snippet" && child.is_string() {
|
|
*child = json!("[RESULT_SNIPPET]");
|
|
continue;
|
|
}
|
|
|
|
match key.as_str() {
|
|
"os" | "arch" if child.is_string() => {
|
|
*child = if key == "os" {
|
|
json!("[OS]")
|
|
} else {
|
|
json!("[ARCH]")
|
|
};
|
|
continue;
|
|
}
|
|
"last_snapshot" | "last_reason" => {
|
|
*child = json!("[LIVE_SAMPLE]");
|
|
continue;
|
|
}
|
|
"current_capacity_pct" => {
|
|
*child = json!(100);
|
|
continue;
|
|
}
|
|
"shrink_count" | "grow_count" => {
|
|
*child = json!(0);
|
|
continue;
|
|
}
|
|
"recent_decisions" => {
|
|
*child = json!([]);
|
|
continue;
|
|
}
|
|
"topology_class" => {
|
|
*child = json!("many_core_single_socket");
|
|
continue;
|
|
}
|
|
"logical_cpus" => {
|
|
*child = json!(128);
|
|
continue;
|
|
}
|
|
"physical_cores" => {
|
|
*child = json!(64);
|
|
continue;
|
|
}
|
|
"sockets" | "numa_nodes" => {
|
|
*child = json!(1);
|
|
continue;
|
|
}
|
|
"llc_groups" => {
|
|
*child = json!(8);
|
|
continue;
|
|
}
|
|
"smt_threads_per_core" => {
|
|
*child = json!(2);
|
|
continue;
|
|
}
|
|
"semantic_batchers" => {
|
|
*child = json!(8);
|
|
continue;
|
|
}
|
|
"steady_batch_fetch_conversations" => {
|
|
*child = json!(1024);
|
|
continue;
|
|
}
|
|
"startup_batch_fetch_conversations" => {
|
|
*child = json!(32);
|
|
continue;
|
|
}
|
|
"controller_loadavg_high_watermark_1m" => {
|
|
if child.is_string() {
|
|
*child = json!("121.0");
|
|
} else {
|
|
*child = json!(121.0);
|
|
}
|
|
continue;
|
|
}
|
|
"controller_loadavg_low_watermark_1m" => {
|
|
if child.is_string() {
|
|
*child = json!("120.0");
|
|
} else {
|
|
*child = json!(120.0);
|
|
}
|
|
continue;
|
|
}
|
|
_ => {}
|
|
}
|
|
|
|
if let Some(text) = child.as_str() {
|
|
if text.starts_with("planned from ") {
|
|
*child = json!(
|
|
"planned from ManyCoreSingleSocket: 128 logical CPUs, 64 physical cores, 1 socket(s), 1 NUMA node(s), 8 LLC group(s)"
|
|
);
|
|
continue;
|
|
}
|
|
if text.starts_with("reserve ") && text.contains(" logical CPUs ") {
|
|
*child = json!(
|
|
"reserve 16 of 128 logical CPUs for interactive work, IO, and NUMA/LLC service headroom"
|
|
);
|
|
continue;
|
|
}
|
|
}
|
|
|
|
normalize_live_robot_values(child);
|
|
}
|
|
}
|
|
Value::Array(values) => {
|
|
for child in values {
|
|
normalize_live_robot_values(child);
|
|
}
|
|
}
|
|
_ => {}
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn live_value_scrubbing_preserves_response_schema_properties() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let input = serde_json::to_string_pretty(&json!({
|
|
"response_schemas": {
|
|
"health": {
|
|
"type": "object",
|
|
"properties": {
|
|
"current_capacity_pct": {
|
|
"type": "integer",
|
|
"description": "Published capacity scalar."
|
|
},
|
|
"semantic_batchers": {
|
|
"type": "integer"
|
|
},
|
|
"last_snapshot": {
|
|
"type": ["object", "null"],
|
|
"properties": {
|
|
"load_per_core": {
|
|
"type": "number"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"schema_fragment": {
|
|
"title": "Runtime budget schema",
|
|
"type": "object",
|
|
"properties": {
|
|
"logical_cpus": {
|
|
"type": "integer"
|
|
}
|
|
}
|
|
},
|
|
"state": {
|
|
"resource_policy": {
|
|
"topology": {
|
|
"topology_class": "single_socket",
|
|
"logical_cpus": 999
|
|
},
|
|
"advisory_budgets": {
|
|
"semantic_batchers": 99,
|
|
"steady_batch_fetch_conversations": 768,
|
|
"startup_batch_fetch_conversations": 16
|
|
},
|
|
"watchdog": {
|
|
"last_snapshot": {
|
|
"load_per_core": 7.5
|
|
},
|
|
"last_reason": null
|
|
}
|
|
}
|
|
}
|
|
}))
|
|
.expect("serialize fixture");
|
|
|
|
let scrubbed = scrub_robot_json(&input, test_home.path());
|
|
let scrubbed: Value = serde_json::from_str(&scrubbed).expect("parse scrubbed fixture");
|
|
|
|
assert_eq!(
|
|
scrubbed["response_schemas"]["health"]["properties"]["current_capacity_pct"]["type"],
|
|
"integer"
|
|
);
|
|
assert_eq!(
|
|
scrubbed["response_schemas"]["health"]["properties"]["semantic_batchers"]["type"],
|
|
"integer"
|
|
);
|
|
assert_eq!(
|
|
scrubbed["response_schemas"]["health"]["properties"]["last_snapshot"]["properties"]["load_per_core"]
|
|
["type"],
|
|
"number"
|
|
);
|
|
assert_eq!(
|
|
scrubbed["schema_fragment"]["properties"]["logical_cpus"]["type"],
|
|
"integer"
|
|
);
|
|
assert_eq!(
|
|
scrubbed["state"]["resource_policy"]["topology"]["topology_class"],
|
|
"many_core_single_socket"
|
|
);
|
|
assert_eq!(
|
|
scrubbed["state"]["resource_policy"]["topology"]["logical_cpus"],
|
|
128
|
|
);
|
|
assert_eq!(
|
|
scrubbed["state"]["resource_policy"]["advisory_budgets"]["semantic_batchers"],
|
|
8
|
|
);
|
|
assert_eq!(
|
|
scrubbed["state"]["resource_policy"]["advisory_budgets"]["steady_batch_fetch_conversations"],
|
|
1024
|
|
);
|
|
assert_eq!(
|
|
scrubbed["state"]["resource_policy"]["advisory_budgets"]["startup_batch_fetch_conversations"],
|
|
32
|
|
);
|
|
assert_eq!(
|
|
scrubbed["state"]["resource_policy"]["watchdog"]["last_snapshot"],
|
|
"[LIVE_SAMPLE]"
|
|
);
|
|
assert_eq!(
|
|
scrubbed["state"]["resource_policy"]["watchdog"]["last_reason"],
|
|
"[LIVE_SAMPLE]"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn live_value_scrubbing_normalizes_host_topology_without_erasing_its_shape() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let input = serde_json::to_string_pretty(&json!({
|
|
"topology_budget": {
|
|
"topology": {
|
|
"source": "linux_sysfs",
|
|
"topology_class": "dual_socket",
|
|
"logical_cpus": 96,
|
|
"physical_cores": 48,
|
|
"sockets": 2,
|
|
"numa_nodes": 2,
|
|
"llc_groups": 4,
|
|
"smt_threads_per_core": 2,
|
|
"memory_total_bytes": 123,
|
|
"memory_available_bytes": 45
|
|
},
|
|
"reserved_core_policy": {
|
|
"reserved_cores": 8,
|
|
"policy": "host-specific policy",
|
|
"reason": "host-specific reason"
|
|
},
|
|
"advisory_budgets": {},
|
|
"fallback_active": false,
|
|
"decision_reason": "planned from DualSocket",
|
|
"proof_notes": ["host-specific proof"]
|
|
},
|
|
"near_miss_without_proof_notes": {
|
|
"topology": {
|
|
"source": "linux_sysfs",
|
|
"memory_total_bytes": 999,
|
|
"memory_available_bytes": 888
|
|
},
|
|
"reserved_core_policy": {},
|
|
"advisory_budgets": {},
|
|
"fallback_active": false,
|
|
"decision_reason": "must remain live"
|
|
}
|
|
}))
|
|
.expect("serialize fixture");
|
|
|
|
let scrubbed = scrub_robot_json(&input, test_home.path());
|
|
let scrubbed: Value = serde_json::from_str(&scrubbed).expect("parse scrubbed fixture");
|
|
let topology_budget = &scrubbed["topology_budget"];
|
|
|
|
assert_eq!(topology_budget["topology"]["source"], "fallback");
|
|
assert!(topology_budget["topology"]["memory_total_bytes"].is_null());
|
|
assert!(topology_budget["topology"]["memory_available_bytes"].is_null());
|
|
assert_eq!(topology_budget["fallback_active"], true);
|
|
assert!(topology_budget["reserved_core_policy"].is_object());
|
|
assert!(topology_budget["advisory_budgets"].is_object());
|
|
assert!(topology_budget["proof_notes"].is_array());
|
|
|
|
let near_miss = &scrubbed["near_miss_without_proof_notes"];
|
|
assert_eq!(near_miss["topology"]["source"], "linux_sysfs");
|
|
assert_eq!(near_miss["topology"]["memory_total_bytes"], "[LIVE_BYTES]");
|
|
assert_eq!(
|
|
near_miss["topology"]["memory_available_bytes"],
|
|
"[LIVE_BYTES]"
|
|
);
|
|
assert_eq!(near_miss["fallback_active"], false);
|
|
assert_eq!(near_miss["decision_reason"], "must remain live");
|
|
}
|
|
|
|
#[test]
|
|
fn live_value_scrubbing_normalizes_runtime_objects_with_type_fields() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let input = serde_json::to_string_pretty(&json!({
|
|
"response_schemas": {
|
|
"health": {
|
|
"type": "object",
|
|
"properties": {
|
|
"logical_cpus": {
|
|
"type": "integer"
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"runtime": {
|
|
"type": "resource_policy",
|
|
"description": "runtime status, not JSON Schema",
|
|
"default": "host sampled",
|
|
"logical_cpus": 999,
|
|
"advisory_budgets": {
|
|
"semantic_batchers": 99
|
|
}
|
|
}
|
|
}))
|
|
.expect("serialize fixture");
|
|
|
|
let scrubbed = scrub_robot_json(&input, test_home.path());
|
|
let scrubbed: Value = serde_json::from_str(&scrubbed).expect("parse scrubbed fixture");
|
|
|
|
assert_eq!(
|
|
scrubbed["response_schemas"]["health"]["properties"]["logical_cpus"]["type"],
|
|
"integer"
|
|
);
|
|
assert_eq!(scrubbed["runtime"]["type"], "resource_policy");
|
|
assert_eq!(
|
|
scrubbed["runtime"]["description"],
|
|
"runtime status, not JSON Schema"
|
|
);
|
|
assert_eq!(scrubbed["runtime"]["default"], "host sampled");
|
|
assert_eq!(scrubbed["runtime"]["logical_cpus"], 128);
|
|
assert_eq!(
|
|
scrubbed["runtime"]["advisory_budgets"]["semantic_batchers"],
|
|
8
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn live_value_scrubbing_redacts_repo_paths_and_result_content() -> Result<(), String> {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let repo_root = env!("CARGO_MANIFEST_DIR");
|
|
let legacy_fixture_root = "/data/projects/coding_agent_session_search";
|
|
let input = serde_json::to_string_pretty(&json!({
|
|
"results": [
|
|
{
|
|
"source_path": format!("{repo_root}/.aider.chat.history.md"),
|
|
"workspace": repo_root,
|
|
"line_number": 42,
|
|
"agent": "aider",
|
|
"snippet": "private snippet",
|
|
"content": "private prompt and assistant transcript"
|
|
},
|
|
{
|
|
"source_path": format!("{legacy_fixture_root}/tests/fixtures/aider/session.md"),
|
|
"workspace": legacy_fixture_root,
|
|
"line_number": 7,
|
|
"agent": "aider",
|
|
"snippet": "legacy private snippet",
|
|
"content": "legacy private prompt and assistant transcript"
|
|
}
|
|
]
|
|
}))
|
|
.expect("serialize fixture");
|
|
|
|
let scrubbed = scrub_robot_json(&input, test_home.path());
|
|
let scrubbed: Value = serde_json::from_str(&scrubbed).expect("parse scrubbed fixture");
|
|
|
|
require_json_string_eq(
|
|
&scrubbed,
|
|
"/results/0/source_path",
|
|
"[REPO]/.aider.chat.history.md",
|
|
)?;
|
|
require_json_string_eq(&scrubbed, "/results/0/workspace", "[REPO]")?;
|
|
require_json_string_eq(&scrubbed, "/results/0/snippet", "[RESULT_SNIPPET]")?;
|
|
require_json_string_eq(&scrubbed, "/results/0/content", "[RESULT_CONTENT]")?;
|
|
require_json_string_eq(
|
|
&scrubbed,
|
|
"/results/1/source_path",
|
|
"[REPO]/tests/fixtures/aider/session.md",
|
|
)?;
|
|
require_json_string_eq(&scrubbed, "/results/1/workspace", "[REPO]")?;
|
|
require_json_string_eq(&scrubbed, "/results/1/snippet", "[RESULT_SNIPPET]")?;
|
|
require_json_string_eq(&scrubbed, "/results/1/content", "[RESULT_CONTENT]")?;
|
|
let serialized = serde_json::to_string(&scrubbed).expect("serialize scrubbed fixture");
|
|
if serialized.contains("private prompt") {
|
|
return Err("scrubbed fixture leaked private prompt text".to_string());
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
fn require_json_string_eq(value: &Value, pointer: &str, expected: &str) -> Result<(), String> {
|
|
let actual = value
|
|
.pointer(pointer)
|
|
.and_then(Value::as_str)
|
|
.ok_or_else(|| format!("missing string at JSON pointer {pointer}"))?;
|
|
match actual.cmp(expected) {
|
|
std::cmp::Ordering::Equal => Ok(()),
|
|
_ => Err(format!(
|
|
"string at JSON pointer {pointer} was {actual:?}, expected {expected:?}"
|
|
)),
|
|
}
|
|
}
|
|
|
|
/// Strip non-deterministic values from a robot-mode JSON payload so the
|
|
/// golden captures *shape* rather than ephemeral facts.
|
|
///
|
|
/// - `crate_version` / top-level `version` → `"[VERSION]"` so the test survives cargo version bumps
|
|
/// - ISO timestamps → `"[TIMESTAMP]"`
|
|
/// - Absolute paths under the test `HOME` → `"[PATH]"`
|
|
/// - UUID-ish tokens → `"[UUID]"`
|
|
fn scrub_robot_json(input: &str, test_home: &std::path::Path) -> String {
|
|
let mut out = input.to_string();
|
|
|
|
// 1. `crate_version` field in capabilities output. Match the exact JSON
|
|
// key so we don't inadvertently touch version strings inside features.
|
|
let crate_version_re = regex::Regex::new(r#""crate_version"\s*:\s*"[^"]*""#).unwrap();
|
|
out = crate_version_re
|
|
.replace_all(&out, r#""crate_version": "[VERSION]""#)
|
|
.to_string();
|
|
let top_level_version_re = regex::Regex::new(r#"(?m)^ "version"\s*:\s*"[^"]*""#).unwrap();
|
|
out = top_level_version_re
|
|
.replace_all(&out, r#" "version": "[VERSION]""#)
|
|
.to_string();
|
|
|
|
// 1b. Build identity (GH #399): the embedded commit/date vary per build
|
|
// (and per checkout state). Keep the fields, scrub the values.
|
|
let build_commit_re = regex::Regex::new(r#""build_commit"\s*:\s*("[^"]*"|null)"#).unwrap();
|
|
out = build_commit_re
|
|
.replace_all(&out, r#""build_commit": "[BUILD_COMMIT]""#)
|
|
.to_string();
|
|
let build_commit_date_re =
|
|
regex::Regex::new(r#""build_commit_date"\s*:\s*("[^"]*"|null)"#).unwrap();
|
|
out = build_commit_date_re
|
|
.replace_all(&out, r#""build_commit_date": "[BUILD_COMMIT_DATE]""#)
|
|
.to_string();
|
|
|
|
// 2. ISO-8601 timestamps (match with optional fractional seconds / tz).
|
|
let ts_re =
|
|
regex::Regex::new(r#"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d+)?(Z|[+-]\d{2}:\d{2})?"#)
|
|
.unwrap();
|
|
out = ts_re.replace_all(&out, "[TIMESTAMP]").to_string();
|
|
|
|
// 3. Absolute paths rooted at the isolated test HOME. Anything else is
|
|
// either a constant relative path or a configured mount — both are
|
|
// shape-relevant and stay in the golden.
|
|
let home_str = test_home.display().to_string();
|
|
if !home_str.is_empty() {
|
|
// macOS may report the same temporary directory through its canonical
|
|
// `/private` alias even when `tempfile` returned the `/var/...` spelling.
|
|
// Replace the longer alias first so both spellings collapse to one
|
|
// portable fixture root instead of leaving `/private[TEST_HOME]`.
|
|
let private_home_str = format!("/private{home_str}");
|
|
out = out.replace(&private_home_str, "[TEST_HOME]");
|
|
out = out.replace(&home_str, "[TEST_HOME]");
|
|
}
|
|
|
|
// 3b. Fixture databases can intentionally contain the repository root as
|
|
// historical source metadata. Keep the field shape, but never freeze the
|
|
// local checkout path into a public contract golden.
|
|
let repo_root = env!("CARGO_MANIFEST_DIR");
|
|
if !repo_root.is_empty() {
|
|
out = out.replace(repo_root, "[REPO]");
|
|
}
|
|
// The checked-in search_demo_data fixture intentionally preserves source
|
|
// metadata captured on the maintainer machine. CI checkouts live under
|
|
// /home/runner, so scrub that stable fixture root explicitly too.
|
|
out = out.replace("/data/projects/coding_agent_session_search", "[REPO]");
|
|
|
|
// 4. UUIDs.
|
|
let uuid_re =
|
|
regex::Regex::new(r#"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"#)
|
|
.unwrap();
|
|
out = uuid_re.replace_all(&out, "[UUID]").to_string();
|
|
|
|
// 5. Wall-clock durations vary run to run and by host. Keep the fields in
|
|
// the golden to prove the shape, but scrub the values so drift on them
|
|
// does not fail the contract test.
|
|
for (key, replacement) in [
|
|
("latency_ms", "[LATENCY_MS]"),
|
|
("elapsed_ms", "[ELAPSED_MS]"),
|
|
("probe_duration_ms", "[ELAPSED_MS]"),
|
|
("slowest_elapsed_ms", "[ELAPSED_MS]"),
|
|
] {
|
|
let re = regex::Regex::new(&format!(r#""{key}"\s*:\s*\d+"#)).unwrap();
|
|
out = re
|
|
.replace_all(&out, format!(r#""{key}": "{replacement}""#).as_str())
|
|
.to_string();
|
|
}
|
|
let slowest_operation_re = regex::Regex::new(r#""slowest_operation"\s*:\s*"[^"]*""#).unwrap();
|
|
out = slowest_operation_re
|
|
.replace_all(&out, r#""slowest_operation": "[LIVE_OPERATION]""#)
|
|
.to_string();
|
|
|
|
// 5b. The semantic daemon socket path is host-local. `cass_cmd` pins it to
|
|
// the fixture, and scrubbing its value keeps the contract stable for any
|
|
// direct callers that do not use that helper.
|
|
let socket_path_re = regex::Regex::new(r#""socket_path"\s*:\s*"[^"]*""#).unwrap();
|
|
out = socket_path_re
|
|
.replace_all(&out, r#""socket_path": "[DAEMON_SOCKET]""#)
|
|
.to_string();
|
|
|
|
// 6. Live-sampled kernel metrics in health --json (load average per
|
|
// core and PSI CPU pressure). These float values change between runs
|
|
// based on whatever else is happening on the box. Scrub to placeholders
|
|
// so the golden locks the shape without chasing host noise.
|
|
for key in ["load_per_core", "psi_cpu_some_avg10"] {
|
|
let re = regex::Regex::new(&format!(
|
|
r#""{key}"\s*:\s*(-?\d+(\.\d+)?([eE][+-]?\d+)?|null)"#
|
|
))
|
|
.unwrap();
|
|
out = re
|
|
.replace_all(&out, format!(r#""{key}": "[LIVE_METRIC]""#).as_str())
|
|
.to_string();
|
|
}
|
|
|
|
// 7. Watchdog sampler counters in health --json. These tick each time
|
|
// the responsiveness sampler fires; the test can race with that timer
|
|
// (0 ticks before the first sample, 1+ ticks after). Scrub the integer
|
|
// to a placeholder so the golden locks the *shape* of the counter
|
|
// surface without chasing sampler-timing drift.
|
|
for key in [
|
|
"healthy_streak",
|
|
"ticks_total",
|
|
"load_window_len",
|
|
"psi_window_len",
|
|
"observations_total",
|
|
] {
|
|
let re = regex::Regex::new(&format!(r#""{key}"\s*:\s*\d+"#)).unwrap();
|
|
out = re
|
|
.replace_all(&out, format!(r#""{key}": "[LIVE_COUNTER]""#).as_str())
|
|
.to_string();
|
|
}
|
|
|
|
// 8. `last_snapshot` + `last_reason` in health --json vary depending on
|
|
// whether the responsiveness sampler has fired. They are normalized
|
|
// structurally in `normalize_live_robot_values` below. Keeping this out
|
|
// of the regex pass is essential: introspection embeds a nested JSON
|
|
// Schema named `last_snapshot`, and a brace-matching regex can truncate
|
|
// that schema into invalid JSON.
|
|
|
|
// Resource policy status reports include host-live CPU and memory budgets.
|
|
// The shape is contractual; the sampled worker/byte counts are not.
|
|
for key in [
|
|
"available_parallelism",
|
|
"reserved_cores",
|
|
"max_workers",
|
|
"effective_worker_ceiling",
|
|
"workers",
|
|
"tantivy_writer_threads",
|
|
"shard_builders",
|
|
"merge_workers",
|
|
"staged_shard_builders",
|
|
"staged_merge_workers",
|
|
"page_prep_workers",
|
|
] {
|
|
let re = regex::Regex::new(&format!(r#""{key}"\s*:\s*("?\d+"?)"#)).unwrap();
|
|
out = re
|
|
.replace_all(&out, format!(r#""{key}": "[LIVE_COUNTER]""#).as_str())
|
|
.to_string();
|
|
}
|
|
|
|
for key in [
|
|
"memory_available_bytes",
|
|
"memory_total_bytes",
|
|
"cache_cap_bytes",
|
|
"available_bytes",
|
|
"max_inflight_bytes",
|
|
"pipeline_max_message_bytes_in_flight",
|
|
// WS-B.4a: the archive footprint depends on the engine's page layout
|
|
// and on whether a WAL sidecar happens to exist; keep the keys, scrub
|
|
// the values.
|
|
"db_bytes",
|
|
"wal_bytes",
|
|
] {
|
|
let re = regex::Regex::new(&format!(r#""{key}"\s*:\s*("?\d+"?)"#)).unwrap();
|
|
out = re
|
|
.replace_all(&out, format!(r#""{key}": "[LIVE_BYTES]""#).as_str())
|
|
.to_string();
|
|
}
|
|
|
|
let age_seconds_re = regex::Regex::new(r#""age_seconds"\s*:\s*(\d+|null)"#).unwrap();
|
|
out = age_seconds_re
|
|
.replace_all(&out, r#""age_seconds": "[AGE_SECONDS]""#)
|
|
.to_string();
|
|
|
|
let last_read_re = regex::Regex::new(r#""last_read_at_ms"\s*:\s*(\d+|null)"#).unwrap();
|
|
out = last_read_re
|
|
.replace_all(&out, r#""last_read_at_ms": "[LAST_READ_MS]""#)
|
|
.to_string();
|
|
|
|
if let Ok(mut parsed) = serde_json::from_str::<Value>(&out) {
|
|
normalize_live_robot_values(&mut parsed);
|
|
sort_example_paths(&mut parsed);
|
|
if let Ok(canonical) = serde_json::to_string_pretty(&parsed) {
|
|
out = canonical;
|
|
}
|
|
}
|
|
|
|
out
|
|
}
|
|
|
|
/// Compare `actual` against the golden at `tests/golden/<name>`. Writes /
|
|
/// overwrites the golden when `UPDATE_GOLDENS=1` is set in the env.
|
|
fn describe_golden_difference(
|
|
line_number: usize,
|
|
expected_line: Option<&str>,
|
|
actual_line: Option<&str>,
|
|
expected_len: usize,
|
|
actual_len: usize,
|
|
) -> String {
|
|
match (expected_line, actual_line) {
|
|
(Some(expected_line), Some(actual_line)) => {
|
|
format!("line {line_number}: expected {expected_line:?}, actual {actual_line:?}")
|
|
}
|
|
(Some(expected_line), None) => {
|
|
format!("line {line_number}: expected {expected_line:?}, actual reached end of file")
|
|
}
|
|
(None, Some(actual_line)) => {
|
|
format!("line {line_number}: expected reached end of file, actual {actual_line:?}")
|
|
}
|
|
(None, None) => format!(
|
|
"line contents match, but byte lengths differ (expected {expected_len}, actual {actual_len})"
|
|
),
|
|
}
|
|
}
|
|
|
|
fn first_golden_difference(expected: &str, actual: &str) -> String {
|
|
let mut expected_lines = expected.lines();
|
|
let mut actual_lines = actual.lines();
|
|
|
|
for line_number in 1.. {
|
|
let expected_line = expected_lines.next();
|
|
let actual_line = actual_lines.next();
|
|
if expected_line.is_some() && expected_line == actual_line {
|
|
continue;
|
|
}
|
|
return describe_golden_difference(
|
|
line_number,
|
|
expected_line,
|
|
actual_line,
|
|
expected.len(),
|
|
actual.len(),
|
|
);
|
|
}
|
|
|
|
describe_golden_difference(0, None, None, expected.len(), actual.len())
|
|
}
|
|
|
|
fn strip_one_final_line_ending(input: &str) -> &str {
|
|
input
|
|
.strip_suffix("\r\n")
|
|
.or_else(|| input.strip_suffix('\n'))
|
|
.unwrap_or(input)
|
|
}
|
|
|
|
fn assert_golden(name: &str, actual: &str) {
|
|
let golden_path = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
|
|
.join("tests")
|
|
.join("golden")
|
|
.join(name);
|
|
|
|
serde_json::from_str::<Value>(actual)
|
|
.unwrap_or_else(|err| panic!("generated golden {name} is not valid JSON: {err}"));
|
|
|
|
if std::env::var("UPDATE_GOLDENS").is_ok() {
|
|
std::fs::create_dir_all(golden_path.parent().unwrap()).expect("create golden parent dir");
|
|
std::fs::write(&golden_path, actual).expect("write golden file");
|
|
eprintln!("[GOLDEN] Updated: {}", golden_path.display());
|
|
return;
|
|
}
|
|
|
|
let expected = std::fs::read_to_string(&golden_path).unwrap_or_else(|err| {
|
|
panic!(
|
|
"Golden file missing or unreadable: {}\n{err}\n\n\
|
|
Run with UPDATE_GOLDENS=1 to create it, then review and commit:\n\
|
|
\tUPDATE_GOLDENS=1 rch exec -- env CARGO_TARGET_DIR=/data/tmp/cass-golden-target cargo test --test golden_robot_json\n\
|
|
\tgit diff tests/golden/\n\
|
|
\tgit add tests/golden/",
|
|
golden_path.display(),
|
|
)
|
|
});
|
|
|
|
let comparable_expected = strip_one_final_line_ending(&expected);
|
|
let comparable_actual = strip_one_final_line_ending(actual);
|
|
if comparable_actual != comparable_expected {
|
|
let first_difference = first_golden_difference(comparable_expected, comparable_actual);
|
|
// Dump actual next to golden for easy diffing.
|
|
let actual_path = golden_path.with_extension("actual");
|
|
std::fs::write(&actual_path, actual).expect("write .actual file");
|
|
panic!(
|
|
"GOLDEN MISMATCH: {name}\n\n\
|
|
Expected: {}\n\
|
|
Actual: {} (first difference: {first_difference})\n\n\
|
|
diff the two files to see the drift, then either:\n\
|
|
\t- fix the code if this was unintentional, or\n\
|
|
\t- regenerate: UPDATE_GOLDENS=1 rch exec -- env CARGO_TARGET_DIR=/data/tmp/cass-golden-target cargo test --test golden_robot_json \\\n\
|
|
\t && git diff tests/golden/ && git add tests/golden/",
|
|
golden_path.display(),
|
|
actual_path.display(),
|
|
);
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn robot_json_goldens_do_not_embed_repo_paths_or_raw_session_content() {
|
|
if std::env::var("UPDATE_GOLDENS").is_ok() {
|
|
return;
|
|
}
|
|
|
|
let golden_root = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
|
|
.join("tests")
|
|
.join("golden")
|
|
.join("robot");
|
|
let repo_root = env!("CARGO_MANIFEST_DIR");
|
|
let forbidden = [
|
|
repo_root,
|
|
"/data/projects/coding_agent_session_search",
|
|
".venv/bin/aider --no-git --message",
|
|
"Using openrouter/deepseek",
|
|
"Aider v0.",
|
|
"https://aider.chat/HISTORY.html",
|
|
];
|
|
|
|
for entry in WalkDir::new(&golden_root) {
|
|
let entry = entry.expect("walk robot goldens");
|
|
if !entry.file_type().is_file()
|
|
|| entry.path().extension().and_then(|ext| ext.to_str()) != Some("golden")
|
|
{
|
|
continue;
|
|
}
|
|
let contents = fs::read_to_string(entry.path())
|
|
.unwrap_or_else(|err| panic!("read {}: {err}", entry.path().display()));
|
|
for needle in forbidden {
|
|
assert!(
|
|
needle.is_empty() || !contents.contains(needle),
|
|
"{} contains unredacted golden-only sensitive fixture text: {needle}",
|
|
entry.path().display()
|
|
);
|
|
}
|
|
}
|
|
}
|
|
|
|
fn should_scan_for_bare_golden_regeneration_recipe(rel_path: &str) -> bool {
|
|
if rel_path == "AGENTS.md" || rel_path == "README.md" {
|
|
return true;
|
|
}
|
|
if rel_path.starts_with("docs/artifacts/") || rel_path == "docs/planning/UPGRADE_LOG.md" {
|
|
return false;
|
|
}
|
|
if rel_path.starts_with("docs/") {
|
|
return rel_path.ends_with(".md");
|
|
}
|
|
if rel_path.starts_with("tests/golden/") {
|
|
return rel_path.ends_with(".golden")
|
|
|| rel_path.ends_with(".json")
|
|
|| rel_path.ends_with(".md");
|
|
}
|
|
rel_path.starts_with("tests/") && rel_path.ends_with(".rs")
|
|
}
|
|
|
|
#[test]
|
|
fn golden_regeneration_hints_do_not_use_bare_cargo() {
|
|
let repo_root = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
|
|
let forbidden = concat!("UPDATE_GOLDENS=1 ", "cargo test");
|
|
let mut violations = Vec::new();
|
|
|
|
for root in ["AGENTS.md", "README.md", "docs", "tests"] {
|
|
let root = repo_root.join(root);
|
|
for entry in WalkDir::new(&root) {
|
|
let entry = entry.expect("walk golden recipe policy files");
|
|
if !entry.file_type().is_file() {
|
|
continue;
|
|
}
|
|
|
|
let rel_path = entry
|
|
.path()
|
|
.strip_prefix(&repo_root)
|
|
.expect("entry under repo root")
|
|
.to_string_lossy()
|
|
.replace('\\', "/");
|
|
if !should_scan_for_bare_golden_regeneration_recipe(&rel_path) {
|
|
continue;
|
|
}
|
|
|
|
let contents = fs::read_to_string(entry.path())
|
|
.unwrap_or_else(|err| panic!("read {}: {err}", entry.path().display()));
|
|
if contents.contains(forbidden) {
|
|
violations.push(rel_path);
|
|
}
|
|
}
|
|
}
|
|
|
|
assert!(
|
|
violations.is_empty(),
|
|
"golden regeneration hints must use rch offload, not bare cargo:\n{}",
|
|
violations.join("\n")
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn agents_md_does_not_recommend_bare_cargo_commands() {
|
|
let agents_path = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("AGENTS.md");
|
|
let contents = fs::read_to_string(&agents_path)
|
|
.unwrap_or_else(|err| panic!("read {}: {err}", agents_path.display()));
|
|
let bare_cargo_line = regex::Regex::new(
|
|
r"^(UPDATE_GOLDENS=1\s+)?(env\s+CARGO_TARGET_DIR=[^\s]+\s+)?cargo\s+(build|test|check|clippy|fmt)\b",
|
|
)
|
|
.expect("compile bare cargo command regex");
|
|
|
|
let violations = contents
|
|
.lines()
|
|
.enumerate()
|
|
.filter_map(|(index, line)| {
|
|
let trimmed = line.trim();
|
|
bare_cargo_line
|
|
.is_match(trimmed)
|
|
.then(|| format!("{}:{}", index + 1, trimmed))
|
|
})
|
|
.collect::<Vec<_>>();
|
|
|
|
assert!(
|
|
violations.is_empty(),
|
|
"AGENTS.md must show rch-offloaded cargo command examples:\n{}",
|
|
violations.join("\n")
|
|
);
|
|
}
|
|
|
|
/// Capture stdout of `cass <args>` in the isolated test home and return
|
|
/// the scrubbed canonical-JSON form (keys-sorted by serde_json's default
|
|
/// `BTreeMap` insertion preservation, pretty-printed, dynamic values
|
|
/// scrubbed). Returns the parsed-then-reserialized string so the golden
|
|
/// survives whitespace drift.
|
|
///
|
|
/// `expect_status` selects the exit-code contract: `ExitOk` for commands
|
|
/// that must succeed (capabilities, models status), `ExitAny` for
|
|
/// commands that legitimately exit non-zero when reporting a problem
|
|
/// (health, which exits 1 when the DB / index is not initialised — that
|
|
/// non-zero status *is* part of the contract and we freeze its JSON).
|
|
fn capture_robot_json(
|
|
test_home: &std::path::Path,
|
|
args: &[&str],
|
|
expect_status: ExpectStatus,
|
|
) -> String {
|
|
let output = cass_cmd(test_home)
|
|
.args(args)
|
|
.output()
|
|
.unwrap_or_else(|err| panic!("run cass {args:?}: {err}"));
|
|
if matches!(expect_status, ExpectStatus::ExitOk) {
|
|
assert!(
|
|
output.status.success(),
|
|
"cass {args:?} exited non-zero: status={:?}\nstdout:\n{}\nstderr:\n{}",
|
|
output.status,
|
|
String::from_utf8_lossy(&output.stdout),
|
|
String::from_utf8_lossy(&output.stderr),
|
|
);
|
|
}
|
|
let stdout = String::from_utf8(output.stdout).expect("utf8 stdout");
|
|
let parsed: serde_json::Value = serde_json::from_str(&stdout)
|
|
.unwrap_or_else(|err| panic!("cass {args:?} stdout is not JSON: {err}\nstdout:\n{stdout}"));
|
|
let canonical = serde_json::to_string_pretty(&parsed).expect("pretty-print JSON");
|
|
scrub_robot_json(&canonical, test_home)
|
|
}
|
|
|
|
fn capture_robot_json_value(
|
|
test_home: &std::path::Path,
|
|
args: &[&str],
|
|
expect_status: ExpectStatus,
|
|
) -> Value {
|
|
let output = cass_cmd(test_home)
|
|
.args(args)
|
|
.output()
|
|
.unwrap_or_else(|err| panic!("run cass {args:?}: {err}"));
|
|
if matches!(expect_status, ExpectStatus::ExitOk) {
|
|
assert!(
|
|
output.status.success(),
|
|
"cass {args:?} exited non-zero: status={:?}\nstdout:\n{}\nstderr:\n{}",
|
|
output.status,
|
|
String::from_utf8_lossy(&output.stdout),
|
|
String::from_utf8_lossy(&output.stderr),
|
|
);
|
|
}
|
|
serde_json::from_slice(&output.stdout)
|
|
.unwrap_or_else(|err| panic!("cass {args:?} stdout is not JSON: {err}"))
|
|
}
|
|
|
|
#[derive(Clone, Copy)]
|
|
enum ExpectStatus {
|
|
ExitOk,
|
|
ExitAny,
|
|
}
|
|
|
|
#[test]
|
|
fn capabilities_json_matches_golden() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let scrubbed = capture_robot_json(
|
|
test_home.path(),
|
|
&["capabilities", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
assert_golden("robot/capabilities.json.golden", &scrubbed);
|
|
}
|
|
|
|
#[test]
|
|
fn capabilities_shape_matches_golden() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let capabilities = capture_robot_json_value(
|
|
test_home.path(),
|
|
&["capabilities", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&capabilities)).expect("pretty-print JSON");
|
|
assert_golden("robot/capabilities_shape.json.golden", &canonical);
|
|
}
|
|
|
|
#[test]
|
|
fn models_status_json_matches_golden() {
|
|
// `cass models status --json` reads XDG_DATA_HOME for the model cache
|
|
// directory. In our isolated test home the cache is always empty, so
|
|
// the output is deterministic: state=not_installed across every field.
|
|
// Absolute paths inside the payload (`model_dir`, `files[].actual_path`)
|
|
// get scrubbed by `scrub_robot_json` → `[TEST_HOME]` prefix.
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let scrubbed = capture_robot_json(
|
|
test_home.path(),
|
|
&["models", "status", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
assert_golden("robot/models_status.json.golden", &scrubbed);
|
|
}
|
|
|
|
#[test]
|
|
fn models_status_shape_matches_golden() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let status = capture_robot_json_value(
|
|
test_home.path(),
|
|
&["models", "status", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&status)).expect("pretty-print JSON");
|
|
assert_golden("robot/models_status_shape.json.golden", &canonical);
|
|
}
|
|
|
|
#[test]
|
|
fn fleet_upgrade_rehearsal_shape_matches_golden() {
|
|
// `cass fleet upgrade-rehearsal --json` (bead sc8sp) against an isolated
|
|
// empty data dir + config dir covers only the local host. The exact
|
|
// version/platform/disposition values depend on the build host and target,
|
|
// so we pin the JSON *shape* (field names + types) — deterministic across
|
|
// hosts and the canonical contract lock for this surface. Concrete values
|
|
// (disposition labels, safe commands, mode, verification) are asserted by
|
|
// tests/e2e_fleet_upgrade_rehearsal_gate.rs against the real binary.
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let rehearsal = capture_robot_json_value(
|
|
test_home.path(),
|
|
&["fleet", "upgrade-rehearsal", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&rehearsal)).expect("pretty-print JSON");
|
|
assert_golden(
|
|
"robot/fleet_upgrade_rehearsal_shape.json.golden",
|
|
&canonical,
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn health_json_matches_golden() {
|
|
// `cass health --json` reports readiness for an isolated empty HOME:
|
|
// status=not_initialized, healthy=false, db.exists=false,
|
|
// state.index.status=missing, state.semantic.availability=...
|
|
// All paths scrub to [TEST_HOME], latency_ms scrubs to [LATENCY_MS].
|
|
// The golden freezes the full readiness contract (ibuuh.9 scope):
|
|
// top-level status/healthy/initialized/errors/recommended_action
|
|
// plus the per-subsystem state.* nested blocks.
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
// `cass health` exits 1 when reporting an unhealthy / uninitialised
|
|
// state — that non-zero exit is part of the contract and the golden
|
|
// below freezes the JSON body that accompanies it. ExitAny lets the
|
|
// capture proceed regardless of status.
|
|
let scrubbed = capture_robot_json(
|
|
test_home.path(),
|
|
&["health", "--json"],
|
|
ExpectStatus::ExitAny,
|
|
);
|
|
assert_golden("robot/health.json.golden", &scrubbed);
|
|
}
|
|
|
|
#[test]
|
|
fn health_shape_matches_golden() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let health = capture_robot_json(
|
|
test_home.path(),
|
|
&["health", "--json"],
|
|
ExpectStatus::ExitAny,
|
|
);
|
|
let health: Value = serde_json::from_str(&health).expect("parse scrubbed health JSON");
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&health)).expect("pretty-print JSON");
|
|
assert_golden("robot/health_shape.json.golden", &canonical);
|
|
}
|
|
|
|
#[test]
|
|
fn selftest_json_matches_golden() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let scrubbed = capture_robot_json(
|
|
test_home.path(),
|
|
&["selftest", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
assert_golden("robot/selftest.json.golden", &scrubbed);
|
|
}
|
|
|
|
#[test]
|
|
fn selftest_shape_matches_golden() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let selftest = capture_robot_json_value(
|
|
test_home.path(),
|
|
&["selftest", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&selftest)).expect("pretty-print JSON");
|
|
assert_golden("robot/selftest_shape.json.golden", &canonical);
|
|
}
|
|
|
|
#[test]
|
|
fn onboarding_json_matches_golden() {
|
|
// `cass onboarding --json` on an isolated empty HOME: no providers detected,
|
|
// no semantic model, no archive DB → recommended_action=discover_sources,
|
|
// mutation_free=true. Read-only; deterministic; paths scrub to [TEST_HOME].
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let scrubbed = capture_robot_json(
|
|
test_home.path(),
|
|
&["onboarding", "--json"],
|
|
ExpectStatus::ExitAny,
|
|
);
|
|
assert_golden("robot/onboarding.json.golden", &scrubbed);
|
|
}
|
|
|
|
#[test]
|
|
fn onboarding_shape_matches_golden() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let onboarding = capture_robot_json(
|
|
test_home.path(),
|
|
&["onboarding", "--json"],
|
|
ExpectStatus::ExitAny,
|
|
);
|
|
let onboarding: Value =
|
|
serde_json::from_str(&onboarding).expect("parse scrubbed onboarding JSON");
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&onboarding)).expect("pretty-print JSON");
|
|
assert_golden("robot/onboarding_shape.json.golden", &canonical);
|
|
}
|
|
|
|
#[test]
|
|
fn diag_json_matches_golden() {
|
|
// `cass diag --json` is the artifact-inventory surface that
|
|
// ibuuh.36's verification matrix wants frozen alongside manifest
|
|
// snapshots and golden-query digests: version, platform, paths,
|
|
// database counts, index presence, and per-connector detection. Under
|
|
// an isolated empty HOME every field is deterministic (no connectors
|
|
// detected, database/index absent, paths scrub to [TEST_HOME]).
|
|
// Freezing this makes drift on any connector-detection or path-layout
|
|
// field fail in CI instead of silently misreporting to operators.
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let scrubbed = capture_robot_json(test_home.path(), &["diag", "--json"], ExpectStatus::ExitOk);
|
|
assert_golden("robot/diag.json.golden", &scrubbed);
|
|
}
|
|
|
|
#[test]
|
|
fn diag_shape_matches_golden() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let diag =
|
|
capture_robot_json_value(test_home.path(), &["diag", "--json"], ExpectStatus::ExitOk);
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&diag)).expect("pretty-print JSON");
|
|
assert_golden("robot/diag_shape.json.golden", &canonical);
|
|
}
|
|
|
|
#[test]
|
|
fn storage_json_matches_golden() {
|
|
// `cass storage --json` (#292.4) is the read-only footprint surface:
|
|
// per-component bytes that sum to a total. Against an isolated empty
|
|
// HOME with an absent data dir every component is zero and the layout
|
|
// is fully deterministic, so freezing it catches drift on any
|
|
// component key, ordering, or the bytes-sum-to-total invariant.
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let scrubbed = capture_robot_json(
|
|
test_home.path(),
|
|
&["storage", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
assert_golden("robot/storage.json.golden", &scrubbed);
|
|
}
|
|
|
|
#[test]
|
|
fn storage_shape_matches_golden() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let storage = capture_robot_json_value(
|
|
test_home.path(),
|
|
&["storage", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&storage)).expect("pretty-print JSON");
|
|
assert_golden("robot/storage_shape.json.golden", &canonical);
|
|
}
|
|
|
|
#[test]
|
|
fn dedup_dry_run_no_db_matches_golden() {
|
|
// `cass dedup --json` (#302 ask #2) defaults to a dry-run. Against an
|
|
// isolated empty HOME the database is absent, so the surface reports a
|
|
// deterministic "nothing to do" envelope (db_exists=false, zero
|
|
// counts, dry_run=true) that we freeze to catch contract drift.
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let scrubbed = capture_robot_json(test_home.path(), &["dedup", "--json"], ExpectStatus::ExitOk);
|
|
assert_golden("robot/dedup_no_db.json.golden", &scrubbed);
|
|
}
|
|
|
|
#[test]
|
|
fn dedup_shape_matches_golden() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let dedup =
|
|
capture_robot_json_value(test_home.path(), &["dedup", "--json"], ExpectStatus::ExitOk);
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&dedup)).expect("pretty-print JSON");
|
|
assert_golden("robot/dedup_shape.json.golden", &canonical);
|
|
}
|
|
|
|
#[test]
|
|
fn quarantine_list_empty_matches_golden() {
|
|
// `cass quarantine list --json` (#292 ask #3) against an isolated empty
|
|
// HOME has no quarantine_state.json, so the surface reports an empty,
|
|
// deterministic envelope (zero entries, breaker inactive) that we freeze.
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let scrubbed = capture_robot_json(
|
|
test_home.path(),
|
|
&["quarantine", "list", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
assert_golden("robot/quarantine_list_empty.json.golden", &scrubbed);
|
|
}
|
|
|
|
#[test]
|
|
fn quarantine_clear_shape_matches_golden() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let clear = capture_robot_json_value(
|
|
test_home.path(),
|
|
&["quarantine", "clear", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&clear)).expect("pretty-print JSON");
|
|
assert_golden("robot/quarantine_clear_shape.json.golden", &canonical);
|
|
}
|
|
|
|
#[test]
|
|
fn quarantine_retry_shape_matches_golden() {
|
|
// `cass quarantine retry --json` is the read-only plan by default. An
|
|
// isolated empty HOME yields a deterministic empty RetryPlan whose schema
|
|
// freezes the new xaztn robot contract without applying any mutation.
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let retry = capture_robot_json_value(
|
|
test_home.path(),
|
|
&["quarantine", "retry", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&retry)).expect("pretty-print JSON");
|
|
assert_golden("robot/quarantine_retry_shape.json.golden", &canonical);
|
|
}
|
|
|
|
#[test]
|
|
fn diag_quarantine_json_matches_golden() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let data_dir = seed_diag_quarantine_fixture(test_home.path());
|
|
let output = cass_cmd(test_home.path())
|
|
.env("CASS_LEXICAL_PUBLISH_BACKUP_RETENTION", "1")
|
|
.args([
|
|
"diag",
|
|
"--json",
|
|
"--quarantine",
|
|
"--data-dir",
|
|
data_dir.to_str().expect("utf8 path"),
|
|
])
|
|
.output()
|
|
.expect("run cass diag --json --quarantine");
|
|
assert!(
|
|
output.status.success(),
|
|
"cass diag --json --quarantine exited non-zero: status={:?}\nstdout:\n{}\nstderr:\n{}",
|
|
output.status,
|
|
String::from_utf8_lossy(&output.stdout),
|
|
String::from_utf8_lossy(&output.stderr),
|
|
);
|
|
let stdout = String::from_utf8(output.stdout).expect("utf8 stdout");
|
|
let parsed: serde_json::Value = serde_json::from_str(&stdout).unwrap_or_else(|err| {
|
|
panic!("diag --quarantine stdout is not JSON: {err}\nstdout:\n{stdout}")
|
|
});
|
|
let canonical = serde_json::to_string_pretty(&parsed).expect("pretty-print JSON");
|
|
let scrubbed = scrub_robot_json(&canonical, test_home.path());
|
|
assert_golden("robot/diag_quarantine.json.golden", &scrubbed);
|
|
}
|
|
|
|
#[test]
|
|
fn doctor_quarantine_json_matches_golden() -> Result<(), Box<dyn Error>> {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
isolate_doctor_fixture_from_parent_repo(test_home.path())?;
|
|
let data_dir = seed_diag_quarantine_fixture(test_home.path());
|
|
let output = cass_cmd(test_home.path())
|
|
.env("CASS_LEXICAL_PUBLISH_BACKUP_RETENTION", "1")
|
|
.args([
|
|
"doctor",
|
|
"--json",
|
|
"--data-dir",
|
|
data_dir.to_str().expect("utf8 path"),
|
|
])
|
|
.output()
|
|
.expect("run cass doctor --json");
|
|
assert!(
|
|
output.status.success(),
|
|
"cass doctor --json exited non-zero: status={:?}\nstdout:\n{}\nstderr:\n{}",
|
|
output.status,
|
|
String::from_utf8_lossy(&output.stdout),
|
|
String::from_utf8_lossy(&output.stderr),
|
|
);
|
|
let stdout = String::from_utf8(output.stdout).expect("utf8 stdout");
|
|
let parsed: serde_json::Value = serde_json::from_str(&stdout)
|
|
.unwrap_or_else(|err| panic!("doctor stdout is not JSON: {err}\nstdout:\n{stdout}"));
|
|
let canonical = serde_json::to_string_pretty(&parsed).expect("pretty-print JSON");
|
|
let scrubbed = scrub_robot_json(&canonical, test_home.path());
|
|
assert_golden("robot/doctor_quarantine.json.golden", &scrubbed);
|
|
Ok(())
|
|
}
|
|
|
|
#[test]
|
|
fn status_quarantine_json_matches_golden() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let data_dir = seed_diag_quarantine_fixture(test_home.path());
|
|
let output = cass_cmd(test_home.path())
|
|
.env("CASS_LEXICAL_PUBLISH_BACKUP_RETENTION", "1")
|
|
.args([
|
|
"status",
|
|
"--json",
|
|
"--data-dir",
|
|
data_dir.to_str().expect("utf8 path"),
|
|
])
|
|
.output()
|
|
.expect("run cass status --json");
|
|
assert!(
|
|
output.status.success(),
|
|
"cass status --json exited non-zero: status={:?}\nstdout:\n{}\nstderr:\n{}",
|
|
output.status,
|
|
String::from_utf8_lossy(&output.stdout),
|
|
String::from_utf8_lossy(&output.stderr),
|
|
);
|
|
let stdout = String::from_utf8(output.stdout).expect("utf8 stdout");
|
|
let parsed: serde_json::Value = serde_json::from_str(&stdout)
|
|
.unwrap_or_else(|err| panic!("status stdout is not JSON: {err}\nstdout:\n{stdout}"));
|
|
let canonical = serde_json::to_string_pretty(&parsed).expect("pretty-print JSON");
|
|
let scrubbed = scrub_robot_json(&canonical, test_home.path());
|
|
assert_golden("robot/status_quarantine.json.golden", &scrubbed);
|
|
}
|
|
|
|
#[test]
|
|
fn status_quarantine_full_json_matches_golden() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let data_dir = seed_diag_quarantine_fixture(test_home.path());
|
|
let output = cass_cmd(test_home.path())
|
|
.env("CASS_LEXICAL_PUBLISH_BACKUP_RETENTION", "1")
|
|
.args([
|
|
"status",
|
|
"--json",
|
|
"--data-dir",
|
|
data_dir.to_str().expect("utf8 path"),
|
|
])
|
|
.output()
|
|
.expect("run cass status --json");
|
|
assert!(
|
|
output.status.success(),
|
|
"cass status --json exited non-zero: status={:?}\nstdout:\n{}\nstderr:\n{}",
|
|
output.status,
|
|
String::from_utf8_lossy(&output.stdout),
|
|
String::from_utf8_lossy(&output.stderr),
|
|
);
|
|
let stdout = String::from_utf8(output.stdout).expect("utf8 stdout");
|
|
let parsed: serde_json::Value = serde_json::from_str(&stdout)
|
|
.unwrap_or_else(|err| panic!("status stdout is not JSON: {err}\nstdout:\n{stdout}"));
|
|
let canonical = serde_json::to_string_pretty(&parsed).expect("pretty-print JSON");
|
|
let scrubbed = scrub_robot_json(&canonical, test_home.path());
|
|
assert_golden("robot/status_quarantine_full.json.golden", &scrubbed);
|
|
}
|
|
|
|
#[test]
|
|
fn quarantine_summary_shape_matches_golden() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let data_dir = seed_diag_quarantine_fixture(test_home.path());
|
|
|
|
fn command_json(
|
|
test_home: &std::path::Path,
|
|
data_dir: &std::path::Path,
|
|
args: &[&str],
|
|
) -> Value {
|
|
let output = cass_cmd(test_home)
|
|
.env("CASS_LEXICAL_PUBLISH_BACKUP_RETENTION", "1")
|
|
.args(args)
|
|
.arg(data_dir)
|
|
.output()
|
|
.expect("run cass command");
|
|
assert!(
|
|
output.status.success(),
|
|
"cass {args:?} exited non-zero: status={:?}\nstdout:\n{}\nstderr:\n{}",
|
|
output.status,
|
|
String::from_utf8_lossy(&output.stdout),
|
|
String::from_utf8_lossy(&output.stderr),
|
|
);
|
|
serde_json::from_slice(&output.stdout).expect("valid JSON")
|
|
}
|
|
|
|
let status = command_json(
|
|
test_home.path(),
|
|
&data_dir,
|
|
&["status", "--json", "--data-dir"],
|
|
);
|
|
let diag = command_json(
|
|
test_home.path(),
|
|
&data_dir,
|
|
&["diag", "--json", "--quarantine", "--data-dir"],
|
|
);
|
|
let doctor = command_json(
|
|
test_home.path(),
|
|
&data_dir,
|
|
&["doctor", "--json", "--data-dir"],
|
|
);
|
|
|
|
let status_shape = json_value_schema(&status["quarantine"]["summary"]);
|
|
let diag_shape = json_value_schema(&diag["quarantine"]["summary"]);
|
|
let doctor_shape = json_value_schema(&doctor["quarantine"]["summary"]);
|
|
|
|
assert_eq!(
|
|
status_shape, diag_shape,
|
|
"status and diag quarantine summaries must expose the same schema"
|
|
);
|
|
assert_eq!(
|
|
status_shape, doctor_shape,
|
|
"status and doctor quarantine summaries must expose the same schema"
|
|
);
|
|
|
|
let canonical = serde_json::to_string_pretty(&status_shape).expect("pretty-print JSON");
|
|
assert_golden("robot/quarantine_summary_shape.json.golden", &canonical);
|
|
}
|
|
|
|
#[test]
|
|
fn api_version_json_matches_golden() {
|
|
// `cass api-version --json` is the smallest LLM contract surface —
|
|
// three fields (crate_version, api_version, contract_version) that
|
|
// together tell an agent "am I talking to a compatible cass build".
|
|
// A silent bump of api_version or contract_version without a
|
|
// coordinated client update breaks every downstream agent. Freezing
|
|
// here catches the drift at commit time.
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let scrubbed = capture_robot_json(
|
|
test_home.path(),
|
|
&["api-version", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
assert_golden("robot/api_version.json.golden", &scrubbed);
|
|
}
|
|
|
|
#[test]
|
|
fn api_version_shape_matches_golden() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let api_version = capture_robot_json_value(
|
|
test_home.path(),
|
|
&["api-version", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&api_version)).expect("pretty-print JSON");
|
|
assert_golden("robot/api_version_shape.json.golden", &canonical);
|
|
}
|
|
|
|
#[test]
|
|
fn stats_json_missing_db_error_envelope_matches_golden() {
|
|
// `cass stats --json` against an isolated empty HOME emits the
|
|
// error-envelope variant of the robot-mode JSON contract: a structured
|
|
// `{"error": {"code", "kind", "message", "hint", "retryable"}}` payload
|
|
// documented in robot-docs' exit-codes topic. Freezing this catches
|
|
// silent drift in the error-envelope shape — important because agent
|
|
// error-handling branches key on these exact fields.
|
|
//
|
|
// Robot-mode success payloads emit on stdout; fatal error envelopes
|
|
// emit on stderr so stdout remains data-only for successful commands.
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let out = cass_cmd(test_home.path())
|
|
.args([
|
|
"stats",
|
|
"--json",
|
|
"--data-dir",
|
|
test_home.path().to_str().expect("utf8 path"),
|
|
])
|
|
.output()
|
|
.expect("run cass stats --json");
|
|
let stderr = String::from_utf8(out.stderr).expect("utf8 stderr");
|
|
let parsed: serde_json::Value =
|
|
serde_json::from_str(&stderr).expect("stats error envelope is JSON on stderr");
|
|
let canonical = serde_json::to_string_pretty(&parsed).expect("pretty-print JSON");
|
|
let scrubbed = scrub_robot_json(&canonical, test_home.path());
|
|
assert_golden("robot/stats_missing_db.json.golden", &scrubbed);
|
|
}
|
|
|
|
#[test]
|
|
fn stats_json_happy_path_matches_golden() -> Result<(), Box<dyn Error>> {
|
|
// `coding_agent_session_search-zefv4`: the error envelope has been
|
|
// pinned (stats_missing_db* goldens) but the success envelope had no
|
|
// freeze — regressions to a field name or a new mandatory key on
|
|
// the common-case happy-path would pass CI silently. Seeds the
|
|
// existing search_demo_data fixture (324 KB canonical DB with a
|
|
// known conversation/message count), invokes `cass stats --json`,
|
|
// and freezes the scrubbed envelope.
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let data_dir = isolated_search_demo_data(test_home.path())?;
|
|
let out = cass_cmd(test_home.path())
|
|
.args([
|
|
"stats",
|
|
"--json",
|
|
"--data-dir",
|
|
data_dir.to_str().expect("utf8 path"),
|
|
])
|
|
.output()
|
|
.expect("run cass stats --json on fixture DB");
|
|
assert!(
|
|
out.status.success(),
|
|
"cass stats --json must succeed on fixture DB; status={:?}\nstdout:\n{}\nstderr:\n{}",
|
|
out.status,
|
|
String::from_utf8_lossy(&out.stdout),
|
|
String::from_utf8_lossy(&out.stderr),
|
|
);
|
|
let stdout = String::from_utf8(out.stdout).expect("utf8 stdout");
|
|
let parsed: serde_json::Value =
|
|
serde_json::from_str(&stdout).expect("stats happy-path envelope is JSON");
|
|
let canonical = serde_json::to_string_pretty(&parsed).expect("pretty-print JSON");
|
|
let scrubbed = scrub_robot_json(&canonical, test_home.path());
|
|
assert_golden("robot/stats_full_payload.json.golden", &scrubbed);
|
|
Ok(())
|
|
}
|
|
|
|
#[test]
|
|
fn stats_json_happy_path_shape_matches_golden() -> Result<(), Box<dyn Error>> {
|
|
// Shape-only pin for the happy-path envelope so a future refactor
|
|
// of the scrubber (or drift in fixture contents) can't accidentally
|
|
// mask structural regressions. json_value_schema diff tolerates
|
|
// value changes; keys + types must hold.
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let data_dir = isolated_search_demo_data(test_home.path())?;
|
|
let out = cass_cmd(test_home.path())
|
|
.args([
|
|
"stats",
|
|
"--json",
|
|
"--data-dir",
|
|
data_dir.to_str().expect("utf8 path"),
|
|
])
|
|
.output()
|
|
.expect("run cass stats --json on fixture DB");
|
|
assert!(out.status.success(), "stats must succeed");
|
|
let parsed: serde_json::Value =
|
|
serde_json::from_slice(&out.stdout).expect("stats happy-path envelope is JSON");
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&parsed)).expect("pretty-print JSON");
|
|
assert_golden("robot/stats_full_payload_shape.json.golden", &canonical);
|
|
Ok(())
|
|
}
|
|
|
|
#[test]
|
|
fn stats_json_missing_db_error_envelope_shape_matches_golden() {
|
|
// Error envelope lives on stderr; stdout stays reserved for successful data.
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let out = cass_cmd(test_home.path())
|
|
.args([
|
|
"stats",
|
|
"--json",
|
|
"--data-dir",
|
|
test_home.path().to_str().expect("utf8 path"),
|
|
])
|
|
.output()
|
|
.expect("run cass stats --json");
|
|
let parsed: serde_json::Value =
|
|
serde_json::from_slice(&out.stderr).expect("stats error envelope is JSON on stderr");
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&parsed)).expect("pretty-print JSON");
|
|
assert_golden("robot/stats_missing_db_shape.json.golden", &canonical);
|
|
}
|
|
|
|
#[test]
|
|
fn analytics_incidents_json_matches_golden() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let db_path = seed_analytics_incidents_fixture(test_home.path());
|
|
let db_path = db_path.to_str().expect("utf8 fixture db path");
|
|
let scrubbed = capture_robot_json(
|
|
test_home.path(),
|
|
&[
|
|
"--db",
|
|
db_path,
|
|
"analytics",
|
|
"incidents",
|
|
"--json",
|
|
"--limit",
|
|
"2",
|
|
"--max-sessions",
|
|
"10",
|
|
"--max-messages",
|
|
"100",
|
|
"--max-bytes",
|
|
"1000000",
|
|
"--budget-ms",
|
|
"60000",
|
|
],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
let scrubbed = format!("{scrubbed}\n");
|
|
assert_golden("robot/analytics_incidents.json.golden", &scrubbed);
|
|
}
|
|
|
|
#[test]
|
|
fn introspect_json_matches_golden() {
|
|
// `cass introspect --json` is the full API schema surface — every
|
|
// subcommand, its flags, positional args, and response-schema
|
|
// references. Agents that bind to cass programmatically use this
|
|
// to generate typed clients; silent drift breaks every downstream
|
|
// client.
|
|
//
|
|
// Was #[ignore]'d when first captured (HashMap-based schema registry
|
|
// emitted non-deterministic key order — filed as bead
|
|
// coding_agent_session_search-8sl73). The underlying HashMap was
|
|
// swapped for BTreeMap in the same commit that re-enabled this test;
|
|
// byte-identical output is now verified across independent runs.
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let scrubbed = capture_robot_json(
|
|
test_home.path(),
|
|
&["introspect", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
assert_golden("robot/introspect.json.golden", &scrubbed);
|
|
}
|
|
|
|
#[test]
|
|
fn introspect_shape_matches_golden() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let introspect = capture_robot_json_value(
|
|
test_home.path(),
|
|
&["introspect", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&introspect)).expect("pretty-print JSON");
|
|
assert_golden("robot/introspect_shape.json.golden", &canonical);
|
|
}
|
|
|
|
#[test]
|
|
fn introspect_subsystem_coverage_matches_executable_matrix() -> Result<(), String> {
|
|
let test_home = tempfile::tempdir().map_err(|error| error.to_string())?;
|
|
let introspect = capture_robot_json_value(
|
|
test_home.path(),
|
|
&["introspect", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
let published = introspect
|
|
.get("subsystem_coverage")
|
|
.ok_or_else(|| "introspect output omits subsystem_coverage".to_string())?;
|
|
let expected = serde_json::to_value(matrix_report()).map_err(|error| error.to_string())?;
|
|
if !published.eq(&expected) {
|
|
return Err("introspect subsystem_coverage drifted from matrix_report()".to_string());
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
#[derive(Debug)]
|
|
struct PackContractRequirement {
|
|
level: &'static str,
|
|
requirement: &'static str,
|
|
proof_tests: &'static [&'static str],
|
|
status: &'static str,
|
|
}
|
|
|
|
const PACK_CONTRACT_MATRIX: &[PackContractRequirement] = &[
|
|
PackContractRequirement {
|
|
level: "MUST",
|
|
requirement: "introspect exposes the pack command and response schema",
|
|
proof_tests: &[
|
|
"introspect_json_matches_golden",
|
|
"introspect_shape_matches_golden",
|
|
"pack_introspect_contract_matrix_is_current",
|
|
],
|
|
status: "covered",
|
|
},
|
|
PackContractRequirement {
|
|
level: "MUST",
|
|
requirement: "pack accepts agent, workspace, source, time, and sessions-from filters",
|
|
proof_tests: &[
|
|
"parse_pack_robot_contract_flags",
|
|
"pack_introspect_contract_matrix_is_current",
|
|
],
|
|
status: "covered",
|
|
},
|
|
PackContractRequirement {
|
|
level: "MUST",
|
|
requirement: "pack exposes token, session, evidence, context, and excerpt budgets",
|
|
proof_tests: &[
|
|
"parse_pack_robot_contract_flags",
|
|
"exact_token_budget_boundary_selects_until_budget_exhausted",
|
|
"pack_introspect_contract_matrix_is_current",
|
|
],
|
|
status: "covered",
|
|
},
|
|
PackContractRequirement {
|
|
level: "MUST",
|
|
requirement: "pack reports readiness, freshness, realized mode, privacy, evidence, and omissions",
|
|
proof_tests: &[
|
|
"introspect_json_matches_golden",
|
|
"render_json_includes_readiness_and_warning_fields",
|
|
"pack_introspect_contract_matrix_is_current",
|
|
],
|
|
status: "covered",
|
|
},
|
|
PackContractRequirement {
|
|
level: "MUST",
|
|
requirement: "pack robot errors keep stdout data-only and use JSON error envelopes",
|
|
proof_tests: &[
|
|
"pack_empty_query_json_error_uses_stderr_only",
|
|
"pack_invalid_field_with_sessions_from_stdin_is_json_error",
|
|
"pack_rejects_sessions_robot_format_before_search",
|
|
],
|
|
status: "covered",
|
|
},
|
|
PackContractRequirement {
|
|
level: "SHOULD",
|
|
requirement: "operators have an rch-based golden regeneration workflow",
|
|
proof_tests: &[
|
|
"golden_regeneration_hints_do_not_use_bare_cargo",
|
|
"pack_robot_docs_contract_matrix_is_current",
|
|
],
|
|
status: "covered",
|
|
},
|
|
];
|
|
|
|
fn assert_pack_contract_matrix_complete() {
|
|
assert!(
|
|
PACK_CONTRACT_MATRIX
|
|
.iter()
|
|
.all(|row| matches!(row.level, "MUST" | "SHOULD")),
|
|
"pack contract matrix levels must be explicit MUST/SHOULD entries"
|
|
);
|
|
assert!(
|
|
PACK_CONTRACT_MATRIX
|
|
.iter()
|
|
.all(|row| row.status == "covered"),
|
|
"pack contract matrix contains uncovered rows: {PACK_CONTRACT_MATRIX:#?}"
|
|
);
|
|
assert!(
|
|
PACK_CONTRACT_MATRIX
|
|
.iter()
|
|
.all(|row| !row.requirement.trim().is_empty() && !row.proof_tests.is_empty()),
|
|
"pack contract matrix rows must list a requirement and proof tests"
|
|
);
|
|
}
|
|
|
|
fn find_introspect_command<'a>(introspect: &'a Value, name: &str) -> &'a Value {
|
|
introspect["commands"]
|
|
.as_array()
|
|
.expect("introspect.commands is an array")
|
|
.iter()
|
|
.find(|command| command["name"] == name)
|
|
.unwrap_or_else(|| panic!("introspect missing command {name}"))
|
|
}
|
|
|
|
fn find_introspect_argument<'a>(command: &'a Value, name: &str) -> &'a Value {
|
|
command["arguments"]
|
|
.as_array()
|
|
.expect("command.arguments is an array")
|
|
.iter()
|
|
.find(|argument| argument["name"] == name)
|
|
.unwrap_or_else(|| panic!("command missing argument {name}"))
|
|
}
|
|
|
|
fn assert_introspect_argument(
|
|
command: &Value,
|
|
name: &str,
|
|
arg_type: &str,
|
|
value_type: Option<&str>,
|
|
) {
|
|
let argument = find_introspect_argument(command, name);
|
|
assert_eq!(argument["arg_type"], arg_type, "{name} arg_type drifted");
|
|
if let Some(value_type) = value_type {
|
|
assert_eq!(
|
|
argument["value_type"], value_type,
|
|
"{name} value_type drifted"
|
|
);
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn pack_introspect_contract_matrix_is_current() {
|
|
assert_pack_contract_matrix_complete();
|
|
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let capabilities = capture_robot_json_value(
|
|
test_home.path(),
|
|
&["capabilities", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
let features = capabilities["features"]
|
|
.as_array()
|
|
.expect("capabilities.features is an array");
|
|
assert!(
|
|
features
|
|
.iter()
|
|
.any(|feature| feature == "answer_pack_command"),
|
|
"capabilities must advertise answer_pack_command"
|
|
);
|
|
|
|
let introspect = capture_robot_json_value(
|
|
test_home.path(),
|
|
&["introspect", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
let pack = find_introspect_command(&introspect, "pack");
|
|
assert_eq!(pack["has_json_output"], true);
|
|
|
|
for (name, arg_type, value_type) in [
|
|
("query", "positional", Some("string")),
|
|
("agent", "option", Some("string")),
|
|
("workspace", "option", Some("string")),
|
|
("limit", "option", Some("integer")),
|
|
("json", "flag", None),
|
|
("fields", "option", Some("string")),
|
|
("max-tokens", "option", Some("integer")),
|
|
("max-sessions", "option", Some("integer")),
|
|
("max-evidence", "option", Some("integer")),
|
|
("context-lines", "option", Some("integer")),
|
|
("max-excerpt-chars", "option", Some("integer")),
|
|
("request-id", "option", Some("string")),
|
|
("display", "option", Some("enum")),
|
|
("data-dir", "option", Some("path")),
|
|
("days", "option", Some("integer")),
|
|
("source", "option", Some("string")),
|
|
("sessions-from", "option", Some("string")),
|
|
("mode", "option", Some("enum")),
|
|
("freshness-policy", "option", Some("enum")),
|
|
("freshness-window-seconds", "option", Some("integer")),
|
|
("require-evidence", "flag", None),
|
|
("explain-selection", "flag", None),
|
|
("refresh", "flag", None),
|
|
("timeout", "option", Some("integer")),
|
|
("robot-format", "option", Some("enum")),
|
|
] {
|
|
assert_introspect_argument(pack, name, arg_type, value_type);
|
|
}
|
|
|
|
let pack_schema = &introspect["response_schemas"]["pack"]["properties"];
|
|
for field in [
|
|
"schema_version",
|
|
"query",
|
|
"_meta",
|
|
"limits",
|
|
"realized",
|
|
"health",
|
|
"freshness",
|
|
"pack",
|
|
"evidence",
|
|
"omitted",
|
|
"privacy",
|
|
"warnings",
|
|
] {
|
|
assert!(
|
|
pack_schema.get(field).is_some(),
|
|
"pack response schema missing {field}"
|
|
);
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn search_robot_json_matches_golden() -> Result<(), Box<dyn Error>> {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let data_dir = isolated_search_demo_data(test_home.path())?;
|
|
let output = cass_cmd(test_home.path())
|
|
.args([
|
|
"search",
|
|
"hello",
|
|
"--json",
|
|
"--limit",
|
|
"2",
|
|
"--data-dir",
|
|
data_dir.to_str().expect("utf8 path"),
|
|
])
|
|
.output()
|
|
.expect("run cass search --json");
|
|
assert!(
|
|
output.status.success(),
|
|
"cass search --json exited non-zero: status={:?}\nstdout:\n{}\nstderr:\n{}",
|
|
output.status,
|
|
String::from_utf8_lossy(&output.stdout),
|
|
String::from_utf8_lossy(&output.stderr),
|
|
);
|
|
let stdout = String::from_utf8(output.stdout).expect("utf8 stdout");
|
|
let parsed: serde_json::Value = serde_json::from_str(&stdout)
|
|
.unwrap_or_else(|err| panic!("search stdout is not JSON: {err}\nstdout:\n{stdout}"));
|
|
let canonical = serde_json::to_string_pretty(&parsed).expect("pretty-print JSON");
|
|
let scrubbed = scrub_robot_json(&canonical, test_home.path());
|
|
assert_golden("robot/search_robot.json.golden", &scrubbed);
|
|
Ok(())
|
|
}
|
|
|
|
#[test]
|
|
fn search_robot_shape_matches_golden() -> Result<(), Box<dyn Error>> {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let data_dir = isolated_search_demo_data(test_home.path())?;
|
|
let output = cass_cmd(test_home.path())
|
|
.args([
|
|
"search",
|
|
"hello",
|
|
"--json",
|
|
"--limit",
|
|
"2",
|
|
"--data-dir",
|
|
data_dir.to_str().expect("utf8 path"),
|
|
])
|
|
.output()
|
|
.expect("run cass search --json");
|
|
assert!(
|
|
output.status.success(),
|
|
"cass search --json exited non-zero: status={:?}\nstdout:\n{}\nstderr:\n{}",
|
|
output.status,
|
|
String::from_utf8_lossy(&output.stdout),
|
|
String::from_utf8_lossy(&output.stderr),
|
|
);
|
|
let parsed: serde_json::Value =
|
|
serde_json::from_slice(&output.stdout).expect("valid search JSON");
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&parsed)).expect("pretty-print JSON");
|
|
assert_golden("robot/search_robot_shape.json.golden", &canonical);
|
|
Ok(())
|
|
}
|
|
|
|
// ========================================================================
|
|
// Bead coding_agent_session_search-v4kz1 (child of ibuuh.10):
|
|
// Golden-artifact freeze for `cass export-html --json` envelope.
|
|
//
|
|
// Existing tests/pages_export_golden.rs spot-asserts three fields on
|
|
// the export-html JSON payload (`success`, `exported.encrypted`,
|
|
// `exported.messages_count`) but nothing pins the ENVELOPE SHAPE.
|
|
// Any regression that renamed / added / removed fields across the
|
|
// `success=true` branch ships through every consumer silently.
|
|
//
|
|
// Freeze the schema (types + keys, values scrubbed) exactly the way
|
|
// the sibling `capabilities_shape_matches_golden`,
|
|
// `health_shape_matches_golden`, and `quarantine_summary_shape_matches_golden`
|
|
// tests do. The golden file lives at
|
|
// `tests/golden/robot/export_html_shape.json.golden` and follows the
|
|
// standard UPDATE_GOLDENS=1 regeneration procedure documented at
|
|
// the top of this file.
|
|
// ========================================================================
|
|
|
|
#[test]
|
|
fn export_html_shape_matches_golden() {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let session_path = test_home.path().join("rollout-export-shape.jsonl");
|
|
// Minimal but complete Codex rollout: session_meta + one user +
|
|
// one assistant message. Matches the shape the main
|
|
// pages_export_golden.rs suite uses so the fixture mirrors real
|
|
// export input.
|
|
fs::write(
|
|
&session_path,
|
|
concat!(
|
|
r#"{"timestamp":"2024-04-24T00:00:00Z","type":"session_meta","payload":{"id":"export-golden","cwd":"/tmp","cli_version":"0.42.0"}}"#,
|
|
"\n",
|
|
r#"{"timestamp":"2024-04-24T00:00:01Z","type":"response_item","payload":{"type":"message","role":"user","content":[{"type":"input_text","text":"hello"}]}}"#,
|
|
"\n",
|
|
r#"{"timestamp":"2024-04-24T00:00:02Z","type":"response_item","payload":{"type":"message","role":"assistant","content":[{"type":"text","text":"hi"}]}}"#,
|
|
"\n",
|
|
),
|
|
)
|
|
.expect("write session fixture");
|
|
|
|
let output_dir = test_home.path().join("export-out");
|
|
fs::create_dir_all(&output_dir).expect("create output dir");
|
|
|
|
let output = cass_cmd(test_home.path())
|
|
.arg("export-html")
|
|
.arg(&session_path)
|
|
.arg("--json")
|
|
.arg("--no-cdns")
|
|
.arg("--output-dir")
|
|
.arg(&output_dir)
|
|
.arg("--filename")
|
|
.arg("shape-probe")
|
|
.output()
|
|
.expect("run cass export-html");
|
|
|
|
assert!(
|
|
output.status.success(),
|
|
"cass export-html --json must succeed on a valid rollout; status={:?}\n\
|
|
stdout:\n{}\nstderr:\n{}",
|
|
output.status,
|
|
String::from_utf8_lossy(&output.stdout),
|
|
String::from_utf8_lossy(&output.stderr),
|
|
);
|
|
|
|
let payload: Value = serde_json::from_slice(&output.stdout).expect("export-html emits JSON");
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&payload)).expect("pretty-print JSON");
|
|
assert_golden("robot/export_html_shape.json.golden", &canonical);
|
|
}
|
|
|
|
// `coding_agent_session_search-oy4fd`: README line 103 advertises
|
|
// sessions / models-verify / models-check-update as golden-pinned
|
|
// JSON contract surfaces, but no goldens existed for them. The three
|
|
// tests below close that gap with shape goldens (json_value_schema
|
|
// diffs tolerate run-time values like timestamps while still pinning
|
|
// the envelope keys and types). Each test seeds the minimal state
|
|
// needed to reach a deterministic branch: sessions hits the
|
|
// missing-db error envelope (stderr, mirrors stats_missing_db
|
|
// convention); models verify/check-update run against an empty
|
|
// data_dir where the model is not yet acquired, so they reach the
|
|
// stable `not_acquired` / `model_not_installed` branches.
|
|
|
|
// `coding_agent_session_search-q931h`: status and doctor had only
|
|
// variant-scoped goldens (status_quarantine{_full}, status_semantic_*,
|
|
// doctor_quarantine). The base not-initialized envelopes emitted
|
|
// for `cass status --json` / `cass doctor --json` against a fresh
|
|
// empty data_dir — the most common shape agent harnesses see before
|
|
// the first index — had no shape pin at all. A regression that
|
|
// added, removed, or re-typed a field in the base envelope would
|
|
// compile clean and pass the existing suite. The two tests below
|
|
// close that gap via json_value_schema diffs (same pattern as
|
|
// health_shape_matches_golden / diag_shape_matches_golden).
|
|
|
|
// `coding_agent_session_search-ut3v8`: the --quarantine subset of
|
|
// cass doctor --json is frozen via doctor_quarantine.json.golden,
|
|
// but the DEFAULT base-state invocation (no --quarantine, no seeded
|
|
// fixture) had no instance freeze. Regressions to the top-level
|
|
// status / recommended_action / checks[] envelope on the fresh
|
|
// empty data_dir — the shape agent harnesses see before any index
|
|
// exists — would not fail at golden time. Closes the instance-side
|
|
// of the pin; the shape-side lives in doctor_shape.json.golden
|
|
// (bead q931h).
|
|
#[test]
|
|
fn doctor_json_matches_golden() -> Result<(), Box<dyn Error>> {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
isolate_doctor_fixture_from_parent_repo(test_home.path())?;
|
|
let scrubbed = capture_robot_json(
|
|
test_home.path(),
|
|
&["doctor", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
assert_golden("robot/doctor.json.golden", &scrubbed);
|
|
Ok(())
|
|
}
|
|
|
|
#[test]
|
|
fn status_shape_matches_golden() -> Result<(), &'static str> {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let mut status = capture_robot_json_value(
|
|
test_home.path(),
|
|
&["status", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
// The load-average controller is Linux-only by default, so these fields
|
|
// are numbers on Linux and null elsewhere. Validate that runtime contract,
|
|
// then pin one representative number so the shape golden is portable.
|
|
for (pointer, representative) in [
|
|
(
|
|
"/rebuild/pipeline/controller_loadavg_high_watermark_1m",
|
|
121.0,
|
|
),
|
|
(
|
|
"/rebuild/pipeline/controller_loadavg_low_watermark_1m",
|
|
120.0,
|
|
),
|
|
] {
|
|
let Some(value) = status.pointer_mut(pointer) else {
|
|
return Err("status shape fixture missing a load-average controller field");
|
|
};
|
|
if !(value.is_null() || value.is_number()) {
|
|
return Err("status shape fixture load-average field was not null|number");
|
|
}
|
|
*value = json!(representative);
|
|
}
|
|
for pointer in [
|
|
"/topology_budget/topology/memory_total_bytes",
|
|
"/topology_budget/topology/memory_available_bytes",
|
|
] {
|
|
let Some(value) = status.pointer_mut(pointer) else {
|
|
return Err("status shape fixture missing a topology memory field");
|
|
};
|
|
if !(value.is_null() || value.is_u64()) {
|
|
return Err("status shape fixture topology memory field was not null|u64");
|
|
}
|
|
*value = json!(536_870_912_u64);
|
|
}
|
|
// Keep the warnings array item schema pinned even when this fixture has no
|
|
// warning instances.
|
|
if let Some(warnings) = status
|
|
.pointer_mut("/quarantine/warnings")
|
|
.and_then(Value::as_array_mut)
|
|
&& warnings.is_empty()
|
|
{
|
|
warnings.push(Value::String("[SHAPE_STRING]".to_string()));
|
|
}
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&status)).expect("pretty-print JSON");
|
|
assert_golden("robot/status_shape.json.golden", &canonical);
|
|
Ok(())
|
|
}
|
|
|
|
#[test]
|
|
fn doctor_shape_matches_golden() -> Result<(), Box<dyn Error>> {
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
isolate_doctor_fixture_from_parent_repo(test_home.path())?;
|
|
let doctor = capture_robot_json_value(
|
|
test_home.path(),
|
|
&["doctor", "--json"],
|
|
ExpectStatus::ExitOk,
|
|
);
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&doctor)).expect("pretty-print JSON");
|
|
assert_golden("robot/doctor_shape.json.golden", &canonical);
|
|
Ok(())
|
|
}
|
|
|
|
#[test]
|
|
fn sessions_json_missing_db_error_envelope_shape_matches_golden() {
|
|
// Mirrors stats_json_missing_db_error_envelope_shape_matches_golden:
|
|
// no DB on a fresh data_dir means cass emits the `missing-db` error
|
|
// envelope on stderr with exit 3. Pinning the envelope shape lets agent
|
|
// harnesses branch on kind="missing-db" without worrying about
|
|
// silent contract drift.
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let out = cass_cmd(test_home.path())
|
|
.args([
|
|
"sessions",
|
|
"--current",
|
|
"--json",
|
|
"--data-dir",
|
|
test_home.path().to_str().expect("utf8 path"),
|
|
])
|
|
.output()
|
|
.expect("run cass sessions --current --json");
|
|
let parsed: serde_json::Value =
|
|
serde_json::from_slice(&out.stderr).expect("sessions error envelope is JSON on stderr");
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&parsed)).expect("pretty-print JSON");
|
|
assert_golden("robot/sessions_missing_db_shape.json.golden", &canonical);
|
|
}
|
|
|
|
#[test]
|
|
fn models_verify_json_not_acquired_shape_matches_golden() {
|
|
// Empty data_dir ⇒ model is not acquired, `cass models verify
|
|
// --json` emits the stable not_acquired envelope on stdout with
|
|
// exit 0. Shape golden pins: status, state_detail, next_step,
|
|
// lexical_fail_open, model_dir, all_valid, cache_lifecycle
|
|
// (nested), error.
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let out = cass_cmd(test_home.path())
|
|
.args([
|
|
"models",
|
|
"verify",
|
|
"--json",
|
|
"--data-dir",
|
|
test_home.path().to_str().expect("utf8 path"),
|
|
])
|
|
.output()
|
|
.expect("run cass models verify --json");
|
|
let stdout = String::from_utf8(out.stdout).expect("utf8 stdout");
|
|
let parsed: serde_json::Value = serde_json::from_str(&stdout)
|
|
.unwrap_or_else(|err| panic!("models verify stdout is not JSON: {err}\nstdout:\n{stdout}"));
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&parsed)).expect("pretty-print JSON");
|
|
assert_golden(
|
|
"robot/models_verify_not_acquired_shape.json.golden",
|
|
&canonical,
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn models_check_update_json_not_installed_shape_matches_golden() {
|
|
// Empty data_dir ⇒ `cass models check-update --json` returns
|
|
// `reason=model_not_installed` with current_revision=null +
|
|
// latest_revision=<pinned sha>. Shape golden pins the 4-field
|
|
// envelope so a regression that renamed/removed any field would
|
|
// trip CI.
|
|
let test_home = tempfile::tempdir().expect("create temp home");
|
|
let out = cass_cmd(test_home.path())
|
|
.args([
|
|
"models",
|
|
"check-update",
|
|
"--json",
|
|
"--data-dir",
|
|
test_home.path().to_str().expect("utf8 path"),
|
|
])
|
|
.output()
|
|
.expect("run cass models check-update --json");
|
|
let stdout = String::from_utf8(out.stdout).expect("utf8 stdout");
|
|
let parsed: serde_json::Value = serde_json::from_str(&stdout).unwrap_or_else(|err| {
|
|
panic!("models check-update stdout is not JSON: {err}\nstdout:\n{stdout}")
|
|
});
|
|
let canonical =
|
|
serde_json::to_string_pretty(&json_value_schema(&parsed)).expect("pretty-print JSON");
|
|
assert_golden(
|
|
"robot/models_check_update_not_installed_shape.json.golden",
|
|
&canonical,
|
|
);
|
|
}
|