2 Commits

Author	SHA1	Message	Date
Sanderhoff-alt	8e66c397a9	fix(memory-defense): correct displayed pattern count (#2369)	2026-06-23 15:03:20 +02:00
Nicolò Boschi	a0e6bedcf1	refactor(memory-defense): per-bank regex defense, webhooks, drop dead surface (#2077) ... * Implement Memory Guard Lite for OSS Allow users to prevent token and secret leakage in agent memory. feat(memory-defense): reject quarantine action in policy parser refactor(retain): drop quarantine branch from orchestrator test(retain): remove quarantine-path tests refactor(memory-defense): remove DefenseAction.QUARANTINE enum value refactor(api): remove include_quarantined query parameter refactor(recall): drop include_quarantined parameter from memory engine test(memory-defense): replace stale parser-reject test with full-union accept test The previous parametrized test asserted parse_policy() should 422 on any detector name other than sensitive_data. That contract was deliberately widened on 2026-06-07 so cloud-style policies pass through api-slim's parser unchanged. The test was stale; the runtime is correct. Replaced with test_parse_policy_accepts_full_detector_union, which proves the actual contract: all 7 detector names are valid in the parser, with dispatch and entitlement enforcement deferred to the loaded extension. Memory defense UI * i18n labels * Fix tests * Client changes to fix breaking tests * Test fixes * chore: regenerate API clients via generate-clients.sh The clients were previously hand-generated in a way that diverged from the project's tooling — including a non-standard hindsight-clients/typescript/client/ directory the generator never produces (the standard output is typescript/generated/), plus ~150 spurious files. Revert the entire hindsight-clients/ tree to main and regenerate from the OpenAPI spec using ./scripts/generate-clients.sh (Rust via progenitor build.rs, Python/Go via openapi-generator, TypeScript via @hey-api/openapi-ts). The spec itself is unchanged (a code-regenerated spec is byte-identical to what was already committed). Net result is the real API delta only: the new nullable MemoryItem.receipt_uri field propagated to the Python, TypeScript and Go models. * refactor(memory-defense): per-bank regex defense, webhooks, drop dead surface Review cleanup of the memory-defense feature: - Rename the OSS extension Lite -> Regex (MemoryDefenseRegexExtension, memory_defense_regex.py). It is pure regex redaction now. - Drop the agent_memory_guard (OWASP) dependency entirely — the SensitiveDataDetector fallback and to_owasp_policy are gone; nothing cloud-tier remains in api-slim. - Trim the policy to what OSS enforces: { enabled, rules:[{on:sensitive_data, action}] }. Removed default_action, protected/immutable namespaces, detector_overrides, min_severity, and the unused memory_defense_enabled_default server default. Per-bank override stays (memory_defense is a configurable field) and the UI writes the trimmed shape. - Fire a memory_defense.triggered webhook on every non-allow decision (redact and block) when one is configured, via the retain orchestrator. Adds WebhookEventType.MEMORY_DEFENSE_TRIGGERED + MemoryDefenseEventData. Replaces the no-op record_violation hook. - Block is now actually enforced (drop item / 422 when all blocked) instead of being silently downgraded to redact. - Remove the unused 'status' lifecycle: the add_status migration + its two merge migrations, the recall quarantine filter, the status column reads in search, and MemoryFact.status. Branch now adds zero migrations (single head). - Remove receipt_uri from the API (MemoryItem) and clients — it was always None and carried no value. Tests updated/renamed accordingly; OWASP smoke + enabled-default tests removed. * fix(memory-defense): address code-review findings - Delete test_migration_status.py (asserted the removed status column/constraint). - Remove receipt_uri from the Rust CLI (memory.rs + integration_test.rs) — the generated client struct no longer has the field, so it wouldn't compile. - Type the blocked-violations as a BlockedViolation dataclass instead of raw dicts (serialized via asdict() in the 422 body); type the webhook helper's decision param as DefenseDecision. - Add an end-to-end test asserting a redact decision queues a memory_defense.triggered webhook delivery. - Drop the unrelated docs/ entry from .gitignore (local scratch, not this PR). * test(memory-defense): consolidate into a single test_memory_defense.py Merge the 10 scattered memory-defense test modules (policy parser, regex engine/screen, redaction benchmark, extension loader, extension-context wiring, bank-config validation, and the three retain e2e files) into one test_memory_defense.py, deduping the overlapping unit screen tests and the duplicated retain redact e2e. 36 tests, same coverage. * docs(memory-defense): document memory_defense.triggered webhook + block action - Add memory_defense.triggered to the control-plane webhook event-type selector (it was firing but wasn't selectable in the UI). - Document the memory_defense.triggered event (payload + data fields) on the webhooks API page, and link it from the Memory Defense page. - Document the block action (the page only described redact) and add a Notifications section. Regenerate the docs skill copies. * docs: remove Memory Defense page from version-0.7 (unreleased feature) The feature was snapshotted into the 0.7 versioned docs by mistake — 0.7 never shipped Memory Defense. Remove the page and its (sole) Security sidebar category. * test(memory-defense): assert webhook payload fields + cover block path - test_retain_fires_webhook_on_redact now parses the queued delivery and asserts the MemoryDefenseEventData payload (action/detector/matched_types/ message + event status), not just that the event type was queued. - Add test_retain_fires_webhook_on_block: a block decision fires the webhook (before the 422 is raised) with action=block. Confirms the delivery persists despite the blocked retain returning 422. - Factor out _memory_defense_webhook_events() helper. * fix(control-plane): render structured API error details as a string A blocked retain returns 422 with detail {violations: [{message, ...}]}; the proxy forwards it as `details` and the client passed that object straight into the sonner toast, crashing with "Objects are not valid as a React child". Add describeErrorDetails() to reduce details to a string — joining violation messages when present (so a Memory Defense block shows e.g. "Sensitive data pattern matched: aws_access_key"), else JSON-stringifying. * docs(webhooks): clarify WebhookEvent.status covers the memory_defense action * feat(memory-defense): record redact/block actions in the audit log Emit a fire-and-forget 'memory_defense' audit entry for each non-allow decision (alongside the webhook), with the action/detector/document_id/matched_types in metadata. Threads the engine's AuditLogger into retain_batch like the webhook manager; gated by the existing audit_log_enabled switch (off by default). - Add the memory_defense option to the audit-logs UI action filter + the actionMemoryDefense i18n key across all locales. - Document it on the Memory Defense page and the audit-logging config section. - Test: a redact retain writes a memory_defense audit row with the expected metadata (audit enabled on the test engine). --------- Co-authored-by: Chris Latimer <chris.latimer@vectorize.io>	2026-06-10 14:12:46 +02:00