4 Commits

Author	SHA1	Message	Date
Nicolò Boschi	d936d4931c	feat(webhooks): emit X-Hub-Signature-256 and a timestamped signature (#3986) ... Webhook deliveries signed only `X-Hindsight-Signature`, a vendor name for a construction that is byte-for-byte the one GitHub popularised: `sha256=<hex>` HMAC-SHA256 over the raw body. Every receiver therefore needed a Hindsight-specific shim to verify a signature it already knew how to check. Emit `X-Hub-Signature-256` alongside it, carrying the identical value. Same secret, same algorithm, same bytes, so duplicating it grants no new capability to an attacker, and existing consumers of `X-Hindsight-Signature` keep working. Preferred over a per-webhook configurable header name: that would add a config field (plus migration, API, control plane, clients, CLI coverage, docs) to let users type the one string this already sends, and would leave every SDK verifier asking which header the sender was configured for. Two adjacent gaps found while in here: - The body-only signature has no notion of freshness, so a delivery captured off the wire stays verifiable forever. Add `X-Hindsight-Signature-V2` (`t=<unix>,v1=<hex>` over `<t>.<raw body>`, Stripe-style), signed at attempt time so retries re-sign. The timestamp is inside the MAC, so receivers can trust it and reject anything outside a tolerance window. The existing headers keep their body-only meaning — `X-Hub-Signature-256` is body-only by convention and must not be redefined. - `http_config.headers` was spread *after* `X-Hindsight-Event`, so a webhook's custom headers could overwrite the event type a receiver keys off. Spread user headers first and set the Hindsight-controlled headers after, so neither the event type nor any signature can be clobbered. Content-Type stays overridable (some receivers insist on a vendor media type; the body is JSON regardless). Document the whole header set with a verification example, which the webhooks page previously did not cover at all. The two unrelated `skills/hindsight-docs/` hunks are pre-existing generated drift from #3896, picked up by re-running generate-docs-skill.sh. Closes #3207	2026-09-01 12:44:36 +02:00
Nicolò Boschi	9452ac29da	feat(retain): report zero-fact documents at write time (#3040) (#3044) ... * feat(retain): report zero-fact documents at write time (#3040) A document whose fact extraction legitimately returns zero facts is stored but unreachable: only memory_units carry embeddings, so recall and reflect cannot reach a document that owns none. The retain still succeeds, the operation reports completed, and nothing in the response, the webhook or the metrics says the document produced no memories — the operator has no way to know it needs a reprocess. FAIL_ON_EXTRACTION_ERRORS (#2721) cannot help by construction: there is no error to fail on. #2861 made retain.completed fire for zero-fact batches, but the payload is byte-identical to a successful one, so it still carries no signal. Add the count to all three write-time surfaces: - retain.completed gains data.memory_unit_count, filled inside the outbox callback on the retain's own connection so units written by the enclosing transaction are visible. - The synchronous retain response gains memory_units_created. - New counter hindsight.retain.documents.total{outcome=facts|no_facts}, emitted per document at both extraction exits. The webhook and the metric report the document's total *after* the retain, not what the call created: the delta path skips unchanged chunks, so an idempotent re-retain creates zero units while the document keeps every memory it had. Reporting units created would raise a false alarm on every re-submit. The count query only runs when the call created nothing, which is the path where no work was done anyway. Docs: how a retain mission trades away retrieval of the raw source, the three signals, the non-determinism caveat, and reprocess as the way back. * fix(retain): drop memory_units_created from the retain response The synchronous response field reported units created by that call, which is a different number from the one the webhook and the metric report (the document's total after the retain) and only ever populated on the sync path. The async path is the one that matters, and it is already covered by retain.completed carrying data.memory_unit_count. Removing it also takes the API surface back to identical with main — the webhook payload is now the only public shape change — so the regenerated Python/TypeScript/Go clients and the OpenAPI spec carry no delta. Also renames the metric's parameter to memory_unit_count to match what it is actually handed: the document total, not units created.	2026-07-29 15:04:58 +02:00
Nicolò Boschi	a0e6bedcf1	refactor(memory-defense): per-bank regex defense, webhooks, drop dead surface (#2077) ... * Implement Memory Guard Lite for OSS Allow users to prevent token and secret leakage in agent memory. feat(memory-defense): reject quarantine action in policy parser refactor(retain): drop quarantine branch from orchestrator test(retain): remove quarantine-path tests refactor(memory-defense): remove DefenseAction.QUARANTINE enum value refactor(api): remove include_quarantined query parameter refactor(recall): drop include_quarantined parameter from memory engine test(memory-defense): replace stale parser-reject test with full-union accept test The previous parametrized test asserted parse_policy() should 422 on any detector name other than sensitive_data. That contract was deliberately widened on 2026-06-07 so cloud-style policies pass through api-slim's parser unchanged. The test was stale; the runtime is correct. Replaced with test_parse_policy_accepts_full_detector_union, which proves the actual contract: all 7 detector names are valid in the parser, with dispatch and entitlement enforcement deferred to the loaded extension. Memory defense UI * i18n labels * Fix tests * Client changes to fix breaking tests * Test fixes * chore: regenerate API clients via generate-clients.sh The clients were previously hand-generated in a way that diverged from the project's tooling — including a non-standard hindsight-clients/typescript/client/ directory the generator never produces (the standard output is typescript/generated/), plus ~150 spurious files. Revert the entire hindsight-clients/ tree to main and regenerate from the OpenAPI spec using ./scripts/generate-clients.sh (Rust via progenitor build.rs, Python/Go via openapi-generator, TypeScript via @hey-api/openapi-ts). The spec itself is unchanged (a code-regenerated spec is byte-identical to what was already committed). Net result is the real API delta only: the new nullable MemoryItem.receipt_uri field propagated to the Python, TypeScript and Go models. * refactor(memory-defense): per-bank regex defense, webhooks, drop dead surface Review cleanup of the memory-defense feature: - Rename the OSS extension Lite -> Regex (MemoryDefenseRegexExtension, memory_defense_regex.py). It is pure regex redaction now. - Drop the agent_memory_guard (OWASP) dependency entirely — the SensitiveDataDetector fallback and to_owasp_policy are gone; nothing cloud-tier remains in api-slim. - Trim the policy to what OSS enforces: { enabled, rules:[{on:sensitive_data, action}] }. Removed default_action, protected/immutable namespaces, detector_overrides, min_severity, and the unused memory_defense_enabled_default server default. Per-bank override stays (memory_defense is a configurable field) and the UI writes the trimmed shape. - Fire a memory_defense.triggered webhook on every non-allow decision (redact and block) when one is configured, via the retain orchestrator. Adds WebhookEventType.MEMORY_DEFENSE_TRIGGERED + MemoryDefenseEventData. Replaces the no-op record_violation hook. - Block is now actually enforced (drop item / 422 when all blocked) instead of being silently downgraded to redact. - Remove the unused 'status' lifecycle: the add_status migration + its two merge migrations, the recall quarantine filter, the status column reads in search, and MemoryFact.status. Branch now adds zero migrations (single head). - Remove receipt_uri from the API (MemoryItem) and clients — it was always None and carried no value. Tests updated/renamed accordingly; OWASP smoke + enabled-default tests removed. * fix(memory-defense): address code-review findings - Delete test_migration_status.py (asserted the removed status column/constraint). - Remove receipt_uri from the Rust CLI (memory.rs + integration_test.rs) — the generated client struct no longer has the field, so it wouldn't compile. - Type the blocked-violations as a BlockedViolation dataclass instead of raw dicts (serialized via asdict() in the 422 body); type the webhook helper's decision param as DefenseDecision. - Add an end-to-end test asserting a redact decision queues a memory_defense.triggered webhook delivery. - Drop the unrelated docs/ entry from .gitignore (local scratch, not this PR). * test(memory-defense): consolidate into a single test_memory_defense.py Merge the 10 scattered memory-defense test modules (policy parser, regex engine/screen, redaction benchmark, extension loader, extension-context wiring, bank-config validation, and the three retain e2e files) into one test_memory_defense.py, deduping the overlapping unit screen tests and the duplicated retain redact e2e. 36 tests, same coverage. * docs(memory-defense): document memory_defense.triggered webhook + block action - Add memory_defense.triggered to the control-plane webhook event-type selector (it was firing but wasn't selectable in the UI). - Document the memory_defense.triggered event (payload + data fields) on the webhooks API page, and link it from the Memory Defense page. - Document the block action (the page only described redact) and add a Notifications section. Regenerate the docs skill copies. * docs: remove Memory Defense page from version-0.7 (unreleased feature) The feature was snapshotted into the 0.7 versioned docs by mistake — 0.7 never shipped Memory Defense. Remove the page and its (sole) Security sidebar category. * test(memory-defense): assert webhook payload fields + cover block path - test_retain_fires_webhook_on_redact now parses the queued delivery and asserts the MemoryDefenseEventData payload (action/detector/matched_types/ message + event status), not just that the event type was queued. - Add test_retain_fires_webhook_on_block: a block decision fires the webhook (before the 422 is raised) with action=block. Confirms the delivery persists despite the blocked retain returning 422. - Factor out _memory_defense_webhook_events() helper. * fix(control-plane): render structured API error details as a string A blocked retain returns 422 with detail {violations: [{message, ...}]}; the proxy forwards it as `details` and the client passed that object straight into the sonner toast, crashing with "Objects are not valid as a React child". Add describeErrorDetails() to reduce details to a string — joining violation messages when present (so a Memory Defense block shows e.g. "Sensitive data pattern matched: aws_access_key"), else JSON-stringifying. * docs(webhooks): clarify WebhookEvent.status covers the memory_defense action * feat(memory-defense): record redact/block actions in the audit log Emit a fire-and-forget 'memory_defense' audit entry for each non-allow decision (alongside the webhook), with the action/detector/document_id/matched_types in metadata. Threads the engine's AuditLogger into retain_batch like the webhook manager; gated by the existing audit_log_enabled switch (off by default). - Add the memory_defense option to the audit-logs UI action filter + the actionMemoryDefense i18n key across all locales. - Document it on the Memory Defense page and the audit-logging config section. - Test: a redact retain writes a memory_defense audit row with the expected metadata (audit enabled on the test engine). --------- Co-authored-by: Chris Latimer <chris.latimer@vectorize.io>	2026-06-10 14:12:46 +02:00
Nicolò Boschi	abbf874d84	feat: webhook system with retain.completed event, UI, and docs (#487) ... * doc: update cookbook * fix(cookbook): preserve tag keys during sync, strip local .md links - Fix extract_tags_from_readme/notebook to return dict[str,str] preserving sdk/topic keys instead of bare values, preventing topics like "Customer Service" from being misclassified as SDK - Add strip_local_md_links() to remove relative .md references that would cause broken link errors in Docusaurus build * ci: run test-doc-examples independently without waiting for test-rust-cli Build the CLI directly in the job instead of downloading the artifact, so test-doc-examples can start at the beginning in parallel with all other jobs. * feat: webhook system with task-owned retry, retain.completed event, and UI - New webhook system: register per-bank webhooks with HMAC signing, configurable HTTP method/timeout/headers/params (http_config JSONB), and PATCH support - Webhook deliveries run as async_operations (webhook_delivery type) with task-owned retry via RetryTaskAt exception and exponential backoff (60s / 5m / 30m / 2h / 8h, max 6 attempts) - New retain.completed event fires per-document for both sync and async retain - Delivery debug info (status code, response body) stored in result_metadata - Control plane UI: webhooks tab per bank with create/edit/delete and a deliveries table with cursor pagination and expandable response details - 28 webhook tests covering HMAC signing, delivery retries, CRUD endpoints, PATCH update, and retain.completed queuing - Docs page at developer/api/webhooks documenting event payloads and delivery - OpenAPI spec and all client SDKs (Python, TypeScript, Rust, Go) regenerated * fix: update tests for task-owned retry model and guard _webhook_manager attribute - test_worker.py: test_executor_exception_triggers_retry now raises RetryTaskAt (plain exceptions are immediate failures in the new system); rename test_executor_exception_marks_failed_after_max_retries to test_executor_exception_marks_failed_immediately to reflect new semantics - test_batch_api.py: remove max_retries kwarg from WorkerPoller constructor - memory_engine.py: use getattr for _webhook_manager in _fire_retain_webhook to avoid AttributeError when engine is created without __init__ (tests) * fix: remove max_retries from benchmark WorkerPoller call * fix(webhooks): transactional outbox, observations_deleted tracking, sidebar - Queue webhook delivery rows atomically with the primary operation using the transactional outbox pattern — prevents lost events on process crash: - Retain (sync + async): outbox_callback passed into orchestrator.retain_batch and called inside the DB transaction, replacing the post-commit fire call - Consolidation: new _mark_operation_completed_and_fire_webhook combines the status UPDATE and webhook INSERT in one transaction - Added fire_event_with_conn() to WebhookManager for in-connection delivery - Track observations_deleted count in consolidation stats and expose it in the consolidation.completed webhook payload (was always None) - Add Webhooks page to docs sidebar - Document at-least-once delivery guarantee with operation_id dedup guidance * fix(ui): add retain.completed to available webhook event types * feat(ui): add delete confirmation dialog for webhooks * fix(webhooks): include operation_id in task_payload so delivery is marked completed The task_payload JSON was missing the operation_id field, causing execute_task to see operation_id=None and skip _mark_operation_completed — leaving every delivery row stuck in 'pending' forever. Added a test that inserts a real async_operations row and verifies the status transitions to 'completed' after a successful execute_task call. * style: fix prettier formatting in webhooks-view	2026-03-04 14:17:01 +01:00