* ci: add pre-commit hook to keep skills/hindsight-docs in sync
The CI verify-generated-files job has been failing on ~82% of recent
runs because PRs touch hindsight-docs/src/pages/changelog/ or
hindsight-docs/static/openapi.json without re-running
./scripts/generate-docs-skill.sh, leaving the committed
skills/hindsight-docs/references/ copy stale.
Catch the drift locally instead. The hook regenerates and, if the
working tree diverges from the index after regen, fails the commit
with a clear message pointing the author at `git add skills/hindsight-docs/`.
The pre-commit dispatcher (.githooks/pre-commit) already iterates every
*.sh in scripts/hooks/, so the new file is picked up automatically.
* fix(retain): stop mutating caller-provided content dicts
PR #1398 (memory pressure) added an in-place pop of the "content" key
on contents_dicts after building combined_content, to release per-item
strings the engine no longer needs. Because the engine forwarded the
caller's dict objects all the way through (memory_engine →
_retain_batch_async_internal → orchestrator.retain_batch), the pop
reached back through the same references and stripped the key from
the caller's input. Any code path that holds onto the contents list
after retain_batch_async returns then trips KeyError: 'content'.
This is what was making test_extensions.py::TestOperationHooksParameters::
test_retain_pre_hook_receives_all_parameters fail intermittently on
main (the streaming path triggers the pop; non-streaming paths skip it).
Fix:
- memory_engine.py: take an engine-owned shallow copy of contents
after the validator hook so the orchestrator can mutate freely
without leaking to the caller. Strings are shared by reference,
so the copy adds only ~150 bytes of dict overhead per item —
negligible vs the multi-MB strings.
- orchestrator.py (_streaming_retain_batch): clear combined_content
immediately after handle_document_tracking / upsert_document_metadata
in all three first-batch paths (no-facts skip, mini-batch DB work,
post-loop fallback). Once tracking persists the document, nothing
reads combined_content again, so releasing it shrinks the lifetime
of the per-document text from "until function returns" to "until DB
write completes" — recovering the bulk of #1398's memory savings
without the caller-mutation side effect. nonlocal declarations on
_process_db_batch and _run_mini_batch_db_work are required because
Python infers combined_content as local once any branch assigns to it.
Memory profile vs PR #1398:
- #1398 benchmark shape (caller releases its reference at call time):
identical sustained, brief 2x peak during the combined_content +
per-item-strings overlap window before tracking completes. Other
PR #1398 savings (chunks, batch lists, sanitized_content) untouched.
- HTTP / FastAPI callers (request body holds strings until the handler
returns): no observable change — those strings were going to live
through the request anyway.
Nicolò Boschi