2 Commits

Author	SHA1	Message	Date
efenex	e68d4ebe9b	fix(summarize): chunk large sessions to fit LLM context window (#472) ... * fix(summarize): chunk large sessions to fit LLM context window JSONL-imported sessions can have far more observations than the 500-cap MAX_OBS_PER_SESSION that constrains native sessions. mem::summarize previously built one prompt containing every observation and shipped it as a single LLM call, which exceeded the provider's context window for sessions >~7,000 observations and returned an unhelpful 400 from upstream — silently leaving large bulk-imported sessions out of the semantic tier. Approach: map-reduce inside mem::summarize. - Sessions ≤ SUMMARIZE_CHUNK_SIZE (default 400) take the legacy single-call path with no overhead - Larger sessions are split into chunks, each summarized with the existing per-session prompt in parallel batches of SUMMARIZE_CHUNK_CONCURRENCY (default 6), and partial summaries merged via a new REDUCE_SYSTEM prompt - Per-chunk retry-once on transient parse / provider errors - Persistently-failing chunks are skipped (not propagated) so a flaky chunk doesn't waste 30+ already-completed LLM calls on the same session - Bail with too_many_chunks_skipped only if >50% of chunks fail Companion operator tool: scripts/backfill-imported-sessions.sh walks jsonl-imported sessions and POSTs mem::summarize per session, with project / agent / obs-count filters, cost estimation, and per-failure payload dumping for debugging provider rejections. Validated locally against a real corpus: - 5,392-obs session (14 chunks, c=6): 39s - 10,704-obs session (27 chunks, c=6): 34s - 105,966-obs session (265 chunks, c=50): handler completes server-side and persists - 52-session bulk backfill → 25 new semantic facts + 6 new reflect insights produced by consolidate-pipeline Known limit: iii-engine has a hardcoded 180s function-invocation timeout. Sessions large enough that chunked summarize wallclock exceeds that will return a timeout/500 to the HTTP client even though the handler completes and persists server-side. High-RPM providers (Novita / DeepInfra / DeepSeek typically allow 100+ concurrent) can raise SUMMARIZE_CHUNK_CONCURRENCY to push the cliff well past any realistic session size. True fix is an async-job pattern; left as follow-up. - src/prompts/summary.ts: add REDUCE_SYSTEM + buildReducePrompt - src/functions/summarize.ts: chunking, retry, skip, parallelism - test/summarize.test.ts: 9 cases covering single-call path, chunking, env-override, retry-then-success, persistent skip, too-many-skipped bail, provider error after retry, concurrency - .env.example: document SUMMARIZE_CHUNK_SIZE / _CONCURRENCY - .gitignore: agentmemory-debug/, data-*/ (operator artefacts) - scripts/backfill-imported-sessions.sh: bulk-import backfill tool 9/9 new tests pass; existing tests untouched. * fix(summarize): address CodeRabbit review on #472 Four nits flagged by the automated reviewer, all worth fixing: - scripts/backfill: add curl --connect-timeout + --max-time profiles (META_CURL_OPTS vs WORK_CURL_OPTS). Metadata reads fail fast and retry on transient blips; LLM-backed work calls get a wide 30-min cap and no retry (retrying a half-finished LLM job double-spends). - scripts/backfill: sanitize sessionId before joining with DEBUG_DIR in dump_failure() (otherwise a session id containing `/` or `..` could escape the debug dir). UUIDs in practice, but the server doesn't enforce that. - scripts/backfill: switch the observations query to `--get --data-urlencode "sessionId=$id"` so special characters can't corrupt the query string. - scripts/backfill: guard `jq` on summarize + consolidate responses with `jq -e . </dev/null 2>&1` first. iii's HTTP layer occasionally returns non-JSON (HTML 5xx, empty body on timeout). Without the guard, `set -e` aborts the whole backfill loop on a single bad response — now it logs `invalid_json_response` and moves on. - test/summarize.test.ts: fix `vi.mock("./audit.js", ...)` path to `"../src/functions/audit.js"`. The old path resolved to `test/audit.js` (nonexistent), so the mock was a silent no-op. Tests passed anyway because `safeAudit` writes to a mocked KV. 9/9 tests still pass; backfill dry-run still resolves the corpus cleanly.	2026-05-19 19:29:06 +01:00
Rohit Ghumare	8b98432853	chore(release): v0.9.13 — env-example + init + CJK + load bench + deploy templates + Gemini GA bumps (#383) ... * chore(release): v0.9.13 — env-example discovery + CJK tokenizer + load harness + deploy templates + Gemini GA bumps + 14 advisories closed Six PRs landed since v0.9.12: - #372 .env.example discovery (this commit) — repo-root template + `init` CLI command + CI sync-checker - #362 CJK BM25 tokenizer (`@node-rs/jieba` + tiny-segmenter + Hangul) - #363 `benchmark/load-100k.ts` harness with p50/p90/p99 + per-release results dir - #361 one-click deploy templates for fly.io / Railway / Render / Coolify (multi-stage Dockerfile, `iiidev/iii` base, `gosu` privilege drop, first-boot HMAC, verified end-to-end on fly.io) - #364 Python ecosystem via `iii-sdk` example (replaces closed PR #360) - #370 Gemini GA bumps (LLM default → gemini-2.5-flash, embedding → gemini-embedding-001 + L2-norm + 768 dims) Plus 14 open Dependabot advisories closed in PR #348 via Next.js → 16.2.6 and PostCSS → 8.5.10 overrides. Bumped: - src/version.ts: VERSION 0.9.12 → 0.9.13 - package.json: 0.9.12 → 0.9.13, files += ".env.example", build script copies .env.example into dist/ - packages/mcp/package.json: 0.9.12 → 0.9.13 (lockstep with main) - plugin/.claude-plugin/plugin.json, plugin/.codex-plugin/plugin.json: 0.9.12 → 0.9.13 - src/types.ts: ExportData.version union extended with "0.9.13" - src/functions/export-import.ts: supportedVersions Set extended - test/export-import.test.ts: expected version updated New surface: - .env.example at repo root — every env var read by src/ documented in one place, grouped by surface (LLM, embedding, auth, search tuning, behaviour flags, CLI runtime, ports, iii engine pin, Claude Code bridge, Obsidian export). Every line commented out by default so the file is a template. - agentmemory init — copies bundled .env.example to ~/.agentmemory/.env if absent, refuses to overwrite, prints a diff command. Wired into CLI dispatch + help block. - scripts/check-env-example.mjs — walks src/ for env-read patterns, fails CI on drift in either direction. Plugged into ci.yml after npm test. Initial bootstrap: 60 keys in sync. Verified: npm test 903/903, npm run build clean, init smoke pass (creates ~/.agentmemory/.env on first run, refuses overwrite on second). * fix(init): atomic copy via COPYFILE_EXCL; address CodeRabbit review Two valid findings from the CodeRabbit pass on PR #383. 1. `runInit` race between existsSync(target) + copyFile(template, target). A parallel `agentmemory init` (or any other process touching ~/.agentmemory/.env between the two calls) would silently overwrite the config the operator just wrote. Switch to a single atomic `copyFile(template, target, fsConstants.COPYFILE_EXCL)` and treat the EEXIST error as the "already configured" signal — same warning + diff hint as before, but the check + copy now happen in one syscall so they cannot race. Other failure paths still surface as process exit 1. 2. Comment on `scripts/check-env-example.mjs::walk` claimed it matched ".ts / .mts / .mjs" but the regex also matched ".js". Rewrote the comment to match the regex (".ts / .mts / .mjs / .js"). Same comment pass: noted that test/ never enters because the walk is rooted at src/, not because of an explicit skip. Skipped findings: - WHAT-style comment on `findEnvExample` — kept a one-liner explaining the package-vs-source priority since both paths are real; reduced the block from 4 lines to 2 instead of removing it entirely. - "Add trailing newline to .env.example" — file already ends with `\n` (verified `tail -c 5` shows `tion\n`). Verified locally: - `npm run build` clean. - `npm test` 903 / 903 pass. - First `agentmemory init` against a clean HOME creates the file. - Second init against the same HOME hits EEXIST and prints the "leaving it untouched" warning + diff hint without overwriting. - `node scripts/check-env-example.mjs` — in sync (60 keys).	2026-05-15 10:11:37 +01:00