Abdullah Alaqeel
8c90741c63
* chore(deps): migrate @xenova/transformers to @huggingface/transformers v4 @xenova/transformers@2.x is deprecated and silently broken on Node 22+ (see #479). The project was renamed to @huggingface/transformers; same Apache-2.0 license, same code. v4 ships onnxruntime-node/web and sharp as hard deps, so they're dropped from our optionalDependencies. Pipeline / RawImage.fromBlob / tolist / text-classification output shape all unchanged. Three behavior-preserving adjustments needed: - All 4 pipeline call sites pass { dtype: "q8" }. v4's default on Node is fp32 (DEFAULT_DEVICE_DTYPE = "fp32"); v2 defaulted to quantized=true. Without explicit dtype, all 4 sites silently regress (~3.5x larger download, slower inference). dtype "q8" maps to model_quantized.onnx per v4's DEFAULT_DTYPE_SUFFIX_MAPPING; file exists in all 3 Xenova models. This was the regression that prompted the test additions below. - src/providers/embedding/local.ts: split import try/catch from pipeline() call so model-load errors (network, missing q8 variant, disk) propagate with their actual message, not masked as "Install @huggingface/transformers...". - src/providers/embedding/{local,clip}.ts: type module from typeof import("@huggingface/transformers") so PretrainedModelOptions flows through; drop hand-rolled aliases and @ts-ignore. Cast at assignment sites (pipeline return union isn't structurally assignable to our narrow FeatureExtractor / ClipPipeline shapes). Tests added where coverage was zero (would have caught the dtype regression): - test/local-embedding-provider.test.ts (3 tests): unavailable-path install hint; pipeline called with dtype:q8 + extractor options + mapped Float32Array result; embedBatch shape. - test/clip-embedding-provider.test.ts (5 tests): unavailable-path; text pipeline dtype:q8 + result; embedBatch; embedImage with data: URL decode; custom model ID propagation. - test/reranker.test.ts: positive-path using vi.doMock + resetModules. Other: - src/huggingface.d.ts deleted (package ships its own types). - src/xenova.d.ts removed. - src/providers/embedding/clip.ts: inline single-use DIMENSIONS constant. - tsdown.config.ts: trim neverBundle list and comment. - README.md L1267: BGE-small -> Xenova/all-MiniLM-L6-v2 (was always wrong). - 16 docs: install commands + prose mentions across main README, 11 translations, SECURITY.md, 2 benchmark docs, benchmark script. - Model IDs (Xenova/all-MiniLM-L6-v2, Xenova/clip-vit-base-patch32, Xenova/ms-marco-MiniLM-L-6-v2) kept — HF Hub repo names, still valid. Closes #1095. Fixes #479. Verified: 1424/1424 tests pass, build clean, tsc clean on migrated files. * test(embedding): add v4 smoke test, harden import errors, expand CI matrix Review follow-ups for #1096: - env-guarded non-mocked smoke test (RUN_HF_SMOKE=1) loading real Xenova/all-MiniLM-L6-v2, asserts 384 finite dims; skipped by default - selective ERR_MODULE_NOT_FOUND handling in local/clip providers so real init errors propagate (checks err.code and err.cause.code to handle vitest mock-factory wrapping) - CLIP install hint made embedding-agnostic (loader serves text + image) - afterEach mock cleanup in doMock-based provider/reranker tests - CI Node matrix: [20, 22] -> [20, 22, 24, 26] across ubuntu/macos * refactor(embedding): drop err.cause check, use manual mock for missing-module tests The .cause branch in the ERR_MODULE_NOT_FOUND check existed only to accommodate vitest's mock-factory wrapping, not a real Node loader behavior. Replace it with a manual mock fixture (__mocks__/@huggingface/ transformers.ts) that throws a Node-shaped error at module top-level, bypassing vitest's factory wrapper so the import rejects with err.code set directly. Production code now checks only err.code === 'ERR_MODULE_NOT_FOUND', matching real Node behavior. Tests verify the same public contract without coupling production code to the test framework.
benchmark/
Two kinds of numbers live in this directory:
-
Quality / retrieval —
longmemeval-bench.ts,quality-eval.ts,real-embeddings-eval.ts,scale-eval.ts. Recall, precision, token savings. Documented inLONGMEMEVAL.md,QUALITY.md,REAL-EMBEDDINGS.md,SCALE.md. -
Load shape —
load-100k.ts. p50 / p90 / p99 latency and throughput against a running daemon. This is the file you want when somebody asks "what's p99 at 100k memories under concurrency 100?".
load-100k.ts
Hand-rolled, dependency-free load harness. Issues real HTTP against a
local agentmemory daemon at http://localhost:3111, records per-request
latency with performance.now(), and writes a JSON report per run.
What it measures
For each cell in the matrix (N, concurrency, endpoint) it records:
p50_ms,p90_ms,p99_ms— nearest-rank percentiles.min_ms,max_ms,ops,errors.throughput_per_sec— wall-clock ops / sec for that cell.
Default matrix:
N∈ {1000, 10000, 100000} — number of memories seeded before the cell runs.C∈ {1, 10, 100} — concurrent in-flight requests during the cell.- Endpoints under test:
POST /agentmemory/rememberPOST /agentmemory/smart-searchGET /agentmemory/memories?latest=true
Each cell issues BENCH_OPS=200 requests by default — enough samples
for stable p99 without dragging a 100k-seed run past tens of minutes.
Why p99 is the number that matters
p50 tells you the median request feels fast. p90 tells you the bulk of requests feel fast. p99 tells you the request your tail user hits when they really need it feels fast. Capacity planning lives here — if you want to size a fleet, scale your daemon, or set an SLO, p99 is the number to plan against. p50 will lie to you.
Running it
# 1. Start the daemon however you normally do (npx, Docker, etc.)
npx @agentmemory/agentmemory
# 2. From the repo root, in another shell:
npm run bench:load
To override the matrix:
BENCH_N=1000 BENCH_C=1,10 BENCH_OPS=100 npm run bench:load
To have the harness spawn a daemon for the run (after npm run build):
AGENTMEMORY_BENCH_AUTOSTART=1 npm run bench:load
Other env knobs (see the file header for the canonical list):
AGENTMEMORY_URL— base URL of the daemon (defaulthttp://localhost:3111).BENCH_SEED— seed for themulberry32content RNG. Same seed + same daemon build = byte-identical seed corpus.BENCH_OUT_DIR— where the JSON report lands (defaultbenchmark/results/).
Where results land
benchmark/results/load-100k-<short-git-sha>.json. The harness
mkdir -ps the directory. The file has a schema_version: 1 field so
future format changes don't silently break consumers.
Content generation is seedable
Synthetic memory content is built from a small noun / verb / concept
vocabulary fed by a mulberry32(BENCH_SEED) PRNG. Same seed + same
build = same corpus. The point isn't "realistic" content (there isn't
one realistic content); the point is reproducibility — re-running
the harness against the same git sha should give the same content
mixture going in, so latency variance comes from the daemon and not
from JSON payload jitter.
Publishing numbers per release
The release flow appends a ## Performance section to CHANGELOG.md
referencing the JSON in benchmark/results/ for that release's git
sha. p99 is the headline number; the JSON is the receipt.