mirror of https://github.com/rohitg00/agentmemory.git synced 2026-09-14 20:16:33 +08:00

Files

T

Abdullah Alaqeel 8c90741c63 chore(deps): migrate @xenova/transformers to @huggingface/transformers v4 (#1096 )

* chore(deps): migrate @xenova/transformers to @huggingface/transformers v4

@xenova/transformers@2.x is deprecated and silently broken on Node 22+
(see #479). The project was renamed to @huggingface/transformers; same
Apache-2.0 license, same code. v4 ships onnxruntime-node/web and sharp
as hard deps, so they're dropped from our optionalDependencies.

Pipeline / RawImage.fromBlob / tolist / text-classification output
shape all unchanged. Three behavior-preserving adjustments needed:

- All 4 pipeline call sites pass { dtype: "q8" }. v4's default on Node
  is fp32 (DEFAULT_DEVICE_DTYPE = "fp32"); v2 defaulted to quantized=true.
  Without explicit dtype, all 4 sites silently regress (~3.5x larger
  download, slower inference). dtype "q8" maps to model_quantized.onnx
  per v4's DEFAULT_DTYPE_SUFFIX_MAPPING; file exists in all 3 Xenova
  models. This was the regression that prompted the test additions below.
- src/providers/embedding/local.ts: split import try/catch from
  pipeline() call so model-load errors (network, missing q8 variant,
  disk) propagate with their actual message, not masked as
  "Install @huggingface/transformers...".
- src/providers/embedding/{local,clip}.ts: type module from
  typeof import("@huggingface/transformers") so PretrainedModelOptions
  flows through; drop hand-rolled aliases and @ts-ignore. Cast at
  assignment sites (pipeline return union isn't structurally assignable
  to our narrow FeatureExtractor / ClipPipeline shapes).

Tests added where coverage was zero (would have caught the dtype
regression):

- test/local-embedding-provider.test.ts (3 tests): unavailable-path
  install hint; pipeline called with dtype:q8 + extractor options +
  mapped Float32Array result; embedBatch shape.
- test/clip-embedding-provider.test.ts (5 tests): unavailable-path;
  text pipeline dtype:q8 + result; embedBatch; embedImage with data:
  URL decode; custom model ID propagation.
- test/reranker.test.ts: positive-path using vi.doMock + resetModules.

Other:
- src/huggingface.d.ts deleted (package ships its own types).
- src/xenova.d.ts removed.
- src/providers/embedding/clip.ts: inline single-use DIMENSIONS constant.
- tsdown.config.ts: trim neverBundle list and comment.
- README.md L1267: BGE-small -> Xenova/all-MiniLM-L6-v2 (was always wrong).
- 16 docs: install commands + prose mentions across main README, 11
  translations, SECURITY.md, 2 benchmark docs, benchmark script.
- Model IDs (Xenova/all-MiniLM-L6-v2, Xenova/clip-vit-base-patch32,
  Xenova/ms-marco-MiniLM-L-6-v2) kept — HF Hub repo names, still valid.

Closes #1095. Fixes #479.

Verified: 1424/1424 tests pass, build clean, tsc clean on migrated files.

* test(embedding): add v4 smoke test, harden import errors, expand CI matrix

Review follow-ups for #1096:
- env-guarded non-mocked smoke test (RUN_HF_SMOKE=1) loading real
  Xenova/all-MiniLM-L6-v2, asserts 384 finite dims; skipped by default
- selective ERR_MODULE_NOT_FOUND handling in local/clip providers so real
  init errors propagate (checks err.code and err.cause.code to handle
  vitest mock-factory wrapping)
- CLIP install hint made embedding-agnostic (loader serves text + image)
- afterEach mock cleanup in doMock-based provider/reranker tests
- CI Node matrix: [20, 22] -> [20, 22, 24, 26] across ubuntu/macos

* refactor(embedding): drop err.cause check, use manual mock for missing-module tests

The .cause branch in the ERR_MODULE_NOT_FOUND check existed only to
accommodate vitest's mock-factory wrapping, not a real Node loader
behavior. Replace it with a manual mock fixture (__mocks__/@huggingface/
transformers.ts) that throws a Node-shaped error at module top-level,
bypassing vitest's factory wrapper so the import rejects with err.code
set directly.

Production code now checks only err.code === 'ERR_MODULE_NOT_FOUND',
matching real Node behavior. Tests verify the same public contract
without coupling production code to the test framework.

2026-07-29 10:21:54 +01:00

3.2 KiB

Raw Permalink Blame History

agentmemory v0.6.0 — Real Embeddings Quality Evaluation

Date: 2026-03-18T07:38:21.450Z Platform: darwin arm64, Node v20.20.0 Dataset: 240 observations, 30 sessions, 20 labeled queries Embedding model: Xenova/all-MiniLM-L6-v2 (384d, local, no API key)

Head-to-Head: Real Embeddings vs Keyword Search

System	Recall@5	Recall@10	Precision@5	NDCG@10	MRR	Avg Latency	Tokens/query
Built-in (grep all)	37.0%	55.8%	78.0%	80.3%	82.5%	0.44ms	19,462
BM25-only (stemmed+synonyms)	43.8%	55.9%	95.0%	82.7%	95.5%	0.26ms	1,571
Dual-stream (BM25+Xenova)	43.8%	64.1%	98.0%	94.9%	100.0%	2.39ms	1,571
Triple-stream (BM25+Xenova+Graph)	43.8%	64.1%	98.0%	94.9%	100.0%	2.07ms	1,571

Improvement from Real Embeddings

Adding real vector embeddings to BM25 improves recall@10 by 8.2 percentage points. Token savings vs loading everything: 92% (1,571 vs 19,462 tokens).

Per-Query: Where Real Embeddings Win

Queries where dual-stream (real embeddings) outperforms BM25-only:

Query	Category	BM25 Recall@10	+Vector Recall@10	Delta
How did we set up authentication?	semantic	25.0%	45.0%	+20.0pp **
Playwright test configuration	exact	50.0%	90.0%	+40.0pp **
database performance optimization	semantic	0.0%	40.0%	+40.0pp **
test infrastructure and factories	exact	50.0%	80.0%	+30.0pp **
Prisma ORM configuration	entity	14.3%	28.6%	+14.3pp **
CI/CD pipeline configuration	exact	20.0%	40.0%	+20.0pp **

By Category Comparison

Category	Built-in grep	BM25 (stemmed)	+Real Vectors	+Graph
exact	48.0%	54.0%	72.0%	72.0%
semantic	35.5%	33.3%	41.9%	41.9%
cross-session	77.8%	77.8%	77.8%	77.8%
entity	79.0%	76.2%	79.0%	79.0%

Embedding Performance

System	Embedding Time	Model	Dimensions
Dual-stream (BM25+Xenova)	3.1s	Xenova/all-MiniLM-L6-v2	384
Triple-stream (BM25+Xenova+Graph)	2.9s	Xenova/all-MiniLM-L6-v2	384

Embedding is a one-time cost at ingestion. Search is sub-millisecond after indexing.

Key Findings

Semantic queries improve most: 8.6pp recall@10 gain from real embeddings
"database performance optimization" — the hardest query — goes from BM25 0.0% to vector-augmented 40.0%
Entity/exact queries are already well-served by BM25+stemming — vectors add marginal value
Local embeddings (Xenova) run without API keys — zero cost, zero latency concerns

Recommendation

Enable local embeddings by default (EMBEDDING_PROVIDER=local or install @huggingface/transformers). This gives agentmemory genuine semantic search that built-in agent memories cannot match — understanding that "database performance optimization" relates to "N+1 query fix" and "eager loading".

All measurements use Xenova/all-MiniLM-L6-v2 local embeddings (384 dimensions, no API calls).

3.2 KiB Raw Permalink Blame History