Rohit Ghumare
d1517465ca
Adds a reproducible, dependency-free load harness so we can answer "what's p99 at 100k memories under concurrency 100?" with a number instead of a shrug. The harness seeds N synthetic memories against a local agentmemory daemon (defaults to http://localhost:3111, optional autostart via AGENTMEMORY_BENCH_AUTOSTART=1), then drives a matrix of (N, concurrency, endpoint) cells with hand-rolled Promise.allSettled batches. Per-request latency is collected via performance.now() and summarized as nearest-rank p50 / p90 / p99 plus min / max / errors and wall-clock throughput. Results are written to benchmark/results/load-100k-<short-git-sha>.json with a schema_version field so future format changes don't silently break consumers. Defaults match issue #346: N in {1000, 10000, 100000} x C in {1, 10, 100} x three endpoints (POST /agentmemory/remember, POST /agentmemory/smart-search, GET /agentmemory/memories?latest=true). Each cell issues BENCH_OPS=200 requests by default — enough samples for stable p99 without dragging a 100k-seed run past tens of minutes. Content is generated by a small noun/verb/concept vocabulary fed by a mulberry32(BENCH_SEED) PRNG so re-running the harness against the same daemon build yields the same seed corpus. Reproducibility, not realism, is the point — latency variance comes from the daemon, not JSON payload jitter. Files: - benchmark/load-100k.ts: main harness - benchmark/lib/percentiles.ts: zero-dep pXX helper, nearest-rank - benchmark/README.md: how to run, what gets measured, where results land, and why p99 is the number you want for capacity planning - benchmark/results/load-100k-96c0ed0.json: example result from a small-N (N=1000, C=10) verification run against a fresh daemon - package.json: wires `npm run bench:load` - CHANGELOG.md: Unreleased entry + a Performance section placeholder describing where per-release numbers should land going forward Verified locally at BENCH_N=1000 BENCH_C=10 BENCH_OPS=200 — three cells, zero errors, JSON written. Full 100k matrix is intentionally deferred to CI/release time. Closes #346.
benchmark/
Two kinds of numbers live in this directory:
-
Quality / retrieval —
longmemeval-bench.ts,quality-eval.ts,real-embeddings-eval.ts,scale-eval.ts. Recall, precision, token savings. Documented inLONGMEMEVAL.md,QUALITY.md,REAL-EMBEDDINGS.md,SCALE.md. -
Load shape —
load-100k.ts. p50 / p90 / p99 latency and throughput against a running daemon. This is the file you want when somebody asks "what's p99 at 100k memories under concurrency 100?".
load-100k.ts
Hand-rolled, dependency-free load harness. Issues real HTTP against a
local agentmemory daemon at http://localhost:3111, records per-request
latency with performance.now(), and writes a JSON report per run.
What it measures
For each cell in the matrix (N, concurrency, endpoint) it records:
p50_ms,p90_ms,p99_ms— nearest-rank percentiles.min_ms,max_ms,ops,errors.throughput_per_sec— wall-clock ops / sec for that cell.
Default matrix:
N∈ {1000, 10000, 100000} — number of memories seeded before the cell runs.C∈ {1, 10, 100} — concurrent in-flight requests during the cell.- Endpoints under test:
POST /agentmemory/rememberPOST /agentmemory/smart-searchGET /agentmemory/memories?latest=true
Each cell issues BENCH_OPS=200 requests by default — enough samples
for stable p99 without dragging a 100k-seed run past tens of minutes.
Why p99 is the number that matters
p50 tells you the median request feels fast. p90 tells you the bulk of requests feel fast. p99 tells you the request your tail user hits when they really need it feels fast. Capacity planning lives here — if you want to size a fleet, scale your daemon, or set an SLO, p99 is the number to plan against. p50 will lie to you.
Running it
# 1. Start the daemon however you normally do (npx, Docker, etc.)
npx @agentmemory/agentmemory
# 2. From the repo root, in another shell:
npm run bench:load
To override the matrix:
BENCH_N=1000 BENCH_C=1,10 BENCH_OPS=100 npm run bench:load
To have the harness spawn a daemon for the run (after npm run build):
AGENTMEMORY_BENCH_AUTOSTART=1 npm run bench:load
Other env knobs (see the file header for the canonical list):
AGENTMEMORY_URL— base URL of the daemon (defaulthttp://localhost:3111).BENCH_SEED— seed for themulberry32content RNG. Same seed + same daemon build = byte-identical seed corpus.BENCH_OUT_DIR— where the JSON report lands (defaultbenchmark/results/).
Where results land
benchmark/results/load-100k-<short-git-sha>.json. The harness
mkdir -ps the directory. The file has a schema_version: 1 field so
future format changes don't silently break consumers.
Content generation is seedable
Synthetic memory content is built from a small noun / verb / concept
vocabulary fed by a mulberry32(BENCH_SEED) PRNG. Same seed + same
build = same corpus. The point isn't "realistic" content (there isn't
one realistic content); the point is reproducibility — re-running
the harness against the same git sha should give the same content
mixture going in, so latency variance comes from the daemon and not
from JSON payload jitter.
Publishing numbers per release
The release flow appends a ## Performance section to CHANGELOG.md
referencing the JSON in benchmark/results/ for that release's git
sha. p99 is the headline number; the JSON is the receipt.