Commit Graph

7 Commits

Author SHA1 Message Date
Yingfeng
30739b7f8e Fix compilation workflow (#16819)
Dataset_nav can not support large number of documents, introducing AHC
clustering as well as retrieval engine as the clustering database
2026-07-10 21:11:19 +08:00
Kevin Hu
2c59d07bdb Feat: add wiki folder (#16749)
### Summary

Add wiki folders.
2026-07-08 20:08:14 +08:00
Mattie Schraeder
8a19c6aa5a Make RAPTOR GMM robust on small reduced clusters (#16632) 2026-07-06 21:09:35 +08:00
Mattie Schraeder
0c2fb622e9 Collapse small RAPTOR layers in one step instead of one node per layer (#16633) 2026-07-06 21:06:04 +08:00
Mattie Schraeder
0fcfb38365 Cap RAPTOR UMAP n_neighbors to prevent OOM on large datasets (#16627)
## Problem
`raptor.py` computes `n_neighbors = int((len(embeddings) - 1) ** 0.8)`
and
passes it to `umap.UMAP(...)`. In a dataset-scope RAPTOR build the first
layer's `embeddings` is the entire KB's chunk set, so this is
effectively
unbounded: ~93k chunks → n_neighbors ≈ 9,446.

UMAP's k-NN graph is `N × n_neighbors`; at these values the raw neighbor
arrays alone are ~14 GB (93k × 9446 × 16 B), and the symmetrized fuzzy
simplicial set + spectral init push peak well past 30 GB. The task
executor is OOM-killed inside `fit_transform` before any clustering runs
—
the log shows "Task has been received" with no "Cluster one layer" line
—
after which the unacked task re-queues and OOMs again in a loop.

The line above already flags this: `# Degrade too much ??`.

## Fix
Cap `n_neighbors` at 100. UMAP's neighborhood size has strongly
diminishing returns well below this (default 15; a few dozen already
captures global structure), so the ceiling preserves — likely improves —
cluster quality while bounding memory to O(N). Mirrors the existing
`n_components=min(12, len(embeddings) - 2)` clamp two lines down.

​```diff
-        n_neighbors = int((len(embeddings) - 1) ** 0.8)
+        n_neighbors = min(int((len(embeddings) - 1) ** 0.8), 100)
​```

## Repro
Dataset-scope RAPTOR over a KB with ~90k+ chunks on a box with <~64 GB
available: executor OOM-killed in the first-layer UMAP `fit_transform`.
With the cap, first-layer UMAP peaks in low single-digit GB and the
build
proceeds to completion.

## Scope
Only affects large dataset-scope builds; file-scope RAPTOR already had
n_neighbors well under 100. No behavior change beyond the ceiling.
2026-07-04 17:47:43 +08:00
Wang Qi
6a4b9be426 Refactor: reformat all code for lefthook using ruff and gofmt (#16585) 2026-07-03 12:53:39 +08:00
Kevin Hu
62f94cd59b Feat: Add knowledge compilation workflows (#16515)
## Summary
- Add knowledge compilation template APIs, services, and builtin
template seed data
- Add advanced knowledge compile structure/artifact/RAPTOR workflow
support
- Update parsing, dataset/document APIs, and supporting services for
compilation workflows
2026-07-02 23:22:07 +08:00