mirror of
https://github.com/mims-harvard/ToolUniverse.git
synced 2026-09-19 07:31:47 +08:00
Shanghua Gao
60ce9a8c3b
* Add generalizable datastore and euhealth tool (#21) * Add generalizable datastore and EUHealth tools * HF repo for euhealth tools and generalizable new tools points to agenticx and is public so everyone can download datasets there * added logic for how users can contribute personal tools to the public ToolUniverse for the community and upload it to the agenticx HF * moved workflow for euhealth into workflow folder * moved euhealth workflow into the workflow folder from .github general folder * import statements working * import change * updated all --local to --collection for CLI given more confusing with both. Always is a collection they are uploading or downloading. * typo * output into CLI with main now * made it correct so CLI specific commands are clear * clearer instruction * clearer instruction * cleaner comprehension * made deep tutorial clean for true, simply comprehension * made alternative (no JSON) option work * made alternative (no JSON) option work * Cleanup tutorials: remove quickstart, rename deepdive to make_your_data_searchable * added confirmed Copilot reviews --------- Co-authored-by: Reza Shamji <rezashamji@college.harvard.edu> Co-authored-by: rezashamji <112912895+rezashamji@users.noreply.github.com> * move test pos * Make datastore & EUHealth plug-and-play: cache-dir defaults, auto-dim, personal HF sync, user-first docs (#27) * Move generic_embedding_tool.json example to docs/tools/ (for tutorial reference) * added detail to point to example tool in JSON form when user creating own tool from JSON rather than python file * added required field (in this case nothing required) * Added path to json example * refactor: move datastore defaults to user cache dir (~/.cache/tooluniverse/embeddings) * Refactor datastore CLI + HF sync: - Auto-detect embedding dimensions (remove --dim flag) - Default HF uploads to user's own namespace via HF_TOKEN - Integrate unified download_from_hf helper - Add overwrite support for FAISS rebuild - Fix imports and minor UX/log improvements * made changes to tutorials for make_your_data_searchable and euhealth_tools post changes * updated euhealth_tools rst to have correct cache * updated make_your_data_searchable.rst to have correct cache * made directions for .env more clear * auto created cache directory * made sure overwrite works * updates * debugging why faiss not outputting in cache embeddings folder * feat(datastore): unify cache directory via get_user_cache_dir and simplify CLI defaults - Removed hardcoded ~/.cache/tooluniverse paths from docs and code - Made --db optional; defaults to get_user_cache_dir()/embeddings/<collection>.db - Added --overwrite support to quickbuild - Updated RST docs to remove <user_cache_dir> confusion and reflect automatic path handling * made cli more user friendly and less required arguments * made it more clear with updated cli.py * Refine datastore and EUHealth documentation: - Major overhaul of 'make_your_data_searchable.rst' for clarity and usability - Added clear Tool → Agent → ToolUniverse model and 3 integration paths - Unified HF sync, caching, and reproducibility instructions - Updated EUHealth docs for consistency with new datastore flow - Verified examples for CLI, Python, and agent-level usage * made rst more cohesive * rename make_your_data_searchable.rst → build_search_and_share_datastores.rst for clarity * removed test_sync_hf.py as relied on folder which would alter the flow * updated comprehensive tutorial of datastore addition * got rid of hf note given test_sync_hf.py was deleted * made cli and syncing to HF cleaner * made directions more clear and also made sure EmbeddingCollection tool was registered, and that custom tool naming actually says the tool name that is registered rather than the tool class in some cases * made your username instead of 'username' more clear * cleaner * forced trial * made it clear that euhealth exists at agenticx HF as public datastore * undid yaml change * Improved EUHealth tool behavior, embedding fallback logic, and Codex integration (#38) * Move generic_embedding_tool.json example to docs/tools/ (for tutorial reference) * added detail to point to example tool in JSON form when user creating own tool from JSON rather than python file * added required field (in this case nothing required) * Added path to json example * refactor: move datastore defaults to user cache dir (~/.cache/tooluniverse/embeddings) * Refactor datastore CLI + HF sync: - Auto-detect embedding dimensions (remove --dim flag) - Default HF uploads to user's own namespace via HF_TOKEN - Integrate unified download_from_hf helper - Add overwrite support for FAISS rebuild - Fix imports and minor UX/log improvements * made changes to tutorials for make_your_data_searchable and euhealth_tools post changes * updated euhealth_tools rst to have correct cache * updated make_your_data_searchable.rst to have correct cache * made directions for .env more clear * auto created cache directory * made sure overwrite works * updates * debugging why faiss not outputting in cache embeddings folder * feat(datastore): unify cache directory via get_user_cache_dir and simplify CLI defaults - Removed hardcoded ~/.cache/tooluniverse paths from docs and code - Made --db optional; defaults to get_user_cache_dir()/embeddings/<collection>.db - Added --overwrite support to quickbuild - Updated RST docs to remove <user_cache_dir> confusion and reflect automatic path handling * made cli more user friendly and less required arguments * made it more clear with updated cli.py * Refine datastore and EUHealth documentation: - Major overhaul of 'make_your_data_searchable.rst' for clarity and usability - Added clear Tool → Agent → ToolUniverse model and 3 integration paths - Unified HF sync, caching, and reproducibility instructions - Updated EUHealth docs for consistency with new datastore flow - Verified examples for CLI, Python, and agent-level usage * made rst more cohesive * rename make_your_data_searchable.rst → build_search_and_share_datastores.rst for clarity * removed test_sync_hf.py as relied on folder which would alter the flow * updated comprehensive tutorial of datastore addition * got rid of hf note given test_sync_hf.py was deleted * made cli and syncing to HF cleaner * made directions more clear and also made sure EmbeddingCollection tool was registered, and that custom tool naming actually says the tool name that is registered rather than the tool class in some cases * made your username instead of 'username' more clear * cleaner * forced trial * made it clear that euhealth exists at agenticx HF as public datastore * undid yaml change * made tutorial for user made searchable datastore with agents more clear. Removed docs/tutorials/build_search_and_share_datastores.rst and replaced with docs/tutorials/make_your_data_agent_searchable * removed euhealth refresh here given too much cost, will update if on our own schedule and add auto-refresh to another PR * altered language to point to new md * moved example JSON for user created tool to examples/make_your_data_agent_searchable_example/make_your_data_agent_searchable_example_JSON.json * got rid of duplicate imports * added test_examples * Restore embedding_tools.rst from main * removed logic to make naming convention of what the tool names are, from this PR and put into another branch, euhealth-refresh-and-other-additions * removed logic register EmbeddingCollectionSearchTool in the tool_registry, from this PR and put into another branch, euhealth-refresh-and-other-additions * skipped pytests that use api or special imports * chore: update pre-commit hooks and apply auto-fixes (black, autoflake, trailing spaces) * made it more clear : * changed tools_runtime.py to work with a user that both has azure model and doesn't use embeddings for search when downloading from online as well as allows them to make their own euhealth db and faiss with their own embeddings * updated it so it still keeps docs without themes * updated euhealth_tools.rst to include explanation of official build need for azure and text embedding small 3 or how to use own models * made it so codex can understand when a user asks for embedding, keyword, or hybrid search, and if there is no env it auto does keyword even if embedding/hybrid asked for --------- Co-authored-by: Reza Shamji <rezashamji@college.harvard.edu> Co-authored-by: rezashamji <112912895+rezashamji@users.noreply.github.com> Co-authored-by: rezashamji <rezamshamji@gmail.com>