diff --git a/docs/administrator/admin/admin_service.md b/docs/administrator/admin/admin_service.md index 7ec27c390b..e4b4df1943 100644 --- a/docs/administrator/admin/admin_service.md +++ b/docs/administrator/admin/admin_service.md @@ -27,7 +27,7 @@ With its unified interface design, the Admin Service combines the convenience of python admin/server/admin_server.py ``` - The service will start and listen for incoming connections from the CLI on the configured port. + The service will start and listen for incoming connections from the CLI on the configured port. ### Using Docker Image diff --git a/docs/administrator/configurations/config_ssl_cert.md b/docs/administrator/configurations/config_ssl_cert.md index 48a146af44..707381e07e 100644 --- a/docs/administrator/configurations/config_ssl_cert.md +++ b/docs/administrator/configurations/config_ssl_cert.md @@ -102,4 +102,4 @@ docker restart docker-ragflow-cpu-1 :::tip IMPORTANT Changes made via `docker cp` and `docker exec` are lost if the container is removed or stopped via `docker-compose down`. **Recommendation**: After a successful test, store the certificates on the host machine and use `volumes` in your `docker-compose.yaml` to mount the certificates and `ragflow.conf` permanently. -::: \ No newline at end of file +::: diff --git a/docs/administrator/tracing.mdx b/docs/administrator/tracing.mdx index da80c5c0e8..7d869cd946 100644 --- a/docs/administrator/tracing.mdx +++ b/docs/administrator/tracing.mdx @@ -17,10 +17,10 @@ This document is contributed by our community contributor [jannikmaierhoefer](ht RAGFlow ships with a built-in [Langfuse](https://langfuse.com) integration so that you can **inspect and debug every retrieval and generation step** of your RAG pipelines in near real-time. -Langfuse stores traces, spans and prompt payloads in a purpose-built observability backend and offers filtering and visualisations on top. +Langfuse stores traces, spans and prompt payloads in a purpose-built observability backend and offers filtering and visualisations on top. :::info NOTE -• RAGFlow **≥ 0.18.0** (contains the Langfuse connector) +• RAGFlow **≥ 0.18.0** (contains the Langfuse connector) • A Langfuse workspace (cloud or self-hosted) with a _Project Public Key_ and _Secret Key_ ::: @@ -28,9 +28,9 @@ Langfuse stores traces, spans and prompt payloads in a purpose-built observabili ## 1. Collect Your Langfuse Credentials -1. Sign in to your Langfuse dashboard. -2. Open **Settings ▸ Projects** and either create a new project or select an existing one. -3. Copy the **Public Key** and **Secret Key**. +1. Sign in to your Langfuse dashboard. +2. Open **Settings ▸ Projects** and either create a new project or select an existing one. +3. Copy the **Public Key** and **Secret Key**. 4. Note the Langfuse **host** (e.g. `https://cloud.langfuse.com`). Use the base URL of your own installation if you self-host. > The keys are _project-scoped_: one pair of keys is enough for all environments that should write into the same project. @@ -41,10 +41,10 @@ Langfuse stores traces, spans and prompt payloads in a purpose-built observabili RAGFlow stores the credentials _per tenant_. You can configure them either via the web UI or the HTTP API. -1. Log in to RAGFlow and click your avatar in the top-right corner. -2. Select **API ▸ Scroll down to the bottom ▸ Langfuse Configuration**. -3. Fill in you Langfuse **Host**, **Public Key** and **Secret Key**. -4. Click **Save**. +1. Log in to RAGFlow and click your avatar in the top-right corner. +2. Select **API ▸ Scroll down to the bottom ▸ Langfuse Configuration**. +3. Fill in you Langfuse **Host**, **Public Key** and **Secret Key**. +4. Click **Save**. Once saved, RAGFlow starts emitting traces automatically – no code change required. @@ -53,14 +53,14 @@ Once saved, RAGFlow starts emitting traces automatically – no code change requ ## 3. Run a Pipeline and Watch the Traces -1. Execute any chat or retrieval pipeline in RAGFlow (e.g. the Quickstart demo). -2. Open your Langfuse project ▸ **Traces**. +1. Execute any chat or retrieval pipeline in RAGFlow (e.g. the Quickstart demo). +2. Open your Langfuse project ▸ **Traces**. 3. Filter by **name ~ `ragflow-*`** (RAGFlow prefixes each trace with `ragflow-`). For every user request you will see: -• a **trace** representing the overall request -• **spans** for retrieval, ranking and generation steps +• a **trace** representing the overall request +• **spans** for retrieval, ranking and generation steps • the complete **prompts**, **retrieved documents** and **LLM responses** as metadata diff --git a/docs/basics/rag.md b/docs/basics/rag.md index 1d6e6dabad..45b3280fb4 100644 --- a/docs/basics/rag.md +++ b/docs/basics/rag.md @@ -86,22 +86,22 @@ They are highly consistent at the technical base (e.g., vector retrieval, keywor RAG has demonstrated clear value in several typical scenarios: -1. Enterprise Knowledge Q&A and Internal Search +1. Enterprise Knowledge Q&A and Internal Search By vectorizing corporate private data and combining it with an LLM, RAG can directly return natural language answers based on authoritative sources, rather than document lists. While meeting intelligent Q&A needs, it inherently aligns with corporate requirements for data security, access control, and compliance. -2. Complex Document Understanding and Professional Q&A +2. Complex Document Understanding and Professional Q&A For structurally complex documents like contracts and regulations, the value of RAG lies in its ability to generate accurate, verifiable answers while maintaining context integrity. Its system accuracy largely depends on text chunking and semantic understanding strategies. -3. Dynamic Knowledge Fusion and Decision Support +3. Dynamic Knowledge Fusion and Decision Support In business scenarios requiring the synthesis of information from multiple sources, RAG evolves into a knowledge orchestration and reasoning support system for business decisions. Through a multi-path recall mechanism, it fuses knowledge from different systems and formats, maintaining factual consistency and logical controllability during the generation phase. ## The Future of RAG The evolution of RAG is unfolding along several clear paths: -1. RAG as the data foundation for Agents +1. RAG as the data foundation for Agents RAG and agents have an architecture vs. scenario relationship. For agents to achieve autonomous and reliable decision-making and execution, they must rely on accurate and timely knowledge. RAG provides them with a standardized capability to access private domain knowledge and is an inevitable choice for building knowledge-aware agents. -2. Advanced RAG: Using LLMs to optimize retrieval itself +2. Advanced RAG: Using LLMs to optimize retrieval itself The core feature of next-generation RAG is fully utilizing the reasoning capabilities of LLMs to optimize the retrieval process, such as rewriting queries, summarizing or fusing results, or implementing intelligent routing. Empowering every aspect of retrieval with LLMs is key to breaking through current performance bottlenecks. -3. Towards context engineering 2.0 +3. Towards context engineering 2.0 Current RAG can be viewed as Context Engineering 1.0, whose core is assembling static knowledge context for single Q&A tasks. The forthcoming Context Engineering 2.0 will extend with RAG technology at its core, becoming a system that automatically and dynamically assembles comprehensive context for agents. The context fused by this system will come not only from documents but also include interaction memory, available tools/skills, and real-time environmental information. This marks the transition of agent development from a "handicraft workshop" model to the industrial starting point of automated context engineering. The essence of RAG is to build a dedicated, efficient, and trustworthy external data interface for large language models; its core is Retrieval, not Generation. Starting from the practical need to solve private data access, its technical depth is reflected in the optimization of retrieval for complex unstructured data. With its deep integration into agent architectures and its development towards automated context engineering, RAG is evolving from a technology that improves Q&A quality into the core infrastructure for building the next generation of trustworthy, controllable, and scalable intelligent applications. diff --git a/docs/develop/contributing.md b/docs/develop/contributing.md index afaaafc706..4370c697f7 100644 --- a/docs/develop/contributing.md +++ b/docs/develop/contributing.md @@ -34,7 +34,7 @@ The list below mentions some contributions you can make, but it is not a complet 1. Fork our GitHub repository. 2. Clone your fork to your local machine: `git clone git@github.com:/ragflow.git` -3. Create a local branch: +3. Create a local branch: `git checkout -b my-branch` 4. Provide sufficient information in your commit message `git commit -m 'Provide sufficient info in your commit message'` @@ -56,4 +56,4 @@ The list below mentions some contributions you can make, but it is not a complet ### Reviewing & Merging a PR -Ensure that your PR passes all Continuous Integration (CI) tests before merging it. \ No newline at end of file +Ensure that your PR passes all Continuous Integration (CI) tests before merging it. diff --git a/docs/develop/launch_ragflow_from_source.md b/docs/develop/launch_ragflow_from_source.md index c0019a902e..7d1d8bf5d6 100644 --- a/docs/develop/launch_ragflow_from_source.md +++ b/docs/develop/launch_ragflow_from_source.md @@ -40,7 +40,7 @@ cd ragflow/ ### Install Python Dependencies 1. Install uv: - + ```bash pipx install uv ``` @@ -98,7 +98,7 @@ docker compose -f docker/docker-compose-base.yml up -d ``` 4. Check the configuration in **conf/service_conf.yaml**, ensuring all hosts and ports are correctly set. - + 5. Run the **entrypoint.sh** script to launch the backend service: ```shell @@ -127,10 +127,10 @@ docker compose -f docker/docker-compose-base.yml up -d 3. Start up the RAGFlow frontend service: ```bash - npm run dev + npm run dev ``` - *The following message appears, showing the IP address and port number of your frontend service:* + *The following message appears, showing the IP address and port number of your frontend service:* ![](https://github.com/user-attachments/assets/0daf462c-a24d-4496-a66f-92533534e187) diff --git a/docs/develop/mcp/launch_mcp_server.md b/docs/develop/mcp/launch_mcp_server.md index eada002861..ef073f44dc 100644 --- a/docs/develop/mcp/launch_mcp_server.md +++ b/docs/develop/mcp/launch_mcp_server.md @@ -13,13 +13,13 @@ Launch an MCP server from source or via Docker. --- -A RAGFlow Model Context Protocol (MCP) server is designed as an independent component to complement the RAGFlow server. Note that an MCP server must operate alongside a properly functioning RAGFlow server. +A RAGFlow Model Context Protocol (MCP) server is designed as an independent component to complement the RAGFlow server. Note that an MCP server must operate alongside a properly functioning RAGFlow server. -An MCP server can start up in either self-host mode (default) or host mode: +An MCP server can start up in either self-host mode (default) or host mode: -- **Self-host mode**: +- **Self-host mode**: When launching an MCP server in self-host mode, you must provide an API key to authenticate the MCP server with the RAGFlow server. In this mode, the MCP server can access *only* the datasets of a specified tenant on the RAGFlow server. -- **Host mode**: +- **Host mode**: In host mode, each MCP client can access their own datasets on the RAGFlow server. However, each client request must include a valid API key to authenticate the client with the RAGFlow server. Once a connection is established, an MCP server communicates with its client in MCP HTTP+SSE (Server-Sent Events) mode, unidirectionally pushing responses from the RAGFlow server to its client in real time. @@ -35,7 +35,7 @@ If you wish to try out our MCP server without upgrading RAGFlow, community contr ## Launch an MCP Server -You can start an MCP server either from source code or via Docker. +You can start an MCP server either from source code or via Docker. ### Launch from Source Code @@ -52,7 +52,7 @@ uv run mcp/server/server.py --host=127.0.0.1 --port=9382 --base-url=http://127.0 # uv run mcp/server/server.py --host=127.0.0.1 --port=9382 --base-url=http://127.0.0.1:9380 --mode=host ``` -Where: +Where: - `host`: The MCP server's host address. - `port`: The MCP server's listening port. @@ -98,7 +98,7 @@ The MCP server is designed as an optional component that complements the RAGFlow # - --no-json-response # Disables JSON responses for the streamable-HTTP transport ``` -Where: +Where: - `mcp-host`: The MCP server's host address. - `mcp-port`: The MCP server's listening port. @@ -123,13 +123,13 @@ Run `docker compose -f docker-compose.yml up` to launch the RAGFlow server toget docker-ragflow-cpu-1 | Starting MCP Server on 0.0.0.0:9382 with base URL http://127.0.0.1:9380... docker-ragflow-cpu-1 | Starting 1 task executor(s) on host 'dd0b5e07e76f'... docker-ragflow-cpu-1 | 2025-04-18 15:41:18,816 INFO 27 ragflow_server log path: /ragflow/logs/ragflow_server.log, log levels: {'peewee': 'WARNING', 'pdfminer': 'WARNING', 'root': 'INFO'} - docker-ragflow-cpu-1 | + docker-ragflow-cpu-1 | docker-ragflow-cpu-1 | __ __ ____ ____ ____ _____ ______ _______ ____ docker-ragflow-cpu-1 | | \/ |/ ___| _ \ / ___|| ____| _ \ \ / / ____| _ \ docker-ragflow-cpu-1 | | |\/| | | | |_) | \___ \| _| | |_) \ \ / /| _| | |_) | docker-ragflow-cpu-1 | | | | | |___| __/ ___) | |___| _ < \ V / | |___| _ < docker-ragflow-cpu-1 | |_| |_|\____|_| |____/|_____|_| \_\ \_/ |_____|_| \_\ - docker-ragflow-cpu-1 | + docker-ragflow-cpu-1 | docker-ragflow-cpu-1 | MCP launch mode: self-host docker-ragflow-cpu-1 | MCP host: 0.0.0.0 docker-ragflow-cpu-1 | MCP port: 9382 @@ -142,13 +142,13 @@ Run `docker compose -f docker-compose.yml up` to launch the RAGFlow server toget docker-ragflow-cpu-1 | 2025-04-18 15:41:23,263 INFO 27 init database on cluster mode successfully docker-ragflow-cpu-1 | 2025-04-18 15:41:25,318 INFO 27 load_model /ragflow/rag/res/deepdoc/det.onnx uses CPU docker-ragflow-cpu-1 | 2025-04-18 15:41:25,367 INFO 27 load_model /ragflow/rag/res/deepdoc/rec.onnx uses CPU - docker-ragflow-cpu-1 | ____ ___ ______ ______ __ + docker-ragflow-cpu-1 | ____ ___ ______ ______ __ docker-ragflow-cpu-1 | / __ \ / | / ____// ____// /____ _ __ docker-ragflow-cpu-1 | / /_/ // /| | / / __ / /_ / // __ \| | /| / / - docker-ragflow-cpu-1 | / _, _// ___ |/ /_/ // __/ / // /_/ /| |/ |/ / - docker-ragflow-cpu-1 | /_/ |_|/_/ |_|\____//_/ /_/ \____/ |__/|__/ - docker-ragflow-cpu-1 | - docker-ragflow-cpu-1 | + docker-ragflow-cpu-1 | / _, _// ___ |/ /_/ // __/ / // /_/ /| |/ |/ / + docker-ragflow-cpu-1 | /_/ |_|/_/ |_|\____//_/ /_/ \____/ |__/|__/ + docker-ragflow-cpu-1 | + docker-ragflow-cpu-1 | docker-ragflow-cpu-1 | 2025-04-18 15:41:29,088 INFO 27 RAGFlow version: v0.18.0-285-gb2c299fa full docker-ragflow-cpu-1 | 2025-04-18 15:41:29,088 INFO 27 project base: /ragflow docker-ragflow-cpu-1 | 2025-04-18 15:41:29,088 INFO 27 Current configs, from /ragflow/conf/service_conf.yaml: @@ -157,12 +157,12 @@ Run `docker compose -f docker-compose.yml up` to launch the RAGFlow server toget docker-ragflow-cpu-1 | * Running on all addresses (0.0.0.0) docker-ragflow-cpu-1 | * Running on http://127.0.0.1:9380 docker-ragflow-cpu-1 | * Running on http://172.19.0.6:9380 - docker-ragflow-cpu-1 | ______ __ ______ __ + docker-ragflow-cpu-1 | ______ __ ______ __ docker-ragflow-cpu-1 | /_ __/___ ______/ /__ / ____/ _____ _______ __/ /_____ _____ docker-ragflow-cpu-1 | / / / __ `/ ___/ //_/ / __/ | |/_/ _ \/ ___/ / / / __/ __ \/ ___/ - docker-ragflow-cpu-1 | / / / /_/ (__ ) ,< / /____> -``` \ No newline at end of file +``` diff --git a/docs/guides/chat/set_chat_variables.md b/docs/guides/chat/set_chat_variables.md index 22ab55508d..193fc37abd 100644 --- a/docs/guides/chat/set_chat_variables.md +++ b/docs/guides/chat/set_chat_variables.md @@ -101,7 +101,7 @@ from ragflow_sdk import RAGFlow rag_object = RAGFlow(api_key="", base_url="http://:9380") assistant = rag_object.list_chats(name="Miss R") assistant = assistant[0] -session = assistant.create_session() +session = assistant.create_session() print("\n==================== Miss R =====================\n") print("Hello. What can I do for you?") @@ -109,9 +109,9 @@ print("Hello. What can I do for you?") while True: question = input("\n==================== User =====================\n> ") style = input("Please enter your preferred style (e.g., formal, informal, hilarious): ") - + print("\n==================== Miss R =====================\n") - + cont = "" for ans in session.ask(question, stream=True, style=style): print(ans.content[len(cont):], end='', flush=True) diff --git a/docs/guides/chat/start_chat.md b/docs/guides/chat/start_chat.md index 02b51c0f7f..ebcfc26b99 100644 --- a/docs/guides/chat/start_chat.md +++ b/docs/guides/chat/start_chat.md @@ -46,8 +46,8 @@ You start an AI conversation by creating an assistant. - **Rerank model** sets the reranker model to use. It is left empty by default. - If **Rerank model** is left empty, the hybrid score system uses keyword similarity and vector similarity, and the default weight assigned to the vector similarity component is 1-0.7=0.3. - If **Rerank model** is selected, the hybrid score system uses keyword similarity and reranker score, and the default weight assigned to the reranker score is 1-0.7=0.3. - - [Cross-language search](../../references/glossary.mdx#cross-language-search): Optional - Select one or more target languages from the dropdown menu. The system’s default chat model will then translate your query into the selected target language(s). This translation ensures accurate semantic matching across languages, allowing you to retrieve relevant results regardless of language differences. + - [Cross-language search](../../references/glossary.mdx#cross-language-search): Optional + Select one or more target languages from the dropdown menu. The system’s default chat model will then translate your query into the selected target language(s). This translation ensures accurate semantic matching across languages, allowing you to retrieve relevant results regardless of language differences. - When selecting target languages, please ensure that these languages are present in the dataset to guarantee an effective search. - If no target language is selected, the system will search only in the language of your query, which may cause relevant information in other languages to be missed. - **Variable** refers to the variables (keys) to be used in the system prompt. `{knowledge}` is a reserved variable. Click **Add** to add more variables for the system prompt. @@ -59,23 +59,23 @@ You start an AI conversation by creating an assistant. 4. Update Model-specific Settings: - In **Model**: you select the chat model. Though you have selected the default chat model in **System Model Settings**, RAGFlow allows you to choose an alternative chat model for your dialogue. - - **Creativity**: A shortcut to **Temperature**, **Top P**, **Presence penalty**, and **Frequency penalty** settings, indicating the freedom level of the model. From **Improvise**, **Precise**, to **Balance**, each preset configuration corresponds to a unique combination of **Temperature**, **Top P**, **Presence penalty**, and **Frequency penalty**. + - **Creativity**: A shortcut to **Temperature**, **Top P**, **Presence penalty**, and **Frequency penalty** settings, indicating the freedom level of the model. From **Improvise**, **Precise**, to **Balance**, each preset configuration corresponds to a unique combination of **Temperature**, **Top P**, **Presence penalty**, and **Frequency penalty**. This parameter has three options: - **Improvise**: Produces more creative responses. - **Precise**: (Default) Produces more conservative responses. - **Balance**: A middle ground between **Improvise** and **Precise**. - - **Temperature**: The randomness level of the model's output. + - **Temperature**: The randomness level of the model's output. Defaults to 0.1. - Lower values lead to more deterministic and predictable outputs. - Higher values lead to more creative and varied outputs. - A temperature of zero results in the same output for the same prompt. - - **Top P**: Nucleus sampling. + - **Top P**: Nucleus sampling. - Reduces the likelihood of generating repetitive or unnatural text by setting a threshold *P* and restricting the sampling to tokens with a cumulative probability exceeding *P*. - Defaults to 0.3. - - **Presence penalty**: Encourages the model to include a more diverse range of tokens in the response. + - **Presence penalty**: Encourages the model to include a more diverse range of tokens in the response. - A higher **presence penalty** value results in the model being more likely to generate tokens not yet been included in the generated text. - Defaults to 0.4. - - **Frequency penalty**: Discourages the model from repeating the same words or phrases too frequently in the generated text. + - **Frequency penalty**: Discourages the model from repeating the same words or phrases too frequently in the generated text. - A higher **frequency penalty** value results in the model being more conservative in its use of repeated tokens. - Defaults to 0.7. diff --git a/docs/guides/dataset/add_data_source/add_bitbucket.md b/docs/guides/dataset/add_data_source/add_bitbucket.md index 15f868e20d..0acd81e6cd 100644 --- a/docs/guides/dataset/add_data_source/add_bitbucket.md +++ b/docs/guides/dataset/add_data_source/add_bitbucket.md @@ -50,4 +50,4 @@ Credentials alone do not trigger indexing. You must link the data source to a sp 1. Navigate to the **Dataset** tab. 2. Select or create the target Dataset. 3. Navigate to the Dataset's **Configuration** page and select **Link data source**. -4. Choose the previously created Bitbucket connector in the popup window. \ No newline at end of file +4. Choose the previously created Bitbucket connector in the popup window. diff --git a/docs/guides/dataset/add_data_source/add_discord.md b/docs/guides/dataset/add_data_source/add_discord.md index 39a9bf79d9..d3ea674d75 100644 --- a/docs/guides/dataset/add_data_source/add_discord.md +++ b/docs/guides/dataset/add_data_source/add_discord.md @@ -57,4 +57,4 @@ With the bot active in your server, you can finalize the integration inside RAGF 1. Navigate to the **Dataset** tab. 2. Select or create the target Dataset. 3. Navigate to the Dataset's **Configuration** page and select **Link data source**. -4. Choose the previously created Discord connector in the popup window. \ No newline at end of file +4. Choose the previously created Discord connector in the popup window. diff --git a/docs/guides/dataset/add_data_source/add_google_drive.md b/docs/guides/dataset/add_data_source/add_google_drive.md index 7a289ee431..7df04b80cc 100644 --- a/docs/guides/dataset/add_data_source/add_google_drive.md +++ b/docs/guides/dataset/add_data_source/add_google_drive.md @@ -20,7 +20,7 @@ This document provides step-by-step instructions for integrating Google Drive as You can either create a dedicated project for RAGFlow or use an existing Google Cloud external project. In this case, we create a Google Cloud project from scratch: -1. Open the project creation page `https://console.cloud.google.com/projectcreate`: +1. Open the project creation page `https://console.cloud.google.com/projectcreate`: ![Google Drive image 1](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image1.jpeg) 2. Under **App Information**, provide an App name and your Gmail account as user support email: ![Google Drive image 2](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image2.png) @@ -34,7 +34,7 @@ You can either create a dedicated project for RAGFlow or use an existing Google You need to configure the OAuth Consent Screen because it is the step where you define how your app asks for permission and what specific data it wants to access on behalf of a user. It's a mandatory part of setting up OAuth 2.0 authentication with Google. Think of it as creating a standardized permission slip for your app. Without it, Google will not allow your app to request access to user data. 1. Go to **APIs & Services** → **OAuth consent screen**. -2. Ensure **User Type** is set to **External**: +2. Ensure **User Type** is set to **External**: ![Google Drive image 4](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image4.jpeg) 3. Under **Test Users**, click **+ Add users** to add test users: ![Google Drive image 5](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image5.jpeg) @@ -43,12 +43,12 @@ You need to configure the OAuth Consent Screen because it is the step where you ## 3. Create OAuth Client Credentials 1. Navigate to `https://console.cloud.google.com/auth/clients`. -2. Select **Web Application** as **Application type** for the created project: +2. Select **Web Application** as **Application type** for the created project: ![Google Drive image 7](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image7.png) 3. Enter a client name. 4. Add `http://localhost:9380/api/v1/connectors/google-drive/oauth/web/callback` as **Authorized redirect URIs**: 5. Add **Authorized JavaScript origins**: - - If deploying RAGFlow from Docker, use `http://localhost:80`: + - If deploying RAGFlow from Docker, use `http://localhost:80`: ![Google Drive image 8](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image8.png) - If building RAGFlow from source, use `http://localhost:9222` ![Google Drive image 9](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image9.png) @@ -81,15 +81,15 @@ https://www.googleapis.com/auth/admin.directory.user.readonly You need to enable the required APIs (such as the Google Drive API) to formally grant your Google Cloud project permission to communicate with Google's services on behalf of your application. These APIs act as a gateway; even if you have valid OAuth credentials, Google will block requests to a disabled API. Enabling them ensures that when RAGFlow attempts to list or retrieve files, Google's servers recognize and authorize the request. -1. Navigate to the Google API Library `https://console.cloud.google.com/apis/library`: +1. Navigate to the Google API Library `https://console.cloud.google.com/apis/library`: ![Google Drive image 14](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image14.png) -2. Enable the following APIs: - - Google Drive API - - Admin SDK API - - Google Sheets API +2. Enable the following APIs: + - Google Drive API + - Admin SDK API + - Google Sheets API - Google Docs API - + ![Google Drive image 15](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image15.png) ![Google Drive image 16](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image16.png) ![Google Drive image 17](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image17.png) @@ -104,12 +104,12 @@ You need to enable the required APIs (such as the Google Drive API) to formally ![Google Drive image 22](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image22.jpeg) 3. Enter the url of the shared Google Drive folder link: ![Google Drive image 23](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image23.png) -4. Click **Authorize with Google** - _A browser window appears showing that Google hasn't verified this app._ +4. Click **Authorize with Google** + _A browser window appears showing that Google hasn't verified this app._ ![Google Drive image 25](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image25.jpeg) 5. Click **Continue** → **Select All** → **Continue**. 6. When the authorization succeeds, select **OK** to add the data source. ![Google Drive image 26](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image26.jpeg) ![Google Drive image 27](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image27.jpeg) ![Google Drive image 28](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image28.png) -![Google Drive image 29](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image29.png) \ No newline at end of file +![Google Drive image 29](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image29.png) diff --git a/docs/guides/dataset/add_data_source/add_notion.md b/docs/guides/dataset/add_data_source/add_notion.md index b3af4e9419..82ad818ebf 100644 --- a/docs/guides/dataset/add_data_source/add_notion.md +++ b/docs/guides/dataset/add_data_source/add_notion.md @@ -82,4 +82,4 @@ Credentials alone do not trigger indexing. You must link the data source to a sp 1. Navigate to the **Dataset** tab. 2. Select or create the target Dataset. 3. Navigate to the Dataset's **Configuration** page and select **Link data source**. -4. Choose the previously created Notion connector in the popup window. \ No newline at end of file +4. Choose the previously created Notion connector in the popup window. diff --git a/docs/guides/dataset/advanced/autokeyword_autoquestion.mdx b/docs/guides/dataset/advanced/autokeyword_autoquestion.mdx index 51fe3cd483..907ed87298 100644 --- a/docs/guides/dataset/advanced/autokeyword_autoquestion.mdx +++ b/docs/guides/dataset/advanced/autokeyword_autoquestion.mdx @@ -24,14 +24,14 @@ Enabling this feature increases document indexing time and uses extra tokens, as Auto-keyword refers to the auto-keyword generation feature of RAGFlow. It uses a chat model to generate a set of keywords or synonyms from each chunk to correct errors and enhance retrieval accuracy. This feature is implemented as a slider under **Page rank** on the **Configuration** page of your dataset. -**Values**: +**Values**: -- 0: (Default) Disabled. -- Between 3 and 5 (inclusive): Recommended if you have chunks of approximately 1,000 characters. -- 30 (maximum) +- 0: (Default) Disabled. +- Between 3 and 5 (inclusive): Recommended if you have chunks of approximately 1,000 characters. +- 30 (maximum) :::tip NOTE -- If your chunk size increases, you can increase the value accordingly. Please note, as the value increases, the marginal benefit decreases. +- If your chunk size increases, you can increase the value accordingly. Please note, as the value increases, the marginal benefit decreases. - An Auto-keyword value must be an integer. If you set it to a non-integer, say 1.7, it will be rounded down to the nearest integer, which in this case is 1. ::: @@ -41,12 +41,12 @@ Auto-question is a feature of RAGFlow that automatically generates questions fro **Values**: -- 0: (Default) Disabled. -- 1 or 2: Recommended if you have chunks of approximately 1,000 characters. +- 0: (Default) Disabled. +- 1 or 2: Recommended if you have chunks of approximately 1,000 characters. - 10 (maximum) :::tip NOTE -- If your chunk size increases, you can increase the value accordingly. Please note, as the value increases, the marginal benefit decreases. +- If your chunk size increases, you can increase the value accordingly. Please note, as the value increases, the marginal benefit decreases. - An Auto-question value must be an integer. If you set it to a non-integer, say 1.7, it will be rounded down to the nearest integer, which in this case is 1. ::: diff --git a/docs/guides/dataset/configure_child_chunking_strategy.md b/docs/guides/dataset/configure_child_chunking_strategy.md index 12d0902437..81ec4c2206 100644 --- a/docs/guides/dataset/configure_child_chunking_strategy.md +++ b/docs/guides/dataset/configure_child_chunking_strategy.md @@ -36,4 +36,3 @@ Through this dual-layer structure of "precise localization + contextual suppleme 3. This configuration applies to the **Chunker** component when it comes to ingestion pipeline settings: ![](https://raw.githubusercontent.com/infiniflow/ragflow-docs/main/images/child_chunking_chunker.png) - diff --git a/docs/guides/dataset/enable_excel2html.md b/docs/guides/dataset/enable_excel2html.md index ac60f1c635..fcb362a96f 100644 --- a/docs/guides/dataset/enable_excel2html.md +++ b/docs/guides/dataset/enable_excel2html.md @@ -43,4 +43,4 @@ Works with complex tables that cannot be represented as key-value pairs. Example ### Should I Enable This Feature for PDFs with Complex Tables? -Nope. This feature applies to spreadsheet files only. Enabling **Excel to HTML** does not affect your PDFs. \ No newline at end of file +Nope. This feature applies to spreadsheet files only. Enabling **Excel to HTML** does not affect your PDFs. diff --git a/docs/guides/dataset/manage_metadata.md b/docs/guides/dataset/manage_metadata.md index 3242782573..4da1ece204 100644 --- a/docs/guides/dataset/manage_metadata.md +++ b/docs/guides/dataset/manage_metadata.md @@ -23,7 +23,7 @@ From v0.23.0 onwards, RAGFlow allows you to manage metadata both at the dataset ![Click metadata](https://raw.githubusercontent.com/infiniflow/ragflow-docs/main/images/click_metadata.png) -2. On the **Manage Metadata** page, you can do either of the following: +2. On the **Manage Metadata** page, you can do either of the following: - Edit Values: You can modify existing values. If you rename two values to be identical, they will be automatically merged. - Delete: You can delete specific values or entire fields. These changes will apply to all associated files. diff --git a/docs/guides/dataset/run_retrieval_test.md b/docs/guides/dataset/run_retrieval_test.md index 6c83e2a435..6f8baee529 100644 --- a/docs/guides/dataset/run_retrieval_test.md +++ b/docs/guides/dataset/run_retrieval_test.md @@ -57,7 +57,7 @@ The switch is disabled by default. When enabled, RAGFlow performs the following 3. Find similar entities and their N-hop relationships from the graph using the embeddings of the extracted query entities. 4. Retrieve similar relationships from the graph using the query embedding. 5. Rank these retrieved entities and relationships by multiplying each one's PageRank value with its similarity score to the query, returning the top n as the final retrieval. -6. Retrieve the report for the community involving the most entities in the final retrieval. +6. Retrieve the report for the community involving the most entities in the final retrieval. *The retrieved entity descriptions, relationship descriptions, and the top 1 community report are sent to the LLM for content generation.* :::danger IMPORTANT diff --git a/docs/guides/dataset/select_pdf_parser.md b/docs/guides/dataset/select_pdf_parser.md index 5d6f2b26b1..c7bd88ec6f 100644 --- a/docs/guides/dataset/select_pdf_parser.md +++ b/docs/guides/dataset/select_pdf_parser.md @@ -59,12 +59,12 @@ Starting from v0.22.0, RAGFlow includes MinerU (≥ 2.6.3) as an optional PDF p - `"vlm-mlx-engine"` - `"vlm-vllm-async-engine"` - `"vlm-lmdeploy-engine"`. - - `MINERU_SERVER_URL`: (optional) The downstream vLLM HTTP server (e.g., `http://vllm-host:30000`). Applicable when `MINERU_BACKEND` is set to `"vlm-http-client"`. + - `MINERU_SERVER_URL`: (optional) The downstream vLLM HTTP server (e.g., `http://vllm-host:30000`). Applicable when `MINERU_BACKEND` is set to `"vlm-http-client"`. - `MINERU_OUTPUT_DIR`: (optional) The local directory for holding the outputs of the MinerU API service (zip/JSON) before ingestion. - `MINERU_DELETE_OUTPUT`: Whether to delete temporary output when a temporary directory is used: - `1`: Delete. - `0`: Retain. -3. In the web UI, navigate to your dataset's **Configuration** page and find the **Ingestion pipeline** section: +3. In the web UI, navigate to your dataset's **Configuration** page and find the **Ingestion pipeline** section: - If you decide to use a chunking method from the **Built-in** dropdown, ensure it supports PDF parsing, then select **MinerU** from the **PDF parser** dropdown. - If you use a custom ingestion pipeline instead, select **MinerU** in the **PDF parser** section of the **Parser** component. diff --git a/docs/guides/dataset/set_context_window.md b/docs/guides/dataset/set_context_window.md index 37f9a2e921..a42bf3960f 100644 --- a/docs/guides/dataset/set_context_window.md +++ b/docs/guides/dataset/set_context_window.md @@ -26,4 +26,4 @@ To address this, RAGFlow 0.23.0 introduces the **Image & table context window** 2. Adjust the number of context tokens according to your needs. - *The number in the red box indicates that approximately **N tokens** of text from above and below the image/table will be captured and inserted into the image or table chunk as contextual information. The capture process intelligently optimizes boundaries at punctuation marks to preserve semantic integrity. * \ No newline at end of file + *The number in the red box indicates that approximately **N tokens** of text from above and below the image/table will be captured and inserted into the image or table chunk as contextual information. The capture process intelligently optimizes boundaries at punctuation marks to preserve semantic integrity. * diff --git a/docs/guides/dataset/set_page_rank.md b/docs/guides/dataset/set_page_rank.md index 432394d896..8c3a2a2e28 100644 --- a/docs/guides/dataset/set_page_rank.md +++ b/docs/guides/dataset/set_page_rank.md @@ -40,4 +40,4 @@ If you set the page rank value to a non-integer, say 1.7, it will be rounded dow If you configure a chat assistant's **similarity threshold** to 0.2, only chunks with a hybrid score greater than 0.2 x 100 = 20 will be retrieved and sent to the chat model for content generation. This initial filtering step is crucial for narrowing down relevant information. -If you have assigned a page rank of 1 to dataset A (2024 news) and 0 to dataset B (2023 news), the final hybrid scores of the retrieved chunks will be adjusted accordingly. A chunk retrieved from dataset A with an initial score of 50 will receive a boost of 1 x 100 = 100 points, resulting in a final score of 50 + 1 x 100 = 150. In this way, chunks retrieved from dataset A will always precede chunks from dataset B. \ No newline at end of file +If you have assigned a page rank of 1 to dataset A (2024 news) and 0 to dataset B (2023 news), the final hybrid scores of the retrieved chunks will be adjusted accordingly. A chunk retrieved from dataset A with an initial score of 50 will receive a boost of 1 x 100 = 100 points, resulting in a final score of 50 + 1 x 100 = 150. In this way, chunks retrieved from dataset A will always precede chunks from dataset B. diff --git a/docs/guides/dataset/use_tag_sets.md b/docs/guides/dataset/use_tag_sets.md index d0595ba7dd..e9aa382bcf 100644 --- a/docs/guides/dataset/use_tag_sets.md +++ b/docs/guides/dataset/use_tag_sets.md @@ -47,7 +47,7 @@ A tag set is *not* involved in document indexing or retrieval. Do not specify a 1. Click **+ Create dataset** to create a dataset. 2. Navigate to the **Configuration** page of the created dataset, select **Built-in** in **Ingestion pipeline**, then choose **Tag** as the default chunking method from the **Built-in** drop-down menu. -3. Go back to the **Files** page and upload and parse your table file in XLSX, CSV, or TXT formats. +3. Go back to the **Files** page and upload and parse your table file in XLSX, CSV, or TXT formats. A tag cloud appears under the **Tag view** section, indicating the tag set is created. 4. Click the **Table** tab to view the tag frequency table. @@ -62,12 +62,12 @@ Once a tag set is created, you can apply it to your dataset: If the tag set is missing from the dropdown, check that it has been created or configured correctly. ::: -3. Re-parse your documents to start the auto-tagging process. +3. Re-parse your documents to start the auto-tagging process. _In an AI chat scenario using auto-tagged datasets, each query will be tagged using the corresponding tag set(s) and chunks with these tags will have a higher chance to be retrieved._ ## 3. Update Tag Set -Creating a tag set is *not* for once and for all. Oftentimes, you may find it necessary to update or delete existing tags or add new entries. +Creating a tag set is *not* for once and for all. Oftentimes, you may find it necessary to update or delete existing tags or add new entries. - You can update the existing tag set in the tag frequency table. - To add new entries, you can add and parse new table files in XLSX, CSV, or TXT formats. diff --git a/docs/guides/models/deploy_local_llm.mdx b/docs/guides/models/deploy_local_llm.mdx index cb92aea31c..6cdb705b25 100644 --- a/docs/guides/models/deploy_local_llm.mdx +++ b/docs/guides/models/deploy_local_llm.mdx @@ -57,9 +57,9 @@ $ sudo docker exec ollama ollama pull llama3.2 ``` ```bash -$ sudo docker exec ollama ollama pull bge-m3 -> pulling daec91ffb5dd... 100% ▕████████████████▏ 1.2 GB -> success +$ sudo docker exec ollama ollama pull bge-m3 +> pulling daec91ffb5dd... 100% ▕████████████████▏ 1.2 GB +> success ``` ### 2. Find Ollama URL and Ensure It Is Accessible @@ -108,7 +108,7 @@ Max retries exceeded with url: /api/chat (Caused by NewConnectionError('** **Model providers** **>** **System Model Settings** to update your model: - + - *You should now be able to find **llama3.2** from the dropdown list under **Chat model**, and **bge-m3** from the dropdown list under **Embedding model**.* ### 6. Update Chat Configuration @@ -128,7 +128,7 @@ To deploy a local model, e.g., **Mistral**, using Xinference: ### 1. Check Firewall Settings -Ensure that your host machine's firewall allows inbound connections on port 9997. +Ensure that your host machine's firewall allows inbound connections on port 9997. ### 2. Start an Xinference Instance @@ -150,13 +150,13 @@ In RAGFlow, click on your logo on the top right of the page **>** **Model provid ### 5. Complete Basic Xinference Settings -Enter an accessible base URL, such as `http://:9997/v1`. +Enter an accessible base URL, such as `http://:9997/v1`. > For rerank model, please use the `http://:9997/v1/rerank` as the base URL. ### 6. Update System Model Settings Click on your logo **>** **Model providers** **>** **System Model Settings** to update your model. - + *You should now be able to find **mistral** from the dropdown list under **Chat model**.* ### 7. Update Chat Configuration @@ -172,7 +172,7 @@ To deploy a local model, e.g., **Qwen2**, using IPEX-LLM-accelerated Ollama: ### 1. Check Firewall Settings Ensure that your host machine's firewall allows inbound connections on port 11434. For example: - + ```bash sudo ufw allow 11434/tcp ``` @@ -181,7 +181,7 @@ sudo ufw allow 11434/tcp #### 2.1 Install IPEX-LLM for Ollama -:::tip NOTE +:::tip NOTE IPEX-LLM's supports Ollama on Linux and Windows systems. ::: @@ -193,7 +193,7 @@ For detailed information about installing IPEX-LLM for Ollama, see [Run llama.cp #### 2.2 Initialize Ollama -1. Activate the `llm-cpp` Conda environment and initialize Ollama: +1. Activate the `llm-cpp` Conda environment and initialize Ollama: - + ```bash conda activate llm-cpp init-ollama @@ -220,7 +220,7 @@ For detailed information about installing IPEX-LLM for Ollama, see [Run llama.cp 2. If the installed `ipex-llm[cpp]` requires an upgrade to the Ollama binary files, remove the old binary files and reinitialize Ollama using `init-ollama` (Linux) or `init-ollama.bat` (Windows). - + *A symbolic link to Ollama appears in your current directory, and you can use this executable file following standard Ollama commands.* #### 2.3 Launch Ollama Service @@ -228,7 +228,7 @@ For detailed information about installing IPEX-LLM for Ollama, see [Run llama.cp 1. Set the environment variable `OLLAMA_NUM_GPU` to `999` to ensure that all layers of your model run on the Intel GPU; otherwise, some layers may default to CPU. 2. For optimal performance on Intel Arc™ A-Series Graphics with Linux OS (Kernel 6.2), set the following environment variable before launching the Ollama service: - ```bash + ```bash export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1 ``` 3. Launch the Ollama service: diff --git a/internal/ingestion/pipeline/pipeline_params_test.go b/internal/ingestion/pipeline/pipeline_params_test.go index 2e732329fe..52fe83fc41 100644 --- a/internal/ingestion/pipeline/pipeline_params_test.go +++ b/internal/ingestion/pipeline/pipeline_params_test.go @@ -214,6 +214,48 @@ func TestBuildParserConfig_AllComponentsPresent(t *testing.T) { } } +func TestBuildParserConfig_BuiltinExtractorKeepsTagFileID(t *testing.T) { + registry, err := DefaultRegistry() + if err != nil { + t.Fatalf("DefaultRegistry: %v", err) + } + checked := 0 + for _, ref := range registry.Refs() { + tpl, ok := registry.Get(ref) + if !ok { + t.Fatalf("registry.Get(%q) failed", ref) + } + dslJSON, err := json.Marshal(tpl.DSL) + if err != nil { + t.Fatalf("marshal DSL %q: %v", ref, err) + } + schemas, err := ExtractAllComponentParams(dslJSON) + if err != nil { + t.Fatalf("ExtractAllComponentParams %q: %v", ref, err) + } + for _, s := range schemas { + if s.ComponentName != "Extractor" { + continue + } + checked++ + overrides := map[string]any{ + s.CpnID: map[string]any{"tag_file_id": "file-123"}, + } + result := BuildParserConfig(dslJSON, overrides) + params, ok := result[s.CpnID].(map[string]any) + if !ok { + t.Fatalf("template %q: expected component %q in result", ref, s.CpnID) + } + if params["tag_file_id"] != "file-123" { + t.Errorf("template %q: expected tag_file_id to survive BuildParserConfig, got %v", ref, params["tag_file_id"]) + } + } + } + if checked == 0 { + t.Fatal("expected at least one builtin template with an Extractor component") + } +} + // --- ResolveComponentParamsDefaults --- func TestResolveComponentParamsDefaults_Basic(t *testing.T) { diff --git a/internal/ingestion/pipeline/template/ingestion_pipeline_audio.json b/internal/ingestion/pipeline/template/ingestion_pipeline_audio.json index 1a20de075c..90b6e82366 100644 --- a/internal/ingestion/pipeline/template/ingestion_pipeline_audio.json +++ b/internal/ingestion/pipeline/template/ingestion_pipeline_audio.json @@ -136,6 +136,7 @@ "auto_questions": 0, "llm_id": "", "auto_tags": 0, + "tag_file_id": "", "enable_metadata": 0, "metadata": [] } diff --git a/internal/ingestion/pipeline/template/ingestion_pipeline_book.json b/internal/ingestion/pipeline/template/ingestion_pipeline_book.json index 1b6336f78c..ddc987eabe 100644 --- a/internal/ingestion/pipeline/template/ingestion_pipeline_book.json +++ b/internal/ingestion/pipeline/template/ingestion_pipeline_book.json @@ -197,6 +197,7 @@ "auto_questions": 0, "llm_id": "", "auto_tags": 0, + "tag_file_id": "", "enable_metadata": 0, "metadata": [] } diff --git a/internal/ingestion/pipeline/template/ingestion_pipeline_email.json b/internal/ingestion/pipeline/template/ingestion_pipeline_email.json index 25947234e1..805a5df922 100644 --- a/internal/ingestion/pipeline/template/ingestion_pipeline_email.json +++ b/internal/ingestion/pipeline/template/ingestion_pipeline_email.json @@ -141,6 +141,7 @@ "auto_questions": 0, "llm_id": "", "auto_tags": 0, + "tag_file_id": "", "enable_metadata": 0, "metadata": [] } diff --git a/internal/ingestion/pipeline/template/ingestion_pipeline_general.json b/internal/ingestion/pipeline/template/ingestion_pipeline_general.json index b540ff1b57..ff4a3165ba 100644 --- a/internal/ingestion/pipeline/template/ingestion_pipeline_general.json +++ b/internal/ingestion/pipeline/template/ingestion_pipeline_general.json @@ -225,6 +225,7 @@ "auto_questions": 0, "llm_id": "", "auto_tags": 0, + "tag_file_id": "", "enable_metadata": 0, "metadata": [] } diff --git a/internal/ingestion/pipeline/template/ingestion_pipeline_laws.json b/internal/ingestion/pipeline/template/ingestion_pipeline_laws.json index 7b6e5ca7b5..b08be5d874 100644 --- a/internal/ingestion/pipeline/template/ingestion_pipeline_laws.json +++ b/internal/ingestion/pipeline/template/ingestion_pipeline_laws.json @@ -210,6 +210,7 @@ "auto_questions": 0, "llm_id": "", "auto_tags": 0, + "tag_file_id": "", "enable_metadata": 0, "metadata": [] } diff --git a/internal/ingestion/pipeline/template/ingestion_pipeline_manual.json b/internal/ingestion/pipeline/template/ingestion_pipeline_manual.json index c227ccd188..e83e0d17e5 100644 --- a/internal/ingestion/pipeline/template/ingestion_pipeline_manual.json +++ b/internal/ingestion/pipeline/template/ingestion_pipeline_manual.json @@ -177,6 +177,7 @@ "auto_questions": 0, "llm_id": "", "auto_tags": 0, + "tag_file_id": "", "enable_metadata": 0, "metadata": [] } diff --git a/internal/ingestion/pipeline/template/ingestion_pipeline_one.json b/internal/ingestion/pipeline/template/ingestion_pipeline_one.json index 3a0005c821..3406161af5 100644 --- a/internal/ingestion/pipeline/template/ingestion_pipeline_one.json +++ b/internal/ingestion/pipeline/template/ingestion_pipeline_one.json @@ -190,6 +190,7 @@ "auto_questions": 0, "llm_id": "", "auto_tags": 0, + "tag_file_id": "", "enable_metadata": 0, "metadata": [] } diff --git a/internal/ingestion/pipeline/template/ingestion_pipeline_paper.json b/internal/ingestion/pipeline/template/ingestion_pipeline_paper.json index b90d09b8b3..07a49d26e2 100644 --- a/internal/ingestion/pipeline/template/ingestion_pipeline_paper.json +++ b/internal/ingestion/pipeline/template/ingestion_pipeline_paper.json @@ -158,6 +158,7 @@ "auto_questions": 0, "llm_id": "", "auto_tags": 0, + "tag_file_id": "", "enable_metadata": 0, "metadata": [] } diff --git a/internal/ingestion/pipeline/template/ingestion_pipeline_picture.json b/internal/ingestion/pipeline/template/ingestion_pipeline_picture.json index cf1e9dbea4..effb4e5b7d 100644 --- a/internal/ingestion/pipeline/template/ingestion_pipeline_picture.json +++ b/internal/ingestion/pipeline/template/ingestion_pipeline_picture.json @@ -150,6 +150,7 @@ "auto_questions": 0, "llm_id": "", "auto_tags": 0, + "tag_file_id": "", "enable_metadata": 0, "metadata": [] } diff --git a/internal/ingestion/pipeline/template/ingestion_pipeline_presentation.json b/internal/ingestion/pipeline/template/ingestion_pipeline_presentation.json index 14e9db6912..b4746fff1c 100644 --- a/internal/ingestion/pipeline/template/ingestion_pipeline_presentation.json +++ b/internal/ingestion/pipeline/template/ingestion_pipeline_presentation.json @@ -136,6 +136,7 @@ "auto_questions": 0, "llm_id": "", "auto_tags": 0, + "tag_file_id": "", "enable_metadata": 0, "metadata": [] } diff --git a/internal/ingestion/pipeline/template/ingestion_pipeline_resume.json b/internal/ingestion/pipeline/template/ingestion_pipeline_resume.json index 7f449c4d34..4721777419 100644 --- a/internal/ingestion/pipeline/template/ingestion_pipeline_resume.json +++ b/internal/ingestion/pipeline/template/ingestion_pipeline_resume.json @@ -50,6 +50,7 @@ "auto_keywords": 0, "auto_questions": 0, "auto_tags": 0, + "tag_file_id": "", "enable_metadata": 0, "metadata": [] } diff --git a/internal/ingestion/task/pipeline_executor_defaults_test.go b/internal/ingestion/task/pipeline_executor_defaults_test.go index acc2113a8e..b7cb5ba9dd 100644 --- a/internal/ingestion/task/pipeline_executor_defaults_test.go +++ b/internal/ingestion/task/pipeline_executor_defaults_test.go @@ -38,18 +38,18 @@ import ( // Comparison is done per-component (see assertComponentsMatch), so the JSON // serialization order inside each entry is irrelevant. var builtinComponentParamsGolden = map[string]string{ - "audio": "{\"File\": {}, \"Parser:SongsFillAir\": {\"audio\": {\"output_format\": \"text\", \"preprocess\": [\"main_content\"], \"suffix\": [\"aac\", \"aiff\", \"ape\", \"au\", \"da\", \"flac\", \"midi\", \"mp3\", \"ogg\", \"oggvorbis\", \"realaudio\", \"vqf\", \"wav\", \"wave\", \"wma\"]}}, \"TokenChunker:BlueSkiesLaugh\": {}, \"Tokenizer:KindEyesWatch\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"Extractor:AutoExtractDefault\": {\"field_name\": \"\", \"auto_keywords\": 0, \"auto_questions\": 0, \"llm_id\": \"\", \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": []}}", - "book": "{\"File\": {}, \"Parser:HipSignsRhyme\": {\"doc\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"doc\"]}, \"docx\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"docx\"], \"vlm\": {}}, \"html\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"htm\", \"html\"]}, \"pdf\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"remove_toc\": true, \"suffix\": [\"pdf\"], \"vlm\": {}}, \"text&code\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"txt\"]}}, \"TitleChunker:GrumpyGarlicsBake\": {\"hierarchy\": 5, \"include_heading_content\": true, \"levels\": [[\"^#[^#]\", \"^##[^#]\", \"^###[^#]\", \"^####[^#]\"], [\"第[零一二三四五六七八九十百0-9]+(分?编|部分)\", \"第[零一二三四五六七八九十百0-9]+章\", \"第[零一二三四五六七八九十百0-9]+节\", \"第[零一二三四五六七八九十百0-9]+条\", \"[\\\\(（][零一二三四五六七八九十百]+[\\\\)）]\"], [\"第[0-9]+章\", \"第[0-9]+节\", \"[0-9]{1,2}[\\\\. 、]\", \"[0-9]{1,2}\\\\.[0-9]{1,2}($|[^a-zA-Z/%~.-])\", \"[0-9]{1,2}\\\\.[0-9]{1,2}\\\\.[0-9]{1,2}\"], [\"第[零一二三四五六七八九十百0-9]+章\", \"第[零一二三四五六七八九十百0-9]+节\", \"[零一二三四五六七八九十百]+[ 、]\", \"[\\\\(（][零一二三四五六七八九十百]+[\\\\)）]\", \"[\\\\(（][0-9]{,2}[\\\\)）]\"], [\"PART (ONE|TWO|THREE|FOUR|FIVE|SIX|SEVEN|EIGHT|NINE|TEN)\", \"Chapter (I+V?|VI*|XI|IX|X)\", \"Section [0-9]+\", \"Article [0-9]+\"]], \"method\": \"hierarchy\"}, \"Tokenizer:HotDonutsRing\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"Extractor:AutoExtractDefault\": {\"field_name\": \"\", \"auto_keywords\": 0, \"auto_questions\": 0, \"llm_id\": \"\", \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": []}}", - "email": "{\"File\": {}, \"Parser:BirdsFlutterHigh\": {\"email\": {\"fields\": [\"from\", \"to\", \"cc\", \"bcc\", \"date\", \"subject\", \"body\", \"attachments\"], \"output_format\": \"text\", \"preprocess\": [\"main_content\"], \"suffix\": [\"eml\"]}}, \"TokenChunker:WarmBreadSmells\": {\"children_delimiters\": [], \"chunk_token_size\": 512, \"delimiter_mode\": \"token_size\", \"delimiters\": [\"\\n\", \"!\", \"?\", \"。\", \"；\", \"！\", \"？\"], \"image_context_size\": 0, \"overlapped_percent\": 0, \"table_context_size\": 0}, \"Tokenizer:NiceWordsSpoken\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"Extractor:AutoExtractDefault\": {\"field_name\": \"\", \"auto_keywords\": 0, \"auto_questions\": 0, \"llm_id\": \"\", \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": []}}", - "general": "{\"File\": {}, \"Parser:HipSignsRhyme\": {\"doc\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"doc\"]}, \"docx\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"docx\"], \"vlm\": {}}, \"html\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"htm\", \"html\"]}, \"markdown\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"md\", \"markdown\", \"mdx\"], \"vlm\": {}}, \"pdf\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"pages\": [[1, 100000]], \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"pdf\"], \"vlm\": {}}, \"spreadsheet\": {\"flatten_media_to_text\": false, \"output_format\": \"html\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"xls\", \"xlsx\", \"csv\"], \"vlm\": {}}, \"text&code\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"txt\", \"py\", \"js\", \"java\", \"c\", \"cpp\", \"h\", \"php\", \"go\", \"ts\", \"sh\", \"cs\", \"kt\", \"sql\"]}}, \"TokenChunker:SixApplesFall\": {\"children_delimiters\": [], \"chunk_token_size\": 512, \"delimiter_mode\": \"token_size\", \"delimiters\": [\"\\n\", \"!\", \"?\", \"。\", \"；\", \"！\", \"？\"], \"image_context_size\": 0, \"overlapped_percent\": 0, \"table_context_size\": 0}, \"Tokenizer:LegalReadersDecide\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"Extractor:AutoExtractDefault\": {\"field_name\": \"\", \"auto_keywords\": 0, \"auto_questions\": 0, \"llm_id\": \"\", \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": []}}", - "laws": "{\"File\": {}, \"Parser:HipSignsRhyme\": {\"doc\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"doc\"]}, \"docx\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"docx\"], \"vlm\": {}}, \"html\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"htm\", \"html\"]}, \"markdown\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"md\", \"markdown\", \"mdx\"], \"vlm\": {}}, \"pdf\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"remove_toc\": true, \"suffix\": [\"pdf\"], \"vlm\": {}}, \"text&code\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"txt\"]}}, \"TitleChunker:SpicyKeysKick\": {\"hierarchy\": 2, \"include_heading_content\": false, \"levels\": [[\"^#[^#]\", \"^##[^#]\", \"^###[^#]\", \"^####[^#]\"], [\"第[零一二三四五六七八九十百0-9]+(分?编|部分)\", \"第[零一二三四五六七八九十百0-9]+章\", \"第[零一二三四五六七八九十百0-9]+节\", \"第[零一二三四五六七八九十百0-9]+条\", \"[\\\\(（][零一二三四五六七八九十百]+[\\\\)）]\"], [\"第[0-9]+章\", \"第[0-9]+节\", \"[0-9]{1,2}[\\\\. 、]\", \"[0-9]{1,2}\\\\.[0-9]{1,2}($|[^a-zA-Z/%~.-])\", \"[0-9]{1,2}\\\\.[0-9]{1,2}\\\\.[0-9]{1,2}\"], [\"第[零一二三四五六七八九十百0-9]+章\", \"第[零一二三四五六七八九十百0-9]+节\", \"[零一二三四五六七八九十百]+[ 、]\", \"[\\\\(（][零一二三四五六七八九十百]+[\\\\)）]\", \"[\\\\(（][0-9]{,2}[\\\\)）]\"], [\"PART (ONE|TWO|THREE|FOUR|FIVE|SIX|SEVEN|EIGHT|NINE|TEN)\", \"Chapter (I+V?|VI*|XI|IX|X)\", \"Section [0-9]+\", \"Article [0-9]+\"]], \"method\": \"hierarchy\"}, \"Tokenizer:PublicJobsTake\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"Extractor:AutoExtractDefault\": {\"field_name\": \"\", \"auto_keywords\": 0, \"auto_questions\": 0, \"llm_id\": \"\", \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": []}}", - "manual": "{\"File\": {}, \"Parser:HipSignsRhyme\": {\"doc\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"doc\"]}, \"docx\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"docx\"], \"vlm\": {}}, \"pdf\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"pdf\"], \"vlm\": {}}}, \"TitleChunker:NineInsectsFind\": {\"hierarchy\": 0, \"include_heading_content\": false, \"levels\": [[\"^#[^#]\", \"^##[^#]\", \"^###[^#]\", \"^####[^#]\"], [\"第[零一二三四五六七八九十百0-9]+(分?编|部分)\", \"第[零一二三四五六七八九十百0-9]+章\", \"第[零一二三四五六七八九十百0-9]+节\", \"第[零一二三四五六七八九十百0-9]+条\", \"[\\\\(（][零一二三四五六七八九十百]+[\\\\)）]\"], [\"第[0-9]+章\", \"第[0-9]+节\", \"[0-9]{1,2}[\\\\. 、]\", \"[0-9]{1,2}\\\\.[0-9]{1,2}($|[^a-zA-Z/%~.-])\", \"[0-9]{1,2}\\\\.[0-9]{1,2}\\\\.[0-9]{1,2}\"], [\"第[零一二三四五六七八九十百0-9]+章\", \"第[零一二三四五六七八九十百0-9]+节\", \"[零一二三四五六七八九十百]+[ 、]\", \"[\\\\(（][零一二三四五六七八九十百]+[\\\\)）]\", \"[\\\\(（][0-9]{,2}[\\\\)）]\"], [\"PART (ONE|TWO|THREE|FOUR|FIVE|SIX|SEVEN|EIGHT|NINE|TEN)\", \"Chapter (I+V?|VI*|XI|IX|X)\", \"Section [0-9]+\", \"Article [0-9]+\"]], \"method\": \"group\"}, \"Tokenizer:FunnyBalloonsGrin\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"Extractor:AutoExtractDefault\": {\"field_name\": \"\", \"auto_keywords\": 0, \"auto_questions\": 0, \"llm_id\": \"\", \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": []}}", - "one": "{\"File\": {}, \"Parser:HipSignsRhyme\": {\"doc\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"doc\"]}, \"docx\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"docx\"], \"vlm\": {}}, \"html\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"htm\", \"html\"]}, \"markdown\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"md\", \"markdown\", \"mdx\"], \"vlm\": {}}, \"pdf\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"pdf\"], \"vlm\": {}}, \"spreadsheet\": {\"flatten_media_to_text\": false, \"output_format\": \"html\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"xls\", \"xlsx\"], \"vlm\": {}}, \"text&code\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"txt\"]}}, \"OneChunker:DryDrinksVisit\": {}, \"Tokenizer:FrankWeeksListen\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"Extractor:AutoExtractDefault\": {\"field_name\": \"\", \"auto_keywords\": 0, \"auto_questions\": 0, \"llm_id\": \"\", \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": []}}", - "paper": "{\"File\": {}, \"Parser:HipSignsRhyme\": {\"pdf\": {\"enable_multi_column\": true, \"flatten_media_to_text\": false, \"output_format\": \"json\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"pdf\"], \"vlm\": {}}}, \"TitleChunker:SparklySchoolsTravel\": {\"hierarchy\": 0, \"include_heading_content\": false, \"levels\": [[\"^#[^#]\", \"^##[^#]\", \"^###[^#]\", \"^####[^#]\"], [\"第[零一二三四五六七八九十百0-9]+(分?编|部分)\", \"第[零一二三四五六七八九十百0-9]+章\", \"第[零一二三四五六七八九十百0-9]+节\", \"第[零一二三四五六七八九十百0-9]+条\", \"[\\\\(（][零一二三四五六七八九十百]+[\\\\)）]\"], [\"第[0-9]+章\", \"第[0-9]+节\", \"[0-9]{1,2}[\\\\. 、]\", \"[0-9]{1,2}\\\\.[0-9]{1,2}($|[^a-zA-Z/%~.-])\", \"[0-9]{1,2}\\\\.[0-9]{1,2}\\\\.[0-9]{1,2}\"], [\"第[零一二三四五六七八九十百0-9]+章\", \"第[零一二三四五六七八九十百0-9]+节\", \"[零一二三四五六七八九十百]+[ 、]\", \"[\\\\(（][零一二三四五六七八九十百]+[\\\\)）]\", \"[\\\\(（][0-9]{,2}[\\\\)）]\"], [\"PART (ONE|TWO|THREE|FOUR|FIVE|SIX|SEVEN|EIGHT|NINE|TEN)\", \"Chapter (I+V?|VI*|XI|IX|X)\", \"Section [0-9]+\", \"Article [0-9]+\"]], \"method\": \"group\"}, \"Tokenizer:GreatCarsWash\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"Extractor:AutoExtractDefault\": {\"field_name\": \"\", \"auto_keywords\": 0, \"auto_questions\": 0, \"llm_id\": \"\", \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": []}}", - "picture": "{\"File\": {}, \"Parser:ViewsCaptureLight\": {\"image\": {\"output_format\": \"json\", \"parse_method\": \"ocr\", \"preprocess\": [\"main_content\"], \"suffix\": [\"bmp\", \"gif\", \"jpeg\", \"jpg\", \"png\", \"svg\", \"tif\", \"tiff\", \"webp\"]}, \"video\": {\"output_format\": \"text\", \"preprocess\": [\"main_content\"], \"suffix\": [\"3gp\", \"3gpp\", \"avi\", \"flv\", \"mkv\", \"mov\", \"mp4\", \"mpeg\", \"mpg\", \"webm\", \"wmv\"]}}, \"TokenChunker:BrightColorsGlow\": {}, \"Tokenizer:SharpLensFocus\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"Extractor:AutoExtractDefault\": {\"field_name\": \"\", \"auto_keywords\": 0, \"auto_questions\": 0, \"llm_id\": \"\", \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": []}}", - "presentation": "{\"File\": {}, \"Parser:HipSignsRhyme\": {\"pdf\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"pdf\"], \"vlm\": {}}, \"slides\": {\"output_format\": \"json\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"pptx\", \"ppt\"]}}, \"Tokenizer:TallTreesDance\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"PresentationChunker:HappyHillsGlow\": {}, \"Extractor:AutoExtractDefault\": {\"field_name\": \"\", \"auto_keywords\": 0, \"auto_questions\": 0, \"llm_id\": \"\", \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": []}}", + "audio": "{\"File\": {}, \"Parser:SongsFillAir\": {\"audio\": {\"output_format\": \"text\", \"preprocess\": [\"main_content\"], \"suffix\": [\"aac\", \"aiff\", \"ape\", \"au\", \"da\", \"flac\", \"midi\", \"mp3\", \"ogg\", \"oggvorbis\", \"realaudio\", \"vqf\", \"wav\", \"wave\", \"wma\"]}}, \"TokenChunker:BlueSkiesLaugh\": {}, \"Tokenizer:KindEyesWatch\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"Extractor:AutoExtractDefault\": {\"field_name\": \"\", \"auto_keywords\": 0, \"auto_questions\": 0, \"llm_id\": \"\", \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": [], \"tag_file_id\": \"\"}}", + "book": "{\"File\": {}, \"Parser:HipSignsRhyme\": {\"doc\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"doc\"]}, \"docx\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"docx\"], \"vlm\": {}}, \"html\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"htm\", \"html\"]}, \"pdf\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"remove_toc\": true, \"suffix\": [\"pdf\"], \"vlm\": {}}, \"text&code\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"txt\"]}}, \"TitleChunker:GrumpyGarlicsBake\": {\"hierarchy\": 5, \"include_heading_content\": true, \"levels\": [[\"^#[^#]\", \"^##[^#]\", \"^###[^#]\", \"^####[^#]\"], [\"第[零一二三四五六七八九十百0-9]+(分?编|部分)\", \"第[零一二三四五六七八九十百0-9]+章\", \"第[零一二三四五六七八九十百0-9]+节\", \"第[零一二三四五六七八九十百0-9]+条\", \"[\\\\(（][零一二三四五六七八九十百]+[\\\\)）]\"], [\"第[0-9]+章\", \"第[0-9]+节\", \"[0-9]{1,2}[\\\\. 、]\", \"[0-9]{1,2}\\\\.[0-9]{1,2}($|[^a-zA-Z/%~.-])\", \"[0-9]{1,2}\\\\.[0-9]{1,2}\\\\.[0-9]{1,2}\"], [\"第[零一二三四五六七八九十百0-9]+章\", \"第[零一二三四五六七八九十百0-9]+节\", \"[零一二三四五六七八九十百]+[ 、]\", \"[\\\\(（][零一二三四五六七八九十百]+[\\\\)）]\", \"[\\\\(（][0-9]{,2}[\\\\)）]\"], [\"PART (ONE|TWO|THREE|FOUR|FIVE|SIX|SEVEN|EIGHT|NINE|TEN)\", \"Chapter (I+V?|VI*|XI|IX|X)\", \"Section [0-9]+\", \"Article [0-9]+\"]], \"method\": \"hierarchy\"}, \"Tokenizer:HotDonutsRing\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"Extractor:AutoExtractDefault\": {\"field_name\": \"\", \"auto_keywords\": 0, \"auto_questions\": 0, \"llm_id\": \"\", \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": [], \"tag_file_id\": \"\"}}", + "email": "{\"File\": {}, \"Parser:BirdsFlutterHigh\": {\"email\": {\"fields\": [\"from\", \"to\", \"cc\", \"bcc\", \"date\", \"subject\", \"body\", \"attachments\"], \"output_format\": \"text\", \"preprocess\": [\"main_content\"], \"suffix\": [\"eml\"]}}, \"TokenChunker:WarmBreadSmells\": {\"children_delimiters\": [], \"chunk_token_size\": 512, \"delimiter_mode\": \"token_size\", \"delimiters\": [\"\\n\", \"!\", \"?\", \"。\", \"；\", \"！\", \"？\"], \"image_context_size\": 0, \"overlapped_percent\": 0, \"table_context_size\": 0}, \"Tokenizer:NiceWordsSpoken\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"Extractor:AutoExtractDefault\": {\"field_name\": \"\", \"auto_keywords\": 0, \"auto_questions\": 0, \"llm_id\": \"\", \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": [], \"tag_file_id\": \"\"}}", + "general": "{\"File\": {}, \"Parser:HipSignsRhyme\": {\"doc\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"doc\"]}, \"docx\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"docx\"], \"vlm\": {}}, \"html\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"htm\", \"html\"]}, \"markdown\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"md\", \"markdown\", \"mdx\"], \"vlm\": {}}, \"pdf\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"pages\": [[1, 100000]], \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"pdf\"], \"vlm\": {}}, \"spreadsheet\": {\"flatten_media_to_text\": false, \"output_format\": \"html\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"xls\", \"xlsx\", \"csv\"], \"vlm\": {}}, \"text&code\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"txt\", \"py\", \"js\", \"java\", \"c\", \"cpp\", \"h\", \"php\", \"go\", \"ts\", \"sh\", \"cs\", \"kt\", \"sql\"]}}, \"TokenChunker:SixApplesFall\": {\"children_delimiters\": [], \"chunk_token_size\": 512, \"delimiter_mode\": \"token_size\", \"delimiters\": [\"\\n\", \"!\", \"?\", \"。\", \"；\", \"！\", \"？\"], \"image_context_size\": 0, \"overlapped_percent\": 0, \"table_context_size\": 0}, \"Tokenizer:LegalReadersDecide\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"Extractor:AutoExtractDefault\": {\"field_name\": \"\", \"auto_keywords\": 0, \"auto_questions\": 0, \"llm_id\": \"\", \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": [], \"tag_file_id\": \"\"}}", + "laws": "{\"File\": {}, \"Parser:HipSignsRhyme\": {\"doc\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"doc\"]}, \"docx\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"docx\"], \"vlm\": {}}, \"html\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"htm\", \"html\"]}, \"markdown\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"md\", \"markdown\", \"mdx\"], \"vlm\": {}}, \"pdf\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"remove_toc\": true, \"suffix\": [\"pdf\"], \"vlm\": {}}, \"text&code\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"txt\"]}}, \"TitleChunker:SpicyKeysKick\": {\"hierarchy\": 2, \"include_heading_content\": false, \"levels\": [[\"^#[^#]\", \"^##[^#]\", \"^###[^#]\", \"^####[^#]\"], [\"第[零一二三四五六七八九十百0-9]+(分?编|部分)\", \"第[零一二三四五六七八九十百0-9]+章\", \"第[零一二三四五六七八九十百0-9]+节\", \"第[零一二三四五六七八九十百0-9]+条\", \"[\\\\(（][零一二三四五六七八九十百]+[\\\\)）]\"], [\"第[0-9]+章\", \"第[0-9]+节\", \"[0-9]{1,2}[\\\\. 、]\", \"[0-9]{1,2}\\\\.[0-9]{1,2}($|[^a-zA-Z/%~.-])\", \"[0-9]{1,2}\\\\.[0-9]{1,2}\\\\.[0-9]{1,2}\"], [\"第[零一二三四五六七八九十百0-9]+章\", \"第[零一二三四五六七八九十百0-9]+节\", \"[零一二三四五六七八九十百]+[ 、]\", \"[\\\\(（][零一二三四五六七八九十百]+[\\\\)）]\", \"[\\\\(（][0-9]{,2}[\\\\)）]\"], [\"PART (ONE|TWO|THREE|FOUR|FIVE|SIX|SEVEN|EIGHT|NINE|TEN)\", \"Chapter (I+V?|VI*|XI|IX|X)\", \"Section [0-9]+\", \"Article [0-9]+\"]], \"method\": \"hierarchy\"}, \"Tokenizer:PublicJobsTake\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"Extractor:AutoExtractDefault\": {\"field_name\": \"\", \"auto_keywords\": 0, \"auto_questions\": 0, \"llm_id\": \"\", \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": [], \"tag_file_id\": \"\"}}", + "manual": "{\"File\": {}, \"Parser:HipSignsRhyme\": {\"doc\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"doc\"]}, \"docx\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"docx\"], \"vlm\": {}}, \"pdf\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"pdf\"], \"vlm\": {}}}, \"TitleChunker:NineInsectsFind\": {\"hierarchy\": 0, \"include_heading_content\": false, \"levels\": [[\"^#[^#]\", \"^##[^#]\", \"^###[^#]\", \"^####[^#]\"], [\"第[零一二三四五六七八九十百0-9]+(分?编|部分)\", \"第[零一二三四五六七八九十百0-9]+章\", \"第[零一二三四五六七八九十百0-9]+节\", \"第[零一二三四五六七八九十百0-9]+条\", \"[\\\\(（][零一二三四五六七八九十百]+[\\\\)）]\"], [\"第[0-9]+章\", \"第[0-9]+节\", \"[0-9]{1,2}[\\\\. 、]\", \"[0-9]{1,2}\\\\.[0-9]{1,2}($|[^a-zA-Z/%~.-])\", \"[0-9]{1,2}\\\\.[0-9]{1,2}\\\\.[0-9]{1,2}\"], [\"第[零一二三四五六七八九十百0-9]+章\", \"第[零一二三四五六七八九十百0-9]+节\", \"[零一二三四五六七八九十百]+[ 、]\", \"[\\\\(（][零一二三四五六七八九十百]+[\\\\)）]\", \"[\\\\(（][0-9]{,2}[\\\\)）]\"], [\"PART (ONE|TWO|THREE|FOUR|FIVE|SIX|SEVEN|EIGHT|NINE|TEN)\", \"Chapter (I+V?|VI*|XI|IX|X)\", \"Section [0-9]+\", \"Article [0-9]+\"]], \"method\": \"group\"}, \"Tokenizer:FunnyBalloonsGrin\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"Extractor:AutoExtractDefault\": {\"field_name\": \"\", \"auto_keywords\": 0, \"auto_questions\": 0, \"llm_id\": \"\", \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": [], \"tag_file_id\": \"\"}}", + "one": "{\"File\": {}, \"Parser:HipSignsRhyme\": {\"doc\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"doc\"]}, \"docx\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"docx\"], \"vlm\": {}}, \"html\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"htm\", \"html\"]}, \"markdown\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"md\", \"markdown\", \"mdx\"], \"vlm\": {}}, \"pdf\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"pdf\"], \"vlm\": {}}, \"spreadsheet\": {\"flatten_media_to_text\": false, \"output_format\": \"html\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"xls\", \"xlsx\"], \"vlm\": {}}, \"text&code\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"txt\"]}}, \"OneChunker:DryDrinksVisit\": {}, \"Tokenizer:FrankWeeksListen\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"Extractor:AutoExtractDefault\": {\"field_name\": \"\", \"auto_keywords\": 0, \"auto_questions\": 0, \"llm_id\": \"\", \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": [], \"tag_file_id\": \"\"}}", + "paper": "{\"File\": {}, \"Parser:HipSignsRhyme\": {\"pdf\": {\"enable_multi_column\": true, \"flatten_media_to_text\": false, \"output_format\": \"json\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"pdf\"], \"vlm\": {}}}, \"TitleChunker:SparklySchoolsTravel\": {\"hierarchy\": 0, \"include_heading_content\": false, \"levels\": [[\"^#[^#]\", \"^##[^#]\", \"^###[^#]\", \"^####[^#]\"], [\"第[零一二三四五六七八九十百0-9]+(分?编|部分)\", \"第[零一二三四五六七八九十百0-9]+章\", \"第[零一二三四五六七八九十百0-9]+节\", \"第[零一二三四五六七八九十百0-9]+条\", \"[\\\\(（][零一二三四五六七八九十百]+[\\\\)）]\"], [\"第[0-9]+章\", \"第[0-9]+节\", \"[0-9]{1,2}[\\\\. 、]\", \"[0-9]{1,2}\\\\.[0-9]{1,2}($|[^a-zA-Z/%~.-])\", \"[0-9]{1,2}\\\\.[0-9]{1,2}\\\\.[0-9]{1,2}\"], [\"第[零一二三四五六七八九十百0-9]+章\", \"第[零一二三四五六七八九十百0-9]+节\", \"[零一二三四五六七八九十百]+[ 、]\", \"[\\\\(（][零一二三四五六七八九十百]+[\\\\)）]\", \"[\\\\(（][0-9]{,2}[\\\\)）]\"], [\"PART (ONE|TWO|THREE|FOUR|FIVE|SIX|SEVEN|EIGHT|NINE|TEN)\", \"Chapter (I+V?|VI*|XI|IX|X)\", \"Section [0-9]+\", \"Article [0-9]+\"]], \"method\": \"group\"}, \"Tokenizer:GreatCarsWash\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"Extractor:AutoExtractDefault\": {\"field_name\": \"\", \"auto_keywords\": 0, \"auto_questions\": 0, \"llm_id\": \"\", \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": [], \"tag_file_id\": \"\"}}", + "picture": "{\"File\": {}, \"Parser:ViewsCaptureLight\": {\"image\": {\"output_format\": \"json\", \"parse_method\": \"ocr\", \"preprocess\": [\"main_content\"], \"suffix\": [\"bmp\", \"gif\", \"jpeg\", \"jpg\", \"png\", \"svg\", \"tif\", \"tiff\", \"webp\"]}, \"video\": {\"output_format\": \"text\", \"preprocess\": [\"main_content\"], \"suffix\": [\"3gp\", \"3gpp\", \"avi\", \"flv\", \"mkv\", \"mov\", \"mp4\", \"mpeg\", \"mpg\", \"webm\", \"wmv\"]}}, \"TokenChunker:BrightColorsGlow\": {}, \"Tokenizer:SharpLensFocus\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"Extractor:AutoExtractDefault\": {\"field_name\": \"\", \"auto_keywords\": 0, \"auto_questions\": 0, \"llm_id\": \"\", \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": [], \"tag_file_id\": \"\"}}", + "presentation": "{\"File\": {}, \"Parser:HipSignsRhyme\": {\"pdf\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"pdf\"], \"vlm\": {}}, \"slides\": {\"output_format\": \"json\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"pptx\", \"ppt\"]}}, \"Tokenizer:TallTreesDance\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"PresentationChunker:HappyHillsGlow\": {}, \"Extractor:AutoExtractDefault\": {\"field_name\": \"\", \"auto_keywords\": 0, \"auto_questions\": 0, \"llm_id\": \"\", \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": [], \"tag_file_id\": \"\"}}", "qa": "{\"File\": {}, \"Parser:HipSignsRhyme\": {\"docx\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"docx\"], \"vlm\": {}}, \"markdown\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"md\", \"markdown\", \"mdx\"], \"vlm\": {}}, \"pdf\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"pdf\"], \"vlm\": {}}, \"spreadsheet\": {\"flatten_media_to_text\": false, \"output_format\": \"html\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"xls\", \"xlsx\", \"csv\"], \"vlm\": {}}, \"text&code\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"txt\"]}}, \"Tokenizer:ColdCloudsDream\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}, \"QAChunker:TidyCloudsThink\": {}}", - "resume": "{\"Extractor:ThreeDrinksAct\": {\"field_name\": \"metadata\", \"frequencyPenaltyEnabled\": true, \"frequency_penalty\": 0.7, \"llm_id\": \"THUDM/GLM-4.1V-9B-Thinking@SILICONFLOW\", \"maxTokensEnabled\": false, \"max_tokens\": 256, \"presencePenaltyEnabled\": true, \"presence_penalty\": 0.4, \"prompts\": [{\"content\": \"Content: {TitleChunker:FlatMiceFix@chunks}\", \"role\": \"user\"}], \"sys_prompt\": \"Act as a precise resume metadata extractor. Extract stable, chunk-supported metadata from the provided resume content.\\n\\nRules:\\n1. Use only information explicitly stated in the content. Do not infer, guess, normalize, or add missing facts.\\n2. The input may be only one chunk of a resume. Extract only what this content directly supports.\\n3. Use only these field names:\\ncandidate_name, gender, phone, email, city, location, nationality, linkedin, github, website, highest_degree, degree_levels, school_names, majors, graduation_years, work_experience_years, current_job_title, job_titles, company_names, job_experience, industries, target_job_titles, target_locations, employment_types, skills, certificates, awards, summary_tags\\n4. Ignore detailed responsibilities, project descriptions, achievement narratives, self-evaluation, and other low-value local details.\\n5. Keep values in the same language as the source text whenever possible.\\n6. Remove duplicates and keep only concise, high-value metadata.\\n7. Return only fields that are explicitly supported by the content. Do not return empty or unsupported fields.\\n\\nField guidance:\\n- highest_degree: highest explicit degree level mentioned\\n- degree_levels: all explicit degree levels mentioned\\n- school_names: explicit school, college, or university names\\n- majors: explicit fields of study\\n- graduation_years: explicit graduation years only\\n- work_experience_years: only if explicitly stated\\n- current_job_title: only if explicitly current or most recent\\n- job_titles: explicit role titles\\n- company_names: explicit employer names\\n- job_experience: concise structured work entries explicitly supported by the content, preferably including title, company, and time information when available\\n- industries: explicit industry names only\\n- target_job_titles: explicit desired roles only\\n- target_locations: explicit desired work locations only\\n- skills: concise, core, search-useful skills explicitly mentioned\\n- certificates: explicit certificate names only\\n- awards: explicit award names only\\n- summary_tags: short, high-value tags strictly supported by the content\\n\\nReturn only the extracted metadata. Do not output explanatory text.\", \"temperature\": 0.1, \"temperatureEnabled\": true, \"tenant_llm_id\": 29, \"topPEnabled\": true, \"top_p\": 0.3, \"auto_keywords\": 0, \"auto_questions\": 0, \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": []}, \"File\": {}, \"Parser:HipSignsRhyme\": {\"docx\": {\"flatten_media_to_text\": true, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"docx\"], \"vlm\": {}}, \"pdf\": {\"flatten_media_to_text\": true, \"output_format\": \"json\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"pdf\"], \"vlm\": {}}, \"text&code\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"txt\"]}}, \"TitleChunker:FlatMiceFix\": {\"hierarchy\": 1, \"include_heading_content\": false, \"levels\": [[\"^\\\\s*(?i:(?:\\\\d+[\\\\.\\\\)]\\\\s*)?(?:EDUCATION|ACADEMIC\\\\s*BACKGROUND|ACADEMIC\\\\s*HISTORY|EDUCATIONAL\\\\s*BACKGROUND|RELEVANT\\\\s*COURSEWORK|COURSEWORK|EXPERIENCE|WORK\\\\s*EXPERIENCE|PROFESSIONAL\\\\s*EXPERIENCE|RELEVANT\\\\s*EXPERIENCE|EMPLOYMENT\\\\s*HISTORY|CAREER\\\\s*HISTORY|INTERNSHIP\\\\s*EXPERIENCE|PROJECTS|PROJECT\\\\s*EXPERIENCE|ACADEMIC\\\\s*PROJECTS|PROFESSIONAL\\\\s*PROJECTS|SKILLS|TECHNICAL\\\\s*SKILLS|CORE\\\\s*COMPETENCIES|COMPETENCIES|QUALIFICATIONS|SUMMARY\\\\s*OF\\\\s*QUALIFICATIONS|CERTIFICATIONS|LICENSES|CERTIFICATES|AWARDS|HONORS|HONOURS|ACHIEVEMENTS|PUBLICATIONS|RESEARCH|RESEARCH\\\\s*EXPERIENCE|LEADERSHIP|LEADERSHIP\\\\s*EXPERIENCE|ACTIVITIES|EXTRACURRICULAR\\\\s*ACTIVITIES|ACTIVITIES\\\\s*(?:&|AND)\\\\s*SKILLS|INVOLVEMENT|CAMPUS\\\\s*INVOLVEMENT|VOLUNTEER\\\\s*EXPERIENCE|VOLUNTEERING|COMMUNITY\\\\s*SERVICE|LANGUAGES|INTERESTS|HOBBIES|PROFILE|PROFESSIONAL\\\\s*PROFILE|SUMMARY|PROFESSIONAL\\\\s*SUMMARY|CAREER\\\\s*SUMMARY|OBJECTIVE|CAREER\\\\s*OBJECTIVE|PERSONAL\\\\s*INFORMATION|CONTACT\\\\s*INFORMATION|ADDITIONAL\\\\s*INFORMATION|TRAINING))\\\\s*[:：]?\\\\s*$\"], [\"^\\\\s*(?:\\\\d+[\\\\.、\\\\)]\\\\s*)?(?:教育背景|教育经历|学历背景|学术背景|技术背景|工作经历|工作经验|实习经历|项目经历|项目经验|科研经历|研究经历|校园经历|实践经历|专业经历|职业经历|技能|专业技能|技能特长|核心技能|技术栈|个人技能|工作技能|职业技能|技能与评价|技能与自我评价|工作技能与自我评价|职业技能与自我评价|证书|资格证书|职业资格|资质证书|获奖情况|获奖经历|荣誉|荣誉奖项|奖项|科研成果|论文发表|发表论文|领导经历|学生工作|校园活动|社团经历|活动经历|志愿经历|志愿服务|社会实践|语言能力|语言|自我评价|个人评价|自我总结|个人总结|个人优势|个人简介|个人信息|基本信息|联系方式|求职意向|应聘意向|职业目标|求职目标|兴趣爱好|兴趣特长|培训经历|其他信息|附加信息)\\\\s*[:：]?\\\\s*$\"]], \"method\": \"hierarchy\"}, \"Tokenizer:KindHandsWin\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}}", + "resume": "{\"Extractor:ThreeDrinksAct\": {\"field_name\": \"metadata\", \"frequencyPenaltyEnabled\": true, \"frequency_penalty\": 0.7, \"llm_id\": \"THUDM/GLM-4.1V-9B-Thinking@SILICONFLOW\", \"maxTokensEnabled\": false, \"max_tokens\": 256, \"presencePenaltyEnabled\": true, \"presence_penalty\": 0.4, \"prompts\": [{\"content\": \"Content: {TitleChunker:FlatMiceFix@chunks}\", \"role\": \"user\"}], \"sys_prompt\": \"Act as a precise resume metadata extractor. Extract stable, chunk-supported metadata from the provided resume content.\\n\\nRules:\\n1. Use only information explicitly stated in the content. Do not infer, guess, normalize, or add missing facts.\\n2. The input may be only one chunk of a resume. Extract only what this content directly supports.\\n3. Use only these field names:\\ncandidate_name, gender, phone, email, city, location, nationality, linkedin, github, website, highest_degree, degree_levels, school_names, majors, graduation_years, work_experience_years, current_job_title, job_titles, company_names, job_experience, industries, target_job_titles, target_locations, employment_types, skills, certificates, awards, summary_tags\\n4. Ignore detailed responsibilities, project descriptions, achievement narratives, self-evaluation, and other low-value local details.\\n5. Keep values in the same language as the source text whenever possible.\\n6. Remove duplicates and keep only concise, high-value metadata.\\n7. Return only fields that are explicitly supported by the content. Do not return empty or unsupported fields.\\n\\nField guidance:\\n- highest_degree: highest explicit degree level mentioned\\n- degree_levels: all explicit degree levels mentioned\\n- school_names: explicit school, college, or university names\\n- majors: explicit fields of study\\n- graduation_years: explicit graduation years only\\n- work_experience_years: only if explicitly stated\\n- current_job_title: only if explicitly current or most recent\\n- job_titles: explicit role titles\\n- company_names: explicit employer names\\n- job_experience: concise structured work entries explicitly supported by the content, preferably including title, company, and time information when available\\n- industries: explicit industry names only\\n- target_job_titles: explicit desired roles only\\n- target_locations: explicit desired work locations only\\n- skills: concise, core, search-useful skills explicitly mentioned\\n- certificates: explicit certificate names only\\n- awards: explicit award names only\\n- summary_tags: short, high-value tags strictly supported by the content\\n\\nReturn only the extracted metadata. Do not output explanatory text.\", \"temperature\": 0.1, \"temperatureEnabled\": true, \"tenant_llm_id\": 29, \"topPEnabled\": true, \"top_p\": 0.3, \"auto_keywords\": 0, \"auto_questions\": 0, \"auto_tags\": 0, \"enable_metadata\": 0, \"metadata\": [], \"tag_file_id\": \"\"}, \"File\": {}, \"Parser:HipSignsRhyme\": {\"docx\": {\"flatten_media_to_text\": true, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"docx\"], \"vlm\": {}}, \"pdf\": {\"flatten_media_to_text\": true, \"output_format\": \"json\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"pdf\"], \"vlm\": {}}, \"text&code\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"txt\"]}}, \"TitleChunker:FlatMiceFix\": {\"hierarchy\": 1, \"include_heading_content\": false, \"levels\": [[\"^\\\\s*(?i:(?:\\\\d+[\\\\.\\\\)]\\\\s*)?(?:EDUCATION|ACADEMIC\\\\s*BACKGROUND|ACADEMIC\\\\s*HISTORY|EDUCATIONAL\\\\s*BACKGROUND|RELEVANT\\\\s*COURSEWORK|COURSEWORK|EXPERIENCE|WORK\\\\s*EXPERIENCE|PROFESSIONAL\\\\s*EXPERIENCE|RELEVANT\\\\s*EXPERIENCE|EMPLOYMENT\\\\s*HISTORY|CAREER\\\\s*HISTORY|INTERNSHIP\\\\s*EXPERIENCE|PROJECTS|PROJECT\\\\s*EXPERIENCE|ACADEMIC\\\\s*PROJECTS|PROFESSIONAL\\\\s*PROJECTS|SKILLS|TECHNICAL\\\\s*SKILLS|CORE\\\\s*COMPETENCIES|COMPETENCIES|QUALIFICATIONS|SUMMARY\\\\s*OF\\\\s*QUALIFICATIONS|CERTIFICATIONS|LICENSES|CERTIFICATES|AWARDS|HONORS|HONOURS|ACHIEVEMENTS|PUBLICATIONS|RESEARCH|RESEARCH\\\\s*EXPERIENCE|LEADERSHIP|LEADERSHIP\\\\s*EXPERIENCE|ACTIVITIES|EXTRACURRICULAR\\\\s*ACTIVITIES|ACTIVITIES\\\\s*(?:&|AND)\\\\s*SKILLS|INVOLVEMENT|CAMPUS\\\\s*INVOLVEMENT|VOLUNTEER\\\\s*EXPERIENCE|VOLUNTEERING|COMMUNITY\\\\s*SERVICE|LANGUAGES|INTERESTS|HOBBIES|PROFILE|PROFESSIONAL\\\\s*PROFILE|SUMMARY|PROFESSIONAL\\\\s*SUMMARY|CAREER\\\\s*SUMMARY|OBJECTIVE|CAREER\\\\s*OBJECTIVE|PERSONAL\\\\s*INFORMATION|CONTACT\\\\s*INFORMATION|ADDITIONAL\\\\s*INFORMATION|TRAINING))\\\\s*[:：]?\\\\s*$\"], [\"^\\\\s*(?:\\\\d+[\\\\.、\\\\)]\\\\s*)?(?:教育背景|教育经历|学历背景|学术背景|技术背景|工作经历|工作经验|实习经历|项目经历|项目经验|科研经历|研究经历|校园经历|实践经历|专业经历|职业经历|技能|专业技能|技能特长|核心技能|技术栈|个人技能|工作技能|职业技能|技能与评价|技能与自我评价|工作技能与自我评价|职业技能与自我评价|证书|资格证书|职业资格|资质证书|获奖情况|获奖经历|荣誉|荣誉奖项|奖项|科研成果|论文发表|发表论文|领导经历|学生工作|校园活动|社团经历|活动经历|志愿经历|志愿服务|社会实践|语言能力|语言|自我评价|个人评价|自我总结|个人总结|个人优势|个人简介|个人信息|基本信息|联系方式|求职意向|应聘意向|职业目标|求职目标|兴趣爱好|兴趣特长|培训经历|其他信息|附加信息)\\\\s*[:：]?\\\\s*$\"]], \"method\": \"hierarchy\"}, \"Tokenizer:KindHandsWin\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}}", "table": "{\"File\": {}, \"Parser:HipSignsRhyme\": {\"spreadsheet\": {\"flatten_media_to_text\": false, \"output_format\": \"html\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"xls\", \"xlsx\", \"csv\"], \"vlm\": {}, \"column_mode\": \"auto\", \"column_roles\": {}, \"column_names\": []}, \"text&code\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"txt\"]}}, \"TableChunker:FastFoxesJump\": {}, \"Tokenizer:DeepLakesShine\": {\"fields\": \"text\", \"filename_embd_weight\": 0.1, \"search_method\": [\"embedding\", \"full_text\"]}}", "knowledge_compiler": "{\"File\": {}, \"KnowledgeCompiler:KnownSwiftLions\": {\"language\": \"English\", \"variant\": \"structure\"}, \"Parser:HipSignsRhyme\": {\"setups\": {\"doc\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"doc\"]}, \"docx\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"docx\"], \"vlm\": {}}, \"html\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"htm\", \"html\"]}, \"markdown\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"md\", \"markdown\", \"mdx\"], \"vlm\": {}}, \"pdf\": {\"flatten_media_to_text\": false, \"output_format\": \"json\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"pdf\"], \"vlm\": {}}, \"spreadsheet\": {\"flatten_media_to_text\": false, \"output_format\": \"html\", \"parse_method\": \"DeepDOC\", \"preprocess\": [\"main_content\"], \"suffix\": [\"xls\", \"xlsx\", \"csv\"], \"vlm\": {}}, \"text&code\": {\"output_format\": \"json\", \"preprocess\": [\"main_content\"], \"suffix\": [\"txt\", \"py\", \"js\", \"java\", \"c\", \"cpp\", \"h\", \"php\", \"go\", \"ts\", \"sh\", \"cs\", \"kt\", \"sql\"]}}}, \"TokenChunker:SixApplesFall\": {\"children_delimiters\": [], \"chunk_token_size\": 512, \"delimiter_mode\": \"token_size\", \"delimiters\": [\"\\n\", \"!\", \"?\", \"。\", \"；\", \"！\", \"？\"], \"image_context_size\": 0, \"overlapped_percent\": 0, \"table_context_size\": 0}}", } diff --git a/web/src/pages/agent/form/extractor-form/index.tsx b/web/src/pages/agent/form/extractor-form/index.tsx index e296adda31..599e06f097 100644 --- a/web/src/pages/agent/form/extractor-form/index.tsx +++ b/web/src/pages/agent/form/extractor-form/index.tsx @@ -81,7 +81,7 @@ const ExtractorForm = ({ const ownerTenantId = useOwnerTenantId(); - const { treeData, loadData } = useTagFileTree(); + const { treeData, loadData } = useTagFileTree(form.watch('tag_file_id')); return (
diff --git a/web/src/pages/agent/form/extractor-form/use-tag-file-tree.ts b/web/src/pages/agent/form/extractor-form/use-tag-file-tree.ts index 7ddcfa3b74..8cc7896a9a 100644 --- a/web/src/pages/agent/form/extractor-form/use-tag-file-tree.ts +++ b/web/src/pages/agent/form/extractor-form/use-tag-file-tree.ts @@ -2,9 +2,10 @@ import { TreeNodeType } from '@/components/ui/async-tree-select'; import { useFetchPureFileList } from '@/hooks/use-file-request'; import { IFile } from '@/interfaces/database/file-manager'; import { isFolderType } from '@/pages/files/util'; +import fileManagerService from '@/services/file-manager-service'; import { getExtension } from '@/utils/document-util'; import { uniqBy } from 'lodash'; -import { useCallback, useState } from 'react'; +import { useCallback, useEffect, useRef, useState } from 'react'; const AllowedExtensions = ['xlsx', 'xls', 'csv', 'txt']; @@ -12,41 +13,77 @@ export function canSelectTagFile(node: TreeNodeType) { return Boolean(node.isLeaf); } -export function useTagFileTree() { +function toTreeNode(x: IFile): TreeNodeType { + return { + id: x.id, + parentId: x.parent_id, + title: x.name, + // has_child_folder only counts child folders, so folders + // must always be expandable to reach files inside them + isLeaf: !isFolderType(x.type), + }; +} + +export function useTagFileTree(value?: string) { const { fetchList } = useFetchPureFileList(); const [treeData, setTreeData] = useState([]); + const appendFiles = useCallback((files: IFile[]) => { + setTreeData((tree) => + uniqBy( + tree.concat( + files + .filter( + (x) => + (isFolderType(x.type) && x.name.toLowerCase() !== 'skills') || + AllowedExtensions.includes(getExtension(x.name)), + ) + .map(toTreeNode), + ), + 'id', + ), + ); + }, []); + const loadData = useCallback( async ({ id }: TreeNodeType) => { const ret = await fetchList(id as string); if (ret.code === 0) { - setTreeData((tree) => - uniqBy( - tree.concat( - ret.data.files - .filter( - (x: IFile) => - (isFolderType(x.type) && - x.name.toLowerCase() !== 'skills') || - AllowedExtensions.includes(getExtension(x.name)), - ) - .map((x: IFile) => ({ - id: x.id, - parentId: x.parent_id, - title: x.name, - // has_child_folder only counts child folders, so folders - // must always be expandable to reach files inside them - isLeaf: !isFolderType(x.type), - })), - ), - 'id', - ), - ); + appendFiles(ret.data.files); } }, - [fetchList], + [fetchList, appendFiles], ); + // Resolve a previously saved selection: the tree is lazy-loaded, so a file + // nested in subfolders is not in treeData on mount and its title cannot be + // displayed. Walk the ancestor chain top-down and load each level. + const resolvedValueRef = useRef(''); + useEffect(() => { + if ( + !value || + resolvedValueRef.current === value || + treeData.some((x) => x.id === value) + ) { + return; + } + resolvedValueRef.current = value; + (async () => { + const { data } = await fileManagerService.getAllParentFolder( + {}, + `${value}/ancestors`, + ); + const folders: IFile[] = data?.data?.parent_folders?.toReversed() ?? []; + setTreeData((tree) => uniqBy(tree.concat(folders.map(toTreeNode)), 'id')); + for (const folder of folders) { + const ret = await fetchList(folder.id); + if (ret.code === 0) { + appendFiles(ret.data.files); + } + } + })(); + }, [value, treeData, fetchList, appendFiles]); + return { treeData, loadData }; } diff --git a/web/src/pages/dataset/compilation/update-button.tsx b/web/src/pages/dataset/compilation/update-button.tsx index 3a189f8c89..66dfcdb78a 100644 --- a/web/src/pages/dataset/compilation/update-button.tsx +++ b/web/src/pages/dataset/compilation/update-button.tsx @@ -10,10 +10,7 @@ import { TooltipTrigger, } from '@/components/ui/tooltip'; import { GenerateStatus, GenerateType } from '@/constants/knowledge'; -import { - ITraceInfo, - useGenerateStatus, -} from '@/hooks/use-dataset-generate'; +import { ITraceInfo, useGenerateStatus } from '@/hooks/use-dataset-generate'; import { UpdateRunProgress } from './update-run-progress'; diff --git a/web/src/pages/dataset/compilation/wiki-left-panel/hooks/use-wiki-update.ts b/web/src/pages/dataset/compilation/wiki-left-panel/hooks/use-wiki-update.ts index bb3dc6fe99..71a4edb7ce 100644 --- a/web/src/pages/dataset/compilation/wiki-left-panel/hooks/use-wiki-update.ts +++ b/web/src/pages/dataset/compilation/wiki-left-panel/hooks/use-wiki-update.ts @@ -10,7 +10,8 @@ type UseWikiUpdateOptions = { export function useWikiUpdate({ onUpdate }: UseWikiUpdateOptions = {}) { const { data, loading: queryLoading } = useFetchArtifactAlteration('wiki'); - const { runArtifactIndex, loading: mutationLoading } = useRunArtifactIndex('wiki'); + const { runArtifactIndex, loading: mutationLoading } = + useRunArtifactIndex('wiki'); const newlyUploaded = data?.newly_uploaded ?? 0; const removed = data?.removed ?? 0; diff --git a/web/src/pages/dataset/compilation/wiki-left-panel/index.tsx b/web/src/pages/dataset/compilation/wiki-left-panel/index.tsx index e1256f5c7c..2e27f5b1bb 100644 --- a/web/src/pages/dataset/compilation/wiki-left-panel/index.tsx +++ b/web/src/pages/dataset/compilation/wiki-left-panel/index.tsx @@ -2,10 +2,7 @@ import { ConfirmDeleteDialog } from '@/components/confirm-delete-dialog'; import { Button } from '@/components/ui/button'; import { Tabs, TabsList, TabsTrigger } from '@/components/ui/tabs'; import { GenerateStatus, GenerateType } from '@/constants/knowledge'; -import { - ITraceInfo, - useGenerateStatus, -} from '@/hooks/use-dataset-generate'; +import { ITraceInfo, useGenerateStatus } from '@/hooks/use-dataset-generate'; import { IArtifact } from '@/interfaces/database/dataset'; import { Trash2 } from 'lucide-react'; import { useCallback } from 'react'; diff --git a/web/src/pages/next-chats/chat/chat-box/next-multiple-chat-box.tsx b/web/src/pages/next-chats/chat/chat-box/next-multiple-chat-box.tsx index a311d6e1e1..22cf808fd4 100644 --- a/web/src/pages/next-chats/chat/chat-box/next-multiple-chat-box.tsx +++ b/web/src/pages/next-chats/chat/chat-box/next-multiple-chat-box.tsx @@ -138,7 +138,13 @@ const ChatCard = forwardRef(function ChatCard( // resend with the card's model settings (llm_id, temperature, ...). const sendCardMessage = useCallback( ({ message, messages }: { message: IMessage; messages?: IMessage[] }) => - sendMessage({ message, messages, ...form.getValues(), storeHistoryMessages: false, omitSessionId: true }), + sendMessage({ + message, + messages, + ...form.getValues(), + storeHistoryMessages: false, + omitSessionId: true, + }), [sendMessage, form], ); @@ -181,7 +187,12 @@ const ChatCard = forwardRef(function ChatCard( useImperativeHandle( ref, (): HandlePressEnterType => (params) => - handlePressEnter({ ...params, ...form.getValues(), storeHistoryMessages: false, omitSessionId: true }), + handlePressEnter({ + ...params, + ...form.getValues(), + storeHistoryMessages: false, + omitSessionId: true, + }), ); useEffect(() => {