### What problem does this PR solve?
Before migration
Web API: POST /v1/document/metadata/update
After migration, Restful API
PATCH /api/v2/datasets/<dataset_id>/documents/metadatas
### Type of change
- [x] Refactoring
### What problem does this PR solve?
Fix: Recall Test Page Metadata Not Displaying.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Before migration
Web API: POST /v1/document/update_metadata_setting
After consolidation, Restful API
PUT
/api/v1/datasets/<dataset_id>/documents/<document_id>/metadata/config
### Type of change
- [x] Refactoring
### What problem does this PR solve?
Fix: The number of chunks in the file list is not displayed.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Before consolidation
Web API: POST /v1/document/list
Http API - GET /api/v1/datasets/<dataset_id>/documents
After consolidation, Restful API -- GET
/api/v1/datasets/<dataset_id>/documents
### Type of change
- [x] Refactoring
Closes#9078
### What problem does this PR solve?
The `retrieval_test` endpoint in `chunk_app.py` never forwarded the
`highlight` request parameter to `retriever.retrieval()`, so the search
engine never produced highlight snippets. Additionally, the frontend
always rendered `content_with_weight` instead of preferring the
`highlight` field, and the CSS rule color `var(--accent-primary)` didn't
work because the variable stores an RGB triplet `(45,212,191)` requiring
the `rgb()` wrapper.
### Before
- Search page: displayed raw content_with_weight as a wall of plain
white text with no term highlighting, including markdown headings
rendered as literal text
- Retrieval testing page: showed `content_with_weight` in a plain `<p>`
tag, no `<em>` tags rendered, no highlight coloring
- Children chunks: when child chunks were consolidated into a parent via
`retrieval_by_children`, any highlight data from children was discarded
- TOC chunks: chunks fetched via `retrieval_by_toc` had no `highlight`
field, appearing as plain text while other chunks had highlights
**Retrieval testing**:
<img width="1449" height="1178"
alt="before-retrieval-no-highlight-cropped"
src="https://github.com/user-attachments/assets/5c6f5a5e-6c11-461a-bdb4-049d7dfb7a33"
/>
**Search**:
<img width="1378" height="711" alt="before-search-no-highlight-cropped"
src="https://github.com/user-attachments/assets/be7b5152-72ef-40da-a8fd-921e997ae7d3"
/>
### After
- Search page: displays the highlight field with search terms rendered
in teal/cyan color (`rgb(var(--accent-primary))`)
- Retrieval testing page: sends highlight: true in the request, uses
`HighLightMarkdown` component to render `<em>` tags with proper coloring
- Children chunks: highlights from child chunks are joined and preserved
on the parent
- TOC chunks: when other chunks have highlights, TOC-fetched chunks use
`content_with_weight` as a highlight fallback
**Retrieval testing**:
<img width="1410" height="1015" alt="05-retrieval-testing-results"
src="https://github.com/user-attachments/assets/f0cff8cf-0962-4320-b559-cd5037f622d2"
/>
**Search**:
<img width="1294" height="455" alt="03-search-highlight-results"
src="https://github.com/user-attachments/assets/a90e0e3e-3837-46be-8ddd-2412ff7cbc19"
/>
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Consolidation WEB API & HTTP API for document upload
Before consolidation
Web API: POST /v1/document/upload
Http API - POST /api/v1/datasets/<dataset_id>/documents
After consolidation, Restful API -- POST
/api/v1/datasets/<dataset_id>/documents
### Type of change
- [x] Refactoring
### What problem does this PR solve?
1. Remove unused token related API
2. Fix typo
### Type of change
- [x] Refactoring
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fix: The file count in the file header did not change after uploading or
deleting files.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
Fix: When creating a dataset, if no `chunk_method` is selected, there is
no indication that this is a required field.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Refactor: merge document.rename into document.update_document
### Type of change
- [x] Refactoring
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a unified document update API (PUT) supporting name, metadata,
parser/chunk settings, and status changes.
* **Breaking Changes**
* Legacy single-parameter rename endpoint removed; renames now require
dataset + document identifiers.
* `/list` now reads dataset id from a different query parameter.
* **Validation / Bug Fixes**
* Stricter meta_fields and parser-config validation; unauthenticated
requests return 401.
* **Frontend**
* UI now sends dataset id when saving document names.
* **Tests**
* Numerous unit and HTTP tests adjusted or removed to match new API and
validations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
Co-authored-by: MkDev11 <94194147+MkDev11@users.noreply.github.com>
Co-authored-by: mkdev11 <YOUR_GITHUB_ID+MkDev11@users.noreply.github.com>
Co-authored-by: mkdev11 <MkDev11@users.noreply.github.com>
Co-authored-by: Qi Wang <wangq8@outlook.com>
Co-authored-by: dataCenter430 <161712630+dataCenter430@users.noreply.github.com>
Co-authored-by: balibabu <cike8899@users.noreply.github.com>
### What problem does this PR solve?
Refactor: Remove unused API code
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Style**
* Updated table header styling in dataset settings by removing a
hard-coded background color class, allowing the header to use default or
inherited component styling instead.
* **Refactor**
* Removed token management endpoints from the API service. Token
creation, listing, and removal functions are no longer available.
* Removed the statistics data endpoint from available API routes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
### What problem does this PR solve?
Fix: Linter error message: Use 'const' instead.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Updated variable declarations across form components, agent utilities,
memory management hooks, and data handling functions to enhance code
consistency and maintainability throughout the application codebase.
* **Style**
* Added ESLint suppressions to document intentional constant-condition
patterns in asynchronous event streaming operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Problem Description
When a user creates Dataset A using the **Tag parser** (for CSV/Excel
files with tag definitions), and then creates Dataset B, the Tag Sets
dropdown in Dataset B's Configuration page cannot display Dataset A.
### Steps to Reproduce
1. Create Dataset A with **Tag** as the chunking method
2. Upload a CSV file to Dataset A to generate tags
3. Create Dataset B
4. Navigate to Dataset B → Configuration → Tag Sets
5. **Expected**: Dataset A should appear in the dropdown
6. **Actual**: The dropdown is empty, Dataset A is not visible
---
## Root Cause Analysis
After thorough code review, **the original code logic is correct**. The
`chunk_method` field flows properly through the system:
### Data Flow
```mermaid
sequenceDiagram
participant Frontend
participant Pydantic
participant API
participant Database
Note over Frontend,Database: Creating a Tag Dataset
Frontend->>Pydantic: POST {chunk_method: "tag"}
Pydantic->>API: serialization_alias converts<br/>chunk_method → parser_id
API->>Database: INSERT {parser_id: "tag"}
Note over Frontend,Database: Querying Datasets
Frontend->>API: GET /api/v1/datasets
API->>Database: SELECT parser_id, ...
Database-->>API: Returns {parser_id: "tag"}
API->>API: remap_dictionary_keys()<br/>parser_id → chunk_method
API-->>Frontend: {chunk_method: "tag"}
Note over Frontend: Filter: x.chunk_method === 'tag'
Note over Frontend: ✅ Match found!
```
### Field Mapping
**Location**: `api/utils/api_utils.py:657-662`
```python
DEFAULT_KEY_MAP = {
"chunk_num": "chunk_count",
"doc_num": "document_count",
"parser_id": "chunk_method", # Maps DB field to API response
"embd_id": "embedding_model",
}
```
### Frontend Filtering (Already Correct)
**Location**:
`web/src/pages/dataset/dataset-setting/components/tag-item.tsx:24`
```typescript
const knowledgeOptions = knowledgeList
.filter((x) => x.chunk_method === 'tag') // ✅ Correct field
.map((x) => ({...}));
```
---
## Actual Issue
The most likely causes for the "bug" are:
1. **Browser Cache**: Old data cached before proper deployment
2. **Stale Data**: Datasets created before the code was fully deployed
3. **Container Not Restarted**: Changes not applied to running container
---
## Resolution
**No code changes are needed.** The existing code correctly:
1. Accepts `chunk_method` from frontend
2. Converts to `parser_id` via Pydantic serialization_alias
3. Stores in database as `parser_id`
4. Maps back to `chunk_method` in API response
5. Frontend filters by `chunk_method === 'tag'`
### What problem does this PR solve?
Fix: The chunk method of the knowledge base cannot be saved.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix: Fix the issue of errors when creating datasets.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
Fix: Metadata,chunk,dataset Related bugs
- metadata not show add button #13731
- chunk edit question style
- dataset modified chunk method bug
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix: The retrieval_test interface is continuously requested when the
user enters a question. #13719
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
When using pagination in the Dataset file list or File Manager,
selecting row N on page 1 would incorrectly cause row N on page 2 (and
subsequent pages) to also appear selected. This is a state pollution
bug.
### Root Cause
TanStack React Table defaults to using array indices (0, 1, 2...) as
`rowSelection` keys. With server-side (manual) pagination, each page's
rows start from index 0, so a selection like `{2: true}` on page 1 also
matches index 2 on every other page.
### Fix
- Added `getRowId: (row) => row.id` to `useReactTable` in both
`DatasetTable` and `FilesTable`, so selection state is keyed by unique
document/file IDs instead of positional indices.
- Updated the `useSelectedIds` helper to support ID-based selection keys
while maintaining backward compatibility with index-based keys.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Files Changed
| File | Change |
|------|--------|
| `web/src/pages/dataset/dataset/dataset-table.tsx` | Added `getRowId`
to table config |
| `web/src/pages/files/files-table.tsx` | Added `getRowId` to table
config |
| `web/src/hooks/logic-hooks/use-row-selection.ts` | Updated
`useSelectedIds` to handle ID-based selection |
### What problem does this PR solve?
1. Split dataset api to gateway and service, and modify web UI to use
restful http api.
2. Old KB releated APIs are commented.
### Type of change
- [x] Refactoring
---------
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
Fix: The dataset description should not be a required field.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
1. Split dataset api to gateway and service, and modify web UI to use
restful http api.
2. Old KB releated APIs are commented.
### Type of change
- [x] Refactoring
### What problem does this PR solve?
1. Split dataset api to gateway and service, and modify web UI to use
restful http api.
2. Old KB releated APIs are commented.
### Type of change
- [x] Refactoring
### What problem does this PR solve?
min value and message force users to input a descript in datasets. Also
had a wrong error message.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
- Unify top level pages structure
- Standardize locale language codes (BCP 47) and time zones (IANA tz)
> **Note:**
> Newly created user info brings non-standard default values `timezone:
"UTC+8\tAsia/Shanghai"` and `language: "English"`.
### Type of change
- [x] Refactoring
### What problem does this PR solve?
- Adjust UI styles in **Dataset** pages.
- Adjust several shared components styles
- Modify files and directory structure in `src/layouts`
### Type of change
- [x] Refactoring
### What problem does this PR solve?
Renovate global navigation bar, align styles to the design.
(May causes minor layout issues in sub-pages, will check and fix soon)
### Type of change
- [x] Refactoring
### What problem does this PR solve?
Fix: UI Placeholder and Hint Optimization
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
This PR adds end-to-end Arabic support in production. It also adds a
full Arabic README
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Documentation Update
### What problem does this PR solve?
Necessary ids for implementing the new testing suite with playwright for
UI
### Type of change
- [x] Other (please describe): Testing IDs
Co-authored-by: Liu An <asiro@qq.com>
### What problem does this PR solve?
Fix: Metadata mult-selected display error
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._
### Type of change
- [x] Refactoring
### What problem does this PR solve?
Fix: replace session page icons and fix nested list search functionality
in filters
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
…, clicking "Parse" will still ask if you want to clear the chunks of
the already parsed files.
### What problem does this PR solve?
Fix: After selecting all and then unchecking the already parsed files,
clicking "Parse" will still ask if you want to clear the chunks of the
already parsed files.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix:Optimize metadata and optimize the empty state style of the agent
page.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
This PR updates and extends the german language support in the frontend.
Additionally two more elements are handled dynamically now. The
interactive Agent is also titled and described in german now.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: Jakob <16180662+hauberj@users.noreply.github.com>
### What problem does this PR solve?
Fix: Issues with metadata parameter addition failures and single-file
chunk saving failures.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
