### Summary
Aligns Go and Python error codes/messages so both backends honor the
same RESTful API contract, removing implementation-specific error leaks
(MySQL errors, `ValueError`, `AttributeError`, Gin validator format) in
favor of clean business error codes.
**Chat list** — invalid `orderby` now returns code 101 (was: raw Python
`AttributeError` code 100); invalid `page`/`page_size` values fall back
to defaults (was: raw `ValueError`/`ProgrammingError` code 100).
**Dataset create/update/delete** — adds UUID validation (101),
extra-field rejection (101), duplicate-id detection (101), content-type
/ JSON-syntax / object-shape checks (101), and "lacks permission" for
nonexistent datasets (IDOR). Create auto-deduplicates dataset names.
Pagerank updates tolerate a missing ES index. List response includes
`parser_config` and `pagerank`.
**Session list/update** — adds filtering, sorting, and pagination
support. Empty payloads are valid no-ops. Authorization errors map to
code 109.
**Chunk list** — doc object uses Python key names (`chunk_count`,
`dataset_id`, `chunk_method`, run text status). Add validates list
element types.
**Document update** — adds `chunk_method` alias, pydantic-style Field
error messages, metadata index auto-create with refresh, and "These
documents do not belong to dataset" messages. List validates
`metadata_condition` and reports ownership errors for unmatched name/id
filters.
**Search completion** — `kb_ids` ownership failure returns code 102
instead of 109.
Released 31 contract tests from `GO_ONLY_SKIPS` (all verified passing on
both Go and Python backends with real LLM keys).
## Summary
Fixes the 6 pre-existing failures in `internal/handler`, aligns one
response
field with the Python API contract, and re-enables packages in CI that
were
previously excluded because of (or unrelated to) those failures.
### Test/handler fixes
- **plugin.go**: return `"success"` (lowercase) instead of `"SUCCESS"`
so the
response envelope matches the Python backend, keeping backend-swap
transparent.
- **search_handler_test.go**: expect lowercase `"no authorization"` to
match the
service error message, which mirrors Python.
- **agent_test.go**: seed versions with `CreateTime` instead of
`UpdateTime` so
`ListVersions` ordering (`create_time DESC`) is exercised correctly.
- **agent_wait_for_user_test.go**: a clean run may emit only the
`[DONE]` frame;
relax the SSE assertion to require a non-empty stream ending in
`[DONE]`.
- **bot_test.go**: align attachment-download assertions with actual
behavior
(`Content-Disposition: attachment; filename="file"`, default
`application/octet-stream`), matching Python
`resolve_attachment_content_type`.
### CI
- **tests.yml / sep-tests.yml**:
- Remove the `grep -v '/internal/handler$'` filter — the 6 failures that
motivated it are now fixed.
- Remove the `grep -v '/internal/storage$'` filter — `internal/storage`
tests self-skip via `t.Skipf` when MinIO is unavailable, so the package
is
safe to run in CI.
- Remove the `grep -v '/internal/agent$'` filter (only present in the
tests.yml infinity job) — the exclusion was undocumented and
inconsistent
with the other jobs that already run `internal/agent`.
- Keep the `internal/tokenizer` exclusion: it is a genuine environmental
dependency (dict files at `/usr/share/infinity/resource`, absent in the
Go
test environment).
## Test plan
- `build.sh --test ./internal/handler/` is green (previously 6 FAIL).
- `build.sh --test ./internal/storage/ ./internal/agent/` should pass;
the
MinIO-backed `internal/storage` tests skip gracefully without a MinIO
server.
- After this PR, `internal/handler`, `internal/storage`, and
`internal/agent`
run again in CI unit-test jobs; `internal/tokenizer` stays excluded.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Ports doc-level auto-metadata extraction to Go and adds the
knowledge_compiler component with scheduler/routing. Fixes Extractor
metadata injection type assertion and enable_metadata default-on.
## Summary
Adds page-range parsing support to the Go-native pipeline path and
introduces strict `parse_type` validation for both dataset and document
update endpoints.
## What changed
### Pages range parsing
- **`internal/utility/pdf_pages.go`** — `NormalizePDFPages`: normalizes
raw page ranges (list of `[from,to]` 1-indexed inclusive ranges) into
sorted, merged, deduplicated `[][]int`. Invalid ranges are dropped.
- **`internal/ingestion/pipeline/pdf_pages.go`** —
`NormalizeParserConfigPages`: walks any parser_config map and normalizes
`"pages"` values under every component → filetype setup, so the
persisted config always carries clean, merged ranges.
- **`internal/deepdoc/parser/pdf/parser.go`** — integrates
`resolvePagesToProcess` to filter parsed PDF pages by the configured
ranges.
- Pipeline integration (parser pages):
`internal/parser/parser/pdf_parser_common.go`, `chunk_process.go`, plus
associated e2e and unit tests.
### Parse type validation (shared logic)
- **`internal/service/parser_mode.go`** (new) — `ValidateParseTypeMode`:
shared function that validates `parse_type` (1=BuiltIn/parser_id,
2=Pipeline/pipeline_id) and ensures the corresponding field is present.
Used by both dataset and document update endpoints.
- **`internal/service/dataset/crud.go`** / `update.go` — replaces inline
`isPipelineMode`/`isBuiltinMode` computation with the shared
`service.ValidateParseTypeMode`.
- **`internal/service/document/document_dataset_update.go`** — adds
strict `parse_type` validation in `validateDatasetDocumentUpdate`,
simplifies the reparse logic to a two-way switch (isBuiltin/isPipeline)
now that parse_type is always valid.
- **`internal/service/document/document.go`** — adds `ParseType` field
to `UpdateDatasetDocumentRequest`.
- **`internal/service/document/document_dataset_update.go`** —
`updateDocumentParserConfig` fallback path when DSL loading fails.
- **`internal/service/parser_mode_test.go`** (new) — test coverage for
nil, invalid, and missing-field scenarios.
### Frontend
- **`web/src/interfaces/request/document.ts`** — adds `parseType` to
`IChangeParserRequestBody`.
- **`web/src/hooks/use-document-request.ts`** —
`useSetDocumentPipelineParser` sends `parse_type` in the PATCH payload.
- **`web/src/pages/dataset/dataset/use-change-document-parser.ts`** —
Go/Python branching for the document parser config dialog.
-
**`web/src/components/document-pipeline-dialog/use-document-pipeline-form.ts`**
— `buildSubmitData` returns `parseType` (bugfix: was dropped from the
return value).
### Test changes
- **Removed**: 2 tests that verified the old "mutually exclusive" error
(replaced by `ValidateParseTypeMode` coverage).
- **Modified**: 6 tests across document and dataset packages to include
`ParseType` in request structs.
- **Added**: new e2e tests for pages parsing (`pages_e2e_test.go`,
`pdf_parser_pages_e2e_test.go`) and unit tests for `NormalizePDFPages`,
`NormalizeParserConfigPages`, `resolvePagesToProcess`.
## Backward compatibility
- The `parse_type` field is **required** when `parser_id` or
`pipeline_id` is sent. This changes the contract for both dataset and
document PATCH endpoints, but aligns the Go backend with the existing
frontend behavior (the frontend already sends `parse_type`). Callers
that omit `parse_type` when updating parser/pipeline selections will
receive a clear error message.
- Existing callers that only update fields like `name`, `enabled`, or
`meta_fields` are unaffected.
- Test updates ensure all known call sites are compliant.
