## Problem
During ingestion, `indexdoc.ProcessChunksForPipeline` stamped
`ck["kb_id"]`
on every chunk. This was both:
- **a dead write** — `elasticsearch.InsertChunks` unconditionally
overwrites
the value with `datasetID` (`chunk.go:211`), so the producer's value
never
reached the index;
- **the wrong shape** — it was emitted as `[]string`, while both engines
actually need a single string.
This is the `kb_id` slice of the ingestion -> engine schema leak tracked
in
#17371: ingestion was carrying index-physical schema knowledge it should
not
own.
## Fix
Make the search engines the single owner of `kb_id` at the write
boundary,
and stop ingestion from emitting it:
- **Elasticsearch** (`chunk.go:211`) already sets `docCopy["kb_id"] =
datasetID`
— unchanged.
- **Infinity** (`chunk.go`) `InsertChunks` now stamps
`insertChunks[i]["kb_id"] = datasetID` right after
`transformChunkFields`
(previously it only *read/normalized* the producer value, which forced
ingestion to supply it). Both engines are now consistent.
- `ProcessChunksForPipeline` no longer stamps `kb_id` and the now-leaky
`kbID` parameter is removed. The same removal is propagated to
`ProcessPipelineOutputForGolden` and the `compare_pipeline_golden` dev
tool
(its `-kb-id` flag is dropped).
The stored `kb_id` value is byte-for-byte unchanged: `datasetID` passed
to
`InsertChunks` is `taskCtx.Doc.KbID`, i.e. the same id that was
previously
set on the producer chunk.
## Verification
- `bash build.sh --test ./internal/ingestion/task/indexdoc/...
./internal/engine/infinity/...`
— both green.
- `internal/ingestion/task` has **two pre-existing** failures
(`TestPipelineExecutor_Run_RealCanvasDSL_UsesGeneralPipeline`,
`TestRunPipeline_RealPipelineOutput_ProducesIndexFields`) that assert
`inserted chunk count = 1, want 2` — a parser/assertion mismatch (the Go
parser merges the 2-paragraph fixture into 1 chunk). They are unrelated
to
this change, which never touches chunk counting. The `kb_id`-related
test
failure this change would otherwise introduce is fixed by updating the
tests
below.
- Updated the pinning unit test:
`TestProcessChunksForPipeline_SetsDocID`
(formerly `...SetsDocIDAndKBID`) now asserts `kb_id` is **not** set by
the
producer. Removed the `kb_id` assertion and the now-dead
`taskChunkFieldEqualsStr` helper from
`pipeline_real_integration_test.go`.
## Scope
This closes only the `kb_id` portion of #17371. The remaining
index-physical
fields (`docnm_kwd`, `create_timestamp_flt`, `page_num_int`/`top_int`/
`position_int`, etc.) are intentionally left for a follow-up (P2).
### Summary
Aligns Go and Python error codes/messages so both backends honor the
same RESTful API contract, removing implementation-specific error leaks
(MySQL errors, `ValueError`, `AttributeError`, Gin validator format) in
favor of clean business error codes.
**Chat list** — invalid `orderby` now returns code 101 (was: raw Python
`AttributeError` code 100); invalid `page`/`page_size` values fall back
to defaults (was: raw `ValueError`/`ProgrammingError` code 100).
**Dataset create/update/delete** — adds UUID validation (101),
extra-field rejection (101), duplicate-id detection (101), content-type
/ JSON-syntax / object-shape checks (101), and "lacks permission" for
nonexistent datasets (IDOR). Create auto-deduplicates dataset names.
Pagerank updates tolerate a missing ES index. List response includes
`parser_config` and `pagerank`.
**Session list/update** — adds filtering, sorting, and pagination
support. Empty payloads are valid no-ops. Authorization errors map to
code 109.
**Chunk list** — doc object uses Python key names (`chunk_count`,
`dataset_id`, `chunk_method`, run text status). Add validates list
element types.
**Document update** — adds `chunk_method` alias, pydantic-style Field
error messages, metadata index auto-create with refresh, and "These
documents do not belong to dataset" messages. List validates
`metadata_condition` and reports ownership errors for unmatched name/id
filters.
**Search completion** — `kb_ids` ownership failure returns code 102
instead of 109.
Released 31 contract tests from `GO_ONLY_SKIPS` (all verified passing on
both Go and Python backends with real LLM keys).
### Summary
**Issue**
When calling Infinity under concurrent load, one shared connection was
reused across concurrent operations, it just hangs.
**Solution**
Avoid hangs with pooled connections and bounded timeouts
### Summary
Plan to start api_server, admin_server and ingestor in one binary:
- ./ragflow_main --admin
- ./ragflow_main --api
- ./ragflow_main --ingestor
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### Summary
```
/api/v1/chats/<chat_id>/sessions/<session_id>/messages/<msg_id> DELETE
/api/v1/chats/<chat_id>/sessions/<session_id>/messages/<msg_id>/feedback PUT
```
Migrates the chat session message delete and feedback APIs to the Go
server, matching the Python behavior for authorization, session
ownership checks, message/reference updates, and feedback validation.
## Summary
After #16407 merged, 44 of the original 93 CodeQL alerts were still open
on the default branch. This PR closes the remaining ones by:
1. **Moving 32 existing `// codeql[...]` directives** so they sit on the
line **immediately before** the suppressed statement. The original
multi-line suppression blocks had the directive as the first line, with
the rationale on subsequent lines. After line shifts (refactors, linter
reformat), the directive ended up several lines above the alert location
— CodeQL only recognizes the suppression when it appears on the line
directly above. (32 alerts across 27 files.)
2. **Adding 9 new `// codeql[...]` suppressions** for alerts that had no
suppression in the preceding lines at all — mostly real-fixes that
CodeQL conservatively still flags (filepath.Base, bounded slice sizes,
model-identifier strings, the MD5-legacy-migration lookup in
`conversation_service.py`).
## Files changed
- `api/db/services/conversation_service.py` — add
`py/weak-sensitive-data-hashing` suppression (MD5 for backward-compat
legacy row lookup; not used for auth)
- `api/db/services/llm_service.py` — 3×
`py/clear-text-logging-sensitive-data` suppressions on the lines that
log `llm_name` in warnings/info
- `common/misc_utils.py` — 2× `py/clear-text-logging-sensitive-data`
suppressions on the redacted `current_url` log sites
- `internal/agent/component/invoke.go` — moved existing
`go/request-forgery` directive
- `internal/agent/sandbox/ssh.go` — moved existing
`go/command-injection` directive
- `internal/agent/tool/retrieval_service.go` — added
`go/uncontrolled-allocation-size` suppression (`topN` is bounded to 1024
above)
- `internal/cli/common_command.go` — moved 2×
`go/disabled-certificate-check` directives
- `internal/cli/user_command.go` — added `go/clear-text-logging`
suppression (filepath.Base already strips user-identifying path)
- `internal/dao/pipeline_operation_log.go` — moved 2× `go/sql-injection`
directives
- `internal/dao/user_canvas.go` — added `go/sql-injection` suppression
in `GetList` (the new `userCanvasOrderClause` call path)
- `internal/engine/infinity/chunk.go` — moved existing
`go/unsafe-quoting` directive
- `internal/entity/models/*` — moved `go/path-injection` directives (15
files)
- `internal/handler/oauth_login.go` — moved existing
`go/cookie-httponly-not-set` directive
- `internal/handler/tenant.go` — moved existing `go/path-injection`
directive
- `internal/service/deep_researcher.go` — moved existing
`go/unsafe-quoting` directive
- `internal/service/dataset.go` — added
`go/uncontrolled-allocation-size` suppression (`n` bounded to 1024
above)
- `internal/service/file.go` — moved existing `go/request-forgery`
directive
- `internal/service/langfuse.go` — moved 2× `go/request-forgery`
directives
- `internal/utility/mcp_client.go` — moved 3× `go/request-forgery`
directives
- `internal/utility/smtp.go` — moved existing `go/email-injection`
directive
- `rag/prompts/generator.py` — added
`py/clear-text-logging-sensitive-data` suppression
- `web/.../use-provider-fields.tsx` — added
`js/prototype-pollution-utility` suppression (FORBIDDEN_KEYS guard is on
the line above)
## Why the previous PR left alerts open
`// codeql[query-id] explanation` must be on the line **immediately
before** the suppressed statement per the [GitHub CodeQL suppression
spec](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/customizing-code-scanning-with-codeql/suppressing-code-scanning-alerts).
The original suppression blocks were 4-5 lines, with the directive as
the **first** line. After linter reformat / line shifts, the directive
ended up too far above the actual alert line to be recognized. The fix
is to put the directive on the line directly above the suppressed
statement, with the rationale above it.
## Test plan
- All 9 modified Python files `ast.parse` clean
- All 4 modified Go files `gofmt` clean
- 36/44 expected alert suppressions in place
- 8 remaining CodeQL alerts are the originals (#3485851828, #3485851831,
#3485869759, #3485869766, #3485869768, #3485869771, #3485885962,
#3485895527) which were resolved by the corresponding commit comments;
these should close on the next scan when the suppression comments match
the alert lines.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
### What problem does this PR solve?
As title:
implement:
```
/api/v1/messages/search GET
/api/v1/messages GET
/api/v1/messages/<memory_id>:<message_id>/content GET
/api/v1/memories/<memory_id>/config GET
/api/v1/messages/<memory_id>:<message_id> PUT
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Implement OpenAI chat completions in GO
POST /api/v1/openai/<chat_id>/chat/completions
OpenAI chat cli: internal/development.md
### Type of change
- [x] Refactoring
