### Summary
Aligns Go and Python error codes/messages so both backends honor the
same RESTful API contract, removing implementation-specific error leaks
(MySQL errors, `ValueError`, `AttributeError`, Gin validator format) in
favor of clean business error codes.
**Chat list** — invalid `orderby` now returns code 101 (was: raw Python
`AttributeError` code 100); invalid `page`/`page_size` values fall back
to defaults (was: raw `ValueError`/`ProgrammingError` code 100).
**Dataset create/update/delete** — adds UUID validation (101),
extra-field rejection (101), duplicate-id detection (101), content-type
/ JSON-syntax / object-shape checks (101), and "lacks permission" for
nonexistent datasets (IDOR). Create auto-deduplicates dataset names.
Pagerank updates tolerate a missing ES index. List response includes
`parser_config` and `pagerank`.
**Session list/update** — adds filtering, sorting, and pagination
support. Empty payloads are valid no-ops. Authorization errors map to
code 109.
**Chunk list** — doc object uses Python key names (`chunk_count`,
`dataset_id`, `chunk_method`, run text status). Add validates list
element types.
**Document update** — adds `chunk_method` alias, pydantic-style Field
error messages, metadata index auto-create with refresh, and "These
documents do not belong to dataset" messages. List validates
`metadata_condition` and reports ownership errors for unmatched name/id
filters.
**Search completion** — `kb_ids` ownership failure returns code 102
instead of 109.
Released 31 contract tests from `GO_ONLY_SKIPS` (all verified passing on
both Go and Python backends with real LLM keys).
## Summary
Adds page-range parsing support to the Go-native pipeline path and
introduces strict `parse_type` validation for both dataset and document
update endpoints.
## What changed
### Pages range parsing
- **`internal/utility/pdf_pages.go`** — `NormalizePDFPages`: normalizes
raw page ranges (list of `[from,to]` 1-indexed inclusive ranges) into
sorted, merged, deduplicated `[][]int`. Invalid ranges are dropped.
- **`internal/ingestion/pipeline/pdf_pages.go`** —
`NormalizeParserConfigPages`: walks any parser_config map and normalizes
`"pages"` values under every component → filetype setup, so the
persisted config always carries clean, merged ranges.
- **`internal/deepdoc/parser/pdf/parser.go`** — integrates
`resolvePagesToProcess` to filter parsed PDF pages by the configured
ranges.
- Pipeline integration (parser pages):
`internal/parser/parser/pdf_parser_common.go`, `chunk_process.go`, plus
associated e2e and unit tests.
### Parse type validation (shared logic)
- **`internal/service/parser_mode.go`** (new) — `ValidateParseTypeMode`:
shared function that validates `parse_type` (1=BuiltIn/parser_id,
2=Pipeline/pipeline_id) and ensures the corresponding field is present.
Used by both dataset and document update endpoints.
- **`internal/service/dataset/crud.go`** / `update.go` — replaces inline
`isPipelineMode`/`isBuiltinMode` computation with the shared
`service.ValidateParseTypeMode`.
- **`internal/service/document/document_dataset_update.go`** — adds
strict `parse_type` validation in `validateDatasetDocumentUpdate`,
simplifies the reparse logic to a two-way switch (isBuiltin/isPipeline)
now that parse_type is always valid.
- **`internal/service/document/document.go`** — adds `ParseType` field
to `UpdateDatasetDocumentRequest`.
- **`internal/service/document/document_dataset_update.go`** —
`updateDocumentParserConfig` fallback path when DSL loading fails.
- **`internal/service/parser_mode_test.go`** (new) — test coverage for
nil, invalid, and missing-field scenarios.
### Frontend
- **`web/src/interfaces/request/document.ts`** — adds `parseType` to
`IChangeParserRequestBody`.
- **`web/src/hooks/use-document-request.ts`** —
`useSetDocumentPipelineParser` sends `parse_type` in the PATCH payload.
- **`web/src/pages/dataset/dataset/use-change-document-parser.ts`** —
Go/Python branching for the document parser config dialog.
-
**`web/src/components/document-pipeline-dialog/use-document-pipeline-form.ts`**
— `buildSubmitData` returns `parseType` (bugfix: was dropped from the
return value).
### Test changes
- **Removed**: 2 tests that verified the old "mutually exclusive" error
(replaced by `ValidateParseTypeMode` coverage).
- **Modified**: 6 tests across document and dataset packages to include
`ParseType` in request structs.
- **Added**: new e2e tests for pages parsing (`pages_e2e_test.go`,
`pdf_parser_pages_e2e_test.go`) and unit tests for `NormalizePDFPages`,
`NormalizeParserConfigPages`, `resolvePagesToProcess`.
## Backward compatibility
- The `parse_type` field is **required** when `parser_id` or
`pipeline_id` is sent. This changes the contract for both dataset and
document PATCH endpoints, but aligns the Go backend with the existing
frontend behavior (the frontend already sends `parse_type`). Callers
that omit `parse_type` when updating parser/pipeline selections will
receive a clear error message.
- Existing callers that only update fields like `name`, `enabled`, or
`meta_fields` are unaffected.
- Test updates ensure all known call sites are compliant.
### Summary
1. list and get by id API for builtin DSL
2. add DSL default component param values validation
3. remove all hard code keys for parser config
## Summary
`ComponentBase.variable_ref_patt` (and its duplicate in
`agent.canvas.Graph.get_value_with_variable`) is the regex the canvas
runtime uses to find `cpn_id@var_nm` template refs in component prompts.
The `cpn_id` half was constrained to `[a-zA-Z:0-9]+`, which silently
dropped underscores. Component ids emitted by the frontend all contain
underscores (`userfillup_abc`, `retrieval_xyz`, `llm_0`, `message_0`,
…), so any template ref like `{userfillup_abc@line}` failed to match.
The placeholder then leaked through to the LLM verbatim, and the Agent
answered only its system-prompt directive.
This is exactly the "unconsidered await response" symptom in #16758:
```
Begin(Task) -> Await response -> Agent -> Message
```
Widen `cpn_id` from `[a-zA-Z:0-9]+` to `[a-zA-Z0-9_]+`. Bare `{line}`
(no cpn_id) remains unrecognised so it stays literal until the user
wires it up — matching the existing `VARIABLE_REF_PATTERN` shape used by
`agent.dsl_migration` for the same purpose.
## Changes
- `agent/component/base.py` — fix `variable_ref_patt` class attribute.
- `agent/canvas.py` — same fix applied to the inline regex inside
`Graph.get_value_with_variable` (kept as the literal regex to avoid
coupling the two unrelated sites).
-
`test/testcases/test_web_api/test_canvas_app/test_variable_ref_pattern_unit.py`
— new regression test pinning both the regex shape and end-to-end
resolution.
## Regression coverage
```
test_variable_ref_patt_matches_underscored_component_ids PASSED
test_variable_ref_patt_still_matches_legacy_ids PASSED
test_get_input_elements_from_text_resolves_underscored_id PASSED
test_string_format_substitutes_underscored_ref PASSED
test_variable_ref_patt_does_not_match_bare_var_name PASSED
```
All five regression tests fail against the pre-fix regex (verified via
`git stash` round trip — drop fix, tests fail, restore fix, tests pass).
The two targeted existing tests in the same directory
(`test_fillup_unit.py`, `test_iterationitem_unit.py`) continue to pass.
## Repro before the fix
```python
import re
patt = r"\{* *\{([a-zA-Z:0-9]+@[A-Za-z0-9_.-]+|sys\.[A-Za-z0-9_.]+|env\.[A-Za-z0-9_.]+)\} *\}*"
list(re.finditer(patt, "{userfillup_abc@line}"))
# => [] # <-- bug
```
## Repro after the fix
```python
import re
patt = r"\{* *\{([a-zA-Z0-9_]+@[A-Za-z0-9_.-]+|sys\.[A-Za-z0-9_.]+|env\.[A-Za-z0-9_.]+)\} *\}*"
list(re.finditer(patt, "{userfillup_abc@line}"))
# => [<re.Match object; span=(0, 24), match='{userfillup_abc@line}'>]
```
Fixes#16758
## Test plan
- [x] New unit tests pass
- [x] Reverse-apply the fix and confirm the regression tests fail (they
do)
- [x] `test_fillup_unit.py` (existing sibling suite) still passes
- [x] `test_iterationitem_unit.py` (existing sibling suite) still passes
- [ ] Project CI green
---------
Co-authored-by: Taranum01 <taranum01@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
### Summary
In Go and python implementation, the dataset / KB id isn't validated if
it is accessible by this user.
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## What
An **Await Response** (`UserFillUp`) node placed inside a **Loop** now
pauses and waits for a fresh user response on **every** iteration,
instead of only on the first one.
## Problem
When a `UserFillUp` node lives inside a `Loop`, it only paused for input
on the first iteration. On subsequent iterations the loop ran straight
through, silently reusing the answer the user gave the first time.
Root cause is in `UserFillUp._invoke` / the canvas wait-check
(`agent/canvas.py`). The wait-check decides whether to pause by calling
`Canvas._is_input_field_satisfied` on the node's form fields — a field
counts as satisfied as soon as its `value` is not `None`:
```python
@staticmethod
def _is_input_field_satisfied(field):
...
if value is None:
return False
return True
```
The same component object is reused across loop iterations, and
`UserFillUp._invoke` writes the answer into
`self._param.inputs[...]["value"]` via `set_input_value`. Nothing
cleared those values when the node was re-entered for the next
iteration, so:
| Iteration | Entry (no answer yet) | Field value | Satisfied? | Result
|
|---|---|---|---|---|
| 1 | fresh | `None` | no | pauses ✅ |
| 1 | resume w/ answer | `answer` | yes | continues ✅ |
| 2 | fresh | `answer` (**stale**) | yes | continues ❌ (should pause) |
## Fix
When a `UserFillUp` is entered without a fresh user answer
(`merged_inputs` is empty), clear the retained form values so the
wait-check treats the form as unsatisfied and pauses again:
```python
merged_inputs = self._merge_runtime_inputs(kwargs.get("inputs", {}))
if not merged_inputs:
self._clear_form_values()
```
- Fresh entry / new loop iteration → no answer supplied → values cleared
→ node pauses and waits.
- Resume with an answer → `merged_inputs` is non-empty → values applied
normally, nothing cleared.
- Non-loop behavior is unchanged: the first entry already had `None`
values, so clearing is a no-op there.
`Begin` overrides `_invoke` and is unaffected.
## Tests
Added to
`test/testcases/test_web_api/test_canvas_app/test_fillup_unit.py`:
- `test_user_fillup_clears_stale_values_on_reentry_without_answer` — a
retained value is cleared on a fresh entry with no answer (loop
re-entry).
- `test_user_fillup_keeps_values_when_answer_supplied` — a supplied
answer is applied and not cleared.
All unit tests pass and `ruff check` is clean.
## Scope
This targets the Python agent runtime (`agent/`). It is independent of
any other in-flight Await Response change.
### Summary
Handle searching dataset without embedding model
In this PR, Searching datasets with different embedding models or
searching dataset with/without embedding models are not allowed. We will
improve the behavior later.
## Related issues
Closes#15375
### What problem does this PR solve?
`GET /api/v1/messages` and `GET /api/v1/messages/search` accepted
unbounded `limit` / `top_n` query parameters while other REST list
endpoints enforce `REST_API_MAX_PAGE_SIZE` (100) via
`validate_rest_api_page_size()`. Oversized values can trigger expensive
memory index queries and large result sets (DoS risk).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Changes
| File | Change |
|------|--------|
| `api/apps/restful_apis/memory_api.py` | Cap `limit` and `top_n` with
`validate_rest_api_page_size`; return argument error when exceeded |
|
`test/testcases/test_web_api/test_message_app/test_message_routes_unit.py`
| Regression tests for oversized `limit` / `top_n` |
### Test plan
- [x] Unit tests added
- [ ] `pytest
test/testcases/test_web_api/test_message_app/test_message_routes_unit.py`
Co-authored-by: Cursor <cursoragent@cursor.com>
Fixes#16168
## Summary
- Add session-scoped authorization for `GET
/api/v1/documents/artifact/<filename>`
- Allow download only when the artifact filename appears in the caller's
`api_4_conversation` message and
`UserCanvasService.accessible(dialog_id, user_id)` passes
- Deny with generic `"Artifact not found."` before storage access (no
cross-user enumeration)
- Return 4xx when the blob is missing (existing behavior preserved)
## Approach
Sandbox artifacts are runtime CodeExec outputs, not KB documents — this
uses the same session gate pattern as `agent_chat_completion`, not
`DocumentService.accessible`.
## Test plan
- [x] Unit: denied when filename not referenced in user sessions
- [x] Unit: denied when agent canvas is not accessible
- [x] Unit: authorized user receives bytes; missing blob returns
`"Artifact not found."`
- [ ] `pytest
test/testcases/test_web_api/test_document_app/test_document_metadata.py
-k get_artifact`
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
Closes#15425. The agent **Invoke** (HTTP Request) component now calls
`assert_url_is_safe` and `pin_dns` before `requests.*`, matching Crawler
and SearXNG.
## Changes
- `agent/component/invoke.py`: SSRF guard + DNS pinning on outbound
requests.
- `test_invoke_component_unit.py`: unit test blocks loopback URL without
calling `requests.get`.
## Test plan
- [x] `pytest
test/testcases/test_web_api/test_canvas_app/test_invoke_component_unit.py::test_invoke_blocks_loopback_url_with_ssrf_guard`
(requires project test env / `ZHIPU_AI_API_KEY` in CI)
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
### What problem does this PR solve?
Fixes#14882
Agent webhook execution currently fails open when the saved webhook
`security` block is missing/empty, or when `auth_type` is set to `none`.
This allows unauthenticated webhook invocation without an explicit
operator opt-in.
This PR makes anonymous webhook access explicit:
- Rejects missing or empty webhook security config.
- Requires `allow_anonymous: true` when `auth_type` is `none`.
- Preserves explicit anonymous webhooks by having the frontend serialize
`allow_anonymous: true` when the user selects `None` auth.
- Updates webhook unit tests to cover both denied implicit-anonymous
configs and allowed explicit-anonymous configs.
### Type of change
- [x] Bug Fix
- [x] Security hardening
- [x] Test
### Tests
- [x] `ZHIPU_AI_API_KEY=dummy uv run python -m pytest
--confcutdir=test/testcases/test_web_api/test_agent_app
test/testcases/test_web_api/test_agent_app/test_agents_webhook_unit.py`
- [x] `uv run ruff check api/apps/restful_apis/agent_api.py
test/testcases/test_web_api/test_agent_app/test_agents_webhook_unit.py`
- [x] `npm exec eslint src/pages/agent/utils.ts
src/pages/agent/form/begin-form/schema.ts`
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
- harden reopened advisory fixes across REST connector, invoke, document
downloads, and markdown rendering
- add targeted regression coverage for redirect-safe SSRF handling,
invoke SSRF checks, document access control, and markdown sanitization
- verify each referenced GHSA against the original GitHub advisory text
and align the closed-advisory plan with the implemented remediation
## What changed
- add tenant access checks to document download endpoints to avoid
cross-tenant document disclosure
- add per-hop SSRF validation, DNS pinning, redirect handling, and
redirect limits to the REST API connector
- ensure invoke requests validate and pin the resolved host and never
follow redirects implicitly
- keep the generic rate-limited request path wrapped, not just GET and
POST helpers
- sanitize markdown HTML before rendering in the highlight markdown
component
## Validation
- `cd web && npm test -- --runInBand
src/components/highlight-markdown/__tests__/index.test.tsx`
- `.venv/bin/python -m pytest -q
test/unit_test/data_source/test_rest_api_connector.py`
- targeted `test/testcases/test_web_api/...` unit additions were
reviewed, but the suite cannot be executed end-to-end in this
environment because parent `test/testcases/conftest.py` requires a local
service on `127.0.0.1:9380`
## Notes
- all GHSA entries referenced by the plan were checked against the
original GitHub advisory text, not sampled
- the closed-advisory plan document was updated locally during review,
but is intentionally not included in this PR
### What problem does this PR solve?
Fixes the PubMed tool always emitting `Authors: Unknown Authors`. The
`safe_find` closure in `_format_pubmed_content` was hardcoded to search
from the article root, so the per-author `LastName`/`ForeName` lookups
never matched.
`safe_find` now accepts an optional `base` node (defaults to `child`,
preserving the existing field lookups), and the author loop passes the
current `<Author>` element.
Closes#16328
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Add test cases
### Testing
Added `test/testcases/test_web_api/test_canvas_app/test_pubmed_unit.py`
covering per-author parsing, intact title/journal/DOI fields, and the
no-authors fallback.
Before: `Authors: Unknown Authors`
After: `Authors: Furqan Khan, Jane Smith`
Replaces the Python agent canvas runtime with a Go implementation that
runs inside `cmd/server_main`.
The canvas compiles into an eino Workflow that pauses on wait-for-user
via native Interrupt/Resume (no sentinel flag) and resumes from a
Redis-backed CheckPointStore.
All 21 Python agent components and ~35 tools are ported with functional
parity.
Sandbox providers now read their JSON config from the admin-panel
system_settings table with env fallback.
234 files / +35,413 / -6,111. All Go files are gofmt-clean (CI gate
added); drops the v2 DSL E2E step and the gap-analysis plan (both
redundant after the port ships).
## Type of change
- [x] Refactoring
- [x] New feature
- [x] Bug fix
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
