7 Commits

Author	SHA1	Message	Date
nonoge	183280ebfc	fix browser extra detection for indentless YAML (#4367)	2026-07-26 09:56:17 +08:00
Ryker_Feng	fa496c0c8d	feat(browser): add agentic browser control (#4187) ... * feat(browser): add agentic browser control * fix(frontend): format browser view changes * fix(browser): keep browser optional and isolate sidecar layout * fix(browser): address PR review security and IME findings - Nginx: add a browser-stream WebSocket location before the generic /api/threads regex so Live upgrades instead of downgrading to HTTP (both nginx.conf and nginx.local.conf). - Ownership: require an existing owned thread for the WS stream and REST navigate, and tear down the browser session on thread deletion so a later caller cannot reuse a retained page/cookies by guessing the id. - SSRF: enforce the URL policy at the browser request boundary via a context-level route guard covering redirects, popups, iframes, and subresources (skipped for CDP-attached Chrome). - IME: skip key forwarding while a composition is active so confirming a CJK candidate with Enter no longer submits the remote page form. Adds regression tests for the request guard, session teardown on delete, and the composing-Enter key decision. * fix(frontend): smooth streaming in long tool threads * Revert "fix(frontend): smooth streaming in long tool threads" This reverts commit f0462516ea. * fix(browser): address review security and lifecycle findings - Reject cross-origin WebSocket upgrades on the live browser stream (Origin allow-list reuse of CORS/same-origin helpers) to close a WS-CSRF hole, and fail closed when the ownership store is absent. - Warn when a CDP-attached session runs with the SSRF request guard off, and drop the unreachable CDP screencast teardown dead code. - Read browser session launch config from a single canonical source (browser_navigate) so it is deterministic regardless of call order. - Bound per-thread Chromium accumulation with idle-timeout eviction and an LRU max-sessions cap. - Reset the Live reconnect counter on a successful open so the stream can't permanently stall after the cumulative attempt cap. * fix(frontend): reduce long tool thread render stalls Reuse stable historical message groups during streaming, defer heavy Markdown and browser previews, and lazy-decode message images. * fix(browser): keep live control responsive during continuous input Why: Manual browser control felt laggy — a physical click ran the remote Playwright click three times and each non-move input synchronously awaited a JPEG screenshot, so events queued behind capture (queue wait up to ~237ms). The first async attempt used a trailing-edge debounce, which froze the visible page until a wheel/keyboard gesture stopped ("scroll finishes, then it jumps"). What: - Frontend forwards one `click` per physical click instead of also emitting `down`/`up`, so the remote page is not clicked twice per gesture. - Backend detaches live-frame capture from input dispatch: non-move actions start a rate-limited background refresh loop (leading frame + bounded cadence) that keeps emitting frames while input continues and never blocks dispatch. - Add regression tests: input dispatch no longer awaits the screenshot, rapid inputs coalesce, and continuous input keeps refreshing before it stops. Scenarios: Verified in the live Browser panel — a single click completes in ~57ms (was blocked behind a 171ms capture), and a 1.14s sustained wheel gesture renders ~7 frames throughout the scroll instead of one frame after it ends. * fix(browser): harden worker and session lifecycle * fix(browser): address latest review feedback * fix(frontend): preserve optimistic new-chat message * test(e2e): preserve mocked message run ids * fix(browser): address capability review feedback --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-21 11:46:33 +08:00
Aari	5eb59cb130	fix(sandbox): stop multi-worker orphan reconcile from killing peer sandboxes (#4221) ... * fix(sandbox): stop multi-worker orphan reconcile from killing peer sandboxes Docker sandboxes are shared across gateway workers, but each worker kept its own in-memory warm pool. Startup reconciliation adopted every running container, so a peer idle reaper could destroy sandboxes another worker still owned and tool calls hit 502 / Connection refused. Add file-based ownership leases under sandbox-leases/, only adopt true orphans, refuse idle/replica/shutdown destroy while a foreign lease is live, and renew the lease on create/get/release/reclaim. Fixes #4206 * fix(sandbox): close lease fail-open, hot-path IO, and check→destroy race Address review of the multi-worker orphan lease (#4206): - read_lease returns None only for a genuinely-absent lease and raises (CorruptLeaseError/OSError) when a lease is unreadable or corrupt, so the ownership check fails closed instead of mistaking an unprovable peer lease for a free container. clear_lease still removes a stuck/corrupt file. - get() no longer renews the lease (blocking mkdir/fsync/os.replace on the event loop path used by ensure_sandbox_initialized_async); active leases are renewed off the event loop from the idle checker (_renew_active_leases). - The ownership check and container stop run under a per-sandbox flock guard (lease_ownership_guard); every lease write takes the same guard so a peer's touch cannot interleave with a destroy. Same-host multi-worker scope, not a multi-pod distributed lock. Also fixes the ruff format lint on the branch. Adds regression tests: corrupt and unreadable lease fail closed, a tests/blocking_io anchor keeping get() non-blocking on the event loop, and a peer-touch/destroy interleave test. * fix(sandbox): share container ownership across gateway instances Rework of the #4206 fix per review: ownership state is shared through a third-party service instead of being maintained per gateway instance, following the stream_bridge precedent (sandbox.ownership.type: memory | redis). The file lease and its same-host flock guard are deleted, not ported — they only covered workers on one host, while the deployment that hits #4206 is a load-balanced multi-instance gateway. A lease answers "who reaps this container", not "who may use it". Containers are deterministic per (user, thread), so consecutive turns legitimately land on different instances: take() transfers ownership on acquire, while claim() gates every adopt/reap path. Leases carry a state — own: or del: — so a takeover is refused against a teardown in progress. Without it an unconditional take() would overwrite a destroyer's claim and the peer's container stop would land on a sandbox the new owner had already handed to an agent. renew() distinguishes a lapsed lease from one a peer took; only the latter drops the sandbox. Collapsing them meant a Redis restart evicted every in-flight sandbox on every instance at once. Renewal runs on its own thread with a TTL derived from its interval, never from idle_timeout: renewal used to ride the idle checker, which does not start at idle_timeout: 0, so leases silently lapsed on a supported config. Ownership establishment is fail-closed: a sandbox whose ownership cannot be published is never handed out, and a just-created container is destroyed rather than leaked as an adoptable orphan. Every destroy path claims before untracking. The memory store is single-instance only and says so; the resolver reads app_config.stream_bridge and the env var in the bridge's own order, so deployments already using Redis get a redis ownership store without extra config. * fix(sandbox): wait out a recovery grace before adopting a keyless container An absent ownership lease meant two opposite things on two paths. Renewal reads it as LAPSED and re-establishes it: nobody took the lease, so the container is still ours. Reconciliation read the same absent key as "orphan" and adopted on sight. After the store loses its keys (a Redis restart without persistence, or eviction under maxmemory) every owner is alive and merely pre-renewal-tick. Whichever instance reconciled first therefore adopted every live container; each real owner's next renewal reported LOST and dropped a sandbox it was serving mid-turn, leaving it for the adopter to idle-destroy — #4206 through the back door, in the very case the LAPSED handling was added to make safe. Not limited to startup: an already-running instance hits the same window from the idle checker's periodic reconcile. _adoptable_after_grace requires an untracked container to be seen unowned across a full lease TTL before it can be adopted. That rebuilds the delay the state loss erased: a live owner republishes within one renewal interval, shorter than the TTL by construction, while a crashed owner never does, so its containers are still adopted one grace later rather than leaking. A republished lease resets the grace; a pausing-only timer would still expire over a live owner's lease. The peek is read-only — the atomic claim still gates adoption. The grace is skipped when the store cannot coordinate across processes: no peer can hold a lease such a store would show us, so single-instance deployments keep instant orphan cleanup, and a grace could not help a multi-worker gateway on memory anyway. * fix(sandbox): hold the teardown lease for as long as the container stop runs claim(..., for_destroy=True) wrote the del: marker with the ordinary lease TTL and nothing refreshed it. renew() extends only own: and deliberately reports a teardown as LOST, and the destroy paths drop the sandbox from the maps the renewal loop iterates — so a container stop that outlived the TTL let the marker lapse, a peer's take() succeeded against the still-running container, and the stop then landed on the turn that had just been handed it. That is the exact window the del: state exists to close, reopened by its own expiry. The two lease states alone never made the per-sandbox flock redundant, as I claimed when deleting it: a held lock cannot expire, a lease can. The exclusion has to be held deliberately rather than assumed to outlast the work it guards. _held_teardown_lease wraps both _backend.destroy() call sites and re-claims the marker every renewal_interval_seconds until the stop returns. No store change is needed: claim(for_destroy=True) already refreshes an existing del: marker on both backends. Reachable without an abnormal backend. The schema bounds only renewal_interval_seconds (> 0) and ttl_multiplier (>= 2), so a legal config puts the TTL below a normal container stop; and LocalContainerBackend._stop_container passes no timeout to subprocess.run, so a wedged daemon blocks unbounded even at the default 120s TTL. The TTL stays finite on purpose: the heartbeat dies with the process, so a destroyer that crashes mid-stop still releases the container one TTL later instead of marking it undestroyable forever. * fix(sandbox): hold the teardown lease on every del: stop, and pin the claims that had no test 90936b49 said `_held_teardown_lease` wrapped "both" `_backend.destroy()` call sites. There are three. `_drop_unhealthy_sandbox` marks `del:` and then blocks on the same unbounded stop, and it untracks *before* claiming, so `_renew_owned_leases` cannot see the id either — nothing refreshed the marker. Reproduced against a real redis: the peer's `take()` succeeds 1.0s into a 2.5s stop. Same window, third path. That miss came from the habit the rest of this commit addresses: a property asserted in prose, with no test that could falsify it. Auditing every load-bearing claim in this feature — AGENTS.md, the store docstrings, the provider's design comments — against the test that would go red turned up several more, each verified by mutating the code and watching the suite stay green. Tests that could not fail: - `test_reconcile_fails_closed_when_ownership_unknown` reached the grace gate, not the claim. A bare MagicMock answers `owner()` with a truthy mock, so the container read as peer-owned and deferred; `claim()` was never called. It stayed green with `_claim_ownership` failing open. Adding the grace ahead of the claim is what hollowed it out — inserting a gate can silently disarm the tests for the gate behind it. - `test_adoption_grace_restarts_when_a_live_owner_republishes` never distinguished reset from pause. Those diverge only on a *second* lapse, which it never drove, so it passed with the reset deleted. Claims with no test at all, each now pinned (mutation → red, per test): - `destroy()`, `_evict_oldest_warm`, `_reclaim_warm_pool_sandbox`, `_register_created_sandbox` and `shutdown()`'s warm loop were each the one untested sibling of an "every path does X" enumeration. `shutdown()` was never driven with a non-empty warm pool, so a loop bypassing the ownership claim — stopping a live peer's container on our exit — went unnoticed. - Renewal's unknown-is-not-lost rule, the single deliberate exception to fail-closed. Inverting it drops every active and warm sandbox on every instance the moment the store blinks. - Both hops of the stream-bridge redis inference. Deleting either left the suite green while every config.yaml-native multi-instance deployment silently fell back to memory — #4206 reopened on exactly the deployments the inference exists for. Claims narrowed instead, because they promised more than the code delivers: - "run against both backends ... cannot drift" — CI provisions no redis, so the merge gate runs the memory tier only and the Lua never executes there. - "Every destroy path claims before untracking" — `_drop_unhealthy_sandbox` untracks first, deliberately, under its `expected_info` TOCTOU guard. - "Atomic: concurrent claims from different instances cannot both succeed" — true via Lua on redis, vacuous on the single-instance memory store, and pinned by neither, since the contract suite drives sequential calls. A concurrency test against the memory store would make the claim look covered while the mechanism that carries it still never runs in CI. * fix(sandbox): release the teardown marker when a destroy() stop fails The three `del:`-marked stop paths disagreed on failure. `_destroy_warm_entry` releases on both outcomes and says why: the stop failed, so the container is probably still up, and a marker left behind refuses its own thread's `take()` until the TTL lapses. `_drop_unhealthy_sandbox` does the same. `destroy()` had no such guard — a raising backend propagated straight past `_release_ownership`, and the thread could not re-acquire for a full TTL. Fails safe rather than fatal: a stuck marker stops peers from touching the container, it is not the cross-instance kill. But the paths must agree, and this one is the odd one out. Release, then re-raise. Swallowing would be the easier symmetry with `_destroy_warm_entry`'s `return False`, but `destroy()` has no failure return and `shutdown()` logs per sandbox off the exception, so swallowing would silently narrow what callers can see. Found by comparing the three paths after @fancyboi999 asked for release to be handled "consistently with the other destroy paths" on the unhealthy path — which 0d2377b2 already does. This is the sibling that wasn't. * fix(deploy): bump chart config_version to 27 for sandbox.ownership config.example.yaml went to 27 with the new sandbox.ownership section, but the chart embeds its own copy and stayed at 26, so validate-chart failed. A bare bump: the chart already sets stream_bridge.type=redis, which is what resolve_ownership_config infers a redis ownership store from, so no field change is needed. * fix(sandbox): release the teardown lease from its heartbeat, not the caller `_held_teardown_lease` joined its heartbeat only briefly and the caller cleared the `del:` marker right after the stop. A refresh `claim` still in flight (`RedisOwnershipStore` had no socket timeout, so a round trip could block) could land *after* that release and rewrite `del:` on a container whose stop had already completed — refusing a fresh `take()` (or rolling back a fresh create) until the TTL. Move the release into the heartbeat's own `finally`, after its loop stops, so no refresh can run after it. The three destroy paths no longer release after the `with` (`destroy()`'s no-container branch still does, since no lease was held there). Bound every store round trip with a socket timeout so the in-flight refresh — and thus the deferred release — stays finite, and broaden the heartbeat's `except` so an unexpected error cannot strand the marker during a long stop. Also fold in the review follow-ups: stop re-resolving an already-resolved ownership config in the factory, document the Redis-outage-vs-TTL boundary in config.example.yaml, and add a tests/blocking_io anchor pinning that `release()`'s store round trip stays off the event loop. * fix(sandbox): refuse a non-destroy claim that would unwind our own teardown `claim(for_destroy=False)` against our own `del:` lease fell through and overwrote it with `own:`, cancelling a teardown that was already in flight. The container stop cannot be recalled, so downgrading the marker would let a `take()` hand out a container that is about to die -- #4206, self-inflicted. No caller does this today: the two non-destroy callers run against an absent key (the LAPSED re-claim) or an unowned one (post-grace reconcile). The contract has to forbid it rather than rely on that staying true. Fixed in both backends. The redis rule lives in Lua and the memory rule in Python, so fixing one only would let them drift silently -- and the shared contract suite is what is supposed to catch that drift, so it now covers this. Also adds a contention test for `claim`. The suite drove sequential calls only, so it pinned the exclusion predicate but not the atomicity that predicate depends on; eight instances now race for one container and exactly one must win. * fix(sandbox): bound the container stop so it cannot outlive its teardown lease `_stop_container` passed no `timeout` to `subprocess.run`, so a wedged container runtime blocks it forever. The `del:` marker is what keeps a peer from re-acquiring the container while the stop runs, but a marker is a lease and a lease can lapse: a store outage longer than the TTL frees it, a peer's `take()` succeeds against the still-running container, and the stop then lands on the turn that was just handed it -- the exact #4206 failure. The teardown heartbeat already covers the case where the store stays reachable. This bounds the worst case independently of the ownership layer, which is the point: it holds even when the ownership layer is the thing that failed. A timeout is not swallowed like a `CalledProcessError`. That error means the runtime answered "I could not stop it"; a timeout means we do not know, and the container is probably still running -- returning normally would let `_destroy_warm_entry` report a clean stop and drop the warm entry, leaking a running container nothing tracks. * fix(sandbox): exclude this instance's own reapers from its acquire path An ownership lease excludes peers and nothing else. `claim()` and `take()` both succeed against our own `own:` lease by design -- that is what lets a destroy path claim what it already owns -- so `del:` says nothing to this process's other threads. Meanwhile every reaper decides outside `_lock`, because a store round trip must not be held under the lock that guards every acquire. So each reaper acts on a decision its own acquire path may already have invalidated, and the store cannot see the difference. Six paths end in an irreversible act (a container stop, or closing a host-side client) on a decision made outside the lock. All six reproduce: _evict_oldest_warm re-checks warm membership, then releases the lock _reap_expired_warm no re-check at all _cleanup_idle_sandboxes re-verifies idle, then releases the lock _renew_owned_leases acts on a stale renew() -> LOST release() same staleness on its own refresh _drop_unhealthy_sandbox untracks before claiming, opening discovery Both warm reapers are a regression from the deferred pop this branch introduced: `WarmPoolLifecycleMixin` popped under the lock, so a reclaim's membership check failed and the race could not occur. Deferring the pop is still right (popping first loses the container on a refused claim), so the exclusion has to be made explicit instead. The idle path is pre-existing in shape, but this branch widened it from a few instructions to a network round trip by claiming ownership before untracking. Two guards, because the two directions want opposite answers: Reaping -- nothing may promote it. The reaper reserves the id, and every promote path refuses a reserved id exactly as it refuses a peer's `del:` (drop and cold-start). The "is this still reapable?" test travels with the reservation as a predicate and runs in the same critical section, because checking first and reserving second is the window, not a narrower version of it. Forgetting -- the peer legitimately wins, so the promote is what to detect. `_publish_ownership` bumps a per-id acquire epoch; the callers that decide from a store round trip snapshot it first, and the pop is skipped if it moved. Object identity cannot substitute: the reuse path re-publishes ownership while handing out the same tracked `AioSandbox`, so an identity check sees nothing and the pop closes a client mid-turn. `still_reapable` is required rather than defaulting to unconditional -- the safe default is the one that makes a new call site think about it. That diverges from the mixin hook, which is safe because this provider overrides both mixin callers, and loud rather than silent if those are ever dropped. Also closes a client leak on the discover path: "nothing to roll back" was true of the container but not of the HTTP client constructed before the publish, which the sibling create path already closes. The shared-store test view rebound `owner_id` outside the store's lock, so a concurrent claim could execute under the wrong id and read its own lease as a peer's. Serialized, so the heartbeat-hold tests stop flaking. * fix(sandbox): mark acquire intent before the ownership round trip A guard must become visible no later than the transition it guards. The acquire epoch cannot manage that for `take()`: the takeover is durable before `take()` returns -- redis has committed the SET while the reply is still in flight -- and the epoch can only be written afterwards. In that interval the store already says the container is ours while the epoch still reads as it did when a renewal decided `LOST`, so the stale forget walks through, drops the maps and closes the client the acquire is about to hand back. Acquire then returns an id the provider no longer tracks and `get()` answers `None` for the rest of the turn. `_publish_ownership` now publishes an intent mark under `_lock` before the round trip; the epoch keeps covering the other half, "an acquire completed since you decided". `_forget_lost_sandbox` honours the intent mark unconditionally rather than only when an epoch is supplied -- today's epoch-less callers cannot reach the window, but "no epoch" reading as "no guard" is how the next caller of a dangerous primitive gets written. The same invariant had four more instances, all reproduced: reuse returns a decision the forget already invalidated -- before the mark is set a `LOST` is both current and correct, so the forget legitimately runs and the entry reuse decided to hand out is gone. Re-check after publishing and fall through to discovery instead. reclaim installs an entry a reaper reserved after its check -- the warm entry is still visible during the stop, and the reaper's claim succeeds because reclaim's own take() just made the lease ours. Re-check likewise. the reservation was released before the entry was removed -- the pop belonged to the caller, leaving a gap where the container is stopped, the entry is still in `_warm_pool`, and nothing marks it. `_destroy_warm_entry` removes it itself, inside the reservation; the pop stays deferred relative to the stop, just not to the reservation. reconcile adopts a container this instance is tearing down -- adoption is a promote and needs the same reservation check as the others. Neither existing guard excludes it: the claim succeeds because the lease is ours, and on `memory` the recovery grace is skipped outright. The pre-round-trip checks in reuse and reclaim are kept as early-outs, since they skip a health check and a store round trip on a doomed entry, and are pinned to that job rather than to a correctness role they no longer hold. The teardown reservation predicate runs under `_lock`, so it must not touch the lock. Documented rather than engineered around: making the lock reentrant to tolerate it would trade a loud hang for a quiet class of re-entrancy bugs across the rest of the provider. * fix(sandbox): honor local teardown after ownership publish * fix(sandbox): clear a stale warm entry when an id becomes active Active and warm are exclusive states, and the two register paths were the only place that could hold both: they inserted into `_sandboxes` without popping `_warm_pool`, so one container ended up with two reapers. `_reap_expired_warm` judges an entry by its warm timestamp and never consults `_last_activity`, so it stops a container an agent is actively using while `_sandboxes` still hands out its client. Reachable because `_reconcile_orphans` adopts an untracked-but-running container into the warm pool inside the register's publish -> track window, and on the `memory` store it adopts on sight: `_adoptable_after_grace` short-circuits when `supports_cross_process` is False, so an id carrying this process's own lease reads as adoptable. That window is new to this branch -- on main the track was a single locked insert with nothing before it. Both register paths now pop the warm entry inside the same locked section that installs the active one. * fix(sandbox): harden ownership renewal teardown --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-07-21 09:09:40 +08:00
Janlay	72f033fbbe	feat(gateway): add redis stream bridge (#3191) ... * feat: add redis stream bridge * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix(gateway): address redis stream bridge review Redis was imported eagerly through deerflow.runtime and declared as a hard dependency, which made memory-only installs load redis.asyncio at startup and left the lazy factory import ineffective. Move redis behind an optional extra, remove the public eager re-export, and keep make_stream_bridge as the only runtime import path with an actionable install hint when the extra is missing. Because Docker deployments now default the stream bridge to Redis via DEER_FLOW_STREAM_BRIDGE_REDIS_URL, install the redis extra explicitly in Docker/dev container flows and teach the local uv-extra detector to infer redis from both stream_bridge.type and the Redis URL env var. This keeps Docker working while preserving slim non-Docker installs. Harden the Redis bridge by batching XREAD replay, replacing brittle ResponseError string matching with a single fallback to 0-0 for malformed Last-Event-ID values, documenting connection/retention/fail-hard behavior, and adding fake plus opt-in real Redis coverage for XADD/XREAD, replay, invalid IDs, and MAXLEN trimming. * fix(config): bump config version for stream bridge * fix redis stream bridge terminal handling * fix: repair uv.lock, format redis.py, and align Dockerfile extras test The uv.lock file was missing a closing bracket for the redis extras section, redis.py had a formatting issue caught by ruff, and the Dockerfile extras test did not account for the hardcoded --extra redis flag. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>	2026-07-04 09:21:19 +08:00
Zhou Kai	dd05e1a76d	fix(docker): production Postgres UV extras detection (#3897) ... * Fix production postgres UV extras detection * fix(backend): validate Docker build UV extras	2026-07-01 23:40:35 +08:00
Yi Tang	48e038f752	feat(channels): enhance Discord with mention-only mode, thread routing, and typing indicators (#2842) ... * feat(channels): enhance Discord with mention-only mode, thread routing, and typing indicators Add mention_only config to only respond when bot is mentioned, with allowed_channels override. Add thread_mode for Hermes-style auto-thread creation. Add periodic typing indicators while bot is processing. * fix(discord): include allowed_channels in mention_only skip condition (line 274) * docs: fix Discord config example to match boolean thread_mode implementation * style: format with ruff * fix(discord): apply Copilot review fixes and resolve lint errors - Remove unused Optional import - Fix thread_ts type hints to str | None - Fix has_mention logic for None values - Implement thread_mode fallback to channel replies on thread creation failure - Fix thread_mode docstring alignment - Fix allowed_channels comment formatting in config.example.yaml * fix(discord): reset context for orphaned threads in mention_only mode When a message arrives in a thread not tracked by _active_threads, clear thread_id and typing_target so the message falls through to the standard channel handling pipeline, which creates a fresh thread instead of incorrectly routing to the stale thread. * fix(discord): create new thread on @ when channel has existing tracked thread When mention_only is enabled and a user @-s the bot in a channel that already has a tracked thread, create a new thread instead of incorrectly routing to the old one. * fix(discord): allow no-@ thread replies while skipping no-@ channel messages The skip block for no-@ messages was too aggressive — it blocked continuation replies within tracked threads AND incorrectly routed no-@ channel messages to the existing thread. Now: - Thread message, no @ → routed to existing tracked thread - Channel message, no @ → skipped - Channel message, with @ → creates new thread * feat(discord): add checkmark reaction to acknowledge received messages * Move discord.py to optional dependency and auto-detect from config.yaml - Add discord extra to [project.optional-dependencies] in pyproject.toml - Update detect_uv_extras.py to map channels.discord.enabled: true -> --extra discord - Set UV_EXTRAS=discord in docker-compose-dev.yaml gateway env * fix(discord): persist thread-channel mappings to store for recovery after restart Discord's _active_threads dict was purely in-memory, so all channel-to-thread mappings were lost on server restart. This fix bridges ChannelStore into DiscordChannel: - Save thread mappings to store.json after every thread creation - Restore active threads from store on DiscordChannel startup - Pass channel_store to all channels via service.py config injection Store keys follow the pattern: discord:<channel_id>:<thread_id> * fix(discord): address Copilot review — fix types, typing targets, cross-thread safety, and config comments * fix(tests): add multitask_strategy param to mock for clarification follow-up test * fix(tests): explicitly set model_name=None for title middleware test isolation * fix(discord): use trigger_typing() instead of typing() for typing indicators discord.py 2.x TextChannel.typing() and Thread.typing() are async context managers, not one-shot coroutines. Use trigger_typing() for periodic typing indicator pings. * fix(discord): cancel typing tasks on channel shutdown Prevents 'Task was destroyed but it is pending' warnings when the Discord client stops while typing indicator loops are still running. * fix(scripts): detect nested YAML config for discord extra section_value() only matched top-level YAML sections. Added nested_section_value() that handles two-level nesting (e.g., channels.discord.enabled), so auto-detection of the discord extra works when config uses the standard nested format. * fix(docker): remove hard-coded UV_EXTRAS=discord from dev compose Relies on auto-detection via detect_uv_extras.py instead of forcing discord.py install even when channels.discord.enabled is false. Matches production docker-compose.yaml behavior (UV_EXTRAS:-). * refactor(nginx): move proxy_buffering/proxy_cache to server level DRY cleanup — these directives were repeated in 14 location blocks. Set at server level once, reducing duplication and risk of drift. * fix(discord): use dedicated JSON file for thread persistence Replace ChannelStore usage for Discord thread-ID persistence with a dedicated discord_threads.json file. ChannelStore is designed to map IM conversations to DeerFlow thread IDs — using it to persist Discord thread IDs was semantically wrong and confusing. Changes: - _save_thread() now reads/writes a simple {channel_id: thread_id} JSON dict - _load_active_threads() reads directly from the JSON file - File path derived from ChannelStore directory (when available) or defaults to ~/.deer-flow/channels/discord_threads.json - Removed unused ChannelStore import * fix(discord): address WillemJiang's code review comments on PR #2842 1. Remove semantically incorrect message_in_thread variable. At this code point (after the Thread case is handled above), we're guaranteed to be in a channel, not a thread. Always apply mention_only check here. 2. Add _active_thread_ids reverse-lookup set for O(1) thread ID membership checks instead of O(n) scan of _active_threads.values(). Keep the set in sync with _active_threads in _load_active_threads() and _save_thread(). 3. Add _thread_store_lock (threading.Lock) to protect _active_threads and the JSON file from concurrent access between the Discord loop thread (_run_client) and the main thread (_load_active_threads, _save_thread).	2026-05-15 22:30:05 +08:00
Xinmin Zeng	94da8f67d7	fix(scripts): preserve uv extras across make dev restarts (#2754) (#2767) ... `make dev` ran `uv sync` unconditionally on every restart, wiping any optional extras the user had installed manually with `uv sync --all-packages --extra postgres`. The Docker image-build path already solved this via the `UV_EXTRAS` build-arg in backend/Dockerfile; the local serve.sh path and the docker-compose-dev startup command were the remaining outliers. `scripts/serve.sh` now resolves extras before `uv sync`: 1. honors `UV_EXTRAS` (parity with backend/Dockerfile and docker/docker-compose.yaml — no new convention introduced); 2. falls back to parsing config.yaml — `database.backend: postgres` or legacy `checkpointer.type: postgres` auto-pins `--extra postgres`, so the common case needs zero extra config. 3. detector stderr is no longer suppressed, so whitelist warnings or crashes surface to the dev terminal (review feedback). Detection lives in `scripts/detect_uv_extras.py` (stdlib-only — has to run before the venv exists). Extra names are validated against `^[A-Za-z][A-Za-z0-9_-]*$` so a stray shell metacharacter in `.env` cannot reach `uv sync` downstream (defense in depth). `docker/docker-compose-dev.yaml`'s startup command is now extracted to `docker/dev-entrypoint.sh` (review feedback — the inline command had grown to a ~350-char one-liner). The script: - parses comma/whitespace-separated UV_EXTRAS, applying the same `^[A-Za-z][A-Za-z0-9_-]*$` whitelist as the local detector; - emits one `--extra X` flag per token, so `UV_EXTRAS=postgres,ollama` works in Docker dev too (harmonized with local — review feedback); - calls `uv sync --all-packages` (PR #2584) so workspace member extras (deerflow-harness's postgres extra) are installed; - keeps the existing self-heal `(uv sync || (recreate venv && retry))` branch; - exposes `--print-extras` for dry-run testing. The compose file mounts the script read-only at runtime, so script edits take effect on `make docker-restart` without an image rebuild. The `--no-sync` alternative (a separate suggestion in the issue thread) was considered but rejected for dev paths because it would drop the self-heal branch and the auto-pickup of new pyproject deps. `--no-sync` is already in use for the production CMD (`backend/Dockerfile:101`) where it's appropriate. Updates the asyncpg-missing error message to include the `--all-packages` flag (matching #2584) plus the persistent install flow, and expands `config.example.yaml` so all three install paths (local / docker dev / docker image build) are documented with their multi-extra capabilities. Tests: - `tests/test_detect_uv_extras.py` (21 tests) — local-path env parsing, YAML edge cases, env-vs-config precedence, whitelist rejection of shell metacharacters. - `tests/test_dev_entrypoint.py` (15 tests) — docker-path validation via `--print-extras`, multi-extra parsing, metacharacter abort. - `tests/test_persistence_scaffold.py` (22 tests, unchanged) — passes with the merged `--all-packages --extra postgres` error message. Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-05-10 22:28:29 +08:00