`.nullish()` is defined as `.nullable().optional()` in Zod — same
validation behavior and inferred type (`T | null | undefined`). Pure
mechanical de-duplication, no behavior change.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* 🔧 chore(memory-user-memory): expose prompts/providers/converters subpath exports
The package only exported `.`, `./schemas` and `./types`, so consumers importing
`@lobechat/memory-user-memory/prompts`, `/providers` or `/converters/*` (e.g. the
agent-eval memory scenarios) failed to resolve under strict node_modules. Add the
matching subpath export entries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Update package.json
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* ✨ feat(database): add llm_generation_tracing schema + tracing package (LOBE-9462)
Foundation layer for per-call observability of `generateObject` calls.
- New Drizzle table `llm_generation_tracing` with identity / context / model /
result / usage / storage / feedback / audit columns and full single-column
index coverage (Postgres bitmap-scan friendly). Migration 0103 is idempotent
(CREATE TABLE/INDEX IF NOT EXISTS) for safe re-runs.
- `LlmGenerationTracingModel` with `record` / `updateFeedback` / `findById` /
`listRecent`, all userId-scoped to prevent cross-user leaks.
- New package `@lobechat/llm-generation-tracing` mirroring agent-tracing's
shape: `ITracingStore` interface, `FileTracingStore` (local/dev, scenario
subfolders + latest.json symlink), `computePromptHash` (6-char sha256 of
systemPrompt + schema), and `TRACING_SCENARIO_REGISTRY` + `resolveScenario`
with explicit scenario override.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* ✨ feat(model-runtime): wire llm_generation_tracing into ModelRuntime.generateObject (LOBE-9462)
Per-call interception layer — one hook covers all generateObject callers.
- New `onGenerateObjectComplete` hook on `ModelRuntimeHooks`: always fires
(success or failure) with latency, usage, output/error. Fixes the gap where
`onGenerateObjectFinal` only fires when the runtime invokes `onUsage`.
- `S3TracingStore` (zstd level 3, key
`llm-generation-tracing/{scenario}/{v}-{hash}/{date}/{id}.json.zst`) and
`LLMGenerationTracingService` that does DB insert → store.save → patch
storage_key. Store failures preserve the row with `metadata.store_error`.
- `createLLMGenerationTracingHook` + `mergeModelRuntimeHooks` wired into
`initModelRuntimeFromDB`; tracing runs alongside business (billing) hooks
via `next/server.after()` when available, microtask fallback otherwise.
Unknown metadata keys (e.g. `parent_memory_trace_key`) pass through.
- Memory extractor accepts `parentMemoryTraceKey` option for the job-level
backlink. Follow-up-action caller given an explicit `scenario: 'follow_up'`
metadata override — it was the only OSS caller missing trigger metadata.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* ✅ test(llm-generation-tracing): type vi.fn mocks so tsgo accepts mock.calls indexing
The hook + service tests destructured `mock.calls[0][0]` and accessed nested
fields, which tsgo flagged as TS2493 / TS18046 because `vi.fn()` defaults to a
zero-arg signature. Add explicit type parameters to the mocks so tsgo can
infer the call tuple, and cast `call.payload` at the access point.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* ♻️ refactor(model-runtime): move mergeModelRuntimeHooks into the package
It's a generic utility for composing `ModelRuntimeHooks` instances — same
import surface as `ModelRuntime` and the hooks interface — so it belongs
alongside them rather than tucked under a server-side consumer.
- New `packages/model-runtime/src/core/mergeHooks.ts` exports
`mergeModelRuntimeHooks` and is re-exported from the package index.
- Move the unit tests to `packages/model-runtime/src/core/mergeHooks.test.ts`,
including a new case covering the "a throws → b is skipped" load-bearing
semantics.
- `src/server/services/llmGenerationTracing/hook.ts` drops the local copy and
the consumer (`src/server/modules/ModelRuntime/index.ts`) imports from
`@lobechat/model-runtime`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* ♻️ refactor(llm-generation-tracing): version lives with the prompt, not in a central table
`promptVersion` was baked into `TRACING_SCENARIO_REGISTRY`, far from any
prompt definition — editing a prompt + forgetting to bump the entry in a
completely different file was an obvious foot-gun.
- Registry is now `Record<string, string>` mapping trigger → scenario only;
it's the stable concern that rarely changes.
- `resolveScenario` always passes `promptVersion` through from the caller,
defaulting to `UNKNOWN_PROMPT_VERSION` ('v0') when absent.
- Each call site declares its own `*_PROMPT_VERSION` constant next to the
prompt it describes. `followUpAction` ships the first one:
`FOLLOW_UP_PROMPT_VERSION` in `prompts/index.ts`, threaded through
`metadata.promptVersion` at the `generateObject` call. Other callers can
add the same constant when they next touch their prompts.
The 6-char prompt hash on the row still catches forgotten bumps.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* ✨ feat(input-completion): wire prompt-version metadata at the auto-complete call site
Aligns input auto-complete with the FOLLOW_UP_PROMPT_VERSION convention so
each prompt iteration is recordable as the chat-side tracing lands.
- `INPUT_COMPLETION_PROMPT_VERSION = 'v1.0'` declared next to
`chainInputCompletion` — bump together with the prompt body.
- `fetchPresetTaskResult` accepts optional `metadata` and forwards it to
`getChatCompletion`; the existing chat path already plumbs metadata to
`ModelRuntime.chat` options.
- `InputEditor` call site passes
`{ scenario: 'input_completion', promptVersion }`.
Note: `llm_generation_tracing` currently only fires from
`onGenerateObjectComplete`. Input completion is a `chat` call, so this
metadata is forward-looking until a chat-side tracing hook lands.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* 🐛 fix(llm-generation-tracing): collapse bucketDir path.join args to silence turbopack glob warning
Turbopack's static analyzer treats `path.join(root, dyn1, dyn2)` as a
multi-segment glob pattern and warned that it could match ~12k files in
the project. Compose the relative subdir as a single string first, so
`path.join` only sees one dynamic segment.
Behavior unchanged — the resulting path is identical.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* ✨ feat(input-completion): route auto-complete through generateObject for tracing
Auto-complete is the first preset-task caller migrated to the structured-
output path so it lands in `llm_generation_tracing` via the existing
`onGenerateObjectComplete` hook. No new server hook, no global chat-side
tracing.
- `chainInputCompletion` now returns `{ messages, schema }` with a minimal
`{ completion: string }` schema and a stable `INPUT_COMPLETION_SCHEMA_NAME`
constant. JSON wrapping costs ~15-30 tokens against a 100-token completion
budget — negligible for the observability win.
- `StructureOutputSchema` / `StructureOutputParams` accept optional
`metadata`; `aiChatRouter.outputJSON` merges caller metadata over the
default trigger so `{ scenario, promptVersion, schemaName }` reach
`ModelRuntime.generateObject` options unchanged.
- `IStructureSchema.description` is now optional to match the zod schema —
previously the TS type was stricter than runtime validation accepted.
- `InputEditor` switches from `chatService.fetchPresetTaskResult` to
`aiChatService.generateJSON`, reading `response.completion`. Streaming
is dropped because auto-complete already buffers the full result before
inserting; no UX change.
- Reverts the unused `metadata` field that was added to
`fetchPresetTaskResult` in the previous commit — no current caller needs
it now that input completion uses the generateObject path.
Bumps `INPUT_COMPLETION_PROMPT_VERSION` to v2.0 because the system prompt
gained an "output the completion field" instruction.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* ♻️ refactor(aiGeneration): extract the runtime-init + generateObject dance into a service
Every server-side caller that produces structured output was repeating the
same two-step ritual: `initModelRuntimeFromDB(...)` → `runtime.generateObject(payload, { metadata })`.
`AiGenerationService` collapses it into one call so future cross-cutting
concerns (default metadata, retry, observability hooks) have one place to
land.
- New `src/server/services/aiGeneration/index.ts` exposes
`generateObject<T>(input, options)` and is unit-tested for provider
resolution + payload/metadata pass-through.
- `aiChatRouter.outputJSON` and `FollowUpActionService.extract` migrated to
the service (other callers move organically when next touched).
- Drops the unused `keyVaultsPayload` field from `StructureOutputParams`
and the placeholder at the InputEditor call site — key vaults are
server-resolved from DB, the client never supplies them.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* ♻️ refactor(tracing): centralize TRACING_SCENARIOS const + inject AiGenerationService via trpc ctx
- New `packages/const/src/llmGenerationTracing.ts` exports `TRACING_SCENARIOS`
+ `TracingScenario` type — the single directory where every known scenario
name lives. Adds `@lobechat/const` as a workspace dep on llm-generation-
tracing so `TRACING_SCENARIO_REGISTRY` can reference the same literals.
- Callers (FollowUpActionService, InputEditor) replace `'follow_up'` /
`'input_completion'` string literals with `TRACING_SCENARIOS.FollowUp` /
`.InputCompletion`, so a typo or a rename fails the type-check instead of
silently drifting on the row.
- `AiGenerationService` is now injected into the `aiChatProcedure` ctx
middleware alongside `aiChatService`; `outputJSON` consumes it via
`ctx.aiGenerationService` instead of new-ing it inside the handler.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* ✨ feat(llm-generation-tracing): add lt/llm-tracing CLI + drop local-only storage_key
- Add `lt` / `llm-tracing` CLI under @lobechat/llm-generation-tracing with
`list` (recent records, --scenario filter, --json) and `inspect` (by
tracing_id prefix or latest, --full, --json).
- `FileTracingStore.save` now returns `{ key: null }` so dev DB rows leave
`storage_key` empty instead of recording a non-resolvable local path; S3
store remains the source of truth for the real key. Add helpers
`findByTracingId` / `getLatest` used by the CLI.
- Wire `agentId` and `topicId` into `input_completion` tracing metadata
from the chat input auto-complete call site.
- Default `FileTracingStore` whenever NODE_ENV=development (drop the
ENABLE_LLM_GENERATION_TRACING_LOCAL opt-in env var).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* 💄 style(llm-generation-tracing): prettier CLI output (tree + colors)
Mirror the @lobechat/agent-tracing viewer style:
- Inline ANSI color helpers (dim/bold/cyan/magenta/green/yellow/red).
- Compact single-line header with id, scenario, version, model, status,
time — replaces the multi-line bullet list.
- Tree structure with `├─`/`└─` connectors instead of `── section ──`
banners.
- input arrays render per-message (role + char count + preview) rather
than dumping raw JSON.
- Small single-key outputs (e.g. `{ completion: "怎么样" }`) collapse
to inline `key: "value"`.
- `lt list` switches to a colored, properly padded table.
Default view stays compact; --full expands system_prompt / input /
schema bodies.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* ♻️ refactor(llm-generation-tracing): split `tracing` config out of `metadata`
`options.metadata` was overloaded — half tracing-specific structured fields
(scenario / promptVersion / schemaName / agentId / topicId / ...), half
free-form jsonb passthrough. Callers couldn't tell which was which, and the
inputHint was always auto-extracted (useless when the prompt wraps the user's
text in a template).
This commit introduces a dedicated `tracing` option:
- Add `TracingOptions` to @lobechat/llm-generation-tracing — the typed shape
callers import (agentId / topicId / inputHint / scenario / promptVersion /
schemaName / systemPrompt / parentTracingId / metadata).
- Add loose `tracing?: Record<string, unknown>` to GenerateObjectOptions and
StructureOutputParams / StructureOutputSchema so the field flows through
the runtime + TRPC.
- Tracing hook now reads `context.options.tracing` for structured fields; it
still falls back to `metadata.trigger` for the cross-cutting trigger string
(ModelRuntime itself uses metadata.trigger for timing logs, so trigger
stays on metadata).
- Service `record()` accepts an explicit `inputHint`; otherwise falls back
to auto-extraction from the first user message. Always truncated.
- Free-form jsonb fields move to `tracing.metadata` (was unknown-key passthrough
on `metadata`).
- Call sites updated:
- FollowUpAction now passes `tracing: { scenario, promptVersion, schemaName,
topicId }` (previously `metadata`).
- InputCompletion now passes `tracing: { agentId, topicId, inputHint: input,
scenario, promptVersion, schemaName }` — `inputHint` is the user's actual
typed text, not the wrapper prompt's first user message.
- `aiChat.outputJSON` router forwards both metadata and tracing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Update inputCompletion.ts
* 🐛 fix(llm-generation-tracing): stop duplicating provider into the row's metadata jsonb
`provider` is already a first-class column on the `llm_generation_tracing`
row, so auto-stamping it into the `metadata` jsonb column on every call was
pure noise. The hook now writes the caller-supplied `tracing.metadata`
verbatim — empty/undefined when the caller had nothing to add.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
♻️ refactor: remove promptfoo configs and dependencies from packages
Migrate all prompt evaluation tests to the cloud repo's agent-evals framework.
Remove promptfoo directories, configs, dependencies, and generator scripts
from @lobechat/prompts, @lobechat/memory-user-memory, and @lobechat/builtin-tool-memory.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ♻️ refactor: implement SkillResolver to replace ad-hoc skill assembly
Introduces a two-layer skill resolution architecture mirroring ToolsEngine + ToolResolver:
- SkillEngine (assembly layer): accepts raw skills + enableChecker, outputs OperationSkillSet
- SkillResolver (resolution layer): merges operation + step delta + accumulated activations
Key changes:
- Add SkillResolver, OperationSkillSet, StepSkillDelta, ActivatedStepSkill types
- Enhance SkillEngine with enableChecker and generate() method
- Wire SkillResolver into RuntimeExecutors call_llm
- Replace manual skillMetas assembly in aiAgent with SkillEngine.generate()
- Update client-side skillEngineering to use SkillEngine + enableChecker
- Add activatedStepSkills to AgentState for step-level skill accumulation
Fixes: agent-browser content injected into non-desktop scenarios (Discord bot)
due to missing filterBuiltinSkills call in aiAgent
LOBE-6410
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ♻️ refactor: extract agent-templates to standalone package and inject documents server-side
- Create @lobechat/agent-templates package with types, templates, and registry
- Move DocumentLoadPosition, DocumentLoadFormat, DocumentLoadRule, etc. to new package
- Move claw templates (AGENTS, BOOTSTRAP, IDENTITY, SOUL) with .md file imports
- Add BOOTSTRAP.md as new onboarding template (priority 1, system-append)
- Fix template positions: AGENTS→before-system, IDENTITY/SOUL→system-append
- Update database package to re-export from @lobechat/agent-templates
- Migrate all consumers to import directly from @lobechat/agent-templates
- Add agent documents injection in server-side RuntimeExecutors (was missing)
- Support -p CLI flag in devStartupSequence for port configuration
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* 🐛 fix: correct import statement for non-type exports from agent-templates
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* 📦 build: add @lobechat/agent-templates to root dependencies
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ♻️ refactor: remove template proxy files from database package
Stop re-exporting template/templates from database — consumers import
directly from @lobechat/agent-templates. Keep types.ts re-exports for
internal database code only.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ♻️ refactor: create BaseSystemRoleProvider to unify system message append pattern
All providers that append to the system message now inherit from
BaseSystemRoleProvider and only implement buildSystemRoleContent().
The base class handles find-or-create and join logic.
Migrated providers:
- EvalContextSystemInjector
- BotPlatformContextInjector
- SystemDateProvider
- ToolSystemRoleProvider
- HistorySummaryProvider
- SkillContextProvider
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* 🐛 fix: restore metadata tracking in BaseSystemRoleProvider via onInjected hook
Add onInjected() callback to BaseSystemRoleProvider so subclasses can
update pipeline metadata after successful injection. Also add raw-md
plugin to context-engine vitest config for .md imports.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ✨ feat: add enabled field to AgentDocumentInjector config
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ♻️ refactor: add enabled field to all providers, remove spread conditionals in MessagesEngine
All providers now accept an `enabled` config field. MessagesEngine
pipeline is a flat array with no spread conditionals — each provider
is always instantiated and uses `enabled` to skip internally.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* 💄 style: clean up MessagesEngine pipeline comments
Remove numbered prefixes, keep descriptive comments for each provider.
Only phase headers use separator blocks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ♻️ refactor: reorganize MessagesEngine pipeline phases by injection target
Phase 1: History Truncation
Phase 2: System Message Assembly (all BaseSystemRoleProvider)
Phase 3: Context Injection (before first user message, BaseFirstUserContentProvider)
Phase 4: User Message Augmentation (last user message injections)
Phase 5: Message Transformation (flatten, template, variables)
Phase 6: Content Processing & Cleanup (multimodal, tool calls, cleanup)
Moved SkillContext, ToolSystemRole, HistorySummary from Phase 3 to
Phase 2 since they append to system message, not user context.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* 💄 style: split Phase 6 into Content Processing (6) and Cleanup (7)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ♻️ refactor: split AgentDocumentInjector into three position-based injectors
- AgentDocumentSystemInjector (Phase 2): before-system, system-append, system-replace
- AgentDocumentContextInjector (Phase 3): before-first-user
- AgentDocumentMessageInjector (Phase 4): after-first-user, context-end
Shared utilities (filterByRules, formatDocument, sortByPriority) extracted
to AgentDocumentInjector/shared.ts. Old monolithic injector removed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ♻️ refactor: split AgentDocumentSystemInjector into three separate injectors
- AgentDocumentBeforeSystemInjector: prepends as separate system message (before-system)
- AgentDocumentSystemAppendInjector: appends to system message (system-append)
- AgentDocumentSystemReplaceInjector: replaces entire system message (system-replace)
Each has distinct semantics and correct pipeline placement:
- BeforeSystem → before SystemRoleInjector
- SystemAppend → after HistorySummary (end of Phase 2)
- SystemReplace → last in Phase 2 (destructive)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ✨ feat: auto-enable agent-documents tool when agent has documents
- Add AgentDocumentsManifest to defaultToolIds
- Add hasAgentDocuments rule in server createServerAgentToolsEngine
- Query agent documents in AiAgentService.execAgent to determine flag
- Pattern matches KnowledgeBase auto-enable via enableChecker rules
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* 🔨 chore: add agent documents status to execAgent operation log
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* update content
* fix tests
* 🐛 fix: add raw-md plugin to database vitest configs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ✨ feat(desktop): implement subscription pages embedding with webview
- Add SubscriptionIframeWrapper component for embedding subscription pages in Electron webview
- Replace compile-time ENABLE_BUSINESS_FEATURES with runtime serverConfigSelectors
- Add useIsCloudActive hook to detect cloud sync status
- Add setupSubscriptionWebviewSession service for webview session management
Resolves LOBE-4589
* ✨ feat: enhance SubscriptionIframeWrapper for improved configuration
- Updated iframe URL to use OFFICIAL_URL for better maintainability.
- Integrated server configuration to conditionally enable business features.
- Cleaned up code by removing commented-out lines and unnecessary eslint suppressions.
This update improves the flexibility and readability of the SubscriptionIframeWrapper component.
Signed-off-by: Innei <tukon479@gmail.com>
* ✨ feat: enhance link interception in SubscriptionIframeWrapper
- Updated the script to intercept both link clicks and window.open calls for better URL handling.
- Added functionality to resolve relative URLs to absolute before logging.
This improvement enhances the tracking of external link interactions within the iframe.
Signed-off-by: Innei <tukon479@gmail.com>
* 🔧 chore: clean up useIsCloudActive hook
- Removed unnecessary eslint suppressions related to react-hooks rules.
- Improved code readability by eliminating commented-out lines.
This update enhances the clarity and maintainability of the useIsCloudActive hook.
Signed-off-by: Innei <tukon479@gmail.com>
---------
Signed-off-by: Innei <tukon479@gmail.com>
* v2 init
* chore: update eslint suppressions and package dependencies
- Removed several eslint suppressions related to array sorting and reversing from eslint-suppressions.json to clean up the configuration.
- Updated @lobehub/lint package version from 2.0.0-beta.6 to 2.0.0-beta.7 in package.json for improvements and bug fixes.
- Made minor formatting adjustments in vitest.config.mts and various SKILL.md files for better readability and consistency.
Signed-off-by: Innei <tukon479@gmail.com>
* fix: clean up import statements and formatting
- Removed unnecessary whitespace in replaceComponentImports.ts for improved readability.
- Standardized import statements in contextEngineering.ts and createAgentExecutors.ts by adding missing spaces for consistency.
Signed-off-by: Innei <tukon479@gmail.com>
* chore: update eslint suppressions and clean up code formatting
* 🐛 fix: use vi.hoisted for mock variable initialization
Fix TDZ error in persona service test by using vi.hoisted() to ensure
mock variables are available when vi.mock factory runs.
---------
Signed-off-by: Innei <tukon479@gmail.com>
AmAzing-