mirror of
https://github.com/nexu-io/open-design.git
synced 2026-09-20 06:15:06 +08:00
maoxin1234
55b3762afe
* feat(daemon): capture per-request request_id + token usage for claude_code runs The claude-stream-json parser read every assistant message's `message.id` only to dedup streamed text/thinking and then dropped it, and never read `message.usage` at all — so a run with N model requests collapsed into a single run-level `result.usage` record with no request id. That left request-level cost/percentile analysis stuck at run-level (~0.9% coverage). Surface the per-request granularity the wire already carries: - claude-stream.ts emits a new `request_usage` event per assistant message, keyed by `message.id` (the provider `msg_…` request id), carrying the per-message input/output/cache tokens via a `perRequestUsageFrom` helper. - packages/contracts adds `request_usage` to both the live SSE `DaemonAgentPayload` and the `PersistedAgentEvent` union; server.ts maps the live event to its persisted form. - mocks/lib/format-claude.mjs now emits per-message `usage` summing to the run-level `result.usage`, so the capture path is verifiable by local replay without burning provider budget. The per-request token sum reconciles against the run-level `result.usage` aggregate — the correctness anchor. New daemon test plus regenerated claude golden cover both the synthetic stream and the mock-replay path. No change to run execution; only telemetry/contract surface is added. claude-stream- json only for now; other runtime families ship per-request shapes in follow-ups. Closes #4610 * feat(daemon): thread per-request usage through Langfuse + PostHog run telemetry Addresses review on #4620: the parser emitted the per-request records but they stopped at SSE/message persistence and never reached the telemetry sinks named in #4610's acceptance criteria, so PostHog/Langfuse still had no per-request request_id + token coverage. Wire the records through the existing run telemetry choke point: - run-analytics-observability.ts adds `scanRunEventsForPerRequestUsageAnalytics`, a pure scanner that collects the per-request records keyed by request_id and checks the reconciliation invariant (per-request token sum == run-level `result.usage`). Both sinks read this one source. - langfuse-bridge.ts `collectAgentEvents` exports one `agent-request-usage` Langfuse event per model request, keyed by the provider request id. - routes/runs.ts adds `run_finished` coverage properties (`request_usage_count`, per-request token sums, and `request_usage_reconciles_aggregate`) so the PostHog request-level surface lifts off the ~0.9% floor; contract fields added to RunFinishedProps. Test extended: a multi-request claude run produces telemetry records keyed by request_id whose token sum reconciles with `result.usage`, plus a non-reconciling guard. langfuse-bridge + run-analytics suites stay green. * fix(daemon): fold cache tokens into per-request usage reconciliation Addresses review on #4620: request_usage_reconciles_aggregate compared only input/output tokens, so a run whose prompt/completion tokens matched but whose cache creation/read tokens drifted still reported `true` — letting the new PostHog/Langfuse telemetry certify the #4610 invariant when part of `result.usage` had not reconciled. scanRunEventsForPerRequestUsageAnalytics now reads the aggregate `cache_creation_input_tokens` / `cache_read_input_tokens` and folds every field `result.usage` reported into the boolean (each compared only when the aggregate carries it, mirroring the input/output leniency). Added a fixture where cache tokens mismatch while input/output still match, so the flag stays false and this cannot regress. * feat(daemon): map request_usage to durable PersistedAgentEvent * feat(daemon): wire per-request usage rollup into run_finished analytics lifecycle * ci: trigger fresh validation run for approved changes * feat(daemon): preserve per-request usage in truncation-proof run side ledger