mirror of
https://github.com/nexu-io/open-design.git
synced 2026-09-20 06:15:06 +08:00
7787556cf1
* fix(daemon): full ACP tool transcript and richer usage telemetry Map all ACP tool_call frames to canonical tool_use/tool_result for run.events, PostHog run_finished, and Langfuse spans. Preserve artifact_count via Write/Edit normalization, tighten path extraction, forward truncated results, and emit real tool startedAt. Enrich usage scanning (thought_tokens, anthropic vs openai cache semantics), canonical tool-family PostHog props, analytics recovery finish snapshot, and shared content-tool redaction. * fix(daemon): prefer ACP kind over noncanonical tool names Normalize recognized ACP kinds (e.g. kind:read + name:read_file → Read) before emitting tool transcripts so Langfuse content-tool redaction matches canonical families and does not leak file contents. Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build) * fix(daemon): fail-closed redaction for unknown ACP tool payloads Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build) * fix(daemon): flush open ACP tools as errors on session fail When a session dies mid-tool (timeout, exit, RPC error), emit pending tool_use/tool_result pairs with isError so transcripts stay complete. Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build) * fix(daemon): redact ACP Bash stdout in tool transcripts Execute tools can dump private files (e.g. cat .env); replace Bash tool_result content with a length summary before the canonical transcript ships so Langfuse cannot receive raw execute output. Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build) * fix(daemon): redact Linux home paths in Langfuse Bash tool spans Extend local-path masking for ACP Bash partial redaction to cover /home and /root (not only /Users and Windows Users), so Linux home paths do not leak into Langfuse tool inputs when content telemetry is on. Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build) * fix(daemon): redact custom ACP tool names before Langfuse telemetry Sanitize path/token-like kind:other tool names at transcript emit and canonicalize non-allowlisted names for Langfuse span labels, toolName metadata, and redaction placeholders so untrusted custom names cannot leak even when content telemetry is off. Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build) * test(e2e): force-click unstable comment targets in preview iframe UI P0 project-workspace timed out on comment attachment when Playwright kept waiting for hero-title stability under auto-fit zoom / bridge inject. Click once visible with force so the critical flow stays reliable. Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build) * fix(daemon): merge partial usage frames in reverse analytics scan Keep provider_usage for thought/cache-only trailing usage events while continuing the reverse scan to the latest primary input/output/total record so run_finished and Langfuse do not drop complete token counts. Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build) * fix(daemon): flush open ACP tools before AMR no-output fail Clean-flush pending non-think tools on successful session/prompt before evaluating emittedConcreteToolEvent so missing terminal tool_call_update does not mark a productive run as acp_no_visible_output. Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build) * fix(daemon): merge partial primary usage and flush tools on ACP abort Require both input and output before ending the reverse usage scan so output-only or input-only trailing frames still merge earlier counters. Flush deferred open ACP tools as errored on abort so user-canceled runs keep complete tool transcripts for Langfuse/PostHog. Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build) * fix(daemon): hash path-like ACP toolCallIds before telemetry Adapters can embed user paths in toolCallId (e.g. read:/home/alice/.env). Hash those values at tool_use/tool_result emit so Langfuse span ids and metadata.toolCallId cannot leak secrets, while keeping the raw id as the local frame-correlation map key. Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build) * fix(daemon): always hash ACP adapter toolCallIds for telemetry Identifier-shaped secrets (sk-proj-*, ghp_*, JWTs) passed the prior allowlist because they contain no path characters. Hash every non-empty adapter toolCallId before tool_use/tool_result emission so Langfuse span ids and metadata.toolCallId cannot leak tokens. Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build) * fix(daemon): keep reverse usage scan open for missing cache fields When a newer usage frame already has input+output but omits cache counters, continue scanning older frames so cache_read/cache_creation still merge into run_finished and Langfuse. Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build) * fix(daemon): retain flushed ACP tool IDs against late terminals Keep acpToolRunEventState entries after flushOpenAcpTools so a racy tool_call_update cannot recreate the same id and emit a second tool_use/tool_result pair after timeout, abort, or prompt flush. Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build) * fix(daemon): block kind:other Bash-like tool names Custom ACP tools with kind:other must not emit Bash/shell/execute/terminal names, which would unlock Langfuse partial redaction and leak rawInput. Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build) * fix(daemon): fail closed on Bash names without execute kind ACP tools missing kind must not preserve name/title Bash labels that unlock Langfuse partial redaction; only recognized execute-family kinds may emit Bash. Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build)
164 lines
4.2 KiB
TypeScript
164 lines
4.2 KiB
TypeScript
import { describe, expect, it } from 'vitest';
|
|
|
|
import { formatUsage } from '../src/agent-protocol/acp/rpc.js';
|
|
import { scanRunEventsForUsageAnalytics } from '../src/run-analytics-observability.js';
|
|
|
|
describe('formatUsage', () => {
|
|
it('normalizes camelCase ACP usage including thought_tokens', () => {
|
|
expect(
|
|
formatUsage({
|
|
inputTokens: 100,
|
|
outputTokens: 20,
|
|
cachedReadTokens: 40,
|
|
thoughtTokens: 8,
|
|
totalTokens: 128,
|
|
}),
|
|
).toEqual({
|
|
input_tokens: 100,
|
|
output_tokens: 20,
|
|
cached_read_tokens: 40,
|
|
thought_tokens: 8,
|
|
total_tokens: 128,
|
|
});
|
|
});
|
|
|
|
it('normalizes snake_case usage aliases including cache creation', () => {
|
|
expect(
|
|
formatUsage({
|
|
input_tokens: 200,
|
|
output_tokens: 30,
|
|
cached_read_tokens: 50,
|
|
cache_creation_input_tokens: 10,
|
|
thought_tokens: 12,
|
|
total_tokens: 242,
|
|
}),
|
|
).toEqual({
|
|
input_tokens: 200,
|
|
output_tokens: 30,
|
|
cached_read_tokens: 50,
|
|
// Anthropic-style creation key is preserved (not collapsed to generic).
|
|
cache_creation_input_tokens: 10,
|
|
thought_tokens: 12,
|
|
total_tokens: 242,
|
|
});
|
|
});
|
|
|
|
it('preserves anthropic cache_read_input_tokens (does not map to cached_read_tokens)', () => {
|
|
expect(
|
|
formatUsage({
|
|
input_tokens: 10,
|
|
cache_read_input_tokens: 4,
|
|
}),
|
|
).toEqual({
|
|
input_tokens: 10,
|
|
cache_read_input_tokens: 4,
|
|
});
|
|
});
|
|
|
|
it('preserves openai-like cached_read_tokens separately from anthropic keys', () => {
|
|
expect(
|
|
formatUsage({
|
|
input_tokens: 100,
|
|
cached_read_tokens: 40,
|
|
cached_write_tokens: 5,
|
|
}),
|
|
).toEqual({
|
|
input_tokens: 100,
|
|
cached_read_tokens: 40,
|
|
cached_write_tokens: 5,
|
|
});
|
|
});
|
|
|
|
it('returns null for empty or non-object input', () => {
|
|
expect(formatUsage(null)).toBeNull();
|
|
expect(formatUsage(undefined)).toBeNull();
|
|
expect(formatUsage({})).toBeNull();
|
|
expect(formatUsage({ unknown: 1 })).toBeNull();
|
|
});
|
|
|
|
it('ignores non-finite numbers', () => {
|
|
expect(
|
|
formatUsage({
|
|
inputTokens: Number.NaN,
|
|
outputTokens: -1,
|
|
thoughtTokens: 3,
|
|
}),
|
|
).toEqual({ thought_tokens: 3 });
|
|
});
|
|
|
|
it('formatUsage anthropic cache fields scan as additive (cache_token_source anthropic)', () => {
|
|
const formatted = formatUsage({
|
|
cache_read_input_tokens: 10,
|
|
input_tokens: 5,
|
|
});
|
|
expect(formatted).toEqual({
|
|
input_tokens: 5,
|
|
cache_read_input_tokens: 10,
|
|
});
|
|
|
|
const scanned = scanRunEventsForUsageAnalytics(
|
|
[
|
|
{
|
|
event: 'agent',
|
|
data: {
|
|
type: 'usage',
|
|
usage: formatted,
|
|
},
|
|
},
|
|
],
|
|
'claude-sonnet',
|
|
0,
|
|
);
|
|
|
|
expect(scanned).toMatchObject({
|
|
input_tokens_provider: 5,
|
|
// Additive: effective = input + cache_read
|
|
input_tokens_effective: 15,
|
|
cache_read_input_tokens: 10,
|
|
uncached_input_tokens: 5,
|
|
cache_token_source: 'anthropic',
|
|
token_count_source: 'provider_usage',
|
|
});
|
|
expect(scanned.cache_hit_ratio).toBeCloseTo(10 / 15);
|
|
});
|
|
|
|
it('formatUsage openai cacheCreationTokens scan as inclusive (not anthropic additive)', () => {
|
|
const formatted = formatUsage({
|
|
inputTokens: 100,
|
|
cachedReadTokens: 40,
|
|
cacheCreationTokens: 5,
|
|
});
|
|
expect(formatted).toEqual({
|
|
input_tokens: 100,
|
|
cached_read_tokens: 40,
|
|
cache_creation_tokens: 5,
|
|
});
|
|
|
|
const scanned = scanRunEventsForUsageAnalytics(
|
|
[
|
|
{
|
|
event: 'agent',
|
|
data: {
|
|
type: 'usage',
|
|
usage: formatted,
|
|
},
|
|
},
|
|
],
|
|
'gpt-4.1',
|
|
0,
|
|
);
|
|
|
|
expect(scanned).toMatchObject({
|
|
input_tokens_provider: 100,
|
|
// Inclusive: effective = input (cache read is a subset)
|
|
input_tokens_effective: 100,
|
|
cache_read_input_tokens: 40,
|
|
cache_creation_input_tokens: 5,
|
|
uncached_input_tokens: 60,
|
|
cache_token_source: 'openai',
|
|
token_count_source: 'provider_usage',
|
|
});
|
|
expect(scanned.cache_hit_ratio).toBeCloseTo(40 / 100);
|
|
});
|
|
});
|