Files
nexu-io__open-design/apps/daemon/tests/acp-format-usage.test.ts
Marc Chan 7787556cf1 fix(daemon): full ACP tool transcript and richer usage telemetry (#6057)
* fix(daemon): full ACP tool transcript and richer usage telemetry

Map all ACP tool_call frames to canonical tool_use/tool_result for
run.events, PostHog run_finished, and Langfuse spans. Preserve
artifact_count via Write/Edit normalization, tighten path extraction,
forward truncated results, and emit real tool startedAt.

Enrich usage scanning (thought_tokens, anthropic vs openai cache
semantics), canonical tool-family PostHog props, analytics recovery
finish snapshot, and shared content-tool redaction.

* fix(daemon): prefer ACP kind over noncanonical tool names

Normalize recognized ACP kinds (e.g. kind:read + name:read_file → Read)
before emitting tool transcripts so Langfuse content-tool redaction matches
canonical families and does not leak file contents.

Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build)

* fix(daemon): fail-closed redaction for unknown ACP tool payloads

Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build)

* fix(daemon): flush open ACP tools as errors on session fail

When a session dies mid-tool (timeout, exit, RPC error), emit pending
tool_use/tool_result pairs with isError so transcripts stay complete.

Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build)

* fix(daemon): redact ACP Bash stdout in tool transcripts

Execute tools can dump private files (e.g. cat .env); replace Bash
tool_result content with a length summary before the canonical transcript
ships so Langfuse cannot receive raw execute output.

Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build)

* fix(daemon): redact Linux home paths in Langfuse Bash tool spans

Extend local-path masking for ACP Bash partial redaction to cover
/home and /root (not only /Users and Windows Users), so Linux home
paths do not leak into Langfuse tool inputs when content telemetry is on.

Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build)

* fix(daemon): redact custom ACP tool names before Langfuse telemetry

Sanitize path/token-like kind:other tool names at transcript emit and
canonicalize non-allowlisted names for Langfuse span labels, toolName
metadata, and redaction placeholders so untrusted custom names cannot
leak even when content telemetry is off.

Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build)

* test(e2e): force-click unstable comment targets in preview iframe

UI P0 project-workspace timed out on comment attachment when Playwright
kept waiting for hero-title stability under auto-fit zoom / bridge inject.
Click once visible with force so the critical flow stays reliable.

Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build)

* fix(daemon): merge partial usage frames in reverse analytics scan

Keep provider_usage for thought/cache-only trailing usage events while
continuing the reverse scan to the latest primary input/output/total
record so run_finished and Langfuse do not drop complete token counts.

Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build)

* fix(daemon): flush open ACP tools before AMR no-output fail

Clean-flush pending non-think tools on successful session/prompt before
evaluating emittedConcreteToolEvent so missing terminal tool_call_update
does not mark a productive run as acp_no_visible_output.

Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build)

* fix(daemon): merge partial primary usage and flush tools on ACP abort

Require both input and output before ending the reverse usage scan so
output-only or input-only trailing frames still merge earlier counters.
Flush deferred open ACP tools as errored on abort so user-canceled runs
keep complete tool transcripts for Langfuse/PostHog.

Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build)

* fix(daemon): hash path-like ACP toolCallIds before telemetry

Adapters can embed user paths in toolCallId (e.g. read:/home/alice/.env).
Hash those values at tool_use/tool_result emit so Langfuse span ids and
metadata.toolCallId cannot leak secrets, while keeping the raw id as the
local frame-correlation map key.

Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build)

* fix(daemon): always hash ACP adapter toolCallIds for telemetry

Identifier-shaped secrets (sk-proj-*, ghp_*, JWTs) passed the prior
allowlist because they contain no path characters. Hash every non-empty
adapter toolCallId before tool_use/tool_result emission so Langfuse span
ids and metadata.toolCallId cannot leak tokens.

Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build)

* fix(daemon): keep reverse usage scan open for missing cache fields

When a newer usage frame already has input+output but omits cache
counters, continue scanning older frames so cache_read/cache_creation
still merge into run_finished and Langfuse.

Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build)

* fix(daemon): retain flushed ACP tool IDs against late terminals

Keep acpToolRunEventState entries after flushOpenAcpTools so a racy
tool_call_update cannot recreate the same id and emit a second
tool_use/tool_result pair after timeout, abort, or prompt flush.

Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build)

* fix(daemon): block kind:other Bash-like tool names

Custom ACP tools with kind:other must not emit Bash/shell/execute/terminal
names, which would unlock Langfuse partial redaction and leak rawInput.

Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build)

* fix(daemon): fail closed on Bash names without execute kind

ACP tools missing kind must not preserve name/title Bash labels that unlock
Langfuse partial redaction; only recognized execute-family kinds may emit Bash.

Generated-By: looper 0.11.0 (runner=fixer, agent=grok-build)
2026-07-27 04:08:27 +00:00

164 lines
4.2 KiB
TypeScript

import { describe, expect, it } from 'vitest';
import { formatUsage } from '../src/agent-protocol/acp/rpc.js';
import { scanRunEventsForUsageAnalytics } from '../src/run-analytics-observability.js';
describe('formatUsage', () => {
it('normalizes camelCase ACP usage including thought_tokens', () => {
expect(
formatUsage({
inputTokens: 100,
outputTokens: 20,
cachedReadTokens: 40,
thoughtTokens: 8,
totalTokens: 128,
}),
).toEqual({
input_tokens: 100,
output_tokens: 20,
cached_read_tokens: 40,
thought_tokens: 8,
total_tokens: 128,
});
});
it('normalizes snake_case usage aliases including cache creation', () => {
expect(
formatUsage({
input_tokens: 200,
output_tokens: 30,
cached_read_tokens: 50,
cache_creation_input_tokens: 10,
thought_tokens: 12,
total_tokens: 242,
}),
).toEqual({
input_tokens: 200,
output_tokens: 30,
cached_read_tokens: 50,
// Anthropic-style creation key is preserved (not collapsed to generic).
cache_creation_input_tokens: 10,
thought_tokens: 12,
total_tokens: 242,
});
});
it('preserves anthropic cache_read_input_tokens (does not map to cached_read_tokens)', () => {
expect(
formatUsage({
input_tokens: 10,
cache_read_input_tokens: 4,
}),
).toEqual({
input_tokens: 10,
cache_read_input_tokens: 4,
});
});
it('preserves openai-like cached_read_tokens separately from anthropic keys', () => {
expect(
formatUsage({
input_tokens: 100,
cached_read_tokens: 40,
cached_write_tokens: 5,
}),
).toEqual({
input_tokens: 100,
cached_read_tokens: 40,
cached_write_tokens: 5,
});
});
it('returns null for empty or non-object input', () => {
expect(formatUsage(null)).toBeNull();
expect(formatUsage(undefined)).toBeNull();
expect(formatUsage({})).toBeNull();
expect(formatUsage({ unknown: 1 })).toBeNull();
});
it('ignores non-finite numbers', () => {
expect(
formatUsage({
inputTokens: Number.NaN,
outputTokens: -1,
thoughtTokens: 3,
}),
).toEqual({ thought_tokens: 3 });
});
it('formatUsage anthropic cache fields scan as additive (cache_token_source anthropic)', () => {
const formatted = formatUsage({
cache_read_input_tokens: 10,
input_tokens: 5,
});
expect(formatted).toEqual({
input_tokens: 5,
cache_read_input_tokens: 10,
});
const scanned = scanRunEventsForUsageAnalytics(
[
{
event: 'agent',
data: {
type: 'usage',
usage: formatted,
},
},
],
'claude-sonnet',
0,
);
expect(scanned).toMatchObject({
input_tokens_provider: 5,
// Additive: effective = input + cache_read
input_tokens_effective: 15,
cache_read_input_tokens: 10,
uncached_input_tokens: 5,
cache_token_source: 'anthropic',
token_count_source: 'provider_usage',
});
expect(scanned.cache_hit_ratio).toBeCloseTo(10 / 15);
});
it('formatUsage openai cacheCreationTokens scan as inclusive (not anthropic additive)', () => {
const formatted = formatUsage({
inputTokens: 100,
cachedReadTokens: 40,
cacheCreationTokens: 5,
});
expect(formatted).toEqual({
input_tokens: 100,
cached_read_tokens: 40,
cache_creation_tokens: 5,
});
const scanned = scanRunEventsForUsageAnalytics(
[
{
event: 'agent',
data: {
type: 'usage',
usage: formatted,
},
},
],
'gpt-4.1',
0,
);
expect(scanned).toMatchObject({
input_tokens_provider: 100,
// Inclusive: effective = input (cache read is a subset)
input_tokens_effective: 100,
cache_read_input_tokens: 40,
cache_creation_input_tokens: 5,
uncached_input_tokens: 60,
cache_token_source: 'openai',
token_count_source: 'provider_usage',
});
expect(scanned.cache_hit_ratio).toBeCloseTo(40 / 100);
});
});