Files
callstack__agent-device/test/integration/provider-scenarios/transcript.test.ts
Michał Pierzchała 95f838f514 test: stop pinning settle capture counts against a wall-clock loop (#1307)
* test: model the UI in settle-observation fixtures instead of a capture count

settle-observation's "contention flakes" were a zero-margin comparison
meeting a 1ms clock skew, not generic load.

runStableCaptureLoop derives pollMs = min(300, max(25, quietMs)), so the
test's settleQuietMs: 25 made pollMs === quietMs, and the settle check
(one sleep(25) plus capture time, against >= 25) a 0ms margin. Node's
setTimeout(25) advances Date.now() by only 24ms in 0.13% of calls idle
and 0.63% under load, because libuv's timers and Date.now() read
different clocks. On a 24 the loop takes a third capture that the
transcript never scripted, and settle's best-effort catch reports the
resulting throw as settled: false.

The fixtures now model the surface rather than the runner's speed: a
quiet UI serves the settled tree to every capture, a busy one a fresh
tree per capture, via new transcript `repeat` entries and result
factories. The snapshot-floor economy guard survives as a bound (2-3
captures), and the follow-up's "no fresh capture" cost — previously
implied by the exact transcript — is now asserted directly.

Production is untouched: the same zero margin only costs a wasted extra
capture and poll there, filed separately as #1306.

* test: stop pinning the settle capture count in the iOS contract scenario too

A sweep for the same bug class found direct-ios-selector's settleObservation
scenario carrying the identical 0ms margin: settleQuietMs: 25 (so pollMs ===
quietMs) against a consume-once transcript scripting exactly two settle
captures, with no injected clock. It has not lost the coin flip in CI yet,
but it fails the same way when it does — a third capture finds no entry and
settle's best-effort catch reports settled: false.

Same fix: the fixture models a quiet UI (every settle capture sees the same
tree) instead of scripting how many captures fit in a wall-clock window.

The rest of the sweep was clean. The other contract scenarios and the
interaction runtime tests already use clamped mocks that repeat the last
snapshot, so any capture count is tolerated; the fake-clock tests are correct
to pin exact counts.

* style: oxfmt quietRunnerSnapshotEntry signature

* test: make one-shot-outranks-repeat a real transcript rule (P2 review)

The review is right: `one-shot entries still outrank a repeat entry` asserted
a guarantee the lookup did not provide. It passed only because the one-shot
happened to be declared first — unordered lookup took the first match, so a
repeat declared ahead of a matching one-shot shadowed it forever and left it
permanently unconsumed. Both failure modes reproduce; the reverse-order test
added here fails on the previous implementation.

Unordered lookup now searches matching one-shots before repeats, so
outranking holds whatever the declaration order. A repeat is documented as
its command's fallback.

Ordered transcripts now reject repeats at construction: ordered lookup only
ever reads the head, so a repeat there never advances and strands every entry
behind it. Refusing the combination beats failing later as a confusing
"Provider command mismatch".

Coverage added for both: reverse declaration order, and ordered + repeat.
2026-07-16 20:13:07 +02:00

116 lines
4.6 KiB
TypeScript

import assert from 'node:assert/strict';
import { test } from 'vitest';
import { createOrderedProviderTranscript, createProviderTranscript } from './transcript.ts';
test('provider transcript matches expected calls without requiring incidental order', () => {
const transcript = createProviderTranscript([
{ command: 'ios.runner.snapshot', result: { ok: 'snapshot' } },
{ command: 'ios.runner.tap', request: { command: 'tap', x: 10, y: 20 }, result: { ok: 'tap' } },
]);
assert.deepEqual(transcript.next('ios.runner.tap', { command: 'tap', x: 10, y: 20 }), {
ok: 'tap',
});
assert.deepEqual(transcript.next('ios.runner.snapshot'), { ok: 'snapshot' });
transcript.assertComplete();
});
test('repeat entries serve every matching call and never count as unconsumed', () => {
const transcript = createProviderTranscript([
{ command: 'ios.runner.snapshot', repeat: true, result: { ok: 'snapshot' } },
{ command: 'ios.runner.tap', result: { ok: 'tap' } },
]);
assert.deepEqual(transcript.next('ios.runner.snapshot'), { ok: 'snapshot' });
assert.deepEqual(transcript.next('ios.runner.snapshot'), { ok: 'snapshot' });
assert.deepEqual(transcript.next('ios.runner.tap'), { ok: 'tap' });
// The repeat entry stays pending forever; only the one-shot tap had to land.
transcript.assertComplete();
assert.equal(transcript.calls.length, 3);
});
test('one-shot entries still outrank a repeat entry and remain required', () => {
const transcript = createProviderTranscript([
{ command: 'ios.runner.snapshot', result: { ok: 'first' } },
{ command: 'ios.runner.snapshot', repeat: true, result: { ok: 'rest' } },
]);
assert.deepEqual(transcript.next('ios.runner.snapshot'), { ok: 'first' });
assert.deepEqual(transcript.next('ios.runner.snapshot'), { ok: 'rest' });
assert.deepEqual(transcript.next('ios.runner.snapshot'), { ok: 'rest' });
transcript.assertComplete();
});
test('a repeat declared before a matching one-shot does not shadow it', () => {
// Outranking is a rule, not an accident of declaration order: taking the
// first match would serve the repeat forever and strand the one-shot.
const transcript = createProviderTranscript([
{ command: 'ios.runner.snapshot', repeat: true, result: { ok: 'rest' } },
{ command: 'ios.runner.snapshot', result: { ok: 'first' } },
]);
assert.deepEqual(transcript.next('ios.runner.snapshot'), { ok: 'first' });
assert.deepEqual(transcript.next('ios.runner.snapshot'), { ok: 'rest' });
assert.deepEqual(transcript.next('ios.runner.snapshot'), { ok: 'rest' });
transcript.assertComplete();
});
test('ordered transcripts reject repeat entries outright', () => {
// Ordered lookup only ever reads entry 0, so a repeat there never advances
// and makes every later entry unreachable. Refuse the combination instead of
// failing later as a confusing command mismatch.
assert.throws(
() =>
createOrderedProviderTranscript([
{ command: 'ios.runner.snapshot', repeat: true, result: { ok: 'snapshot' } },
{ command: 'ios.runner.tap', result: { ok: 'tap' } },
]),
/repeat/i,
);
});
test('unconsumed one-shot entries still fail assertComplete alongside a repeat entry', () => {
const transcript = createProviderTranscript([
{ command: 'ios.runner.snapshot', repeat: true, result: { ok: 'snapshot' } },
{ command: 'ios.runner.tap', result: { ok: 'tap' } },
]);
transcript.next('ios.runner.snapshot');
assert.throws(() => transcript.assertComplete(), /Unconsumed provider transcript entries.*tap/s);
});
test('a result factory is invoked per call, so repeated calls can return fresh results', () => {
let capture = 0;
const transcript = createProviderTranscript([
{
command: 'ios.runner.snapshot',
repeat: true,
result: () => {
capture += 1;
return { label: `Loading ${capture}` };
},
},
]);
assert.deepEqual(transcript.next('ios.runner.snapshot'), { label: 'Loading 1' });
assert.deepEqual(transcript.next('ios.runner.snapshot'), { label: 'Loading 2' });
// The recorded call carries the resolved result, not the factory.
assert.deepEqual(
transcript.calls.map((call) => call.result),
[{ label: 'Loading 1' }, { label: 'Loading 2' }],
);
});
test('ordered provider transcript remains available when ordering is the contract', () => {
const transcript = createOrderedProviderTranscript([
{ command: 'ios.runner.snapshot', result: { ok: 'snapshot' } },
{ command: 'ios.runner.tap', result: { ok: 'tap' } },
]);
assert.throws(
() => transcript.next('ios.runner.tap'),
/Provider command mismatch|Unexpected provider call/,
);
});