schpet__linear-cli/evals/linear-cli-skill/fixtures/comment.md at main

mirror of https://github.com/schpet/linear-cli.git synced 2026-09-14 14:26:50 +08:00

Files

T

Peter Schilling 07e7d4b54a Add codex-based eval harness for the linear-cli skill, with baseline

Issue #207 claims agents reading the skill skip dedicated CLI subcommands
and reach for raw GraphQL via `linear api`. Before changing the skill text,
this adds an eval that can actually measure that: codex exec runs each task
prompt in a fully isolated environment (fresh CODEX_HOME + fake HOME so the
globally installed skill can't leak in, recording shims for linear/curl/
npx/npm, workspace-write sandbox) and a deterministic grader classifies
route choice and flag correctness from the recorded invocations.

Cases: five recipe families with development + holdout prompts, plus two
controls where GraphQL is genuinely the right route. Outcome rules were
declared before the baseline ran (see evals/linear-cli-skill/README.md).

Baseline result, 36 trials at low effort on gpt-5.6-sol: 30/30 supported
tasks full success, 6/6 controls on linear api. The premise of #207 did not
reproduce in this configuration — with the skill actually read, routing is
already perfect. Two earlier baseline runs were voided during harness
development because stateless canned outputs (issue view contradicting the
subject's own update; ENG-prefixed identifiers for OPS-team requests)
baited subjects into GraphQL investigation and contaminated the signal;
the shim is now consistency-aware.

Related to #207

2026-07-23 10:46:18 -07:00

289 B

Raw Permalink Blame History

Reproduced this on staging with the following steps:

Log in via SSO
Open the billing page in a second tab
Refresh the first tab

The session cookie is refreshed with a mismatched domain, which is why the redirect loops. Fix candidate: pin the cookie domain in the auth callback.

289 B Raw Permalink Blame History

289 B

Raw Permalink Blame History