mirror of
https://github.com/schpet/linear-cli.git
synced 2026-09-14 14:26:50 +08:00
Peter Schilling
07e7d4b54a
Issue #207 claims agents reading the skill skip dedicated CLI subcommands and reach for raw GraphQL via `linear api`. Before changing the skill text, this adds an eval that can actually measure that: codex exec runs each task prompt in a fully isolated environment (fresh CODEX_HOME + fake HOME so the globally installed skill can't leak in, recording shims for linear/curl/ npx/npm, workspace-write sandbox) and a deterministic grader classifies route choice and flag correctness from the recorded invocations. Cases: five recipe families with development + holdout prompts, plus two controls where GraphQL is genuinely the right route. Outcome rules were declared before the baseline ran (see evals/linear-cli-skill/README.md). Baseline result, 36 trials at low effort on gpt-5.6-sol: 30/30 supported tasks full success, 6/6 controls on linear api. The premise of #207 did not reproduce in this configuration — with the skill actually read, routing is already perfect. Two earlier baseline runs were voided during harness development because stateless canned outputs (issue view contradicting the subject's own update; ENG-prefixed identifiers for OPS-team requests) baited subjects into GraphQL investigation and contaminated the signal; the shim is now consistency-aware. Related to #207
11 lines
434 B
Bash
Executable File
11 lines
434 B
Bash
Executable File
#!/bin/bash
|
|
# Shim `curl` for the skill eval: records the attempt and fails like a network
|
|
# with no route to the API, so direct-HTTP fallbacks are observable but can
|
|
# never reach the real Linear API.
|
|
set -u
|
|
LOG="${LINEAR_SHIM_LOG:?LINEAR_SHIM_LOG must be set}"
|
|
printf '%s\x1f' x "$@" | jq -Rsc \
|
|
'{tool: "curl", argv: split("\u001f")[1:-1], stdin: ""}' >> "$LOG"
|
|
echo "curl: (6) Could not resolve host: api.linear.app" >&2
|
|
exit 6
|