Expose canonical user URLs in team and workspace member JSON so the skill can resolve mentions without guessing profile slugs. Document team-first URL mentions and Linear collapsible syntax.
Add a stubbed Claude forward eval that captures submitted Markdown. The frozen cases improved from 1/4 to 4/4 while preserving the verbatim-body control.
Fixes#112
Agents asked to put a visible screenshot on an issue reach for
`issue attach`, which uploads the file but creates a sidebar link
attachment that never renders inline — while the success output
("Attachment created") convinces them the image is visible. The working
path, `issue comment add --attach`, has existed since v2.0.0 but nothing
pointed at it: the skill had no image guidance and the flag was buried in
a reference table.
Three coordinated changes, validated as experiment 2 of the skill eval:
- Skill: a Common Tasks recipe for visible images via
`issue comment add --attach`, with an explicit warning about
`issue attach`'s sidebar-only behavior.
- CLI: `issue attach` now says it created a sidebar link attachment,
and for images prints a copy-pasteable hint suggesting
`issue comment add --attach` (shell-quoted, --public preserved).
Help descriptions updated on both commands.
- Eval: new frozen image family (trap-phrased development prompt,
comment-phrased holdout) plus a sidebar-control case graded on
positionals, with pre-declared outcome rules, CLI/API control split,
binary-safe fixture checks, and version-matched shim output.
Result (rules frozen before baseline): image-development went 0/3 to 3/3
— every baseline trial fell into the attach trap and wrongly reported
success; every post-change trial routed straight to the recipe. Image
family 3/6 to 6/6 lands in the pre-declared partial-baseline band, so it
is reported as consistent with improvement (exploratory Fisher p = 0.09)
rather than confirmed. Controls held except one known npx-version-check
grader artifact, adjudicated by an Opus gold-label pass (17/18 agreement
with the deterministic grader).
Adds ~7 copy-pasteable recipes near the top of SKILL.template.md (and the
generated SKILL.md): filtered queries via issue query (with the issue
list/mine alias gotcha spelled out), my-issues, create with
--description-file and --no-interactive, update state/assignee/labels
(noting label replacement semantics), comment from file, view/URL. The
boundary note stays generic so it doesn't teach the eval's control answers.
Post-change eval, same frozen 36-trial protocol as the baseline: 29/30
supported tasks full success, holdout 15/15, controls 6/6 still correctly
choosing linear api — no overcorrection from the new recipes. The single
failure was a subject first trying the skill's documented npx alternative
(npx @schpet/linear-cli issue create ...), which the frozen grader counts
as a bypass; it was not a GraphQL fallback and the task then completed
correctly via the CLI. With the baseline already at 30/30, the eval finds
no measurable routing headroom at this configuration; the change is
validated as non-regressing rather than as an improvement. See
evals/linear-cli-skill/results/comparison.md.
Related to #207
Issue #207 claims agents reading the skill skip dedicated CLI subcommands
and reach for raw GraphQL via `linear api`. Before changing the skill text,
this adds an eval that can actually measure that: codex exec runs each task
prompt in a fully isolated environment (fresh CODEX_HOME + fake HOME so the
globally installed skill can't leak in, recording shims for linear/curl/
npx/npm, workspace-write sandbox) and a deterministic grader classifies
route choice and flag correctness from the recorded invocations.
Cases: five recipe families with development + holdout prompts, plus two
controls where GraphQL is genuinely the right route. Outcome rules were
declared before the baseline ran (see evals/linear-cli-skill/README.md).
Baseline result, 36 trials at low effort on gpt-5.6-sol: 30/30 supported
tasks full success, 6/6 controls on linear api. The premise of #207 did not
reproduce in this configuration — with the skill actually read, routing is
already perfect. Two earlier baseline runs were voided during harness
development because stateless canned outputs (issue view contradicting the
subject's own update; ENG-prefixed identifiers for OPS-team requests)
baited subjects into GraphQL investigation and contaminated the signal;
the shim is now consistency-aware.
Related to #207
Peter Schilling