2 Commits

Author	SHA1	Message	Date
Brad Simantel	3039201f1b	Add should-flag-change advisory skill + eval suite (#101) ... * Add should-flag-change advisory skill + eval suite New read-only skill that decides whether a code change should ship behind a LaunchDarkly feature flag. Invoked ad hoc or in CI on a PR: it reads the diff and surrounding code, weighs a decision framework (favoring flags for user-facing and risky changes, weighting false negatives over false positives), and ends with a structured recommend-flag verdict. It never creates or modifies flags. - skills/feature-flags/should-flag-change: SKILL.md (read-only scope boundary, confusion-matrix decision framework, recommend-flag verdict contract) plus README and marketplace.json - evals: recommend-flag tool definition + mock, git_diff provider var injected as a <git_diff> tag, and a two-tier suite (judgment vs agentic providers, targeted per fixture) with 5 labeled fixtures - evals/scripts/build-pr-fixture.js: build a fixture from a real PR via gh pr diff + refs/pull/N/head, bootstrapping the label from LD SDK call sites - wiring: package.json scripts, _manifest.js, README skill list, skills.json * Add already-gated fixtures, PR-fixture tooling, and templating fix Extends the should-flag-change eval suite and its fixture tooling based on testing against real pull requests. - promptfooconfig.yaml: add three synthetic fixtures for the "already gated" case the suite didn't cover — a change that ships behind a flag should get recommend: false ("already handled"), and its ungated mirror should get recommend: true. Uses public SDK idioms; no proprietary source. - provider: fix a Nunjucks var-render crash on diffs containing `{{ ... }}` (JSX props, Go templates). Fixtures may wrap such content in a `{% raw %}` block; stripRawWrapper removes it before the agent sees the diff. - build-pr-fixture.js: read changed files via the GitHub contents API instead of fetching refs/pull/N/head (avoids cloning a large external repo); add --flag-pattern / --flag-token / --flag-file-pattern to teach it a codebase's flag conventions at call time, and --counterfactual to build positive-recall fixtures by stripping the flag gate. Kept repo-agnostic — no baked-in conventions. - tests: unit-test stripRawWrapper and the builder's pure functions (addedLines, detectLdUsage, splitDiffByFile, dropFilesFromDiff, extractFlagTokens, stripFlagGate). * Combine should-flag-change and flag-and-release-change onto one branch Bring the flag-and-release-change skill (originally PR #102) onto the should-flag-change branch (#101) so the advisory "decide" step and the "apply" step ship together. Only the content of #102's two flag-and-release commits is included -- not that branch's unrelated observability skills (which sit on an unmerged #99 commit). README skill list updated and skills.json regenerated from the catalog script. * Improve both flag skills with ideas from auto-factory and gonfalon should-flag-change: - Ancestor-gate analysis: detect an enclosing flag the change already sits behind, and judge on its rollout state. - Prerequisite/dependency signal as a reason to flag. - Optional risk/blast-radius field on the verdict, orthogonal to confidence (added to the eval tool schema, additive/non-breaking). - Refactor-that-changes-a-contract case; net-new-vs-incremental measurability nuance; unverified-claim confidence discipline. flag-and-release-change: - Capture and honor human release intent (release / hold / notBefore / segment / prerequisite), fail-closed; three-layer precedence (intent -> override -> policy -> default). - "The deploy is not the release" framing. - Fail-closed on non-409 flag-creation errors. - Paired flag-on/flag-off tests run to green before push. - Metric-adequacy check for guarded policies; prerequisite coupling. - Off-path-invokes-no-new-code invariant; edge cases for duplicate config, registering before the PR exists, and double-toggling. Sources: launchdarkly-labs/launchdarkly-auto-factory (AI-config prompts, ADRs) and launchdarkly/gonfalon .agents/skills. Portable-safe only -- repo-specific specifics (env matrix, release tags, context kinds) left out as the per-repo customization surface. * Extract flag-release skill; slim flag-and-release-change to an orchestrator Decompose the "apply" half into composable, harness-callable units, matching gonfalon's planning/add-flag/rollout shape while staying generic: - New skill feature-flags/flag-release: records a flag's automated rollout for a PR (match-release-policies preview, release intent + fail-closed hold, precedence, prerequisites, metric-adequacy). Operates on an existing flag; creates no flags and edits no code. auto-release.md moves here as its core reference. - flag-and-release-change becomes a thin portable orchestrator that composes should-flag-change -> launchdarkly-flag-create -> flag-release, owning only the PR workflow (clone/diff/push) and plan->implement sequencing. Notes that an automation harness can bypass it and invoke the three atoms directly. Repo-specific values (env matrix, release tags, targeting context) remain out of these generic skills -- that is the AgentControl customization seam. * Add eval coverage for the new flag decision/release logic should-flag-change: - New fixtures: ancestor-gate (agentic; explores to an enclosing flag not in the diff) and prerequisite-dependency (judgment). - risk-calibration asserts folded into the auth (expect high) and docs-only (expect low/absent) fixtures. All pass. New flag-release suite (adds match-release-policies / list-release-policies / create-automated-rollout-config tool defs + mocks; environmentKey mock replacement): - Release-on-merge happy path: previews the policy, records both envs. PASS. - Hold intent: KNOWN-RED tracked signal. sonnet-4-6 records the held env as `policy` (wrongly believing policy holds); opus omits it correctly. The assertion is correct (a held env must be omitted); do not weaken it. New flag-and-release-change suite (both PASS): - Plan phase is side-effect-free (no mutating MCP calls during planning). - Fails closed: a create-flag 403 (via the restricted-project mock hook) stops the run before any release is recorded. Wiring: manifest + package.json scripts for both new suites; READMEs for flag-release and flag-and-release-change. Sharpened flag-release's hold-honoring wording (helps stronger models; sonnet gap tracked above). 90/90 unit tests pass. * Address review feedback on the should-flag-change PR 1. Catalog leaked third-party skills from evals/node_modules (dotenv, dotenvx, playwright) into the public skills.json, and CI's generate_catalog.py --check failed on the mismatch. Add node_modules to the validator's EXCLUDED_DIRS (generate_catalog reuses it) and regenerate skills.json — back to 44 skills, --check clean. 2. Enforce the recommend-flag contract. A new suite-wide verdict_contract assertion requires the tool to be called EXACTLY once and as the final tool call; a run that calls it early, twice, or keeps working afterward now fails instead of passing on the first call's boolean. 3. Guard the read-only safety contract. A new read_only_guard assertion bans Edit/Write and mutating shell commands (redirects, rm/mv/cp, git commit/push, package installs) across the suite, so the agentic tier can't modify code or state and still pass. Previously only flag-mutating MCP tools were checked. Both new assertions pass on all 10 should-flag-change fixtures. * Declare js-yaml as a direct dependency of the eval tooling build-pr-fixture.js required js-yaml via an explicit ../node_modules path, but js-yaml was only present transitively (via promptfoo). A promptfoo bump that dropped or relocated it would break the fixture builder and its unit test. Add js-yaml (^4.1.1, already the resolved version) to evals devDependencies, update the lockfile, and use a plain require. Also drop the now-unused node:path import. * Enrich should-flag-change + add targeting context-availability reference (#108) * feat(feature-flags): enrich should-flag-change and add context-availability reference Layer portable, de-LaunchDarkly-internal lessons from our flag-planning skills onto the public feature-flag skills. should-flag-change: - Add an explicit, named user-observability test as a gate before any `recommend: false` verdict. - Add a `verdict` field (suggested | already-flagged | not-suited) to the recommend-flag output, keeping already-flagged (protected by an existing or ancestor gate) distinct from not-suited (nothing to flag). `recommend` stays the boolean a CI check keys on. - Generalize a decision-posture tie-breaker (conservative vs. low-overhead) for genuinely balanced calls, without internal "dogfood" wording. Add a new SDK-agnostic targeting context-availability reference: match the context kind to the surface where the flag is read (server/client/anonymous), key vs attribute, and rollout bucketing. Wire it into flag-targeting and flag-create, and reference it from should-flag-change. Bump versions and regenerate skills.json. Co-authored-by: Cursor <cursoragent@cursor.com> * test(feature-flags): cover verdict taxonomy and targeting-context reasoning - should-flag-change suite: assert the verdict field across the already-flagged, not-suited, and suggested fixtures, and validate the enum plus recommend/verdict agreement in the ambiguous fixture. - flag-create suite: add a fixture where a client-side flag is asked to target a server-only signal, asserting the agent flags that browser context can't carry it and suggests an available approach. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> * Let the eval gate tolerate documented known-red fixtures The flag-release suite carries an intentionally-red fixture (Sonnet mishandles a held production environment) that must stay red until the skill or model closes the gap. Previously any suite below 75% failed the CI job, so this tracked signal blocked merge. Declare such fixtures via a knownRed allowlist in _manifest.js (matched by the fixture description, which promptfoo stores under testCase.description). aggregate.js now keeps the honest score/badge red while gating on a gateScore that excludes known-red fixtures, so only unexpected failures fail CI. A known-red fixture that starts passing is surfaced so its entry can be removed. * Revert "Let the eval gate tolerate documented known-red fixtures" This reverts commit 5e870937e4. * Restore known-red eval gate tolerance for flag-release hold fixture Sonnet still records held production as `policy` (auto-releases on merge), so the intentionally-red hold-intent fixture keeps failing the 75% suite gate. Re-apply the gateScore allowlist so the tracked signal stays visible without blocking CI. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(flag-release): make held-environment handling a mechanical step The hold-intent fixture failed because the model maps "hold production" to `releaseType: policy` — the tool describes `policy` as "defer to the environment's release policy on merge", and "defer" reads like "hold". The skill fought this with prose caveats, which the authors noted hadn't closed the gap. Restructure the Implement phase around a forced RELEASE-vs-HOLD bucket sort: the environments array is built only from the RELEASE bucket, HOLD means absence from the array, and the "policy defers to the policy, not to you" misreading is called out at the point of the call. Remove the known-red gate allowlist so the fixture counts again and the skill fix carries the suite. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(flag-release): de-trap the rollout tool description; name the hold-field hallucination Root-cause dig on the hold-intent fixture: the model doesn't fail from a bare reasoning gap — the eval's mock tool description diverged from the real hosted tool and fed it a trap. The real create-automated-rollout-config says policy "waits until merge, then performs the release"; the mock said policy "defers to the release policy", and "defer" reads as "hold", steering the model to keep a held env as policy. With louder omit-prose the model instead fabricated a holdUntil field — which the mock silently accepts (.passthrough), so it never learns the field is invalid. - Make the mock tool description faithful to the real tool: drop "defer", say "waits then performs", and state explicitly there is NO hold/notBefore/date field (omit the env to hold it; unknown fields are rejected). - Skill: name the exact tell — wanting holdUntil/notBefore/hold (or keeping an env as policy "so it waits") means the env is HOLD; drop the entry, don't invent a field. Date/reason go in the report. - Keep the known-red gate as a backstop since the 2-test suite is a binary gate that can flake even when the fix lands; the honest score still shows red if it regresses. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(evals): drop known-red backstop now flag-release hold fixture passes The de-trapped tool description + skill hallucination-naming flips Sonnet to the correct behavior: it omits the held environment (recorded [{staging,simple}], production absent) and scores 0.83 > 0.75. With a real fix in place the known-red gate tolerance would only mask a future regression, so remove it and let the fixture gate on its own. Co-authored-by: Cursor <cursoragent@cursor.com> * test(flag-release): broaden hold coverage and lock in the no-fabricated-field fix Expand the flag-release suite from 2 to 5 fixtures, all exercising the release-vs-hold sort the earlier fix addressed: - add a no_fabricated_fields assertion (rejects holdUntil/notBefore/hold on any recorded env) to the hold fixtures, locking in the regression fix - multi-env mixed intent (dev+staging release, production hold) — the sort must scale past two envs - single-target date hold (the only env is held, so nothing releases) - fail closed on ambiguous intent (release staging, hold unconfirmed production) Retire the stale KNOWN-RED comment on the original hold fixture (now a passing regression guard). More fixtures also stabilize the 75% gate: a single stochastic miss on a 5-test suite stays green, where on 2 tests it went red. Co-authored-by: Cursor <cursoragent@cursor.com> * test(flag-and-release-change): cover the orchestrator's decide gate and create step The orchestrator suite only had plan-side-effect-free and fail-closed fixtures. Add two more targeting its own distinct, sandbox-reachable decisions (the record step needs a git push the MCP-only provider can't do, and is covered directly by the flag-release suite): - decide gate: a docs-only change is judged not flag-worthy and creates/records nothing, even when told to "flag and release if warranted" - create step: an approved change produces exactly one boolean kill-switch, created OFF, never toggled on by hand (over-flag + created-OFF guards) Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Ramon Niebla <rniebla@launchdarkly.com> Co-authored-by: Cursor <cursoragent@cursor.com>	2026-07-21 11:50:05 -07:00
Ramon Niebla	d67914a5ca	Feat: Add skill scaffolding and Flag cleanup skill (#1)	2026-02-03 07:43:05 -08:00