mirror of
https://github.com/adobe/skills.git
synced 2026-09-19 08:06:09 +08:00
david catalan
8443af4f3a
The new css-four-scope evals exercise the prototype skill which is
Phase 3 in the v2 pipeline (Phase 1=extract, Phase 2=direct, Phase
3=prototype, Phase 4=migrate). Was incorrectly labeled Phase 2 in
ef9d204.
Refs #147
Stardust v2 evals
Evaluation suite for the four-phase redesign pipeline plus the "open and reasoned" intent-reasoning principle that governs the master skill.
The v1 evals (brand-extract-from-url, briefings-from-prompt,
prototype-iterate, stardust-navigator, wireframes-render) were
tied to v1's stage decomposition (brand → briefings →
wireframes → prototype) and have been replaced wholesale to
match the v2 surface.
Format
Each eval lives in its own directory and contains exactly two files:
task.md— Setup, User prompt, Expected behavior. Human-readable scenario specification.criteria.json— Weighted scoring rubric. Each criterion has anid, aweight, and adescription.totalshould equal the sum of weights. Used by the eval runner to score the agent's output.
This format mirrors v1's structure (and the format other Adobe-skills plugins use), so the eval runner that worked for v1 should work for v2 evals without modification.
Evals in this suite
| Directory | Stage tested | What it pins down |
|---|---|---|
extract-multipage/ |
Phase 1 (extract) |
Multi-page crawl with cap + Playwright (not WebFetch) + correct file shapes + direct authoring of current PRODUCT.md / DESIGN.md. |
direct-from-phrase/ |
Phase 2 (direct) |
Dimensional restatement + at most two questions + plan-before-execution + direct authoring of target spec + direction.md trace. |
prototype-before-after/ |
Phase 3 (prototype) |
One proposed file per page + :root block + data attributes + content preserved + delegates to $impeccable craft + opens in browser. |
migrate-incremental/ |
Phase 4 (migrate) |
Both render paths (A and B) + nested index.html output + content preservation + idempotent skip on re-run. |
migrate-multi-template/ |
Phase 4 (migrate) |
Three render branches (Path A / A′ / B) + canon + modules + bespoke-slot promotion + broken-link reporting + color-reservation refusal. |
migrate-self-contained-bundle/ |
Phase 4 (migrate) |
Self-contained zip-and-deploy bundle — six asset detection shapes, nine edge cases, six acceptance criteria, state.json migrate block with selfContained: true. |
intent-reasoning-style/ |
Master skill principle | "Open and reasoned" — vague phrases get clarified, never silently mapped to commands. Pending direction persisted. |
css-four-scope-global/ |
Phase 3 (prototype) |
Discipline 11 GLOBAL scope — marker comments for tokens / resets / default-content / compound utility / motion; unscoped-class refusal path. |
css-four-scope-section/ |
Phase 3 (prototype) |
Discipline 11 SECTION scope — marker names matching data-section, token override + default-content override grouping, cascade, orphan refusal. |
css-four-scope-block/ |
Phase 3 (prototype) |
Discipline 11 BLOCK scope — data-module + data-template fold into BLOCK, internals don't bleed to default-content, dead-block refusal. |
css-four-scope-default-content/ |
Phase 3 (prototype) |
Discipline 11 default-content scope — direct-child combinator load-bearing, canonical element set, isolation from block internals. |
Coverage map
| Layer of the v2 design | Covered by |
|---|---|
| Layer 1 — intent abstraction (open + reasoned) | intent-reasoning-style, direct-from-phrase |
| Layer 2 — navigator orchestrator (4 phases) | extract-multipage, direct-from-phrase, prototype-before-after, migrate-incremental |
| Layer 3 — migration tooling (per-page, incremental, idempotent) | migrate-incremental, migrate-multi-template |
| Layer 3 — migrate output contract (self-contained bundle) | migrate-self-contained-bundle |
| Layer 3 — four-scope CSS extraction contract (Discipline 11) | css-four-scope-global, css-four-scope-section, css-four-scope-block, css-four-scope-default-content |
The cross-cutting properties are pinned across multiple evals:
- Hard impeccable dependency — every eval expects the master setup to verify impeccable before proceeding.
- Direct authoring (no interview duplication via
$impeccable teach/documentfor files stardust seeds) — checked inextract-multipageanddirect-from-phrase. - Provenance everywhere — each eval has a provenance criterion.
- No EDS / framework / CMS leakage — every eval has a
no_eds_referencescriterion. - Stale-on-direction-change — covered in
migrate-incremental(the eval can be extended later with a re-direct mid-run scenario; for v0.2.0 we test the idempotent baseline).
Running
The eval runner is not bundled with this plugin (it lives elsewhere
in adobe/skills). Each eval is self-describing: a runner reads
task.md for the scenario, executes it against a clean stardust
project, and scores the output against criteria.json.
A criterion passes if its description is satisfied as judged by
the runner. Per-criterion verdicts are combined as a weighted sum
out of total (100 per eval).
What stardust v2 evals deliberately do NOT test
- Visual quality of the redesigned output. That's
$impeccable critiqueand$impeccable auditterritory; running them as a side effect of a stardust eval is not the right boundary. A redesign that is technically correct (passes all criteria) can still be ugly; that's a different evaluation. - Specific brand outcomes. The evals pin the procedure and the artifacts, not whether the resulting brand identity is "good for Stripe" or "young enough" — those are user judgements.
- Live URL availability. The
extract-multipageeval uses stripe.com as a stable, public, well-structured target. If stripe.com changes structure dramatically the eval may need updating; that's expected maintenance.