skills/efremidze__swift-architecture-skill

mirror of https://github.com/efremidze/swift-architecture-skill.git synced 2026-09-14 20:49:06 +08:00

Files

T

Lasha Efremidze 3a998f6407 Relocate evals and scripts into tooling/

Move the evals/ and scripts/ trees into tooling/ (tooling/evals, tooling/scripts) and update all references accordingly. Adjusted CI workflow, docs (AGENTS.md, CONTRIBUTING.md, evals README), manifests and runner scripts to use the new paths, updated .gitignore, and fixed script path resolution (parents index and ROOT_DIR adjustments) so tooling runs and validations continue to work from their new location.

2026-02-26 00:46:38 -08:00

1.5 KiB

Raw Permalink Blame History

Skill Evaluation Harness

This folder provides scenario-based evaluations for swift-architecture-skill.

Goal

Check whether the skill catches architecture/code smells and proposes concrete, pattern-consistent fixes.

Layout

tooling/evals/cases.json: all eval cases in one file
tooling/evals/rubric.md: manual scoring rubric
tooling/scripts/run/evals.sh: runner

Case Format (`cases.json`)

Each case includes:

id
notes
prompt
required_keywords (legacy all-of keywords)
required_any (optional grouped any-of checks; preferred)
forbidden_keywords
forbidden_any (optional grouped any-of checks)

Run a Batch

Prepare a run package (manual mode):

./tooling/scripts/run/evals.sh

This creates tooling/evals/runs/<timestamp>/ with:

packet.md (all prompts)
responses/<id>.md
scorecards/<id>.md
cases.json snapshot

Optional: run with a command that reads stdin and writes stdout:

./tooling/scripts/run/evals.sh --cmd "your-command-here"

The command is executed per case as:

bash -lc "$CMD" < prompt_text > responses/<id>.md

Grade a completed run:

./tooling/scripts/run/evals.sh --grade-only --run-dir tooling/evals/runs/<timestamp>

Scoring Guidance

Keyword scoring is a quick heuristic. Use manual scorecards as source of truth.

Manual scorecard dimensions:

Smell detection
Architecture fit
Fix quality
Safety (error/cancellation guidance)
Clarity

1.5 KiB Raw Permalink Blame History