Rename the tag step to calculate the next version and run the tag action in dry-run mode. Add steps to update .claude-plugin/plugin.json and .claude-plugin/marketplace.json with the computed version, commit and push that change, then create and push the release tag. This ensures plugin metadata is bumped and committed before the release tag is created.
Move the evals/ and scripts/ trees into tooling/ (tooling/evals, tooling/scripts) and update all references accordingly. Adjusted CI workflow, docs (AGENTS.md, CONTRIBUTING.md, evals README), manifests and runner scripts to use the new paths, updated .gitignore, and fixed script path resolution (parents index and ROOT_DIR adjustments) so tooling runs and validations continue to work from their new location.
Make benchmark & corpus tests deterministic and add stricter validation across tooling and CI. Key changes:
- CI: install Python 3.11 in validate-skill workflow.
- Tests: replace nondeterministic sleeps with Task.yield/explicit scheduling, use subjects and collectors, add small TestError enums and helpers, adapt TCA tests to use deterministic stubs and assertions.
- Manifests/corpus: tighten regexes, add semantic description for an intentionally-broken case, and pin upstream SOURCE URLs to specific commits.
- Scripts: use sys.executable for subprocess calls, validate assertion scopes, compile/validate heading regex, better JSON error messages, surface unknown README entries, and improve behavior for architecture assertion format checks.
- Shell validation: detect unclosed Swift fences by parsing filtered blocks and reporting swiftc parse errors.
- Docs: minor CONTRIBUTING guidance and reference updates for clean-architecture and MVI examples.
Overall this commit improves test reliability and hardens validation scripts used by CI and local tooling.
Add many benchmark and real-world corpus cases, update manifest assertions, and introduce validation tooling and CI steps. New files add architecture-focused benchmark cases (clean, mvi, reactive, viper, tca/mvvm adjustments) under quality/benchmarks/cases and several real-world corpus examples under quality/corpus/cases. Added scripts/validate-benchmark-coverage.py to ensure each architecture has positive/negative cases and scripts/run-realworld-corpus-eval.py to invoke the existing runner with a corpus manifest; run-testing-benchmark-suite.py was enhanced to extract Swift code blocks and make architecture assertion evaluation more robust (scoped matching and regex error handling). Updated quality/benchmarks/manifest.json to include new cases, improved regexes (TCA cancellable pattern and MVVM stale-assert), and set architecture fields; docs (AGENTS.md, CONTRIBUTING.md, SOURCES.md) and CI (.github/workflows/validate-skill.yml) were updated to surface and run the new validations.
Add a new architecture consistency validator script and CI job (scripts/validate-architecture-consistency.py + workflow entry) to verify references in swift-architecture-skill. Update docs (AGENTS.md, CONTRIBUTING.md, README.md) to surface the new validation step. Tighten the testing-quality contract to require at least one cancellation/stale-response focused test. Update reference materials: add cancellation-focused async test and blocking stub in clean-architecture, rename a reactive test for clarity, and refactor viper stubs/tests to use an async closure-based interactor plus an extra cancellation behavior test and guidance about using a cancellation-aware presenter.
Introduce a testing-quality benchmark suite and validators: add quality/benchmarks (cases, manifest, SOURCES) and a testing-quality-contract.json; add scripts/run-testing-benchmark-suite.py and scripts/validate-testing-quality.py to run cases and validate playbooks; update CI (.github/workflows/validate-skill.yml) with validate-testing-quality and validate-testing-benchmarks jobs; update docs (README.md, AGENTS.md, CONTRIBUTING.md) to document the new checks; tweak scripts/validate-testing-snippets.sh to accept an optional playbook dir and verify it exists; and update several playbook references to emphasize cancellation handling and deterministic tests. These changes enable automated evaluation of snippet syntax and guidance quality against curated benchmark cases.
Lasha Efremidze