- Rename test_pop_cancellation_onEmptyPath_doesNotCrash to test_pop_onEmptyPath_doesNotCrash
to accurately reflect empty-path guard behaviour (no actual cancellation involved)
- Rename test_addChild_retainsChildCoordinatorUntilRemoved to
test_cancelChildFlow_removesChildCoordinator, which genuinely exercises
sub-flow cancellation and satisfies the contract cancellation-pattern requirement
- Fix StubUserRepository to return a fixed UUID literal instead of UUID() per call,
making the stub deterministic as stated in the section description
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add MockCoordinatorDelegate stub, delegate wiring, and test names covering
success, failure, and cancellation paths so the case passes all required_any_groups
and required_patterns checks in the testing-quality contract.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- mvvm.md: add State(wrappedValue:) and StateObject(wrappedValue:)
initializer patterns to @Observable and ObservableObject view
examples so ViewModels can be injected from the assembly layer
- viper.md: remove duplicate ProfilePresenter body from Concurrency
section; replace with a reference to the Wiring Pattern section
- reactive.md: make SearchPresenter generic over S: Scheduler so the
UIKit example is testable, consistent with SearchViewModel
- tca.md: remove redundant $0.continuousClock = clock from the
cancellation test (numberFact.fetch captures clock directly)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Move the evals/ and scripts/ trees into tooling/ (tooling/evals, tooling/scripts) and update all references accordingly. Adjusted CI workflow, docs (AGENTS.md, CONTRIBUTING.md, evals README), manifests and runner scripts to use the new paths, updated .gitignore, and fixed script path resolution (parents index and ROOT_DIR adjustments) so tooling runs and validations continue to work from their new location.
Refactor and enhance testing examples and validation tooling:
- CONTRIBUTING.md: fix checklist indentation for readability.
- quality/benchmarks/cases/*: replace placeholder asserts with real async XCTest checks (mvvm stale-overwrite and async cases), add helper functions and TestError, and ensure stale-request ordering is tested correctly.
- scripts/run-testing-benchmark-suite.py: add a 30s COMMAND_TIMEOUT, handle subprocess.TimeoutExpired, and treat unsupported assertion scopes as failures instead of raising.
- scripts/validate-benchmark-coverage.py: validate that architecture assertions are objects and skip malformed entries.
- scripts/validate-testing-quality.py: validate requirement scopes ('section' or 'file') and catch invalid regex errors with clear messages.
- swift-architecture-skill/references/mvi.md: correct unit test expected counter value from 3 to 0.
These changes improve test quality, make validation tools more robust, and provide clearer error reporting.
Make benchmark & corpus tests deterministic and add stricter validation across tooling and CI. Key changes:
- CI: install Python 3.11 in validate-skill workflow.
- Tests: replace nondeterministic sleeps with Task.yield/explicit scheduling, use subjects and collectors, add small TestError enums and helpers, adapt TCA tests to use deterministic stubs and assertions.
- Manifests/corpus: tighten regexes, add semantic description for an intentionally-broken case, and pin upstream SOURCE URLs to specific commits.
- Scripts: use sys.executable for subprocess calls, validate assertion scopes, compile/validate heading regex, better JSON error messages, surface unknown README entries, and improve behavior for architecture assertion format checks.
- Shell validation: detect unclosed Swift fences by parsing filtered blocks and reporting swiftc parse errors.
- Docs: minor CONTRIBUTING guidance and reference updates for clean-architecture and MVI examples.
Overall this commit improves test reliability and hardens validation scripts used by CI and local tooling.
Add many benchmark and real-world corpus cases, update manifest assertions, and introduce validation tooling and CI steps. New files add architecture-focused benchmark cases (clean, mvi, reactive, viper, tca/mvvm adjustments) under quality/benchmarks/cases and several real-world corpus examples under quality/corpus/cases. Added scripts/validate-benchmark-coverage.py to ensure each architecture has positive/negative cases and scripts/run-realworld-corpus-eval.py to invoke the existing runner with a corpus manifest; run-testing-benchmark-suite.py was enhanced to extract Swift code blocks and make architecture assertion evaluation more robust (scoped matching and regex error handling). Updated quality/benchmarks/manifest.json to include new cases, improved regexes (TCA cancellable pattern and MVVM stale-assert), and set architecture fields; docs (AGENTS.md, CONTRIBUTING.md, SOURCES.md) and CI (.github/workflows/validate-skill.yml) were updated to surface and run the new validations.
Add architecture-focused benchmark fixtures and update tooling to evaluate architecture assertions.
- Add new benchmark cases for MVVM and TCA: mvvm-stale-overwrite-good, mvvm-missing-stale-overwrite-should-fail-architecture, tca-cancel-id-good, and tca-missing-cancel-id-should-fail-architecture.
- Update quality/benchmarks/manifest.json to register the new cases and include architecture_assertions (regex-based checks, with dotall/ignore-case flags where needed).
- Update README.md and SOURCES.md to mention architecture-specific fixtures.
- Enhance scripts/run-testing-benchmark-suite.py: import re, add evaluate_architecture_assertions to run regex checks against case content, extend evaluate_case return values, and adjust main() to consider expect_architecture when computing pass/fail and printing architecture failure output.
These changes enable the benchmark suite to validate not only syntax/semantic expectations but also pattern-level architecture guarantees (e.g., TCA cancel IDs and MVVM stale-overwrite protections).
Add a new architecture consistency validator script and CI job (scripts/validate-architecture-consistency.py + workflow entry) to verify references in swift-architecture-skill. Update docs (AGENTS.md, CONTRIBUTING.md, README.md) to surface the new validation step. Tighten the testing-quality contract to require at least one cancellation/stale-response focused test. Update reference materials: add cancellation-focused async test and blocking stub in clean-architecture, rename a reactive test for clarity, and refactor viper stubs/tests to use an async closure-based interactor plus an extra cancellation behavior test and guidance about using a cancellation-aware presenter.
Introduce a testing-quality benchmark suite and validators: add quality/benchmarks (cases, manifest, SOURCES) and a testing-quality-contract.json; add scripts/run-testing-benchmark-suite.py and scripts/validate-testing-quality.py to run cases and validate playbooks; update CI (.github/workflows/validate-skill.yml) with validate-testing-quality and validate-testing-benchmarks jobs; update docs (README.md, AGENTS.md, CONTRIBUTING.md) to document the new checks; tweak scripts/validate-testing-snippets.sh to accept an optional playbook dir and verify it exists; and update several playbook references to emphasize cancellation handling and deterministic tests. These changes enable automated evaluation of snippet syntax and guidance quality against curated benchmark cases.
Lasha Efremidze