Commit Graph

1 Commits

Author SHA1 Message Date
Bo 39f0850640 ci(validation): test-count non-regression gate for cli/internal (ag-h2z #test-count-noregress) (#562)
## Summary

Adds a blocking T1 CI gate that ratchets test coverage in
`cli/internal/` packages: once tests are added, a PR cannot delete them
without a deliberate, recorded justification. Motivated by PR #558–#561,
which added 248 tests across 6 internal packages with nothing preventing
a future PR from silently removing them.

- `scripts/check-test-count-regression.sh` — for each
`cli/internal/<pkg>` with a changed `_test.go` file between the PR base
and HEAD, sums `^func Test` declarations per package at both refs. A net
per-package decrease fails the gate **unless** a `Test-Removal-Reason:`
trailer is present in a commit body in the range. Counts are summed per
package (not per file), so moving a test between files in the same
package is net-zero. Emergency bypass:
`AGENTOPS_TEST_COUNT_NOREGRESS=skip` (logged). Fail-open when the base
ref is unresolvable (shallow clone), fail-closed only on a verifiable
decrease.
- `tests/scripts/check-test-count-regression.bats` — 8 scenarios (add /
remove-no-trailer / remove-with-trailer / rename / skip-bypass / no-op /
unresolvable-base / cross-file move) driven by throwaway git repos.
Auto-discovered by the `bats-tests` job.
- Wired as blocking `validate-test-count-noregress` (T1): job block +
`summary.needs` + `validate-pr-evidence-claims.needs` in
`.github/workflows/validate.yml`, manifest entry in
`docs/contracts/ci-jobs.yaml`, and regenerated `AGENTS-CI.md` (parity
gate green, 66 rows). `fetch-depth: 0` so the base ref resolves for the
before/after diff.

Green by construction on this PR: it changes no
`cli/internal/**/_test.go`, so the gate is a no-op PASS.

## Acceptance scenarios (from ag-h2z)

- PR adds tests to `cli/internal/X` → gate passes (count up or stable).
✅ bats #1
- PR removes a test without a trailer → gate fails, naming the test +
pkg. ✅ bats #2
- PR removes a test **with** `Test-Removal-Reason:` trailer → gate
passes. ✅ bats #3
- PR renames a test (`Test_X` → `TestX`) → net count stable → gate
passes. ✅ bats #4

## Test plan

- `bats tests/scripts/check-test-count-regression.bats` — 8/8 green
- `shellcheck scripts/check-test-count-regression.sh` — clean
- `bash scripts/validate-ci-policy-parity.sh` — PASS (66 rows)

## Follow-ups (out of scope, immutable)

- Promoting `validate-test-count-noregress` to a GitHub
branch-protection *required status* is repo-admin (immutable scope) —
not done here. The job is blocking via `summary.needs`.

Closes-scenario: ag-h2z#test-count-noregress
Bounded-context: BC2-Validation
Evidence: PASS: no cli/internal/*_test.go changes
2026-05-27 23:22:53 -04:00