fission-ai__openspec/openspec/changes/add-qa-smoke-harness/proposal.md at main

mirror of https://github.com/Fission-AI/OpenSpec.git synced 2026-09-14 20:16:53 +08:00

Files

T

Tabish Bidiwale 4ba26902df feat: simplify skill installation with profiles and smart defaults (#726 )

* feat: implement simplified skill installation with profiles and smart defaults

Introduces a profile system (core/custom) to reduce the default workflow
count from 10 to 4, auto-detects AI tools during init, adds a new
`propose` workflow combining new+ff, fixes multi-select keybindings,
and adds backwards-compatible migration for existing users.

* feat: harden update config drift, command-only detection, and init profile validation

Address post-implementation review findings: update now detects profile/delivery
drift even when template versions are current, recognizes command-only installs
as configured tools, init validates --profile values and applies delivery cleanup
on re-init. Specs and design docs updated with new scenarios and rationale.

* fix: address AI reviewer feedback on config, detection, and test cleanup

- Add error handling for execSync in config profile apply and use static import
- Fix config list showing "(explicit)" for core profile workflows misleadingly
- Add missing 'openspec-onboard' to SKILL_NAMES for parity with COMMAND_IDS
- Remove unused fsSync import in init tests
- Fix configTempDir leak in test afterEach cleanup

* docs: add qa smoke harness change proposal

2026-02-19 18:21:06 -08:00

2.3 KiB

Raw Permalink Blame History

Why

We need a faster, more reliable way to manually validate CLI behavior changes like profile/delivery sync, migration behavior, and tool-detection UX.

Today, manual review is mostly ad hoc: each developer sets up state differently, runs a different command order, and checks outputs informally. This makes regressions easy to miss and slows iteration on CLI UX work.

An 80/20 solution is to add a lightweight smoke harness for deterministic non-interactive flows, plus a short manual checklist for interactive prompt behavior.

What Changes

Add a lightweight QA smoke harness for OpenSpec CLI behavior with isolated per-run sandbox state
Use Makefile targets as the primary entrypoint:
- make qa (default local QA entrypoint)
- make qa-smoke (deterministic non-interactive suite)
- make qa-interactive (prints/opens manual interactive checklist)
Implement smoke logic in a script (invoked by Make targets), not in Make itself
Ensure each scenario runs in an isolated sandbox with temporary HOME, XDG_CONFIG_HOME, XDG_DATA_HOME, and CODEX_HOME
Capture scenario artifacts for inspection (command output, exit code, and before/after filesystem state)
Add a focused scenario set for high-risk behavior:
- init core output generation
- non-interactive detected-tool behavior
- migration when profile is unset
- delivery cleanup (both -> skills, both -> commands)
- commands-only update detection
- new tool directory detection messaging
- invalid profile override validation
Add a short interactive checklist for keypress/prompt UX verification (Space toggle, Enter confirm, detected pre-selection)
Wire CI to run the smoke suite on Linux as a fast regression gate

Capabilities

New Capabilities

qa-smoke-harness: Deterministic, sandboxed CLI smoke validation with a single developer entrypoint

Modified Capabilities

developer-qa-workflow: Standardized local/CI QA flow for CLI behavior and migration-sensitive scenarios

Impact

Makefile - Add qa, qa-smoke, and qa-interactive targets
scripts/qa-smoke.sh (or equivalent) - Implement sandbox setup, scenario execution, and assertions
docs/ - Add/update contributor-facing QA instructions and interactive checklist usage
CI workflow - Add smoke target execution as a lightweight regression gate

2.3 KiB Raw Permalink Blame History

Why