fission-ai__openspec/OPENSPEC_COMPREHENSIVE_RETROSPECTIVE.md

Comprehensive Retrospective: Creating an OpenSpec Change Proposal

Executive Summary

This document consolidates learnings from creating the bulk-validation-interactive-selection change proposal for OpenSpec. The process revealed critical gaps in documentation, unhelpful error messages, and areas where the system could be more user-friendly. While OpenSpec's core functionality works correctly, the user experience for creating changes needs significant improvement.

Table of Contents

1. Errors Encountered
2. System Issues vs User Errors
3. Documentation Gaps Analysis
4. Key Learnings
5. Recommendations
6. Conclusion

---

Errors Encountered

Error 1: Misunderstanding Delta Structure

Error Message: ✗ [ERROR] deltas: Change must have at least one delta

What Happened: Initially attempted to define deltas directly in the proposal.md file using markdown sections like:

### Delta: Add validate-all command
**Type**: Feature addition
**Effort**: Small (< 100 lines)

Root Cause: Fundamental misunderstanding of how OpenSpec processes deltas. Deltas are derived from spec files in the change's specs/ directory, not from the proposal itself.

Discovery Process:

• Examined ChangeParser class in /src/core/parsers/change-parser.ts
• Found that parseDeltaSpecs() method looks for spec files in specs/ subdirectory
• Learned that deltas are extracted by comparing spec files against existing specs

Error 2: Missing Operation Prefix in Section Headers

Error Message: ✗ [ERROR] deltas: Change must have at least one delta

What Happened: Created new spec files with standard ## Requirements headers instead of operation-prefixed headers.

Root Cause: Failed to understand that ALL spec files in a change need operation prefixes (ADDED, MODIFIED, etc.) in their section headers, regardless of whether they're new specs or modifications.

Discovery Process:

• Created new specs with ## Requirements → No deltas detected
• Changed to ## ADDED Requirements → Deltas detected successfully!
• Realized creating new specs works perfectly fine once properly formatted

Important Clarification: OpenSpec fully supports creating new specs. They appear as ADDED operations in the deltas. My initial analysis incorrectly suggested this was a limitation, but it was actually just a formatting issue.

Error 3: Improper Scenario Formatting

Error Message:

✗ [ERROR] deltas.0.requirements.0.scenarios: Requirement must have at least one scenario
✗ [ERROR] deltas.1.requirements.0.scenarios: Requirement must have at least one scenario

What Happened: Formatted scenarios as bullet lists under a bold "Scenarios:" label:

**Scenarios:**
- **WHEN** executing command
- **THEN** expected behavior

Root Cause: OpenSpec's parser expects scenarios to be defined as level 4 headers (####) with specific formatting:

#### Scenario: Descriptive scenario name

- **WHEN** executing command
- **THEN** expected behavior

Discovery Process:

• Checked parsed JSON output: npx openspec change show bulk-validation-interactive-selection --json
• Saw "scenarios": [] empty array despite having scenario content
• Examined working spec files and found the #### Scenario: header pattern

Error 4: File Path Confusion

Initial Confusion: Wasn't clear whether to create specs that would become part of the main openspec/specs/ or just define them in the change.

Resolution: Learned that changes can:

1. Create new specs (they start in changes/{change-name}/specs/ and move to openspec/specs/ when archived)
2. Modify existing specs (by creating a spec file with the same name as one in openspec/specs/)
3. The validation system detects both patterns and creates appropriate deltas

---

System Issues vs User Errors

System Issues / Bugs

1. Unhelpful Error Messages ⚠️

Issue: ✗ [ERROR] deltas: Change must have at least one delta

Why This Is a System Problem:

• Error message provides no guidance on HOW to create deltas
• Doesn't mention that deltas come from specs/ subdirectory
• Doesn't explain the required section headers
• A better error would be: "No deltas found. Ensure your change has a specs/ directory with .md files containing sections like '## ADDED Requirements'"

2. Silent Scenario Parsing Failures ⚠️

Issue: When scenarios were formatted incorrectly, they were silently ignored

Why This Is a System Problem:

• Parser silently returns empty scenarios array instead of warning
• No validation error explaining the format issue
• User gets "Requirement must have at least one scenario" without knowing their scenarios exist but aren't parsed

Evidence:

{
  "text": "The CLI SHALL provide a top-level `show` command with interactive selection.",
  "scenarios": []  // Silent failure - scenarios existed but weren't parsed
}

3. No Validation for Proposal Structure During Creation

Issue: System allows creating invalid proposals without early feedback

Why This Is a System Problem:

• No scaffolding or template commands
• No incremental validation as you build
• Must fully create the change before discovering structural issues

User Errors (My Mistakes)

1. Trying to Define Deltas in Proposal.md

• Incorrectly assumed deltas could be inline in the proposal
• System correctly expects deltas in separate spec files

2. Using Wrong Section Headers

• Used ## Requirements instead of ## ADDED Requirements
• Convention is documented in existing changes, but I didn't examine carefully

3. Wrong Scenario Format

• Used bullet lists instead of #### Scenario: headers
• Made assumptions instead of checking existing patterns

Gray Areas

1. Documentation gaps - While examples exist, there's no comprehensive "How to Create a Change" guide
2. Lack of tooling - No scaffolding commands to create properly structured changes

---

Documentation Gaps Analysis

Critical Gaps in openspec/README.md

1. Scenario Format - COMPLETELY MISSING ⚠️

What README Shows: No scenario examples at all

What's Actually Required:

#### Scenario: Descriptive name
- **WHEN** condition
- **THEN** expected outcome
- **AND** additional outcomes

Impact: This was the biggest struggle. The README mentions requirements but never shows how to write scenarios. Without this, requirements fail validation.

2. Complete Spec File Example - MISSING

What README Shows: Only fragments

What's Actually Needed: A complete working example showing:

• Full spec file structure
• Proper requirement format
• Scenario formatting
• All required elements

3. Validation Commands - NOT MENTIONED

Missing from README:

• npx openspec change validate <change-name>
• npx openspec change show <change-name> --json
• The --strict flag for catching warnings

4. Delta Detection Explanation - INCOMPLETE

What's Missing:

• WHERE the system looks for specs (specs/ subdirectory)
• THAT deltas are automatically extracted
• HOW to debug when deltas aren't detected
• WHAT error messages mean

Misleading Documentation

"Store only the changes" - MISLEADING

Line 145: # - Store only the changes (not complete future state)

Problem: This suggests storing diffs or partial content. In reality, you need:

• Complete requirements in their final form
• Full scenario definitions
• The entire requirement text

Documentation That Was Helpful

1. Delta section headers (## ADDED Requirements) - clearly documented
2. Directory structure - excellent visualization
3. When to create proposals - well defined

---

Key Learnings

1. OpenSpec's Delta Detection Algorithm

The system follows this process:

1. Scans openspec/changes/{change-name}/specs/ directory
2. For each spec file found, parses for delta sections (ADDED, MODIFIED, REMOVED, RENAMED)
3. Creates delta objects with operation type, affected spec, and requirements
4. Validates that at least one delta exists for the change to be valid

Important: Creating entirely new specs is fully supported! New specs use ## ADDED Requirements and appear as ADDED operations in the deltas.

2. Spec File Structure Requirements

Valid spec files must follow this structure:

# Spec Title

## [ADDED|MODIFIED|REMOVED|RENAMED] Requirements

### Requirement: Clear requirement statement

The requirement description using SHALL/SHOULD/MAY.

#### Scenario: Scenario name

- **WHEN** condition
- **THEN** expected outcome
- **AND** additional outcomes

3. Change Proposal Structure

A valid change must have:

• ## Why section - explaining the motivation
• ## What Changes section - summarizing the changes
• specs/ directory with properly formatted spec files containing deltas
• Each delta must have at least one requirement with at least one scenario

4. Validation Commands Are Essential

# Basic validation
npx openspec change validate {change-name}

# Strict validation (recommended)
npx openspec change validate {change-name} --strict

# Debug delta detection
npx openspec change show {change-name} --json | jq '.deltas'

---

Recommendations

High Priority (System Bugs to Fix)

1. Improve Error Messages

   • Add actionable guidance to error messages
   • Example: "No deltas found. Check: 1) specs/ directory exists, 2) Files use ## ADDED Requirements headers, 3) Each requirement has #### Scenario: sections"

2. Add Warnings for Malformed Content

   • Warn when scenarios exist but aren't properly formatted
   • Show which line/file has the issue

3. Add Delta Detection Debugging

   • Command like openspec change debug-deltas {change-name}
   • Show which files were scanned, what was found, what was rejected

Medium Priority (Documentation Improvements)

1. Add Complete Working Example to README

   • Full change proposal with all files
   • Properly formatted specs with scenarios
   • Show the validation output

2. Add Troubleshooting Section

   • Common errors and their solutions
   • How to debug delta detection
   • Scenario formatting requirements

3. Add Validation Best Practices

   • When to use --strict
   • How to use JSON output for debugging
   • Common validation patterns

Low Priority (Developer Experience)

1. Add Scaffolding Command

   openspec change scaffold {change-name}
   

   • Creates proper directory structure
   • Includes template files with correct formatting
   • Adds example scenarios

2. Add Interactive Creation Wizard

   • Guide users through change creation
   • Validate as they go
   • Suggest fixes for common issues

3. Add Auto-fix Capability

   • --fix flag to correct common formatting issues
   • Convert bullet list scenarios to proper headers
   • Add missing operation prefixes

---

Conclusion

The OpenSpec system works correctly for its intended design, but the user experience for creating changes needs significant improvement. The core issues stem from:

System Issues

• Unhelpful error messages that don't guide users to solutions
• Silent parsing failures that provide no feedback about malformed content
• Lack of debugging tools to understand what went wrong

Documentation Issues

• Critical formatting requirements missing (especially scenario format)
• No complete working examples showing all required elements
• Validation commands not documented despite being essential

User Issues

• Incorrect assumptions about how the system works
• Not examining existing patterns carefully enough
• Trying to shortcut instead of following established conventions

The Path Forward

With better error messages, complete documentation, and basic tooling support, most of the errors encountered could be prevented. The system's delta-centric approach is powerful and ensures changes are atomic and trackable, but it needs to be more discoverable and user-friendly.

The most impactful improvements would be:

1. Adding scenario format documentation to the README
2. Improving error messages with actionable guidance
3. Creating a scaffolding command for new changes

These changes would transform OpenSpec from a system that works correctly but is hard to use, into one that actively helps developers succeed.