Commit Graph

303 Commits

Author SHA1 Message Date
Justin van der Merwe 7067f6f7a5 Organize specs as per-feature .feature files with optional --scope
### Problem

Specs were stored as a single file per branch. This coupled specs to branches rather than product features, and made it harder to mentally organize the growing set of behaviours as a product evolves across branches.

### Solution

We split spec storage into multiple files — one per Feature block in the Gherkin output, named by slugifying the feature name (e.g., "CLI spec update" becomes cli-spec-update.feature). Specs now describe product features, not branch-scoped work.

The LLM pipeline is unchanged. It continues to operate on a single concatenated string — file boundaries are invisible to it. We introduce a virtualized I/O boundary: on read, all feature files are globbed, sorted, and concatenated. On write, the output is parsed by Feature block, existing files are removed, and each block is written to its own file. The removal before writing is safe because the content was already consumed and re-expressed by the LLM.

We also add a --scope flag to direct specs into a subdirectory (e.g., --scope derive writes to .machinen/specs/derive/), and remove the init command (no longer needed with multi-file storage).

The architecture blueprint is updated throughout — system flow, behaviour spec, API reference, invariants, and directory mapping.
2026-03-04 23:38:30 +02:00
justinvdm b1324e9b91 Split specs into feature files. 2026-03-04 23:37:02 +02:00
justinvdm 4f4be8ca5b Organize specs as per-feature .feature files with optional --scope. 2026-03-04 23:33:54 +02:00
justinvdm a9b47a2fd6 Implement --scope flag for spec subdirectory. 2026-03-04 22:59:41 +02:00
justinvdm 5f0687d445 Implement CLI context detection, conversation discovery, incremental spec updates, multi-file spec storage, one-shot spec update, reset mode, spec file content and format, spec file location, and watch mode. 2026-03-04 22:51:13 +02:00
justinvdm 6f5f1d8924 Remove spec file path from database schema. 2026-03-04 22:45:30 +02:00
Peter Pistorius 277b5e43a8 test: add workspace copy integration tests (#17)
- Extract copyWorkspace() into cleanup.ts as a testable function
- Wire it into local-job.ts (replaces inline copy logic)
- Add 4 integration tests using real temp git repos:
  - copies tracked files
  - copies untracked-but-not-ignored files
  - excludes gitignored files (node_modules, dist, *.log)
  - preserves nested directory structure
2026-03-04 22:41:38 +02:00
justinvdm d5c5ce4b6c Remove outdated documentation rules. 2026-03-04 22:36:52 +02:00
justinvdm e6c3a139f7 Implement feature file split for multi-file spec storage. 2026-03-04 22:36:39 +02:00
Peter Pistorius 5bb11c6506 fix: use per-file cp -c instead of full directory clone (#17)
The full-directory cp -c -R was copying _/ (15GB) and node_modules
since it doesn't respect .gitignore. Switch to git ls-files | xargs cp -c
for per-file APFS CoW clones that only copy tracked files.
2026-03-04 22:31:43 +02:00
Peter Pistorius f6ea800eb8 feat: add disk usage visibility API (#17)
- GET /disk-usage returns workspace count, sizes, ages, pnpm/playwright/logs sizes
- DELETE /disk-usage/workspaces removes specific or all stale workspaces
- Add API test for disk-usage endpoint

Fixes #17 (Fix 6)
2026-03-04 19:02:59 +02:00
Peter Pistorius c212f2e20b fix: prune stale workspaces at server startup (#17)
- Scan _/work/oa-runner-* directories on startup
- Remove any older than 24 hours
- Log pruned workspace names

Fixes #17 (Fix 5)
2026-03-04 19:01:10 +02:00
Peter Pistorius 465b943416 fix: add signal handlers for workspace cleanup (#17)
- Register SIGINT/SIGTERM handlers to ensure cleanup on kill
- Handlers clean containerWorkDir, shimsDir, diagDir
- Deregister handlers when normal cleanup path runs

Fixes #17 (Fix 4)
2026-03-04 18:59:43 +02:00
Peter Pistorius 42cff42c00 fix: use APFS CoW copies for workspace isolation (#17)
- On macOS, use cp -c -R (APFS clone) for near-instant zero-disk copies
- Clone shares physical blocks until a file is actually modified
- Falls back to rsync/fs.cpSync on non-APFS volumes or Linux

Fixes #17 (Fix 3)
2026-03-04 18:58:25 +02:00
Peter Pistorius 008d9d6d43 fix: eliminate container-side workspace copy (#17)
- Place workspace files directly in containerWorkDir/<repo>/<repo>/
- Remove /tmp/oa-workspace bind-mount and container-side cp -r
- Halves NVMe writes per run by eliminating Copy 2

Fixes #17 (Fix 2)
2026-03-04 18:56:42 +02:00
Peter Pistorius 83cb328915 fix: cleanup containerWorkDir on exit (#17)
- Clean containerWorkDir on success, retain on failure for debugging
- Add cleanup.ts module with pruneStaleWorkspaces and getDiskUsage
- Add cleanup.test.ts with TDD tests for all cleanup behaviors

Fixes #17 (Fix 1)
2026-03-04 18:54:38 +02:00
Peter Pistorius f3f0ce0b11 Merge pull request #16 from redwoodjs/pp-concurrency-limiting
Finally support playwright tests: concurrency, docker in docker emulation.
2026-03-04 16:59:11 +02:00
Peter Pistorius a79d4435f8 Finally support playwright tests: concurrency, docker in docker emulation. 2026-03-04 16:13:50 +02:00
Justin van der Merwe 509900bc86 Add derive - spec extraction from Claude Code conversations
## Problem

Engineers build features on branches using Claude Code. Those conversations implicitly contain the evolving behavioural intent of the feature — but that intent is trapped in JSONL logs, not captured in a structured, testable form.

## Solution

We add `derive`, a CLI tool that reads Claude Code conversation logs and maintains a living Gherkin behaviour spec per git branch. It extracts testable product behaviours from conversations and writes them to `.machinen/specs/<branch>.gherkin` in the project directory, where they travel with the branch via git.

The tool provides four modes — all with explicit token-spend control:

- **`derive`** — one-shot update. Discover conversations, read new messages, update the spec, exit.
- **`derive --reset`** — regenerate the spec from scratch. Reprocess all conversations sequentially, with a single review pass at the end. Supports `--keep-spec` to preserve hand-written content as starting context.
- **`derive watch`** — opt-in continuous mode. Run an initial update, then watch for conversation changes on the current branch (debounced, branch-scoped).
- **`derive init`** — create an empty spec file for manual seeding before any conversations are processed.

Under the hood, `derive` maintains a lightweight SQLite routing index (`~/.machinen/machinen.db`) that maps conversations to repos and branches and tracks read cursors. The spec pipeline is stateless: each update is a fresh `claude -p` call (Sonnet, no tools, low effort) that reads the current spec from disk, combines it with new conversation excerpts, and produces an updated spec. A two-pass architecture — extraction then review — ensures specs contain only externally observable behaviours (the "black-box test") and are deduplicated, consolidated, and simplified.

We have dogfooded `derive` on itself: `.machinen/specs/specs.gherkin` is the tool's own behaviour spec, generated from the conversations that built it. When we add test generation from specs, we will use this generated spec to dogfood further — closing the loop from conversations to specs to tests.

Example snippet of generated spec:

```gherkin
Feature: Conversation discovery
  Scenario: New conversations are discovered on first run
    Given the user is in a git repository on branch "feature-x"
    And Claude Code conversations exist for this repository and branch
    And no previous derive run has occurred
    When the user runs derive
    Then the conversations are discovered and indexed
    And their messages are included in the spec update

  Scenario: Conversations for other branches are ignored
    Given the user is in a git repository on branch "feature-x"
    And Claude Code conversations exist for both "feature-x" and "other-branch"
    When the user runs derive
    Then only conversations for "feature-x" contribute to the spec

  Scenario: Multiple conversations are associated with a single branch
    Given two or more Claude Code conversations exist for the same branch
    When the user runs derive
    Then all conversations for that branch contribute to the spec update
```
2026-03-04 01:54:01 +02:00
justinvdm 6fdff3587a Add derive CLI tool for spec extraction from Claude Code conversations. 2026-03-04 01:51:27 +02:00
justinvdm 6018913208 Refactor conversation discovery and spec file handling. 2026-03-04 01:45:10 +02:00
justinvdm 2b8011da09 Prevent ghost conversations from session persistence. 2026-03-04 01:30:17 +02:00
justinvdm 36d137566a Update Sonnet model in derive pipeline. 2026-03-04 01:08:57 +02:00
justinvdm a683ca4e5f Skip review pass on intermediate specs in reset mode. 2026-03-04 00:59:10 +02:00
justinvdm 309a0b0094 Disable tools and set low effort for spawned agent. 2026-03-04 00:47:46 +02:00
justinvdm 83d1f5a0ca Update spec pipeline to use stream-json activity log. 2026-03-04 00:44:58 +02:00
justinvdm 5e257a3e05 Refactor system prompt handling to avoid OS arg length limits. 2026-03-04 00:34:11 +02:00
justinvdm f1c2aea4b7 Update spec file generation and handling. 2026-03-04 00:02:13 +02:00
justinvdm 874fbcd8f3 Implement --reset --keep-spec flag to preserve existing spec file. 2026-03-04 00:01:45 +02:00
justinvdm 5d571ebfe3 Expand filter pass to deduplicate and simplify specs. 2026-03-03 23:55:59 +02:00
justinvdm 79936c526a Move documentation files to docs directory. 2026-03-03 23:43:15 +02:00
justinvdm b569387251 Move documentation files out of internal directory. 2026-03-03 23:42:45 +02:00
justinvdm c939f3487b Remove internal error handling scenarios from specs. 2026-03-03 23:33:50 +02:00
justinvdm 76e320a4ff Remove internal error handling scenarios and update spec file location. 2026-03-03 23:29:11 +02:00
justinvdm 6368302e6f Implement conversation indexing and spec generation. 2026-03-03 22:57:31 +02:00
justinvdm de123f1c87 Implement conversation-based Gherkin spec updates for Claude Code. 2026-03-03 22:49:11 +02:00
justinvdm 5a6ef5aaca Implement derive init mode for manual seeding. 2026-03-03 22:38:59 +02:00
justinvdm 639df3e247 Implement project-local spec storage and CLI-driven rearchitect. 2026-03-03 22:31:36 +02:00
justinvdm 977788d692 Implement CLI-driven rearchitect with opt-in watch mode. 2026-03-03 22:09:28 +02:00
justinvdm eda7a87507 Rearchitect derive from daemon to manual CLI tool. 2026-03-03 21:28:12 +02:00
justinvdm f23ea69308 Update .../justin/worklogs/2026-03-03-machinen-setup.md | and 10 other file(s) 2026-03-03 17:04:25 +02:00
Peter Pistorius fb116ba67e Merge pull request #11 from redwoodjs/pp-support-matrix-worflows
Add matrix support for matrix workflows.
2026-03-03 14:49:34 +02:00
Peter Pistorius f8d699b388 Add matrix support for matrix workflows. 2026-03-03 14:48:13 +02:00
Peter Pistorius 3e18e270ce Merge pull request #9 from redwoodjs/pp-dx-get-errors
Improve DX and artifact handling
2026-03-03 14:04:02 +02:00
Peter Pistorius a704e35244 Fix test again. 2026-03-03 13:45:05 +02:00
Peter Pistorius 32457ad2cd Fix tests. 2026-03-03 13:26:15 +02:00
Peter Pistorius b253f3a8af Add basic support for secrets. 2026-03-03 12:59:42 +02:00
Peter Pistorius bd530313f0 Add dev container. 2026-03-03 12:20:56 +02:00
Peter Pistorius fc28e71311 Add secrets. 2026-03-03 12:11:44 +02:00
Peter Pistorius 73d02497f0 Fix artifact uploading. 2026-03-03 11:29:44 +02:00