nexu-io__open-design/docs/testing/syntax-acceptance.md

Deliverable syntax: local acceptance and outcome metrics

This runbook validates the branch's actual daemon before a PR or remote evaluation. A unit-test pass is not a deployment receipt. Local investigation history remains private; this file contains the portable contract and reproduction steps.

Confirmed behavior

Alignment status: READY — non-blocking delivery (2026-09-08)

The current user decision supersedes the earlier syntax-blocking requirement: syntax validation must not prevent delivery. Unrepairable output is still made available to the user, with durable, content-free evidence of the warning.

ID	Confirmed behavior and source	Acceptance / traceability
N1	User: syntax checking must not block final delivery	Host finalizer returns warn, not a Run failure, on repair refusal or incomplete checks; original artifact remains available
N2	User: preserve evidence for unsuccessful repairs	Run/SSE and Langfuse retain the real checker status and finalization reason; deliveredWithSyntaxWarningCount=1 only for a succeeded physical Run with complete warning evidence
N3	User: continue programmatic repair	Existing safe-edit, staging, verification, commit and time budgets remain; no model repair turn
N4	Existing execution failures remain failures	Agent/protocol, missing artifact, cancellation and snapshot failures are not converted to success; a syntax warning is never counted as repaired or blocked delivery
N5	User-approved review follow-up: internal engine defects must remain visible without blocking delivery	The finalizer throws a dedicated internal error; only the delivery boundary recovers it, retaining internal_error evidence and ERROR-level observation; expected filesystem errors remain ordinary incomplete warnings

The earlier implementation baseline 259ee527d used the blocking policy; its remote evaluation cannot validate N1–N4. The non-blocking increment is now based directly on main (d07102a3b). PR submission does not imply merge or release. No critical decision remains unresolved in this scope.

ID	Contract	Acceptance
Q1	Preserve the historical six-error file locally; publish a privacy-safe derivative	Source hashes and six changed lines in fixture provenance
Q2	Only deterministic, unambiguous syntax patches	Static quote mismatch and static HTML attribute-quote rules; dynamic expressions refuse
Q3	Bounded, program-only staging	At most 8 patches, 32 edited characters, 1 second cooperative repair budget; no model turn
Q4	Verify before commit	Whole-candidate checker after each patch; original unchanged on ambiguity, limit or commit conflict
O1	Content-free summary on the existing physical Run terminal	No new telemetry request, source/diff upload, prompt or Repair Agent in this increment
O2	Staged is not committed	Initial repairable + proven committed patch + final pass + allow + succeeded required for recovery
O3	Warning delivery is not repaired or blocked delivery	Original bytes on budget/commit-conflict/incomplete warnings; true execution failures remain failures; partial summaries remain unknown
O4	Versioned, allowlisted metadata	Fixed rule/reason enums and counts; missing evidence is unknown
O5	Agent and Host budgets do not share candidate state	Exhausted Agent state cannot consume Host program-patch budget
O6	Real local deployment witness	Build source, start isolated tools-dev daemon, invoke HTTP Run, read terminal and actual artifact

Architecture and impact

After a physical Agent Run exits successfully, the daemon resolves its canonical touched HTML deliverable and settled process tree. It checks syntax, stages safe patches in memory, fully rechecks and commits only a passing candidate through a guarded atomic replacement. Unsupported, ambiguous or incomplete work refuses the patch, returns warn, and continues delivery of the original artifact. This is fail-closed patching, not a delivery gate. A passing staged candidate that cannot be committed also warns; it does not prove that the delivered original parses.

The shared daemon path covers eligible UI/CLI and ordinary/OD Next Runs, not only ODEval or production-phase Runs. No valid HTML entry skips. When touchedPaths are provided, the canonical entry must have been touched; otherwise existing artifact evidence determines applicability. Inspection is limited to inline scripts and supplied related code paths; it does not recursively fetch every script URL or validate CSS, visuals, runtime or business behavior. Parsing success does not prove author intent.

Existing checker limits are 100 files, 2 MiB/file and 8 MiB total. Incomplete checks, including oversized HTML, retain incomplete evidence and warn without blocking delivery. The one-second repair budget is cooperative, not preemption of synchronous parsing or fsync. Normal checks do not acquire the repair-only deadline. Agent or protocol failures, missing required artifacts, cancellation and HTML snapshot failures are outside this syntax-only policy and keep their existing failure paths.

The Host finalizer defaults on. OD_DELIVERABLE_SYNTAX_FINALIZER=off skips its checks, safe repair and warning evidence. It does not restore the older Agent prompt/tool loop removed by the parent PR change. This increment adds no Prompt/OD Next mapping change. Acorn is a daemon-local runtime dependency; the strict checker stays authoritative.

Local deployment acceptance

Alignment: explicit synthetic telemetry canary (2026-09-09)

The user authorized executing the proposed local upload acceptance. This does not authorize a push, release, real-model batch or upload of existing user data.

ID	Confirmed behavior	Acceptance
S1	Upload is explicitly opt-in; existing replay and real lanes remain offline	Default consent stays off; upload refuses real/dataset/external-fixture inputs
S2	Only three built-in synthetic outputs: clean, safely repairable, unrepairable	One repeat, isolated data and AMR home, no inherited user auth; real daemon performs finalization
S3	Exercise the real exporter through the configured official test relay	Both existing consent gates explicitly on, no direct credentials or manual metrics ingestion; await durable delivery outcome
S4	Distinguish synthetic records and verify actual ingestion	Unique synthetic-test environment, existing low-cardinality fixture model tag, exact Trace/Observation readback; relay acceptance alone is insufficient

The native Langfuse environment reuses the existing telemetry environment source. No new production privacy policy or sink-priority rule is introduced.

Explicit, fixed-synthetic upload canary (never accepts an external dataset, fixture manifest, real mode, production profile or more than one repeat):

OPEN_DESIGN_TELEMETRY_RELAY_URL=https://telemetry-test.open-design.ai/api/langfuse \
  pnpm exec tsx scripts/syntax-acceptance.ts --mode replay --upload-telemetry --repeat 1

The successful Run is followed by the normal client final-message GET/PUT with telemetryFinalized: true. Only the actual durable assistant message is reused; the harness does not manufacture usage, diagnostics or outcome metrics. It then waits for the isolated Run's durable exporter checkpoint. failed, not_expected, zero attempts and a missing receipt do not pass. Automatic repository dotenv loading is disabled for all tools-dev commands. Installation and legacy migration overrides are cleared; the upload lane also isolates AMR home and clears inherited account/direct-ingestion keys. PostHog remains disabled.

Run from e2e/:

pnpm exec tsx scripts/syntax-acceptance.ts --mode replay \
  --fixture-manifest ../apps/daemon/tests/artifacts/fixtures/syntax-quotes/replay.manifest.json \
  --repeat 3

The harness builds the current worktree and dependencies, starts tools-dev in an isolated data root, and captures source/build identity and cleanup. The fake CLI only supplies a fixed file; the real built daemon performs checks, patches and commit. This is not real-model generation. Each fixture repeats three times:

• Six quote errors: six committed patches and exact reference bytes.
• Valid control: zero patches and unchanged bytes.
• Ambiguous expression: succeeded physical Run with repairable / warn, zero committed patches and exact original bytes still readable as the delivered artifact.

The new-policy expectation is nine succeeded Runs: three repaired deliveries, three clean controls and three warning deliveries. This is an acceptance target, not a result until the current source is built and replayed. Historical runs that intentionally blocked delivery do not validate this policy. The harness rejects old action: fail syntax fixture manifests so the old oracle cannot silently pass. The committed fixture is about 120 KB after removing original images/contact data. Its six error locations are preserved. The 1.3 MB original remains local and ignored; do not infer its timing from the smaller derivative. See fixture README/provenance.

The opt-in --mode real lane requires dataset path/hash/row count, pinned runner path/version, Vela binary and test profile. It runs AMR / deepseek-v4-flash / explicit OD Next against a few unchanged pressure cases, without creating a remote batch. It validates integration, not deterministic repair coverage. It requires a runner that projects the complete Host terminal summary and, for warning delivery, deliveredWithSyntaxWarningCount=1, recoveredDeliveryCount=0 and blockedBrokenDeliveryCount=0. Missing summaries or terminal evidence do not pass acceptance, even if the last checker status is pass. This lane's projected metrics alone do not prove original-byte preservation; fixed replay reads and compares the actual artifact to supply that evidence. No production-wallet fallback is permitted. Both lanes disable telemetry/content/manifest upload by default; only the explicit fixed-synthetic canary enables Trace upload. Neither offline replay nor the anonymous Trace canary proves R2 object materialization.

Synthetic upload receipt: 2026-09-09

Receipt od-syntax-acceptance-c7ZU1U rebuilt the current local worktree and ran three fixtures through the real daemon, terminal-message route and test relay. All three local oracles passed, all three exporter receipts were accepted on the first attempt, source identity stayed stable and runtime cleanup was confirmed. The fixture process is synthetic: its token counts are test values, not model billing or generation performance evidence.

Fixture	Remote terminal evidence	Checker total	Discovery to terminal delivery
Clean	pass / allow; committed=0, recovered=0, warning=0	4.73 ms	Not applicable
Missing closing array delimiter	pass / allow; committed=1, recovered=1, warning=0	14.08 ms	91 ms
Missing expression	repairable / warn / no_safe_fix; committed=0, recovered=0, warning=1	2.62 ms	13 ms

The repaired file exactly matched the frozen expected bytes; the clean and warning files stayed byte-identical to their originals. All blocked-delivery counts were zero. Transport waiting was 2.46–3.85 seconds, after Run delivery; it is not checker or repair cost. These tiny synthetic timings are not a production latency promise.

Exact authenticated browser readback found all three Traces in open-design-test, not the production open-design project. Each showed the fixture model tag, native environment synthetic-test-274a1e44, the actual execution tree and the same syntax timing/outcome fields as its local receipt. Filtering Traces to this unique native environment returned exactly three distinct Trace IDs, each with 16 observations; no duplicate Trace was present. The test relay stamps metadata.env=test while preserving the native synthetic environment. The existing production-scoped read-script credential returns 404 for these test-project IDs; that result must not be mistaken for missing ingestion. No credential was copied.

The earlier interrupted preflight od-syntax-acceptance-udPuj0 is not an acceptance receipt: it stopped before terminal-message telemetry finalization; its isolated runtime was confirmed stopped. The subsequent completed receipt above is the evidence of actual upload. Default-off, fixture/relay isolation, failed checkpoint and durable-message contract tests passed 17/17; daemon telemetry/bridge/terminal tests passed 237/237 and both daemon source/test typechecks passed. These test counts are code tests, distinct from the three deployed-daemon canary executions.

No source was committed, pushed, merged or remotely deployed by that canary. Its receipt predates PR submission. An additional default-off-lane identity-isolation assertion was added after this receipt; it leaves the canary's effective environment unchanged.

Main-based PR validation: 2026-09-09

Commit 5164836e47c8530254459e392051732b38a4c602, based directly on main d07102a3b, passed a fresh build and isolated deployed-daemon replay:

pnpm exec tsx e2e/scripts/syntax-acceptance.ts --mode replay --repeat 1

Private receipt od-syntax-acceptance-mK8tps/report.json records 10/10 PASS: six verified repairs, one clean delivery and three warning deliveries. All Runs succeeded; refused patches retained the original bytes. sourceStable=true and cleanup.stopped=true were verified. Upload was disabled and the CLI emitted fixed fixtures, so this is local deployment acceptance, not real-model or online coverage evidence. Repaired-delivery windows were 21–38 ms for these tiny fixtures; the two observed-error warning windows were 11/14 ms. The oversized incomplete check has no first-error window, not a zero-duration repair.

The same source passed 380 daemon tests across eight focused files, 16 contract tests and 17 harness tests (413 total), repository guard and full workspace typecheck. Initial restricted-sandbox attempts could not create IPC/listening sockets; the affected commands passed with the required local process permission. The subsequent documentation-only commit does not change the validated runtime.

Local receipts: 2026-09-08 non-blocking policy

The new harness assertions first failed against the old oracle (5 failures / 8 passes), then passed 13/13. Additional red checks rejected a bare staged pass without finalization evidence and incorrect delivered bytes. These are harness unit tests, not model or deployment acceptance. pnpm typecheck in e2e/ and git diff --check also passed.

Two subsequent invocations rebuilt the worktree, started the real local tools-dev daemon and submitted HTTP Runs using fixed-output fake CLI processes:

Replay receipt directory	Cases / result	Observed delivery outcomes
od-syntax-acceptance-kOL1U5	Quote manifest, 3 repeats: 9/9 PASS	3 repaired, 3 clean, 3 warning; all 9 physical Runs succeeded
od-syntax-acceptance-sxbOnZ	Built-in boundary fixtures, 1 repeat: 10/10 PASS	6 repaired, 1 clean, 3 warning; all 10 physical Runs succeeded

Each private local directory contains report.json, per-Run JSON, actual before / after files and runtime/build logs. Both receipts record sourceStable=true and confirmed cleanup.stopped=true; no background acceptance daemon remains. They use base commit 259ee527dacccbb83d29fe0b91bcf9c6c3348a6e plus local changes, not a published version. The scoped diff hashes differ between receipts (65ce8a78235c... / 5f6133be5d99...), so these are not identical full-source snapshots. The built server, finalizer, safe-fixer and quote-helper hashes match exactly across both receipts; their complete source/build identities are retained in the reports.

The quote fixture's six patches committed in each repeat and delivered bytes exactly matched the reference. Every warning fixture preserved its original bytes:

• Ambiguous expression, 3 repeats: repairable / warn, zero committed patches.
• Repair limit: repairable / warn / attempt_limit_reached, eight staged but zero committed patches; no partial staged changes leaked into delivery.
• Expression hole: repairable / warn / no_safe_fix, zero committed patches.
• Oversized file: incomplete / warn / check_incomplete, zero committed patches; incomplete is not counted as a detected syntax error.

The three six-error repairs took 257 / 127 / 112 ms from first repairable check to physical success. Ambiguous-expression warning delivery took 6 / 6 / 8 ms; the repair-limit and expression-hole warning windows were 14 / 5 ms. Oversize has no first-repairable timestamp, so that window is unavailable, not zero. These are fixed-file local terminal windows (the quote derivative is about 120 KB), not model generation latency, 1.3 MB original-file timing, production p95 or online repair coverage. No real-model batch, remote deployment or telemetry ingestion test ran.

The nine saved physical Run JSON records from the quote receipt were also passed through the current Open Design bridge, strict safe telemetry schema and flat metadata helper, and separately through the local ODEval projector and Case rollup. Both paths agreed on all nine records: three recovered deliveries, three clean deliveries, three warning deliveries and zero blocked deliveries. Each warning retained no_safe_fix / unsupported_syntax_error, with zero committed patches and zero recovered-delivery count. This is a local serialization and projection witness, not evidence of receipt by the online Langfuse service.

Focused checks additionally passed: daemon finalizer / HTTP / artifact regressions 67/67, telemetry regressions 311/311, contracts / normalized observations / OD Next capability 48/48, and local ODEval projection / rollup / HTTP regressions 117/117. The strengthened original-byte HTTP assertions passed again in a 4/4 rerun. Daemon source/tests type checks and repository guard passed. Cross-Run regression tests ensure a latest warning cannot inherit an earlier recovery, an earlier warning cannot override a latest clean result, and missing latest warning evidence stays unknown rather than becoming an observed zero.

Terminal summary and timing

Review follow-up (2026-09-09): internal finalizer defects now reject from the engine as DeliverableSyntaxInternalError. Only the delivery owner handles this dedicated error, retaining internal_error evidence and a sanitized local ERROR log. The native Run trace exports an ERROR-level deliverable-syntax-internal-error event; ordinary incomplete checks do not emit it. The original cause stays in memory and is not logged or exported. Existing Agent/protocol/snapshot failures are unchanged. Consumers must accept the additive internal_error finalization reason to retain its classification; this PR does not deploy ODEval or prove online ingestion.

The illegal-decision and programming-error tests first failed against ce880a19b (two incorrect warning resolutions), then passed. Updated engine, delivery, HTTP, Langfuse, Task and contract checks passed 373 tests; full typecheck and guard passed. The HTTP fault-injection test proves durable internal-error evidence, successful Run completion and byte-identical original artifact readback. Fresh rebuilt-daemon replay od-syntax-acceptance-R85H3J passed 10/10 with stable source and confirmed cleanup (six repaired, one clean, three warnings; fixed CLI, no model/upload). The preceding replay was invalidated by a concurrent test-source edit and is not an acceptance receipt. These local witnesses are not production monitoring results.

deliverableSyntaxValidation.finalization records summaryVersion 1, repairEngine host-safe-fixer@2, initialStatus, action/reason/refusal, stagedPatchCount, committedPatchCount and committedRepairRules. Staged rules are not committed rules. Physical terminal status completes the success criterion. Missing or partial historical summaries cannot prove Host recovery. The acceptance report separates repairVerified from deliveredWithWarning. The latter requires a succeeded physical Run, complete Host summary, warn, zero committed patches and a supported finalization reason. It may retain repairable, incomplete, or a staged pass with commit failure; no status is rewritten to claim repair. An actual failed/canceled Run cannot count as warning delivery.

Timing	Exact scope
checkerDurationMs	Sum of actual checker invocations in this physical Run
safeFixProposalDurationMs	Proposal evaluation, including refusals
repairDurationMs	Accepted proposal plus commit time; overlaps proposal time, excludes parser time
repairWindowDurationMs	Historical failed-check start to later passing-check start; excludes last check/commit
repairToTerminalDurationMs	First failed-check start to physical terminal, including failures
repairToDeliveryDurationMs	Legacy name for the same terminal window, not proof of success
replay discoveryToWarningDeliveryMs	Terminal window only when the original is delivered with warning; not a successful repair window
replay discoveryToRepairedDeliveryMs	Terminal window only with proven committed repair and succeeded delivery

Do not add proposal to repairDuration. Langfuse and ODEval retain the versioned summary and reason enums without diagnostic text, paths or source. Case aggregation must not turn a failed final Case into a recovered delivery. Legacy exhausted tool state alone does not prove that the Host actually blocked delivery.

For comparable versions with complete eligible terminal evidence, report recovered deliveries / initially broken deliveries as coverage, and recovered / eligible observed deliveries as overall contribution. Separately report delivered-with-warning, incomplete, true execution failure and unobserved records. A warning is neither a recovered delivery nor a blocked broken delivery. Incomplete evidence is not proof of a syntax error. Rule-hit counts mean deliveries matching a rule, not independent error counts. Report clean checker cost, proven repaired-delivery terminal windows and warning-delivery terminal windows separately; the warning window is not time spent successfully repairing the artifact.

Fixed regression is not evidence of online 99% coverage, p95 or causal conversion uplift. Before/after R2 capture, live dashboards and production ingestion acceptance are separate follow-ups; content consent, retention and storage policy do not change.