mirror of https://github.com/openprose/prose.git synced 2026-09-19 05:55:05 +08:00

Files

T

dan 012650a201 [codex] Prepare OpenProse 0.12.0 responsibility runtime release (#57 )

* Document native runtime skill semantics

* Rename native runtime to responsibility runtime

* Add responsibility runtime compiler skeleton

* Add responsibility compilation IR

* Compile Forme manifests into repository IR

* Serve static repository IR

* Tighten responsibility runtime review gaps

* Introduce OpenProse root model

* Harden OpenProse root resolution

* Add responsibility judge activation

* Add responsibility pressure fulfillment path

* Add repository status inspection

* Make triggers concrete gateway registrations

* Add live responsibility trigger adapters

* Refine responsibility runtime validation

* Harden responsibility runtime freshness

* Tighten responsibility runtime guardrails

* docs: replace skill examples with native repositories

* chore: release v0.12.0

* fix: tighten release review gaps

2026-05-04 09:26:26 -07:00

2.5 KiB

Raw Permalink Blame History

name, kind, subject, tier, contract_version

name	kind	subject	tier	contract_version
agent-readiness.eval	test	agent-readiness	service	v1

Agent Readiness — Service Test

Protects the demo-facing service for new users: every run must produce a grounded, short, actionable report the caller can screenshot.

Fixtures

fixture: agent_friendly_domain company_name: OpenProse domain: prose.md focus: all expected_result: overall_score >= 60 and report_path exists
fixture: js_wall_marketing_site company_name: FictionalJsApp domain: a known JS-rendered marketing page with no /llms.txt focus: marketing-site expected_result: parseability score <= 40 and exactly one top_fix names a plain-HTML fallback
fixture: missing_well_knowns company_name: Bare domain: a domain returning 404 on /llms.txt, /sitemap.xml, /.well-known/ai-plugin.json focus: all expected_result: discoverability score <= 30 and top_fixes[0] names /llms.txt
fixture: unreachable_domain company_name: GhostCo domain: a domain that does not respond focus: all expected_result: errors contains domain_unreachable and no overall_score is produced

Expects

path: workspace_dir predicate: exists_on_disk and matches(^./runs/./workspace/agent-readiness/[a-z0-9-]+/$)
path: report_path predicate: exists_on_disk and ends_with("/bindings/agent-readiness.md")
path: overall_score predicate: is_integer_between(0, 100)
path: dimension_scores predicate: every_value_is_integer_between(0, 100)
path: top_fixes predicate: length == 3 and every_item_names_a_file_path_or_url
path: report predicate: contains_section("Raw Probe Results") and every_finding_grounds_in_raw_probe_results

Expects Not

path: report predicate: no_recommendation_without_file_path
path: report predicate: no_ungrounded_score (every score has at least one evidence bullet)
path: execution predicate: no_javascript_execution_attempted
path: execution predicate: no_fetch_outside_target_origin (probes stay on the user's domain)
path: workspace_dir predicate: no_write_outside_workspace_or_declared_binding

Performance Tracked Over Time

metric: wall_clock_seconds source: OpenProse run telemetry direction: down alert_when: p95 > 60s
metric: ungrounded_finding_rate source: human triage of demo runs direction: down alert_when: more than 1 ungrounded finding in a rolling 10 runs
metric: caller_screenshot_rate source: demo feedback ("did you share the report?") direction: up alert_when: share rate drops below 50% in a rolling 10 demos

2.5 KiB Raw Permalink Blame History