- Updated agent-runner documentation to clarify the requirement for exact model ID matching.
- Implemented model identity checks in run_agent.py to ensure the response model matches the configured model.
- Introduced a new script, run_paired_trial.py, to facilitate bounded A/B testing with a constrained skill-guidance protocol.
- Added comprehensive tests for the paired trial CLI in test_run_paired_trial.py, covering various scenarios including credential handling, preflight checks, and model identity validation.
Studied via direct fetches: SakanaAI/AI-Scientist-v2, WecoAI/aideml,
anthropics/skills, obra/superpowers, UniM0cha/claude-self-improving-
skills, SamuelSchmidgall/AgentLaboratory, princeton-nlp/SWE-agent,
openai/preparedness (PaperBench), open-mmlab/mmsegmentation,
facebookresearch/dinov2, karpathy/nanoGPT. Applied the top verified
findings:
- extract_commands: join backslash-continued commands (every dinov2 run
command was previously truncated to an unrunnable `python ... \` stub)
and classify entrypoint-first with word-boundary keywords (nanoGPT's
`train.py ... --eval_iters=20` classified as evaluation, which would
execute training while bypassing the authorization gate). Quick-start
sections now count as run sections. Both real-repo patterns added as
regression fixtures.
- research-thinking-loop: draft/debug/improve iteration types with a
3-attempt debug cap (AIDE journal semantics), no-parsed-metric ⇒ buggy
and never best, replication across 3 seeds before a candidate replaces
current_research (AI-Scientist-v2 multi-seed evaluation), metric-only
best selection note, typed stop reasons, defaults table, ledger-first
grounding.
- annotate_readme: PaperBench-style evidence tiers per annotation
(code-development / execution / result-match) and a weighted 0-1
reproduction score in the header chip and readme_section_coverage.
- lessons_store: touch/prune lifecycle with usage-extended staleness
windows, credential-shape blocklist (AKIA/ghp_/sk-/xox/AIza),
best-effort security wording, what-NOT-to-record and human-reviewed
promotion flow in the policy.
- Demo bundles and preview images regenerated with score and tiers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HT2VAaQodjqTJdjBHSStxA
Research thinking loop (references/research-thinking-loop.md):
- Codifies the greedy, evidence-anchored cycle required in the explore
lane: observe -> ground (paper/code/prior-run/intuition anchors) ->
falsifiable hypothesis -> single-variable design -> bounded run ->
fair comparison vs current best -> greedy keep/rollback -> ledger
record. Greedy applies to selection, never to honesty: keeps require
comparable evidence under the frozen contract.
- Adapts AIDE's greedy code-space search and AI-Scientist-v2's managed
agentic tree search under RigorPilot's comparability-first gates.
- Wired into ai-research-explore SKILL.md and AGENTS.md.
Continuous learning (references/continuous-learning-policy.md):
- Immutable universal rigor core + user-owned lessons overlay.
- shared/scripts/lessons_store.py: JSONL store under ~/.rigorpilot/
(RIGORPILOT_HOME override, RIGORPILOT_LESSONS=0 opt-out), secret-line
refusal, dedup, bounded summarize into PERSONAL_RIGOR.md with an
explicit core-wins disclaimer.
- orchestrate_repro auto-records failure blockers and their later
resolutions per repo fingerprint; payload reports lesson_recorded.
- Hard rules: lessons are advisory only, never relax rigor gates, never
store secrets, never edit skill files; deleting the folder restores
base behavior.
- Installed alongside the skills (shared references + shared scripts).
Docs and tests:
- READMEs (en/zh): new Research Thinking Loop section with the greedy
cycle mermaid replacing the old explore-flow diagram, new Continuous
Learning section, two At-a-Glance rows, counts 48 scripts / 45 tests.
- scripts/test_lessons_store.py: persist/dedup/secret-refusal/summarize/
opt-out/CLI coverage (7 checks); full suite 46/46 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HT2VAaQodjqTJdjBHSStxA
- scripts/run_all_tests.py: cross-platform entrypoint running
validate_repo.py plus every scripts/test_*.py with per-script timing
and failure tails; new tests are picked up automatically.
- .github/workflows/validate.yml: replace the hand-listed 9-test subset
with the full-suite entrypoint (fail-fast off across the
ubuntu/macos/windows matrix; installer smoke steps kept). The three
POSIX crashes fixed earlier in this branch had been invisible
precisely because their tests were missing from the CI list.
- READMEs: CI status badge, run_all_tests as the primary local
validation command, script count 47.
- CHANGELOG: release the Unreleased section as v1.1.0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HT2VAaQodjqTJdjBHSStxA
- Add root AGENTS.md so AGENTS.md-aware agents (OpenAI Codex, Cursor,
GitHub Copilot, Gemini CLI, Aider, Zed, ...) pick up the lane model,
entrypoint table, evidence contract, and hard rules with no install
step, complementing the Agent Skills (SKILL.md) standard the skills
already follow.
- Annotated README gains a rubric-style section-coverage scoreboard in
its header (e.g. 1 green / 2 yellow / 1 purple / 10 read-only of 14
sections), and the counts are recorded machine-readably as
readme_section_coverage in repro_outputs/status.json — inspired by how
replication benchmarks like PaperBench grade reproduction with
hierarchical rubrics instead of a single pass/fail. The annotation now
runs before the bundle writers so status.json carries the coverage and
the annotated-readme output path.
- READMEs: promote the annotated README to flagship position (moved
directly after Start Fast, new At-a-Glance rows), add a Multi-Agent /
Multi-Model section, and document the scoreboard. zh-CN mirrored.
- client-compatibility-policy.md: cover the Agent Skills open standard
and AGENTS.md consistency rules.
- Demo bundles regenerated with the scoreboard; renderer test now checks
the coverage line (17 checks).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HT2VAaQodjqTJdjBHSStxA
Annotation content (annotate_readme.py):
- Executed sections: command, observed metrics, completed steps on
success; blocker, stderr error excerpt, and suggested next action on
partial/blocked, with explicit low/medium/high risk labels.
- Setup / not-executed sections list the actual commands instead of a
bare count.
- Training sections in the trusted lane get a purple decision annotation
(requires explicit authorization) even when another goal ran.
- Data-preparation sections get a yellow data-readiness warning when no
local dataset is detected (fed by the real asset manifest via a new
local_dataset_present flag from the orchestrator).
- Every boxed annotation carries a compact evidence-links line.
Demo (examples/annotated-readme-demo{,-zh}):
- The MiniSeg demo README now mirrors a real project: centered title,
badge row, paper/project/colab links, highlights, news, two model-zoo
tables, pinned install versions, dataset layout tree, single- and
multi-scale eval, torchrun training, project structure, FAQ, citation.
- Two committed scenarios per language, generated by real runs of the
orchestrator: first-run/ (checkpoint missing -> yellow partial with a
real FileNotFoundError excerpt, dataset-not-ready warning, purple
training decision) and after-setup/ (green success with observed
mIoU/aAcc). README/README.zh-CN link both scenarios and show the
richer preview blocks.
Tests: extend test_readme_annotation.py to cover the data-readiness
warning, the training authorization annotation, command listings, and
the error excerpt (16 checks).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HT2VAaQodjqTJdjBHSStxA
- Prose-only sections get a single dim <sub> line instead of a full
admonition box, so annotations no longer dominate the page.
- Annotation content collapses to one summary line (status · command ·
metric · steps); only partial/blocked keep a second blocker line.
- Drop the repeated RigorPilot prefix and per-annotation evidence links;
the header carries the evidence links and a compact legend once.
- Trim trailing blank lines per block for tighter spacing.
- Update README previews (en/zh) and the renderer regression test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HT2VAaQodjqTJdjBHSStxA
ai-research-reproduction now writes repro_outputs/ANNOTATED_README.md:
the target repository's README replayed verbatim, split into heading
blocks (code-fence aware), with a GitHub-renderable annotation under
each block describing what the agent did there, colored by risk —
[!TIP] green success, [!NOTE] blue informational / read-only,
[!WARNING] yellow partial, [!CAUTION] red blocked, [!IMPORTANT] purple
decision required. Every annotation links back to the evidence bundle
(SUMMARY.md, COMMANDS.md, LOG.md, status.json), so the researcher can
see at a glance what ran, what was skipped, and where attention is
needed, then drill into details on demand. Annotations are bilingual
(en/zh) via the existing --user-language flag.
- skills/ai-research-reproduction/scripts/annotate_readme.py: renderer,
also usable standalone via --readme/--context-json/--output.
- orchestrate_repro.py: writes the annotated README after the bundle
and reports it as annotated_readme in the payload.
- Registry, SKILL.md, output-contract updated; READMEs gain a rendered
preview section (en + zh) and updated script counts.
- scripts/test_readme_annotation.py: verifies original content is
preserved verbatim, per-status admonition colors, evidence links,
fence safety, and zh localization.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HT2VAaQodjqTJdjBHSStxA
Correctness (explore lane):
- execution_feasibility: silence stdout/stderr of import/constructor
probes so repo modules with import-time prints no longer corrupt the
orchestrator's JSON stdout contract.
- orchestrate_explore: guard axis_flag_map null; coerce scalar/null
variant_axes, subset_sizes, short_run_steps in normalize_variant_spec;
pass the real campaign into execute_variant_candidates so executed-run
ranking honors evaluation_source.primary_metric; carry
source/source_repo/source_file/source_symbol through
normalize_candidate_ideas so candidate-idea source triples reach
lookup_sources.
- source_mapping: honor a researcher-requested patch class the plan can
satisfy instead of escalating to module-transplant-shim.
- candidate_idea_generation, plan_code_changes, plan_variants: defensive
coercion for scalar/null variant axes and empty subset/step lists.
Correctness (trusted lane):
- analyze_project, plan_code_changes: apply SKIP_PARTS filters to
repo-relative parts, so repos under /tmp (or any ancestor named tmp,
artifacts, *_outputs) no longer produce empty analyses/targets.
- extract_commands: stop treating markdown blockquotes as shell prompts;
ignore fenced code lines when collecting headings.
- orchestrate_repro: shlex.split posix=True (consistent with runners),
fixing quoted args and quoted --config paths.
- run_training, run_command: exclude epoch/step/iter counters from
best-metric selection; degrade gracefully when the git binary is
missing or hangs instead of crashing before the run.
- safe_debug: match "out of memory" instead of bare "oom" substring
(no more bloom/room misclassification).
- bootstrap_env: conda/mamba --dry-run now previews the dependency
install step like the venv path does.
Installer:
- Install shared/scripts bundle writers next to the skills so installed
write_outputs.py wrappers work in copy mode.
- Add explore-variant-spec.md and research-pitfall-checklist.md to the
shared references; fix the two SKILL.md paths pointing at them.
- safe_remove: validate containment lexically and unlink symlinks
without resolving, so --force can replace symlink-mode installs.
- test_install_targets: derive reference list from the installer, run an
installed writer, and cover the symlink-then-copy --force path.
Docs and hygiene:
- README/README.zh-CN: EXPERIMENT_LEDGER.md is generated today, not a
future concept; add the missing lifecycle-view section to zh-CN.
- Strip UTF-8 BOMs from two test scripts and one example doc.
- .gitignore: cover codex-* test scratch dirs and all *_outputs dirs.
- Set exec bits on install_skills.py and bootstrap_env.sh.
- Remove stale one-skill skills-lock.json leftover.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HT2VAaQodjqTJdjBHSStxA
- Introduced `agent-operating-principles.md` to provide directional guardrails for agent behavior across skills.
- Updated `trigger-boundary-policy.md` to refine skill invocation rules based on specific prompt phrases.
- Enhanced `install_skills.py` to include installation of shared reference files.
- Added tests to ensure shared principles are referenced correctly in skills and validate their structure.
- Updated various skills to reference the shared operating principles, ensuring consistent guidance across the repository.
- Improved `research-campaign-spec.md` to clarify the minimal shape and optional guidance fields for campaign inputs.
- Refined `ai-research-explore` and `ai-research-reproduction` skills to align with new principles and clarify their intended use cases.
- Added new test cases to validate the correct routing of prompts based on the updated boundary policies.
- Introduced language policy to ensure human-readable outputs while maintaining machine-readable stability.
- Created output specification detailing required output files and their formats.
- Established patch policy to guide conservative modifications to repository code.
- Defined research safety principles to prioritize observable and reviewable research progress.
- Implemented orchestration script for minimal README-first reproduction, including command extraction and execution handling.
- Added functions for managing assets, estimating training duration, and building context for reproducibility.
- Implemented `lookup_sources.py` to facilitate a free-first, cache-first research lookup pass, including functions for deduplication, seed record collection, and provider record resolution.
- Introduced `source_mapping.py` for source module lookup and interface diffing, featuring functions to select source records, build target location maps, and generate minimal patch plans.
- Enhanced data normalization and evidence class handling to improve the accuracy of source mapping.
- Added utility functions for writing module candidates and interface diffs to markdown files for better documentation.
- Introduced `test_research_explore_campaign_abandon.py` to validate abandonment of far-from-SOTA baselines.
- Added `test_research_explore_campaign_checkpoint.py` to ensure human checkpoint gating for near-tied ideas.
- Created `test_research_explore_campaign_flow.py` to verify the overall campaign flow and execution policies.
- Documented the `research_campaign_spec.md` outlining the structure and requirements for campaign JSON/YAML configurations.
- Added `run_training.py` modifications to improve command execution handling.
- Introduced `explore-module-roadmap.md` to outline the evolution of the explore lane.
- Created `explore-variant-spec.md` for defining variant specifications in exploratory runs.
- Implemented regression tests for non-training and training variant execution in `test_research_explore_nontraining_execution.py` and `test_research_explore_variant_execution.py`.
- Developed `plan_code_changes.py` to generate exploratory code-change plans.
- Added `run_command.py` for executing short non-training commands and normalizing evidence.
- Added `research-explore` skill to facilitate end-to-end exploratory work on top of `current_research`.
- Updated `explore-code` and `explore-run` skills to clarify boundaries and usage.
- Enhanced documentation for all skills to reflect new orchestrator capabilities and requirements.
- Implemented orchestration logic in `scripts/orchestrate_explore.py` to manage exploratory code and run changes together.
- Created output structure for exploratory results in `explore_outputs/`.
- Added regression tests for the new orchestrator to ensure proper functionality and integration.
lama