Two new skills ported from the internal research agent's skill set.
wandb-eval-tables: non-destructive conversion of W&B Table artifacts into
bounded, verified EvalTable previews. Scan-first preflight, truncation and batch
limits, verify-after-write, idempotency, and a safe undo path. The source
skill's UI/page-routing helper is not carried over — it was entirely coupled to
the originating product surface.
wandb-autoresearch: bounded training research through W&B Launch — readiness
gate, serial trials with a budget, noise-aware comparison, and resumable local
state. Depends on the Launch helpers in wandb-primary.
Pins the install commands at wandb>=0.28.1 to match the README Requirements
floor; the video path additionally needs the eval-table-video-support extra,
documented in references/EVAL_TABLES.md.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Merge the W&B-related operational content from the internal research agent's
skill set into wandb-primary, keeping the existing public guidance where it was
already equal or stronger.
New references: ARTIFACT_OPS, ARTIFACTS_AND_REGISTRY, COLLECTION_OVERVIEWS,
RUN_OPS, RUNS_TABLE. New scripts: run ops CLI (with wandb_run_ops_lib), run
vocab, artifact/collection-overview helpers, workspace inspection and guarded
write helpers, bounded Weave agent-conversation ops.
Merged rather than duplicated: Reports, run logs, Workspaces, WANDB_SDK,
WEAVE_SDK, and the Launch helpers gain the additions while keeping draft-first
reports, bounded pagination, guarded writes, retries, and read-back
verification. The removed beta history API is not reintroduced.
Document a Requirements floor in the README: the helpers are validated against
Python 3.13, wandb 0.28.1, wandb-workspaces 0.4.4, and weave 0.52.41. Older
SDKs expose some of these methods as unimplemented stubs.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* feat: add run logs, safe-eval helpers, and richer workspaces to wandb-primary
Second cutover pass over the WandBAgentFactory .production skill (core sync
0a41099), porting only generic, agent-free content. Stacked on the
WBAgent-cutover branch.
New references/RUN_LOGS.md (+190):
- Reading run console logs via the logLines GraphQL connection (paginate or
tail), multipart output.log layout + stitching, and crash/resume gotchas.
Pure SDK/GraphQL, no agent machinery.
weave_helpers safe-eval functions (+197):
- safe_eval_root_summary / safe_eval_child_summary / safe_project_eval_summary
(+ internal helpers): count predict_and_score rows server-side first and cap
payloads, so eval exploration stays memory-safe on large projects. Zero agent
deps; internal helpers stay private under `import *`.
WORKSPACES.md high-value sections (+109):
- 12-row panel-type table, run grouping (workspace/group/panel-level with
groupby_aggfunc/rangefunc), run visibility/pinning/baselines (incl.
cross-project pins + delta columns), list-all-views via GraphQL, and 5 added
gotchas. Stripped all workspace_ops/wandb_run_ops/sandbox references.
SKILL.md (+6): wire the new helpers and reference into the import list,
"When to use what" table, and Reference docs list.
Deliberately skipped: the wandb/autoresearch-base Launch image (agent-specific),
the Weave agents-plane section (weave_agent_ops), and Lazy/Bug signal exemplars
(already covered more thoroughly in SIGNAL_PROMPT_EXAMPLES.md).
Validation: validate_skills.py passes; 54 markdown python blocks parse clean;
no banned tokens; weave_helpers compiles and exports the 3 new functions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: drop the signal-building capability from wandb-primary
Signal building (creating Weave ClassifierMonitors + LLMAsAJudgeScorer prompts)
is a W&B-agent capability, not something a Codex/Claude-code agent reading and
analyzing W&B data needs. The helpers were a faithful copy of the production
source, so this is a scope/fit decision, not a regression fix.
Removed:
- scripts/signal_helpers.py + signal_helpers_impl.py
- references/SIGNALS.md (4-phase signal workflow)
- references/SIGNAL_PROMPT_EXAMPLES.md (13 default-signal prompt patterns)
- SKILL.md: the "Building Weave signals" section, the signal_helpers import
block, the "create/tag/classify" when-to-use row, the SIGNAL_PROMPT_EXAMPLES
reference entry, and "Signal Builder" from the skill description.
Kept: the generic weave.Monitor "set up production monitoring" pointer (raw
Weave SDK, no signal_helpers dependency).
Validation: validate_skills.py passes; no dangling signal references; 54
markdown python blocks still parse clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Port the genuinely portable content from the production W&B agent's
skills into the public wandb-primary skill, stripping all agent-specific
machinery (no ask_user, Modal/sandbox, docs-agent, llm-responses, or
surface framing). The public skill remains the source of truth.
SKILL.md (+256):
- Operating guidance: scope-and-approach (brief vs intense; probe runs
and Weave first), analyzing-a-project, closing-a-substantive-analysis,
respect-the-user's-scope, answer-from-helper-output-with-audit-trail.
- Safety: never delete a W&B project; confirm before destructive ops.
- jobType vs job_type filter gotcha.
- Net-new run-table recipes: artifact inventory, artifact file summary,
training-history/stability diagnosis, cohort/variant comparison.
New references/WORKSPACES.md (+235):
- Pure wandb_workspaces SDK: load/edit saved views, create from scratch,
sections/panels, runset settings, gotchas.
- Self-contained GraphQL fallback for the default (nwuser) workspace that
from_url rejects, with no internal CLI/lib dependency.
New scripts/report_helpers{,_impl}.py (+181):
- save_report_verified / edit_report_verified: save then read back from
the server so success is verified, not assumed. Follows the established
_impl + thin-wrapper packaging convention.
Validation: scripts/validate_skills.py passes; still exactly
{wandb-primary}; no banned tokens.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: sync public wandb-primary skill
Refresh the public primary W&B skill from the main internal source and scrub internal-only names, paths, and data assumptions before publishing.
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore: remove public skill bench workflow
Remove the benchmark workflow from the public skills PR because it exposes an evaluation trigger that should not be runnable by arbitrary contributors.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs: add wandb-workspaces report reference (#37)
* docs: add wandb-workspaces report reference
Document the updated report v2 API in a standalone follow-up reference so reviewers can evaluate the report changes separately from the core wandb-primary port.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs: remove stale report API guidance
Make REPORTS.md the single canonical report reference for wandb-workspaces v2 and guard against reintroducing the stale reports import path.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs: clarify report tag filters
Document structured tag filters for report runsets so single dynamic tags are wrapped as lists and agents avoid invalid string filter syntax.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs: clarify media browser report panels
Align the report reference with wandb-workspaces media browser examples by documenting gallery axes, grid axes, and explicit mode selection.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix eval workflow for WBAF SDK --agent flag change
The WandBAgentFactory SDK refactored run_eval to require --agent as a
mandatory CLI argument. Pass the already-computed agent name from the
setup step output.
* Updated the WandB skill to include better descriptions + report authoring from @ShawnLewis
Instead of hardcoding uv as the package manager, the skill now instructs
agents to detect the user's Python environment first and write the
finding into the skill itself. Falls back to uv only when the
environment cannot be determined.
Replace the placeholder in CONTRIBUTING.md with the actual contributing
instructions from README, and link to CONTRIBUTING.md from README.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Set up contributor-assistant GitHub Action to require CLA signatures
on pull requests, matching the wandb/weave CLA setup.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Update all references from wandb/agent-skills to wandb/skills in
README and CI workflow to reflect the repo rename.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Python blocks run with cwd=wbaf/ (via uv --directory wbaf), so
paths must not include the wbaf/ prefix. Shell keeps the prefix.
- Use WBAF_PAT for update-readme checkout to bypass branch protection.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
uv run --directory wbaf sets cwd to wbaf/, so Python paths must
not include the wbaf/ prefix. Shell commands (mkdir, echo fallback)
run from workspace root and keep the prefix.
Verified end-to-end in a tempdir simulating the CI checkout layout.
The printf command with single quotes and > redirect was being
parsed as YAML syntax rather than a shell command. Switch to
block scalar (|) to avoid ambiguity.
* Fix eval workflow: target main, add Claude Code agent matrix
- ref: wbaf-refact → ref: main
- Matrix strategy: benchmark both codex-mcp and claude-code
- Add ANTHROPIC_API_KEY for Claude Code agent
- Use all.yaml instead of codex-all.yaml (which no longer exists)
- Unique temp agent per matrix cell to avoid config collisions
- Separate README update job that aggregates results across agents
* Add install.sh, replace wandb skill with wandb-primary, update CI
- Add install.sh for easy skill installation (local or global Claude Code)
- Replace skills/wandb with skills/wandb-primary (comprehensive W&B + Weave skill
with helper scripts and reference docs)
- Update README with installation instructions (npx skills + install script)
- Fix CI matrix: use codex and claude-code as base agents (not codex-mcp)
- Fix all wandb-primary-v2 path references to wandb-primary
- Remove stale .badges directory
* Overhaul CI: per-agent badges, wba-all eval source, no task name leaks
- Switch eval source from all.yaml to wba-all.yaml (W&B-specific tasks)
- Run update_all_eval.py before each eval to pick up new tasks
- Replace raw eval-log artifact with results-summary.json (counts only)
- Generate per-agent badge JSON files (.badges/codex.json, .badges/claude-code.json)
- Add Codex and Claude Code shield badges to README
- Keep eval_tasks input override for manual dispatch
- No task names, rubric content, or eval details leak to public artifacts
-- AI was used to create large parts of this PR
Parses the eval summary from the log output to extract:
- passed count / total tasks
- score percentage from correctness or rubric scorer
Badge now shows e.g. "13/24 passing (54%)" instead of just "passing".
Per-agent badges will come later when WBAF supports multi-agent eval reporting.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The workflow no longer depends on pre-existing agent configs in WBAF.
Instead it:
1. Copies skills from this repo into WBAF's skills/
2. Writes a temp agent config (ci-skill-eval) extending codex-mcp with those skills
3. Writes a temp eval config pointing at that agent
4. Runs the eval
Inputs:
- skills: comma-separated list (blank = all)
- base_agent: agent to extend (default: codex-mcp)
- eval_tasks: task source (default: all from codex-all.yaml)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Workflow status badge at top of README
- Skill table with markers (<!-- BEGIN/END SKILL TABLE -->) auto-updated by CI
- CI reads skill descriptions from frontmatter and writes status into table
- Replaces JSON badge files with direct README updates (works for private repos)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- GitHub Actions workflow that runs on push/PR to main (when skills/ changes)
- Clones WandBAgentFactory (wbaf-refact branch) for the eval framework
- Copies skills into the eval framework and runs Codex agent with MCP
- Generates badge data on main merges
- Weekly scheduled run on Sundays at 11 PM PT
- Updated README with skill format docs and contributing guide
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Zubin Aysola