mirror of
https://github.com/streamlit/agent-skills.git
synced 2026-09-14 15:29:04 +08:00
Nico Bellante
d84e3344da
## Summary Replaces the old `developing-with-streamlit` skill (21 sub-skills + 17 templates) with a single lightweight **meta-skill** that discovers Streamlit's bundled skills inside the pip package (>=1.57). One user-level install works across every project; the discovered skills always match each project's pinned Streamlit version. Net diff: **+1,171 / −12,738** across 99 files. ## The contract `developing-with-streamlit/scripts/discover.py` is the actual logic — stdlib-only Python, cross-platform via `pathlib` + the `venv` module's Windows/POSIX awareness. It detects the active Python interpreter (`$VIRTUAL_ENV` → `./.venv` → `../.venv` → `<git-root>/.venv` → `$CONDA_PREFIX` → `pipenv` if `Pipfile` → `poetry` if `poetry.lock` → `pdm` if `pdm.lock` → `uv` if `uv.lock` → system), runs `import streamlit; print(streamlit.__path__[0])`, and emits one of six exit codes: | Exit | Meaning | Agent action | |---|---|---| | 0 | Bundled SKILL.md found | Read printed path, follow its routing | | 1 | Streamlit not installed | Confirm with user, then install with the **single** package-manager-matched command in the error (`poetry add streamlit` for poetry, `uv add streamlit` for uv, `<venv-python> -m pip install streamlit` for venv, etc.) — no buffet | | 2 | Streamlit < 1.57 | Suggest `pip install --upgrade streamlit` for version-matched bundled skills; fall back to `docs.streamlit.io/llms-full.txt` if user can't upgrade | | 3 | No Python interpreter | Ask user to install Python 3.10+ (suggest `uv` as the easy path) | | 4 | Bundled skills exist but expected sub-path missing | Read listed skills directly | | 5 | Invalid argument | Fix invocation | `SKILL.md` is a thin wrapper that documents this contract and tells the agent to invoke `python <SKILL_DIR>/scripts/discover.py --project-dir <USER_PROJECT_DIR>`. The `--project-dir` arg makes resolution deterministic regardless of the agent's CWD at invocation time. `detect_interpreter()` returns `(cmd, tag)` where `tag` identifies which detection branch fired. `install_advice(cmd, tag)` maps the tag to one matching install command, so a poetry project's exit-1 message points at `poetry add streamlit` instead of dumping pip + uv + poetry + pipenv + conda and asking the agent to guess. Why a script instead of prose: same logic runs in production AND in tests, so no parallel reference impl can drift. Pure-prose meta-skills need LLM-in-the-loop evals to validate; this gets the bulk of realistic failure modes from cheap pytest at zero LLM cost — modulo the description-triggering question, which only LLM eval can answer. ## Tier 1 test matrix 19 pytest tests (26 cases including parametrized) in `tests/discovery/test_discovery.py`. Each E2E test creates an isolated `tmp_path` venv via the stdlib `venv` module, installs Streamlit fresh, runs `discover.py`, and asserts the contract. Coverage: - **Detection priority**: each interpreter source in our priority order, including `<git-root>/.venv` (helpful for deep monorepo layouts) - **Conflict**: `$VIRTUAL_ENV` wins over `./.venv` - **Fallbacks**: missing-Streamlit (exit 1), pre-1.57 (exit 2), upstream-restructured (exit 4) - **Invocation contract**: `--project-dir` argument from outside the project - **Tool-marker detection**: `Pipfile` (pipenv), `poetry.lock` (poetry), `pdm.lock` (pdm), `uv.lock` (uv) - **uv branch isn't too eager**: doesn't fire on bare `pyproject.toml` without `uv.lock` - **Tool-aware install advice**: `install_advice` parametrized over all 10 tags (virtual-env, venv-local, venv-parent, venv-git-root, conda, pipenv, poetry, pdm, uv, system) — guarantees the per-tag mapping doesn't regress Negative-test verified: temporarily reversing priority order in `discover.py` causes `test_priority_venv_over_local` to fail with a clear diagnostic; reverting the `uv.lock` marker causes `test_uv_no_lockfile` to fail similarly. Both prove the tests catch the regressions they're designed for. CI runs on `ubuntu-latest` AND `windows-latest`, on PR + push to main + weekly cron. Tests install whatever `streamlit` is currently latest on PyPI — failures here are an honest signal that something broke, either in our code or in upstream's bundled-skill layout. ### Latest passing run [Run 25827312532](https://github.com/streamlit/agent-skills/actions/runs/25827312532) on commit `7fd4fae`: | Job | Result | Duration | |---|---|---| | [`ubuntu-latest`](https://github.com/streamlit/agent-skills/actions/runs/25827312532/job/75883639893) | 26 passed | 4m 33s | | [`windows-latest`](https://github.com/streamlit/agent-skills/actions/runs/25827312532/job/75883639903) | 26 passed | 9m 13s | All 26 cases pass on both runners — no skips. Every Windows-specific concern (Scripts vs bin, `python.exe` vs `python`, path separators, `Pipfile` / `poetry.lock` / `pdm.lock` / `uv.lock` markers) is empirically validated. Each E2E test pins the resolved path to its own venv via `assert_resolves_bundled(..., inside=...)`, so a wrong-but-valid Streamlit install elsewhere on the runner can't silently satisfy the assertion. ## End-to-end verification (LLM-in-the-loop) Hands-on smoke test in a fresh Claude Code v2.1.128 session — done by me before opening this for review, not by an automated script. **Setup:** Built a `streamlit==1.57.0` venv at `/tmp/test-streamlit-install-2/`, installed the skill at `~/.claude/skills/developing-with-streamlit/`. Started Claude Code in the project directory with no prior context, no `CLAUDE.md`, no skill pre-selected. **Prompt:** > *I want to add a small KPI dashboard to a new app.py in this directory — three metric cards showing total revenue, active users, and conversion rate. Use placeholder values for now. Make it look polished.* **Observed (full trajectory captured in the session JSONL — 14 tool calls):** 1. Description match → `Skill(developing-with-streamlit)` fires automatically. The prompt never names the skill or mentions `discover`. 2. `Bash` runs `discover.py --project-dir <project>` with **no permission prompt** — the `allowed-tools` frontmatter pre-approves the bundled script. 3. Stdout returns the bundled `SKILL.md` path inside the venv. 4. `Read` bundled SKILL.md, then `references/dashboards.md` and `references/design.md`. 5. `ls` + `Read` of the `dashboard-metrics` template. 6. `Write` `app.py` using bundled patterns: `st.metric(border=True)`, horizontal container, Material icons, sparkline trends. 7. Step 4 of the bundled SKILL.md respected — agent offered to run the app rather than auto-running. Whole chain held end-to-end. No skill misroutes, no silent fallthroughs, no freelancing of Streamlit code without first reading the bundled refs. ## Acknowledged gaps - **Multi-env disambiguation** — user with both conda + venv pinning different Streamlit versions gets whichever the priority order picks. Acceptable without a `cortex env detect`-style CLI. - **Cross-agent `allowed-tools` support** — the pre-approval works in Claude Code but is ignored by Cursor / Copilot / Gemini / Codex. They fall back to their own permission prompts, which is the safe default — just slightly more friction on first invocation in those agents. - **Cross-agent end-to-end validation** — the smoke test above was Claude Code only. Other agents follow the same Agent Skills spec and should work via `npx skills add`, but individual agents haven't been verified hands-on. ## Manual verification All three confirmed by the cold-start hands-on test above: - [x] Description's trigger phrases auto-fire the skill (no prompt mentioned `developing-with-streamlit` or `discover`). - [x] Agent invokes `scripts/discover.py` rather than winging it from prose. - [x] `allowed-tools` frontmatter pre-approves the bundled `discover.py` (no bash permission prompt on first invocation in Claude Code).
Discovery tests
End-to-end tests for developing-with-streamlit/scripts/discover.py.
Each test exercises one environment shape from the meta-skill's documented
priority order, or one of the fallback codepaths. Tests are pytest-based,
stdlib-only (no third-party deps beyond pytest itself), and cross-platform
— the same source runs on Linux, macOS, and Windows.
Layout
tests/discovery/
conftest.py # shared helpers: make_venv, run_discover, assert_resolves_bundled
test_discovery.py # all tests, 19 functions covering the documented matrix
run-local.sh # convenience wrapper around `pytest tests/discovery/`
README.md
Running locally
pip install pytest # one-time
bash tests/discovery/run-local.sh # full suite
bash tests/discovery/run-local.sh -k pipenv # pytest filter
bash tests/discovery/run-local.sh -x # stop at first failure
Tests that need third-party tools (uv, pipenv, poetry, pdm, conda) skip
cleanly when those tools aren't installed — pytest reports them as SKIPPED
rather than errors. CI installs uv, pipenv, poetry, and pdm so those
tests run; conda is intentionally not installed on CI runners, so test_conda
is reported as SKIPPED there and only exercised locally.
Each test takes ~5–15 seconds (most of which is pip install streamlit into
a fresh venv). A full local run is under 2 minutes.
Running in CI
Triggered by .github/workflows/test-discovery.yml on PRs that touch
developing-with-streamlit/** or tests/**, on push to main, and weekly
via cron. Two OS jobs:
- Linux (
ubuntu-latest): runs the full suite;test_condaskips. - Windows (
windows-latest): runs the full suite;test_condaskips.
Both jobs install the same toolchain (Python 3.12, pytest, uv, pipenv,
poetry, pdm) before running the suite, so coverage is identical across
OSes. conda is deliberately not installed — its tests run locally on
machines that already have it.
What this catches
- Regressions in the priority order (e.g.
./.venvsilently winning over$VIRTUAL_ENV). - Breakage when Streamlit upstream moves
.agents/skills/. - Broken fallback messages for missing Streamlit or pre-1.57 versions.
- Cross-platform regressions (Windows-specific path handling,
Scripts/python.exevsbin/python). discover.pybugs that wouldn't surface on the author's single machine.
What this does NOT catch
- LLM misinterpretation of
SKILL.mdprose (Tier 2 / cold-start eval — out of scope). - Bugs in the bundled skills themselves (upstream repo's concern).
- Environment shapes we don't document: hatch-managed envs without activation,
pyenv-virtualenvwithout activation.
Adding a test
- Add a function to
test_discovery.py. Use thetmp_pathfixture for an isolated working directory, andmake_venv()/run_discover()fromconftest.pyfor setup. - If the test depends on a tool that may not be installed, add
@pytest.mark.skipif(shutil.which("toolname") is None, reason="..."). - CI matrix is OS-only (
ubuntu-latest,windows-latest); pytest discovers new tests automatically — no workflow changes needed.