This PR:
- unpins the examples harness from the staging host: `COMPOSIO_BASE_URL`
now selects the backend and still defaults to staging, with the
structural checks kept (https, bare root, no
path/query/fragment/credentials) — `examples-live.yml` sets staging
explicitly, so CI is unchanged
- renames `harness/staging-backend.mjs` to `harness/backend-url.mjs` and
`requireStagingBaseUrl` to `resolveBackendBaseUrl`, and relaxes the
matching hard-exit in `harness/trace-py/sitecustomize.py`
- fixes the Python candidate swap: `relaxPyClientPin()` drops the
project's exact `composio-client==` pin for the duration of a candidate
sweep, so `COMPOSIO_CLIENT_WHEEL` can win over it, and
`python/pyproject.toml` + `uv.lock` are restored from a pre-run snapshot
afterwards
- adds `harness/README.md` covering backend selection, the
candidate-swap flow for both languages, and how to tell a real parity
pass from a vacuous one
- ignores `.artifacts/`, where every sweep writes its results, traces,
and downloaded client
- extends `harness/run.mjs selftest` with eleven checks over the new
backend-URL behaviour and the pin relaxation; the suite passes locally
## Context
Found while producing a Python examples parity run for the
`composio-client` 2.x bump, which lives on a separate branch. Both
problems blocked that run and neither is specific to it.
The staging pin made a comparison against any other project impossible
without editing the harness in place.
The Python candidate swap was worse than blocking, because it failed
quietly. A candidate sweep layers the wheel onto every entry with `uv
run --with`; seven of the twenty-two Python entries also install the
local `./python` project through `pyWith`, and uv cannot satisfy that
project's exact pin and the candidate wheel together:
```
No solution found when resolving `--with` dependencies:
Because only composio==0.20.0 is available and composio==0.20.0 depends on
composio-client==<pinned>, [...] your requirements are unsatisfiable.
```
Those entries went red in under a tenth of a second for a packaging
reason rather than a client one, and `parity.mjs` only compares entries
green on both sides — so the comparison silently shrank to fifteen
entries and still reported a clean pass. With the pin relaxed, the same
sweep runs 22/22 green and compares all twenty-two.
https://claude.ai/code/session_015Bx2BaSwuq2cxgJSRtAxag
This PR:
- builds on merged PR #4107
- adds `examples-manifest.json`: 93 runnable entrypoints across both
SDKs with tiers (unattended / provisioned / bounded / excluded),
per-entry timeouts, env contracts, and labeled readiness markers for
OAuth flows
- adds `harness/run.mjs`: sweeps entries against staging with per-entry
Composio traces, negative controls (garbage credentials must fail), a
`selftest`, and a baseline/candidate client swap for parity runs
- rejects every non-staging `COMPOSIO_BASE_URL` before the harness or
provisioning script can send credentials
- adds `--llm mock`: LLM examples run against a pinned
`@copilotkit/aimock` server with scripted tool-call fixtures
(`harness/llm-mock/`) — full sweeps spend zero model tokens while
Composio tool execution stays live on staging; 46/48 LLM entries pass
mocked, and the two `@openai/agents` entries are marked `llmMock:false`
- adds `examples-live.yml`: nightly gc-provision → mock sweep →
four-entry live LLM canary; dispatchable per language, client, entry
subset, and LLM mode, and bound to the `staging` GitHub environment
- accepts separate TypeScript tarball and Python wheel inputs for
candidate sweeps, restores exact pre-run workspace files on failure, and
rejects empty or partial parity comparisons
The configured `COMPOSIO_API_KEY` must belong to a dedicated, disposable
staging project. The workflow becomes dispatchable after it exists on
the default branch.
## Context
Examples historically went stale because nothing executed them. PR #4107
makes them runnable and provisionable; this PR adds the machinery that
runs them on a schedule and validates the Stainless→self-managed client
swap via baseline/candidate parity sweeps.
```mermaid
graph TD
M[examples-manifest.json] --> R[harness/run.mjs sweep]
P[examples-provision.mjs --gc] -->|COMPOSIO_EXAMPLES_* ids| R
R -->|tier 1-3 entries| E[TS + Python examples]
A[aimock fixtures] -->|--llm mock| E
E -->|live tool execution + traces| C[Composio staging]
R --> N[nightly live canary: 4 entries]
```
## Verification
- `env -u COMPOSIO_BASE_URL node harness/run.mjs selftest`
- `pnpm test:examples` against a clean tracked snapshot
- manifest integrity: 93 unique entry IDs and 93 existing files
- JavaScript, Python, JSON, YAML, and embedded Bash syntax checks
- `pnpm validate:changesets`
Alberto Schiabel