Alberto Schiabel
5b898e7957
This PR: - unpins the examples harness from the staging host: `COMPOSIO_BASE_URL` now selects the backend and still defaults to staging, with the structural checks kept (https, bare root, no path/query/fragment/credentials) — `examples-live.yml` sets staging explicitly, so CI is unchanged - renames `harness/staging-backend.mjs` to `harness/backend-url.mjs` and `requireStagingBaseUrl` to `resolveBackendBaseUrl`, and relaxes the matching hard-exit in `harness/trace-py/sitecustomize.py` - fixes the Python candidate swap: `relaxPyClientPin()` drops the project's exact `composio-client==` pin for the duration of a candidate sweep, so `COMPOSIO_CLIENT_WHEEL` can win over it, and `python/pyproject.toml` + `uv.lock` are restored from a pre-run snapshot afterwards - adds `harness/README.md` covering backend selection, the candidate-swap flow for both languages, and how to tell a real parity pass from a vacuous one - ignores `.artifacts/`, where every sweep writes its results, traces, and downloaded client - extends `harness/run.mjs selftest` with eleven checks over the new backend-URL behaviour and the pin relaxation; the suite passes locally ## Context Found while producing a Python examples parity run for the `composio-client` 2.x bump, which lives on a separate branch. Both problems blocked that run and neither is specific to it. The staging pin made a comparison against any other project impossible without editing the harness in place. The Python candidate swap was worse than blocking, because it failed quietly. A candidate sweep layers the wheel onto every entry with `uv run --with`; seven of the twenty-two Python entries also install the local `./python` project through `pyWith`, and uv cannot satisfy that project's exact pin and the candidate wheel together: ``` No solution found when resolving `--with` dependencies: Because only composio==0.20.0 is available and composio==0.20.0 depends on composio-client==<pinned>, [...] your requirements are unsatisfiable. ``` Those entries went red in under a tenth of a second for a packaging reason rather than a client one, and `parity.mjs` only compares entries green on both sides — so the comparison silently shrank to fifteen entries and still reported a clean pass. With the pin relaxed, the same sweep runs 22/22 green and compares all twenty-two. https://claude.ai/code/session_015Bx2BaSwuq2cxgJSRtAxag
Examples harness
Runs the entrypoints inventoried in examples-manifest.json against a live
Composio backend, records every backend call each entry makes, and compares two
runs call-for-call.
harness/run.mjs— sweeps entries, writesresults.jsonl+ one trace file per entry under.artifacts/examples-parity/<run-id>/.harness/parity.mjs— compares two run directories.harness/trace/register.mjs,harness/trace-py/sitecustomize.py— the fetch and httpx tracers, injected by the runner. Examples never referenceCOMPOSIO_TRACE_FILEthemselves.scripts/examples-provision.mjs— provisions the auth configs and connected accounts the tier-2/3 entries need, and prints them asCOMPOSIO_EXAMPLES_*exports.
Backend selection
COMPOSIO_BASE_URL picks the backend and defaults to staging. Any bare https
root is accepted; a URL carrying a path, query, fragment, or embedded
credentials is refused, because the tracers pin the backend host and the
provisioner appends its own paths. .github/workflows/examples-live.yml sets
staging explicitly, so CI is unaffected by the default.
Point this at a project whose data you are willing to have the examples touch.
scripts/examples-provision.mjs --gc deletes examples-owned resources older
than 24h across the whole project — never run it against a project you care
about. The outbound-email denylist is enforced in the tracers and is not
overridable per run; --llm mock additionally keeps model traffic on a local
aimock server so no agent can decide to write something.
Comparing a client bump
A parity run is two sweeps over the same entry ids, one on each side of the
change, compared by traced (method, path-template) pairs.
# Provisioned ids for the project you are sweeping. Capture, then eval — a
# failed provisioning run must not be swallowed.
out=$(node scripts/examples-provision.mjs) && eval "$out"
# Baseline: the pinned client, from a checkout of the base branch.
node harness/run.mjs sweep --client baseline --lang py --llm mock --ids "$IDS"
# Candidate: the same entries with the candidate client swapped in.
COMPOSIO_CLIENT_WHEEL=/abs/path/composio_client-<version>-py3-none-any.whl \
node harness/run.mjs sweep --client candidate --lang py --llm mock --ids "$IDS"
node harness/parity.mjs <baseline-run-dir> <candidate-run-dir>
Pass --ids explicitly rather than relying on the default selection, so both
sides run the same set even when the two checkouts disagree about the manifest.
The candidate client comes from a local artifact, not a version spec:
COMPOSIO_CLIENT_TARBALL for TypeScript, COMPOSIO_CLIENT_WHEEL for Python.
Fetch the Python one with pip download composio-client==<version> --no-deps.
Both swaps abort the sweep if the client the project resolves does not actually
change, and both restore the files they touched when the sweep ends.
For Python the runner also drops the project's exact composio-client== pin for
the duration of a candidate sweep. Several entries install the local ./python
project through pyWith, and uv cannot satisfy that pin and the candidate wheel
at once — without this, those entries fail to resolve and go red for a packaging
reason, quietly shrinking the comparison.
parity.mjs only compares entries green in both runs; anything red, skipped,
or missing on either side becomes parity: false with a reason rather than
failing the comparator. Read compared alongside parityGreen, and check that
the traces are non-empty — parity over two empty traces holds vacuously.
Verifying the harness itself
node harness/run.mjs selftest
Covers backend-URL handling, the candidate-swap guards, the known-good and
known-bad fixtures, both tracers, and the comparator's own accept/reject
behaviour. node harness/run.mjs neg is the complementary check on the examples:
every entry must go red under garbage credentials, so an entry that swallows its
errors cannot pass as coverage.