2 Commits

Author	SHA1	Message	Date
Alberto Schiabel	d787107dfa	fix(examples): declare provider deps and drop magic-flow-demo (#4221) ... Follow-up to the live-sweep findings in #4135. Independent of it — this targets `next` directly. ## Python entries could never resolve their provider imports `harness/run.mjs` runs each Python entry as `uv run --project python [--with <dep>]... python <file>`. The base environment is the `python` project only, so the provider packages are present just when an entry names them in `pyWith`. Six entries import a provider and declared none, and have failed with `ModuleNotFoundError` every time the sweep has run since it landed on 2026-08-14: | entry | imports | had | | --- | --- | --- | | `py/tool_router/files` | `composio_openai` | no `pyWith` | | `py/tool_router/preload` | `composio_openai_agents` | `openai-agents>=0.19` only | | `py/tool_router/direct_tools_preset` | `composio_openai_agents` | `openai-agents>=0.19` only | | `py/tool_router/tools` | `composio_openai_agents` | `openai-agents>=0.19` only | | `py/tool_router/tool_router_mcp` | `composio_openai_agents` | `openai-agents>=0.19` only | | `py/custom_tools_agent_test` | `composio_openai_agents` | `openai-agents>=0.19` only | Those entries declared the upstream `openai-agents` SDK but not the Composio provider built on it. ### Why local paths, not package names A bare `--with composio-openai-agents` resolves from **PyPI**, so the sweep would exercise the published provider rather than this repo's — and it drags the published `composio` in with it. Declaring `./python/providers/<name>` plus `./python` keeps both local. Verified in the resulting overlay: ``` composio : file:///…/composio2/python composio-openai : file:///…/composio2/python/providers/openai ``` `py/tool_router/claude_agent` already had `./python` but named its provider, so it was testing the **published** `composio-claude-agent-sdk` against the repo's `composio` (confirmed: that dist has no `direct_url.json`). Pointed at the local path too, so every entry now tests repo code under one convention. ## Deleting `ts/connected-accounts/magic-flow-demo` - Not substantively touched since **2025-06-20** — ~14 months. The only later commits are a model-version bump and its same-day revert, which net to zero. - It calls an endpoint the backend has retired. The sweep gets `400`: *"Creating connections on this endpoint for Composio-managed OAuth auth configs is no longer supported. Use POST /api/v3/connected_accounts/link instead."* - It could not pass unattended even if that were fixed: `waitForConnection()` blocks until a human completes a browser OAuth grant. Removed the source, its manifest entry, and its README bullet. Rewriting it onto `/connected_accounts/link` would be a genuinely new example rather than a repair, so it seemed better to drop it than to carry a broken one. ## Verified against the live staging backend Not just locally — these entries were **executed** against `staging-backend.composio.dev` (`llm=mock`, zero model spend): ``` provisioned state: complete sweep 20260823224239-lk9413-baseline-mock: 5 entries ✓ py/tool_router/toolkits (5.8s) ✓ py/tool_router/preload (11.1s) ✓ py/tool_router/direct_tools_preset (11.2s) ✓ py/tool_router/files (12.3s) ✓ py/auth_configs (3.9s) sweep: 5 green / 0 red / 0 skipped ``` All three previously-broken entries now pass end to end; `toolkits` and `auth_configs` ran as unchanged controls. As far as I can tell from the workflow history, this is the first time any of these examples has completed green — the nightly has failed at `Verify provisioned project state` on every run since it was introduced. The run needed a branch carrying both these fixes and #4135's scoped provisioning, since `next`'s provisioning tries to create a `googledrive` auth config and dies before sweeping anything. That branch was throwaway and is deleted. Other checks: `node harness/run.mjs selftest` passes, `pnpm run test:examples` validates 20 packages, the connected-accounts package typechecks, and all 92 remaining manifest entries point at files that exist. ### Still unproven The other three entries fixed here — `tools`, `tool_router_mcp`, `custom_tools_agent_test` — need github/gmail connected accounts, which staging does not yet have. They get past their imports but cannot be run end to end until those grants exist. `py/modifiers` is `tier=X`, excluded by design. ## Related `py/tool_router/tool_router_mcp` also exits **0** when `COMPOSIO_API_KEY` is unset, so the harness would score it green while it did nothing. Fixed separately in #4222.	2026-08-24 00:47:10 +02:00
Alberto Schiabel	71ec523d25	feat(examples): live-sweep harness, manifest, and scheduled workflow (#4117) ... This PR: - builds on merged PR #4107 - adds `examples-manifest.json`: 93 runnable entrypoints across both SDKs with tiers (unattended / provisioned / bounded / excluded), per-entry timeouts, env contracts, and labeled readiness markers for OAuth flows - adds `harness/run.mjs`: sweeps entries against staging with per-entry Composio traces, negative controls (garbage credentials must fail), a `selftest`, and a baseline/candidate client swap for parity runs - rejects every non-staging `COMPOSIO_BASE_URL` before the harness or provisioning script can send credentials - adds `--llm mock`: LLM examples run against a pinned `@copilotkit/aimock` server with scripted tool-call fixtures (`harness/llm-mock/`) — full sweeps spend zero model tokens while Composio tool execution stays live on staging; 46/48 LLM entries pass mocked, and the two `@openai/agents` entries are marked `llmMock:false` - adds `examples-live.yml`: nightly gc-provision → mock sweep → four-entry live LLM canary; dispatchable per language, client, entry subset, and LLM mode, and bound to the `staging` GitHub environment - accepts separate TypeScript tarball and Python wheel inputs for candidate sweeps, restores exact pre-run workspace files on failure, and rejects empty or partial parity comparisons The configured `COMPOSIO_API_KEY` must belong to a dedicated, disposable staging project. The workflow becomes dispatchable after it exists on the default branch. ## Context Examples historically went stale because nothing executed them. PR #4107 makes them runnable and provisionable; this PR adds the machinery that runs them on a schedule and validates the Stainless→self-managed client swap via baseline/candidate parity sweeps. ```mermaid graph TD M[examples-manifest.json] --> R[harness/run.mjs sweep] P[examples-provision.mjs --gc] -->|COMPOSIO_EXAMPLES_* ids| R R -->|tier 1-3 entries| E[TS + Python examples] A[aimock fixtures] -->|--llm mock| E E -->|live tool execution + traces| C[Composio staging] R --> N[nightly live canary: 4 entries] ``` ## Verification - `env -u COMPOSIO_BASE_URL node harness/run.mjs selftest` - `pnpm test:examples` against a clean tracked snapshot - manifest integrity: 93 unique entry IDs and 93 existing files - JavaScript, Python, JSON, YAML, and embedded Bash syntax checks - `pnpm validate:changesets`	2026-08-14 15:47:47 +02:00