getcargohq__cargo-skills/cargo-diagnostics/references/batch-error-sweep.md

Batch error sweep — group failures by root cause

Use this when many runs are involved and you don't yet know where to look: a batch reports errors, a play's error rate spiked, or "some records didn't come through". The output of a sweep is a small table of failure groups with an exemplar run UUID each — not a list of every failed run.

SQL syntax, table columns, and query caps: ../../cargo-orchestration/references/examples/queries.md. All queries below are read-only and workspace-scoped automatically.

1. Size the problem

# For one batch
cargo-ai orchestration query execute \
  "SELECT status, count() FROM runs WHERE batch_uuid = '<batch-uuid>' GROUP BY status"

# For a play/workflow over time
cargo-ai orchestration query execute \
  "SELECT countIf(status='error') / count() AS error_rate, count() AS total
   FROM runs
   WHERE workflow_uuid = '<workflow-uuid>' AND created_at > now() - INTERVAL 7 DAY"

Calibration: error rates under ~5% on connector-heavy workflows are often provider coverage, not defects (see the over-provision rule in cost-discipline.md). A spike above that, or errors on native nodes, is worth the sweep.

2. Find where failures concentrate

# Which node fails most
cargo-ai orchestration query execute \
  "SELECT node_slug, count() AS failures
   FROM spans
   WHERE execution_status='error' AND execution_started_at > now() - INTERVAL 1 DAY
   GROUP BY node_slug
   ORDER BY failures DESC"

Scope with batch_uuid/workflow_uuid predicates when you have them. Two shapes to distinguish:

• Concentrated (one node owns most failures) → a config, credential, or expression defect at that node. Proceed to step 3 with that node.
• Spread across connector nodes, clustered in time → third-party rate limiting. Confirm with the "signs you are being rate-limited" checklist in troubleshooting.md; the fix is retry config + smaller sub-batches, not per-run debugging.

3. Pick exemplars and read the actual errors

cargo-ai orchestration query execute \
  "SELECT uuid, created_at
   FROM runs
   WHERE batch_uuid = '<batch-uuid>' AND status='error'
   ORDER BY created_at ASC
   LIMIT 3"

Take 2–3 exemplars per failure group and trace each with run-trace.md — the error detail lives in run get's runContext, not in the SQL tables. Failures with the same node + same error pattern are one group; resist tracing every run.

4. Decide: fix, re-run, or report

Root cause shape	Action
Expression/branch defect (same wrong output every time)	Fix the node, re-test on exemplar record IDs, then re-run only the failed records: run download --statuses error → fix → batch create --data '{"kind":"recordIds",...}' (sequence in troubleshooting.md, "Run error recovery")
Expired connector credentials	Re-authenticate the connector (connection connector update or the Cargo app), then re-run failed records
Provider rate limiting	Retry config + sequential sub-batches; don't chase individual rows
Provider coverage (no email exists, company not found)	Expected loss — drop the rows per the over-provision rule; do not re-run them through more providers
CLI/platform behavior contradicts the docs	File it — this is exactly what the report channel is for: cargo-ai workspaceManagement report create --title "..." --description "<commands, errorMessage verbatim, expected vs actual, UUIDs>"

Any re-run of paid nodes is a paid action: pilot gate + receipt per cost-discipline.md.

Presenting a sweep

Per ../../cargo/references/interaction.md: conclusion first ("one root cause explains 18 of 20 failures"), then the groups table, then the recommended action per group. Example shape:

Batch <uuid>: 20 of 250 runs errored. Two groups:

| group | runs | node        | root cause                            | action                    |
|-------|------|-------------|----------------------------------------|---------------------------|
| 1     | 18   | find_email  | connector token expired (401 verbatim) | re-auth, re-run 18 records |
| 2     | 2    | enrich_co   | provider has no data for these domains | drop rows (coverage)      |

Re-running group 1 ≈ <n> credits. Proceed?