getcargohq__cargo-skills/cargo-diagnostics/references/run-trace.md

Run trace — explain one run end-to-end

Use this when you have (or can find) a single run UUID and need to answer "what actually happened to this record?" — a hard failure, or the more common case: status: "success" with wrong or empty output.

Field-by-field semantics for everything used here live in ../../cargo-orchestration/references/troubleshooting.md ("Debugging a workflow run"). This runbook is the ordered procedure.

0. Find the run

Every step below needs a run UUID. Work down this ladder and stop at the first rung that matches what the user actually gave you — most of the time it's a symptom and a company name, not a UUID.

run list cannot answer "the last run". cargo-ai orchestration run list requires --workflow-uuid; there is no unfiltered form, and a play's UUID is not a workflow UUID. Orchestration SQL has no such requirement, so it — not run list — is the entry point whenever you don't already know the workflow. Concluding "the run data isn't accessible" because run list refused is a wrong answer: runs is queryable with no filter at all.

"Look at the last run" / "what just ran" — no UUID, no workflow, nothing:

cargo-ai orchestration query execute \
  "SELECT uuid, workflow_uuid, record_title, status, created_at
   FROM runs
   ORDER BY created_at DESC
   LIMIT 10"

A company, domain, or record the user names ("the run for acme.com") — record_title carries the record's title, or for record-less runs the input payload, so a substring match finds it:

cargo-ai orchestration query execute \
  "SELECT uuid, workflow_uuid, record_title, status, created_at
   FROM runs
   WHERE record_title ILIKE '%acme.com%'
   ORDER BY created_at DESC
   LIMIT 10"

A play or workflow by name — resolve to a workflowUuid first, then filter. runs has no play column, so this hop is mandatory:

cargo-ai orchestration play list        # → find the play, take play.workflowUuid

cargo-ai orchestration query execute \
  "SELECT uuid, status, created_at, credits_used_count
   FROM runs
   WHERE workflow_uuid = '<play.workflowUuid>'
   ORDER BY created_at DESC
   LIMIT 20"

Play anatomy and the rest of the play surface: ../../cargo-orchestration/references/examples/plays.md.

Coming from a batch sweep — you already have exemplar UUIDs; skip ahead.

When the discovery query itself errors

Error	Cause and fix
Limit for number of columns to read exceeded. Requested: 51, maximum: 50.	You ran SELECT *. runs is wider than the 50-column read cap — name the columns you need, as every query above does.
Unknown expression identifier '<col>'	That column doesn't exist. runs has no play_uuid, no name, and no trigger-source column; the ones used here (uuid, workflow_uuid, release_uuid, batch_uuid, record_id, record_title, status, created_at, credits_used_count) are confirmed present.

Where a run was triggered from — the CLI, a scheduled play, or a click in the UI editor — is not supposed to change where it lands: all of them write to runs and are readable with run get. So a run the user can see in the UI but that none of these queries return is a real bug, not a boundary you should work around or explain away. Say so and file a report (skill § "When diagnosis dead-ends"), quoting the queries you ran.

1. Pull the trace

cargo-ai orchestration run get <run-uuid>

Read three fields, in this order:

1. run.executions[] — the node-by-node path. For each node: nodeSlug, status, nextNodeUuid, nodeChildIndex, creditsUsedCount. This tells you where execution went, including which child a branch took (nodeChildIndex 0 = matched/yes, 1 = not matched/no).
2. runContext — per-node output keyed by nodeSlug. This is the actual data downstream expressions saw as {{nodes.<slug>...}}. It is the source of truth; the title on an execution is a truncated summary, never evidence.
3. runComputedConfigs — what each node was actually called with after expression resolution. When a node received garbage, this shows the garbage.

Don't paste the raw response into the conversation — extract the two or three nodes that matter (see "Presenting" below).

2. Diagnose by symptom

Symptom	Where to look	Typical conclusion
Run error	First executions[] item with status: "error"; its runContext.<slug> entry carries the error detail	Failing node identified — match it against the error-pattern table in troubleshooting.md ("Run error recovery")
Run success, output empty	runContext.<upstreamSlug> of the node that produced the empty value	Expression path doesn't exist — commonly agent output nested under .answer ({{nodes.qualify.answer.qualified}}, not {{nodes.qualify.qualified}})
Wrong branch taken	The branch node's nodeChildIndex + the runContext of the node its condition references	Condition resolved falsy because the referenced path is missing/undefined — verify the real shape in runContext
Connector node "worked" but downstream empty	runContext.<connectorSlug>	Partial provider response; the real field names differ from the ones referenced (e.g. contact.email vs email)
One node absurdly slow	Spans timing query below	Rate-limited or retrying connector; see the batch-sizing section of troubleshooting.md

Per-node timing for the slow-node case:

cargo-ai orchestration query execute \
  "SELECT node_slug, execution_status,
          dateDiff('second', execution_started_at, execution_finished_at) AS duration_s
   FROM spans
   WHERE run_uuid = '<run-uuid>'
   ORDER BY duration_s DESC"

3. Confirm the fix on the same record

Stage → approve → deploy → re-run the exact record IDs that exposed the bug — the command sequence is in troubleshooting.md ("Re-run a single record after fixing"). Re-running paid nodes counts as a paid action: pilot gate + receipt per ../../cargo-gtm/references/cost-discipline.md.

Presenting a trace

Per ../../cargo/references/interaction.md: conclusion first, then a compact path table — one row per relevant node (nodeSlug → status → the one field that matters), then the recommended fix. Example shape:

The run "succeeded" but the branch took the no-path: the condition reads
{{nodes.qualify.qualified}}, but the agent's output is nested under .answer.

| node      | status  | evidence                                        |
|-----------|---------|--------------------------------------------------|
| qualify   | success | runContext.qualify.answer.qualified = true       |
| branch_1  | success | nodeChildIndex = 1 (no-path) — condition falsy   |

Fix: change the condition to {{nodes.qualify.answer.qualified}} and re-run
record <id> to confirm (1 record ≈ <n> credits).

For a wrong-branch or wrong-path diagnosis, add the graph with the offending node marked — the user has to see the fork to agree the run went down the wrong side of it:

cargo-ai orchestration node diagram --run-uuid <run-uuid> --highlight branch_1 --raw

Free, runs nothing, and it works for either run shape — an ad-hoc action execute run carries its own nodes, a run of a deployed tool or play carries only a releaseUuid, and --run-uuid follows whichever it has. Flags and mapping rules: ../../cargo-orchestration/references/node-diagram.md.