mirror of
https://github.com/runpod/runpod-plugins-official.git
synced 2026-09-14 18:44:38 +08:00
Justin
4e6ba01676
* docs(runpodctl): cover the v2.9.0 surface — invoke commands, --wait, runtimeStatus
runpodctl v2.9.0 added three serverless commands, a readiness flag on both
creates, and a new pod status field. The skill described none of it, and four
claims across two files were made false by the release.
Additions:
- `serverless run` / `status` / `health`: the payload is the handler payload
(wrapped as `{"input": ...}`, validated and size-checked locally against the
10 MiB /run limit), stdout is the job payload even for a FAILED job while
progress and errors go to stderr, the exit codes, and the one that matters most
for an agent: a `timeout` naming a `serverless status` command means the job is
still running — poll it, never re-invoke and buy a second job.
- `--wait` / `--wait-timeout` on `pod create` and `serverless create`, as a
decision rule (don't hand-roll a poll loop) plus the guard rails: refused with
`--ssh=false`, warns on CPU and on community-without-public-ip, and a wait that
gives up keeps the resource and names it in the error object's `id`.
- `runtimeStatus` / `runtimeStatusReason` / `uptimeSeconds` on `pod get`/`pod
list`, with the two edges that bite: `--status` filters `desiredStatus` only,
and `unknown` means "the lookup failed", not "the pod is down".
- Two evals: invoke via `serverless run` rather than a hand-built curl, and
create-and-wait rather than a hand-rolled readiness loop.
Corrections the release forced:
- "there's no first-class serverless worker-log command, so diagnosis relies on
/health worker counts" — twice, in the runpodctl skill and in
runpod-usage/reference/endpoint-workflows.md. Now points at `serverless health`
for the counts and keeps the true half (worker *logs* remain MCP-only).
- output-and-errors.md said a local wait loop timing out is `cli_error`;
`model add --wait-for-hash` reports `timeout` as of v2.9.0.
- output-and-errors.md pinned the not-ready ssh shape as exactly
`{"error":"pod not ready",…}`; v2.9.0 appends a reason, so an exact match broke
and a prefix match did not. Also records that a stopped pod no longer returns a
dead ssh_command and that empty `connections` is now `[]`.
- The router's invoke row justified MCP with "MCP has first-class job tools",
which no longer differentiates the lanes. Verdict unchanged; the reason is now
that MCP is typed and only MCP streams job output and worker logs.
New codes (`timeout`, `job_failed`, `wait_timeout`, `wait_interrupted`) and the
new `id` error field went into output-and-errors.md, which already owns the code
table, rather than starting a second table in command-reference.md.
Verified against the shipped v2.9.0 binary, not the PR diffs: flag surface and
defaults from live --help; every local payload validation and the 10 MiB body
check by running them; the --wait guard rails with a bogus key so nothing was
created; `serverless health` against a live endpoint; and the `timeout` code by
forcing a 1ms per-call deadline, which also established both levers (`TIMEOUT`
env with no RUNPOD_ prefix, or a top-level `timeout` in config.toml above any
[section] — a trap that silently ate the value in my first attempt).
* docs(runpodctl): correct the error-handling eval for the v2.9.0 codes
* docs(runpodctl): move v2.9.0 mechanics out of the SKILL.md body into reference
* docs(golden-paths): re-verify monitoring path against runpodctl v2.9.0
Golden path 15 still steered agents to raw curl for /health and /status, and
told them runpodctl "covers create/list/delete" — the same claim this branch
corrects elsewhere, left stale in the one document whose whole subject is the
observability loop. Re-ran the path against live infrastructure on the v2.9.0
binary and rewrote it from what came back.
The signals table is now first-choice/fallback rather than a single source:
serverless health and serverless status lead, the curl stays as the
zero-dependency equivalent and the thing you hand a user to copy-paste. The
worker-log half of the old claim is true and survives, stated more precisely —
there is no worker-log command in runpodctl and no worker-log path on REST v1,
which is a statement about the service you call, not about an endpoint's own
version field (a version:1 endpoint's logs read fine through the v2 route).
The live run caught a real failure rather than a clean success, which is the
better teaching case and closes the one code this branch had only source-
verified: a job walked IN_QUEUE to FAILED with a server-side retry, /status
reported "job timed out after 1 retries", and the container log showed the
worker had passed all seven fitness checks and rejected the payload. The path
now says not to trust the error string alone, and records that a retry resets
delayTime rather than lengthening the queue.
Also corrected while re-reading the observed output: the v2 worker-list summary
has no ready key (that exists only on /health), so the two views disagree by
design; serverless get --include-workers returns raw worker records carrying
RUNPOD_AI_API_KEY, RUNPOD_ENDPOINT_SECRET and any handler tokens in env, which
is worth knowing before pasting it into a ticket.
The same overreach appeared in runpod-usage/reference/gotchas.md and
golden-paths/03-whisper-endpoint/README.md; both corrected, keeping the true
half. output-and-errors.md now names timeout as the one documented exception to
its own branch-on-code-never-the-message rule, so the reference and the
error-handling eval no longer read as disagreeing — a code-only handler that
never retries timeout stays correct.
* docs: strip leaked tool-call markup from golden path 19
19-three-region-same-file.md ends with literal </content> and </invoke> tags —
authoring-tool markup that was committed as document text and has been sitting
on main. It renders as stray characters and is the kind of thing a reader takes
for a formatting convention rather than a mistake.
Unrelated to the v2.9.0 work; separated so it can be reverted or cherry-picked
on its own.
* docs(readme): surface the golden paths and prefer the HF model cache
The golden paths were a paragraph buried under the development loop, described
as "acceptance scenarios, not installed skills" — accurate, and precisely the
framing that makes a reader skip them. They are the most expensive artifact in
this repo (someone ran each one on real infrastructure and pasted back what
actually happened, including the parts that went wrong) and the least
advertised. Promoted to their own section, led with the instruction to look for
one before improvising, and given a phrasing a user can hand straight to an
agent.
Added a model-delivery section pointing at the host-side HF cache. The facts
were already in runpodctl/reference/model-caching.md; what was missing was
anything at README level, so the default a reader carries away came from
storage.md's "when in doubt, choose the network volume" — right for datasets and
checkpoints, wrong for a HuggingFace model on GPU serverless. The cache is
host-side, so cold starts drop to seconds and download time is not billed, and
it avoids the constraint that actually hurts: a network volume is pinned to one
data center, which narrows the GPU availability you can schedule against.
Scoped that storage.md default rather than weakening it, since it is correct for
everything except model weights, and pointed it at the model-delivery section
below. Its "Getting a model to the worker" heading also said "Three ways" over a
four-row table — the Model Repository row was added without updating the count —
so that is corrected and given the same routing rule of thumb the runpodctl
reference already carried.
Both the caching guidance and its constraints (GPU-only, runpodctl >= v2.4.0,
HuggingFace-hosted models) are as documented in model-caching.md and verified
against the v2.9.0 binary's flag surface.
* fix(runpodctl): pod/serverless logs are first-class in v2.10.0
runpodctl v2.10.0 (2026-08-19) added `pod logs` and `serverless logs` with
`--follow`/`--tail`/`--since`/`--source`/`--worker`, emitting json lines. That
falsifies eleven assertions across five files that worker logs were reachable
only through MCP or a hand-built SSE read — most of which this branch had just
added while documenting v2.9.0.
The worst of them was an eval that graded an agent *correct* for answering
"worker logs → NOT runpodctl … the MCP lane is the only one that can read
them". A false negative in an eval trains the wrong behavior in rather than
merely misinforming a reader, so that is rewritten to expect
`serverless logs`, with the SSE path demoted to the pre-v2.10.0 fallback.
Golden path 15's section 3 gains the CLI as Route A ahead of the raw v2 REST
stream (Route B) and MCP (Route C); its four-signals table, triage checklist,
gotchas and folded-back facts follow. `stream-job` is now the only genuinely
MCP-only capability, and the router's capability matrix says so — updated in
all three SKILL.md files per AGENTS.md rule 4.
Also fixes a pre-existing gotcha in 03-whisper-endpoint/variant-a-hub.md that
asked for `runpodctl serverless logs <endpoint-id>` by name, which now exists.
Verified against the shipped v2.10.0 binary, not the release notes.
* fix(runpodctl): correct --output and exec behavior for v2.10.0
Both changed in runpodctl #319 and both inverted what the reference said.
`--output` validation: the doc claimed the flag is matched case-sensitively and
that an unrecognized value silently falls back to JSON, so `--output=table`
*and* `--output=YAML` both return JSON. On v2.10.0 exactly the opposite holds —
`table` exits 1 with `{"code":"usage_error"}` before any API call, and `YAML`
returns real YAML because matching is now case-insensitive. Recorded with the
v2.9.0 behavior kept alongside it, since a handler has to know which binary it
is on. (Upstream's own note is worth keeping: that silent fallback is how a
table format the CLI never had ended up documented in the first place.)
`exec` exit code: it printed errors as plaintext and still exited **0**, which
the plaintext-gaps table and the "don't trust the exit code" line both relied
on. It now emits coded JSON and a non-zero exit, leaving `project` as the only
surface where the exit code cannot be trusted. The error-handling eval asserted
the old grouping, so it asserted something false.
These lines sat four rows above this branch's own edits to the same file.
* docs(runpodctl): make command-reference stop restating --help
The file opened by admitting `runpodctl <resource> <action> --help` is
authoritative and then spent 140 lines restating it — every Hub/template/volume/
registry/info/utility flag, hand-copied and silently rotting. Nothing in this
repo can keep that current, and a wrong flag list is worse than no flag list.
Kept only what `--help` cannot tell you: what "ready" is defined as for
`--wait`, which pod status field to trust, the `serverless run` payload/exit-code
contract, `pod logs`/`serverless logs` termination semantics, the `send`/`receive`
one-time-code flow, and the `--include-workers` secret leak. Everything
mechanical is now a pointer at the binary.
Two things this surfaces that were missing entirely:
- `pod get` returns `networkVolumeId` whenever a volume is attached as of
v2.10.0 (runpodctl #322). On earlier binaries both it and `networkVolume` were
dropped on deserialization and read back `null`, so a pre-v2.10.0 binary
cannot answer "does this pod have a volume".
- `serverless update` sends zero values as of v2.10.0 (#313). Before that
`--workers-min 0` was dropped by `omitempty`, which means the SKILL.md
instruction to set a dev endpoint back to `--workers-min 0` was a silent
no-op and the endpoint kept billing. That is a cost-safety claim with no
version floor, so it gets one in both places.
Also notes the two `--help` traps a reader will hit: cobra does not traverse
aliases (`pod remove --help` says unknown command though `pod remove <id>`
works — use `runpodctl help pod remove`), and `get pod`/`get cloud` are hidden
from `--help` but still live.
165 lines, down from 222, and none of them go stale on a flag change.
* feat(hooks): gate "the CLI cannot do X" claims against the real CLI surface
Absence claims are the one kind that rots silently. The
`--help`-is-authoritative rule does not protect them — there is no flag to look
up, so nobody reverifies — and the sentence reads as authoritative right up
until the release that adds the command. It has now happened twice in the same
skill: v2.9.0 added `serverless health`, falsifying "diagnosis relies on
/health worker counts"; v2.10.0 added `serverless logs`/`pod logs`, falsifying
eleven assertions including an eval that graded the false answer as correct.
So gate them, following the precedent rule 8 already sets for runpod-migrate's
Class-3 table: find every absence claim in the skills, extract the command it
names, and fail if that command exists in a vendored surface snapshot.
- `testdata/runpodctl/command-surface.json` — 66 commands walked out of the
v2.10.0 binary, plus the deprecated paths cobra hides from `--help`.
- `hooks/gen_cli_surface.py` — regenerates it against any binary.
- `hooks/check_cli_absence_claims.py` — the gate. Blocking in validate.yml
against the snapshot; `--live` in spec-drift.yml downloads the latest release
weekly, non-blocking, which is what would have caught this PR on the day.
Deliberately narrow, because a noisy gate gets disabled: a claim about a
missing *flag* is exempt (`--help` covers those), so is a negated or
version-floored one ("no longer MCP-only", "needs >= v2.10.0"), and genuinely
true absences live in an `ALLOW` list that requires a reason. Verified both
ways — reintroducing this branch's own "runpodctl still has no serverless logs
command" fails the check; the 16 surviving absence claims pass.
New rule 9 in AGENTS.md: prefer a positive claim or a version floor over an
open-ended "cannot", and never put a false negative in an eval.
* docs(runpod): put "the tools are the source of truth" in the router
The deference posture existed only inside two lanes — runpodctl's SKILL.md
("live --help is authoritative … the binary in front of you wins") and
runpod-mcp's ("the tool list above is a map, not a contract"). The router, which
is what an agent reads first, said nothing about it, and flash and
companion-clis had no equivalent at all.
So state it once at the entrypoint, with the specific thing to ask per lane:
`--help` and a deliberate wrong invocation for runpodctl, the client's tool list
for MCP, the live openapi.json for REST v2. Plus the rule this PR exists to
learn: never assert a tool cannot do something without checking, and if a limit
does hold, name the version it holds for instead of saying "cannot".
Also fixes a pointer this branch broke: SKILL.md still sent readers to
command-reference.md for the "full flag menu" after the previous commit removed
the flag lists from it. It now describes what that file actually holds — the
part --help cannot answer.
* docs(runpod): add the absence-claim eval and finish the source-of-truth pass
Two gaps left by the previous commit.
**Rule 6.** That commit changed router behavior — a new rule about how to treat
capability claims — and shipped no eval, which rule 6 binds. Adds
`no-unchecked-absence-claims.eval.md`, prompting for exactly the question this PR
got wrong: "can I read serverless worker logs from the CLI, or do I need MCP?"
It asserts the agent answers yes and names `serverless logs`, does not answer
from the pre-v2.10.0 limitation, and expresses any version caveat as a floor
rather than a denial. This is the one eval whose failure mode is a model
confidently reciting an older copy of these very docs, so it is worth having
even though the underlying fact is now stated in four places.
**flash and companion-clis.** The router's new table points at `flash --help`
and each companion CLI's `--help`, but neither lane's own SKILL.md mentioned
`--help` at all — zero occurrences in both — so an agent entering the lane
directly never got the pointer. Both now say their tool is authoritative for the
command surface, with the version check to run before concluding something is
missing (`pip show runpod-flash` for flash).
* fix(runpod): make golden paths step 0 of routing instead of an appendix
The examples were structurally unreachable. The router put "How to route" at
line 80, told the agent to "read the matching skill's SKILL.md next", and then
listed the worked examples 67 lines further down — so an agent that followed the
instructions routed into a lane and handed off before ever seeing them. Three of
six lanes (flash, runpod-usage, companion-clis) never mentioned golden paths at
all, and the frontmatter description — the one part always in context — did not
either. Nothing pulled toward the examples, so they only got used when a user
asked for them by name.
Reordering rather than nagging:
- **Step 0 of "How to route"**, ahead of lane selection, with a trigger stated in
observable terms: more than one resource or lane, provisions something
billable, an ask shaped like "get X running", or worth a multi-step plan. And
an explicit skip for a single read or single CRUD call, so "list my pods" does
not pay for a file read.
- **A matching path outranks the lane table** — it was verified end to end on a
real account, so a disagreement is a bug in the table, not in the path.
- The frontmatter description now advertises the 25 examples, and the handoff
sentence sends multi-step tasks to them first.
- Every lane links back, phrased for what that lane gets wrong without them:
MCP (tool calls are easy to issue in the wrong order), flash, runpod-usage
("this skill explains; the paths demonstrate"), companion-clis (each CLI is one
step inside a larger job).
- Partial matches count. The nearest path's ordering and gotchas usually transfer
even when the model or region does not, which is most of the value.
Fixes drift the new rule forbids: **path 21 (storage tiers) had no router row**,
so routing could not reach it even though golden-paths/README.md lists it.
Rule 5 rewritten to bind all of this and merged with the old duplicate
"Golden paths" rule (there were two, both with that title); rules renumbered to
11 and the progressive-disclosure cross-reference repointed.
`consult-golden-path-first.eval.md` asserts **both** directions — a one-line stop
must not open a path, and a host-cached HF deploy must find path 20 specifically
before proposing commands. Widening the trigger to "always read an example
first" breaks that eval on purpose.
* fix: correct review findings — count claim, description length, #322 sourcing, gate holes
Self-review of the branch found four real defects, three of them mine.
**"25 verified" examples was wrong twice.** I put it in the router's frontmatter
description, having read it off the highest filename (`25-bake-vs-mount`). There
are **24** paths (there is no 24), and 23 are live-verified while path 21 is
`documented (launch only)`. Now "two dozen live-verified", which matches
golden-paths/README.md's own Status column.
**The description ran to 3 sentences**, over rule 2's 1-2. It was already 3 before
this branch and I made it longer; condensed to 2 and it now names runpod-migrate,
which it had been omitting since that lane shipped.
**#322 was stated more strongly than the evidence.** I wrote that `pod get`
returns `networkVolumeId` "with or without `--include-network-volume`" as flat
fact. That comes from the PR author's hedged aside ("I believe this is the
desired behaviour"), and only the flagged path has a test. Reworded to attribute
it and to tell the reader to pass the flag rather than rely on it. Re-checked the
neighbouring `uptimeSeconds` claim while there — that one holds, #315 fixed the
`runtime.uptimeInSeconds` mapping and shipped in v2.9.0.
**The gate had two holes.** Probing it with eleven phrasings of the same false
claim, two got through:
- `CMD_IN_TEXT` was case-sensitive, so a claim opening a sentence with a
capitalized resource — "Serverless logs are only available through MCP" —
matched an absence pattern but resolved no command name and passed.
- "you must use MCP for X" / "X requires the MCP server" asserts absence with no
absence wording in it at all, and no pattern covered that form.
Both fixed, plus command names now resolve against the whole line rather than the
matched span (the command a claim is about frequently sits outside it). Guarded
`FLAG_CLAIM` against the resulting false positive on "lacks `templateId`
support", which is a field claim about MCP.
A gate whose patterns can silently stop matching is worth little, so
`--self-test` is now part of the check and runs first in CI: 11 phrasings that
must fail, 5 legitimate ones that must not. Every MUST_CATCH entry is a phrasing
that actually evaded an earlier version of these regexes. Verified the self-test
fails when the case-sensitivity hole is reintroduced.
* docs: stop describing evals as enforcement — nothing runs them
No runner exists in this repo and no workflow reads `evals/*.eval.md`, so
"an eval asserts X" is a claim about coverage that does not exist. AGENTS.md
called them "regression scenarios", which invites exactly that reading, and I
compounded it in rule 5 by saying widening the routing trigger "breaks that eval
on purpose" — it cannot break anything.
Both spots now say plainly that evals are specifications to hand-check a model
against and to re-read when behavior changes, not a gate. This matters for
reading the rest of this PR: the absence-claim check is enforced in CI, the
golden-path routing change is not.
* test(runpod): validate the golden-path routing change against a control
Ran the change instead of asserting it. Same prompts against origin/main and
this branch, agents reporting which files they opened and in what order.
**The router-entry scenario shows no effect.** 4/4 agents opened golden path 20
in both arms. The router's "Want to…" table already has a near-verbatim row for
that task and agents read the file end to end, so where the section sits on the
page never mattered — and origin/main already said "open its golden path first".
Scenario A of this eval is therefore a weak test, and I have said so in the file
rather than leaving it looking like coverage.
**The lane-entry scenario is where the change does the work**, and it is also the
realistic one: an agent already in a lane does not re-read the router. Entering
at runpod-mcp/SKILL.md with a task that has no verbatim row (3-DC HA endpoint):
| | opened a path | plan source | correct architecture |
| without the pointer | 0 / 2 | derived | 0 / 2 |
| with it | 2 / 2 | golden-path | 2 / 2 |
Both treatment runs cited the added sentence by name and found path 19, a closer
match than the path 10 this eval originally predicted.
The control failures are worth recording precisely, because they are not laziness
— origin/main's runpod-mcp SKILL.md mentions golden paths exactly once, in
passing, inside an unrelated endpointType bullet below the tool surface, so the
agent correctly concluded nothing pointed it anywhere. Deprived of the example
both runs then invented a limitation to justify the plan they derived: "you
cannot attach one network volume to a multi-DC endpoint" and "network volumes are
not a mechanism for multi-DC redundancy of weights". Path 10 live-verified the
opposite on 2026-07-10 — one endpoint, --network-volume-ids vol-ro,vol-cz, both
attached, workers serving from both DCs.
That is the same defect class as the docs bug this PR's CI gate exists to stop,
arriving from the other direction: the gate stops the repo asserting a false
absence, and surfacing the examples stops a model inventing one. Adds scenario B
so the eval tests the case that actually differs.