llamaindex, agno, and claude-sdk-python emit AG-UI REASONING_MESSAGE_*
events but pinned @ag-ui/client ^0.0.43, whose @ag-ui/core discriminated
union lacks the REASONING_MESSAGE_* variants — the frontend threw
invalid_union_discriminator and the reasoning demo broke. Pin all three
to exact 0.0.48 (matching the claude-sdk-typescript fix in #5323),
regenerate their lockfiles so @ag-ui/core resolves to 0.0.48 with zero
0.0.43 nodes, and ratchet the validate-pins drift baseline down from 60
to 57 to reflect the now-exact pins. Verified locally on the built-image
showcase rig: D5 green and the D6 reasoning-display probe passes for all
three backends with zero invalid_union_discriminator.
with_fast_sleeper mutated the process-global RETRY_DELAY_SEC via
remove_const/const_set. The clean seam (pin_and_verify(sleeper:)) is not
reachable from cmd.run without changing bin/railway production logic, so keep
the swap but make it bulletproof against run-order state leakage: capture the
original before mutating, track whether the swap happened so a mid-setup
failure never leaves the const perturbed, restore in ensure even on raise,
and silence the "already initialized constant" warning locally.
Also reword the header/test comments so they no longer hardcode the literal
"5 public hosts" count, referring to the EXPECTED_DOMAINS[PRODUCTION_ENV_ID]
set instead so the prose can't drift from the SSOT the fixture derives from.
install_fleet_fixture derived one prod service per SSOT public host and then
unconditionally appended make_prod_service(target). When the target already
owns a public prod host (e.g. "docs" owns docs.copilotkit.ai) this listed the
same prod service twice — one domain-bearing, one with custom_domains:[] —
a malformed fleet shape that contradicts the helper's "no public domain of
its own" contract and was only masked by find_service first-match + .uniq.
Guard the append so the bare target is added only when it is NOT already a
derived domain owner. Add a red-green test asserting the derived prod
snapshot contains no duplicate service names or service_ids and that the
target still appears exactly once.
PinCommand#run resolves the service id via RollbackCommand#resolve_service_id
BEFORE the --dry-run early-return, which issues a real GraphQL call and
die!s (exit) on a tokenless CI runner — aborting the whole minitest
process before the summary. Stub resolve_service_id at the class level so
the colon-split assertions run hermetically. Verified green under an
unset RAILWAY_TOKEN + isolated HOME.
install_fleet_fixture hardcoded the 5 public prod hosts and the tests
hardcoded service names, so any change to the SSOT
(railway-envs.generated.json) would break these tests with confusing
phantom-domain WARN / parity die! failures — a brittle gate around the
promote logic rather than the logic itself.
Derive the fixture's domain-bearing prod services from the same constant
the prod code reads (Railway::EXPECTED_DOMAINS[PRODUCTION_ENV_ID]),
mapping each public host back to its owning SSOT service, and select the
target/sibling from Railway::STAGING_SERVICES instead of bare strings.
Documents the derive-from-SSOT invariant. Behavior is identical for the
current SSOT — all four existing tests stay green.
PinCommand#run and PromoteCommand#image_shape stripped the tag with a
first-colon split(":", 2), which cuts at the registry PORT colon and
corrupts a host:PORT/org/img:tag ref (e.g. localhost:5000/img:latest →
base "localhost"). Switch both call sites to the existing last-colon
String#rsplit_colon helper (already used by GHCR#parse_image_ref) so the
tag-stripping is consistent and port-safe. Canonical ghcr.io/...:tag refs
(no port) are unaffected — this is a latent correctness fix.
Adds red-green unit coverage proving a host:PORT/img:tag ref now parses
and pins correctly, and that an empty-tag port ref is no longer
misclassified as :tag.
The `bin/railway promote` help banner advertised preflight checks and
effects that were never implemented in any commit (verified via full
git history on showcase/bin/railway — all phrases trace to the original
56e85dfd80 add, never as working logic):
- MOVES "autoUpdate=disabled flag": execute_promotion only pins the prod
image digest + redeploys (serviceInstanceUpdate sets source.image only);
auto_updates_disabled is a vestigial snapshot field hardcoded to nil and
never mutated.
- VERIFY-REFUSE "PB superuser auth" / "PB collection parity": no such
checks exist; POCKETBASE_SUPERUSER_* are only key-presence entries in
CRITICAL_ENV_KEYS, and PocketBase auth/collection logic lives entirely in
the harness, never reached by promote.
- VERIFY-REFUSE "cross-env URL leak scan": never implemented; snapshots
capture env-key NAMES only (values are never compared), making such a
scan impossible by construction.
- WARN "sealed-var heuristics": isSealed is fetched in the env-vars query
but never read; no heuristic consumes it.
Rewrite MOVES/VERIFY-REFUSE/WARN/IGNORE to list only the real preflight
(P1 GHCR digest, P2 staging deployment + race, P3 staging live-green, P6
startCommand/healthcheckPath/image-shape parity, service-set parity,
critical env-key parity; P6 region/replicas/restartPolicy/env-key-set and
expected-prod-domains WARNs) and the real effect (pin prod image to the
staging digest + redeploy). Doc-text-only; no logic changed.
Renumber test_snapshot_ivar_lint.rb ALLOWED_LINES by +3 to track the
line shift from the (longer) banner — the lint is line-number-pinned by
design and instructs hand-renumbering on any shift above its region.
## Summary
Closes the fleet-wide reasoning-emission gap: showcase reasoning demo
cells now emit real AG-UI `REASONING_MESSAGE_*` (role `reasoning`) from
each backend's native reasoning channel, so the d5/d6 reasoning cells
render the thinking block.
- **Real reasoning fixes (5 backends):** claude-sdk-python (Anthropic
native `thinking_delta`, multi-block + redacted-thinking history replay
for the tool loop), agno (`RunContentEvent.reasoning_content` tee),
built-in-agent (chat-completions `reasoning_content` adapter),
llamaindex (OpenAI **Responses API** + reasoning model, matching the
langgraph-python gold standard), claude-sdk-typescript (`role:
"reasoning"` fix + `@ag-ui/client` ^0.0.48).
- **Documented genuine SDK limitations (3 backends):** ag2,
crewai-crews, spring-ai cannot surface a model reasoning channel
(bridge/SDK has no reasoning event) — documented in each
`PARITY_NOTES.md`, not faked.
- **Probe coverage restored:** agno + llamaindex reasoning demo ids
renamed to `reasoning-custom`/`reasoning-default` (+ re-added missing
manifest demo blocks) so the d5 reasoning-display probe fires for them.
- Verified via aimock d5/d6 replay (native channel confirmed, not the
inline-tag fallback). langgraph-python is the parity gold standard.
## Verification
- Per-backend AG-UI event-level RED→GREEN (`REASONING_MESSAGE_START`
0→N) under aimock.
- claude-sdk-python multi-block lifecycle + thinking-history signature
replay verified via captured iteration-2 Anthropic request.
- 7-agent CR with two fix rounds + three confirmation rounds → converged
to zero blocking findings.
## Follow-ups (not in this PR)
- Fleet-wide reasoning-id rename for the remaining backends
(ag2/crewai-crews/spring-ai/langgraph-fastapi/langroid/mastra/strands)
so their reasoning-display cells get probed.
- aimock hardening filed upstream: CopilotKit/aimock#253 (validate
Anthropic extended-thinking request invariants) and #254
(model-capability-aware reasoning emission).
- Minor robustness: `id(block)`→monotonic counter for reasoning message
ids; symmetric unparseable-reasoning warning on the chat-completions
transport; reuse Anthropic SDK `*BlockParam` types.
## Test plan
- [ ] CI green
- [ ] Post-merge: showcase `:latest` rebuild + staging redeploy, then
confirm the d5/d6 reasoning cells (chain + reasoning-display) render
green on the dashboard for the 5 fixed backends; the 3
documented-limitation backends remain red-but-documented.
check_service_set_parity scoped only the staging-only arm to the
single-service target; the prod-only arm was still computed over the
full fleet, so any prod-only service (e.g. a deprecated harness-legacy)
REFUSEd every unrelated single-service promote — the exact mirror of
the bug #5324 fixed. Scope both arms to the target for single-service
promotes; full-fleet promotes (target nil) keep both arms at full
strictness.
Tests: add prod-only tolerance red-green test, strengthen the
target-absent test to assert target-scoping (unrelated staging-only
sibling ignored), rewrite the stale snapshot-narrowing comments to
describe the real fleet_*/& [target] contract, drop the dead
FLEET_PUBLIC_PROD_HOSTS constant, and renumber the ivar-lint allowlist
for the one-line shift in bin/railway.
A single-service `promote <svc>` ran check_service_set_parity over the
FULL staging vs FULL prod fleet and REFUSEd whenever staging carried any
service prod lacks. The live staging fleet legitimately contains 13
staging-only services — harness-workers (SSOT-modeled) and 12 starter-*
demos — so an otherwise-clean single-service promote (e.g. docs) is
blocked with `REFUSE: services in staging not in prod`.
Scope the "staging not in prod" REFUSE to the promote TARGET when a
single-service promote is in effect (intersect the staging-only set with
[target]). The target-absent-from-prod footgun still REFUSEs (target is
in the intersection), the "prod not in staging" arm is unchanged, and
full-fleet promotes (no --service) retain full strictness.
Complements #5322.
When a promote fails, the succeeded-service set is empty, so verify-prod
hits its skip branch (`exit 0`). The GitHub job result is therefore
`success`, and the notify step rendered `verify-prod=success` in the
#oss-alerts Slack message — a misleading green, since prod was never
probed.
verify-prod now exports a `status` output: `success` after a real probe
passes, `skipped` on the empty-CSV skip. notify reads that output (via
the new bats-tested verify-prod-display.sh) instead of the raw job
result, so the Slack line accurately reads `verify-prod=skipped` vs
`success` vs `failure`. A genuine probe failure / contract violation
exits non-zero (job result `failure`, status never written), and the
display falls back to the job result. Slack formatting is unchanged.
Extracts the display mapping into showcase/scripts/verify-prod-display.sh
(mirroring promote-fleet.sh) with red-green bats coverage, and adds it to
the showcase_validate.yml shellcheck step.
build_snapshot enumerates every project service and queries each one's
serviceInstance. The only guard was `next if inst.nil?` — it handled a
NULL result but not a THROWN `GraphQL: ServiceInstance not found` error
(a half-deleted service that still appears in the project service list
but has no instance in the env). That error bubbled to Railway.run's
top-level `rescue GraphQL::Error` and aborted the ENTIRE promote with an
opaque exit 2 before any preflight/divergence logic ran (run 27144525566
killed the docs promote this way).
Scope the rescue narrowly to ONLY the per-service "ServiceInstance not
found" message — log+skip that one service exactly like the nil case —
so every other GraphQL failure (auth, rate-limit, schema drift) still
propagates fail-loud. Adds red-green coverage: a single thrown not-found
is skipped (healthy services still snapshot), while an unrelated GraphQL
error still raises.
The /strands/deploy-agentcore and /langgraph/deploy-agentcore pages
rendered only their title — the body was empty. <Content> resolved to a
dead stub in the MDX component registry that rendered nothing, despite
the content being authored in the shared agentcore partial.
- mdx-registry.tsx: replace the dead Content stub with a dedicated
component that renders the agentcore partial via PartialLoader and
threads the page's framework into MDX scope.
- mdx-registry-loader.tsx: PartialLoader accepts an optional scope,
forwarded to MDXRemote options.scope so partials can read bare scope
identifiers (next-mdx-remote binds scope as module identifiers, not as
the rendered component's props).
- agentcore/index.mdx: reference {framework} (bare scope var) instead of
props.framework so AgentCoreCommandTabs collapses to the single
relevant framework per page.
The icon geometry was defined via the Chrome-only CSS `d:` property, so
the spinner/checkmark rendered blank in Safari and Firefox. Move the
geometry into each path's `d` attribute and split the single morphing
path into two overlaid paths — a spinning arc that fades out and a
checkmark that draws itself in (stroke-dashoffset) upright.
Spin the arc with `transform-box: fill-box; transform-origin: center` so
it stays centered in WebKit (view-box is mis-resolved there; a SMIL
animateTransform stalls on first paint in Chrome). Both paths use
`pathLength="1"` so dashes read as fractions, and all motion is gated
behind `prefers-reduced-motion`.
## Summary
- **Cold-start retry before fast-fail (#71), gated to plain-fill turns**
— `conversation-runner` now performs a bounded turn-1 `page.reload()`
retry when an error banner appears on a cold start, recovering transient
boot flaps. The retry shares the single turn deadline (no ~2× budget
blowup, FF20) and is gated to plain-fill turns so it never masks a real
failure: a banner that survives the reload still fast-fails.
- **Fleet teardown surface-state honesty** — adds a
`worker-reclaimed-pending` comm-error kind, a control-plane SIGTERM
drain path that distinguishes graceful teardown from a crash (#70), and
pre-dispatch warm-up health pings (#72). Graceful Railway teardown is
now reported as pending, not as a red.
- **Dashboard pending-surface never masks a real red** — `cell-model`
decodes comm-errors severity-first, guards against stale `observedAt`,
and renders a dedicated pending chip. A pending surface can never
override a genuine red, while transient teardown noise resolves to
pending instead of flapping.
**Dashboard-green impact:** kills false flaps on Railway teardown
WITHOUT masking real reds, and bounds the cold-start retry so a
genuinely-broken cold start still surfaces.
## Test plan
- [x] harness `tsc --noEmit` (clean)
- [x] harness vitest — conversation-runner (46), fleet contracts +
control-plane/job-producer + queue-client (123 tests across 4 touched
suites, all passing)
- [x] shell-dashboard `tsc --noEmit` (clean)
- [x] shell-dashboard vitest — cell-model + depth-chip (148 tests, all
passing)
- [x] oxfmt `--check` clean on all changed files
## Known follow-ups
All pre-existing, tracked for a separate PR (not introduced by this
change):
- `priority` / `leaseSeconds` dead knobs in the fleet contracts
- `modelsEqual` does not compare `jobId`
- `DepthChip` switch is not exhaustiveness-checked
- unknown-state should map to a gray cell in `cell-model`
- `createRailwayAdapter` stale JSDoc
- no-reload-retry test-fake edge case
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Follow-ups to #5309 that further lower the d5/d6 dashboard red floor:
- **byoc browser-pool race:** guard `newContext()` against a
disconnected shared browser (the d5 contention source).
- **D4 probe:** `networkidle`→`load` — networkidle never settles on
CopilotKit's persistent-SSE demo pages, so D4 timed out locally and
gated every cell red on the local rig; staging unaffected but the fix is
correct and unblocks local==staging visual testing.
- **strands + spring-ai declarative-hashbrown/json-render:** dedicated
backend agents/controllers + tuned prompts + regenerated d6 fixtures +
manifest coherence (json-render is a real byoc-feature-type cell, at
parity with strands).
- **beautiful-chart:** port the pie/bar/scheduler generative-UI
renderers to built-in-agent + claude-sdk-python (backends already emit
the tool-calls).
- **cleanup:** remove the temporary x-diag-probe instrumentation (keeps
the permanent CVDIAG + the #5309 forwarding fixes).
Showcase-only; no package releases.
## Test plan
- [ ] CI rebuilds :latest; redeploy
- [ ] PB re-pivot:
byoc/declarative-hashbrown/json-render/beautiful-chart-chart cells green
on the fixed backends
- [ ] d5 contention sawtooth reduced (byoc guard)
## Deferred (separate follow-ups)
- Flap-band: decouple PB sampling from live run / detector cold-start
retry / warm+pool (the ±70 churn band)
- #67 backlog: harness accounting (d6 hung-teardown, d4 body-fallback
false-green), forwarding-shim hardening, spring-ai controller error-path
lifecycle (class-wide), built-in-agent @copilotkit/runtime pin
## Summary
Greens the staging d5/d6 showcase dashboard, which was red across ~450
cells. Root causes were a small set of genuine code/CI bugs (not a
single regression — the 06-01/06-06 flap edges were a harness
error-banner detector turning on + D5 starting to run the already-broken
D6 pill set):
- **ASSET-404 (multimodal):** CI checked out Git LFS files as pointer
stubs (`lfs:false` + no `git lfs pull`), so images served 130-byte
pointers instead of real assets. Now pulls LFS during build.
- **ROUTE-404 (beautiful-chat / declarative-hashbrown /
declarative-json-render):** 5 backends shipped the demo pages but no
per-pill API route → 404. Routes added (mirrored from same-language
siblings; injectA2UITool set per whether the backend already owns the
tool).
- **Gen-UI header forwarding:** the secondary `generate_a2ui` LLM call
dropped `x-aimock-context` — in Python because it runs in a thread-pool
executor that doesn't propagate the ContextVar (ported ag2's
`install_executor_contextvar_propagation` to
agno/ms-agent-python/pydantic-ai/strands), and in .NET because the AG-UI
SSE pump runs on a non-inheriting ExecutionContext (moved AsyncLocal →
HttpContext.Items, removed a racing finally-wipe). pydantic-ai
additionally sent a system-only prompt (StateDeps.copilotkit was always
None → now reads the real conversation). mastra's gen-a2ui threw
AI_UnsupportedModelVersionError (ai v4 vs @ai-sdk/openai v2 → bumped ai
to v5).
- **Harness measurement:** the fleet worker ignored the YAML
`timeout_ms` (used a hardcoded 10-min default → slow backends
false-aborted); now conveyed. Stopped rendering
`ms-agent-harness-dotnet`, which was flipped deployed:true but excluded
from all probes → perpetual stale red.
Showcase-only; no package releases.
## Test plan
- [ ] CI rebuilds `:latest` images with LFS assets; staging redeploys
- [ ] PB re-pivot: multimodal / beautiful-chat / declarative-* / gen-ui
cells green across backends
- [ ] aimock journal: secondary gen-ui calls carry `x-aimock-context`
(no ctx-absent 503s on real-pill traffic)
- [ ] ms-agent-dotnet d5 no longer abort-cascades
## Deferred to follow-ups (out of this PR's subject)
- strands + spring-ai declarative-hashbrown/json-render need backend
prompt specialization (separate PR)
- pre-existing bucket-(c)/(d): diag-hop labeling (reverts with the
temporary x-diag-probe), harness deploy-churn/total-invariant
accounting, .NET secondary-caller error-mapping, forwarding-shim
hardening, baseline-only stale-render partners
Jordan Ritter