mirror of
https://github.com/CopilotKit/CopilotKit.git
synced 2026-09-14 16:26:20 +08:00
Jordan Ritter
7335d58359
## Summary Fixes two correctness bugs in the showcase staging→prod promote path (`showcase/bin/railway`), discovered + live-validated while promoting the full 19-service cluster. **Bug #1 — promote pinned the wrong digest.** `resolved_prod_image` re-resolved the mutable `:latest` tag against *current* GHCR instead of pinning the digest staging is actually *running* (`latestDeployment.meta.imageDigest`). When `:latest` drifted after staging deployed, promote pushed an unvalidated (and once, regressed) image to prod. Now pins staging's running digest. Adds a loud `⚠️ STAGING DRIFT` warning (promote stdout + `STAGING_DRIFT_MARKER:` → `promote-fleet.sh` aggregation → both Slack payloads) when staging's running digest ≠ current `:latest`, so the gap is visible without blocking the promote. **Bug #2 — the pin never activated.** `pin_and_verify` used `serviceInstanceRedeploy`, which replays the *existing* deployment's old image rather than the just-pinned `source.image`. Config showed the new digest while prod kept serving the old one (this is why earlier promotes "succeeded" while prod stayed broken). Switched to `serviceInstanceDeployV2` + a new `verify_serving_digest!` gate that polls the new deployment to SUCCESS and **fails loud** if the running digest ≠ pinned. Plus CR-round hardening: P2 in-flight race check now reads `meta.imageDigest` (was dead on tag-form staging) and skips on `--digest` override; `detect_staging_drift` fails loud (WARN) on GHCR-resolve failure instead of swallowing; `--digest` override suppresses spurious drift; `drift_line` LF-stripped at the `GITHUB_OUTPUT` boundary; multi-service drift join fixed; fallback-log drift preserved. ## Commits 1. `fix(showcase): pin prod to staging's running digest + loud staging-drift warning` 2. `fix(showcase): activate prod pin via serviceInstanceDeployV2 + verify running==pinned` 3. `fix(showcase): harden promote P2 race check + refresh ivar-lint allowlist` ## Validation - Live red-green on real Railway: claude-sdk-python prod flipped from stale → pinned digest via the fixed CLI; all 19 cluster services promoted green; 5 previously-degraded backends (ag2, llamaindex, pydantic-ai, ms-agent-python, strands) recovered. - 15/15 Ruby specs + 15/15 bats green; ruby -c, shellcheck (CI invocation), actionlint clean. - 3-round cr-loop (7 agents/round) converged to zero bucket-(a). ## Follow-ups (not in this PR) - Defensive `.to_s.empty?` on `meta.imageDigest` extraction (can't-happen on real Railway). - Test-quality nits (capture_io scoping, weak bats glob, shared mock fixture). - Pre-existing verify-prod `succeeded_csv`/`GITHUB_OUTPUT` coupling (graceful-degraded, audited STAY_IN_C). - `deploy-to-railway.ts` births prod on `:latest` (the root provisioning gap); CLI↔workflow notify equivalence (PR2). ## Test plan - [ ] CI green on PR HEAD - [ ] (post-merge) a real `gh workflow run showcase_promote.yml` shows the drift line in the #team-showcase notification when staging is behind :latest