Commit Graph

20 Commits

Author SHA1 Message Date
Alem Tuzlak 9df97ee23b fix(showcase): refresh validate-pins ratchet hash after ADK demo set shift
The previous commit fixed the universal ADK loop + agent renames + A2UI
shape, which changed the pin-status of ~half of the google-adk demos in
the validate-pins matrix. Total FAIL count is unchanged at 95 (no
regression elsewhere), but the SET of failing tuples drifted, which the
ratchet correctly caught.

Updates validatePinsFailHash to the new sorted-failing-set SHA from the
CI run (62e06e1e...0b04d5e0). No baseline count change — `_comment`
explicitly forbids raising the count without sign-off.
2026-05-13 17:19:28 +02:00
Tyler Slaton 2f7df90a02 fix(showcase/google-adk): unbreak shell build + ratchet validate-pins to 95
CI surfaced two issues with PR #4792:

1. shell/shell-dojo/shell-docs build-check: the bundler walks the
   manifest's `highlight:` list when bundling demo source for the
   shell's Code tab. Three paths were stale after the parity blitz
   restructured the demos:
   - chat-slots: custom-welcome-screen.tsx → slot-wrappers.tsx (LP's
     current highlight; the old file was replaced when chat-slots was
     ported to LP's Slot Atlas pattern)
   - headless-complete: message-list.tsx → chat/chat.tsx (file moved
     into the chat/ subdir during the LP-verbatim port)
   - declarative-hashbrown: copilotkit-byoc-hashbrown/route.ts →
     copilotkit-declarative-hashbrown/route.ts (route dir was renamed
     when the slug went byoc → declarative)

2. Validate Showcase: validate-pins is a drift ratchet — pin failure
   count can only decrease. Pinning google-adk's frontend +
   ag-ui-adk dropped the count from 98 → 95. Update baseline so the
   improvement locks in.

Verified locally with a script that walks every demo's `highlight:`
list and checks each path resolves on disk.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 21:50:35 -07:00
Alem Tuzlak 45cc89dc99 chore(showcase): ratchet validate-pins baseline 97→98
Pre-existing drift on main (mastra/ms-agent/llamaindex/pydantic-ai/strands)
that the langgraph-python Dockerfile fix is unrelated to. Bump the
baseline in this PR to unblock CI.
2026-05-07 14:03:39 +02:00
Ran Shem Tov 2358034310 chore: ratchet showcase pin baseline to 97 2026-05-01 13:01:30 +02:00
Jordan Ritter 223c52b6c2 fix(showcase): update validate-pins baseline for google-adk openai dep
Adding openai to google-adk/package.json for the voice route introduced
one new pin validation failure (136 vs 135). Update the ratchet baseline.
2026-05-01 01:00:53 -07:00
Jordan Ritter f02c6fa906 chore(showcase): ratchet validate-pins baseline 134→135
New pin drift from ms-agent-dotnet V1→V2 import change.
2026-04-30 17:04:56 -07:00
Alem Tuzlak bea43fb999 chore(showcase): ratchet validate-pins hash baseline
Pre-existing drift on main: count held at 134, hash shifted (one
FAIL healed, another regressed). Captured the new sorted-FAIL hash
locally with the same algorithm CI uses (sort -u | shasum -a 256)
and updated showcase/scripts/fail-baseline.json to match.
2026-04-30 11:41:03 +02:00
Jordan Ritter b4d5b9c0d4 fix(showcase): update pin ratchet baseline for restored eval files
Bump validatePinsFailCount 132→134 and hash to account for
eval-tiers.json and showcase_eval.yml re-added by eval restoration.
2026-04-29 20:22:47 -07:00
Jordan Ritter 5abf0409a8 fix(showcase): D5 test infrastructure and CLI improvements
- Rewrite cmd-test.sh to use harness CLI for D5 probes
- Add run-e2e-with-aimock.sh helper (removed stale version)
- Update fail-baseline hash for D5 coverage
- Fix provider base URLs in .env.example
- Update local-ports.json for spring-ai
- Remove eval tier system (superseded by D5 depth probes)
- Add D5 debugging runbook
2026-04-29 19:40:10 -07:00
Alem Tuzlak 7201a22294 chore(showcase): actually bump validate-pins baseline (prev commit's edit was a no-op) 2026-04-29 13:25:12 +02:00
Martha Schumann 5d5b655b88 chore(showcase): ratchet validate-pins baseline hash for ADK bump
The ADK source pin moved 1.55.2 → 1.56.4, so the corresponding
[FAIL] line in showcase validate-pins now reads "Dojo=1.56.4".
Same FAIL count (129), different hash. Updates the baseline so
CI accepts the new (still-failing-but-known) state.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 12:07:10 -07:00
Jordan Ritter 55a2061615 fix(showcase): restore langgraph-python as reference + update ratchet baseline
Add missing feature IDs (hitl, hitl-in-chat-booking) to langgraph-python
manifest so it reclaims reference status from langgraph-fastapi. Update
catalog test expectations and validate-pins fail-baseline.json hash/count
after PR #4287 dependency changes.
2026-04-24 22:09:13 -07:00
Jordan Ritter ff4bb27945 fix(showcase): integration fixes — lefthook glob, feature-registry, fixture validator, constraints, test snapshots 2026-04-24 21:24:26 -07:00
Alem Tuzlak c0eaa00b73 chore(showcase-scripts): bump validatePinsFailCount to 112 for voice demo
The voice demo adds two new framework deps (@copilotkit/voice, openai)
to showcase/packages/langgraph-python. The Dojo example doesn't pin
either, so validate-pins emits [FAIL] "is not an exact pin in showcase"
for each. Both use the same non-exact spec style as the already-baselined
@copilotkit/react-core / @copilotkit/runtime entries (next-channel dist
tag + caret range). Bump the drift baseline by 2 so CI accepts the new
deps without lowering the overall pin discipline.
2026-04-23 22:31:10 +02:00
Jordan Ritter 008d4e58e8 chore(showcase): ratchet validate-pins baseline after Dojo pin bump 2026-04-21 10:51:55 -07:00
Jordan Ritter 9379b8855a fix(showcase/crewai-crews): upgrade ag-ui-crewai to 0.2.0 + drop shim + -u flag
Root-cause fix for the 04-21 silent-hang incident on the crewai-crews
Railway deploy. Three tightly-coupled changes:

1. Bump ag-ui-crewai pin from `>=0.1.4,<0.1.6` to `>=0.2.0,<0.3.0`.
   0.1.5 had three defects that wedged the agent: unguarded
   `source.state.messages` access, an orphan `asyncio.create_task` with
   no cancel ref, and a sync `completion()` call that pinned the event
   loop. All three are fixed in ag-ui PR #1550, released as 0.2.0 on
   2026-04-18. Our upper ceiling was blocking the fix.

2. Remove the pre-bind LLM crash hardening shim in `agent_server.py`.
   The shim monkey-patched `crewai.cli.crew_chat.generate_*_description_with_ai`
   to static strings so that `ChatWithCrewFlow.__init__` — which
   ag-ui-crewai <= 0.1.5 invoked at endpoint-registration time, BEFORE
   uvicorn bound its port — could not crash the process before the HTTP
   server was listening. 0.2.0 defers `ChatWithCrewFlow` construction to
   first request via a module-scoped `_cached_flow` + `asyncio.Lock`
   inside `add_crewai_crew_fastapi_endpoint`. Any LLM hiccup now
   surfaces as a 5xx on the first request instead of a startup crash,
   which is what the shim was reaching for. The shim is dead code on
   0.2.0 and has been removed (with `logging` import dropped as it was
   only used by the shim).

3. Add `python -u` to the uvicorn invocation in `entrypoint.sh` as a
   belt-and-suspenders complement to the existing `PYTHONUNBUFFERED=1`
   export. The env var can in principle be un-exported by a child;
   `-u` forces unbuffered stdout/stderr at the interpreter level and
   is not overridable by user code. Combined with `awk '{...; fflush()}'`
   in the pipe (already in place), this guarantees uvicorn request
   lines reach Railway's log stream line-at-a-time. During the 04-21
   incident Railway saw only ~15 log lines over 9h of uptime because
   of buffering through a previous `sed` formulation.

Also updates `showcase/scripts/fail-baseline.json`'s `validatePinsFailHash`
to match the new `ag-ui-crewai` spec string. Pin-drift FAIL count is
unchanged (110); the hash changed only because the `ag-ui-crewai` line
in the FAIL set went from `>=0.1.4,<0.1.6` to `>=0.2.0,<0.3.0`.

Verified locally:
- `pip install -r requirements.txt` resolves `ag-ui-crewai-0.2.0` cleanly.
- `python -u -m uvicorn agent_server:app` starts; `/health` returns
  200 `{"status":"ok"}`; request lines appear in real-time logs.
- `pytest tests/python/` — 94/94 pass.
- `pnpm -C showcase/scripts test` (vitest) — 1079/1079 pass.
- `validate-pins.ts` — count=110 matches baseline; hash updated.

Upstream refs:
- crewAI issue: https://github.com/crewAIInc/crewAI/issues/5510
- ag-ui PR #1550: https://github.com/ag-ui-protocol/ag-ui/pull/1550

Intentionally NOT in this PR:
- `showcase/starters/crewai-crews/` parity backport (the starter still
  carries the 0.1.5 pin and the shim).
- The 14-starter watchdog generalisation.
Both belong to the silent-hang vulnerability-class work tracked
separately.
2026-04-21 08:49:21 -07:00
Jordan Ritter 321ceb9fa0 chore(showcase): ratchet validate-pins baseline for main (post-4095)
PR #4095 (agno SDK upgrade to 2.5.17) merged into main and shifted the
`[FAIL]` set produced by validate-pins.ts. The FAIL count is unchanged
(110 -> 110), but the hash of the sorted/deduplicated FAIL lines drifted
because a FAIL healed in one package while another regressed — net zero
on the counter, different tuples in the set.

Re-ratchets `showcase/scripts/fail-baseline.json` to reflect current main:
  validatePinsFailCount: 110 (unchanged)
  validatePinsFailHash:  c87ecdd6... -> 958bcf24...

Main CI is red until this lands; please merge ASAP.

Observed from run 24632821166:
  validate-pins FAIL: actual=110 baseline=110
  validate-pins HASH: actual=958bcf24... baseline=c87ecdd6...
2026-04-19 08:50:31 -07:00
Jordan Ritter 0aa95d7e15 chore(showcase): ratchet validate-pins baseline 109->110 for google-genai pin 2026-04-19 08:07:02 -07:00
Jordan Ritter 5f6a864922 chore(showcase): ratchet validate-pins baseline 111→109 (fixes landed on main)
Real pin-fix PRs have landed on `main` since the baseline was last
refreshed, reducing the FAIL set from 111 to 109 and invalidating the
stored hash. The ratchet gate now rejects every PR (including ones
that don't touch pins) with a "ratchet down" instruction.

Refresh the baseline to reflect the actual state of `origin/main`:

  validatePinsFailCount: 111  -> 109
  validatePinsFailHash:  77b586b7 -> d03716b5

Values computed by running the canonical pipeline from
`.github/workflows/showcase_validate.yml` ("Run validate-pins (ratchet)"
step) against a clean `origin/main` worktree:

  pnpm exec tsx showcase/scripts/validate-pins.ts 2> stderr
  grep ^Summary stdout       -> FAIL=109
  grep '^\[FAIL\]' stderr | LC_ALL=C sort -u | shasum -a 256
                             -> d03716b5...f597e81d

This is a pure ratchet-down to match reality, not a policy change.
No validator behavior, workflow, or pin change is included. Actual
pin drift cleanup (109 -> 0) continues as a separate effort.

Unblocks #4068 and any other PR stalled on the same ratchet.
2026-04-18 10:10:33 -07:00
Jordan Ritter cb0bf1c9b5 feat(showcase/scripts): add validate-pins.ts pin-drift validator
Compares framework dependency pins across showcase/packages/*/ and
the corresponding dojo examples/integrations/* trees, flagging drift
between the two and rejecting non-exact specs on the showcase side.

Key design:
- Parses package.json, requirements.txt, and pyproject.toml
  (including Poetry's [tool.poetry.dependencies] and PEP 621
  [project.dependencies] / optional-dependencies). Separate jsDeps
  and pythonDeps maps prevent cross-ecosystem name collisions.
- isExactSpec enforces exact-version pins per ecosystem (npm: no
  operators, workspace refs, or ranges; Python: ==X / ===X / ~=X
  with PEP 440 body). Symmetric rejection of bare MAJOR-only forms.
- parseRequirementsTxt and parsePyprojectToml thin wrappers throw
  when the detailed form produced skipped[] or dropped[] entries,
  preventing silent data loss in simpler callers.
- canonicalizeDepMap canonicalises names per PEP 503 and surfaces
  same-file collisions with differing specs as warnings.
- First-writer-wins at both file and package levels.
- UnreadableInputError carries an optional partialReport so an
  infra failure mid-slug-loop preserves already-collected drift
  findings for other slugs.
- Exit codes: 0 ok, 1 drift, 2 internal, 3 unreadable.

fail-baseline.json is the single source of truth for the CI ratchet
(validatePinsFailCount + validatePinsFailHash) and the demo-count
floor (baselineDemoCount, cross-checked against validate-parity.ts
in a dedicated sync test).

Test coverage spans every parser variant, EACCES routing via chmod
probe + fs spies, exit-code taxonomy subprocess tests, partial-report
preservation on mid-loop infra throws, and Poetry/PEP 503 edge cases
via committed fixture files under __tests__/fixtures/pins/.
2026-04-17 22:34:18 -07:00