Commit Graph

926 Commits

Author SHA1 Message Date
Jordan Ritter 9f2b859450 fix(showcase): stop ag2 generate_a2ui empty-arg validation loop
The ag2 declarative-gen-ui route pointed its HttpAgent at the root
catch-all mount (agents/agent.py) instead of the dedicated
/declarative-gen-ui mount (a2ui_dynamic.py), and generate_a2ui declared
a required context arg that the model emits as {}. pydantic rejected
every call with "context Field required" and AG2 retried without bound —
a 630-iteration hot loop per pill that flooded logs and starved the
frontend.

Fix: route to the dedicated mount with injectA2UITool:false (the
dedicated agent owns generate_a2ui and emits a2ui_operations itself);
make generate_a2ui a no-arg tool matching the D6 fixtures and the
langgraph-python gold standard, with a constant inner system prompt
(per-pill distinctness comes from the captured user message). Regenerated
the gen-ui-declarative fixture and ported the LP definitions/renderers
catalog (all 7 driver testids) for parity. Eliminates the validation
loop: runsFinished=1, zero validation errors.
2026-06-23 15:12:09 -07:00
Jordan Ritter 7cb43581aa fix(showcase): forward request-time tools in llamaindex AG-UI adapter
The LlamaIndex AG-UI adapter never forwarded RunAgentInput.tools, so
page-injected frontend tools (useFrontendTool / useComponent /
useHumanInTheLoop) were invisible to the LLM — runs the model couldn't
satisfy ended in RUN_ERROR (no RUN_FINISHED), which the harness reported
as sse-missing.

New RequestAwareAGUIChatWorkflow re-implements the chat step as the
upstream body plus three additions: forward RunAgentInput.tools as no-op
FunctionTool stubs carrying the verbatim injected JSON schema; re-role
tool-result messages to role="tool" on the LLM-bound message copy only
(so the hasToolResult second-leg fixture matches) without mutating
stored/snapshot history; and for frontend tool calls dispatch the
ToolCallEvent but skip the duplicate TOOL_CALL_CHUNK (the bare snapshot
already delivers the call). beautiful-chat 0/5 -> 5/5; mcp-apps fixed.
2026-06-23 15:12:09 -07:00
Nathan 🔶 Tarbert 97d1de4682 fix(showcase): use agent.setState in built-in-agent shared-state demo
The built-in-agent shared-state-read-write demo wrote agent state via
direct property assignment (`agent.state = {...}`), which sets the value
but does not fire `onStateChanged` — so subscribed components never
re-render off a UI write. Every other framework demo (including the
langgraph-python gold standard) already uses `agent.setState(...)`, which
both updates state and notifies subscribers. This was the lone divergence.

Switch the seed and the title write to `agent.setState(...)` so the demo
matches the documented pattern (docs/shared-state.mdx says writes go
through `setState`) and the rest of the showcase.

Verified: built-in-agent D5 e2e-deep suite green after rebuild.
2026-06-23 17:21:49 -04:00
Alem Tuzlak 5ecdee36b8 feat(bot): pluggable StateStore persistence + cross-platform transcripts
Adds a durable persistence layer for @copilotkit/bot, replacing the
in-memory-only ActionStore with a pluggable StateStore.

- StateStore interface (kv/list/lock/dedup/queue) with a shared
  conformance suite; MemoryStore default plus @copilotkit/bot-store-redis
  and @copilotkit/bot-store-postgres backends.
- createBot({ store }): typed per-thread state via Standard Schema,
  action snapshots persisted through the store, per-conversation turn
  lock (onLockConflict drop|force), and inbound-event dedup keyed on a
  stable eventId. ActionStore is kept as a deprecated alias.
- Cross-platform transcripts (bot.transcripts + identity resolver) with
  age-bounded retention (prune on append + filter on read), and
  runAgent({ transcript: true }) to auto-inject history and capture the
  reply.
- createBot({ components }) re-registers components so durable actions
  re-fire after a restart; restart-durability demo in examples/slack.
- Dedup is marked seen only after the turn lock is acquired, so a turn
  dropped on lock-conflict does not burn its eventId (no lost retries).
- Release lockstep: bot-store-redis/postgres version with bot + bot-ui.
2026-06-23 18:33:38 +02:00
Ran Shemtov e0d6eeaebe Merge branch 'main' into claude/trusting-babbage-f4d48a 2026-06-23 09:21:35 +02:00
github-actions[bot] 931e046f6d style: auto-fix formatting 2026-06-22 17:35:50 +00:00
Jordan Ritter 1a80d28f4e chore(cvdiag): regenerate schema.json + re-stage TS emitter after attribution-fix union merge
schema.json regenerated from the merged canonical schema.ts (failure_classifier
probe.exit additions UNION backend request.ingress/sse.first_byte/llm.call.*
boundaries + test_id adoption). Per-integration staged schema.ts copies
re-derived via 'showcase cvdiag-stage-ts' so codegen --check and stage --check
are both in sync. No hand-merge of generated artifacts.
2026-06-22 10:31:04 -07:00
Jordan Ritter 4925d0fc60 fix(cvdiag): backend adopts inbound x-test-id as cross-layer test_id + emits request.ingress/sse.first_byte/llm.call.* boundaries (TS integrations) 2026-06-22 10:30:09 -07:00
Ran Shem Tov 8768c8a7e7 Merge remote-tracking branch 'origin/main' into claude/trusting-babbage-f4d48a 2026-06-22 16:11:21 +02:00
Ran Shem Tov 851dc1e90d feat(showcase): strands-typescript parity with strands-python (A2UI, aimock fixtures, header forwarding)
Bring the TypeScript AWS Strands integration to parity with the Python
strands sibling now that the @ag-ui/aws-strands TS adapter is confirmed to
support the same feature surface (per its examples/server):

- Restore A2UI: the declarative-gen-ui + a2ui-fixed-schema demos, their
  routes, qa, specs, the @copilotkit/a2ui-renderer dep, beautiful-chat's
  A2UI catalog, and the manifest entries (generative_ui / features / demos /
  a2ui_pattern). manifest now matches strands-python feature-for-feature.
- Header forwarding: attach `x-aimock-context: strands-typescript` as a
  static defaultHeader on the OpenAI client (model-factory + sub-agent
  client) — the TS analog of the Python integration's _header_forwarding
  shim — so aimock matches this integration's fixtures.
- aimock fixtures: add d6/strands-typescript + d4/strands-typescript
  (ported from the Python sibling, context retargeted).
- playwright.config: X-AIMock-Context → strands-typescript.

Note: the raw tests/e2e Playwright suite is flaky and not a CI merge gate
(demo e2e / `/eval` D5 are comment-triggered, not required) — it fails the
same specs for strands-python too. The auto-gates (build, validate-
constraints, oxlint/oxfmt, unit) are green.
2026-06-22 14:36:04 +02:00
Jordan Ritter 154ffb969c style(cvdiag): oxfmt the new writer-auth TS files + re-stage (preempt auto-format bot) 2026-06-21 12:52:44 -07:00
Jordan Ritter e0cb30e7b9 fix(cvdiag): narrow PB binary to concrete str in live-PB test fixture to satisfy Pyright reportArgumentType 2026-06-21 12:50:59 -07:00
Jordan Ritter 2efc99fa5b fix(cvdiag): bundle a concrete writer-role PB writer into TS integration backends (auth-with-password→Bearer fetch) so backend events persist; was a type-only no-op seam 2026-06-21 12:49:05 -07:00
Jordan Ritter aeb94b95b2 fix(cvdiag): Python PB writer authenticates as cvdiag_api_keys writer role (auth-with-password→Bearer) instead of inert X-Cvdiag-Writer-Key header — backend events now persist 2026-06-21 12:49:01 -07:00
Jordan Ritter ca12d09c36 cvdiag: permanent showcase observability subsystem (probe→backend→aimock→edge) (#5591)
## What

Adds **cvdiag** — a permanent, always-available observability subsystem
for the showcase, built to diagnose the red↔green cell flap on the
staging dashboard and to make that diagnosis a dashboard query rather
than a multi-day forensic hunt in the future.

Captures the full request path with `X-Test-Id` correlation across
**probe → backend → aimock → edge**, across every integration
(TypeScript, Python, Java/spring-ai, .NET):
- Per-language backend emitters (canonical + staged/compile-linked
mirrors), all sharing one schema (`schema.json`, closed-world
`additionalProperties:false`).
- CREATE-only writes to two new PocketBase collections: `cvdiag_events`
and `cvdiag_raw_byte_samples` (additive migrations — no existing data
touched).
- An 8-class flap classifier mapping to the observed failure signatures
(`sse-missing` / `text-unstable` / `dom-missing`).
- DEBUG-tier raw-byte capture (secret-scrubbed) and HMAC-guarded A/B
edge-interference detection.

## Why

The runId flap-fix (`cdc1e90e`, 2026-06-09) did **not** fully resolve
the flap — it was still observed 2026-06-19. cvdiag exists so the
*remaining* cause is observed live with full correlation instead of
inferred.

## Safety / enablement

- **Inert by default.** With `CVDIAG_BACKEND_EMITTER` unset the
subsystem performs zero host mutation (no logging-config changes, no
threads/tasks, no stdout) — verified by
`test_cvdiag_inert_when_disabled`. **To accumulate data, set
`CVDIAG_BACKEND_EMITTER=1` on the showcase services.**
- All per-language scrubbers match the canonical `scrubSecrets`
(sk-/base64url, Bearer, colon-less URL userinfo, size-guard) — verified
with real toolchains (vitest / mvn / dotnet).
- Merged latest `main` (only conflict: a clean `.csproj` include union).

## Verification
- harness `tsc --noEmit` ✓ · `src/cvdiag` vitest 251/251 ✓ ·
`cvdiag-stage-ts --check` in-sync ✓
- Java MessageScrubber 17/17 (mvn) ✓ · .NET CvdiagBackend 5/5 (dotnet
sdk:9.0) ✓ · Python emitters 93/93 (3.12) ✓

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-06-19 20:45:52 -07:00
Tyler Slaton db2fd6539b fix: address merge conflicts and run formatter 2026-06-19 15:51:03 -07:00
Jordan Ritter 679282570b fix(cvdiag): resolve staged cvdiag .js imports in integration next build (extensionAlias / stage rewrite) — unbreaks build-check across all integrations (M6) 2026-06-19 14:09:14 -07:00
github-actions[bot] 691c036789 style: auto-fix formatting 2026-06-19 20:54:20 +00:00
Jordan Ritter fada109b72 Merge remote-tracking branch 'origin/main' into blitz/cvdiag-observability/integration
# Conflicts:
#	showcase/integrations/ms-agent-harness-dotnet/agent/BeautifulChatAgent.csproj
2026-06-19 13:51:02 -07:00
Jordan Ritter 32d851a290 fix(cvdiag): .NET scrubber redacts URL userinfo (scheme://user:pass@host + colon-less) matching scrubSecrets (M6) 2026-06-19 12:33:06 -07:00
Jordan Ritter a5678c118f fix(cvdiag): Java MessageScrubber parity with scrubSecrets — sk- base64url tails, Bearer \S+, colon-less URL userinfo + size-guard (M6) 2026-06-19 12:33:03 -07:00
Jordan Ritter 0a826bf17f fix(cvdiag): re-stage TS cvdiag emitter from canonical — staged copies were stale (leaked sk-ant-/colon-less URL userinfo + half-missing emit.ts) (M6) 2026-06-19 12:33:00 -07:00
Jordan Ritter 802ffbbcc9 fix(cvdiag): _shared logging setup is inert when disabled — no host root-handler teardown at import; capture-when-enabled preserved + inert lever test (M5 CR R4)
setup() ran logging.basicConfig(force=True) unconditionally at import (module
calls setup() at the bottom), tearing down the host app's root-logger handlers
on every backend even when cvdiag was disabled (CVDIAG_BACKEND_EMITTER off,
canary-safe default) — violating the byte-for-byte-inert contract. Replaced the
root basicConfig(force=True) with a scoped StreamHandler attached to the
"agents" logger, installed ONLY on the ENABLED path (after _ENABLED=True), so a
disabled/degraded setup performs zero logging mutation and the host root
handlers survive. capture-when-enabled (agents.* → stdout) preserved.

Call sites: setup() defined cvdiag_bootstrap.py:133, invoked at import-time
:273 (now no longer touches root). is_enabled() :206 unchanged (gates emit).
basicConfig() call REMOVED from :128; only doc references remain. New scoped
_install_agents_log_capture() runs on the enabled path; reset_for_test() now
detaches the handler.
2026-06-19 12:02:55 -07:00
Jordan Ritter 530649864e fix(cvdiag): LGP gates request.ingress/llm.call.*/sse.first_byte to VERBOSE tier matching canonical _BOUNDARY_TIER (M5 CR R3)
The four §6-VERBOSE-only backend boundaries (request.ingress, llm.call.start,
llm.call.response, sse.first_byte) called _emit with no tier_gate, so they
over-emitted at DEFAULT tier — 4 extra events/request vs the middleware family,
breaking the §7 tier budget and cross-backend apples-to-apples parity. Gate
them with tier_gate=_VERBOSE_TIERS, matching emit.ts:58-63 and the agno
_BOUNDARY_TIER. langgraph-fastapi received the identical change (the two LGP
files differ only by docstring/plan-unit/_SLUG). Adds default-suppressed +
verbose-emits red-green coverage; updates the pre-existing first_byte
correlation test to drive at VERBOSE tier (the boundary is VERBOSE-only).
2026-06-19 11:42:52 -07:00
Jordan Ritter 75e2c53286 fix(cvdiag): _shared emit gate consults _ENABLED so the fail-closed DEBUG degrade actually suppresses emission (M5 CR R3)
emit_cvdiag now early-returns when not is_enabled(), in addition to the
per-integration CVDIAG_BACKEND_EMITTER env check, so a degraded setup()
(_ENABLED=False) emits nothing — the degrade wins over the live env toggle.

Call sites: shared emit_cvdiag (this file, the single chokepoint) now gates on
is_enabled() (the previously-dead _ENABLED flag, set False by setup()'s
fail-closed degrade). Per-integration emitter_enabled() (langgraph-python,
langgraph-fastapi) and cvdiag_backend_enabled() (10 other _cvdiag_backend.py
modules) remain env-only and call into emit_cvdiag — out of scope here; the
shared gate is the defense-in-depth backstop for all of them.
2026-06-19 11:42:52 -07:00
Jordan Ritter a9d2dd342e fix(cvdiag): backend scrub URL-userinfo+Bearer-tail parity + size-guard, live-tier consistency, stop_heartbeat cooperative-cancel across 12 emitters (M5 CR R1) 2026-06-19 11:23:35 -07:00
Jordan Ritter 2f2302cb65 fix(cvdiag): _shared pb_writer never-propagate drain + bootstrap degrade-not-crash + idempotent setup (M5 CR R1) 2026-06-19 11:23:31 -07:00
Ran Shem Tov 7af24d7171 chore(showcase): label strands integration "AWS Strands (Python)" for TS parity
Now that strands-typescript ships as "AWS Strands (TypeScript)", rename the
existing Python integration's display name from "AWS Strands" to
"AWS Strands (Python)" so the two appear as a matched Python/TypeScript pair
in the docs framework selector and shell (mirrors LangGraph's
"(Python)"/"(TypeScript)" pairing). Slug, content, and behavior are
unchanged. The shared aws-strands docs language tabs already default per
framework (strands -> Python, strands-typescript -> TypeScript) via
TAB_DEFAULTS_BY_SLUG.
2026-06-19 17:45:02 +02:00
Ran Shem Tov 0e5b2189e7 feat(showcase): add strands-typescript integration with base demos
Add a new node/TypeScript-backed AWS Strands showcase integration at
showcase/integrations/strands-typescript.

Backend: a node/TS agent server (src/agent/) built on @strands-agents/sdk
`Agent`/`tool` wrapped in @ag-ui/aws-strands `StrandsAgent` and served via
@ag-ui/aws-strands/server (`createStrandsApp`/`addStrandsExpressEndpoint`),
modeled on the upstream ag-ui aws-strands TS example server and the
langgraph-typescript infra. A single shared agent at "/" serves most demos
(tools, shared state via toolBehaviors/stateContextBuilder, HITL,
sub-agents), with tool-free specialized agents mounted at /voice,
/byoc-hashbrown, /byoc-json-render. model-factory targets OpenAI chat
completions and honors OPENAI_API_KEY / OPENAI_BASE_URL so it works behind
the showcase aimock proxy. Node-based Dockerfile + entrypoint run the agent
server (:8000) alongside the Next.js frontend.

Frontend mirrors the strands (Python) sibling's demo set and the
langgraph-typescript conventions, with HttpAgent routes proxying to the TS
agent server.

Scope: base integration + standard demos only. A2UI / declarative-gen-ui /
a2ui-fixed-schema is intentionally excluded (no A2UI agents, routes, demos,
or deps) and layered on later.

Platform wiring (mirrors langgraph-typescript): docker-compose local/dev
services on host port 3119, local-ports.json, packages.json, slug-map.ts
(born-in-showcase), showcase_build.yml matrix + path filter + metadata,
shell-docs/dashboard registries, and a logo asset. The python strands
integration is untouched.
2026-06-19 17:38:02 +02:00
Alem Tuzlak c55dad5cc5 Merge branch 'main' into feat/bot-slack-native-streaming
Resolve event-renderer.ts onRunFinishedEvent: keep the native turn stream open
(finalized in finish()) AND retain the legacy per-message stream drain from main
(#5573) as a no-op-in-native safety net. app/index.ts (telegram adapter from
#5520 + showToolStatus:false) and create-bot.test.ts auto-merged.
2026-06-19 15:57:20 +02:00
github-actions[bot] 45424333f0 style: auto-fix formatting 2026-06-19 09:32:19 +00:00
Jordan Ritter 49d2ac16ff feat(showcase): port ms-agent-harness-dotnet frontend (demos, shared components, API routes) 2026-06-19 02:28:27 -07:00
Jordan Ritter 6ddc85f7ee feat(showcase): port ms-agent-harness-dotnet backend agents on the AsHarnessAgent construction surface 2026-06-19 02:28:22 -07:00
Jordan Ritter 4c1ed975c7 chore(showcase): register ms-agent-harness-dotnet column (manifest, csproj, slug-map, compose, dashboard, smoke) 2026-06-19 02:28:18 -07:00
Jordan Ritter 03f33fca48 chore(cvdiag): merge L1-D3 Python backend instrumentation into integration 2026-06-18 14:41:10 -07:00
Jordan Ritter a74f423ef4 chore(cvdiag): merge L1-D2 Python backend instrumentation into integration 2026-06-18 14:41:09 -07:00
Jordan Ritter 24de3d1710 feat(cvdiag): backend instrumentation for google-adk + langroid + langgraph-fastapi (L1-D3) 2026-06-18 14:40:32 -07:00
Jordan Ritter 84369d5887 feat(cvdiag): backend instrumentation for llamaindex + ms-agent-python + claude-sdk-python (L1-D1) 2026-06-18 14:37:47 -07:00
Jordan Ritter 9d66761064 feat(cvdiag): backend instrumentation for pydantic-ai + crewai-crews + agno (L1-D2) 2026-06-18 14:37:10 -07:00
Jordan Ritter 6d72936fcb chore(cvdiag): merge L1-I backend instrumentation into integration 2026-06-18 14:30:51 -07:00
Jordan Ritter 4cfa10d5eb chore(cvdiag): merge L1-G backend instrumentation into integration 2026-06-18 14:30:51 -07:00
Jordan Ritter ac4f582721 chore(cvdiag): merge L1-F backend instrumentation into integration 2026-06-18 14:30:50 -07:00
Jordan Ritter 04e2a19e3c chore(cvdiag): merge L1-E backend instrumentation into integration 2026-06-18 14:30:50 -07:00
Jordan Ritter 12dd238fbc feat(cvdiag): backend 11-boundary instrumentation for strands + ag2 (L1-C) 2026-06-18 14:30:07 -07:00
Jordan Ritter 3737e6ac01 feat(cvdiag): backend 11-boundary instrumentation for 2 .NET integrations (L1-F) 2026-06-18 14:28:38 -07:00
Jordan Ritter 8118ff1dea feat(cvdiag): backend 11-boundary instrumentation for 4 TS integrations (L1-E) 2026-06-18 14:27:25 -07:00
Jordan Ritter 1df8e7adba feat(cvdiag): backend 11-boundary instrumentation for spring-ai (L1-G) 2026-06-18 14:23:41 -07:00
Jordan Ritter b101ffe901 feat(cvdiag): LGP backend 11-boundary schema-v1 instrumentation (L1-I) 2026-06-18 14:19:58 -07:00
Jordan Ritter ed7e318034 chore(cvdiag): merge L0-F codegen binding into integration 2026-06-18 14:07:44 -07:00
Jordan Ritter 96ad26dcf6 chore(cvdiag): merge L0-E codegen binding into integration 2026-06-18 14:07:44 -07:00