mirror of
https://github.com/CopilotKit/CopilotKit.git
synced 2026-09-14 16:26:20 +08:00
Alem Tuzlak
9a501d5e0d
Six defects reported on staging. All six reproduce; five share one cause. ## The shared cause `built-in-agent/src/app/api/copilotkit/route.ts` maps **~20 demos to the same prompt-less `createBuiltInAgent()`**. The reference wires each to its own graph **and its own system prompt** (28 graphs in `langgraph-python/langgraph.json`). aimock replays a scripted tool-call sequence keyed on `userMessage` + `context`, so D6 is green whether or not the model could have reasoned its way there. This is `showcase/GOTCHAS.md` #8 verbatim — including its stated "structural tell": *grep the integration's `route.ts` for demos left in the generic fallthrough that LGP wires to a dedicated graph.* ## Per-defect | Demo | Symptom | Root cause | |---|---|---| | `gen-ui-tool-based` | charts plot zeros | No prompt. The assistant says so itself: *"I used placeholder values since no sales figures were provided."* Nondeterministic — one run gave a real 0..220 axis, the next `0 1 2 3 4` with no bars, which is how it passed review. | | `gen-ui-agent` | wall of text, steps frozen at "step 1 of 4" | `set_steps` + its `/steps` STATE_DELTA already worked; nothing told the model to *walk* pending→in_progress→completed. | | `subagents` | left panel always empty | `grep -rn delegations src/ \| grep -v demos/subagents` → **no output**. Frontend reads `agent.state.delegations`; the converter handles `/steps` and `/notes` and has no `delegations` case. Chat works because the tools do run. | | `declarative-json-render` | raw JSON for pills 2 & 3 | Intercepted the SSE: **the wire is already one closing brace short** — nothing is dropped in transport, and the client parser correctly requires a balanced object. Pill 1 only works because the model copies the prompt's worked example verbatim (`\$1.24M`, `+18% vs Q2`). Model-level JSON enforcement had been removed with the note *"the system prompt already enforces JSON-only output"* — it does not. | | `a2ui-recovery` | 5 identical cards | Reuses declarative-gen-ui's prompt with no "call once" constraint, while the pills literally ask it to *"self-correct a malformed first attempt"*. The retry loop is **inside** the tool and returns on first valid pass, so every supervisor retry is pure duplication. | | `declarative-gen-ui` | D4 though it works live | **The inverse case.** Badges: `UI ✓ BE ✓ 1P ✗ D6 —` (*"gated — blocked by a lower rung"*). Not a live bug — a fixture bug. See below. | ## The D4 cap (`declarative-gen-ui`, `a2ui-recovery`) Both demos' secondary design-call fixtures gated on `match.responseFormat: \"json_object\"`. That matcher **can never match this backend**: - The agent talks to the OpenAI *Responses* API, where JSON mode is `text.format`. - aimock has **no `text.format` handling at all** — `responsesToCompletionRequest` forwards only a top-level `response_format` (absent in 1.19.1, forwarded in 1.37.4), a key the Responses API doesn't accept and this client doesn't send. - So `effective.response_format?.type` is always `undefined` and `router.ts` skips the fixture → the design call never matched → surface never painted → D5 red, D6 never ran. Re-keyed onto `match.toolName`, which aimock *does* normalize out of a Responses request (`responsesToolsToCompletionsTools`): the outer call declares `generate_a2ui`, the in-tool design call declares nothing. Tool-less secondary fixtures moved last, because several pills' brief is a substring of the pill text (`\"Build my Q2 revenue summary …\"` contains `\"Q2 revenue summary\"`) and would otherwise win the outer request. ## Verification **Reproduced live** (staging, real LLM): the zero-axis chart plus its self-incriminating message, the unbalanced JSON on the wire, and the reference rendering the *same pill* correctly for contrast. **Two mutation-verified test suites** — both fail with the fix reverted: - `aimock-a2ui-routing.test.ts` drives aimock's **real `matchFixture`**. On the old fixtures 14/16 cases fail with `no fixture matched the secondary design call for brief \"…\"` — the D5 red, reproduced as a unit test. - `tanstack-factory.test.ts` covers the `/delegations` and `/steps` deltas (3 delegation tests fail with the branch disabled; the 2 controls still pass). **No regressions:** `tsc --noEmit` is 61 errors before *and* after (all pre-existing — `gpt-5.4` missing from the adapter's model union, zod v3/v4 skew); oxlint clean on changed files; the repo-wide fixture validator passes (858 assertions). `text.format` was verified by reading the request mapping in the **pinned** `@tanstack/ai-openai@0.15.6` → `@tanstack/openai-base@0.9.2`: `modelOptions` is spread straight into `responses.create()`, `validateTextProviderOptions` only inspects `metadata`/`conversation`/`previous_response_id`, and the adapter sets `text.format` itself only when an `outputSchema` is passed (none here, so nothing is clobbered). ## What is NOT verified **The four prompt/`text.format` changes have not had a real-LLM click-through** — that needs an OpenAI key this environment doesn't have, and D6/aimock cannot verify them by construction (that's the whole point of gotcha #8). Reasoning is documented inline at each site. This area has burned the repo before: a previous `response_format` attempt made the call return an empty string "verified against real OpenAI", which is why `text.format` — the Responses API's own param — is used instead. **Worth a live click-through on the staging deploy before this is considered closed.** ## Deliberate non-goals - **`a2ui-recovery` only demonstrates recovery under aimock.** The heal/exhaust branches need a designer LLM that emits invalid surfaces on demand; a real one succeeds on attempt 1. The single-call constraint fixes the 5-card bug and makes it honest; making recovery visible live needs deliberate fault injection — a product decision, recorded in `PARITY_NOTES.md`. - **The reference pins `openai:gpt-4o-mini`** (`gen_ui_agent.py:92`). Flagged, not touched. ## Docs `GOTCHAS.md` gains built-in-agent as a second worked instance of #8 (including that the masking runs *both* ways, and that `1P ✗ D6 —` means gated, not failing), plus the `text.format` and dead-`responseFormat` traps. `PARITY_NOTES.md` records that the named-agent registry is **not** prompt-neutral, and that any new `state.<slot>` needs a converter branch.