## Summary
- Showcase packages serve health/smoke at `/api/health` and `/api/smoke`
(Next.js app router convention), but the smoke probe driver was hitting
bare `/smoke` and `/health` for the package shape, causing 17/17
false-red rows every tick for both `smoke:<pkg>` and `health:<pkg>`.
- Update `deriveUrls` in the smoke probe driver so package-shape
discovery uses `/api/smoke` and `/api/health`, matching what starters
already do for `/api/health`.
- Fix `deriveAgentUrl` to handle `/api/smoke` path prefix correctly.
- Update tests to assert the corrected URL paths.
All 17 packages already had `/api/health` and `/api/smoke` route files;
the issue was purely in the probe driver's URL construction for the
package shape.
## Test plan
- [x] `pnpm --filter @copilotkit/showcase-ops test` passes (788/788)
- [ ] CI green
- [ ] After deploy, PB `smoke:<package>` and `health:<package>` rows
turn green for all 17 packages
Showcase packages serve health and smoke routes under /api/
(Next.js app router convention: src/app/api/health/route.ts
maps to /api/health), but the smoke probe driver was hitting
bare /smoke and /health for the package shape. This produced
17/17 false-red smoke and health rows every tick.
Update deriveUrls to use /api/smoke and /api/health for the
package shape, matching the actual route paths. The starter
shape already used /api/health and is unchanged.
## Summary
- Delete `showcase/aimock/Dockerfile` (wrapper image no longer used)
- Remove aimock from `showcase_deploy` workflow matrix + paths-filter +
dispatch options
- Update `showcase/aimock/README.md` to reflect direct-pull architecture
- Keep fixture JSONs (`smoke.json`, `feature-parity.json`) and
`RAILWAY.md` (actively used)
## Why
Phase 3 of the showcase-aimock wrapper elimination. Railway now pulls
`ghcr.io/copilotkit/aimock:latest` directly with a `startCommand` that
fetches fixtures from GitHub raw URLs. The wrapper Dockerfile, its
`ghcr.io/copilotkit/showcase-aimock` GHCR image, and the workflow
trigger are dead code.
## Test plan
- [ ] CI green
- [ ] showcase-aimock Railway service still healthy (unaffected by this
PR -- no runtime dependency on deleted files)
- [ ] `workflow_dispatch` option list no longer shows aimock
## What does this PR do?
Fixes the guillotine cut where long messages hit the flat top of the
chat input and get sliced mid-line. Most visible in
`autoScroll=\"pin-to-send\"` mode (where the user reads at their own
pace and routinely lingers with a line butted against the input), but
applies to all scroll modes.
### The problem
`CopilotChatView` rendered the attachment queue + input as flex siblings
beneath the scroll area, so the scroll content stopped at the input's
flat rectangular boundary. The previously-shipped feather gradient
masked this visually but clashed with host themes whose `--background`
didn't match its hard-coded white / near-black —
[b621e96ee](https://github.com/CopilotKit/CopilotKit/commit/b621e96ee)
defaulted the feather to an empty div, which then revealed the
underlying layout bug.
### The fix
Wrap attachments + input in a single absolute-positioned overlay wrapper
at the `CopilotChatView` level. The scroll content now fills full height
and passes behind the rounded pill, matching ChatGPT's layout. The
scroll content's bottom padding reflects the measured overlay height so
the last line clears the pill when scrolled to the bottom.
Changes:
- `CopilotChatView.tsx` — replace flex-sibling attachments + input with
a single absolute overlay wrapper; move `inputContainerRef` onto the
wrapper so the `ResizeObserver` measures the full stack (attachments +
pill + disclaimer); add `inputContainerHeight` to scroll-content bottom
padding
- `CopilotChatAttachmentQueue.tsx` — add
`data-testid=\"copilot-attachment-queue\"` for test hooks
- `CopilotChatView.stories.tsx` — add a `PinToSend` story as
manual-verification scaffolding
- New test file `CopilotChatView.inputOverlay.test.tsx` with four tests:
overlay wrapper is absolute-positioned, attachments render above the
input inside the wrapper, welcome-screen input is NOT wrapped, scroll
content reserves `inputContainerHeight` as bottom padding
Not changed:
- `CopilotChatInput.tsx` public API (still accepts `positioning:
\"static\" | \"absolute\"`, same `bottomAnchored` flag, same
`containerRef`)
- `use-pin-to-send.ts` anchor math
- The `feather` slot itself — still empty-div default from
[b621e96ee](https://github.com/CopilotKit/CopilotKit/commit/b621e96ee);
hosts who want a themed fade supply their own via `scrollView={{
feather: ... }}`
- Welcome-screen input (stays inline)
- Angular parallel (follow-up if desired)
## Related PRs and Issues
- Builds on [#4158](https://github.com/CopilotKit/CopilotKit/pull/4158)
(pin-to-send anchoring/spacer/feather fixes)
- Builds on
[b621e96ee](https://github.com/CopilotKit/CopilotKit/commit/b621e96ee)
(feather defaulted to empty div)
## Checklist
- [x] I have read the [Contribution
Guide](https://github.com/copilotkit/copilotkit/blob/master/CONTRIBUTING.md)
- [ ] If the PR changes or adds functionality, I have updated the
relevant documentation
## Test plan
- [x] Unit tests — 4 new tests in
`CopilotChatView.inputOverlay.test.tsx` cover overlay structure,
attachment ordering, welcome-screen exclusion, and padding formula
- [x] Full `@copilotkit/react-core` suite (1153 tests) passes
- [x] Manual verification in Storybook (`UI/CopilotChatView → Default`,
`PinToSend`): content flows cleanly under the pill, last line reachable,
no guillotine cut
- [ ] Reviewer: check `WithSuggestions` story (padding formula switches
between `+4` and `+32`)
- [ ] Reviewer: check that `scrollView={{ feather: MyFeather }}`
override still works (the `slots.e2e` test covers this, but worth a
manual sanity check)
aimock is no longer a Docker-built showcase service. Railway
pulls the pre-built upstream image directly from GHCR. Remove:
- aimock from workflow_dispatch service options
- aimock paths-filter (showcase/aimock/**)
- aimock entry from ALL_SERVICES matrix
Railway now pulls ghcr.io/copilotkit/aimock:latest directly
via a startCommand that fetches fixtures from GitHub raw URLs.
The wrapper Dockerfile that copied fixtures into the image is
no longer needed.
Also updates README.md to reflect the new architecture:
- References Railway startCommand instead of Dockerfile
- Removes the Dockerfile entry from the fixtures listing
- Updates related workflow descriptions
⚠️ **Docs sync — MANUAL REVIEW REQUIRED**
This PR was auto-opened because the docs-sync script detected
showcase-local modifications overlapping with upstream changes.
The script attempted a best-effort 3-way merge:
- Where `git merge-file` produced a clean merge, the merged content was
written.
- Where `git merge-file` produced conflict markers, **upstream content
was written as-is** and showcase-local modifications were overridden.
**Manual review required.**
### Review items
```
Files where 3-way merge FAILED — upstream content written as-is, local modifications overridden. Manual review REQUIRED before merging this PR:
- docs/snippets/shared/generative-ui/tool-rendering.mdx
Files auto-merged via 3-way merge (clean, no conflict markers — still worth a glance):
- docs/content/docs/integrations/langgraph/generative-ui/state-rendering.mdx
```
### Source
- Upstream ref:
[`189c45fb4`](https://github.com/CopilotKit/CopilotKit/commit/189c45fb4)
- Workflow run:
https://github.com/CopilotKit/CopilotKit/actions/runs/24736203326
**Review before merging.** Auto-merge is intentionally disabled
for `needs-review` PRs — confirm the upstream-wins sections
preserve any intentional showcase-local divergence you want to
keep, then merge manually.
## Summary
- **Content writing pass** — em-dash overuse removed across ~37 docs
files; bridging text added before live demos, gifs, and snippets on ~15
pages; intro sentences added before code examples that directly follow
headings (11 files); hardcoded \"LangGraph Python\" reference fixed in
quickstart
- **Feature matrix** — built a real 17 backends × 12 features matrix on
the integrations overview page; fixed card underlines and added intro
text
- **IntegrationGrid fixes** — converted to a client component that
returns `null` on framework-scoped routes (was always visible); moved
the \"Choose your AI backend\" heading into the component so it
disappears with the grid; stripped the now-redundant heading from 21 MDX
files
- **Right-rail TOC** — new `DocsToc` client component with
IntersectionObserver scrollspy; heading extractor + slugifier in
`toc.ts`; H2/H3 ID overrides wired in `DocsPageView`; respects `hideTOC:
true` frontmatter (290 of 366 pages show the TOC)
- **Index page** — card and feature-grid styles aligned to site CSS
variables (`var(--border)`, `var(--accent)`, etc.) instead of shadcn
tokens; duplicate \"Explore by AI backend\" header removed
- **Generated data** — registry, status, constraints, demo-content
updated; `langgraph-python` set as quickstart default integration
The initial shell-docs content-import left 21 reference pages with missing
import statements: 8 component pages had empty `## Import` code fences, and
13 hook pages had `## Signature` fences whose leading `import { X } from
"@copilotkit/react-core/v2";` line was stripped.
This commit restores the import(s) in the Import and Signature fences only.
Usage examples and other code blocks are intentionally not touched.
- 8 components: fill empty Import fence with named (or default for
CopilotChatView) import + styles.css side-effect import
- 13 hooks: prepend import line(s) at the top of the Signature fence.
- useComponent: 3 imports (z, type ComponentType, useComponent)
- useRenderTool: imports injected into both Wildcard and Named overload fences
- useCopilotChatConfiguration: imports injected into both Provider and
hook ### Signature subblocks
Replace em-dashes in prose across ~37 shell-docs files with appropriate
punctuation (colons, semicolons, commas, parens, periods). Also fixes a
hardcoded "LangGraph Python" reference in quickstart.mdx bridging text,
adds a user-friendly placeholder when no framework is selected on snippet
pages, and makes heading code font size proportional rather than fixed.
- Update registry.json, demo-content.json, status.json, constraints.json,
docs-status.json across shell/shell-docs/shell-dojo
- Add integration="langgraph-python" default to quickstart InlineDemo so
the base unscoped page shows a demo instead of being empty
- Replace FeatureMatrix stub with a real server-rendered table reading from
registry — 17 integrations × 12 curated feature columns, sticky first column,
✓/— cells, integration names link to /{slug} landing pages
- Fix matrix links from /integrations/{slug} (404) to /{slug}
- Remove redundant "Feature comparison" section from index.mdx (FeatureMatrix
was a stub pointing to same destination as IntegrationGrid)
- Add .reference-content .not-prose a rule to suppress underlines on card grids
inside MDX without breaking prose link styling
- Add intro sentence to "Explore by AI backend" section
- Correct 3 mismatched feature IDs in MDX (generative-ui-tool-based ->
gen-ui-tool-based, frontend-tools-sync -> frontend-tools, reasoning ->
agentic-chat-reasoning) so FeatureIntegrations renders integration chips
- Remove stale 'matrix' from RESERVED_ROUTE_SLUGS (no app/matrix/ route exists)
- Replace broken /matrix links in IntegrationGrid and search-modal with /
since the matrix lives in shell-dashboard, not shell-docs
## Summary
One-line YAML fix aligning the `e2e-smoke` probe's emitted key prefix
with the rest of the system.
- `showcase/ops/config/probes/e2e-smoke.yml` line 56: `key_template:
"e2e-smoke:${name}"` -> `key_template: "e2e_smoke:${name}"`.
## Why
Every other reference in showcase-ops uses the underscore form:
- `kind: e2e_smoke` (same file, line 39)
- `DIMENSIONS` enum (zod schema — `e2e_smoke`)
- Alert rules filter on `dimension: e2e_smoke`
The YAML key_template was the lone hyphen holdout, so the producer wrote
`e2e-smoke:<slug>` rows while the dashboard filtered for `e2e_smoke`. PB
currently has 17 orphan hyphen rows and zero matches on the dashboard.
Driver code is untouched: `deriveSlug` splits on `:` regardless of
prefix, so no test changes required. All 788 showcase-ops tests pass.
Typecheck and build are clean.
## Test plan
- [x] `pnpm --filter @copilotkit/showcase-ops test` — 788/788 green
- [x] `pnpm --filter @copilotkit/showcase-ops typecheck` — clean
- [x] `pnpm --filter @copilotkit/showcase-ops build` — clean
- [x] `oxfmt --check` on edited file — clean
- [x] `oxlint showcase/ops/` — 0 errors (108 pre-existing warnings
unrelated)
- [ ] Post-deploy: confirm new `e2e_smoke:<slug>` rows appear in PB,
hyphen rows go stale
The YAML had key_template "e2e-smoke:${name}" (hyphen) but every other
reference — DIMENSIONS constant, alert rules, kind: e2e_smoke — uses
the underscore form. This mismatch caused the producer to write
e2e-smoke:<slug> while the dashboard filtered for e2e_smoke, leaving
17 orphan hyphen rows and zero matches on the dashboard.
Changing the YAML key_template aligns the emitted probe key with the
dashboard filter. Driver code (deriveSlug) splits on ':' regardless
of prefix, so no driver/test changes are required.
## Summary
- Adds `showcase/aimock/RAILWAY.md` — reconstruction reference for the
Railway service
- Documents image, startCommand flags, env vars, deploy recipe, and the
legacy Dockerfile cleanup path
- Supports Phase 2 of the aimock wrapper-elimination work
## Why
Railway service config (image + startCommand + env) previously lived
only in Railway's state. If Railway loses it, the Notion plan was the
only reconstruction source. This persists an authoritative copy in-repo
so any reviewer can rebuild the service from scratch.
Values (project ID, environment ID, public domain) were queried live
from Railway GraphQL at doc-writing time.
## Test plan
- Docs-only, no runtime impact
- Verify every referenced URL renders on GitHub after merge
- Commitlint + CI green
## Summary
The deployed showcase-ops probe pipeline has been writing `smoke:*`,
`health:*`, and `agent:*` rows for the 7 infra services declared in
`showcase/ops/config/probes/smoke.yml`'s `nameExcludes` list
(showcase-ops, showcase-pocketbase, showcase-shell,
showcase-shell-dashboard, showcase-shell-docs, showcase-shell-dojo,
showcase-aimock) — roughly 21 false-red PocketBase rows per 15-minute
tick.
## Why
The `railway-services` DiscoverySource wrapped its filter schema in an
outer `{filter: FilterSchema}` ConfigSchema, but the probe-invoker at
`src/probes/loader/probe-invoker.ts:184` hands the source
`cfg.discovery.filter ?? {}` — the filter CONTENTS directly, not a
wrapped object. The wrapper never matched the invoker's call shape, so
`cfg.filter` was always `undefined`, and BOTH `namePrefix` and
`nameExcludes` silently defaulted to `undefined` at
`railway-services.ts:234`. Every `showcase-*` service — user-facing
and infra alike — was enumerated and probed.
The fix drops the outer wrapper. `ConfigSchema` IS the filter block, so
`rawConfig` is parsed against it directly. This matches the documented
`enumerate()` contract in `probes/types.ts` ("the second argument is
`discovery.filter`") and the shape `pnpm-packages` has used all along.
## Row-count impact
- Before: 41 services × 3 dimensions = 123 probe rows/tick (7 infra
services × 3 = 21 false-red rows included).
- After: 34 user-facing services × 3 dimensions = 102 probe rows/tick.
The 7 infra services no longer cost a per-service Railway variables
query either (7 round-trips saved per tick).
## Test plan
- [x] Red: added `honours filter passed flat (invoker shape) — drops
all 7 infra services in smoke.yml` regression test in
`railway-services.test.ts`. Confirmed failing against pre-fix
code (returned all 9 services instead of 2).
- [x] Green: same test passes with the schema fix.
- [x] `pnpm --filter @copilotkit/showcase-ops test` → 788 tests pass.
- [x] `pnpm --filter @copilotkit/showcase-ops typecheck` clean.
- [x] `pnpm --filter @copilotkit/showcase-ops build` clean.
- [x] Pre-commit `test-and-check-packages` hook passes.
The railway-services DiscoverySource wrapped its filter schema in an
outer {filter: FilterSchema} ConfigSchema, but the probe-invoker calls
source.enumerate(ctx, cfg.discovery.filter ?? {}) — it passes the filter
contents DIRECTLY, not a wrapped object. At runtime cfg.filter was
always undefined, so namePrefix AND nameExcludes silently defaulted to
undefined and ZERO filtering happened.
Impact: all 7 infra services declared in smoke.yml's nameExcludes
(showcase-ops, showcase-pocketbase, showcase-shell,
showcase-shell-dashboard, showcase-shell-docs, showcase-shell-dojo,
showcase-aimock) produced smoke:/health:/agent: ProbeResults on every
15-minute tick. PocketBase carried ~21 false-red rows (7 services × 3
dimensions) across smoke:*, health:*, agent:*; after this fix the
discovery source filters them out and only the 34 user-facing
package+starter services emit probe rows (102 legitimate rows).
Fix: drop the outer wrapper. ConfigSchema IS the filter block — parse
rawConfig against it directly. Matches the documented enumerate()
contract in probes/types.ts ("config is discovery.filter") and the
shape pnpm-packages has used all along.
Tests updated: the prior discovery tests passed a {filter: {...}}
wrapped argument that only parsed because passthrough() swallowed the
misshape; they now pass the flat filter object the invoker actually
hands the source. Added one regression test that enumerates two real
showcase services plus all 7 infra services and asserts the seven are
dropped before the per-service env fetch (saves seven variables
round-trips per tick as a side benefit).
## Summary
Add a bounded rebase-and-retry loop to the `Commit registry updates`
step in `.github/workflows/showcase_capture-previews.yml` so the
devops-bot push to `main` can recover from concurrent pushes that landed
during the capture job.
## Why
The Showcase: Capture Previews workflow failed twice today (runs
[24799601181](https://github.com/CopilotKit/CopilotKit/actions/runs/24799601181)
at 13:10 PDT and
[24809331709](https://github.com/CopilotKit/CopilotKit/actions/runs/24809331709)
at 17:23 PDT), both at the `Commit registry updates` step with:
```
! [rejected] main -> main (fetch first)
error: failed to push some refs to 'https://github.com/CopilotKit/CopilotKit'
hint: Updates were rejected because the remote contains work that you do not
hint: have locally.
```
The capture step runs Playwright + ffmpeg against every starter and can
take ~30 minutes. During that window, normal PR merges to `main`
routinely land, so the auto-generated `Update preview URLs in registry`
commit loses the push race. Today's activity (showcase-ops swapover
landing a sequence of fixes) made the race visible twice in one
afternoon, but the bug is latent under any normal PR cadence.
The fix: if `git push` is rejected, `git pull --rebase` onto the latest
branch head and retry. Bounded to 5 attempts so a persistent failure
(conflict on `registry.json`, auth issue, etc.) surfaces rather than
looping forever. The commit contains only `registry.json` so rebase
conflicts should be rare; if one occurs, the loop exits and the job
fails loudly.
## Test plan
- [ ] CI runs the workflow on a subsequent `Showcase: Build & Deploy`
completion and the registry push lands on the first attempt
- [ ] Next time the capture job races against a concurrent merge, the
rebase loop kicks in and the push succeeds rather than exit-1'ing
- [ ] If a persistent push failure occurs (not a race), the job fails
after 5 attempts with a clear `::error::push failed after 5 rebase
attempts` annotation
## Summary
The `railway-services` DiscoverySource in `showcase/ops` passed
`environmentId` as an argument to `Service.serviceInstances` in its
project-level GraphQL query. Railway's current schema does not accept
that argument — every discovery tick 400'd with:
```
railway gql 400: Unknown argument "environmentId" on field "Service.serviceInstances".
```
Result: no probes ticked, PocketBase stopped receiving fresh rows, alert
pipeline starved of data.
Fix: drop the invalid field argument. The returned
`serviceInstances.edges` are already filtered by `environmentId`
client-side a few lines down (`svc.serviceInstances.edges.find((e) =>
e.node.environmentId === environmentId)`), so there is no semantic
change — the argument was dead weight that tripped GraphQL validation.
## Why
- Blocks `railway-services` from enumerating anything, which blocks
every downstream probe that fans out across Railway services
(image-drift, smoke, e2e-smoke, redirect-decom).
- Unblocks the fresh-redeploy of showcase-ops that carries the #4180
merge — that PR's fix is in the running image but never reachable
because no probes tick.
## Verification
Introspection confirmed the current shape of `Service.serviceInstances`:
- `Query { project(id: String!) }` exists and returns `Project`
- `Project.services` paginates to `ServiceEdge { node: Service }`
- `Service.serviceInstances` accepts no field arguments
- `ServiceInstance.environmentId` is present, so client-side filtering
is correct
Direct `curl` against `backboard.railway.com/graphql/v2` with the fixed
query returned 41 services for the CopilotKit showcase project, matching
the expected ~35 (17 starters + 17 packages + shell + aimock + a few
infra services).
## Test plan
New test: `omits environmentId arg from Service.serviceInstances
selection to match current Railway schema`
- Mocks a Railway endpoint that rejects any query whose
`serviceInstances` call includes parenthesised arguments with the
production 400 error body, and accepts the arg-less form.
- Red on pre-fix code: `DiscoverySourceBackendError: railway gql 400:
{"errors":[{"message":"Unknown argument \"environmentId\" on field
\"Service.serviceInstances\".",...}]}`
- Green after fix; full suite 743/743.
- [x] `pnpm --filter @copilotkit/showcase-ops typecheck`
- [x] `pnpm --filter @copilotkit/showcase-ops test` (743 passed, was 742
+ 1 new regression test)
- [x] `pnpm format`
- [x] `oxlint` clean on touched files (one pre-existing unused-import
warning left alone — not in this diff)
## Summary
Replace `RUN curl -LsSf https://astral.sh/uv/install.sh | sh` across 20
starter Dockerfiles with `COPY --from=ghcr.io/astral-sh/uv:latest /uv
/uvx /usr/local/bin/` to eliminate a latent pipe-swallow bug.
- 9 `examples/integrations/*/docker/Dockerfile.agent`
- 10 `examples/integrations/*/Dockerfile` (adk, agno, crewai-crews,
langgraph-fastapi, langgraph-python, langgraph-python-threads,
llamaindex, ms-agent-framework-python, pydantic-ai, strands-python)
- 1 `examples/showcases/scene-creator/agent/Dockerfile`
All 20 use the uv image COPY pattern — no `pipefail` fallbacks were
needed.
## Why
When astral.sh returns a 5xx, `curl -LsSf` fails with no stdout, but the
piped `sh` sees empty stdin and exits 0. The Docker layer "succeeds"
with no `uv` binary installed, and the next `RUN uv sync` then crashes
with `uv: not found` (exit 127). The root failure mode is invisible in
logs — you only see the downstream symptom.
This already bit the agno starter today in CI run `24809910399` when
astral.sh had a transient outage. Upstream is fine now, so this is a
**latent-bug cleanup, not a hotfix** — the intent is to make future
astral.sh outages a non-event.
The COPY-from-image pattern is uv's own recommended production pattern
(see https://docs.astral.sh/uv/guides/integration/docker/):
cache-friendly, no network call at build time, no pipe to swallow.
## Test plan
- [x] `grep -rln 'astral.sh/uv/install.sh' --include='Dockerfile*' .`
returns zero matches on this branch
- [x] Local `docker build -f
examples/integrations/agno/docker/Dockerfile.agent
examples/integrations/agno/agent` builds cleanly end-to-end (uv binary
present, `uv sync` completes)
- [ ] CI starter smoke / showcase deploy jobs build all 20 Dockerfiles
successfully
The `railway-services` DiscoverySource was passing
`environmentId` as an argument to `Service.serviceInstances`,
but Railway's current GraphQL schema doesn't accept it. Every
discovery tick 400'd with `Unknown argument "environmentId"
on field "Service.serviceInstances"` and no probes ran.
Drop the field argument; the returned instances are filtered
by `environmentId` client-side in the subsequent loop (existing
behaviour), so no semantic change — the bad arg was dead
weight that tripped validation.
Add a regression test that scripts Railway's exact 400 when
the old query shape is sent and asserts the source succeeds
against a schema-accurate mock. Verified against the live
Railway schema via introspection and a direct curl.
Replace `RUN curl -LsSf https://astral.sh/uv/install.sh | sh` across all
starter Dockerfiles with `COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx
/usr/local/bin/`. The curl|sh form has a pipe-swallow latent bug: when
astral.sh returns a 5xx, curl fails but `sh` gets no stdin and exits 0,
so the layer "succeeds" with no uv binary. A later `RUN uv sync` then
crashes with `uv: not found` (exit 127). This already bit the agno
starter today (run 24809910399) when astral.sh had a transient outage;
upstream recovered on its own so this is latent-bug cleanup, not a
hotfix.
Using the official uv image is uv's own recommended pattern: it's
cache-friendly, network-free at build time, and sidesteps the pipe
failure mode entirely.
Scope: all 20 Dockerfiles under examples/integrations/*/Dockerfile,
examples/integrations/*/docker/Dockerfile.agent, and
examples/showcases/scene-creator/agent/Dockerfile.
Verified locally: `docker build -f docker/Dockerfile.agent ./agent`
for agno succeeds against the new pattern.
## Summary
- Sanitize `errorDesc` before it flows into Slack templates: strip HTML,
script/style bodies (including entity-smuggled), Slack mrkdwn controls
(`<!channel>`, `<!here>`, `<@U…>`), backticks. Cap at 120 chars.
- Section-guard optional link fields in
`{smoke,agent,chat,tools}-red-tick.yml` so absent links emit nothing
instead of `<|smoke>`.
- Switch `{{signal.errorDesc}}` to `{{{signal.errorDesc}}}`
(triple-brace) since values are pre-sanitized; add `errorDesc` to each
dimension's `slackSafeFields` so rule-loader validation passes.
- Introduce cross-service aggregation: new `AggregationBucketStore`,
`CompiledRule.aggregation?`, YAML schema field, engine ingress branch,
composite-flush dedupe via `buildCompositeDedupeKey`.
- New `{smoke,agent,chat,tools}-red-fleet.yml` rules:
`groupBy=[dimension]`, `windowMs=120s`, `minMatches=3`. `<!channel>`
migrates from per-service red-tick rules to the fleet rules.
## Why
PR #4178 (L1-L4 coverage on 34 services) landed and caused an
18-per-tick fanout in `#oss-alerts` with entity-escaped HTML bodies and
empty link brackets. Channel was silenced by removing the webhook env
var on Railway as short-term triage; this PR is the durable fix.
## Test plan
- [x] `sanitizeErrorDesc` unit tests: 10 cases including entity-smuggled
script payload, Slack mrkdwn injection, length cap with U+2026 ellipsis.
- [x] Red-tick render tests: 9 cases across 4 YAMLs covering
missing-link emits-nothing, present-link renders, and errorDesc rendered
as literal characters (no HTML-entity double-escape).
- [x] Aggregation store tests: 8 cases covering threshold flush,
window-expiry flush, below-threshold non-fire, independent groupBy
buckets, composite dedupe across window expiries, bootstrap suppression,
TTL eviction.
- [x] Full `showcase/ops` suite green (731+ tests).
- [x] typecheck clean, lint clean, build clean.
## Rollout notes
- The first probe tick after this lands will exercise all four fleet
rules. Bootstrap suppression (15min from engine boot) delays first
composite by design.
- Existing `SLACK_WEBHOOK_OSS_ALERTS` env var was removed from Railway
`showcase-ops` during triage. Re-setting it (to `#oss-alerts` or a
personal redirect) activates this branch's output.
- Staging verification: no dedicated `showcase-ops` staging env exists;
this PR ships on "land + watch next real bulk deploy" as documented
manual verification.
## Parallel-branch coordination
A separate branch is modifying
`showcase/ops/src/probes/drivers/smoke.ts` for path-resolution. This
PR's `smoke.ts` surface is exactly 1 import + 8 errorDesc-assignment
wraps — rebase-friendly. Sanitization is unconditional; any new
errorDesc sites the parallel branch introduces should be wrapped the
same way.
The Promise branch of applyFlushResult set flushedAt optimistically
before the promise resolved, and the .then handler never reverted it
when the flush was suppressed. Since onAggregationFlush is async, this
broke the A5 "keep bucket live for re-threshold" contract in production.
The sync branch test did not exercise the Promise path. Added
bucket_stays_live_when_flush_suppressed_async to lock the invariant.
probeOne and probeAgent catch-blocks both assigned
`errorDesc: sanitizeErrorDesc(errorDesc)` where the local `errorDesc`
had already been sanitized on the line above. Double-sanitize is
harmless today because sanitizeErrorDesc is idempotent on the token
set we emit, but it would turn into a silent escape-on-escape
regression the moment the sanitizer grew a non-idempotent rule
(e.g. entity-encoding `&` → `&`). Drop the outer wrap in both
sites.
validateTripleBrace previously scanned only rule.template.text and
rule.on_error.template.text. A rule with a malformed
`{{{firstSignal.unsafe}}}` reference in aggregation.template passed
load validation and rendered the unsafe value at runtime via
Mustache.render inside onAggregationFlush — same Slack mrkdwn-injection
surface R21-a closed for on_error.template.
Pull aggregation.template into the same scan:
- firstSignal.* and lastSignal.* are aliases for signal.* in the
aggregation render context; normalize both prefixes to `signal.`
before applying the existing per-dimension slackSafe allowlist.
- `count` and `services` are engine-injected scalars (number / string)
and are always loader-known-safe — allow their bare form for symmetry
with the existing `{{count}}` usage in fleet rules.
Tests: rule-loader.test.ts adds
"rejects aggregation.template with triple-brace on an unsafe signal.*
field" under the new "validateTripleBrace covers aggregation.template"
describe.
A2 — Run link guard
Every L1–L4 red-tick alert used to render the dashboard Run link
unconditionally, so a tick with no runId (invariant probes, cron ticks
without a propagated workflow run) emitted a broken
`<https://dashboard/.../runs/|Run>` in every alert. Wrap the token in
`{{#event.runId}}…{{/event.runId}}` across all four red-tick YAMLs
(smoke/agent/chat/tools) so the entire "· <…|Run>" chunk disappears when
runId is absent. The Showcase dashboard link stays unguarded (always
valid).
A3 — smoke endpoint link from signal.url
The smoke-red-tick template referenced `{{{signal.links.smoke}}}` and
`{{{signal.links.health}}}` section-guarded on those fields. The driver-
emitted SmokeDriverSignal (probes/drivers/smoke.ts) has no `links`
object — it exposes `url` instead, the URL that was actually probed. So
the guard sections were always empty and the link never rendered. Swap
the template to reference `{{{signal.url}}}` directly and emit a single
"endpoint" link when present. The other three red-tick YAMLs never
referenced `signal.links`; leave them untouched.
SMOKE_SLACK_SAFE_FIELDS updated: drop `links.smoke` / `links.health`,
add `url`. errorDesc stays safe.
Tests:
- render-red-tick.test.ts: parametrize across all four YAMLs, assert
renders_no_run_link_when_runId_missing (output contains no
`/runs/|` or `/runs/ |`) and renders_run_link_when_runId_present
(output contains `/runs/run-abc|Run`). Add
renders_endpoint_link_from_signal_url to cover the smoke endpoint
link swap directly.
- rule-loader.test.ts: update the smoke-red-tick full-YAML tests to
seed `signal.url` and assert `<…|endpoint>` instead of seeding
signal.links.
Alert engine (handleStatusChanged):
- A1: call resolveTriggers on aggregated-rule ingress and skip when the
triggered set is purely red_to_green. Prevents recovery events from
ingesting into the fleet bucket and firing <!channel> on recoveries.
error transitions also skip ingest — aggregated rules don't support
onError today.
- B3: dispatchCronAlert returns early when rule.aggregation is set. A
rule ever declaring both cron triggers and aggregation would otherwise
render the empty top-level template or bypass aggregation entirely.
Alert engine (onAggregationFlush + drain wiring):
- A5: return "suppressed" when the bootstrap / rate-limit gate short-
circuits dispatch, so the store keeps the bucket live. Pre-fix a
bootstrap-suppressed threshold flush was silently lost — the bucket's
flushedAt was marked and post-bootstrap ticks couldn't re-trigger.
- A4: drain() is now async. Register a SIGTERM handler that awaits
drain() so in-flight buckets actually dispatch before process exit;
keep beforeExit + stop() as best-effort fire-and-forget (Node
beforeExit cannot truly await). Remove both listeners in stop() for
test-cycle symmetry.
AggregationBucketStore (aggregation.ts):
- drain() returns Promise<void> and awaits each onFlush invocation via
Promise.allSettled.
- OnFlushCallback returns void | "suppressed" | Promise<…>. When the
callback returns "suppressed", flushedAt stays null so the bucket stays
live for re-ingestion after the engine gate reopens.
- A7: groupBy is optional; absent/empty collapses the bucket key to
rule.id alone. Schema updated so AggregationSchema.groupBy is
z.array(string).optional() (pre-fix .min(1) forced a decorative
[dimension] that always resolved to one bucket anyway).
- B4: intra-key field separator is NUL instead of "&" so a groupBy
value containing "&" + "=" cannot collide with a distinct grouping.
buildBucketKey and buildCompositeDedupeKey both match.
Schema (rules/schema.ts):
- B1: remove unused aggregation.targets field. The engine has always
used rule.targets for aggregation dispatch; the schema field was
silently dropped.
Tests (alert-engine.test.ts):
- A1 engine-level: red_to_green on an aggregated rule does NOT dispatch,
green_to_red DOES dispatch (minMatches: 1, empty groupBy).
- A4: drain_awaits_onflush_promises — async onFlush resolves before
drain's promise resolves.
- A5: bucket_stays_live_when_flush_suppressed — onFlush returning
"suppressed" keeps the bucket live; a follow-up ingestion re-fires.
- A6: fix TDZ in bootstrap_suppression_respected — declare rule before
the onFlush closure that captures it.
- A7: no_groupBy_uses_ruleid_only_as_bucket_scope — three signals
across distinct dimensions all collapse into one rule-id bucket when
groupBy is empty.
- B4: bucket_key_collision_safe_with_special_chars — concrete collision
{b:"y&c=z"} vs {b:"y",c:"z"} is rejected by the NUL separator.
Fleet aggregation rules previously listed red_to_green alongside green_to_red
and sustained_red, which meant any recovery event ingested into the fleet
bucket and could fire a <!channel> aggregation alert on recovery. Remove
red_to_green from all four fleet YAMLs (smoke/agent/chat/tools) — the engine
ingress gate that actually skips recovery-only matches is added in the
subsequent commit. While here, drop the decorative groupBy: [dimension] on
every fleet rule: signal.dimension already partitions traffic to one
dimension per rule, so the groupBy expanded to undefined → "" at runtime
and silently collapsed to a single bucket per rule. The new schema treats
absent/empty groupBy as an explicit "single bucket per rule" contract.
Jordan Ritter