mirror of
https://github.com/mongodb/agent-skills.git
synced 2026-09-18 21:15:11 +08:00
main
62 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 kanchana-mongodb
|
1e72df255e |
feat(mongodb-search-and-ai): add Automated Embedding, refine skill after review (#57)
* feat(mongodb-search-and-ai): add Automated Embedding, refine skill after review Add Automated Embedding (autoEmbed) support and polish the skill following a full /review-skill pass (structural validation + LLM scoring + live cluster validation against sample_mflix.movies). Changes: - SKILL.md: add "Automated Embedding" search type, cluster-tier check step, Core Principle #6 (accessible language), condense $regex/$text anti-pattern, trim redundant "Remember" section; restore license/metadata frontmatter to match sibling skills. - references/automated-embedding.md: new reference for autoEmbed index types and text-query (query vs queryVector) syntax. - references/vector-search.md: consolidate pre/post-filter docs, fix dead TOC anchor (#filter-fields), remove duplicated performance notes and score caveats. - references/hybrid-search.md: $rankFusion (8.0+) / $scoreFusion (8.2+) updates. Review outcome: structural validation passed; SKILL.md overall 3.83 -> 4.0, reference aggregate 4.4 -> 4.6, no dimension below 3. Live-validated the highest-novelty autoEmbed query/queryVector distinction against a live Atlas cluster. * test(mongodb-search-and-ai): add Automated Embedding evals + iteration-1 results Add three Automated Embedding eval cases (ids 16-18) covering the autoEmbed index type, plain-text query field (vs queryVector), and model selection (voyage-4-lite for cost, voyage-code-3 for code). Add SUMMARY.md recording the iteration-1 skill-creator run: with_skill 95.2% vs without_skill 43.4% (+51.8%), Claude Opus 4.6, MongoDB MCP against sample_mflix. All 18 evals differentiate; the new autoEmbed cases show +75%/+100%/+75% deltas. * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * chore(mongodb-search-and-ai): sync plugin skill copies Regenerate the mongodb and mongodb-atlas plugin copies of the mongodb-search-and-ai skill from the canonical skills/ source so the plugin mirrors reflect the Automated Embedding + review refinements. Ran tools/sync-plugin-skills.ts; output is idempotent (no further drift). Fixes the failing sync-check CI job on PR #57. * fix(mongodb-search-and-ai): use valid mongosh syntax in autoEmbed index example Address Copilot review on PR #57: the createSearchIndex example used db.<collection>, which isn't valid mongosh syntax. Use db.collection to match the convention in the sibling reference files (placeholders stay inside strings). Regenerated plugin copies via tools/sync-plugin-skills.ts (idempotent). * AGSK-18 fixes version * fix(mongodb-search-and-ai): correct version gates to 8.3+ SME correction: $scoreFusion and self-managed Automated Embedding require MongoDB 8.3+ (previously documented as 8.2+). Normalize the $scoreFusion note in SKILL.md to use the "+" suffix for consistency. Regenerated plugin copies via tools/sync-plugin-skills.ts (idempotent). * fix(mongodb-search-and-ai): address Copilot review feedback - Add "Automated Embedding" to the workflow proceed-gate in SKILL.md so Automated Embedding requests don't dead-end after the Cluster Check. - Make self-managed prerequisites edition-agnostic: drop "Community Edition"/"Enterprise Edition" qualifiers in automated-embedding.md and SKILL.md; key the requirement on MongoDB 8.3+ with mongot + Voyage AI. Regenerated plugin copies via tools/sync-plugin-skills.ts (idempotent). * fix(mongodb-search-and-ai): correct rate-limit unit in autoEmbed upgrade note Address Copilot review on PR #57: the free->paid upgrade note said "667x higher TPM", but 667x is the RPM increase (3 -> 2,000 RPM). Reword to state the RPM change explicitly and note TPM rises correspondingly, so it matches the rate-limit tables above. Regenerated plugin copies via tools/sync-plugin-skills.ts (idempotent). * docs(mongodb-search-and-ai): note Atlas prerequisites can change Address Copilot review on PR #57: the Automated Embedding cluster tier and autoscaling prerequisites were stated as absolutes. Add a brief note in SKILL.md and automated-embedding.md to confirm current requirements in the Atlas UI or official docs, since these can drift across releases. Regenerated plugin copies via tools/sync-plugin-skills.ts (idempotent). * docs(mongodb-search-and-ai): cover item-to-item similarity for autoEmbed Address jeff-allen-mongo's review on PR #57: the decision framework and Automated Embedding query guidance didn't handle "given item A, find similar items" requests. Since autoEmbed $vectorSearch.query only accepts a plain text string (no raw-vector read path to reuse a document's stored embedding), add: - A clarifying Discovery question in SKILL.md (free text vs. similarity to an existing item). - An "Item-to-Item Similarity" callout in automated-embedding.md showing how to resolve the source item to its indexed text field and pass that as the query (with a self-exclusion filter), plus a TOC entry. Regenerated plugin copies via tools/sync-plugin-skills.ts (idempotent). * docs(mongodb-search-and-ai): simplify Automated Embedding autoscaling note Condense the M10+ dedicated cluster prerequisite to "enable storage autoscaling" and align the Item-to-Item Similarity TOC entry as a top-level bullet. Regenerated plugin copies via tools/sync-plugin-skills.ts (idempotent). * docs(mongodb-search-and-ai): simplify autoEmbed auto-scaling prerequisite Condense the Atlas Clusters prerequisite to "M10+ dedicated clusters require storage auto-scaling to be enabled", dropping the detailed max-tier table and explanatory paragraph. Fix subject/verb agreement in the staleness note. Regenerated plugin copies via tools/sync-plugin-skills.ts (idempotent). * docs(mongodb-search-and-ai): address Copilot review nits - SKILL.md: standardize on "auto-scaling" (hyphenated) to match references/automated-embedding.md and the Atlas UI spelling. - references/vector-search.md: remove trailing whitespace on the pre-filtering Definition line. - references/hybrid-search.md: format all referenced filenames in the Scope paragraph as inline code for consistency. Regenerated plugin copies via tools/sync-plugin-skills.ts (idempotent). * test(mongodb-search-and-ai): expect cluster check in autoEmbed evals Align Automated Embedding eval cases 16-18 with the updated skill workflow: each expected_output now requires a cluster check before index creation - confirm Atlas (all tiers incl. M0) vs self-managed (8.3+ with mongot + Voyage AI key), and call out the M10+ dedicated storage auto-scaling prerequisite. * chore(mongodb-search-and-ai): sync plugin mirrors for auto-scaling wording Propagate the canonical SKILL.md edit ("M10+ without storage auto-scaling") to the four plugin copies via tools/sync-plugin-skills.ts. Fixes the failing Plugin Skills Sync check caused by mirror drift. * fix(mongodb-search-and-ai): apply review suggestions and sync plugin mirrors Co-authored-by: kanchana-mongodb <54281287+kanchana-mongodb@users.noreply.github.com> * docs(mongodb-search-and-ai): address review feedback Route search-type selection from step 2 and defer prerequisites to reference files, per PR review. - SKILL.md: merge search-type routing into "Determine Search Type" (step 2); remove standalone Cluster Check / Version Check / Consult Reference Files steps. Each search type links directly to its reference file and defers prerequisite verification to that file. - automated-embedding.md: move cluster/self-managed prerequisites and the M10+ auto-scaling + Voyage AI fallback behavior into Prerequisites; drop "Atlas UI" wording; fix inaccurate "no vector pipelines required" intro; combine intro+scope; remove decorative "---" rules; drop the duplicated "When to Use" routing block; document optional filter type/path fields. - vector-search.md / hybrid-search.md: combine intro + scope to reduce redundancy; move hybrid version-gate fallback behavior into hybrid-search.md. - evals.json: fix missing comma (invalid JSON). - Regenerate plugin skill mirrors via tools/sync-plugin-skills.ts. * docs(mongodb-search-and-ai): fold in remaining Copilot suggestions Address open Copilot review comments on the Automated Embedding reference, then regenerate the plugin mirrors. - automated-embedding.md: use a concrete `db.movies` collection name in the createSearchIndex and item-to-item examples (instead of `db.collection` / `db.<collection>`) for consistency with sibling reference files; use the modern `{ projection: { plot: 1 } }` findOne options form; add a note that `_id` is always available as a filter field on an autoEmbed index (no need to declare it) for the self-exclusion filter. - Regenerate plugin skill mirrors via tools/sync-plugin-skills.ts. * docs(mongodb-search-and-ai): clarify filter-field guidance Address Copilot re-review comments on automated-embedding.md. - Reword the self-referential filter-field inline comment to explain the purpose (index fields as filters to enable pre-filtering / scoped search) instead of "recommended for filter fields". - Clarify the `_id` note so it no longer contradicts the Query Parameters table: filter fields must generally be indexed as type "filter", with `_id` called out as the one implicit exception; add the same caveat to the table's `filter` row. - Regenerate plugin skill mirrors via tools/sync-plugin-skills.ts. * Update MCP config description in SUMMARY.md Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
dylan-good-e
|
0decd50318 | chore: narrow default CODEOWNERS to @mongodb/devdocs (#65) | ||
|
cory
|
d0129e97a3 |
feat(qa): publish actionable eval diagnostics (#63)
* fix(qa): generate non-null eval reports Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(qa): publish per-case eval diagnostics Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(qa): allow required project discovery Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(qa): correct ASP trajectory assertions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(qa): align S3 trajectory expectation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(evals): accept precise clarification for missing inputs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(ci): isolate report validation failures Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(evals): align ASP cases with skill guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(evals): define ASP mutation battery Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor(qa): narrow diagnostics PR scope Remove unrelated eval and metadata changes while retaining QA workflow diagnostics and hardening. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(qa): harden report publication Avoid shell word splitting when iterating skills and emit only explicitly typed public report fields. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
cory
|
8ada610346 |
Bump skill-gate to v0.1.1 (#64)
Use the released symlink-safe scanner in the skill-gate workflow.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
cory
|
47cc46148f |
proposal: functional-grading fields + schema for testing/*/evals.json (draft) (#55)
* proposal: optional functional-grading fields + schema for testing/*/evals.json
Adds optional seed/functional_check/trajectory_checks to the eval-case format plus a JSON
Schema and a hermetic fixtures/ convention, so the automated Inspect QA harness can grade a
skill against a real (local) MongoDB — complementary to the existing setup-slow-queries.ts /
skill-creator live-Atlas workflow, not a replacement. Fully backward compatible: all fields
optional; all current cases validate unchanged.
- testing/evals.schema.json: JSON Schema (draft 2020-12) for evals.json (current shape + the
new optional functional fields).
- testing/validate-evals.mjs: ajv validator to wire into CI (left for the team to hook into
validate-skills.yml).
- testing/mongodb-query-optimizer: a sample functional case (id 20, ESR compound index, graded
on the query plan) + its hermetic fixture (seeds/seeded_events.js).
See PR description for the parallel-tiers-vs-converge question.
* ci: wire eval-case schema validation into CI (validate-eval-cases.yml)
Don't leave the wiring to reviewers. Adds a testing/**-scoped GitHub Actions workflow that
installs the validator deps and runs testing/validate-evals.mjs, so a malformed evals.json
(incl. the optional functional_check/trajectory_checks) fails in CI where it's authored.
- testing/package.json: ajv + glob devDeps + a validate:evals script.
- validate-evals.mjs: use Ajv2020 (schema is draft 2020-12; default Ajv is draft-07) and
resolve paths relative to the file (CWD-independent).
Verified locally: npm install --prefix testing + node testing/validate-evals.mjs → all 7
evals.json (94 cases) valid, exit 0.
* ci: commit testing lockfile + use npm ci for deterministic validator deps
* feat(testing): extend eval schema for called_successfully trajectory checks
Adds "called_successfully" to trajectory_check.type's enum and a
matcher.success property, mirroring the same schema extension in the
Inspect harness repo (10gen/agent-skills-evals). Without this, an eval
case authoring a called_successfully or success: check would fail schema
validation -- the harness gained the capability (joining a tool call to
its result so "attempted" and "succeeded" are distinguishable) before the
schema was updated to let a case actually use it.
Backward compatible: all existing cases validate unchanged, since both
additions are optional.
* test: relocate DB seed fixtures from agent-skills-evals into evals/fixtures/
Fixtures backing functional_check cases (seeded_movies.js, seeded_nlq_lift.js,
seeded_sample.js for mongodb-natural-language-querying; seeded_movies_search.js for
mongodb-search-and-ai) were living in agent-skills-evals/inspect/seeds/, disconnected from
the evals.json cases that reference them by name. This is where an author is already
working when authoring a functional_check case, so the fixture now lives in the same PR/
review as the case that depends on it. mongodb-query-optimizer's seeded_events.js already
had an (identical, unused) copy here from an earlier pass; no change needed there.
agent-skills-evals now resolves fixtures from this location by default (SEED_BASE_DIR),
mirroring how SKILL_BASE_DIR already resolves the skill content itself from this repo.
* test: relocate mutation battery config from agent-skills-evals
Mirrors 8351f8e's seed-fixture move: invert_rule/key_reference config for the mutation
battery was living in agent-skills-evals/cases/qa/<skill>/mutation.yaml, disconnected from
SKILL.md and evals.json. Now lives alongside evals.json, so an author authoring/updating a
skill's mutation battery works in this repo, in one PR, without a separate checkout.
* Draft mutation-battery CI workflow (workflow_dispatch only)
Not yet wired to push/schedule triggers -- pending the Grove-secret
custody review and extending the agent-skills-evals GitHub App's
installation to this repo. See in-file header for details.
* Add asset-existence check and answer-echo lint to eval-case validation
validate-evals.mjs now confirms every case's files[]/seed reference a real
path, instead of failing deep inside the QA harness (case_source.py /
seed_db.py) with no hint which PR broke it.
lint-item-echo.mjs flags eval items whose authored text (prompt/
expected_output/expectations) overlaps heavily with their own skill's
SKILL.md/references -- i.e. items that test whether the agent can quote
the guidance back rather than apply it. Threshold is calibrated from the
corpus's own cross-skill null distribution rather than a guessed constant.
Advisory (reports, doesn't block) until checked against a few real PRs.
* Draft per-PR QA screen-check workflow (workflow_dispatch only)
Tier 2/3 of DESIGN.md's tiering -- runs skill-off then skill-on (k=1) for
a changed skill and posts a sanitized PR comment via
agent-skills-evals/inspect/screen_report.py. Same two blockers as
qa-mutation-battery.yml (Grove-secret custody review; GitHub App
install-scope extension), so pull_request stays commented out in favor
of workflow_dispatch until both land. Tier 1 (schema/lint/drift/echo) is
already live elsewhere and this workflow doesn't duplicate it.
* Correct GitHub App requirement in draft workflow comments
Cross-repo checkout via actions/create-github-app-token only needs the
App installed on the TARGET repo (agent-skills-evals, already true) --
not on the repo the workflow runs in. What these drafts actually still
need is AGENT_SKILLS_EVALS_APP_ID/_PRIVATE_KEY as Actions secrets on this
repo, unrelated to the App's installation scope.
(The App genuinely does need agent-skills in its installation for a
different, unrelated consumer -- agent-skills-evals's T1 GitHubReportWriter,
which commits eval-report.json/md there directly.)
* fix(ci): make the non-T1 eval jobs runnable and the echo lint honest
qa-mutation-battery ran the whole battery at --epochs 1. This repo's own noise
measurement says that cannot answer anything: at k=2 a NULL control (zero damage)
produced a per-item S_i of 0.114, above the 0.10 gate threshold, and only at k=5
did the floor fall to 0.029. mutation_report correctly suppresses the MDD in that
regime, so the job spent a full battery of model calls per skill to publish
"unmeasurable". Default is now 5, exposed as a workflow input with the reasoning
in its description.
Also in that workflow: every matrix job pushed its own baseline commit to
agent-skills-evals main, so with fail-fast:false the first would win and the rest
fail on non-fast-forward -- a 7-skill census quietly losing most of its results.
Reports are now artifacts, committed once by a `publish` job with rebase-retry.
Plus timeout-minutes (~11 mutants at k=5 vs. a 6h default), a concurrency group,
MAX_SAMPLES 2 (it is a CONCURRENCY cap and each sample spins its own agent+mongo
pair -- 4 is 8 containers on a 2-vCPU runner), `grep || true` so a no-match under
pipefail yields [] instead of failing the job, and candidate names intersected
against ALL_SKILLS so testing/skills-boundaries cannot become a matrix entry.
qa-pr-screen: documents fork PRs as a third blocker before the pull_request
trigger goes live. This repo is public and takes community contributions, and a
fork run gets no secrets -- so every external PR would get a red X on an
explicitly advisory check. Handled with a fork-notice job and same-repo scoping;
pull_request_target is the alternative that does get secrets, and running an
untrusted PR head with credentials needs a maintainer gate, which is a decision
to make rather than a default to fall into. Also replaces `always()` (which would
evaluate fromJSON('') after a failed detect), adds a concurrency group so two
pushes cannot double-post the marker comment, and passes --model explicitly since
a drift from screen_report's default surfaces as "no qa_runs doc found".
lint-item-echo claimed its threshold was "calibrated from the corpus's own
cross-skill null rather than a guessed constant". Measured: the null is p99=0.000
(two skills essentially never share an 8-gram), so max(p99, 0.15) always resolved
to the constant it said it was avoiding. Comment now says what is true, and the
floor is named as the operative threshold.
More substantively, the lint pooled prompt and expected_output into one score.
Those mean opposite things -- overlap in the prompt is leakage, overlap in the
expected answer is often just a correctly-stated answer -- so pooling mostly
measured the second and reported it as the first. Now split per field, which
surfaces one finding the pooled version missed. Adds the co-movement check ported
from the private echo.py: an edit that RAISES an item's containment is teaching to
the test, detectable from the diff alone with no held-out set, which makes it the
highest-value check available to a per-PR job. Wired to --base in CI.
Smaller: both validators now report a bad JSON file with its filename instead of
an unhandled stack trace; the echo lint fails loudly when skill_name matches no
skills/ directory, rather than reporting a green "no echoing items" over an empty
corpus; the --strict TODO is dated and owned. New echo-thresholds.json and the
schema's x-fixtureContract single-source the constants and the seed/fixture
convention that agent-skills-evals also resolves against at run time.
* Add qa-eval-status and qa-eval-check workflows
Promotes the reference workflows from agent-skills-evals/docs/ (the T2
QA eval flow's PR-gate half) into real, live workflows here -- the last
item from that flow's "not yet implemented" checklist that was mine to
do. Renamed to Title Case and bumped actions/checkout to v6 to match
this repo's existing workflow conventions; otherwise unchanged from the
reference.
These post a `qa-eval` status context but don't block merges on their
own -- main has no branch-protection required-status-checks configured
today, so this is visible-but-non-enforcing until someone separately
adds "qa-eval" as a required check.
* Pin GitHub Actions to commit SHAs (Semgrep github-actions-mutable-action-tag)
14 findings on PR #55, all the same rule: a mutable major-version tag
(@v6, @v4, @v2) can be silently repointed by the action owner (or an
attacker who compromises their account) without this repo's history
changing at all -- the cited precedent is the trivy-action and
kics-github-action compromises. Pinned every actions/checkout,
create-github-app-token, setup-uv, setup-node, upload-artifact, and
download-artifact reference to its resolved commit SHA, with a
# vX.Y.Z comment for readability. Verified each SHA against the
GitHub API before using it (resolving through the annotated-tag
indirection where a tag object's own SHA isn't the commit SHA, e.g.
astral-sh/setup-uv). No behavior change -- same versions, just pinned.
* fix(ci): derive the skill list from disk, and don't claim co-movement on a new skill
Both found by walking mongodb/agent-skills#50 (a fork PR adding mongodb-laravel)
through this infra.
The ALL_SKILLS allowlist I added was a literal list of today's seven mongodb-*
skills, used to filter path-derived candidates. A skill added since -- exactly
what a community contribution is -- was not in it, so detect silently dropped it:
the PR-triggered screen would have screened nothing, and the battery's census
would have skipped it. That is worse than the deep harness failure the allowlist
replaced, because a silent skip and "no work to do" look identical.
Now derived from skills/*/ (any directory with a SKILL.md). Because pull_request
checks out the merge ref, a skill ADDED by the PR is already on disk during
detect, so it gets screened. Verified against the real tree (derives the 7,
matching skills/*/) and against a simulated merge ref containing mongodb-laravel:
kept, with skills-boundaries still dropped.
Two related fixes: qa-pr-screen now emits the ::notice for dropped candidates (it
was the only one staying silent about them), and a typo'd workflow_dispatch input
now fails with the available list rather than running an empty matrix that reports
success.
Co-movement asks whether THIS PR's guidance edit moved an item's overlap. For a
PR that adds a skill there is no before-state, so every item trivially measures
0 -> n and would be reported as a rise that never happened. readCorpusAtRef now
returns anyPresent alongside the text -- "" is ambiguous, and a file that existed
and was empty is a real before-state while a skill that did not exist is not --
and co-movement skips those skills with a stated reason. An item that copies a NEW
skill's text is still echoing; that is the static check's finding to report, in
its own words.
Verified both paths in a reconstructed merge state of #50: the new skill is
skipped with the reason printed, and appending an item's prompt into an EXISTING
skill's SKILL.md still flags co-movement (0.00 -> 1.00). #50 itself passes schema
validation and produces no echo findings.
* docs(testing): explain why only two skills have a mutation.yaml
2 of 7 reads as arbitrary, and it isn't. Documented in the canonical place
(mongodb-query-optimizer/evals/mutation.yaml's header, which schema-design's file
already points at) so it survives past the PR that introduced it, and summarised in
PR #55's description.
Four points, in the order a reader needs them:
- Absence of the file does NOT mean the skill is unmutated. Every skill gets the
nine SKILL.md operators with no config at all, and one with references/ gets two
more. Verified: mongodb-natural-language-querying, which has neither references/
nor a mutation.yaml, materialises 9 of 9 mutants including real damage
(truncate_body 0.75 takes SKILL.md from 9,626 to 2,662 bytes).
- What the file adds is invert_rule, the operator closest to a real guidance
regression -- document still well-formed, advice now wrong.
- That is only worth authoring where the inverted advice is plausible-but-wrong
rather than incoherent, AND the skill's own cases turn on the rule. Fail the
second and S_i = 0, indistinguishable from "the skill isn't load-bearing here".
mutate_skill raises on a find-string that misses; nothing catches an inversion
that turns the doc into nonsense.
- These two satisfy both and are the two the battery has been run against:
query-optimizer's find-strings were caught 4x by the no-op raise while being
authored, and schema-design is the skill with measured lift (0.767 vs 0.100 at
k=5), so at the time it was the only one where mutation could show anything.
Also records what blocks the other five -- most of their rule-shaped text is
PROCEDURAL ("always consult the reference file", "always warn about billing"), and
inverting process does not change advice the graded output reveals -- and names the
follow-up order: connection (its cases assert singleton-client and maxPoolSize
directly), then search-and-ai, then atlas-stream-processing. mcp-setup and nlq may
never warrant one.
Comment-only; all five existing inversions re-verified to still hit their live files
(3 + 2 invert_rule mutants materialise).
* docs(ci): record that the Actions qa-eval workflows are superseded by Evergreen
The fork-PR problem is not an independent blocker -- it is a consequence of where the
eval executes. If a mongodb/agent-skills Evergreen project runs the QA eval, Evergreen
posts the status under its own GitHub App (which works on fork PRs, unlike a fork's
read-only GITHUB_TOKEN) and gates outside-org patches natively. At that point these
Actions workflows are superseded.
Recorded in their headers because the failure mode is leaving BOTH in place: someone
adds the Actions `qa-eval` context to the ruleset's required checks, it never posts on a
fork PR, and with bypass_actors empty every external contribution wedges permanently --
while a green Evergreen check sits next to it. Whatever posts the status has to be the
single thing named in the ruleset, so the instruction is delete, not coexist.
qa-pr-screen gets the same note on its blocker 3, which reads as a live concern today
and will read as a stale one the moment Evergreen lands.
Comment-only.
* docs: address Copilot review comments on #55
All three were legitimate. Two are stale-comment fixes; one closes a real CI gap.
1. testing/validate-evals.mjs header claimed three things the code does not do: exit on
the FIRST violation (it reports every file, then exits 1), print the skill name (it
prints the file path plus ajv's instancePath), and install with `npm install` (CI uses
`npm ci`). Corrected, and the report-everything behaviour is now stated as deliberate,
since it is the better behaviour -- an author fixes the whole set in one pass instead of
rediscovering the next problem on each CI run. Verified by breaking two case files: two
failures reported, exit 1.
The `npm ci` correction matters beyond tidiness: a documented local command that
resolves different dependency versions than CI is a local command that can disagree
with CI.
2. validate-eval-cases.yml's paths filter omitted testing/package-lock.json. `npm ci`
installs from the lockfile, so a lockfile-only change -- a dependency bump, a regen --
alters what the workflow actually runs while leaving the workflow unrun. An ajv upgrade
that broke validation would have sailed past the check meant to catch it. Added.
3. seeded_events.js's header described an earlier design of its case: "functional case 37",
equality lookups on `userId`, and an index-EXISTENCE scorer. The actual consumer is case
20, whose functional_check is `type: explain` -- filter {type: 'purchase'} with sort
{ts: -1}, requiring an IXSCAN, forbidding an in-memory sort, and capping docsExamined at
1.1x returned. So the graded behaviour is whether the agent builds a compound index
serving BOTH the filter and the sort, not whether any index exists. Rewritten, including
the line-18 comment Copilot flagged as the same problem.
Also confirmed semgrep's two findings on this PR are already resolved by
|
||
|
github-actions[bot]
|
6ebabd86eb |
chore: prepare release v1.2.1 (#60)
Co-authored-by: gagik <17454623+gagik@users.noreply.github.com> |
||
|
Copilot
|
049d8b4652 |
fix: add missing version field to .agy-plugin/plugin.json manifests (#59)
Co-authored-by: gagik <17454623+gagik@users.noreply.github.com> |
||
|
Nikola Irinchev
|
03740fc609 | chore: add plugin readmes (#56) | ||
|
Dachary
|
2fd451405f |
Add skill-gate CI workflows (#51)
* Add skill-gate CI workflows
* Change workflow order to mitigate code execution risks
* Rename skill-gate-required.yml to skill-gate-status.yml
* Fetch Grove key from AWS Secrets Manager via OIDC (ENTSEC-5465, ART-41)
Replace the GitHub environment secret (GROVE_API_KEY) with AWS Secrets
Manager retrieval: assume an OIDC-scoped IAM role (agent-skills-grove-readonly,
docs AWS account 683862839917) and pull the key at runtime into
ANTHROPIC_API_KEY. The Grove key no longer lives in GitHub repo settings.
- permissions: add id-token: write for AWS OIDC role assumption
- configure-aws-credentials@e6de0542 (#v6.2.3) and
aws-secretsmanager-get-secrets@a9a7eb4 (#v2.0.10), both SHA-pinned
(public repo + id-token:write → mutable tags are a supply-chain risk)
- role trust scoped to repo:mongodb/agent-skills:environment:skill-gate
so only this gated environment can assume it
- drop ANTHROPIC_API_KEY from the scan step env; it is now injected by
the Secrets Manager step and flows through to the scan shell
AWS-side prereqs (already in place): secret agent-skills/grove in
us-east-1, role agent-skills-grove-readonly with GetSecretValue on that
secret ARN. GitHub-side: AWS_ROLE_ARN env secret + GROVE_BASE_URL env var
in the skill-gate environment.
* Fix scan loop: restore `read` in while-IFS read line
Previous commit dropped `read` from `while IFS= read -r dir`, which would
break the per-skill scan loop. No behavioral change beyond that fix.
* Split skill-gate into unprivileged checkout + privileged scan jobs
The issue_comment trigger is privileged, so doing the PR-head checkout in the
same job that holds the Grove credentials is what CodeQL flags as an untrusted
checkout. Split the workflow so no job is both:
- collect: no environment, no id-token, no write scope. Does the data-only
sparse skills/ checkout of the PR head and uploads it (plus the changed-bundle
list) as an artifact.
- scan: environment: skill-gate + id-token: write, OIDC -> Secrets Manager for
the Grove key. Downloads the artifact and runs the pinned skill-gate binary
over the markdown. Never calls actions/checkout on a PR ref.
The OIDC trust policy (repo:mongodb/agent-skills:environment:skill-gate) still
matches, since scan is the job declaring the environment. Top-level permissions
dropped to {} with least-privilege grants per job.
ENTSEC-5465, ART-41
* ci(skill-gate): fail closed on gh api errors; non-fatal summary grep
Address Copilot review feedback (3 fail-open/fatality bugs in the workflow bash):
- skill-gate.yml `collect`: add `set -euo pipefail` and fetch the PR file list on
its own line (no `|| true`) so a `gh api` failure fails the step closed —
previously the `|| true` on the whole pipeline could mask an API error, leave
`changed_skills.txt` empty (count=0), skip the scan, and post AUTO-PASS. The
bundle-dir grep is now separate with `^skills/[^/]+/` so non-bundle files
like `skills/OWNERS.yaml` don't enter the list; its `|| true` stays because
count=0 is a legitimate "no skill changes" outcome.
- skill-gate-status.yml `seed`: same fail-open shape — `gh api` failure could
silently seed `skill-gate=success` ("No skill changes") and allow merge with
no scan. Split the gh api call from the grep -c so an API error fails the
seeder closed; the bundle-dir grep -c keeps `|| true` (no match = touched=0,
legitimate).
- skill-gate.yml `scan`: append `|| true` to the `grep -v '^::'` summary filter
so it can't fatal the step under `set -e` when `$out` is all annotations or
empty (a successful scan must not fail the job on cosmetic summary filtering).
Co-Authored-By: Claude <noreply@anthropic.com>
* ci(skill-gate): serialize comment-triggered runs; document fork + size expectations
Addresses review feedback on the PR:
- cancel-in-progress: true -> false. Every issue_comment creates a run in
the per-PR concurrency group (the /skill-gate filter is a job-level if:),
so an unrelated comment could cancel an in-flight scan and leave the
required check pending. Serializing is safe: the verdict posts to the
head SHA resolved at trigger time, and the seeder re-seeds pending on
any new head.
- Note the changed-files API ceiling (3,000 files/PR) at both call sites.
- CONTRIBUTING.md gains a "Pull requests" section: branch-based workflow
expected (fork PRs merge only after a maintainer runs the gate), and
PR-size expectations relative to that ceiling.
* Switch Grove key from AWS Secrets Manager to GHA environment secret
Replace the AWS OIDC + Secrets Manager flow with a GitHub Actions
environment secret stored in the `skill-gate` environment. The two-job
split (untrusted checkout in `collect`, credentials in `scan`) is
preserved — the key still never shares a runner with PR content.
- Remove `id-token: write` permission (no AWS OIDC)
- Remove `aws-actions/configure-aws-credentials` step
- Remove `aws-actions/aws-secretsmanager-get-secrets` step
- Read ANTHROPIC_API_KEY from secrets.GROVE_API_KEY (skill-gate env secret)
- Update comments to describe the GHA-secret flow
Secrets to add (skill-gate environment):
- GROVE_API_KEY: the Grove API key
Existing var (unchanged):
- GROVE_BASE_URL: Grove base URL (repo var)
* ci(skill-gate): address review — merge jobs, fail-closed verdict, concurrency
Address nirinchev review feedback on PR #51:
- Merge the `collect` (untrusted checkout) and `scan` (credentials) jobs into
a single `scan` job, per review request — the token's only write scope is now
`statuses:write`, so the blast radius that justified the split is minimal.
This reintroduces the CodeQL actions/untrusted-checkout/critical alert;
accepted with a header-comment justification (data-only sparse skills/
checkout, nothing from the PR is built/executed, only the pinned skill-gate
binary reads the markdown). The alert should be dismissed in the code-
scanning Security tab with that justification.
- Fix a fail-open in `Report verdict`: an empty verdict_code mapped to success.
Now branch on steps.changed.outputs.count — count=0 (no skill changes, scan
skipped) -> success; count!=0 with empty verdict_code (scan crashed) -> the
fail-closed *) branch.
- Add a concurrency block to skill-gate-status.yml (cancel-in-progress: true,
grouped by PR head sha) and explain why it differs from skill-gate.yml's
cancel-in-progress: false (issue_comment vs pull_request trigger).
- Remove the "See setup doc step 3." comment — it referenced an internal doc
not in this repo.
* ci(skill-gate): correct CodeQL rule IDs in header comment
The alerts that fire on the single-job checkout are the /high variants
(actions/untrusted-checkout/high and actions/untrusted-checkout-toctou/high),
not /critical. Comment-only; no behavioral change.
---------
Co-authored-by: cory <115956901+cbullinger@users.noreply.github.com>
Co-authored-by: Cory Bullinger <cory.bullinger@mongodb.com>
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Nikola Irinchev
|
b4ea8150a0 | chore: update mcp urls to point to prod (#53) | ||
|
Nikola Irinchev
|
999adf7b10 |
feat: add dual plugin infrastructure MCP-547 MCP-550 MCP-555 MCP-553 MCP-558 (#49)
Co-authored-by: Eduarda Scharnhorst <eduardascharnhorst@gmail.com> |
||
|
Paula Stachova
|
be846b4deb | feat(schema-design): Access pattern analysis MCP-476 (#37) | ||
|
Nikola Irinchev
|
9ea7387c7a |
Update mongodb-mcp-server version constraint (#48)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Dachary
|
bcd6518084 |
Chore: Skill metadata enrichment (#45)
* chore: align metadata with Agent Skills spec * chore: rename asp to mongodb-asp for namespace alignment * chore: add an OWNERS.yml manifest * chore: update PR template with OWNERS.yml checkbox |
||
|
Dachary
|
3393704c79 |
chore: add explicit permissions block to validate-skills workflow (#47)
Resolves CodeQL alert actions/missing-workflow-permissions on .github/workflows/validate-skills.yml. The workflow only reads repository content (checkout + validate), so contents: read is sufficient and applies the principle of least privilege. |
||
|
Dachary
|
c49e612059 |
chore: remove redundant CodeQL workflow (#46)
The repo's default code scanning setup covers both GitHub Actions and
JavaScript/TypeScript, which is a superset of this advanced workflow
(Actions only). When both configurations are enabled, GitHub rejects the
SARIF upload from the advanced workflow with:
CodeQL analyses from advanced configurations cannot be processed when
the default setup is enabled
Removing the now-redundant workflow file resolves the conflict and lets
default setup handle scanning.
|
||
|
dylan-good-e
|
f05bda6660 |
fix: update Atlas service account navigation steps (#43)
Correct the UI path for creating a service account — select org first, then use Project Identity and Access > Applications instead of Access Manager. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
MongoCaleb
|
a2bc4ec7f9 | two small file fixes discovered during eval testing and inspecting (#44) | ||
|
Nikola Irinchev
|
5cbde1c81a | chore: fixup the install instructions for Codex MCP-479 (#39) | ||
|
Nikola Irinchev
|
7f33694bf4 | chore: add release docs (#40) | ||
|
Rhys
|
160e31ea50 | feat(natural-language-querying): reduce Compass specific instructions and context size MCP-475 (#35) | ||
|
Nikola Irinchev
|
5f5e2b2edf | feat: add codex plugin support (#38) | ||
|
Rhys
|
10781ada29 | chore(schema-design): add evals for skill boundary and skill activation MCP-474 (#34) | ||
|
Paula Stachova
|
a9e846982a | chore(schema-design): add schema versioning eval MCP-466 (#32) | ||
|
Rhys
|
24529d9540 | chore(schema-design): add schema versioning eval MCP-467 (#33) | ||
|
cory
|
eba6ae3f67 | update README with new install instructions (#31) | ||
|
cory
|
29c95e0753 |
DOCSP-58329: update docs links (#30)
* docs: update docs links * docs: update Claude instructions |
||
|
Dachary
|
c47079f65e |
Switch skill-validator to base CLI (#28)
* Update 'review-skill' to use Claude Code for LLM-as-judge scoring * Update CI workflow and tool script to use base 'skill-validator' tool * Note minimum skill-validator version for claude-cli scoring |
||
|
Nikola Irinchev
|
0f9b1b16dc | chore: update readme and mcp config (#29) | ||
|
github-actions[bot]
|
b6fcb1a2d8 |
chore: prepare release v1.1.0 (#27)
Co-authored-by: nirinchev <2315687+nirinchev@users.noreply.github.com>v1.1.0 |
||
|
AyaanH123
|
337d6e14ba |
Add Search and AI Recommendations skill MCP-423 (#9)
Adds the mongodb-search-and-ai skill — an agent skill for implementing and optimizing Atlas Search (full-text), Vector Search (semantic), and Hybrid Search in MongoDB. Reference Files - lexical-search-indexing.md — Atlas Search index configuration: field types, analyzers, dynamic/explicit/typeSet mappings, storedSource, synonyms, and creating Search indexes on Views - lexical-search-querying.md — Atlas Search query patterns: $search vs $searchMeta, count, pagination with searchSequenceToken, returnScope, compound queries, autocomplete, facet, geo operators, sort optimization, and compound.filter vs $match guidance - vector-search.md — Vector Search index configuration, $vectorSearch query construction, filter fields, numCandidates tuning, ANN vs ENN, and Vector Search on Views - hybrid-search.md — $rankFusion (MongoDB 8.0+), $scoreFusion (MongoDB 8.2+), lexical prefilters using the vectorSearch operator inside $search, and choosing between approaches Key Design Decisions - Skill scope explicitly excludes Voyage API integration, auto-embedding, and reranking - Skill description includes text containment signals (contains, includes, appears in) to correctly route text search use cases away from the NLQ skill Testing - 15 evals covering lexical, vector, hybrid, and edge cases - Skills-boundary tests (natural-language-querying-vs-search-ai.json) with 25 test cases validating correct skill invocation vs mongodb-natural-language-querying |
||
|
Raschid J.F. Rafeally
|
e93609d42c |
feat(skills): mongodb-connection MCP-419 (#5)
* feat(skills): add mongodb-connection * perf(skills): reduce token count and contamination for mongodb-connection * perf(language-patterns): remove language examples * perf: ensure diagnostic questions and add correct timeout error/exception * Update skills/mongodb-connection/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update skills/mongodb-connection/references/language-patterns.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * perf(mongodb-connection): apply feedback from `/review-skill` * test(mongodb-connection): add evals * fix(mongodb-connection): update pool size patter for Node.js to modern versions * fix(connection): list missing timout errors/exceptions in Pool Exhaustion symptoms * fix(connection): update WiredTiger default ticket count. * fix(language-patterns): claim about default pool sizes across drivers * perf: improve clarify and token efficiency * fix(language-patterns): update Pymongo/Motor documentation * style(languate-patterns): normalize best practice verbiage * fix(monitoring): update ticket count information * fix: incorporate Alex's feedback * pref: WIP major rework and improvements * test: update evals * refactor: dropped separate language-patterns section. * test: add more constraints for better resut consistency * style: fix typos and copilot suggestions * Update skills/mongodb-connection/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix: remove stray tickets reference --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
David Jahn
|
09e4b740a8 |
MongoDB Query Optimizer Skill MCP-421
v1 skill to suggest query index or optimization |
||
|
Laura Zhukas
|
b8355cb152 | Add Atlas Stream Processing skill MCP-422 (#10) | ||
|
cory
|
1539c126e0 |
docs: update README TODOs (#26)
* docs: update README TODOs * fix: derp. markdown |
||
|
Nikola Irinchev
|
7016e50be5 | Remove trailing whitespace from readme.md (#25) | ||
|
github-actions[bot]
|
f5f148ffd3 |
chore: bump plugin versions to v1.0.0 (#24)
Co-authored-by: nirinchev <2315687+nirinchev@users.noreply.github.com>v1.0.0 |
||
|
Nikola Irinchev
|
163362d7c2 | Modify the setup skill to avoid credential handling (#23) | ||
|
Anna Henningsen
|
25c7ffa26a |
Schema design skill MCP-420 (#7)
* Copy schema-design files from romiluz13/mongodb-agent-skills * Update to Anthropic guidelines * Updates based on content review * Remove files called out as unnecessary * Expand key principles based on eval testing * Fixup: reformat * Asya CR comments * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Copilot CR * SKILL.md updates * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Reduce token count for reference files * Asya CR * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Copilot CR * Drop explicit schema validation entries * Merge document size concern files * Merge referencing reference files * Fixup: reduce SKILL.md size a bit, guide more actively towards when to use specific files * Bring under 25k limit * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Copilot CR * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Copilot CR * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * CR suggestion * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Polymorphic pattern does not need discriminator * Routing improvements * Dachary CR * Further CR * Update skills/mongodb-schema-design/references/pattern-schema-versioning.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Copilot CR --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Betsy Button
|
ff21280ffd | Add mongodb-query skill with testing infrastructure MCP-425 (#2) | ||
|
github-actions[bot]
|
c76ab57e68 |
chore: prepare release v0.5.0 (#22)
Co-authored-by: nirinchev <2315687+nirinchev@users.noreply.github.com>v0.5.0 |
||
|
Nikola Irinchev
|
76123de46f | Add MCP setup skill (#3) | ||
|
Nikola Irinchev
|
de7309eae7 | Add release workflows (#14) | ||
|
Nikola Irinchev
|
17e87f32d2 | Remove env variables from the config (#21) | ||
|
Nikola Irinchev
|
37f894761f | tweak plugin descriptions (#20) | ||
|
Dachary
|
1d66539b06 |
Add local validate script (#19)
* Add a local validate-skills script, check all skills in CI * Don't continue-on-warn - fail CI if merged skills fail validation * Anticipate GitHub Copilot comment about GH context injection vectors * Confirm skill-validator-ent is installed locally, or suggest install instructions * Fix comments from Copilot review |
||
|
Nikola Irinchev
|
63175effa2 |
Update CODEOWNERS for repository ownership (#13)
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Dachary <dc@dacharycarey.com> |
||
|
Nikola Irinchev
|
08d74d4fbd | Add gemini extension config (#8) | ||
|
Dachary
|
4db73a00ac |
Revert "Rework skill validator script (#11)" (#15)
This reverts commit
|
||
|
Nikola Irinchev
|
7a4ace96f2 |
Rework skill validator script (#11)
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |