dotnet__skills/.github/aw/shared/devops-health.lock.md

DevOps Health Check — Compiled Knowledge

This document contains the health check catalog, fingerprinting rules, output templates, and operational guidance for the DevOps Daily Health Check agentic workflow.

---

1. Fingerprinting Rules

Every health finding MUST be assigned a deterministic fingerprint — a string ID derived from the finding's category and key attributes (but NOT timestamps, run IDs, or other ephemeral data). The same real-world issue MUST produce the same fingerprint on every run.

1.1 Pipeline Fingerprints

fingerprint = "pipeline:{workflow_name}:{job_name}:{failed_step}:{conclusion}"

• Normalize workflow_name by lowercasing and replacing spaces with hyphens
• Normalize job_name and failed_step the same way
• Same workflow + job + step + conclusion = same finding (even across different run IDs)
• A workflow that fails in a different step is a different finding
• For timeouts/cancellations: pipeline:{workflow_name}:{job_name}:timeout
• For aggregate failure rate (P5): pipeline:evaluation:failure-rate:{bucket} where bucket = "critical" or "warning"
• For scheduled cancellation rate (P6): pipeline:evaluation:schedule-cancellation:{bucket} where bucket = "critical" or "warning"

Examples:

Finding	Fingerprint
Evaluation workflow, evaluate job, "Run skill-validator" step failed	pipeline:evaluation:evaluate:run-skill-validator:failure
Evaluation workflow, evaluate job, "Build validator" step failed	pipeline:evaluation:evaluate:build-validator:failure
validate-skills workflow, validate job timed out	pipeline:validate-skills:validate:timeout
Evaluation failure rate > 30% across all branches	pipeline:evaluation:failure-rate:critical
Evaluation failure rate > 15% across all branches	pipeline:evaluation:failure-rate:warning
Evaluation scheduled cancellation rate > 60%	pipeline:evaluation:schedule-cancellation:critical
Evaluation scheduled cancellation rate > 30%	pipeline:evaluation:schedule-cancellation:warning

1.2 Infrastructure Fingerprints

fingerprint = "infra:{config_key}"
  where config_key ∈ { "no-codeowners", "no-dependabot", "relaxed-skill-validation",
                        "verdict-warn-only", "pages-deployment-failed",
                        "unpinned-action:{action_name}",
                        "orphan-skill:{component}:{skill_name}",
                        "orphan-plugin:{directory_basename}" }

1.3 Resource Fingerprints

fingerprint = "resource:{metric}:{threshold_breach}"

• resource:eval-duration:warning — eval avg > 50 min
• resource:eval-duration:critical — eval avg > 55 min
• resource:cost-increase — weekly compute hours up >20%

---

2. Diff Algorithm

previous_fps = cache_memory_load("health-check-fingerprints") ?? {}
current_fps  = {}

for each finding in all_collected_findings:
    fp = compute_fingerprint(finding)
    current_fps[fp] = finding

new_findings      = { fp: f for fp, f in current_fps  if fp NOT IN previous_fps }
existing_findings = { fp: f for fp, f in current_fps  if fp IN previous_fps }
resolved_findings = { fp: f for fp, f in previous_fps if fp NOT IN current_fps }

# Update occurrence tracking
for fp in existing_findings:
    existing_findings[fp].occurrences = previous_fps[fp].occurrences + 1
    existing_findings[fp].first_seen = previous_fps[fp].first_seen

for fp in new_findings:
    new_findings[fp].occurrences = 1
    new_findings[fp].first_seen = today

cache_memory_save("health-check-fingerprints", current_fps)
cache_memory_save("health-check-history", append(
    load("health-check-history"),
    { date: today, new_count, existing_count, resolved_count, by_severity }
))

2.1 Sorting Within Diff Categories

Within each category (NEW, EXISTING, RESOLVED):

1. Primary: Severity descending — 🔴 Critical → 🟡 Warning → 🔵 Info
2. Secondary: Category — pipeline → infra → resource
3. Tertiary: Alphabetical by title

---

3. Severity Rules Reference

Pipeline

Check	Condition	Severity
P1	evaluation workflow failed on main	🔴 Critical
P1	Other workflow failed on main	🟡 Warning
P1	Matches known-noise pattern	🔵 Info (demoted)
P2	Any cancelled/timed-out run on main	🟡 Warning
P3	Eval avg duration > 55 min	🔴 Critical
P3	Eval avg duration > 50 min	🟡 Warning
P5	Eval failure rate > 30% (all branches, 24h)	🔴 Critical
P5	Eval failure rate > 15% (all branches, 24h)	🟡 Warning
P6	Eval scheduled cancellation rate > 60% (24h)	🔴 Critical
P6	Eval scheduled cancellation rate > 30% (24h)	🟡 Warning

Infrastructure

Check	Condition	Severity
I1	No CODEOWNERS file	🟡 Warning
I2	No Dependabot config	🟡 Warning
I3	fail-on-warning: false in validate-skills	🟡 Warning
I4	--verdict-warn-only in evaluation	🔵 Info
I5	Pages deployment failed	🔴 Critical
I6	Unpinned third-party action	🔵 Info
I7	Orphan skill (not registered in any plugin)	🟡 Warning
I8	Orphan plugin (not listed in marketplace.json)	🟡 Warning

Resource

Check	Condition	Severity
U3	Weekly compute up >20%	🟡 Warning

---

4. Known Noise Patterns

The cache-memory key known-noise stores a list of fingerprint prefixes or patterns that should be demoted to 🔵 Info severity. Example patterns:

• pipeline:copilot-code-review — org-level workflow with known chronic failures
• infra:verdict-warn-only — intentional configuration, always Info

When a finding's fingerprint matches any known-noise pattern (prefix match), demote its severity to 🔵 Info. The finding is still reported in the output (in the EXISTING section if recurring) — it is NOT hidden.

New patterns can be added by manually editing the known-noise list in cache-memory.

---

5. Investigation Dispatch Rules

Only 🆕 NEW findings that meet these criteria qualify for investigation dispatch:

Condition	Action
🆕 + 🔴 Critical	Always dispatch
🆕 + 🟡 Warning + pipeline category	Dispatch
🆕 + 🟡 Warning + infra or resource category	Skip
🆕 + 🔵 Info	Never dispatch
📌 EXISTING or ✅ RESOLVED	Never dispatch

Budget cap: Maximum 2 dispatches per run. Priority order when cap is hit:

1. 🔴 Critical findings first
2. Pipeline findings before infrastructure
3. Other categories last

6. Output Templates

6.1 Issue Title

🏥 Repository Health Dashboard

6.2 Issue Label

devops-health

• Color: #0E8A16
• Description: Daily automated health check report

6.3 First Run Notice

If no previous fingerprints exist in cache-memory:

> ⚠️ This is the first health check run. All findings appear as new.
> Starting from the next run, only changes will be highlighted.

6.4 Trends Arrow Legend

Condition	Arrow	Meaning
Δ positive and good (e.g., success rate up)	✅	Improving
Δ positive and bad (e.g., compute hours up)	↗️	Increasing (watch)
Δ negative and good (e.g., open PRs down)	✅	Improving
Δ negative and bad (e.g., success rate down)	⚠️	Degrading
Δ ≈ 0	➡️	Stable

6.5 Investigation Island Template

<!-- investigation:{fingerprint} -->
⏳ Investigation dispatched — results arriving shortly...
<!-- /investigation:{fingerprint} -->

---

7. Operational Guardrails

7.1 API Rate Limits

• Use targeted, date-filtered queries to minimize API calls
• The github MCP toolset handles pagination automatically
• Space dispatches 5 seconds apart

7.2 Issue Body Size

• GitHub issues have a ~65,535 character limit
• If body exceeds 60k: truncate EXISTING section (keep top 20 by severity)
• Footer: > … N additional existing findings omitted
• The daily comment always includes complete summary counts

7.3 Cache Memory Keys

Key	Contents	Updated
health-check-fingerprints	Map of fingerprint → finding (with occurrences, first_seen)	Every run
health-check-history	Array of daily summaries (date, counts by diff type and severity)	Appended each run
health-dashboard-issue	Issue number of the canonical health dashboard issue. Used to update the dashboard by number so it stays stable even when GitHub's label search/list index drops the issue (which otherwise causes a duplicate dashboard to be created).	Every run
known-noise	Array of fingerprint patterns to demote to Info	Manual edit

7.4 Graceful Degradation

If any data source is unavailable:

• Skip that check category entirely
• Note the skip in the output: > ⚠️ Skipped {category} checks: {reason}
• Do NOT fail the entire workflow
• Continue with available data

7.5 Cache Memory Loss

If cache-memory returns no previous state:

• Treat all findings as 🆕 NEW
• Display the first-run notice (§6.3)
• The diff will resume automatically on the next run