dotnet__skills/tests/dotnet-msbuild/binlog-failure-analysis/eval.vally.yaml

name: binlog-failure-analysis
description: Evaluates the dotnet-msbuild/binlog-failure-analysis skill
type: capability
config:
  timeout: 6m
stimuli:
  - name: Diagnose build failures from binlog only (no source files)
    prompt: A .NET solution build has failed. The binary log from the failed build is available at build.binlog in the
      current directory. You do NOT have access to the original source files — only the binlog. Analyze this binlog file
      to diagnose all build failures, identify which errors are root causes versus cascading failures from dependent
      project failures, and suggest targeted fixes for each root cause.
    environment:
      files:
        - src: .
          dest: .
      commands:
        - dotnet build BuildDiag.sln -bl:build.binlog
        - node -e "const fs=require('fs'); fs.readdirSync('.').filter(f=>!f.endsWith('.binlog'))
          .forEach(f=>fs.rmSync(f,{recursive:true,force:true}))"
    graders:
      - type: output-contains
        config:
          substring: CS0246
      - type: output-contains
        config:
          substring: Newtonsoft.Json
      - type: prompt
    rubric:
      - Successfully replayed the binary log to text logs or otherwise extracted errors from the binlog
      - Identified CS0246 compiler errors in the Services project for Newtonsoft.Json types (JsonConvert, JObject,
        Formatting) from the binlog
      - Correctly determined that the root cause is a missing PackageReference for Newtonsoft.Json in the Services
        project
      - "Recognized the cascading failure chain: Services failed → Api could not build because it depends on Services"
      - Suggested adding a PackageReference for Newtonsoft.Json to the Services project as the primary fix