Skill Evals

Regression Testing for Claude Code Skills

Automated verification that Claude Code uses correct Base44 APIs
instead of hallucinating Firebase / Supabase patterns

The Hallucination Problem

Without skill documentation, Claude falls back to pre-training patterns

API	Correct (Base44)	Hallucinated
Create record	`entities.Product.create(data)`	`Product.insert(data)`
Filter records	`entities.Product.filter(query)`	`Product.find(query)` / `.where()`
Auth login	`loginViaEmailPassword(email, pw)`	`signInWithEmailAndPassword()`
OAuth provider	`loginWithProvider('google')`	`signInWithGoogle()`
SDK import	`import { ... } from "@base44/sdk"`	`import { ... } from "firebase"`
Config init	`createClient({ appId })`	`initializeApp({ apiKey })`

What Value Does This Provide?

Regression Testing

Every change to skill documentation is tested against 16 fixtures and 30+ prompts. Catch regressions before they ship.

Strategy Comparison

Compare documentation approaches (AGENTS.md format, verbosity level, context priming) to find the most effective strategy.

Consistency Measurement

Run the same fixture N times to measure how reliably Claude produces correct output. Quantify flakiness.

Architecture Overview

Experiments

CLAUDE.md + AGENTS.md
+ skills/

→

Setup

Inject skills &
docs into fixtures

→

Fixtures

eval.json + project/
16 test scenarios

→

Runner

Copy to /tmp
Run agent

→

Checks

10 check types
Validate output

→

Reports

Pass/fail matrix
Best experiment

src/experiments.ts → src/setup.ts → src/runner.ts → src/checks/ → src/compare-runner.ts

Fixture Anatomy

fixtures/sdk-entity-crud/
  eval.json← prompts + checks
  project/← working directory for agent
    CLAUDE.md← injected by setup
    AGENTS.md← injected by setup
    skills/← injected from experiment
      base44-sdk/
        SKILL.md
        references/← symlink
      base44-cli/
        SKILL.md
        references/← symlink
    src/← optional starter code

Experiment injection:

setup.ts copies experiment's skills/ into each fixture's project/skills/
CLAUDE.md and AGENTS.md copied from experiment dir if present
Baseline experiment = empty dir, no docs injected

At runtime:

Entire fixture copied to /tmp/eval-*
Agent runs inside project/
Checks validate output files + stdout
Temp dir cleaned up after each run

eval.json Format

{
  "$schema": "../../eval.schema.json",
  "name": "anti-hallucination",
  "description": "Verify correct Base44 APIs, block hallucinated patterns",
  "prompts": [
    {
      "name": "anti-hallucination-auth",
      "prompt": "Create a React login page with email/password login,
                 Google OAuth, registration, and logout...",
      "expectedSkills": ["base44-sdk"],
      "checks": [
        {
          "type": "file-exists",
          "description": "Login component created",
          "filePath": "src/components/Login.jsx"
        },
        {
          "type": "file-content",
          "description": "Uses loginViaEmailPassword",
          "filePath": "src/components/Login.jsx",
          "pattern": "loginViaEmailPassword"
        },
        {
          "type": "file-content-excluded",
          "description": "No Firebase signIn pattern",
          "filePath": "src/components/Login.jsx",
          "pattern": "signInWithEmailAndPassword"
        }
      ]
    }
  ]
}

The three-check pattern: file-exists → file-content (correct API) → file-content-excluded (hallucinated API)

The 10 Check Types

File System

file-exists — Verify a file was created at the expected path
file-content — Assert a regex pattern exists in a file
file-content-excluded — Assert a pattern does NOT appear in a file

Output

contains — Check agent stdout for a substring
valid-json — Verify a file contains parseable JSON
command-passes — Run a shell command, assert exit code 0

Structural

json-schema — Validate file contents against a JSON Schema

Domain-Specific (Base44)

entity-config — Validate entity schema: PascalCase name, type:object, properties, required
agent-config — Validate agent schema: snake_case name, instructions, tool_configs
function-def — Validate function config: name + entry (not entrypoint)

Experiments

Feature	baseline	agents-md	agents-md -combined	verbose -index	context -first
SKILL.md files	—	✓	✓	✓	✓
AGENTS.md API index	—	✓	✓	✓	✓
CLAUDE.md priming	—	—	✓	—	✓
Expanded signatures	—	—	—	✓	—
Retrieval-led reasoning hint	—	—	✓	—	✓
Common Mistakes section	—	✓	✓	✓	✓

baseline
No docs at all. Control group — shows what hallucinations look like without guidance.

agents-md
Compact API index in AGENTS.md listing modules + method names. Skills with SKILL.md + references. No CLAUDE.md.

agents-md-combined
Same AGENTS.md index + CLAUDE.md that primes agent to explore project structure first and prefer retrieval over pre-training.

verbose-index
AGENTS.md with expanded method signatures including param types and return types. More verbose, potentially more precise.

context-first
CLAUDE.md with “read docs first” hint + compact AGENTS.md index. Tests if retrieval priming improves accuracy.

Execution Flow

1 Discover fixtures

→

2 Expand multi-prompt

→

3 Group by suite

↓

4 Copy to /tmp

→

5 Run agent in project/

→

6 Check expectedSkills

↓

7 Run check suite

→

8 Aggregate results

→

9 Report pass/fail

Concurrency: up to 5 fixtures run in parallel — configurable via --concurrency flag

Results: Experiment Comparison

5 experiments × 16 fixtures (45 expanded prompts)

Experiment	Passed	Failed	Pass Rate
agents-md-combined	38	7	84%
agents-md-context-first	36	9	80%
agents-md	34	11	76%
agents-md-verbose-index	32	13	71%
baseline	21	24	47%

combined

84%

context-first

80%

agents-md

76%

verbose-index

71%

baseline

47%

bestExperiment: agents-md-combined

CLAUDE.md retrieval priming + AGENTS.md compact index = highest pass rate. Verbose signatures did not help — more docs != better results.

Test Coverage: 16 Fixtures

SDK / Frontend (7)

sdk-entity-crud (6 prompts)
sdk-auth
sdk-functions
sdk-integrations
anti-hallucination (4 prompts)
agent-chat-component
react-task-manager

CLI / Config (5)

cli-entities (6 prompts)
cli-agents (4 prompts)
cli-functions
create-support-agent
skill-check

Cross-Cutting (4)

fullstack (4 prompts)
existing-app-add-feature
nextjs-todo
agent-with-function

Multi-prompt fixtures expand to 30+ individual test prompts. Each prompt has its own expectedSkills and checks[].

Adding a New Fixture

Step 1: Create eval.json

mkdir evals/fixtures/my-new-fixture
# $schema gives autocomplete in VS Code
cat > evals/fixtures/my-new-fixture/eval.json

Step 2: Add project/ with starter files

mkdir -p evals/fixtures/my-new-fixture/project/src
# Add any starter code the agent needs
# project/ becomes the agent's working directory

Step 3: Run it

cd evals
npm run eval -- --fixtures my-new-fixture --verbose

Reference eval.schema.json with "$schema": "../../eval.schema.json" for IDE validation and autocomplete.

Adding a New Experiment

Step 1: Create experiment directory

experiments/my-experiment/
  CLAUDE.md← optional project-level instructions
  AGENTS.md← optional API index
  skills/
    base44-sdk/
      SKILL.md
      references/← symlink

Step 2: Symlink references to avoid duplication

cd experiments/my-experiment/skills/base44-sdk
ln -s ../../../../../skills/base44-sdk/references references

Step 3: Run comparison

cd evals
npm run eval:compare

Daily Workflows

Validate a skill change

# Edit skills/base44-sdk/SKILL.md, then:
cd evals && npm run eval -- --verbose

Run all fixtures against the current skill documentation

Compare documentation strategies

cd evals && npm run eval:compare
# Runs all experiments x all fixtures
# Outputs pass/fail matrix + bestExperiment

Find which AGENTS.md format produces the highest pass rate

Check consistency (repeated runs)

cd evals && tsx src/run-repeated.ts --runs 5 \
  --fixtures anti-hallucination --verbose

Run the same fixture N times to measure flakiness and consistency

Bottom Line

The Problem

Claude's pre-training data contains far more Firebase/Supabase patterns than Base44 APIs. Without skill documentation, it hallucinates wrong method names, wrong imports, and wrong auth flows.

The Solution

16 fixtures with 30+ prompts that verify correct API usage and reject hallucinated patterns. Each fixture uses the three-check pattern: file exists, correct API present, hallucinated API absent.

The Impact

Every skill change is testable. Run npm run eval before merging to catch regressions in documentation that cause hallucinations.

Documentation strategies are comparable. The compare runner quantifies which AGENTS.md format, verbosity level, or priming approach works best.

Consistency is measurable. Repeated runs reveal whether Claude reliably uses correct APIs or just gets lucky sometimes.

Skills that pass evals → fewer hallucinations in production → users get working code on the first try.