mirror of
https://github.com/elevenlabs/skills.git
synced 2026-09-14 20:46:33 +08:00
Tadas Petra
5acd0650d0
* Fix eval harness bugs and repair eval definitions after per-eval review Full review of all functional evals and trigger queries with fresh-context agents against actual run artifacts. Of 15 failing expectations at the ~80% baseline, none were genuine skill-quality failures. Harness (evals/run_all.py): - Resolve --output-dir to absolute: a relative path was re-resolved by cursor-agent against the eval cwd (doubled path), failing every functional eval instantly - Grade outputs/ recursively: nested files (agent_configs/*.json, TS subprojects) were invisible to grading, causing 4 false fails - Kill the whole process group on timeout: cursor-agent grandchildren holding the stdout pipe made subprocess.run block forever Eval definitions: - Rewrite 8 execution-style prompts to script-writing form (the sandbox has no ELEVENLABS_API_KEY; "generate X.mp3" caused refusals, raw-REST fallbacks, and one live agent deployment) - Retexture 25 expectations so the keyword grader checks load-bearing tokens instead of passing/failing vacuously - Remove 1 ungradeable expectation (music eval-2 duration: agents legitimately use music_length_ms or composition-plan section durations) - Replace/sharpen 5 trigger queries (negatives that were structurally unfair in the one-skill-installed eval workspace) Skill content bugs surfaced by false passes: - speech-to-text: fix invalid convert.with_raw_response(...) chain (raises AttributeError; verified against elevenlabs 2.60.0) - text-to-speech: fix audio.pipe(...) on the web ReadableStream returned by JS SDK 2.x (bridge via Readable.fromWeb) - voice-changer: description now excludes voice cloning (was the only hard trigger failure) - setup-api-key: fix duplicate step numbering Verified: triggers 156/156 (68 rerun live), functional 123/124. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix speech engine mic expectation token * Remove comma from getUserMedia token so the whitespace-split grader can match it Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Make functional-eval timeout kill portable to Windows os.killpg/signal.SIGKILL and setsid-based process groups are POSIX-only; use taskkill /F /T on Windows and only pass start_new_session on POSIX. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Stabilize eval expectations * Fix remaining flaky evals surfaced by rerun - Harmonize the ElevenLabs() constructor expectation in the four remaining files (speech-to-text, voice-isolator, music, sound-effects): nested agents alternate between ElevenLabs() and ElevenLabs(api_key=...) run to run; anchor on 'ElevenLabs(' which matches both - setup-api-key eval-2 'provides steps' false-failed a response that gave a full numbered create-key walkthrough (semantic check needed 2 indicator words, response had 1); reworded to stable tokens - Add missing/401/cause to the causes semantic indicators: in the keyless eval workspace, 'no key configured' is the correct 401 diagnosis and should count as suggesting a cause - agents SKILL.md description: qualify 'customer service bots' as voice-based and exclude text-only chatbots — the claude-API chatbot negative was a coin flip because the description invited text-bot reads - setup-api-key trigger: replace the ElevenLabs Scribe capability negative (still ElevenLabs-topical, flaky in the one-skill workspace) with a deepgram key-setup question — same boundary, structurally fair All updated expectations statically verified against artifacts from all four 2026-08-03 runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Require exact filenames in eval expectations * Stabilize three more grader false-fails from third rerun - speech-to-text eval-1: 'Prints or returns result.text' failed a correct script that used getattr(result, "text") + print(text); anchor on 'print(' and 'text' instead of the literal attribute chain - sound-effects eval-1: quote-attached tokens ('thunder / rain') in a pre-existing expectation can never match the whitespace-split grader; unquoted rewording verified against all four runs - setup-api-key eval-2: remove 'Suggests common causes' — every run phrases the (correct) 401 diagnosis differently and the adjacent 'Reports that ELEVENLABS_API_KEY is not configured...' expectation already covers it - speech-engine eval-2: proactively fix the comma-attached onTranscript(transcript, signal, session) signature tokens found by a full punctuation audit (would false-fail on TS-typed signatures); all other flagged tokens are graded by pattern checks and are safe All rewordings statically verified PASS against artifacts from all four 2026-08-03 runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Require the SDK explicitly in prompts that grade SDK usage Fourth rerun failed all three SDK-pattern expectations in speech-to-text eval-1 at once: the nested agent wrote a Python script that shells out to curl against the REST API. With no SDK installed in the sandbox, dodging an unverifiable import is reasonable behavior whenever the prompt only says "using ElevenLabs" — the same root cause as the earlier sound-effects raw-urllib failure. Swept all prompts whose expectations grade SDK usage and pinned the five with the gap (speech-to-text 1-3, text-to-speech 2, music 1) to name the SDK explicitly. Explanatory "show me how" prompts and the Speech Engine server (SDK-only APIs) left as-is. Live-verified: speech-to-text 14/14 and text-to-speech 16/16 after the change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>