Claude Code on Linux headless VMs has no keychain access.
Use --bare mode which reads ANTHROPIC_API_KEY from env instead.
The OAuth access token works as API key in bare mode.
Signed-off-by: Patrick szymkowiak <pszymkowiak@innovtech.eu>
Signed-off-by: Patrick szymkowiak <patrick.szymkowiak@innovtech.eu>
Quality benchmark now auto-detects available LLM CLIs and uses them
for comprehension tests. No API keys needed — uses each tool's native
auth (OAuth for claude/codex, Google auth for gemini, local for ollama).
New --llm flag: claude | gemini | codex | ollama | auto (default)
Each backend is called via CLI pipe (temp file to avoid shell escaping).
All backends run in parallel when --llm=auto detects multiple.
Signed-off-by: Patrick szymkowiak <pszymkowiak@innovtech.eu>
Signed-off-by: Patrick szymkowiak <patrick.szymkowiak@innovtech.eu>
Cloud-init VM now installs 50+ real tools (arch-aware for ARM/x86),
clones 14 open-source projects (ripgrep, Flask, Hugo, Vite, Redis,
Newtonsoft.Json, Polars, dbt-core, scikit-learn, Vue, T3, Sinatra,
Laravel, Spring Petclinic), and pre-builds dependencies.
New Phase 12: Token savings measured on real codebases instead of
toy projects. Covers Rust, Python, Go, JS/TS, C, C#, Ruby, Java.
New Phase 13: AI Comprehension Quality benchmark — asks Claude Haiku
the same factual questions on raw vs filtered output to validate that
RTK preserves semantic content while removing 85-99% of tokens.
Uses claude CLI auth (no API key needed) with ANTHROPIC_API_KEY fallback.
Signed-off-by: Patrick szymkowiak <pszymkowiak@innovtech.eu>
Signed-off-by: Patrick szymkowiak <patrick.szymkowiak@innovtech.eu>
- Run cargo test directly through testCmd instead of faking results
- Add 60s per-test timeout on vmExec to prevent hung suite
- Make report path configurable via --report flag (default: project root)
- Fix shell injection in testRewrite by escaping single quotes
Signed-off-by: Patrick szymkowiak <patrick.szymkowiak@innovtech.eu>
- Strict exit codes: cargo/python/go tests expect exact exit codes
instead of "any" (catches real regressions)
- Binary size limit documented: 8MB for ARM Linux VM vs 5MB x86 stripped
- --phase NaN guard: error message instead of silent no-op
- Verdict: 0 failures = READY, any failure = NOT READY (no more "minor issues" budget)
- rtk err exit code bug tracked in #846
Signed-off-by: Patrick szymkowiak <patrick.szymkowiak@innovtech.eu>
Patrick szymkowiak