rtk-ai__rtk

skills/rtk-ai__rtk

Fork 0

mirror of https://github.com/rtk-ai/rtk.git synced 2026-09-19 07:33:17 +08:00

Commit Graph

Author	SHA1	Message	Date
Patrick szymkowiak	cc15ea7a8c	feat(benchmark): real-project savings + AI comprehension quality tests Cloud-init VM now installs 50+ real tools (arch-aware for ARM/x86), clones 14 open-source projects (ripgrep, Flask, Hugo, Vite, Redis, Newtonsoft.Json, Polars, dbt-core, scikit-learn, Vue, T3, Sinatra, Laravel, Spring Petclinic), and pre-builds dependencies. New Phase 12: Token savings measured on real codebases instead of toy projects. Covers Rust, Python, Go, JS/TS, C, C#, Ruby, Java. New Phase 13: AI Comprehension Quality benchmark — asks Claude Haiku the same factual questions on raw vs filtered output to validate that RTK preserves semantic content while removing 85-99% of tokens. Uses claude CLI auth (no API key needed) with ANTHROPIC_API_KEY fallback. Signed-off-by: Patrick szymkowiak <pszymkowiak@innovtech.eu> Signed-off-by: Patrick szymkowiak <patrick.szymkowiak@innovtech.eu>	2026-04-16 11:27:07 +02:00
Patrick szymkowiak	d22759b8c5	feat(benchmark): add multipass VM integration test suite Bun/TypeScript orchestrator that creates an Ubuntu 24.04 VM via multipass, installs all dev tools (Rust, Go, Node, Python, .NET, Terraform, etc.), builds RTK, and runs 103 tests across 11 phases: - Cargo quality (fmt, clippy, test) - 47 Rust built-in commands (git, ls, grep, cargo, pytest, go, tsc...) - 21 TOML filter commands (df, ps, shellcheck, hadolint, helm...) - Hook rewrite engine (17 rewrite assertions) - Exit code preservation - Token savings verification (avg 81%) - Pipe compatibility - Edge cases (unicode, ANSI, empty output) - Performance (memory < 20MB) - Concurrency (10 parallel executions) Usage: bun run scripts/benchmark/run.ts # Full suite (~3 min) bun run scripts/benchmark/run.ts --quick # Skip perf/concurrency bun run scripts/benchmark/run.ts --phase 3 # Single phase bun run scripts/benchmark/cleanup.ts # Delete VM bun run scripts/benchmark/rebuild.ts # Fast rebuild Prerequisites: brew install multipass, bun Signed-off-by: Patrick szymkowiak <patrick.szymkowiak@innovtech.eu>	2026-04-12 10:36:37 +02:00

Author

SHA1

Message

Date

Patrick szymkowiak

cc15ea7a8c

feat(benchmark): real-project savings + AI comprehension quality tests

Cloud-init VM now installs 50+ real tools (arch-aware for ARM/x86),
clones 14 open-source projects (ripgrep, Flask, Hugo, Vite, Redis,
Newtonsoft.Json, Polars, dbt-core, scikit-learn, Vue, T3, Sinatra,
Laravel, Spring Petclinic), and pre-builds dependencies.

New Phase 12: Token savings measured on real codebases instead of
toy projects. Covers Rust, Python, Go, JS/TS, C, C#, Ruby, Java.

New Phase 13: AI Comprehension Quality benchmark — asks Claude Haiku
the same factual questions on raw vs filtered output to validate that
RTK preserves semantic content while removing 85-99% of tokens.
Uses claude CLI auth (no API key needed) with ANTHROPIC_API_KEY fallback.

Signed-off-by: Patrick szymkowiak <pszymkowiak@innovtech.eu>
Signed-off-by: Patrick szymkowiak <patrick.szymkowiak@innovtech.eu>

2026-04-16 11:27:07 +02:00

Patrick szymkowiak

d22759b8c5

feat(benchmark): add multipass VM integration test suite

Bun/TypeScript orchestrator that creates an Ubuntu 24.04 VM via multipass,
installs all dev tools (Rust, Go, Node, Python, .NET, Terraform, etc.),
builds RTK, and runs 103 tests across 11 phases:

- Cargo quality (fmt, clippy, test)
- 47 Rust built-in commands (git, ls, grep, cargo, pytest, go, tsc...)
- 21 TOML filter commands (df, ps, shellcheck, hadolint, helm...)
- Hook rewrite engine (17 rewrite assertions)
- Exit code preservation
- Token savings verification (avg 81%)
- Pipe compatibility
- Edge cases (unicode, ANSI, empty output)
- Performance (memory < 20MB)
- Concurrency (10 parallel executions)

Usage:
  bun run scripts/benchmark/run.ts           # Full suite (~3 min)
  bun run scripts/benchmark/run.ts --quick   # Skip perf/concurrency
  bun run scripts/benchmark/run.ts --phase 3 # Single phase
  bun run scripts/benchmark/cleanup.ts       # Delete VM
  bun run scripts/benchmark/rebuild.ts       # Fast rebuild

Prerequisites: brew install multipass, bun
Signed-off-by: Patrick szymkowiak <patrick.szymkowiak@innovtech.eu>

2026-04-12 10:36:37 +02:00

2 Commits