f89a97f256
Delete laminar files since it is integrated to the main eval service
2025-06-19 11:26:53 +02:00
0b6ebea431
Merge branch 'main' into mert/new_everything
2025-06-19 09:49:43 +02:00
829eafe982
Merge branch 'main' into eventbus
2025-06-18 10:26:50 -07:00
f89a1aac84
Update evaluation workflow to use laminar_eval.py
...
- Changed the script executed in the evaluation workflow from `eval/service.py` to `eval/laminar_eval.py` for consistency with recent updates.
2025-06-18 16:43:06 +02:00
7f1d256964
Update dependencies in pyproject.toml and rename evaluation workflow
...
- Removed duplicate lmnr dependency from the `dependencies` section in `pyproject.toml`.
- Updated `lmnr` version to `0.6.11` in the `eval` extras group.
- Renamed the evaluation workflow from "Run Evaluation Script" to "Run Laminar Eval Script" for clarity.
- Adjusted the dependency installation command in the workflow to include the `--extra eval` flag.
2025-06-18 16:31:00 +02:00
01a36a587e
gh action for laminar evals
2025-06-18 14:38:27 +01:00
1864e52635
Merge branch 'main' into mert/new_everything
2025-06-18 10:00:25 +02:00
d0ec528802
fixed events
2025-06-17 22:38:59 -07:00
ec32fee074
Merge branch 'main' into eventbus
2025-06-17 15:25:24 -07:00
a114613c09
ignore formatting changes commits
2025-06-17 14:13:34 -07:00
c66880a8fa
Add lmnr package for tracing integration and update eval workflow
2025-06-17 18:58:56 +02:00
0c34c399db
Merge branch 'main' into mert/new_everything
2025-06-17 11:19:06 +02:00
e3d21d33a1
fix evaluate_tasks.py errors in CI
2025-06-16 17:20:27 -07:00
3972f335f9
allow claude github action to use more tools
2025-06-16 15:47:18 -07:00
cdf0706f20
only create pre-release versions manually
2025-06-12 10:49:04 -07:00
06488e11ba
fix clickaction error handling test
2025-06-11 00:05:58 -07:00
1fd8e0ec92
try statuses-write option
2025-06-10 23:57:41 -07:00
4d8f1342e8
try and revert evals change
2025-06-10 23:57:41 -07:00
ffd36eb5da
tweak env vars for CI
2025-06-10 03:58:37 -07:00
4d95a5db75
cleanup project root dir a bit
2025-06-09 22:34:34 -07:00
061f2cf089
Increase timeout duration in evaluation workflow from 60 to 360 minutes to accommodate longer-running jobs.
2025-06-09 20:51:11 +02:00
fe64e35b3c
Add SERPER_API_KEY to evaluation workflow for enhanced API integration
2025-06-09 12:25:50 +02:00
b1af578bf3
Add SERPER_API_KEY to evaluation workflow for enhanced API integration
2025-06-09 11:58:34 +02:00
739a3b2c87
add flag to toggle whether webjudge gets to see final agent result
2025-06-08 17:49:05 -07:00
31dcce5390
decrease workflow timeout
2025-06-07 22:08:29 -07:00
c04710ef61
Merge remote-tracking branch 'upstream/main' into parallel-evals
2025-06-07 21:48:47 -07:00
bc56c73fe7
add ability to append results to existing run, so we can parallelize
2025-06-07 21:47:47 -07:00
eaab9f04d7
Enhance GitHub Actions workflow and evaluate_tasks.py to include detailed task evaluation results. The workflow now captures and displays detailed results in a structured format, while the Python script outputs detailed results as JSON for better integration with GitHub Actions. This improves visibility and understanding of task outcomes in the evaluation process.
2025-06-07 13:39:16 +02:00
3ecee462a2
Update GitHub Actions workflow permissions to allow writing comments on pull requests and issues, enhancing interaction capabilities for automated testing processes.
2025-06-07 13:22:43 +02:00
bdf29c34fb
Add PR comment functionality to GitHub Actions workflow for agent task evaluation results. This includes a summary of passed tasks, percentage score, and status emoji based on task outcomes, enhancing visibility of evaluation results directly in pull requests.
2025-06-07 13:16:30 +02:00
8d9b24b03a
Add summary output for agent tasks evaluation in CI workflow
2025-06-07 11:27:06 +02:00
52c60f8367
Remove dependency on tests job in evaluate-tasks step of CI workflow
2025-06-07 11:11:23 +02:00
576519ee40
Enhance CI workflow by adding agent tasks evaluation step and updating evaluate_tasks.py to output evaluation results
2025-06-07 10:59:24 +02:00
3666d2b077
Add agent tasks evaluation script and update CI workflow to include it
2025-06-07 10:49:03 +02:00
074169f80f
Merge branch 'main' into tests/eval
2025-06-07 09:15:41 +02:00
4358924964
only install chromium in tests
2025-06-06 19:32:20 -07:00
42dac3dce1
Add API key environment variables to GitHub Actions workflow for enhanced test capabilities
2025-06-07 01:11:48 +02:00
8504bc4c7b
parallelize playwright tests using loop scope=session and pytest-xdist
2025-06-06 02:03:25 -07:00
61f9c31a3d
feat: support changing eval task set
2025-06-03 10:27:34 -07:00
15cb992618
fix: expose ANCHOR_BROWSER_API_KEY variable in the evals workflow .yaml
2025-06-01 12:58:14 +02:00
ebe47f6cf6
omg ppl how big do I have to make it
2025-05-28 00:58:55 -07:00
7a4b121647
bruh are people really this blind
2025-05-28 00:48:21 -07:00
bfb6b26274
Merge branch 'main' into new-eval
2025-05-25 18:58:08 -07:00
1d2cb46d73
Update claude.yml
2025-05-25 06:07:37 -04:00
6d0758764a
Add Claude PR Assistant workflow
2025-05-25 03:04:06 -07:00
ad71ba8d29
add branch name selection into workflow
2025-05-24 19:12:01 -07:00
1f113fa640
Merge remote-tracking branch 'upstream/main' into new-eval
2025-05-24 10:25:43 -07:00
6a2e50dd19
Update 3_feature_request.yml
2025-05-24 04:17:32 -04:00
b9bbfad99a
Update 2_bug_report.yml
2025-05-24 04:17:18 -04:00
620a5c0b3e
Update 1_element_detection_bug.yml
2025-05-24 04:17:06 -04:00
Magnus Müller