Nick Sweeting
|
1fd8e0ec92
|
try statuses-write option
|
2025-06-10 23:57:41 -07:00 |
|
Nick Sweeting
|
4d8f1342e8
|
try and revert evals change
|
2025-06-10 23:57:41 -07:00 |
|
Nick Sweeting
|
ffd36eb5da
|
tweak env vars for CI
|
2025-06-10 03:58:37 -07:00 |
|
Magnus Müller
|
061f2cf089
|
Increase timeout duration in evaluation workflow from 60 to 360 minutes to accommodate longer-running jobs.
|
2025-06-09 20:51:11 +02:00 |
|
Magnus Müller
|
fe64e35b3c
|
Add SERPER_API_KEY to evaluation workflow for enhanced API integration
|
2025-06-09 12:25:50 +02:00 |
|
Magnus Müller
|
b1af578bf3
|
Add SERPER_API_KEY to evaluation workflow for enhanced API integration
|
2025-06-09 11:58:34 +02:00 |
|
Alezander9
|
739a3b2c87
|
add flag to toggle whether webjudge gets to see final agent result
|
2025-06-08 17:49:05 -07:00 |
|
Alezander9
|
31dcce5390
|
decrease workflow timeout
|
2025-06-07 22:08:29 -07:00 |
|
Alezander9
|
c04710ef61
|
Merge remote-tracking branch 'upstream/main' into parallel-evals
|
2025-06-07 21:48:47 -07:00 |
|
Alezander9
|
bc56c73fe7
|
add ability to append results to existing run, so we can parallelize
|
2025-06-07 21:47:47 -07:00 |
|
Magnus Müller
|
eaab9f04d7
|
Enhance GitHub Actions workflow and evaluate_tasks.py to include detailed task evaluation results. The workflow now captures and displays detailed results in a structured format, while the Python script outputs detailed results as JSON for better integration with GitHub Actions. This improves visibility and understanding of task outcomes in the evaluation process.
|
2025-06-07 13:39:16 +02:00 |
|
Magnus Müller
|
3ecee462a2
|
Update GitHub Actions workflow permissions to allow writing comments on pull requests and issues, enhancing interaction capabilities for automated testing processes.
|
2025-06-07 13:22:43 +02:00 |
|
Magnus Müller
|
bdf29c34fb
|
Add PR comment functionality to GitHub Actions workflow for agent task evaluation results. This includes a summary of passed tasks, percentage score, and status emoji based on task outcomes, enhancing visibility of evaluation results directly in pull requests.
|
2025-06-07 13:16:30 +02:00 |
|
Magnus Müller
|
8d9b24b03a
|
Add summary output for agent tasks evaluation in CI workflow
|
2025-06-07 11:27:06 +02:00 |
|
Magnus Müller
|
52c60f8367
|
Remove dependency on tests job in evaluate-tasks step of CI workflow
|
2025-06-07 11:11:23 +02:00 |
|
Magnus Müller
|
576519ee40
|
Enhance CI workflow by adding agent tasks evaluation step and updating evaluate_tasks.py to output evaluation results
|
2025-06-07 10:59:24 +02:00 |
|
Magnus Müller
|
3666d2b077
|
Add agent tasks evaluation script and update CI workflow to include it
|
2025-06-07 10:49:03 +02:00 |
|
Magnus Müller
|
074169f80f
|
Merge branch 'main' into tests/eval
|
2025-06-07 09:15:41 +02:00 |
|
Nick Sweeting
|
4358924964
|
only install chromium in tests
|
2025-06-06 19:32:20 -07:00 |
|
Magnus Müller
|
42dac3dce1
|
Add API key environment variables to GitHub Actions workflow for enhanced test capabilities
|
2025-06-07 01:11:48 +02:00 |
|
Nick Sweeting
|
8504bc4c7b
|
parallelize playwright tests using loop scope=session and pytest-xdist
|
2025-06-06 02:03:25 -07:00 |
|
Alezander9
|
61f9c31a3d
|
feat: support changing eval task set
|
2025-06-03 10:27:34 -07:00 |
|
BroskyBrowser
|
15cb992618
|
fix: expose ANCHOR_BROWSER_API_KEY variable in the evals workflow .yaml
|
2025-06-01 12:58:14 +02:00 |
|
Nick Sweeting
|
bfb6b26274
|
Merge branch 'main' into new-eval
|
2025-05-25 18:58:08 -07:00 |
|
Nick Sweeting
|
1d2cb46d73
|
Update claude.yml
|
2025-05-25 06:07:37 -04:00 |
|
Nick Sweeting
|
6d0758764a
|
Add Claude PR Assistant workflow
|
2025-05-25 03:04:06 -07:00 |
|
Alezander9
|
ad71ba8d29
|
add branch name selection into workflow
|
2025-05-24 19:12:01 -07:00 |
|
Alezander9
|
1f113fa640
|
Merge remote-tracking branch 'upstream/main' into new-eval
|
2025-05-24 10:25:43 -07:00 |
|
Nick Sweeting
|
04a3c881df
|
add docker setting for ci
|
2025-05-23 22:22:10 -07:00 |
|
Nick Sweeting
|
cacc7c2020
|
allow running publish manually
|
2025-05-23 22:18:09 -07:00 |
|
Nick Sweeting
|
8e8f9a2381
|
allow running publish manually
|
2025-05-23 22:17:09 -07:00 |
|
Nick Sweeting
|
4196e79faa
|
fix publish action
|
2025-05-23 22:15:34 -07:00 |
|
Nick Sweeting
|
3d10260543
|
fix missing link between find_tests and test job in CI
|
2025-05-23 19:22:55 -07:00 |
|
Nick Sweeting
|
37a36dbd28
|
catch failure case up-front
|
2025-05-23 19:12:46 -07:00 |
|
Nick Sweeting
|
6a1ed628e3
|
properly split filenamees out of ls results in test discovery
|
2025-05-23 19:10:25 -07:00 |
|
Nick Sweeting
|
063f103efd
|
more warning on filure to list tests
|
2025-05-23 19:05:52 -07:00 |
|
Nick Sweeting
|
06ee004a88
|
add assertion to tests discovery
|
2025-05-23 18:33:18 -07:00 |
|
Nick Sweeting
|
9fcd5cd7b2
|
debugging tests discovery
|
2025-05-23 18:32:28 -07:00 |
|
Nick Sweeting
|
e19e1c5dfc
|
fix ci tests
|
2025-05-23 18:29:59 -07:00 |
|
Nick Sweeting
|
815ae48938
|
fix pypi publish action
|
2025-05-23 18:21:18 -07:00 |
|
Alezander9
|
45dd0a26c2
|
update eval workflow with new arguments
|
2025-05-23 14:57:18 -07:00 |
|
Alezander9
|
a3dd8b004b
|
update eval workflow with new arguments
|
2025-05-23 14:46:14 -07:00 |
|
Nick Sweeting
|
27ca169393
|
reduce mumber of redundant CI builds
|
2025-05-23 04:46:21 -04:00 |
|
Nick Sweeting
|
3940462d8d
|
Potential fix for code scanning alert no. 28: Workflow does not contain permissions
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
|
2025-05-23 03:50:15 -04:00 |
|
Nick Sweeting
|
6b8360c475
|
better logging
|
2025-05-22 23:17:21 -07:00 |
|
Alezander9
|
aab470243f
|
update user message default argument
|
2025-05-22 15:53:49 -07:00 |
|
Alezander9
|
529e43fdd1
|
update args in workflow script to match new format
|
2025-05-22 11:36:47 -07:00 |
|
Alezander9
|
4a7e9113ca
|
add claude 4 support and cleanup eval script arguments
|
2025-05-22 10:54:00 -07:00 |
|
Nick Sweeting
|
18554e2834
|
autodetect tests for ci by looking in folder
|
2025-05-22 05:34:04 -07:00 |
|
Alezander9
|
0fbfc82da0
|
switch eval workflow to use new consolidated branch
|
2025-05-21 14:39:09 -07:00 |
|