Commit Graph

333 Commits

Author SHA1 Message Date
Nick Sweeting 1fd8e0ec92 try statuses-write option 2025-06-10 23:57:41 -07:00
Nick Sweeting 4d8f1342e8 try and revert evals change 2025-06-10 23:57:41 -07:00
Nick Sweeting ffd36eb5da tweak env vars for CI 2025-06-10 03:58:37 -07:00
Magnus Müller 061f2cf089 Increase timeout duration in evaluation workflow from 60 to 360 minutes to accommodate longer-running jobs. 2025-06-09 20:51:11 +02:00
Magnus Müller fe64e35b3c Add SERPER_API_KEY to evaluation workflow for enhanced API integration 2025-06-09 12:25:50 +02:00
Magnus Müller b1af578bf3 Add SERPER_API_KEY to evaluation workflow for enhanced API integration 2025-06-09 11:58:34 +02:00
Alezander9 739a3b2c87 add flag to toggle whether webjudge gets to see final agent result 2025-06-08 17:49:05 -07:00
Alezander9 31dcce5390 decrease workflow timeout 2025-06-07 22:08:29 -07:00
Alezander9 c04710ef61 Merge remote-tracking branch 'upstream/main' into parallel-evals 2025-06-07 21:48:47 -07:00
Alezander9 bc56c73fe7 add ability to append results to existing run, so we can parallelize 2025-06-07 21:47:47 -07:00
Magnus Müller eaab9f04d7 Enhance GitHub Actions workflow and evaluate_tasks.py to include detailed task evaluation results. The workflow now captures and displays detailed results in a structured format, while the Python script outputs detailed results as JSON for better integration with GitHub Actions. This improves visibility and understanding of task outcomes in the evaluation process. 2025-06-07 13:39:16 +02:00
Magnus Müller 3ecee462a2 Update GitHub Actions workflow permissions to allow writing comments on pull requests and issues, enhancing interaction capabilities for automated testing processes. 2025-06-07 13:22:43 +02:00
Magnus Müller bdf29c34fb Add PR comment functionality to GitHub Actions workflow for agent task evaluation results. This includes a summary of passed tasks, percentage score, and status emoji based on task outcomes, enhancing visibility of evaluation results directly in pull requests. 2025-06-07 13:16:30 +02:00
Magnus Müller 8d9b24b03a Add summary output for agent tasks evaluation in CI workflow 2025-06-07 11:27:06 +02:00
Magnus Müller 52c60f8367 Remove dependency on tests job in evaluate-tasks step of CI workflow 2025-06-07 11:11:23 +02:00
Magnus Müller 576519ee40 Enhance CI workflow by adding agent tasks evaluation step and updating evaluate_tasks.py to output evaluation results 2025-06-07 10:59:24 +02:00
Magnus Müller 3666d2b077 Add agent tasks evaluation script and update CI workflow to include it 2025-06-07 10:49:03 +02:00
Magnus Müller 074169f80f Merge branch 'main' into tests/eval 2025-06-07 09:15:41 +02:00
Nick Sweeting 4358924964 only install chromium in tests 2025-06-06 19:32:20 -07:00
Magnus Müller 42dac3dce1 Add API key environment variables to GitHub Actions workflow for enhanced test capabilities 2025-06-07 01:11:48 +02:00
Nick Sweeting 8504bc4c7b parallelize playwright tests using loop scope=session and pytest-xdist 2025-06-06 02:03:25 -07:00
Alezander9 61f9c31a3d feat: support changing eval task set 2025-06-03 10:27:34 -07:00
BroskyBrowser 15cb992618 fix: expose ANCHOR_BROWSER_API_KEY variable in the evals workflow .yaml 2025-06-01 12:58:14 +02:00
Nick Sweeting bfb6b26274 Merge branch 'main' into new-eval 2025-05-25 18:58:08 -07:00
Nick Sweeting 1d2cb46d73 Update claude.yml 2025-05-25 06:07:37 -04:00
Nick Sweeting 6d0758764a Add Claude PR Assistant workflow 2025-05-25 03:04:06 -07:00
Alezander9 ad71ba8d29 add branch name selection into workflow 2025-05-24 19:12:01 -07:00
Alezander9 1f113fa640 Merge remote-tracking branch 'upstream/main' into new-eval 2025-05-24 10:25:43 -07:00
Nick Sweeting 04a3c881df add docker setting for ci 2025-05-23 22:22:10 -07:00
Nick Sweeting cacc7c2020 allow running publish manually 2025-05-23 22:18:09 -07:00
Nick Sweeting 8e8f9a2381 allow running publish manually 2025-05-23 22:17:09 -07:00
Nick Sweeting 4196e79faa fix publish action 2025-05-23 22:15:34 -07:00
Nick Sweeting 3d10260543 fix missing link between find_tests and test job in CI 2025-05-23 19:22:55 -07:00
Nick Sweeting 37a36dbd28 catch failure case up-front 2025-05-23 19:12:46 -07:00
Nick Sweeting 6a1ed628e3 properly split filenamees out of ls results in test discovery 2025-05-23 19:10:25 -07:00
Nick Sweeting 063f103efd more warning on filure to list tests 2025-05-23 19:05:52 -07:00
Nick Sweeting 06ee004a88 add assertion to tests discovery 2025-05-23 18:33:18 -07:00
Nick Sweeting 9fcd5cd7b2 debugging tests discovery 2025-05-23 18:32:28 -07:00
Nick Sweeting e19e1c5dfc fix ci tests 2025-05-23 18:29:59 -07:00
Nick Sweeting 815ae48938 fix pypi publish action 2025-05-23 18:21:18 -07:00
Alezander9 45dd0a26c2 update eval workflow with new arguments 2025-05-23 14:57:18 -07:00
Alezander9 a3dd8b004b update eval workflow with new arguments 2025-05-23 14:46:14 -07:00
Nick Sweeting 27ca169393 reduce mumber of redundant CI builds 2025-05-23 04:46:21 -04:00
Nick Sweeting 3940462d8d Potential fix for code scanning alert no. 28: Workflow does not contain permissions
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-05-23 03:50:15 -04:00
Nick Sweeting 6b8360c475 better logging 2025-05-22 23:17:21 -07:00
Alezander9 aab470243f update user message default argument 2025-05-22 15:53:49 -07:00
Alezander9 529e43fdd1 update args in workflow script to match new format 2025-05-22 11:36:47 -07:00
Alezander9 4a7e9113ca add claude 4 support and cleanup eval script arguments 2025-05-22 10:54:00 -07:00
Nick Sweeting 18554e2834 autodetect tests for ci by looking in folder 2025-05-22 05:34:04 -07:00
Alezander9 0fbfc82da0 switch eval workflow to use new consolidated branch 2025-05-21 14:39:09 -07:00