supabase / supabase / 21719409559

73%

push

github

feat(assistant): run 3 trials for Assistant evals in CI (#42510)

Runs 3 trials for Assistant evals in CI to reduce random variation.
Locally, only 1 trial is run.

Also adds `CI` to `studio#build` env in turbo.json. This env var is
[automatically set by GitHub
Actions](https://github.blog/changelog/2020-04-15-github-actions-sets-the-ci-environment-variable-to-true/).

Compare number of trials:
- [Assistant
(mattrossman/ai-398-increase-trial-count-for-assistant-evals-1770305591)](https://www.braintrust.dev/app/supabase.io/p/Assistant/experiments/mattrossman%2Fai-398-increase-trial-count-for-assistant-evals-1770305591)
- [Assistant
(master)](https://www.braintrust.dev/app/supabase.io/p/Assistant/experiments/master-1770305906?c=mattrossman/ai-398-increase-trial-count-for-assistant-evals-1770305591)

References:
- https://www.braintrust.dev/docs/evaluate/run-evaluations#trials

Closes AI-398

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated evaluation configuration to adjust trial counts based on CI
environment
  * Integrated CI environment variable into build system configuration

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Ali Waseem <waseema393@gmail.com>

714 of 822 branches covered (86.86%)

Branch coverage included in aggregate %.

2641 of 4240 relevant lines covered (62.29%)

115.7 hits per line