supabase / supabase / 26045492446

72%

push

github

fix(ai): contextualize cron schedule as SQL writes, score in "Tool Usage" (#45997)

When Assistant tries to schedule crons in read-only mode, it succeeds
but creates the jobs under the `supabase_read_only_user`. This causes
permission errors when user try to delete or unschedule them from the
Cron dashboard. The root fix will be to enforce read-only transactions
for that user. In the meantime, this PR steers Assistant to avoid the
mistake.

**Changes**

- Prompts `execute_sql` to treat side-effecting function calls such as
`cron.schedule()` as write queries.
- Adds tool input assertions for "Tool Usage" scorer and a focused cron
regression eval.
- Updates eval mocks to show pg_cron extension as installed so it can
call `cron.schedule()`

**Verification**

See [this
trace](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=experiment&object_id=4a9e8c0e-83b7-4555-8502-365662c3ec8e&r=e041e69b-b70f-41d1-b88c-e8f7888c3de5&s=e041e69b-b70f-41d1-b88c-e8f7888c3de5)
from Braintrust where the new eval passes "Tool Usage", correctly using
`isWriteQuery` for the `cron.schedule()`

Closes AI-737


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Tool evaluation now validates tool inputs (including exact and
substring matches) in addition to tool presence.

* **Tests**
* Added a test confirming cron-scheduling behavior and that SQL
scheduling/enqueue calls are treated as write operations.

* **Chores**
  * Added pg_cron to mock extension data.
* Clarified description that SQL calls with side effects should be
treated as writes.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/45997?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-genera... (continued)

965 of 1364 branches covered (70.75%)

Branch coverage included in aggregate %.

1485 of 2051 relevant lines covered (72.4%)

240.39 hits per line