supabase / supabase / 21519968519

73%

push

github

feat(assistant): score URL validity and fix support ticket URL guidance (#42227)

**Logic changes**
- Adds function in `helpers.ts` to extract URLs from text via regex
- I also considering using a library like
[linkify-it](https://www.npmjs.com/package/linkify-it) for this but
figured it's not worth the extra dep
- Adds associated tests in `helpers.test.ts`
- Adds "URL Validity" scorer which performs a HEAD request for links in
Assistant response text and determins what portion of links have `.ok`
responses
- Adds eval case to check correctness of support ticket URL answers

**Prompt changes**
- Informs Assistant of https://supabase.com/dashboard/support/new being
the URL to create support tickets
- Encourages Assistant to "self-debug" issues before directing users to
create support tickets

See [Eval
Report](https://github.com/supabase/supabase/pull/42227#issuecomment-3807772871)
and
[Correctness](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=experiment&object_id=1ad0f9b0-5adb-436c-9812-a87aac62c036&r=1ef13459-a98c-4904-925e-6d81276cebb2&s=dbe5c607-a560-462b-8745-41d430744431)
analysis for new support ticket test case.

Resolves AI-384

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added URL validity scoring to evaluations and helper utilities for
extracting/cleaning URLs.
* Added evaluation cases for support-ticket URL handling and OAuth
callback guidance.

* **Documentation**
* Updated assistant guidance to prefer self-resolution, include
support-ticket direction, clarified data-recovery search steps, and
added template-URL notation.

* **Tests**
* Expanded URL extraction and related utility tests to cover many
formats and edge cases.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

714 of 822 branches covered (86.86%)

Branch coverage included in aggregate %.

25 of 25 new or added lines in 1 file covered. (100.0%)

2641 of 4232 relevant lines covered (62.41%)

115.92 hits per line