Kikobeats / html-urls / 30556916444

99%

push

github

feat: fix mailto rewriting by upgrading @metascraper/helpers (#92)

* fix: keep mailto:user@host from becoming an HTTP url

normalizeUrl (via @metascraper/helpers → normalize-url) applies
defaultProtocol `http:` and rewrites `mailto:user@host` into
`http://host`. That made url/uri fetchable, so crawlers following
extracted links could hit internal hosts from addresses like
`mailto:admin@169.254.169.254`.

For absolute non-HTTP URIs, reject any normalizeUrl result that is an
HTTP(S) URL and keep the original scheme in `uri`.

* test: cover mailto:user@host HTTP rewrite regression

Ensure standard mailto links keep their scheme and never surface a
fetchable http:// url/uri for crawlers.

* fix: only guard non-http schemes from normalizeUrl rewrite

The previous guard used `isUri && !isHttpUrl`, which also caught broken
`http://` values and incorrectly kept them as `uri`. Limit the protection
to attributes whose scheme is present and not http/https.

* feat: preserve non HTTP(S) schemes as uri

normalizeUrl applies defaultProtocol `http:`, so `mailto:admin@169.254.169.254`
came back as `url: http://169.254.169.254` and any consumer crawling the output
would fetch a link-local address it was never given.

Non-HTTP(S) schemes now skip that coercion and keep their own value, while still
being normalized when normalization is meaningful (`ftp://EXAMPLE.com/a/../x`
becomes `ftp://example.com/x`).

Side effect, and the reason this is a feat: `tel:` and `javascript:` used to come
back with no `uri` at all because normalizeUrl returned undefined for them, and
now they carry their raw value. Consumers that only want fetchable links filter
on `url`.

Attribute values are trimmed up front so a padded `mailto:` is not dropped. The
HTTP path already got trimming for free inside normalizeUrl.

* chore: improve npm search metadata

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: fix mailto rewriting by upgrading @metascraper/helpers

`normalizeUrl` applied def... (continued)

37 of 37 branches covered (100.0%)

Branch coverage included in aggregate %.

101 of 101 relevant lines covered (100.0%)

118.4 hits per line