harttle / liquidjs / 40740851194

99%

push

github

fix(filters): truncate and slice must not split surrogate pairs (#981)

* fix(filters): truncate and slice must not split surrogate pairs

truncate and slice cut strings by UTF-16 code unit, so a cut that landed
inside an emoji or other astral character left a lone surrogate in the
output, e.g. {{ "a😀b" | slice: 1, 1 }} returned "\ud83d". A lone
surrogate becomes U+FFFD when the output is encoded as UTF-8.

Move a cut that falls between the two halves of a pair to the nearest
boundary inside the range. Lengths are still counted in UTF-16 code
units, so output may be shorter than requested by one unit. Strings
without surrogate pairs are unaffected.

* fix(filters): truncate and slice count and cut by code point

Count and cut by Unicode code point instead of UTF-16 code unit, as
Shopify/liquid does (String#length, String#[] and String#slice are
character based in Ruby). This also stops a cut from leaving half of a
surrogate pair behind, e.g. {{ "a😀b" | slice: 1, 1 }} returned "\ud83d"
and now returns "😀".

truncate counts the length of the ellipsis in code points as well.
Offsets are resolved by walking the string once, so a negative slice
offset or a deep truncate costs O(length of the string).

1203 of 1224 branches covered (98.28%)

Branch coverage included in aggregate %.

16 of 16 new or added lines in 3 files covered. (100.0%)

3128 of 3134 relevant lines covered (99.81%)

19486.99 hits per line