{{ message }}
Repository navigation
genUnicodeString generates invalid unicode #167
Description
The unicode character generator for unicode characters is picking a random CodePoint in the BMP. The unicode string generator just generates an arbitrary array of such code points and turns it into a string. It turns out that this can generate invalid unicode via unpaired surrogates: https://unicode.org/faq/utf_bom.html#utf16-7
One solution here would be to restrict the code points to avoid such cases, another would be to figure out a more complicated but correct way to generate unicode which cannot be done CodePoint by CodePoint.
For context I discovered this while trying to write a quickcheck test for utf8 encoding/decoding, you can see the failing test here
Activity
martyall commentedon Sep 13, 2023
On a related note, it appears that the unicode
Chargenerator includes the code point65536(chooseIntis inlcusive?). Shouldn't this be65535? (FYI, I've verified that this is not causing my error)natefaubion commentedon Sep 13, 2023
It's wrong as a bound, but 65536 is just getting turned into 65535 via
toEnumWithDefaults, which is effectivelyclamp.[-]Generators generate invalid unicode[/-][+]genUnicodeString generates invalid unicode[/+]martyall commentedon Sep 13, 2023
Does something like this seem right?
https://github.com/f-o-a-m/purescript-bytestrings/pull/1/files#diff-71732b478b4808898d86c8591ad7ab46d8122c1e4facec4a9151ac49efba905dR107-R137
At least it passes the utf-8 round trip
garyb commentedon Sep 24, 2023
Seems reasonable to me 👍