The UTF-16 overload of detail::toUtf8() advances to the next code unit as soon as it encounters any surrogate. If the surrogate is not the start of a valid high/low pair, that next code unit has already been consumed.
For example, converting a high surrogate followed by A in replacement mode currently emits only U+FFFD and silently drops the valid A. An isolated low surrogate has the same problem with the code unit following it.
Expected behaviour
- Consume two UTF-16 code units only for a valid high/low surrogate pair.
- In replacement mode, emit U+FFFD for each malformed surrogate and continue converting all subsequent valid code units.
- With
GHC_RAISE_UNICODE_ERRORS, continue to report malformed UTF-16 as an illegal byte sequence.
- Keep valid surrogate-pair conversion unchanged.
The UTF-16 overload of
detail::toUtf8()advances to the next code unit as soon as it encounters any surrogate. If the surrogate is not the start of a valid high/low pair, that next code unit has already been consumed.For example, converting a high surrogate followed by
Ain replacement mode currently emits onlyU+FFFDand silently drops the validA. An isolated low surrogate has the same problem with the code unit following it.Expected behaviour
GHC_RAISE_UNICODE_ERRORS, continue to report malformed UTF-16 as an illegal byte sequence.