emit \uXXXX escapes instead of \0 and \v in _escape_ch by iefa-m · Pull Request #129 · dpranke/pyjson5 · GitHub
Skip to content

emit \uXXXX escapes instead of \0 and \v in _escape_ch - #129

Merged
dpranke merged 1 commit into
dpranke:mainfrom
iefa-m:json-escapes-for-nul-and-vt
Oct 22, 2026
Merged

dpranke merged 1 commit into
dpranke:mainfrom
iefa-m:json-escapes-for-nul-and-vt

Conversation

@iefa-m

@iefa-m iefa-m commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

_escape_ch writes NUL and vertical tab as \0 and \v. Those escapes exist in JSON5 (via ECMAScript) but not in JSON, where the only single-character escapes are \", \\, \/, \b, \f, \n, \r and \t. dumps(quote_keys=True, trailing_commas=False) is documented to produce the same output as json.dumps(), and --as-json in the CLI uses those settings, but for any string containing U+0000 or U+000B the result is "\0" or "\v", which json.loads() and other strict JSON parsers reject with Invalid \escape. I noticed it while comparing dumps() against the json module for every BMP code point; these two were the only characters whose output could not be read back as JSON. Letting them fall through to the generic \uXXXX form gives the same six-character \uXXXX escapes for NUL and VT (\u000b for VT), which is what the json module writes and is equally valid JSON5, so loads() round-trips are unchanged. That also makes the NUL-before-digit special case in _encode_quoted_str from #121 unnecessary, since a \uXXXX escape for NUL is never ambiguous, so it is removed.

>>> import json, json5
>>> json5.dumps('a\0b\v', quote_keys=True, trailing_commas=False)
'"a\\0b\\v"'
>>> json.loads(_)
json.decoder.JSONDecodeError: Invalid \escape: line 1 column 3 (char 2)

JSON5 (like Javascript but unlike JSON) allows `\0` and `\v` as escape
sequences in strings. The encoder used them for NUL and vertical tab,
which made `dumps(quote_keys=True, trailing_commas=False)` (documented
to match `json.dumps()`) and the `--as-json` flag produce output such
as `"\0"` and `"\v"` that the json module and other strict JSON parsers
reject with "Invalid \escape".

Let those two characters fall through to the generic `\uXXXX` form
instead, so they are written as `\u0000` and `\u000b`, which is what
the json module writes and is equally valid JSON5. The NUL-before-digit
special case in `_encode_quoted_str` is no longer needed since
`\u0000` cannot be mistaken for an octal escape, so it is removed.
@dpranke

dpranke commented Oct 20, 2026

Copy link
Copy Markdown
Owner

@iefa-m iefa-m changed the title emit \u0000 and \u000b instead of \0 and \v in _escape_ch emit \uXXXX escapes instead of \0 and \v in _escape_ch Oct 21, 2026
@dpranke

dpranke commented Oct 22, 2026

Copy link
Copy Markdown
Owner

Okay, having looked at this now ...

Given that the docs explicitly say that dumps(..., quote_keys=True, trailing_commas=False) should produce json-compliant output, and it doesn't (and there's no flag combination that you can use that will do what you want if you want JSON-compliant output), this seems like it's probably the right fix.

However, it's perhaps a bit unfortunately if you actually like \0 and \v and were fine with getting them, since now you can't.

What this suggests to me is that there should really be an as_json flag to dumps, just as there is to the tool, and that that should control if something is json-compliant, not the other two.

But that's a patch for another day. This looks fine.

@dpranke
dpranke merged commit d3c9b73 into dpranke:main Oct 22, 2026
9 checks passed
@dpranke

dpranke commented Oct 22, 2026

Copy link
Copy Markdown
Owner

Also, thanks for the full extended description :).

@iefa-m

iefa-m commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants