{{ message }}
fix: correct misleading "Expected any" error for Raw fields in convert() - #1169
Open
Ananthr16 wants to merge 1 commit into
Open
fix: correct misleading "Expected any" error for Raw fields in convert()#1169Ananthr16 wants to merge 1 commit into
Ananthr16 wants to merge 1 commit into
Conversation
…nvert() msgspec.Raw fields are marked with a typecode of 0. JSON/msgpack decoding never reaches typenode_simple_repr for a Raw mismatch, since the raw bytes are captured before type checking. convert() has no such interception, so a non-Raw input against a Raw-typed field falls through to the generic error path, which reported the expected type as "any" -- directly contradicting the ValidationError being raised. Report the expected type as "raw" instead, matching the wording already used for the reverse case (an already-Raw input rejected by a non-Raw target). Also documents that Raw fields can only be populated by json.decode/msgpack.decode, not by convert() (and thus not by yaml.decode/toml.decode), since there's no encoded-byte buffer left to capture once PyYAML/tomllib have already parsed the input. Fixes msgspec#1136.
Ananthr16
had a problem deploying
to
docs-preview
September 2, 2026 20:33 — with
GitHub Actions
Failure
Expected any`` error for Raw fields in convert()
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Problem
Fixes #1136.
A
Raw-typed field fed a plain decoded value throughconvert()(and therefore throughyaml.decode/toml.decode, both of which parse first and callconvert()on the result) raises aValidationErrorthat says the expected type isany:That's self-contradictory -- if
anywere really acceptable, there'd be no error at all.Root cause
msgspec.Rawfields are marked with a typecode of0(see the comment at_core.c's type-node builder: "Raw is marked with a typecode of 0, nothing to do"). Forjson.decode/msgpack.decode, that's fine: the raw byte span for aRawfield is captured up front, before any type-checking againsttype->typeshappens, so a typecode-0 mismatch is never actually reached through those paths.convert()has no equivalent interception -- it dispatches purely on the input Python object's type (convert_int,convert_str,convert_dict, ...), and each of those falls through to a generic "no match" error when none of the target'stype->typesbits are set. That generic path callstypenode_simple_repr()to build the human-readable "expected" string, and that function treats typecode0identically to realAny(bitMS_TYPE_ANYset), returning"any"for both. RealAnyfields always set theMS_TYPE_ANYbit explicitly (state->types = MS_TYPE_ANY), sotypes == 0is unambiguously theRawmarker -- but the repr function was conflating the two.The one already-supported case (
convert({"x": raw_instance}, type=Ex), where the input is already aRawobject) goes through a separateconvert_rawdispatch keyed on the input's Python type, and correctly labels a rejected input as`raw`in the reverse direction. This PR makes the forward direction consistent with that existing wording.What changed
src/msgspec/_core.c:typenode_simple_repr()now returns"raw"for typecode0instead of"any", so the error readsExpected `raw`, got `int`-- accurate, and consistent with the wording already used for the reverse mismatch direction. RealAny-typed fields are unaffected (confirmed they always carry the explicitMS_TYPE_ANYbit, never falling into thetypes == 0branch).tests/unit/test_convert.py: added a parametrized regression test (TestRaw.test_raw_mismatch_error_message) covering int/str/array/object inputs against aRaw-typed field.docs/supported-types.rst: added a note under theRawsection clarifying thatRawfields are only populated byjson.decode/msgpack.decode(which can capture original encoded bytes before decoding further);convert()and anything built on it (yaml.decode,toml.decode) operates on already-parsed Python objects, so there's no encoded-byte buffer left to capture, and only an already-constructedRawobject is accepted as input.Explicitly out of scope
The issue (filed as a question) could also be read as "should
convert()support building aRawfrom an arbitrary already-decoded value?" -- e.g. by re-serializing it to some byte format. I deliberately didn't attempt that here:Raw's whole documented purpose is "already-encoded bytes, so the encoder can skip re-encoding it," and a plain Python dict decoded from YAML isn't already encoded in any particular wire format, so picking one (JSON? msgpack?) to synthesize would be inventing new semantics rather than fixing a bug. If that's wanted, it feels like a separate design decision -- happy to hear ifRawis intended to be constructible viaconvert()at all before attempting it.How it was verified
msgspec.Rawappears not to work withmsgspec.yaml.decode#1136 (yaml.decodewith adict[str, msgspec.Raw]field, five different YAML scalar/collection shapes) against unmodifiedmainfirst, confirming the misleadingExpected `any`message.Expected `raw`, got `<type>`as expected; the existing supported case (convert({"x": raw_instance}, type=Ex)) still round-trips correctly; a genuineAny-typed field mismatch is unaffected.just rebuild=1 test-all: 6414 passed, 113 skipped (0 failed). Ran the identical command against unmodifiedmainfor comparison: 6410 passed, 113 skipped -- the +4 delta is exactly the new parametrized regression test (int/str/array/object), zero regressions.just test-doc: 7 passed (doctests unaffected).just hooks-check(ruff lint, ruff format, codespell): all passed.just doc-build: Sphinx build succeeds cleanly with the newsupported-types.rstnote.