fix: correct misleading "Expected any" error for Raw fields in convert() by Ananthr16 · Pull Request #1169 · msgspec/msgspec · GitHub
Skip to content

fix: correct misleading "Expected any" error for Raw fields in convert() - #1169

Open
Ananthr16 wants to merge 1 commit into
msgspec:mainfrom
Ananthr16:fix/raw-convert-error-message
Open

fix: correct misleading "Expected any" error for Raw fields in convert()#1169
Ananthr16 wants to merge 1 commit into
msgspec:mainfrom
Ananthr16:fix/raw-convert-error-message

Conversation

@Ananthr16

Copy link
Copy Markdown

Problem

Fixes #1136.

A Raw-typed field fed a plain decoded value through convert() (and therefore through yaml.decode/toml.decode, both of which parse first and call convert() on the result) raises a ValidationError that says the expected type is any:

>>> class Repro(msgspec.Struct):
...     tools: dict[str, msgspec.Raw]
>>> msgspec.yaml.decode(b"tools:\n  x: 1\n", type=Repro)
msgspec.ValidationError: Expected `any`, got `int` - at `$.tools[...]`

That's self-contradictory -- if any were really acceptable, there'd be no error at all.

Root cause

msgspec.Raw fields are marked with a typecode of 0 (see the comment at _core.c's type-node builder: "Raw is marked with a typecode of 0, nothing to do"). For json.decode/msgpack.decode, that's fine: the raw byte span for a Raw field is captured up front, before any type-checking against type->types happens, so a typecode-0 mismatch is never actually reached through those paths.

convert() has no equivalent interception -- it dispatches purely on the input Python object's type (convert_int, convert_str, convert_dict, ...), and each of those falls through to a generic "no match" error when none of the target's type->types bits are set. That generic path calls typenode_simple_repr() to build the human-readable "expected" string, and that function treats typecode 0 identically to real Any (bit MS_TYPE_ANY set), returning "any" for both. Real Any fields always set the MS_TYPE_ANY bit explicitly (state->types = MS_TYPE_ANY), so types == 0 is unambiguously the Raw marker -- but the repr function was conflating the two.

The one already-supported case (convert({"x": raw_instance}, type=Ex), where the input is already a Raw object) goes through a separate convert_raw dispatch keyed on the input's Python type, and correctly labels a rejected input as `raw` in the reverse direction. This PR makes the forward direction consistent with that existing wording.

What changed

  • src/msgspec/_core.c: typenode_simple_repr() now returns "raw" for typecode 0 instead of "any", so the error reads Expected `raw`, got `int` -- accurate, and consistent with the wording already used for the reverse mismatch direction. Real Any-typed fields are unaffected (confirmed they always carry the explicit MS_TYPE_ANY bit, never falling into the types == 0 branch).
  • tests/unit/test_convert.py: added a parametrized regression test (TestRaw.test_raw_mismatch_error_message) covering int/str/array/object inputs against a Raw-typed field.
  • docs/supported-types.rst: added a note under the Raw section clarifying that Raw fields are only populated by json.decode/msgpack.decode (which can capture original encoded bytes before decoding further); convert() and anything built on it (yaml.decode, toml.decode) operates on already-parsed Python objects, so there's no encoded-byte buffer left to capture, and only an already-constructed Raw object is accepted as input.

Explicitly out of scope

The issue (filed as a question) could also be read as "should convert() support building a Raw from an arbitrary already-decoded value?" -- e.g. by re-serializing it to some byte format. I deliberately didn't attempt that here: Raw's whole documented purpose is "already-encoded bytes, so the encoder can skip re-encoding it," and a plain Python dict decoded from YAML isn't already encoded in any particular wire format, so picking one (JSON? msgpack?) to synthesize would be inventing new semantics rather than fixing a bug. If that's wanted, it feels like a separate design decision -- happy to hear if Raw is intended to be constructible via convert() at all before attempting it.

How it was verified

  • Reproduced the exact scenario from msgspec.Raw appears not to work with msgspec.yaml.decode #1136 (yaml.decode with a dict[str, msgspec.Raw] field, five different YAML scalar/collection shapes) against unmodified main first, confirming the misleading Expected `any` message.
  • After the fix, all five now raise Expected `raw`, got `<type>` as expected; the existing supported case (convert({"x": raw_instance}, type=Ex)) still round-trips correctly; a genuine Any-typed field mismatch is unaffected.
  • just rebuild=1 test-all: 6414 passed, 113 skipped (0 failed). Ran the identical command against unmodified main for comparison: 6410 passed, 113 skipped -- the +4 delta is exactly the new parametrized regression test (int/str/array/object), zero regressions.
  • just test-doc: 7 passed (doctests unaffected).
  • just hooks-check (ruff lint, ruff format, codespell): all passed.
  • just doc-build: Sphinx build succeeds cleanly with the new supported-types.rst note.

…nvert()

msgspec.Raw fields are marked with a typecode of 0. JSON/msgpack decoding
never reaches typenode_simple_repr for a Raw mismatch, since the raw bytes
are captured before type checking. convert() has no such interception, so
a non-Raw input against a Raw-typed field falls through to the generic
error path, which reported the expected type as "any" -- directly
contradicting the ValidationError being raised.

Report the expected type as "raw" instead, matching the wording already
used for the reverse case (an already-Raw input rejected by a non-Raw
target). Also documents that Raw fields can only be populated by
json.decode/msgpack.decode, not by convert() (and thus not by
yaml.decode/toml.decode), since there's no encoded-byte buffer left to
capture once PyYAML/tomllib have already parsed the input.

Fixes msgspec#1136.
@Ananthr16 Ananthr16 changed the title fix: correct misleading Expected any`` error for Raw fields in convert() fix: correct misleading "Expected any" error for Raw fields in convert() Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

msgspec.Raw appears not to work with msgspec.yaml.decode

1 participant