Fix jsonable_encoder raising an unclear UnicodeDecodeError for non-UTF-8 bytes by usamabhatti1998 · Pull Request #16350 · fastapi/fastapi · GitHub
Skip to content

Fix jsonable_encoder raising an unclear UnicodeDecodeError for non-UTF-8 bytes - #16350

Open
usamabhatti1998 wants to merge 1 commit into
fastapi:masterfrom
usamabhatti1998:fix/jsonable-encoder-bytes-unicode-error
Open

usamabhatti1998 wants to merge 1 commit into
fastapi:masterfrom
usamabhatti1998:fix/jsonable-encoder-bytes-unicode-error

Conversation

@usamabhatti1998

Copy link
Copy Markdown

Summary

jsonable_encoder(some_bytes) currently calls some_bytes.decode() with no error handling. For any bytes value that isn't valid UTF-8 (raw binary data — hashes, encrypted blobs, protobuf, DB blob columns), this raises an unhandled UnicodeDecodeError straight from deep inside FastAPI's internals, with no indication of what went wrong or how to fix it — even though a working escape hatch (custom_encoder={bytes: ...}) already exists and just isn't discoverable from the error.

Fix

Wrap the decode in a named bytes_encoder() function that re-raises as a ValueError (consistent with the ValueError jsonable_encoder already raises for other unencodable types) with a message that explains the problem and points at custom_encoder as the fix. The existing pattern of naming non-trivial encoders as top-level functions (e.g. decimal_encoder, isoformat) is followed rather than keeping an inline lambda.

Testing

Added three tests to tests/test_jsonable_encoder.py:

  • test_encode_bytes_valid_utf8: valid UTF-8 bytes still encode correctly (unchanged behavior)
  • test_encode_bytes_invalid_utf8_raises_clear_error: invalid UTF-8 now raises ValueError with an actionable message
  • test_encode_bytes_invalid_utf8_with_custom_encoder: the existing custom_encoder escape hatch still works and now has a regression test

Not a breaking change for the success path; the only change is the exception type/message for previously-crashing input.

@codspeed

codspeed Bot commented Sep 14, 2026

Copy link
Copy Markdown

@VimalN2005

Copy link
Copy Markdown

@Shriprasad-P Shriprasad-P left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. Looks focused from a first pass — please confirm tests cover the new path.

@jasraj1111 jasraj1111 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pulled this branch locally and verified with pytest tests/test_jsonable_encoder.py and ruff check. The three new tests pass cleanly on Python 3.11 (Windows) and the ValueError message gives clear, actionable guidance on using custom_encoder.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants