{{ message }}
Repository navigation
Count Azure writer memoryview inputs in bytes - #965
Merged
ddelange merged 1 commit intoOct 12, 2026
Merged
Conversation
4 of 6 tasks
5 of 6 tasks
ddelange
added a commit
that referenced
this pull request
Oct 28, 2026
* [s3] Count S3 multipart writer memoryview inputs in bytes
MultipartWriter.write wrapped its argument in memoryview(b) without a
byte cast, so for a typed buffer (memoryview.cast('I'), array('d'))
write() returned and tell() reported the number of elements, and parts
were cut every part_size elements rather than bytes. Cast to bytes, as
the S3 reader already does and as #965/#971 did for Azure and WebHDFS.
* Test that a rejected multipart write leaves close working
---------
Co-authored-by: youdie006 <youdie006@users.noreply.github.com>
Co-authored-by: ddelange <14880945+ddelange@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Azure block and append writers accept typed and multidimensional
memoryviewinputs, but count their elements instead of their bytes. Writing a 16-byte buffer cast toIuploads all 16 bytes whilewrite()returns 4 andtell()reports 4; a 2-by-2 view reports 2. This breaks the byte-count contract for callers streaming array buffers to Azure Blob Storage.Use the byte count returned by the underlying
BytesIO.write()for both writers. The upload contents and buffering thresholds are unchanged. Found during source inspection and reproduced on unmodifieddevelop(418d3ee).Tests
The regression covers block and append modes, buffered and immediately uploaded parts, consecutive writes, and bytes, bytearray, byte views, integer views and multidimensional views. It compares
write()andtell()withBytesIO, then reads back the uploaded bytes through the public API.python -m pytest tests/ -q: 492 passed, 4 skipped.make lint: all checks passed (including Ruff, ty and pydoclint).Azure coverage uses the repository's in-memory fake client; no live Azure service was tested. The four skips are the missing-dependency environment checks and the test requiring a local Moto server.
Checklist
python update_helptext.pyin case there are API changes (not applicable; no API changes)