gh-134938: Add `set_pledged_input_size` to ZstdCompressor by emmatyping · Pull Request #135010 · python/cpython · GitHub
Skip to content

gh-134938: Add set_pledged_input_size to ZstdCompressor - #135010

Merged
serhiy-storchaka merged 17 commits into
python:mainfrom
emmatyping:zstd-set-pledged-input-size
Jun 5, 2025
Merged

serhiy-storchaka merged 17 commits into
python:mainfrom
emmatyping:zstd-set-pledged-input-size

Conversation

@emmatyping

@emmatyping emmatyping commented Jun 1, 2025

Copy link
Copy Markdown
Member

This PR adds a new method set_pledged_input_size(), which allows users to declare up front how much data will be written to the compressor for a given frame. This combined with CompressionParameter.content_size_flag allows users to ensure that even for streaming compression scenarios the content size is written into the zstd frame header, which is beneficial for reducing decompression memory usage.


📚 Documentation preview 📚: https://cpython-previews--135010.org.readthedocs.build/

@emmatyping

Copy link
Copy Markdown
Member Author

Comment thread Doc/library/compression.zstd.rst Outdated
Comment thread Modules/_zstd/compressor.c Outdated
Comment thread Modules/_zstd/compressor.c Outdated
Comment thread Modules/_zstd/compressor.c Outdated
Comment thread Modules/_zstd/compressor.c Outdated
Comment thread Modules/_zstd/compressor.c Outdated
Comment thread Modules/_zstd/compressor.c Outdated
Comment thread Modules/_zstd/compressor.c Outdated
Comment thread Doc/library/compression.zstd.rst Outdated
Comment thread Modules/_zstd/compressor.c
Comment thread Modules/_zstd/compressor.c
Comment thread Modules/_zstd/compressor.c
The *mode* argument is a :class:`ZstdCompressor` attribute, either
:attr:`~.FLUSH_BLOCK`, or :attr:`~.FLUSH_FRAME`.

.. method:: set_pledged_input_size(size)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't currently explain why one would want to use the method.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a bit to describe this, but suggestions welcome.

Comment thread Doc/library/compression.zstd.rst Outdated
Comment thread Modules/_zstd/compressor.c Outdated
Comment thread Modules/_zstd/compressor.c Outdated
Comment thread Modules/_zstd/compressor.c Outdated
Comment thread Modules/_zstd/compressor.c Outdated
Comment thread Modules/_zstd/compressor.c Outdated

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that mark up should be cleaned up, but there are many occurrences of ~. in existing documentation, so this can be done in a following PR.

Otherwise LGTM. 👍

Comment thread Doc/library/compression.zstd.rst Outdated
Comment thread Doc/library/compression.zstd.rst Outdated
Comment thread Modules/_zstd/compressor.c Outdated
Comment thread Doc/library/compression.zstd.rst
@emmatyping

Copy link
Copy Markdown
Member Author

@serhiy-storchaka
Thanks for the approval!

I think that mark up should be cleaned up, but there are many occurrences of ~. in existing documentation, so this can be done in a following PR.

I'll make a follow up after this is merged to resolve these.

@serhiy-storchaka
serhiy-storchaka merged commit 4b44b34 into python:main Jun 5, 2025
@miss-islington-app

Copy link
Copy Markdown

Thanks @emmatyping for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jun 5, 2025
…onGH-135010)

(cherry picked from commit 4b44b34)

Co-authored-by: Emma Smith <emma@emmatyping.dev>
@bedevere-app

bedevere-app Bot commented Jun 5, 2025

Copy link
Copy Markdown

@bedevere-app bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Jun 5, 2025
serhiy-storchaka pushed a commit that referenced this pull request Jun 5, 2025
…135010) (GH-135173)

(cherry picked from commit 4b44b34)

Co-authored-by: Emma Smith <emma@emmatyping.dev>
taegyunkim pushed a commit to taegyunkim/cpython that referenced this pull request Aug 4, 2025
Agent-Hellboy pushed a commit to Agent-Hellboy/cpython that referenced this pull request Aug 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants