{{ message }}
Repository navigation
Conversation
Since decode_gzip switched to a single zlib.decompressobj (mitmproxy#7940), it stops at the end of the first gzip member and drops the rest, while gzip.GzipFile, used before, read every member (RFC 1952 allows any number of them). Keep decoding while the remaining data starts with a gzip header. The input is fed to zlib in chunks of at most 16 KiB: unused_data copies everything that follows the end of a member, so passing the whole remaining body would make decoding quadratic in the number of members.
kayaal34
reviewed
Oct 23, 2026
kayaal34
left a comment
kayaal34
left a comment
Contributor
There was a problem hiding this comment.
Ran the tests locally (test_encoding.py, 16 passed) and compared a few edge cases with Python's gzip module:
- two members: b"hello world", same as gzip.decompress
- trailing zero bytes: first member only, same as gzip.decompress
- trailing garbage after a member: returns the first member, while gzip.decompress raises BadGzipFile
- second member truncated: returns the first member, while gzip.decompress raises EOFError
Decoding 20,000 one-byte members took about 0.05s, so the chunked feeding avoids the quadratic case.
Question: is silently dropping a truncated second member intended? It matches how a single truncated member is handled (test_truncated), so it may well be the right call for a proxy, but it could be worth a short comment in the code.
(I used an AI assistant to help go through the change; I ran the tests myself.)
It is not obvious from the loop what happens when the body ends inside a later member.
Author
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Since #7940 (12.2.1),
decode_gzipuses a singlezlib.decompressobj, which stops at the end of the first gzip member. RFC 1952 allows a gzip stream to consist of several members, andgzip.GzipFile, used before, read all of them. Now the rest of the body is silently dropped:.content,.text, content views and the~bfilters only see the first member, and an addon that edits.textre-encodes and forwards the truncated body.The decoder now keeps going while the remaining data starts with a gzip header. It feeds zlib bounded chunks, because
unused_datacopies everything after the end of a member, so passing the whole rest of the body each time would make bodies with many small members quadratic to decode. zlib-wrapped bodies, which #7940 added support for, are decoded as before.Checklist