You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[1.2] .decompress(..., output_buffer_limit=None) no longer works (which is the default value) #225
importbrotlicffi# Highly compressible data over a SMALL window (lgwin=12 = 4 KiB): a tiny# compressed prefix expands hugely, forcing the many-tiny-buffer decode path.original=b"".join(bytes([65+ (i%26)]) *4096foriinrange(16)) # 64 KiBcompressed=brotlicffi.compress(original, lgwin=12)
assertlen(compressed) >64# 64 is a genuine mid-stream prefixd=brotlicffi.Decompressor()
d.decompress(compressed[:64]) # feed a mid-stream prefix, NO output_buffer_limit# The prefix was fully consumed and nothing is buffered, yet the decoder claims# it cannot accept more input:assertnotd.can_accept_more_data() # BUG: should be Trued.decompress(compressed[64:]) # raises: can_accept_more_data() is False
No limit → tiny output buffers. With no limit and non-empty input, the buffer for each decode iteration is sized at just 5 * len(input):
_calculate_buffer_size, no-limit branch → 5 * input_data_len (
# Allocate a buffer that's hopefully overlarge, but if it's not we
# don't mind: we'll spin around again.
return5*input_data_len
)
For a high-expansion chunk (e.g. 1 KiB compressed → ~1 MiB out, especially with a small lgwin), that buffer fills long before the input is exhausted, so the loop spins many times on NEEDS_MORE_OUTPUT.
On every iteration where input remains, it stashes the remainder. Whenever available_in > 0 after a decode step, the leftover compressed bytes are copied into self._unconsumed_data:
the save-unconsumed block (
The final iteration consumes the rest but never clears the stash. When a later iteration consumes the last bytes and the decoder returns NEEDS_MORE_INPUT, the loop breaks — and that break asserts available_in
== 0, so the save-block in step 2 does not run on this pass:
the NEEDS_MORE_INPUT break (
Nothing ever resets self._unconsumed_data, so it keeps a stale copy of bytes that were already decoded into the output. (My trace: 1024-byte prefix → 205 iterations of 5120-byte buffers; iter 204 had
available_in=4 and stashed those 4 bytes, iter 205 consumed them to 0 and broke — leaving the stale 4 bytes behind, with HasMoreOutput False.)
can_accept_more_data() now lies. It reports "not ready" purely because the stash is non-empty, even though the decoder actually finished the input and has no buffered output:
can_accept_more_data → False when _unconsumed_data is non-empty (
Feeding new data hits the guard that assumes non-empty _unconsumed_data means the caller ignored the protocol, and raises can_accept_more_data() is False: the guard
(
Feeding b"" (to "drain") re-feeds the stale, already-consumed bytes into the decoder, which is now past that stream position → BrotliDecoderDecompressStream returns an error (ERROR_FORMAT*).
@ngoldbaum@sethmlarson Any chance we can get this looked at? This is a pretty critical failure in the library. Users are hitting errors trying to decode normal payloads now.
Activity
Dreamsorcerer commentedon Aug 20, 2026
@ngoldbaum @sethmlarson Any chance we can get this looked at? This is a pretty critical failure in the library. Users are hitting errors trying to decode normal payloads now.
ngoldbaum commentedon Aug 20, 2026
Thanks for the ping, I don't subscribe to issues here. I'm taking a look.
ngoldbaum commentedon Aug 20, 2026
I opened #226, I'd appreciate it if you could test on your real use case.