Bug report
Bug description:
In the free-threaded build, a whole-buffer BytesIO.read() returns self->buf by reference (Py_NewRef) instead of a copy,
|
if (size > 1 && |
|
self->pos == 0 && size == PyBytes_GET_SIZE(self->buf) && |
|
FT_ATOMIC_LOAD_SSIZE_RELAXED(self->exports) == 0) { |
|
self->pos += size; |
|
return Py_NewRef(self->buf); |
|
} |
so resize_buffer_lock_held() can reallocate that same self->buf in place (_PyBytes_Resize)
|
if (SHARED_BUF(self)) { |
|
if (unshare_buffer_lock_held(self, alloc) < 0) |
|
return -1; |
|
} |
|
else { |
|
if (_PyBytes_Resize(&self->buf, alloc) < 0) |
|
return -1; |
|
} |
while another thread concurrently decrefs the read-returned reference (_Py_DecRefShared), racing on the same buffer object without synchronization.
Reproducer:
import io
from threading import Thread, Barrier
shared = io.BytesIO(b'x' * 64)
N_W, N_R = 2, 8
barrier = Barrier(N_W + N_R)
def writer():
barrier.wait()
for i in range(20000):
try:
shared.seek(0)
shared.write(b'a' * (64 + (i & 63)))
except Exception:
pass
def reader():
barrier.wait()
for _ in range(20000):
try:
shared.seek(0)
shared.read()
except Exception:
pass
if __name__ == "__main__":
threads = [Thread(target=writer) for _ in range(N_W)]
threads += [Thread(target=reader) for _ in range(N_R)]
for t in threads: t.start()
for t in threads: t.join()TSAN Report:
==================
WARNING: ThreadSanitizer: data race (pid=2292869)
Read of size 8 at 0x7fffbc080190 by thread T2:
#0 _PyObject_MiRealloc /cpython/Objects/obmalloc.c:360:13
#1 PyObject_Realloc /cpython/Objects/obmalloc.c:1731:12
#2 _PyBytes_Resize /cpython/Objects/bytesobject.c:3389:9
#3 resize_buffer_lock_held /cpython/./Modules/_io/bytesio.c:176:13
#4 write_bytes_lock_held /cpython/./Modules/_io/bytesio.c:216:13
#5 _io_BytesIO_write_impl /cpython/./Modules/_io/bytesio.c:782:20
#6 bytesio_setstate_lock_held /cpython/./Modules/_io/bytesio.c:927:14
#7 bytesio_setstate /cpython/./Modules/_io/bytesio.c:980:11
#8 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h:4261:35
...
Previous atomic write of size 8 at 0x7fffbc080190 by thread T8:
#0 _Py_atomic_compare_exchange_ssize /cpython/./Include/cpython/pyatomic_gcc.h:130:10
#1 _Py_DecRefSharedIsDead /cpython/Objects/object.c:406:15
#2 _Py_DecRefSharedDebug /cpython/Objects/object.c:425:9
#3 _Py_DecRefShared /cpython/Objects/object.c:433:5
#4 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h
...
SUMMARY: ThreadSanitizer: data race /cpython/Objects/obmalloc.c:360:13 in _PyObject_MiRealloc
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Linked PRs
Bug report
Bug description:
In the free-threaded build, a whole-buffer
BytesIO.read()returnsself->bufby reference (Py_NewRef) instead of a copy,cpython/Modules/_io/bytesio.c
Lines 432 to 437 in 9e863fa
so
resize_buffer_lock_held()can reallocate that sameself->bufin place (_PyBytes_Resize)cpython/Modules/_io/bytesio.c
Lines 171 to 178 in 9e863fa
while another thread concurrently decrefs the read-returned reference (
_Py_DecRefShared), racing on the same buffer object without synchronization.Reproducer:
TSAN Report:
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Linked PRs
BytesIObuffer in free-threaded builds (GH-151651) #153151BytesIObuffer in free-threaded builds (GH-151651) #153153