gh-117657: TSAN Fix races in `PyMember_Get` and `PyMember_Set`, for C extensions by dpdani · Pull Request #123211 · python/cpython · GitHub
Skip to content

gh-117657: TSAN Fix races in PyMember_Get and PyMember_Set, for C extensions - #123211

Merged
colesbury merged 32 commits into
python:mainfrom
dpdani:gh-117657-tsan-pymember-c-ext
Dec 3, 2024
Merged

gh-117657: TSAN Fix races in PyMember_Get and PyMember_Set, for C extensions#123211
colesbury merged 32 commits into
python:mainfrom
dpdani:gh-117657-tsan-pymember-c-ext

Conversation

@dpdani

@dpdani dpdani commented Aug 21, 2024

Copy link
Copy Markdown
Contributor

Fix data races that would only be visible when using C extensions.

This is a follow-up on #119368.

I'm intentionally not testing:

  • _Py_T_NONE (deprecated)
  • _Py_T_OBJECT (deprecated)
  • Py_T_STRING (immutable)
  • Py_T_STRING_INPLACE (immutable)

Py_T_CHAR

For some reason Py_T_CHAR is untested also in test_capi.test_structmembers.
In fact, it's not even in the supporting C types, which I'm using for the TSAN tests as well: old api, new api.
I'm wondering if there's a specific reason for this, or if it should be in the test suite instead.

I'm guessing that the TSAN suite should cover the thread-safety of Py_T_CHAR regardless?
Or should we dismiss it for TSAN tests as well?

I've added a new member to the T_CHAR member to the _testcapi module, and the rest of the tests don't seem to break.
I can revert this change if needed.

@picnixz picnixz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quite hard to review since everything looks the same and the functions are not ordered by pairs apparently. I didn't go through everything but I had a question and an observation.

Comment thread Include/cpython/pyatomic.h
Comment thread Include/internal/pycore_pyatomic_ft_wrappers.h Outdated
@dpdani

dpdani commented Aug 22, 2024

Copy link
Copy Markdown
Contributor Author

@picnixz

picnixz commented Aug 22, 2024

Copy link
Copy Markdown
Member

I'm not exactly sure why test_importlib fails, but I don't want to push a dummy commit just to retry it.

I've relaunched the test manually (I can't relaunch the SSL test though, I don't know why)

@colesbury colesbury left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I left a few comments below.

How long do the added tests takes to run?

Comment thread Python/structmember.c Outdated
Comment thread Python/structmember.c Outdated
Comment thread Python/structmember.c Outdated
@dpdani

dpdani commented Nov 2, 2024

Copy link
Copy Markdown
Contributor Author

I finally had some time to come back to this.

I guess that moving the stores after error checking does slightly change the externally-visible behavior.
Should we write a news entry?

@dpdani
dpdani requested a review from colesbury November 2, 2024 17:54

@colesbury colesbury left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@colesbury

Copy link
Copy Markdown
Contributor

!buildbot nogil

@bedevere-bot

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @colesbury for commit 6c5cec5 🤖

The command will test the builders whose names match following regular expression: nogil

The builders matched are:

  • AMD64 Windows Server 2022 NoGIL PR
  • AMD64 Fedora Rawhide NoGIL refleaks PR
  • AMD64 CentOS9 NoGIL Refleaks PR
  • x86-64 MacOS Intel ASAN NoGIL PR
  • AMD64 Windows PGO NoGIL PR
  • aarch64 Fedora Rawhide NoGIL PR
  • PPC64LE Fedora Rawhide NoGIL refleaks PR
  • aarch64 Fedora Rawhide NoGIL refleaks PR
  • ARM64 MacOS M1 NoGIL PR
  • AMD64 CentOS9 NoGIL PR
  • ARM64 MacOS M1 Refleaks NoGIL PR
  • AMD64 Fedora Rawhide NoGIL PR
  • PPC64LE Fedora Rawhide NoGIL PR
  • x86-64 MacOS Intel NoGIL PR

@colesbury
colesbury merged commit 979bf24 into python:main Dec 3, 2024
@colesbury

Copy link
Copy Markdown
Contributor

@bedevere-bot

Copy link