[3.12] gh-116510: Fix a crash due to shared immortal interned strings. by nascheme · Pull Request #124541 · python/cpython · GitHub
Skip to content

[3.12] gh-116510: Fix a crash due to shared immortal interned strings. - #124541

Merged
nascheme merged 2 commits into
python:3.12from
nascheme:intern_dict_share_main
Sep 27, 2024
Merged

[3.12] gh-116510: Fix a crash due to shared immortal interned strings.#124541
nascheme merged 2 commits into
python:3.12from
nascheme:intern_dict_share_main

Conversation

@nascheme

@nascheme nascheme commented Sep 25, 2024 •
edited by bedevere-app Bot
Loading

Copy link
Copy Markdown
Member

Fix a crash caused by immortal interned strings being shared between sub-interpreters that use basic single-phase init. In that case, the string can be used by an interpreter that outlives the interpeter that created and
interned it. For interpreters that share obmalloc state, also share the interned dict with the main interpreter.

@nascheme
pythongh-116510: Fix crash due to shared immortal interned strings.
3aa5e2a 
Fix a crash caused by immortal interned strings being shared between
sub-interpreters that use basic single-phase init.  In that case, the string
can be used by an interpreter that outlives the interpeter that created and
interned it.  For interpreters that share obmalloc state, also share the
interned dict with the main interpreter.
@bedevere-app bedevere-app Bot mentioned this pull request Sep 25, 2024
Closed
@nascheme nascheme changed the title gh-116510: Fix a crash due to shared immortal interned strings. [3.12] gh-116510: Fix a crash due to shared immortal interned strings. Sep 25, 2024
@nascheme

nascheme commented Sep 25, 2024

Copy link
Copy Markdown
Member Author

ericsnowcurrently
ericsnowcurrently requested changes Sep 26, 2024
View reviewed changes

@ericsnowcurrently ericsnowcurrently left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly LGTM

I think there are just a couple spots where we need to clear the PyInterpreterState field for the shared interned dict case.

Comment thread Objects/unicodeobject.c Outdated
Comment thread Objects/unicodeobject.c Outdated
@bedevere-app

bedevere-app Bot commented Sep 26, 2024

Copy link
Copy Markdown

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

@nascheme

nascheme commented Sep 27, 2024

Copy link
Copy Markdown
Member Author

I have made the requested changes; please review again

@bedevere-app

bedevere-app Bot commented Sep 27, 2024

Copy link
Copy Markdown

Thanks for making the requested changes!

@ericsnowcurrently: please review the changes made to this pull request.

ericsnowcurrently
ericsnowcurrently approved these changes Sep 27, 2024
View reviewed changes

@ericsnowcurrently ericsnowcurrently left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nascheme
nascheme merged commit 5dd07eb into python:3.12 Sep 27, 2024
@bedevere-bot

bedevere-bot commented Sep 27, 2024

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Arch Linux TraceRefs 3.12 has failed when building commit 5dd07eb.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1197/builds/893) and take a look at the build logs.
  4. Check if the failure is related to this commit (5dd07eb) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1197/builds/893

Failed tests:

  • test_embed
  • test_syslog
  • test_threading
  • test_import
  • test_multibytecodec
  • test_atexit
  • test_importlib
  • test_int
  • test_capi

Failed subtests:

  • test_subinterps_different_ids - test.test_embed.EmbeddingTests.test_subinterps_different_ids
  • test_single_init_extension_compat - test.test_import.SubinterpImportTests.test_single_init_extension_compat
  • test_python_compat - test.test_import.SubinterpImportTests.test_python_compat
  • test_subinterps_distinct_state - test.test_embed.EmbeddingTests.test_subinterps_distinct_state
  • test_subinterps_main - test.test_embed.EmbeddingTests.test_subinterps_main
  • test_multi_init_extension_compat - test.test_import.SubinterpImportTests.test_multi_init_extension_compat

Summary of the results of the build (if available):

==

Click to see traceback logs 
TracebackTests.test_broken_parent_from) ... ok


TracebackTests.test_broken_submodule) ... ok


TracebackTests.test_import_bug) ... ok


TracebackTests.test_syntax_error) ... ok


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 174, in test_subinterps_main
    for run in self.run_repeated_init_and_subinterpreters():
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7f22e05ffe80\nobject refcount : 0\nobject type     : 0x55b39aff42c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7f22e05ffe80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x000055b39b168908)\n\nCurrent thread 0x00007f22e0989740 (most recent call first):\n  <no Python frame>\n'


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 180, in test_subinterps_different_ids
    for run in self.run_repeated_init_and_subinterpreters():
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7f48cf32fe80\nobject refcount : 0\nobject type     : 0x557a9680d2c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7f48cf32fe80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x0000557a96981908)\n\nCurrent thread 0x00007f48cf6b9740 (most recent call first):\n  <no Python frame>\n'


TracebackTests.test_exec_failure_nested) ... ok


TracebackTests.test_nonexistent_module_nested) ... ok


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1897, in test_multi_init_extension_compat
    self.check_compatible_fresh(module, strict=True)
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1817, in check_compatible_fresh
    _, out, err = script_helper.assert_python_ok('-c', textwrap.dedent(f'''
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 166, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 151, in _assert_python
    res.fail(cmd_line)
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 76, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is -11
command line: ['/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/python', '-X', 'faulthandler', '-I', '-c', '\nimport _testcapi, sys\nassert (\n    \'_testmultiphase\' in sys.builtin_module_names or\n    \'_testmultiphase\' not in sys.modules\n), repr(\'_testmultiphase\')\nret = _testcapi.run_in_subinterp_with_config(\n    "\\nimport os, sys\\n\\ntry:\\n    import _testmultiphase\\nexcept ImportError as exc:\\n    text = \'ImportError: \' + str(exc)\\nelse:\\n    text = \'okay\'\\nos.write(sys.stdout.fileno(), text.encode(\'utf-8\'))\\n",\n    **{\'allow_fork\': False, \'allow_exec\': False, \'allow_threads\': True, \'allow_daemon_threads\': False, \'use_main_obmalloc\': True, \'gil\': 1, \'check_multi_interp_extensions\': True},\n)\nassert ret == 0, ret\n']


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1886, in test_single_init_extension_compat
    self.check_incompatible_fresh(module)
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1841, in check_incompatible_fresh
    _, out, err = script_helper.assert_python_ok('-c', textwrap.dedent(f'''
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 166, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 151, in _assert_python
    res.fail(cmd_line)
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 76, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is -11
command line: ['/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/python', '-X', 'faulthandler', '-I', '-c', '\nimport _testcapi, sys\nassert \'_testsinglephase\' not in sys.modules, \'_testsinglephase\'\nret = _testcapi.run_in_subinterp_with_config(\n    "\\nimport os, sys\\n\\ntry:\\n    import _testsinglephase\\nexcept ImportError as exc:\\n    text = \'ImportError: \' + str(exc)\\nelse:\\n    text = \'okay\'\\nos.write(sys.stdout.fileno(), text.encode(\'utf-8\'))\\n",\n    **{\'allow_fork\': False, \'allow_exec\': False, \'allow_threads\': True, \'allow_daemon_threads\': False, \'use_main_obmalloc\': True, \'gil\': 1, \'check_multi_interp_extensions\': True},\n)\nassert ret == 0, ret\n']


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 188, in test_subinterps_distinct_state
    for run in self.run_repeated_init_and_subinterpreters():
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7fef8cfbfe80\nobject refcount : 0\nobject type     : 0x55da79f132c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7fef8cfbfe80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x000055da7a087908)\n\nCurrent thread 0x00007fef8d34b740 (most recent call first):\n  <no Python frame>\n'


TracebackTests.test_broken_from) ... ok


TracebackTests.test_nonexistent_module) ... ok


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 174, in test_subinterps_main
    for run in self.run_repeated_init_and_subinterpreters():
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7fc0394bfe80\nobject refcount : 0\nobject type     : 0x55c76866b2c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7fc0394bfe80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x000055c7687df908)\n\nCurrent thread 0x00007fc03984b740 (most recent call first):\n  <no Python frame>\n'


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1945, in test_python_compat
    self.check_compatible_fresh(module, strict=True)
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1817, in check_compatible_fresh
    _, out, err = script_helper.assert_python_ok('-c', textwrap.dedent(f'''
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 166, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 151, in _assert_python
    res.fail(cmd_line)
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 76, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is -11
command line: ['/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/python', '-X', 'faulthandler', '-I', '-c', '\nimport _testcapi, sys\nassert (\n    \'threading\' in sys.builtin_module_names or\n    \'threading\' not in sys.modules\n), repr(\'threading\')\nret = _testcapi.run_in_subinterp_with_config(\n    "\\nimport os, sys\\n\\ntry:\\n    import threading\\nexcept ImportError as exc:\\n    text = \'ImportError: \' + str(exc)\\nelse:\\n    text = \'okay\'\\nos.write(sys.stdout.fileno(), text.encode(\'utf-8\'))\\n",\n    **{\'allow_fork\': False, \'allow_exec\': False, \'allow_threads\': True, \'allow_daemon_threads\': False, \'use_main_obmalloc\': True, \'gil\': 1, \'check_multi_interp_extensions\': True},\n)\nassert ret == 0, ret\n']


TracebackTests.test_unencodable_filename) ... ok


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 188, in test_subinterps_distinct_state
    for run in self.run_repeated_init_and_subinterpreters():
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7f14a6de80a0\nobject refcount : 0\nobject type     : 0x5567decca2c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7f14a6de80a0>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x00005567dee3e908)\n\nCurrent thread 0x00007f14a7168740 (most recent call first):\n  <no Python frame>\n'


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 180, in test_subinterps_different_ids
    for run in self.run_repeated_init_and_subinterpreters():
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7f06ef537e80\nobject refcount : 0\nobject type     : 0x5610d3a722c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7f06ef537e80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x00005610d3be6908)\n\nCurrent thread 0x00007f06ef8be740 (most recent call first):\n  <no Python frame>\n'


TracebackTests.test_broken_parent) ... ok


TracebackTests.test_exec_failure) ... ok

nascheme added a commit to nascheme/cpython that referenced this pull request Oct 1, 2024
@nascheme
Revert "[3.12] pythongh-116510: Fix a crash due to shared immortal in…
b2063bd 
…terned strings. (pythongh-124541)"

This reverts commit 5dd07eb.
@bedevere-app

bedevere-app Bot commented Oct 1, 2024

Copy link
Copy Markdown

Yhg1s pushed a commit that referenced this pull request Oct 1, 2024
@nascheme
[3.12] gh-124785: Revert "[3.12] gh-116510: Fix a crash due to shared…
ffc6a10 
… immortal interned strings. (gh-124541)" (#124814)

Revert "[3.12] gh-116510: Fix a crash due to shared immortal interned strings. (gh-124541)"

This reverts commit 5dd07eb.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ericsnowcurrently ericsnowcurrently ericsnowcurrently approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@nascheme @bedevere-bot @ericsnowcurrently