gh-109047: concurrent.futures catches RuntimeError by vstinner · Pull Request #109810 · python/cpython · GitHub
Skip to content

gh-109047: concurrent.futures catches RuntimeError#109810

Merged
vstinner merged 2 commits intopython:mainfrom
vstinner:cf_finalization_error
Sep 29, 2023
Merged

gh-109047: concurrent.futures catches RuntimeError#109810
vstinner merged 2 commits intopython:mainfrom
vstinner:cf_finalization_error

Conversation

@vstinner
Copy link
Copy Markdown
Member

@vstinner vstinner commented Sep 25, 2023 •
edited
Loading

concurrent.futures: The 'executor manager thread' now catches
PythonFinalizationError, it calls terminate_broken(). The exception
occurs while Python is being finalized when adding an item to the
'call queue' tries to create a new 'queue feeder' thread.

Add test_python_finalization_error() to test_concurrent_futures.

concurrent.futures._ExecutorManagerThread changes:

  • terminate_broken() no longer calls shutdown_workers() since the
    queue is no longer working anymore (read and write ends of the
    queue pipe are closed).
  • terminate_broken() now terminates child processes.
  • wait_result_broken_or_wakeup() now uses the short form of
    traceback.format_exception().

multiprocessing.Queue changes:

  • Add _terminate_broken() method.
  • _start_thread() sets _thread to None on exception to prevent
    leaking "dangling threads" even if the thread was not started
    yet.

📚 Documentation preview 📚: https://cpython-previews--109810.org.readthedocs.build/

@bedevere-app bedevere-app Bot mentioned this pull request Sep 25, 2023
Closed
@vstinner
Copy link
Copy Markdown
Member Author

vstinner commented Sep 25, 2023

@vstinner vstinner force-pushed the cf_finalization_error branch from e286574 to 1722fb1 Compare September 25, 2023 00:25
@vstinner vstinner mentioned this pull request Sep 25, 2023
Merged
@vstinner
Copy link
Copy Markdown
Member Author

vstinner commented Sep 25, 2023

Another error, because a pipe file descriptor is closed:

ERROR: test_python_finalization_error (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_python_finalization_error)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_concurrent_futures/test_process_pool.py", line 209, in test_python_finalization_error
    list(executor.map(mul, [(2, 3)] * 10))
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/concurrent/futures/process.py", line 850, in map
    results = super().map(partial(_process_chunk, fn),
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/concurrent/futures/_base.py", line 608, in map
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/concurrent/futures/process.py", line 821, in submit
    self._adjust_process_count()
  File "/home/vstinner/python/main/Lib/concurrent/futures/process.py", line 780, in _adjust_process_count
    self._spawn_process()
  File "/home/vstinner/python/main/Lib/concurrent/futures/process.py", line 798, in _spawn_process
    p.start()
  File "/home/vstinner/python/main/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/multiprocessing/context.py", line 289, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/home/vstinner/python/main/Lib/multiprocessing/popen_spawn_posix.py", line 58, in _launch
    self.pid = util.spawnv_passfds(spawn.get_executable(),
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/multiprocessing/util.py", line 453, in spawnv_passfds
    return _posixsubprocess.fork_exec(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: bad value(s) in fds_to_keep

@vstinner vstinner force-pushed the cf_finalization_error branch from 1722fb1 to 3137689 Compare September 29, 2023 18:20
@vstinner vstinner marked this pull request as ready for review September 29, 2023 18:20
@vstinner vstinner force-pushed the cf_finalization_error branch 2 times, most recently from a77c413 to a035765 Compare September 29, 2023 18:35
@vstinner
pythongh-109047: concurrent.futures catches exc on add_call_item_to_q…
f94feb1 
…ueue()

concurrent.futures: The *executor manager thread* now catches
exceptions when adding an item to the *call queue*. During Python
finalization, creating a new thread can now raise RuntimeError. Catch
the exception and call terminate_broken() in this case.

Add test_python_finalization_error() to test_concurrent_futures.

concurrent.futures._ExecutorManagerThread changes:

* terminate_broken() no longer calls shutdown_workers() since the
  queue is no longer working anymore (read and write ends of the
  queue pipe are closed).
* terminate_broken() now terminates child processes.
* wait_result_broken_or_wakeup() now uses the short form (1 argument,
  not 3) of traceback.format_exception().
* _ExecutorManagerThread.terminate_broken() now holds shutdown_lock
  to prevent race conditons with ProcessPoolExecutor.submit().

multiprocessing.Queue changes:

* Add _terminate_broken() method.
* _start_thread() sets _thread to None on exception to prevent
  leaking "dangling threads" even if the thread was not started
  yet.
@vstinner vstinner force-pushed the cf_finalization_error branch from a035765 to f94feb1 Compare September 29, 2023 18:37
@vstinner vstinner mentioned this pull request Sep 29, 2023
Closed
@vstinner vstinner enabled auto-merge (squash) September 29, 2023 18:43
@vstinner vstinner added the needs backport to 3.12 only security fixes label Sep 29, 2023
@vstinner vstinner changed the title gh-109047: concurrent.futures catches PythonFinalizationError gh-109047: concurrent.futures catches RuntimeError Sep 29, 2023
@vstinner vstinner merged commit 6351842 into python:main Sep 29, 2023
@vstinner vstinner deleted the cf_finalization_error branch September 29, 2023 19:31
@miss-islington
Copy link
Copy Markdown
Contributor

miss-islington commented Sep 29, 2023

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

@miss-islington
Copy link
Copy Markdown
Contributor

miss-islington commented Sep 29, 2023

Sorry, @vstinner, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 635184212179b0511768ea1cd57256e134ba2d75 3.12

vstinner added a commit to vstinner/cpython that referenced this pull request Sep 29, 2023
@vstinner
pythongh-109047: concurrent.futures catches PythonFinalizationError (p…
657f6b1 
…ython#109810)

concurrent.futures: The *executor manager thread* now catches
exceptions when adding an item to the *call queue*. During Python
finalization, creating a new thread can now raise RuntimeError. Catch
the exception and call terminate_broken() in this case.

Add test_python_finalization_error() to test_concurrent_futures.

concurrent.futures._ExecutorManagerThread changes:

* terminate_broken() no longer calls shutdown_workers() since the
  call queue is no longer working anymore (read and write ends of
  the queue pipe are closed).
* terminate_broken() now terminates child processes, not only
  wait until they complete.
* _ExecutorManagerThread.terminate_broken() now holds shutdown_lock
  to prevent race conditons with ProcessPoolExecutor.submit().

multiprocessing.Queue changes:

* Add _terminate_broken() method.
* _start_thread() sets _thread to None on exception to prevent
  leaking "dangling threads" even if the thread was not started
  yet.

(cherry picked from commit 6351842)
vstinner added a commit to vstinner/cpython that referenced this pull request Sep 29, 2023
@vstinner
pythongh-109047: concurrent.futures catches PythonFinalizationError (p…
49ea36d 
…ython#109810)

concurrent.futures: The *executor manager thread* now catches
exceptions when adding an item to the *call queue*. During Python
finalization, creating a new thread can now raise RuntimeError. Catch
the exception and call terminate_broken() in this case.

Add test_python_finalization_error() to test_concurrent_futures.

concurrent.futures._ExecutorManagerThread changes:

* terminate_broken() no longer calls shutdown_workers() since the
  call queue is no longer working anymore (read and write ends of
  the queue pipe are closed).
* terminate_broken() now terminates child processes, not only
  wait until they complete.
* _ExecutorManagerThread.terminate_broken() now holds shutdown_lock
  to prevent race conditons with ProcessPoolExecutor.submit().

multiprocessing.Queue changes:

* Add _terminate_broken() method.
* _start_thread() sets _thread to None on exception to prevent
  leaking "dangling threads" even if the thread was not started
  yet.

(cherry picked from commit 6351842)
@bedevere-app
Copy link
Copy Markdown

bedevere-app Bot commented Sep 29, 2023

GH-110126 is a backport of this pull request to the 3.12 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.12 only security fixes label Sep 29, 2023
@hugovk hugovk mentioned this pull request Sep 29, 2023
Merged
@bedevere-bot
Copy link
Copy Markdown

bedevere-bot commented Sep 29, 2023

Yhg1s pushed a commit that referenced this pull request Oct 2, 2023
@vstinner
[3.12] gh-109047: concurrent.futures catches RuntimeError (#109810) (#…
356de02 
…110126)

gh-109047: concurrent.futures catches PythonFinalizationError (#109810)

concurrent.futures: The *executor manager thread* now catches
exceptions when adding an item to the *call queue*. During Python
finalization, creating a new thread can now raise RuntimeError. Catch
the exception and call terminate_broken() in this case.

Add test_python_finalization_error() to test_concurrent_futures.

concurrent.futures._ExecutorManagerThread changes:

* terminate_broken() no longer calls shutdown_workers() since the
  call queue is no longer working anymore (read and write ends of
  the queue pipe are closed).
* terminate_broken() now terminates child processes, not only
  wait until they complete.
* _ExecutorManagerThread.terminate_broken() now holds shutdown_lock
  to prevent race conditons with ProcessPoolExecutor.submit().

multiprocessing.Queue changes:

* Add _terminate_broken() method.
* _start_thread() sets _thread to None on exception to prevent
  leaking "dangling threads" even if the thread was not started
  yet.

(cherry picked from commit 6351842)
@vstinner vstinner mentioned this pull request Nov 30, 2023
Closed
Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024
@vstinner @Glyphack
pythongh-109047: concurrent.futures catches PythonFinalizationError (p…
68b4f35 
…ython#109810)

concurrent.futures: The *executor manager thread* now catches
exceptions when adding an item to the *call queue*. During Python
finalization, creating a new thread can now raise RuntimeError. Catch
the exception and call terminate_broken() in this case.

Add test_python_finalization_error() to test_concurrent_futures.

concurrent.futures._ExecutorManagerThread changes:

* terminate_broken() no longer calls shutdown_workers() since the
  call queue is no longer working anymore (read and write ends of
  the queue pipe are closed).
* terminate_broken() now terminates child processes, not only
  wait until they complete.
* _ExecutorManagerThread.terminate_broken() now holds shutdown_lock
  to prevent race conditons with ProcessPoolExecutor.submit().

multiprocessing.Queue changes:

* Add _terminate_broken() method.
* _start_thread() sets _thread to None on exception to prevent
  leaking "dangling threads" even if the thread was not started
  yet.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rhettinger rhettinger Awaiting requested review from rhettinger

Assignees

@vstinner vstinner

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@vstinner @miss-islington @bedevere-bot