Repository navigation
Fix loop.close() hanging on in-flight DNS resolution - #783
JoshCLWren wants to merge 6 commits into
Conversation
|
Replaced the executor implementation with native libuv DNS plus deferred cleanup of executing requests at close (cdbfbb4). The earlier performance warning applies to the superseded executor implementation. Current comparison against 07b10a4: both uvloop and libuv built with Paired bootstrap 95% intervals for each throughput ratio include 1.0. Numeric forward uses socktype=0 and AI_NUMERICHOST to exercise native requests rather than uvloop's numeric fast path. Concurrent localhost lookup measured 13,933 vs 13,890 calls/s (-0.31%). With the application's executor occupied, native DNS latency remains about 2 ms rather than waiting for application work. Methodology caveats: unrestricted host scheduling was noisy enough that identical-baseline A/A comparisons differed by ~12%; fixed affinity reduced this noise. Localhost sequential NSS timings remain noisy. These results show no material regression in the measured workloads, rather than proving universal performance parity. Also, the earlier executor comparison used the same unoptimized libuv archive on both sides; this new comparison corrects that build configuration and uses optimized libuv on both sides. A reproducible harness, including A/A controls and executor saturation, is now included at examples/bench/dns.py. All 18 Linux/macOS configurations and the aggregate regression check now pass on cdbfbb4 (CI run); the PR is ready for review. |

Fixes #777.
loop.close()currently waits for executing libuv DNS requests, so a slow or stuck resolver can block shutdown indefinitely. This revision keeps DNS on libuv's existing resolver pool and makes close return without waiting for executing lookups. It replaces the earlier executor-based implementation, which regressed throughput and shared the application's executor queue.DNS requests are tracked with intrusive C links. Close cancels queued requests, drains handle close callbacks without blocking, and hands the native loop to a daemon cleanup thread only if executing DNS remains. That thread retains the loop until native completion; closed-loop callbacks discard results and free returned address data. Normal DNS uses the original native request and callback path without Python executor submissions, locks, or additional Python allocations. Closed-loop
time()avoids accessing the native loop during cleanup.An executing libc resolver cannot be interrupted: a lookup that never returns retains its native loop and cleanup thread. This change makes synchronous loop close nonblocking; it does not promise to interrupt DNS or change process-exit behavior.
Validation includes deterministic Linux subprocess tests for forward/reverse resolution, successful and failed late results, queued cancellation, idempotent close, retained loop lifetime and eventual collection, plus independence from a saturated application executor. Python 3.9 and 3.10 DNS suites pass locally. The Python 3.13 native debug build passes all 535 locally runnable tests (32 existing skips); the original blocked-resolver reproducer returns from close in 1 ms while native DNS sleeps for 10 seconds. A free-threaded debug build passed 180 repeated shutdown/lifetime cases with debug allocation enabled. All 18 Linux/macOS configurations and the aggregate Regression Tests check pass on cdbfbb4 (CI run), covering Python 3.9–3.15 and free-threaded 3.14/3.15, normal/debug builds, and asyncio debug mode on/off. Locally, IPv6 UDP sends to ::1 fail with EPERM even through a plain socket, so two unrelated full-suite tests cannot run successfully in this environment.
The reproducible benchmark is
examples/bench/dns.py. Matching optimized uvloop and libuv builds, 15 alternating process pairs, and fixed CPU affinity give native numeric forward throughput changes of -0.54% sequential / +0.08% concurrent, and reverse changes of -1.29% / +2.14%. All paired bootstrap 95% intervals include parity. Localhost concurrent throughput changes by -0.31%; DNS still completes in about 2 ms with the application executor saturated. These are local measurements, not a guarantee for every workload. The revised implementation is ready for review.