Fix loop.close() hanging on in-flight DNS resolution by JoshCLWren · Pull Request #783 · MagicStack/uvloop · GitHub
Skip to content

Fix loop.close() hanging on in-flight DNS resolution - #783

Open
JoshCLWren wants to merge 6 commits into
MagicStack:masterfrom
JoshCLWren:test/issue-777-dns-close-hang
Open

JoshCLWren wants to merge 6 commits into
MagicStack:masterfrom
JoshCLWren:test/issue-777-dns-close-hang

Conversation

@JoshCLWren

@JoshCLWren JoshCLWren commented Oct 19, 2026 •

Copy link
Copy Markdown

Fixes #777.

loop.close() currently waits for executing libuv DNS requests, so a slow or stuck resolver can block shutdown indefinitely. This revision keeps DNS on libuv's existing resolver pool and makes close return without waiting for executing lookups. It replaces the earlier executor-based implementation, which regressed throughput and shared the application's executor queue.

DNS requests are tracked with intrusive C links. Close cancels queued requests, drains handle close callbacks without blocking, and hands the native loop to a daemon cleanup thread only if executing DNS remains. That thread retains the loop until native completion; closed-loop callbacks discard results and free returned address data. Normal DNS uses the original native request and callback path without Python executor submissions, locks, or additional Python allocations. Closed-loop time() avoids accessing the native loop during cleanup.

An executing libc resolver cannot be interrupted: a lookup that never returns retains its native loop and cleanup thread. This change makes synchronous loop close nonblocking; it does not promise to interrupt DNS or change process-exit behavior.

Validation includes deterministic Linux subprocess tests for forward/reverse resolution, successful and failed late results, queued cancellation, idempotent close, retained loop lifetime and eventual collection, plus independence from a saturated application executor. Python 3.9 and 3.10 DNS suites pass locally. The Python 3.13 native debug build passes all 535 locally runnable tests (32 existing skips); the original blocked-resolver reproducer returns from close in 1 ms while native DNS sleeps for 10 seconds. A free-threaded debug build passed 180 repeated shutdown/lifetime cases with debug allocation enabled. All 18 Linux/macOS configurations and the aggregate Regression Tests check pass on cdbfbb4 (CI run), covering Python 3.9–3.15 and free-threaded 3.14/3.15, normal/debug builds, and asyncio debug mode on/off. Locally, IPv6 UDP sends to ::1 fail with EPERM even through a plain socket, so two unrelated full-suite tests cannot run successfully in this environment.

The reproducible benchmark is examples/bench/dns.py. Matching optimized uvloop and libuv builds, 15 alternating process pairs, and fixed CPU affinity give native numeric forward throughput changes of -0.54% sequential / +0.08% concurrent, and reverse changes of -1.29% / +2.14%. All paired bootstrap 95% intervals include parity. Localhost concurrent throughput changes by -0.31%; DNS still completes in about 2 ms with the application executor saturated. These are local measurements, not a guarantee for every workload. The revised implementation is ready for review.

@JoshCLWren JoshCLWren changed the title test: reproduce loop.close() hang during native DNS resolution Fix loop.close() hanging on in-flight DNS resolution Oct 19, 2026
@JoshCLWren
JoshCLWren marked this pull request as ready for review October 19, 2026 13:01
@JoshCLWren

JoshCLWren commented Oct 20, 2026 •

Copy link
Copy Markdown
Author

@JoshCLWren

JoshCLWren commented Oct 20, 2026 •

Copy link
Copy Markdown
Author

Replaced the executor implementation with native libuv DNS plus deferred cleanup of executing requests at close (cdbfbb4). The earlier performance warning applies to the superseded executor implementation.

Current comparison against 07b10a4: both uvloop and libuv built with -O2, UV_THREADPOOL_SIZE=4, 15 alternating fresh-process pairs per case, affinity to CPUs 0 and 1. Long native cases use 65,536 calls per sample; concurrent cases use 32 tasks.

Native lookup case Baseline calls/s Revision calls/s Change
Numeric forward, sequential 40,993 40,773 -0.54%
Numeric forward, concurrent 84,108 84,173 +0.08%
Reverse, sequential 29,806 29,423 -1.29%
Reverse, concurrent 65,024 66,416 +2.14%

Paired bootstrap 95% intervals for each throughput ratio include 1.0. Numeric forward uses socktype=0 and AI_NUMERICHOST to exercise native requests rather than uvloop's numeric fast path. Concurrent localhost lookup measured 13,933 vs 13,890 calls/s (-0.31%). With the application's executor occupied, native DNS latency remains about 2 ms rather than waiting for application work.

Methodology caveats: unrestricted host scheduling was noisy enough that identical-baseline A/A comparisons differed by ~12%; fixed affinity reduced this noise. Localhost sequential NSS timings remain noisy. These results show no material regression in the measured workloads, rather than proving universal performance parity. Also, the earlier executor comparison used the same unoptimized libuv archive on both sides; this new comparison corrects that build configuration and uses optimized libuv on both sides.

A reproducible harness, including A/A controls and executor saturation, is now included at examples/bench/dns.py. All 18 Linux/macOS configurations and the aggregate regression check now pass on cdbfbb4 (CI run); the PR is ready for review.

@JoshCLWren
JoshCLWren marked this pull request as ready for review October 20, 2026 04:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Loop.close() can hang indefinitely on an in-flight libuv getaddrinfo request

1 participant