{{ message }}
Repository navigation
Stop items() iteration from reading a freed entry in CIMultiDict - #1496
Merged
asvetlov merged 3 commits intoSep 21, 2026
Merged
Conversation
_md_ensure_key() replaced a cached plain str key with its istr and then read entry->key again after dropping the old key. That decref can run a str subclass's __del__, or suspend the critical section on free-threaded builds, either of which lets the table be mutated and the entry freed. Building the istr has the same problem through a str subclass's __str__, which also invalidated entry->identity while IStr_New() still used it. Hold our own references across the conversion, only cache the istr if the version is unchanged, return the computed key directly, and have every caller read the value before the key so the decrefs are the last entry access.
asvetlov
marked this pull request as ready for review
September 21, 2026 09:53
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1496 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 37 37
Lines 6021 6069 +48
Branches 538 538
=========================================
+ Hits 6021 6069 +48
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
This comment has been minimized.
This comment has been minimized.
md_clone_from_ht() copied the source's version, so a str subclass's __str__ calling d.__init__(d.copy()) during key conversion restored the version _md_ensure_key() checked, and the istr was written through the freed entry. A clone now takes a fresh version, as the pure Python implementation already does.
Member
Author
|
@aiolibsbot rr |
4 of 6 tasks
Contributor
asvetlov
added a commit
that referenced
this pull request
Sep 21, 2026
<!-- Thank you for your contribution! --> ## What do these changes do? The ASan test command in `AGENTS.md` and in the CI `asan` job ran `python -Im pytest` with `PYTHONMALLOC=malloc`. `-I` implies `-E`, so Python ignored `PYTHONMALLOC` and kept serving small allocations from pymalloc arenas. Most `htkeys_t` tables are 512 bytes or less, so ASan could not detect use-after-free on them. This drops `-I` in both places and adds a short note explaining why it must stay out. The TSan command and job, and the `Makefile`, do not combine `-I`/`-E` with `PYTHONMALLOC` and are unchanged. ## Are there changes in behavior for the user? No. Contributor-facing only. ## Is it a substantial burden for the maintainers to support this? No. ## Related issue number N/A ## Checklist - [x] I think the code is well written - [ ] Unit tests for the changes exist: N/A, docs and CI invocation only - [x] Documentation reflects the changes - [ ] If you provide code modification, please add yourself to `CONTRIBUTORS.txt`: N/A - [x] Add a new news fragment into the `CHANGES/` folder - [x] `make doc-spelling` passes and any new technical words are added to `docs/spelling_wordlist.txt` Drafted with Claude Opus 5 (Claude Code); reviewed by @asvetlov. <details> <summary>Agent run details (optional, for reviewers)</summary> Allocator check on CPython 3.13.2: `PYTHONMALLOC=malloc python -c ...pymem_getallocatorsname()` reports `malloc`; with `-I` it reports `pymalloc`. Earlier manual check: `tests/test_multidict.py::test_items_iter_key_str_reinits` (from #1496) hits a heap-use-after-free on the pre-fix code under `python -m pytest`, but passes under `python -Im pytest`. Corrected command on current master, ASan+UBSan build, isolated 3.13.2 venv: `LD_PRELOAD=$ASAN_SO ASAN_OPTIONS=detect_leaks=0:halt_on_error=1 PYTHONMALLOC=malloc python -m pytest tests -q --no-cov -k "not test_leak"` gave 2048 passed, 21 skipped, 11 deselected. So turning on real malloc in CI does not make master fail. `make doc-spelling`: passed. Pre-commit hooks: passed. </details>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

What do these changes do?
Fixes a use-after-free in the C extension's key conversion that crashed the cp315t wheel test on master (job,
list(md.items())intest_race_condition_iterator_vs_mutation).When iterating a
CIMultiDictwhose stored key is a plainstr,_md_ensure_key()caches theistrin the entry, drops the old key, and then readentry->keyagain. Dropping the old key can run astrsubclass's__del__, or on free-threaded builds suspend the iterator's critical section; either lets the table be mutated and the entry freed before that reload. Building theistritself can run astrsubclass's__str__, which freedentry->identitywhileIStr_New()was still using it._md_ensure_key()now holds its own references across the conversion, only caches theistrif the version is unchanged, and returns the computed key instead of reloading it.md_clone_from_ht()now assigns a new version instead of copying the source's, so a__str__that callsd.__init__(d.copy())can no longer restore the version that check relies on; the pure Python implementation already behaved this way.md_next(),md_prev()and both finders read the value before the key, so the decrefs are the last access to the entry. The pure Python implementation does not cache the converted key and is unaffected.Are there changes in behavior for the user?
No, apart from the crash being gone.
getversion()of acopy()or of a multidict re-initialized from another one now differs from the source's on the C extension too, as it already did in pure Python. A mutation from a key's__str__/__del__during iteration now surfaces as the usualRuntimeError: MultiDict is changed during iteration.Is it a substantial burden for the maintainers to support this?
No.
Related issue number
N/A
Checklist
CONTRIBUTORS.txt(N/A)CHANGES/foldermake doc-spellingpasses and any new technical words are added todocs/spelling_wordlist.txtDrafted with Claude Code (Claude Opus 5); reviewed by @asvetlov.
Agent run details (optional, for reviewers)
Both new tests segfault the unpatched extension on a plain CPython 3.14 release build; ASan pins them to
hashtable.h:869(Py_NewRef(entry->key)) andIStr_New().With the patch:
--no-c-extensions): 1042 passedMULTIDICT_DEBUG_BUILD=1 MULTIDICT_ASAN_BUILD=1,PYTHONMALLOC=malloc(run without-I, which would drop it): 2058 passed, no ASan/UBSan reportsMULTIDICT_DEBUG_BUILD=1: 2072 passedFree-threaded mechanism: with a
Py_BEGIN_ALLOW_THREADS+nanosleep()window injected right after the old key decref (not committed), a 4-threadCIMultiDictwriter/reader stress on 3.14.7t segfaulted 10/10 runs before the fix and 0/10 after.make doc-spellingand pre-commit pass.