gh-150942: Speed up asyncio Future.remove_done_callback by muhamedfazalps · Pull Request #153013 · python/cpython · GitHub
Skip to content

gh-150942: Speed up asyncio Future.remove_done_callback#153013

Open
muhamedfazalps wants to merge 1 commit into
python:mainfrom
muhamedfazalps:speedup-asyncio-callback-compaction
Open

gh-150942: Speed up asyncio Future.remove_done_callback#153013
muhamedfazalps wants to merge 1 commit into
python:mainfrom
muhamedfazalps:speedup-asyncio-callback-compaction

Conversation

@muhamedfazalps

@muhamedfazalps muhamedfazalps commented Jul 4, 2026

Copy link
Copy Markdown

Summary

Use _PyList_AppendTakeRef instead of PyList_Append to avoid an extra incref/decref pair when compaction appends callbacks to the new list.

Fixes gh-150942

Motivation

The issue lists this as a possible performance improvement:

Modules/_asynciomodule.c:1213 | asyncio future done-callback compaction | _PyList_AppendTakeRef

The current code:

PyObject *item = PyList_GET_ITEM(self->fut_callbacks, i);
Py_INCREF(item);
ret = PyObject_RichCompareBool(PyTuple_GET_ITEM(item, 0), fn, Py_EQ);
if (ret == 0) {
    if (j < len) {
        PyList_SET_ITEM(newlist, j, item);
        j++;
        continue;
    }
    ret = PyList_Append(newlist, item);
}
Py_DECREF(item);

The item is already owned (Py_INCREF was called earlier), so PyList_Append increments the refcount and then Py_DECREF decrements it - an unnecessary pair.

Changes

PyObject *item = PyList_GET_ITEM(self->fut_callbacks, i);
Py_INCREF(item);
ret = PyObject_RichCompareBool(PyTuple_GET_ITEM(item, 0), fn, Py_EQ);
if (ret == 0) {
    if (j < len) {
        PyList_SET_ITEM(newlist, j, item);
        j++;
        continue;
    }
    ret = _PyList_AppendTakeRef((PyListObject *)newlist, item);
}
else {
    Py_DECREF(item);
}

_PyList_AppendTakeRef takes ownership of the reference without incrementing it, eliminating the unnecessary incref/decref pair.

Testing

  • Existing asyncio tests pass
  • The change is a performance optimization with no behavioral change

Discovered during open source research. https://buymeacoffee.com/muhamedfazalps

@muhamedfazalps
pythongh-150942: Speed up asyncio Future.remove_done_callback
ef15bd9 
Use _PyList_AppendTakeRef instead of PyList_Append to avoid an
extra incref/decref pair when compaction appends callbacks to the
new list.

The item is already owned (Py_INCREF was called earlier in the loop),
so _PyList_AppendTakeRef takes ownership of that reference directly
without incrementing it again.
@bedevere-app

bedevere-app Bot commented Jul 4, 2026

Copy link
Copy Markdown

@python-cla-bot

python-cla-bot Bot commented Jul 4, 2026

Copy link
Copy Markdown

The following commit authors need to sign the Contributor License Agreement:

CLA not signed

@bedevere-app bedevere-app Bot mentioned this pull request Jul 4, 2026
Open
@picnixz

picnixz commented Jul 4, 2026

Copy link
Copy Markdown
Member

Please, we need to see benchmarks and how it improved the resutl. JUst changing the code is not sufficient.

@picnixz

picnixz commented Jul 4, 2026

Copy link
Copy Markdown
Member

See #150942 (comment).

picnixz
picnixz requested changes Jul 4, 2026
View reviewed changes

@picnixz picnixz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please include benchmarks first on the issue, sign the CLA and add a NEWS entry.

@bedevere-app

bedevere-app Bot commented Jul 4, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@picnixz picnixz picnixz requested changes
@1st1 1st1 Awaiting requested review from 1st1 1st1 is a code owner
@asvetlov asvetlov Awaiting requested review from asvetlov asvetlov is a code owner
@kumaraditya303 kumaraditya303 Awaiting requested review from kumaraditya303 kumaraditya303 is a code owner
@willingc willingc Awaiting requested review from willingc willingc is a code owner

Assignees

No one assigned

Labels

awaiting changes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Improve performance by using reference stealing methods

2 participants

@muhamedfazalps @picnixz