BLD/RFC: compatibility with Python 3.15's Limited API

Context: there are currently 2 competing PEPs (803 and 809) that aim to specify a new Limited API for Python 3.15 that would be compatible with both GIL-enabled and free-threaded builds. Since astropy is already compliant with Python 3.11's Limited API my hope is that it would not be a great deal more work to also get compatibility with this other API, but of course it's too early to tell and I'm merely speculating here. This issue is intended to help keep track of progress on this matter, both locally and upstream

Upstream issues:

@kumaraditya303

I didn't realize astropy and yt had both been converted to use abi3 builds. Since astropy has more nontrivial use of the threading module, I thought I'd try getting a build working.

tl;dr: this is going to take some pretty substantial surgery in astropy C internals to get working.

In order to get everything to build, you'll need to install the branches of packaging tools that have experimental support for the opaque PyObject build. You'll also need to compile NumPy, SciPy, and PyErfa from source. For NumPy you'll need to use the experimental opaque branch on @kumaraditya303's fork of NumPy.

Once you do that, you'll also need to patch extension-helpers:

diff --git a/extension_helpers/_setup_helpers.py b/extension_helpers/_setup_helpers.py
index 3d27971..fe56596 100644
--- a/extension_helpers/_setup_helpers.py
+++ b/extension_helpers/_setup_helpers.py
@@ -9,6 +9,7 @@ import os
 import shutil
 import subprocess
 import sys
+import sysconfig
 from collections import defaultdict
 
 from setuptools import Extension, find_packages
@@ -25,6 +26,8 @@ __all__ = ["get_compiler", "get_extensions", "pkg_config"]
 
 log = logging.getLogger(__name__)
 
+FREE_THREADED_BUILD = bool(sysconfig.get_config_var("Py_GIL_DISABLED"))
+
 
 def get_compiler():
     """
@@ -154,6 +157,8 @@ def get_extensions(srcdir="."):
         for ext in ext_modules:
             ext.py_limited_api = True
             ext.define_macros.append(("Py_LIMITED_API", version_hex))
+            if FREE_THREADED_BUILD:
+                ext.define_macros.append(("_Py_OPAQUE_PYOBJECT", 1))
 
     return ext_modules

And astropy itself:

diff --git a/setup.py b/setup.py
index 71e7fcf65e..dddd107ad0 100755
--- a/setup.py
+++ b/setup.py
@@ -46,7 +46,7 @@ class EditableInstallWithStubs(editable_wheel):
 # Specify the minimum version for the Numpy C-API
 for ext in ext_modules:
     if ext.include_dirs and "numpy" in ext.include_dirs[0]:
-        ext.define_macros.append(("NPY_TARGET_VERSION", "NPY_1_25_API_VERSION"))
+        ext.define_macros.append(("NPY_TARGET_VERSION", "NPY_2_5_API_VERSION"))
         ext.define_macros.append(("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION"))
 
 setup(

That's sufficient to start a build with the correct NumPy API version, but unfortunately it leads to errors like this:

Details

astropy/wcs/include/astropy_wcs/astropy_wcs.h:14:3: error: unknown type name 'PyObject_HEAD'
   14 |   PyObject_HEAD
      |   ^
astropy/wcs/include/astropy_wcs/astropy_wcs.h:15:13: error: expected ';' at end of declaration list
   15 |   pipeline_t x;
      |             ^
      |             ;
In file included from astropy/wcs/src/astropy_wcs.c:7:
astropy/wcs/include/astropy_wcs/wcslib_wrap.h:14:3: error: unknown type name 'PyObject_HEAD'
   14 |   PyObject_HEAD
      |   ^
In file included from astropy/wcs/src/astropy_wcs.c:8:
astropy/wcs/include/astropy_wcs/wcslib_tabprm_wrap.h:15:3: error: unknown type name 'PyObject_HEAD'
   15 |   PyObject_HEAD
      |   ^
In file included from astropy/wcs/src/astropy_wcs.c:9:
astropy/wcs/include/astropy_wcs/wcslib_auxprm_wrap.h:10:3: error: unknown type name 'PyObject_HEAD'
   10 |   PyObject_HEAD
      |   ^
In file included from astropy/wcs/src/astropy_wcs.c:10:
astropy/wcs/include/astropy_wcs/wcslib_prjprm_wrap.h:10:5: error: unknown type name 'PyObject_HEAD'
   10 |     PyObject_HEAD
      |     ^
In file included from astropy/wcs/src/astropy_wcs.c:11:
astropy/wcs/include/astropy_wcs/wcslib_celprm_wrap.h:10:5: error: unknown type name 'PyObject_HEAD'
   10 |     PyObject_HEAD
      |     ^
In file included from astropy/wcs/src/astropy_wcs.c:12:
astropy/wcs/include/astropy_wcs/wcslib_units_wrap.h:15:3: error: unknown type name 'PyObject_HEAD'
   15 |   PyObject_HEAD
      |   ^
In file included from astropy/wcs/src/astropy_wcs.c:13:
astropy/wcs/include/astropy_wcs/wcslib_wtbarr_wrap.h:15:3: error: unknown type name 'PyObject_HEAD'
   15 |   PyObject_HEAD
      |   ^
In file included from astropy/wcs/src/astropy_wcs.c:14:
astropy/wcs/include/astropy_wcs/distortion_wrap.h:15:3: error: unknown type name 'PyObject_HEAD'
   15 |   PyObject_HEAD
      |   ^
astropy/wcs/include/astropy_wcs/distortion_wrap.h:16:22: error: expected ';' at end of declaration list
   16 |   distortion_lookup_t                    x;
      |                      ^
      |                      ;
In file included from astropy/wcs/src/astropy_wcs.c:15:
astropy/wcs/include/astropy_wcs/sip_wrap.h:15:3: error: unknown type name 'PyObject_HEAD'
   15 |   PyObject_HEAD
      |   ^
astropy/wcs/include/astropy_wcs/sip_wrap.h:16:8: error: expected ';' at end of declaration list
   16 |   sip_t x;
      |        ^
      |        ;
astropy/wcs/src/astropy_wcs.c:92:24: error: no member named 'x' in 'struct (unnamed at astropy/wcs/include/astropy_wcs/astropy_wcs.h:13:9)'
   92 |   pipeline_free(&self->x);
      |                  ~~~~  ^
astropy/wcs/src/astropy_wcs.c:109:27: error: no member named 'x' in 'struct (unnamed at astropy/wcs/include/astropy_wcs/astropy_wcs.h:13:9)'
  109 |     pipeline_clear(&self->x);
      |                     ~~~~  ^
astropy/wcs/src/astropy_wcs.c:155:13: error: no member named 'x' in 'struct (unnamed at astropy/wcs/include/astropy_wcs/astropy_wcs.h:13:9)'
  155 |       self->x.det2im[i] = &(((PyDistLookup*)py_det2im[i])->x);
      |       ~~~~  ^
astropy/wcs/src/astropy_wcs.c:155:60: error: no member named 'x' in 'struct (unnamed at astropy/wcs/include/astropy_wcs/distortion_wrap.h:14:9)'
  155 |       self->x.det2im[i] = &(((PyDistLookup*)py_det2im[i])->x);
      |                             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  ^
astropy/wcs/src/astropy_wcs.c:170:11: error: no member named 'x' in 'struct (unnamed at astropy/wcs/include/astropy_wcs/astropy_wcs.h:13:9)'
  170 |     self->x.sip = &(((PySip*)py_sip)->x);
      |     ~~~~  ^
astropy/wcs/src/astropy_wcs.c:170:39: error: no member named 'x' in 'struct (unnamed at astropy/wcs/include/astropy_wcs/sip_wrap.h:14:9)'
  170 |     self->x.sip = &(((PySip*)py_sip)->x);
      |                     ~~~~~~~~~~~~~~~~  ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]

All of the types that rely on extending PyObject statically will need to be updated to use heap types. Additionally, all the C extensions defined with PyModuleDef_HEAD_INIT will need to be converted to use the new PEP 793 initialization API.

I didn't realize astropy had raw C extensions as well as cython-generated extensions. Packages using bindings generators with no "manual" C extensions will have an easier time supporting the new stable ABI.

I don't think removing raw C extensions is feasible request. I think some are bundled and beyond our control.

Thanks for plugging in, Nathan.

To some extent, the patch you proposed for extension helpers would be a minimal version of astropy/extension-helpers#132, and I'd support upstreaming it since my PR is stuck in limbo.

Packages using bindings generators with no "manual" C extensions will have an easier time supporting the new stable ABI.

Yeah, that's been my experience with complying to the current abi3. Astropy took much longer to adapt than any other packages I worked on for precisely this reason. That said I'm still unsure that it'd be worth converting to generated low level code, but I'm considering exploring this idea, FWIW. In the mean time I can do the manual conversion to heap types as I already did recently.

I don't think removing raw C extensions is feasible request.

Didn't mean to imply that.

and I'd support upstreaming it since my PR is stuck in limbo

It shouldn't be upstreamed until/if PEP 803 is accepted by the Python steering council. This is all to facilitate testing in support of PEP 803.

In the mean time I can do the manual conversion to heap types as I already did recently.

Nice, that's the hardest refactor. Supporting PEP 793 module initalization is a lot easier.

It shouldn't be upstreamed until/if PEP 803 is accepted by the Python steering council.

good point. I'll keep your patch to a branch then.

I checked everything you described to my fork of astropy under a exp/pep803 branch, and tried building locally with

uv sync --only-group build -p 3.15 && EXTENSION_HELPERS_PY_LIMITED_API=cp315 uv sync --inexact

and to my surprise, the builds appears successful. I must be missing something. Do you see any relevant difference with what you did ?

I think it's _Py_OPAQUE_PYOBJECT that you're missing

Ah, of course, I forgot it was only enabled with free-threaded builds. I'll try again with 3.15t

Ah, of course, I forgot it was only enabled with free-threaded builds. I'll try again with 3.15t

No - that shouldn't actually be necessary. The idea is that it works the same on both freethreaded and non-freethreaded builds. Of course, eventually you'll want to test it on both, but you can build it with either.

[...] Ah, but Nathan's set up extensionhelpers to only add that flag on freethreaded builds.... That isn't required though - it's just the way he's set it up.

setuptools needs new UI to enable that and I didn't want to bikeshed over it until I get the attention of a setuptools maintainer

@ngoldbaum

I tried again building my branch with

uv venv -p 3.15t
uv pip install --group build pyproject.toml # automatically use patched numpy and extension-helpers
EXTENSION_HELPERS_PY_LIMITED_API=cp315 uv pip install -e . --no-build-isolation

but this immediately stops from setuptools:

ValueError: `py_limited_api='cp315'` not supported. `Py_LIMITED_API` is currently incompatible with `Py_GIL_DISABLED`. See https://github.com/python/cpython/issues/111506.

@ngoldbaum, did you also patch setuptools in your attempt ? or am I still missing something ?

TST: Add direct tests for join_inner in table/_np_utils Cython extension

Yes, I patched CPython, pip, packaging, and setuptools. See the discourse post I made for more details if you want to replicate what I did:

https://discuss.python.org/t/pep-803-stable-abi-for-free-threaded-builds-packaging-thread/104976/108

But also no need to do that if you don't want to. You don't need to replicate my build errors to do the needed refactors.

Hopefully by 3.15.0b1 in a month or two the new ABI will all be ironed out and be easier to experiment with. Until then it'll be a little painful to try it.

Ok, thanks for all the details !
Do you know whether PEP 803 needs to be approved before beta 1 ? I've been following the progress from afar but I don't know when to expect a decision would be made.

We should have an update from the steering council in the next week or two. The CPython API and ABI is frozen after beta 1. While fixes can still happen, the intention is for downstream projects to be able to build and distribute wheels generated with beta 1 or newer.

for visibility: PEP 803 was accepted yesterday (congratz, Nathan !)

I'll update the description to track the implementation.

I think for astropy to support the new stable ABI, it first needs to migrate all its heap types to use PEP 697 ¹ layout which allows defining object without PyObject header.

https://peps.python.org/pep-0697/ ↩

Yes, that's already been pointed out and is tracked in #19479, as sub issue to this one. Or is there something else I'm missing ?

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository navigation

BLD/RFC: compatibility with Python 3.15's Limited API #19249

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Sunbelt Computer Software

PL/B Language Development and Support

BLD/RFC: compatibility with Python 3.15's Limited API #19249

Description

Activity

ngoldbaum commented on Mar 17, 2026

pllim commented on Mar 17, 2026

neutrinoceros commented on Mar 17, 2026

ngoldbaum commented on Mar 17, 2026

neutrinoceros commented on Mar 17, 2026

neutrinoceros commented on Mar 17, 2026

da-woods commented on Mar 17, 2026

neutrinoceros commented on Mar 17, 2026

da-woods commented on Mar 17, 2026

da-woods commented on Mar 17, 2026

ngoldbaum commented on Mar 17, 2026

neutrinoceros commented on Mar 18, 2026

ngoldbaum commented on Mar 18, 2026

neutrinoceros commented on Mar 18, 2026

ngoldbaum commented on Mar 18, 2026

neutrinoceros commented on Mar 30, 2026

kumaraditya303 commented on Mar 30, 2026

Footnotes

neutrinoceros commented on Mar 30, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions