There were a few suggestions spread across different places about typing_extensions.disjoint_base vs. typing.disjoint_base. I believe these are potentially fueled because of misunderstandings of how typing, typing_extensions, Python versions and typecheckers interact. Let me explain why the current status of this PR currently is actually spec compliant and the more correct solution.

TL;DR

# mod_te.pyi`
from typing_extensions import disjoint_base

@disjoint_base
class Foo: ...

# use_te.py
from mod_te import Foo

$ mypy --no-site-packages --python-version 3.10 use_te.py
Success: no issues found in 1 source file

=> Even if targetting Python version 3.10 in the type checker, this checks just fine.

If I use typing.disjoint_base instead, it does not work and mypy even suggests exactly what this PR is already doing:

$ mypy --no-site-packages --python-version 3.10 use_ty.py
mod_ty.pyi:1: error: Module "typing" has no attribute "disjoint_base"  [attr-defined]
mod_ty.pyi:1: note: Use `from typing_extensions import disjoint_base` instead
mod_ty.pyi:1: note: See https://mypy.readthedocs.io/en/stable/runtime_troubles.html#using-new-additions-to-the-typing-module
Found 1 error in 1 file (checked 1 source file)

Some facts

typeshed currently has 17 imports of disjoint_base, all of them from typing_extensions: https://github.com/search?q=repo%3Apython%2Ftypeshed+%22import+disjoint_base%22&type=code
If you search for a bit, you will even find INVERSE Python version checks to the one (I believe) you're suggesting: https://github.com/python/typeshed/blob/f40e0da70e10e818692c3a771d5d5ac7010042f9/stdlib/crypt.pyi#L16 (in this case of course because of some semantic changes, similar but opposite to one like this, but not becaue of Python version typing compatibility)

Every Python version >= 3.0 shall support everything in typing_extensions: https://github.com/python/typeshed/blob/f40e0da70e10e818692c3a771d5d5ac7010042f9/stdlib/VERSIONS#L322

And this explicit rule about features not present in every Python version having to be imported from typing_extensions:

Features from the typing module that are not present in all supported Python versions must be imported from typing_extensions instead in typeshed stubs.

Why is that?

Every type checker ships its own copy of typeshed internally as a fallback (making imports from typing_extensions not a runtime dependency), which includes typing.pyi but also typing_extensions.pyi. pyright for example does it here, mypy here. As stub files are not read at runtime but exclusively by the type checker, just the version of the type checker and its bundled typing_extensions define what can be used and what not. Even the "Python version" flags don't have any influence on this, since (as stated above) every Python version >=3.0 has to support everything from typing_extensions.

I asked at python/typing#1532 which seems to propose that typing_extensions be recognised as part of the spec.

I would therefore like to go for the following proposal:

1. the PyO3 macros should emit typing.disjoint_base, as that is the eventual future import, and it keeps the macro code simple.
2. pyo3-introspection should recognise this import and re-write it to typing_extensions.disjoint_base.
   • We can then make pyo3-introspection accept a setting to set a minimum Python version for which to emit stubs; it could then choose not to do the re-write if the minimum Python version is high enough.
   • For the pyo3-introspection binary we could attempt to infer from the pyproject.toml; maturin would probably already know the minimum Python version and could pass this trivially.

I think for this PR it would be good enough to just land the typing.disjoint_base -> typing_extensions.disjoint_base re-write in pyo3-introspection without the setting to set the Python version floor.

@davidhewitt Thank you!

I tend to slightly prefer the approach in #6384 that pushes the choice between typing and typing_extensions into the macro: the pyo3 crate is already aware of the minimum supported Python version. This avoid having pyo3-introspection aware of the minimal supported python version and drop the error prone thing of having to inject the minimal python version into it and make sure it's actually in sync with what pyo3 compiled with.

An other "middle" ground option is to set the minimum supported python version in the introspection data and make pyo3-introspection rely on it.

WDYT? I can live with any of the three options.

the pyo3 crate is already aware of the minimum supported Python version

Sadly I don't think this is true; often PyO3 is built with a version-specific API which happens to be the current interpreter rather than a minimum version. This statement is only true if an abi3-pyXY (or abi3t-pyXY) feature is enabled.

Sadly I don't think this is true; often PyO3 is built with a version-specific API which happens to be the current interpreter rather than a minimum version.

Good point. But in this case, aren't all type checkers resolving imports using the current interpreter too? So, the version used in pyo3 will still be the correct one?

Stubs are build/wheel-specific, the "minimum python version" should be understood as the one of the build/wheel not the one of the crate/package

I guess we haven't fully resolved exactly when we expect packages to generate their stubs. I am unsure whether best practice is going to be to run stub generation as a static step and commit stubs into the repository, or to generate stubs as part of wheel build.

In the latter case - you're definitely right that current interpreter will match what the stubs contain.

If stubs are checked in - I guess as long as users are aware that they should generate stubs with their minimum supported interpreter, it'll be fine.

For the sake of unblocking these, let's go with the approach in #6384 and document that stub generation is sensitive to the interpreter version, so checked-in stubs should use the minimum-supported interpreter.

@davidhewitt & @Tpt thanks so much for the investigation and discussion!

For now I switched to the #6384 approach: the macro now emits disjoint_base from typing when PyO3 targets 3.15+ and from typing_extensions otherwise, via the same typing_or_extensions_if_less node and a cfg!(Py_3_15) const fn in the pyo3 crate. I'd suggest whichever PR lands second just drops the duplicated helper on rebase. The guide now says stubs depend on the target Python version, so checked-in stubs should be generated with the minimum supported interpreter. Checked locally: 3.14 reproduces the committed stubs, 3.15t produces from typing import disjoint_base.

To give a datapoint: To make docs rendering and typechecking statically a bit easier, in our packages we tend to check generated stubs in, instead of having to always dynamically generate them for mentioned CI jobs. Usually we do ABI compliant Python 3.12+ builds, and then have a CI job that checks whether the a generated typestub matches byte-to-byte with the one already checked in.

Great to see this moving forward. Thank you!

If stubs are checked in - I guess as long as users are aware that they should generate stubs with their minimum supported interpreter, it'll be fine.

Sadly this is not fully true because stubs might still be partial: if an exposed function is gated by Py_3_XX with 3.XX higher than the current minimum supported interpreter it won't be in the generated stubs. So, I tend to think the best way will be to generate stubs as part of the wheel build (maturin already supports that). But indeed it makes some validation/CI harder.

The amazing thing would be to generate stubs with proper version and platform gates but it's sadly not doable with the "introspection data is generated during macro expansion" idea (if you have #[cfg(windows)]#[pyfunction], the function is invisible for macro expansion when building on linux)