Draft: explore a RustPython-owned ABI facade for pristine PyO3 by smarcd · Pull Request #7647 · RustPython/RustPython · GitHub
Skip to content

Draft: explore a RustPython-owned ABI facade for pristine PyO3#7647

Draft
smarcd wants to merge 83 commits intoRustPython:mainfrom
smarcd:abi-facade-pyo3
Draft

Draft: explore a RustPython-owned ABI facade for pristine PyO3#7647
smarcd wants to merge 83 commits intoRustPython:mainfrom
smarcd:abi-facade-pyo3

Conversation

@smarcd
Copy link
Copy Markdown

@smarcd smarcd commented Apr 21, 2026

Summary

This PR explores a RustPython-owned ABI facade in crates/capi so that pristine or near-pristine PyO3 can work against RustPython without requiring a large RustPython-specific backend split inside PyO3.

This direction is explicitly informed by:

Current verified status

  • pristine pyo3 0.28.x compiles against crates/capi
  • unchanged downstream packages:
    • rpds: package-owned tests green
    • jiter: package-owned tests green
    • blake3: package-owned tests green except the numpy-dependent case

Scope and claims

This is not claiming:

  • ready abi3 support
  • all PyO3 packages now work
  • broad drop-in CPython compatibility

What it does claim is:

  • the ABI-facade direction is viable
  • the compatibility burden can sit mostly in RustPython
  • real unchanged PyO3 packages can already be driven through this facade

Reviewer asks

  1. Is the current core/facade boundary acceptable?
  2. Are the current core hooks minimal enough?
  3. Should crates/capi be the preferred direction relative to #7562?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 21, 2026

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Apr 21, 2026

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: 0fa10971-4f0e-456d-b92b-142fa96ee755

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@bschoenmaeckers
Copy link
Copy Markdown
Contributor

Why are you sidestepping my MR. Without participating in the api discussion there?

@smarcd
Copy link
Copy Markdown
Author

smarcd commented Apr 21, 2026

Why are you sidestepping my MR. Without participating in the api discussion there?

This is a PR taking your ideas and running with them.

Comment thread crates/capi/src/symbols.rs Outdated
Comment on lines +129 to +150
#[unsafe(export_name = "PyBaseObject_Type")]
pub static mut PYBASEOBJECT_TYPE_HANDLE: *mut PyTypeObject = ptr::null_mut();
#[unsafe(export_name = "PyBool_Type")]
pub static mut PYBOOL_TYPE_HANDLE: *mut PyTypeObject = ptr::null_mut();
#[unsafe(export_name = "PyByteArray_Type")]
pub static mut PYBYTEARRAY_TYPE_HANDLE: *mut PyTypeObject = ptr::null_mut();
#[unsafe(export_name = "PyBytes_Type")]
pub static mut PYBYTES_TYPE_HANDLE: *mut PyTypeObject = ptr::null_mut();
#[unsafe(export_name = "PyDict_Type")]
pub static mut PYDICT_TYPE_HANDLE: *mut PyTypeObject = ptr::null_mut();
#[unsafe(export_name = "PyList_Type")]
pub static mut PYLIST_TYPE_HANDLE: *mut PyTypeObject = ptr::null_mut();
#[unsafe(export_name = "PyLong_Type")]
pub static mut PYLONG_TYPE_HANDLE: *mut PyTypeObject = ptr::null_mut();
#[unsafe(export_name = "PyModule_Type")]
pub static mut PYMODULE_TYPE_HANDLE: *mut PyTypeObject = ptr::null_mut();
#[unsafe(export_name = "PyTuple_Type")]
pub static mut PYTUPLE_TYPE_HANDLE: *mut PyTypeObject = ptr::null_mut();
#[unsafe(export_name = "PyType_Type")]
pub static mut PYTYPE_TYPE_HANDLE: *mut PyTypeObject = ptr::null_mut();
#[unsafe(export_name = "PyUnicode_Type")]
pub static mut PYUNICODE_TYPE_HANDLE: *mut PyTypeObject = ptr::null_mut();
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed in my PR, this will not work.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Author

@smarcd smarcd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two high-confidence issues from a focused review of the new crates/capi facade (the rest looks consistent with the draft/exploration intent).

Comment thread crates/capi/src/abstract_.rs Outdated
Comment on lines +97 to +113
#[unsafe(no_mangle)]
pub extern "C" fn PyObject_CallMethodObjArgs(
receiver: *mut PyObject,
name: *mut PyObject,
arg1: *mut PyObject,
arg2: *mut PyObject,
arg3: *mut PyObject,
arg4: *mut PyObject,
arg5: *mut PyObject,
arg6: *mut PyObject,
arg7: *mut PyObject,
arg8: *mut PyObject,
) -> *mut PyObject {
let raw = [arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8];
let nargs = raw.iter().position(|arg| arg.is_null()).unwrap_or(raw.len());
RustPython_PyObject_CallMethodObjArgsArray(receiver, name, raw.as_ptr(), nargs)
}
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate exported symbol PyObject_CallMethodObjArgs — link error / ABI corruption.

This Rust definition collides with the variadic C definition in crates/capi/csrc/pyobject_callmethodobjargs.c#L13-L20. Both have external linkage with the same name (Rust uses #[unsafe(no_mangle)], C is non-static), both end up in the same rustpython-capi rlib (the C file is compiled by cc::Build in crates/capi/build.rs), and lib.rs even adds #[used] static KEEP_PYOBJECT_CALL_METHOD_OBJ_ARGS to force this Rust version to be retained.

Most linkers (gnu-ld with --whole-archive, lld in cdylib mode, etc.) will fail with a multiple-definition error. On macOS the linker can silently pick one — and the two signatures are not interchangeable: the C entry is variadic with a NULL sentinel, this Rust entry takes 8 fixed *mut PyObject slots. If the Rust version wins, real CPython callers that pass more than 8 args (or that pass the NULL terminator) get garbage in the trailing slots; if the C version wins, this Rust function is dead code.

Since the C wrapper already implements the variadic ABI and just dispatches to RustPython_PyObject_CallMethodObjArgsArray, the simplest fix is to delete this Rust function (and the matching #[used] static in lib.rs).

Comment thread crates/capi/src/abstract_.rs Outdated
Comment on lines +138 to +142
let args = unsafe { slice::from_raw_parts(args, args_len) }
.iter()
.map(|arg| unsafe { &*resolve_object_handle(*arg) }.to_owned())
.collect::<Vec<_>>();

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Vectorcall drops keyword-argument values — wrong result / runtime panic when any kwargs are passed.

This slice reads only args_len (positional count) entries from the C-supplied args buffer. CPython's vectorcall ABI requires that when kwnames is non-NULL the buffer holds nargs + len(kwnames) entries — positional values at args[0..nargs] and keyword values at args[nargs..nargs+nkwargs]. See Python docs / vectorcall:

If kwnames is not NULL, it must be a tuple of strings naming the keyword arguments. The values of these arguments are stored in args after the positional arguments.

Downstream, FuncArgs::from_vectorcall_owned does args.drain(nargs..nargs + kw_count) (see crates/vm/src/function/argument.rs#L177-L186), which will index past the end of this truncated Vec whenever kwnames is non-empty (debug-asserts in debug, OOB drain panic in release). Any vectorcall through this facade with keyword arguments is broken.

PyObject_VectorcallMethod at L167-L172 builds its own slice the same way, but since it then re-enters this function through args.as_ptr() (still pointing into the original C buffer where the kwarg values live), fixing only PyObject_Vectorcall is enough: resolve kwnames length first and slice args_len + kwnames_len from the raw pointer.

Copy link
Copy Markdown
Author

smarcd commented Apr 22, 2026

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants