..."))` rather than `pointer_checked(None)`. The latter rejects named capsules outright with CPython's "incorrect name" error. * `SessionContext.with_logical_extension_codec` and `with_physical_extension_codec` now wrap the returned internal context in `SessionContext` so the result has the full Python surface. The pre-existing logical setter was returning a raw internal object that lacked `sql()` and friends. `examples/datafusion-ffi-example/Cargo.toml` gains `datafusion` and `datafusion-proto` workspace dependencies for the new Rust impls. Co-Authored-By: Claude Opus 4.7 (1M context) * refactor: tighten PR1 scope to codec plumbing only Review feedback pass. PR1 is now strictly the composable codec layer + session routing + class-method serialization API. Anything that touches actual Python UDF inline encoding or Python expression wrapping moves to PR2 alongside the pickle work. Dropped: * `encode_python_scalar_udf` / `decode_python_scalar_udf` helpers from `crates/core/src/codec.rs`, along with cloudpickle and pyarrow imports. The wrapper codecs now delegate every method to `inner`. `DFPYUDF1` magic constant is kept (marked `dead_code` for now) as a reservation so PR2 has a single definition site. * `udf.rs` reverted to pre-PR1 shape. The codec no longer needs `func()` / `input_fields()` / `volatility()` / `from_parts()` accessors. Re-added by PR2 when scalar-UDF inlining lands. * `PyPhysicalExpr` class + Python wrapper + `__init__` export + `MyPhysicalExpr` FFI fixture + `_test_physical_expr.py`. No consumer in PR1 or PR2 plan documents; symmetry with `PyExecutionPlan` is not enough to justify the surface area. * Rust-side `PyLogicalPlan::to_proto` / `from_proto` and `PyExecutionPlan::to_proto` / `from_proto` deprecated wrappers. The deprecation lives entirely in the Python wrapper layer, which emits `DeprecationWarning` and forwards to `to_bytes` / `from_bytes`. Less Rust duplication. * `PythonLogicalCodec::with_default_inner` / `PythonPhysicalCodec::with_default_inner` — redundant with `impl Default`. Logic moved into `Default::default`. * `PySessionContext::default_logical_codec` / `default_physical_codec` helpers. Inlined as `Arc::new(PythonLogicalCodec::default())` at the three call sites. Tests (root: 1076, FFI example: 36) all green after the cuts. Co-Authored-By: Claude Opus 4.7 (1M context) * remove unuseful code comments * docs: rewrite codec module comments around purpose, not PR sequence The previous doc-block framed PythonLogicalCodec / PythonPhysicalCodec in terms of "PR1 delegates, PR2 will add encoding" — useful for review, useless for someone reading the code later. Reframed in terms of what the codecs exist to do: encode Python-side plan references (pure-Python UDFs, etc.) into the proto wire format so plans can cross process boundaries without the receiver having to pre-register every callable. The wrappers sit at the top of the session's codec stack and delegate non-Python encoding to a composable inner codec. Magic-prefix registry table loses the "reserved" column. Doc still notes that the in-module impls currently delegate and that encoder/decoder hooks land alongside the corresponding Python-side serialization work. Co-Authored-By: Claude Opus 4.7 (1M context) * feat(codec): forward every LogicalExtensionCodec / PhysicalExtensionCodec method to inner PythonLogicalCodec previously only overrode the four required methods on the trait plus the scalar UDF pair, so the default trait impls (returning "LogicalExtensionCodec is not provided") shadowed any downstream FFI codec for file formats, aggregate UDFs, and window UDFs. A user installing their own codec via `SessionContext.with_logical_extension_codec(...)` would silently lose access to its `try_*_file_format`, `try_*_udaf`, `try_*_udwf` implementations. Forward every trait method to `inner` so the user-installed codec is fully reachable. Same change on the physical side, including `try_*_expr`, `try_*_udaf`, `try_*_udwf` — the corresponding Python-aware paths can layer on later by intercepting before delegation. Co-Authored-By: Claude Opus 4.7 (1M context) * docs: tighten codec dispatch test docstrings The previous docstrings claimed the tests verify "PythonLogicalCodec delegates non-Python UDFs to the inner codec." That's forward-looking — the codecs currently delegate every UDF unconditionally, so the test would behave identically for Python and non-Python UDFs. Rewrite to describe what the test actually proves: the dispatch chain `PyLogicalPlan.to_bytes -> session.logical_codec -> PythonLogicalCodec -> FFI -> user impl` (and the physical mirror) forwards correctly, observable via the user codec's atomic counter incrementing after one encode pass. Co-Authored-By: Claude Opus 4.7 (1M context) * refactor(ffi-example): MyExecutionPlan emits real data via MemorySourceConfig Was a one-column `EmptyExec` stub useful only as a capsule-handoff target. Promoted to a minimal reference impl that a downstream Rust crate can copy when exposing a custom `ExecutionPlan` to datafusion-python: configurable `num_rows`, produces a single batch of sequential `Int32` values under column `value`, wrapped in `DataSourceExec` via `MemorySourceConfig::try_new_exec`. Header comment explains the typical use case (remote backend, streaming source, synthetic data generator) and the `__datafusion_execution_plan__` capsule shape downstream crates should follow. Test asserts the schema-bearing plan survives the FFI hop: a `DataSourceExec` arrives with the expected partitioning and no children. Schema details are not surfaced through the FFI display path (only the wrapping `ForeignExecutionPlan` name + inner plan name appear), so the test does not assert the column name. `to_bytes` round-trip of an FFI-imported plan is not exercised: encoding requires a physical codec that knows how to serialize `ForeignExecutionPlan`, which the default codec does not. A downstream user round-tripping such a plan must install their own codec via `with_physical_extension_codec`. Documented in the test file rather than asserted on. Co-Authored-By: Claude Opus 4.7 (1M context) * refactor: drop dormant ExecutionPlan PyCapsule round-trip `PyExecutionPlan::from_pycapsule` and the matching `__datafusion_execution_plan__` exporter have no consumer in this repo, on the POC `poc_ffi_query_planner` branch, or on any sibling branch (`testing/datafusion-distributed`, `testing/ffi-library-marker`, `tmp/ffi-with-codecs`). The pair was wired up speculatively for FFI plan handoff that no Python code path actually performs today. Drop the whole capsule round-trip for `ExecutionPlan`: * Rust `PyExecutionPlan::from_pycapsule` and `__datafusion_execution_plan__`. * Python `ExecutionPlan.from_pycapsule` and `__datafusion_execution_plan__` wrappers. * `MyExecutionPlan` FFI fixture + `_test_execution_plan.py` + lib.rs registration. Was solely a test fixture for the dropped path. * `test_execution_plan_pycapsule_protocol` in `python/tests/test_plans.py`. `PyExecutionPlan.to_bytes` / `from_bytes` survive — they encode through the session's physical codec and have real coverage. Capsule round-trip can be re-added when a concrete consumer (distributed worker, bridge library) lands. Co-Authored-By: Claude Opus 4.7 (1M context) * feat: PyExpr.to_bytes / from_bytes via session logical codec Mirrors PyLogicalPlan / PyExecutionPlan: encode through the session's installed `LogicalExtensionCodec` (or a default-inner `PythonLogicalCodec` when no `ctx` is supplied), decode against the session's function registry + codec via `parse_expr`. Rust side calls `datafusion_proto::logical_plan::to_proto::serialize_expr` and `from_proto::parse_expr`. Python wrapper threads an optional `SessionContext` through. Tests cover the session-routed roundtrip and the no-ctx default-codec encode path. Adds a third consumer of `session.logical_codec()` alongside `PyLogicalPlan` and the codec dispatch tests in the FFI example, broadening coverage of the codec stack. This is the last piece of the PR1 codec surface — follow-up pickle work (`Expr.__reduce__`, worker-scoped context, multiprocessing) can build on this without bundling the byte-level serialization API. Co-Authored-By: Claude Opus 4.7 (1M context) * test(ffi-example): assert codec roundtrip restores plan output PR review feedback: weak `is not None` checks let regressions slip past. Mirror python/tests/test_plans.py — logical compares `df.collect() == round_trip.collect()`; physical compares `str(original) == str(restored)`. Co-Authored-By: Claude Opus 4.7 (1M context) --------- Co-authored-by: Claude Opus 4.7 (1M context)

Sunbelt Computer Software

PL/B Language Development and Support