Correctness

This fixes the missing error in #5741: a manually declared ordinary struct can reach the native reader even when the Parquet field is annotated as VARIANT. Checking the physical extension marker against the requested marker is the right distinction. Matching only the storage child names would incorrectly classify ordinary structs. The change carries the ignore setting through the scan protocol, gates the Parquet datasource check to Spark 4.1+, and translates the new error through the Spark 4.x shim.

I found two [P2] correctness issues, detailed inline. The factory validates the full relation schema rather than the actual projection, so selecting an unrelated scalar column can fail because an omitted column has a VARIANT annotation. The map recursion also matches the entries children by name or field ID, while the reader consumes map keys and values by position. A differently named physical value field can therefore bypass the annotation check.

The existing field-ID remap preserves physical metadata, and absent requested fields do not acquire an annotation to reject. Marked Variant requests and the explicit ignore setting remain accepted by the guard. The PR documents a remaining version limitation: the Arrow marker loses the annotation's spec version. That limitation predates this guard's acceptance path and is not counted as a new finding here.

Validation

Reviewed 03f9da00857d469afb7dccdf6bcfdd1f3dd8dcb4 against 38a6ec362096c7c205b379c5a5d7a7fc81c6b9e9, including all 13 changed files. I checked maintained Spark 3.5/4.0 sources for schema clipping and map matching, plus the exact DataFusion 55.1.0 and parquet-rs 59.3.0 sources verified against the lockfile checksums. An isolated Rust probe compiled and exercised the exact guard and factory block with minimal type doubles. It confirmed the two control-flow cases and checked marked requests, the ignore setting, and renamed or missing field IDs. It did not execute Arrow, DataFusion, JNI, Spark, or a real Parquet scan. Maintained Spark 3.4/4.1 branches were unavailable, so no coverage of those sources is claimed.

The PR adds 15 native cases and reports seven passing Spark 4.1.3 suite tests. I have not independently rerun them. At the September 15, 04:40 UTC refresh, only labeling passed. CI, CodeQL and the Delta gate were approval-required with zero jobs. The cached merge has the exact reviewed base/head parents and the same whole tree as the head, but there is no executed product CI to credit.

Performance

The guard walks in-memory schema metadata when the adapter is created. It adds no data-row loop or metadata fetch. It does allocate paths and lookup maps while visiting nested fields, including currently unrequested roots. Restricting validation to the actual read schema addresses that unnecessary work together with the first correctness issue. The existing ASCII fast path and cached non-ASCII name folding remain available. No benchmark or measured speedup is claimed for this validation change.

Design

The native reader is the appropriate boundary because the requested Spark schema alone cannot reveal a file annotation. An eager check also preserves rejection of an empty file when the incompatible field is requested. The check needs the same selection rules as the read itself: the requested schema for its scope, field-ID/name resolution for structs, and positional resolution for map children. Those changes address the findings without restricting legitimate projections or moving the error into per-row execution.

Abstraction & complexity

The option and structured error carrier fit the existing scan and error-conversion interfaces. The new recursive matcher duplicates struct-resolution logic already centralized in match_struct_fields. Reusing that helper for structs and handling map children explicitly would keep validation aligned with conversion, including ambiguity handling. The file-writing tests and probe factory have a useful purpose: they distinguish a real Parquet annotation reaching the adapter from an Arrow-schema hint that merely looks annotated. Their missing boundary cases are an omitted annotated root and a differently named map value, which the inline findings request.