[SPARK-59511][SQL] Reduce ANSI type coercion allocations - #58802
bhollis-dbx wants to merge 1 commit into
Conversation
ANSI type coercion creates temporary tuples, closures, and partial-function chains while traversing expressions that remain unchanged. Large expression trees amplify this analysis cost. Reuse immutable coercion chains and avoid per-expression helper allocations without changing rule ordering or coercion behavior.
HyukjinKwon
left a comment
There was a problem hiding this comment.
0 blocking, 0 non-blocking, 0 nits.
Approve. Every hunk is behavior-preserving and I verified the equivalences directly; the change only removes per-traversal allocations (extractor tuples, an Option.map closure, and per-call PartialFunction chain construction). No user-facing behavior changes. The added ordering test is a sensible guard for the rule-composition the refactor touches.
Verification
Confirmed BinaryOperator.unapply returns Some((e.left, e.right)) (Expression.scala:869), so case b: BinaryOperator + b.left/b.right matches exactly what the tuple extractor did in both AnsiStringPromotion and ImplicitTypeCoercion. For ImplicitTypeCoercion, traced that case _ => b reproduces both old fall-throughs (None, and Some(ct) where !inputType.acceptsType(ct)) and the Some+accepts branch keeps the same Cast-if-needed logic. For DecimalPrecision: applyOrElse(expr, identity) equals the old lift(expr).getOrElse(expr); both cached chains reproduce the exact orElse order decimalAndDecimal -> integralAndDecimalLiteral -> nondecimalAndDecimal(true/false); and critically, nondecimalAndDecimal uses its literalPickMinimumPrecision parameter only inside case guards (lines 173, 178) and reads no conf at construction, so caching the chains as object vals is safe and the live flag is still read per call via if (conf.literalPickMinimumPrecision) (line 130) - the conf.getConf at line 214 is inside the unchanged widerDecimalType method, not a val initializer. Validated the new test's API: TypeCoercionRule.transform is the overridable PF and AnsiCombinedTypeCoercionRule(rules) (AnsiTypeCoercion.scala:261) chains them, so the assertion genuinely exercises in-order invocation. Changed lines are within 100 chars and ASCII (the flagged long lines are the pre-existing scalastyle:off conversion table and import block). All three scanners (contract_claim_verifier, local_efficiency over 28 lines, text_quality) returned no findings and dropped nothing.

What changes were proposed in this pull request?
This pull request reduces temporary allocations in ANSI type coercion by matching
BinaryOperatorexpressions without tuple-producing extractors, replacing anOption.mapchain with direct matching, and reusing the two immutable decimal coercion rule chains selected byspark.sql.legacy.literal.pickMinimumPrecision.It also adds a regression test confirming that combined ANSI coercion invokes every rule in order when earlier rules return the original expression.
Why are the changes needed?
ANSI type coercion allocates temporary tuples, closures, and partial-function chains while traversing expressions that remain unchanged. Large expression trees amplify this allocation cost and increase analysis memory and latency.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Added an ordering regression test and ran:
build/sbt 'catalyst/testOnly *AnsiTypeCoercionSuite *DecimalPrecisionSuite'All 62 tests passed.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)