[SPARK-59511][SQL] Reduce ANSI type coercion allocations by bhollis-dbx · Pull Request #58802 · apache/spark · GitHub
Skip to content

[SPARK-59511][SQL] Reduce ANSI type coercion allocations - #58802

Open
bhollis-dbx wants to merge 1 commit into
apache:masterfrom
bhollis-dbx:reduce-ansi-type-coercion-allocations
Open

bhollis-dbx wants to merge 1 commit into
apache:masterfrom
bhollis-dbx:reduce-ansi-type-coercion-allocations

Conversation

@bhollis-dbx

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This pull request reduces temporary allocations in ANSI type coercion by matching BinaryOperator expressions without tuple-producing extractors, replacing an Option.map chain with direct matching, and reusing the two immutable decimal coercion rule chains selected by spark.sql.legacy.literal.pickMinimumPrecision.

It also adds a regression test confirming that combined ANSI coercion invokes every rule in order when earlier rules return the original expression.

Why are the changes needed?

ANSI type coercion allocates temporary tuples, closures, and partial-function chains while traversing expressions that remain unchanged. Large expression trees amplify this allocation cost and increase analysis memory and latency.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Added an ordering regression test and ran:

build/sbt 'catalyst/testOnly *AnsiTypeCoercionSuite *DecimalPrecisionSuite'

All 62 tests passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-5)

ANSI type coercion creates temporary tuples, closures, and partial-function chains while traversing expressions that remain unchanged. Large expression trees amplify this analysis cost. Reuse immutable coercion chains and avoid per-expression helper allocations without changing rule ordering or coercion behavior.

@HyukjinKwon HyukjinKwon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 blocking, 0 non-blocking, 0 nits.
Approve. Every hunk is behavior-preserving and I verified the equivalences directly; the change only removes per-traversal allocations (extractor tuples, an Option.map closure, and per-call PartialFunction chain construction). No user-facing behavior changes. The added ordering test is a sensible guard for the rule-composition the refactor touches.

Verification

Confirmed BinaryOperator.unapply returns Some((e.left, e.right)) (Expression.scala:869), so case b: BinaryOperator + b.left/b.right matches exactly what the tuple extractor did in both AnsiStringPromotion and ImplicitTypeCoercion. For ImplicitTypeCoercion, traced that case _ => b reproduces both old fall-throughs (None, and Some(ct) where !inputType.acceptsType(ct)) and the Some+accepts branch keeps the same Cast-if-needed logic. For DecimalPrecision: applyOrElse(expr, identity) equals the old lift(expr).getOrElse(expr); both cached chains reproduce the exact orElse order decimalAndDecimal -> integralAndDecimalLiteral -> nondecimalAndDecimal(true/false); and critically, nondecimalAndDecimal uses its literalPickMinimumPrecision parameter only inside case guards (lines 173, 178) and reads no conf at construction, so caching the chains as object vals is safe and the live flag is still read per call via if (conf.literalPickMinimumPrecision) (line 130) - the conf.getConf at line 214 is inside the unchanged widerDecimalType method, not a val initializer. Validated the new test's API: TypeCoercionRule.transform is the overridable PF and AnsiCombinedTypeCoercionRule(rules) (AnsiTypeCoercion.scala:261) chains them, so the assertion genuinely exercises in-order invocation. Changed lines are within 100 chars and ASCII (the flagged long lines are the pre-existing scalastyle:off conversion table and import block). All three scanners (contract_claim_verifier, local_efficiency over 28 lines, text_quality) returned no findings and dropped nothing.

@uros-b

uros-b commented Sep 15, 2026

Copy link
Copy Markdown
Member

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants