[SPARK-59507][SQL] Parse interval fractional seconds without building a padded string by david-mollitor-db · Pull Request #58796 · apache/spark · GitHub
Skip to content

[SPARK-59507][SQL] Parse interval fractional seconds without building a padded string - #58796

Open
david-mollitor-db wants to merge 1 commit into
apache:masterfrom
david-mollitor-db:SPARK-59507
Open

david-mollitor-db wants to merge 1 commit into
apache:masterfrom
david-mollitor-db:SPARK-59507

Conversation

@david-mollitor-db

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

IntervalUtils.parseNanos converts the fractional-second digits of a day-time interval to
nanoseconds. It right-padded the digits to 9 characters with a concatenation and a substring
before parsing:

val alignedStr = if (nanos.length < maxNanosLen) {
  (nanos + "000000000").substring(0, maxNanosLen)
} else nanos
val nanoSecond = toLongWithRange(nanosStr, alignedStr, 0L, 999999999L)

This PR parses the digits directly and scales by the corresponding power of ten, removing
the intermediate strings:

val raw = toLongWithRange(nanosStr, nanos, 0L, 999999999L)
val nanoSecond = raw * nanosMultiplier(maxNanosLen - nanos.length)

nanosMultiplier is a small 10^0 .. 10^8 lookup allocated once.

Why are the changes needed?

The old form allocated two throwaway strings per call (the concatenation and the substring)
purely to zero-pad the value ahead of an integer parse. The fractional part is guaranteed to
be 1-9 ASCII digits by the interval grammar, so it can be parsed and scaled arithmetically
with no string allocation. JFR profiling of interval parsing attributed these allocations to
StringConcatHelper.newString.

Does this PR introduce any user-facing change?

No. The result is identical: the parsed value is always in [0, 999999999] and scaling keeps
it within that range, so the range check and error behavior are unchanged.

How was this patch tested?

Existing IntervalUtilsSuite and IntervalExpressionsSuite pass; they cover day-time
interval casting with fractional seconds.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Isaac

This pull request and its description were written by Isaac.

@HyukjinKwon

Copy link
Copy Markdown
Member

… a padded string

`IntervalUtils.parseNanos` right-padded the fractional-second digits to 9 characters with a
string concatenation and a substring before parsing:

    (nanos + "000000000").substring(0, maxNanosLen)

That allocated two throwaway strings per call purely to zero-pad ahead of an integer parse.

Since the fractional part is guaranteed to be 1-9 ASCII digits by the interval grammar,
parse it directly and scale by the corresponding power of ten instead. The result is
identical -- the parsed value stays in [0, 999999999], so the range check and error
behavior are unchanged -- and no intermediate string is allocated.

Co-authored-by: Isaac <no-reply@databricks.com>
@david-mollitor-db

Copy link
Copy Markdown
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants