{{ message }}
[SPARK-58876][SQL][FOLLOWUP] Map sub-microsecond Oracle TIMESTAMP to nanos NTZ - #58809
Open
aleksandar-trajkovic-db wants to merge 1 commit into
Open
aleksandar-trajkovic-db wants to merge 1 commit into
aleksandar-trajkovic-db wants to merge 1 commit into
Conversation
…nanos NTZ Oracle DATE/TIMESTAMP mapping short-circuited the shared TIMESTAMP resolver, regressing TIMESTAMP(7-9) to microsecond TimestampNTZType under the nanos preview. Extract the micro-vs-nanos decision into JdbcUtils.resolveTimestampType and route Oracle's NTZ mapping through it (forcing NTZ). Co-authored-by: Isaac <no-reply@databricks.com>
Member
uros-b
reviewed
Sep 15, 2026
| JdbcUtils.getSchema(mock(classOf[Connection]), rs, OracleDialect(), | ||
| preferTimestampNanos = preferNanos).fields.head.dataType | ||
| } | ||
| } |
Member
There was a problem hiding this comment.
Nit: resolve only returns the data type. SPARK-58876 also stamps READ_TIMESTAMP_NTZ_WALL_CLOCK so a later flag flip cannot desync the microsecond NTZ read. That marker is redundant for TimestampNTZNanosType (the nanos getter already uses getObject(LocalDateTime)), but it is still part of the Oracle contract.
Consider returning the StructField (as the nearby preferTimestampNTZ test does) and asserting the marker is still present for both the nanos and microsecond outcomes, e.g. scale=9 with both flags on and scale=6 / flags off.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Follow-up to SPARK-58876. That change mapped Oracle
DATE/TIMESTAMPtoTimestampNTZTypedirectly inOracleDialect.getCatalystType, which short-circuitedthe shared
Types.TIMESTAMPresolution inJdbcUtilsthat widens a sub-microsecondcolumn to a nanosecond-capable type.
This PR extracts that micro-vs-nanos decision into
JdbcUtils.resolveTimestampType(isTimestampNTZ, scale, preferTimestampNanos)(thegeneric
Types.TIMESTAMPbranch now calls it) and routes Oracle's mapping through itwith
isTimestampNTZ = true. OracleDATE/TIMESTAMPtherefore stays NTZ, but aTIMESTAMP(7-9)now maps toTimestampNTZNanosType(scale)when the nanos preview isengaged.
Why are the changes needed?
Before SPARK-58876, Oracle
DATE/TIMESTAMPreturnedNonefrom the dialect and fellthrough to the shared resolver, so
TIMESTAMP(7-9)was mapped to a nanosecond type whenthe
preferTimestampNanosread option andspark.sql.timestampNanosTypes.enabledwereboth on. SPARK-58876 accidentally regressed this, truncating such columns to microsecond
TimestampNTZType. This restores the nanosecond precision while keeping the newzoneless-NTZ behavior.
Does this PR introduce any user-facing change?
Yes — relative to the unreleased SPARK-58876 change, not to any released version. With
preferTimestampNanos=trueandspark.sql.timestampNanosTypes.enabled=true, an OracleTIMESTAMP(7-9)column now reads asTimestampNTZNanos(scale)instead of microsecondTimestampNTZType.DATE,TIMESTAMP(0-6), and the default (preview off) areunchanged. The value is read time-zone-independently (wall-clock).
How was this patch tested?
Added a
getSchema-level unit test inJDBCSuiteasserting that an OracleTIMESTAMP(7-9)resolves toTimestampNTZNanosType(scale)only when both thepreferTimestampNanosoption and the preview flag are on, and stays microsecondTimestampNTZTypefor coarser scales or with either flag off. This replaces the priortest that pinned the regressed microsecond behavior.
Ran the SPARK-58876
JDBCSuitegroup locally; all 7 tests pass:Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Anthropic Claude Opus 4.8)
This is my original work and I license it to the project under the project's open source license.