Fix date_time_overflow_behavior being ignored for integer and float casts to DateTime64/Time64#101512
Fix date_time_overflow_behavior being ignored for integer and float casts to DateTime64/Time64#101512yariks5s wants to merge 26 commits into
date_time_overflow_behavior being ignored for integer and float casts to DateTime64/Time64#101512Conversation
…64/Time64 numeric casts
|
The Stress test (arm_msan) failure is fixed by #101239, which should be merged first. After it is merged, please update the branch to include the fix. |
|
The failures of "Flaky check" in "functions_bad_arguments" will be fixed by #101994. |
|
This was fixed by #105146. Let's update the branch. |
The previous fix only routed `DataTypeUInt64` through `ToTime64TransformUnsigned` and `ToDateTime64TransformUnsigned`, so `DataTypeUInt8`/`UInt16`/`UInt32` fell back to the generic decimal conversion path and bypassed the `date_time_overflow_behavior` check. This meant cases like `CAST(3600000::UInt32, 'Time64') SETTINGS date_time_overflow_behavior='throw'` silently saturated instead of raising `VALUE_IS_OUT_OF_RANGE_OF_DATA_TYPE`. Extend the dispatch to include all narrower unsigned integer sources and template the transforms on `FromDataType::FieldType`. Guard the overflow check with a `constexpr` predicate so widths whose entire range fits inside the target bounds (UInt8/UInt16 to Time64, UInt8/UInt16/UInt32 to DateTime64) skip it entirely. Addresses a review comment on #101512
…_overflow_behavior-ignored
Extend the explicit ConvertImpl branches for `DateTime64`/`Time64` to cover `Int128`/`Int256` and `UInt128`/`UInt256` source types, so that `date_time_overflow_behavior` is honored when casting wide integers through `CAST`. Previously these sources fell through to the generic decimal path and silently ignored the setting. The transforms now compare and clamp in the source type's domain so values above `INT64_MAX` are not truncated before the bounds check. The constexpr skip is reformulated in terms of `sizeof(FromType)` so it correctly identifies wide ints as requiring a bounds check. Includes `base/wide_integer_to_string.h` to make the `fmt::formatter` specialization for `wide::integer` visible to the `Exception` message.
Without this, casts from `Int128`/`Int256`/`UInt128`/`UInt256` to `DateTime64`/`Time64` were rejected at the wrapper-construction stage with `CANNOT_CONVERT_TYPE` (code 70), so the routing into the overflow-aware `ToDateTime64Transform*` / `ToTime64Transform*` paths added in `bce5fc0` was never reached. The `serverError VALUE_IS_OUT_OF_RANGE_OF_DATA_TYPE` assertions in `04050_time64_cast_uint64_no_wrap.sql` therefore observed the wrong error code in Fast test. Allow the wide unsigned/signed integer source families through the `DataTypeDateTime64`/`DataTypeTime64` wrapper so `date_time_overflow_behavior` applies uniformly, matching the dispatch in `ConvertImpl`.
…_overflow_behavior-ignored
…_overflow_behavior-ignored
…_overflow_behavior-ignored
The float conversion paths `ToDateTime64TransformFloat` and `ToTime64TransformFloat` compared the source value against whole-second bounds (`MAX_DATETIME64_TIMESTAMP` / `MAX_TIME_TIMESTAMP`). However, `DateTime64` and `Time64` can represent fractional seconds within the boundary second, so valid values such as `10413791999.1` (a valid `DateTime64(1)` = `2299-12-31 23:59:59.1`) and `3599999.1` (a valid `Time64(1)` = `999:59:59.1`) were incorrectly rejected with `VALUE_IS_OUT_OF_RANGE_OF_DATA_TYPE` under `date_time_overflow_behavior = 'throw'`, and clamped down to the whole second under the other modes. Extend the bounds by the largest fraction representable at the target scale (`MAX + (1 - 10^-scale)`) so boundary-second fractions are accepted, and out-of-range values saturate to the maximum representable value rather than to the truncated whole second. This addresses the review feedback at `src/Functions/FunctionsConversion.h:605` and `:742`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`ToDateTime64TransformFloat`/`Signed`/`Unsigned` computed overflow bounds from the calendar maximum second (`MAX_DATETIME64_TIMESTAMP`), but at scale 9 that second does not fit in the `Int64` native storage once multiplied by the scale (`10413791999 * 1e9 > Int64::max`). As a result a value that passed the bound check could still raise `DECIMAL_OVERFLOW` inside the conversion even with `date_time_overflow_behavior = 'saturate'`, where it should clamp instead. Derive the bounds from the maximum representable native value, capped at `Int64::max`, so scale 9 saturates to the largest representable value. The float path now also produces saturated values directly in the native domain to avoid `Float64` rounding losing the last representable fraction (e.g. `CAST(1e20 AS DateTime64(6))` now yields `...59.999999` instead of the imprecise `...59.999998`). Also initialize `clamped` in the signed transforms (fixes `cppcoreguidelines-init-variables` in the `arm_tidy` build) and apply the same native-domain saturation to `ToTime64TransformFloat` for consistency. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…_overflow_behavior-ignored
`03011_definitive_guide_to_cast`: `CAST(1e20 AS DateTime64(6))` now saturates to the exact maximum fraction (`...59.999999`) instead of the float-imprecise `...59.999998`, and `CAST(1e20 AS DateTime64(9))` now saturates to the maximum representable value instead of raising `DECIMAL_OVERFLOW` (the comment is updated accordingly). `04050_time64_cast_uint64_no_wrap`: add `throw`/`saturate` coverage for `DateTime64(9)` with `Float64`, `UInt64` and `Int64` sources, proving `throw` raises `VALUE_IS_OUT_OF_RANGE_OF_DATA_TYPE` (not `DECIMAL_OVERFLOW`) and `saturate` clamps to the maximum representable value. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Picked up the branch: merged latest
I updated the Built on ARM and verified locally: I did not approve, since this is not my own PR. |
|
📊 Cloud Performance Report ✅ AI verdict: This PR changes only the numeric-to-DateTime64/Time64 CAST overflow logic (clamping, saturate/throw behavior, and wide-integer support) in the conversion functions, plus the accompanying tests. None of the four flagged ClickBench queries (Q4, Q15, Q28, Q33) cast numbers into DateTime64 or Time64, so the modified code path is never executed by them and cannot plausibly produce these improvements. All four flagged improvements are downgraded to not-sure as run-to-run variance rather than real PR effects. No genuine cross-query performance pattern is present. clickbenchFlagged queries (4 of 43)
q-value = BH-FDR adjusted p; smaller is stronger evidence. MIRAI flags a query when q < fdr_q (default 0.10) — the value the verdict is based on. tpch_adapted_1_official🟢 No significant changes Debug info
|
…_overflow_behavior-ignored
…_overflow_behavior-ignored
Addresses review on the numeric->DateTime64/Time64 cast overflow path. The `ToDateTime64TransformFloat` / `ToTime64TransformFloat` transforms decided overflow by narrowing the scaled native bound back to the source float type (`max_native / scale_multiplier` in `FromType`). For `Float32` that division rounds to the wrong side of the true bound, so under `date_time_overflow_behavior = 'throw'` an out-of-range value could pass the check, and an in-range value could be clamped. Decide overflow instead by comparing the whole-second part as an integer and the sub-second part in the `Float64` domain, which is exact enough at every scale and deterministic across architectures. The in-range conversion is now also done in the `Float64` domain rather than through `convertToDecimal`, whose range check runs in the source-float precision: for a `Float32` value just below the `DateTime64(9)` maximum the scaled product is indistinguishable from `Int64::max` at `Float32` precision, so `convertToDecimal` would spuriously raise `DECIMAL_OVERFLOW`. The `Float64` computation reproduces `convertToDecimal` exactly for `Float64` sources and is strictly more faithful for `Float32` ones, so such a value is preserved (`CAST(9223371776::Float32, 'DateTime64(9, 'UTC')')` -> `2262-04-11 23:42:56`) instead of being rejected. Adds `Float32` regression coverage to `04050_time64_cast_uint64_no_wrap` and completes the `01734_datetime64_from_float` reference for the `DateTime64(9)` saturation cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Updated the branch: merged latest The The in-range conversion is also now done in the Built on this worktree and verified locally: The |
…_overflow_behavior-ignored
…version path `toDateTime64(toFloat32(1665519765), 3)` now scales in the `Float64` domain instead of going through `convertToDecimal` in source-float precision, so the result is the exact value of the `Float32` input (`1665519744` -> `2022-10-11 22:22:24.000`) rather than a value distorted by `Float32`-domain scaling (`2022-10-11 22:21:54.304`). Fixes the `Fast test` failure: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=101512&sha=d3e60548107aff956887b00c070e42a364550fb8&name_0=PR&name_1=Fast%20test Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ed scaled-native value Two fixes in `ToDateTime64TransformFloat` and `ToTime64TransformFloat`: 1. `NaN` left both range-comparison flags false and reached the final `static_cast<NativeType>`, which is undefined behavior (the pre-existing `convertToDecimal` path rejected non-finite values). `NaN` is now routed through the out-of-range path: `throw` raises `VALUE_IS_OUT_OF_RANGE_OF_DATA_TYPE`, `saturate` clamps to a representable bound. Infinities were already caught by the range checks. 2. The overflow decision compared the binary-rounded fractional part against the maximum representable fraction, falsely rejecting valid boundary inputs: `10413791999.999::Float64` for `DateTime64(3)` rounds to a fractional part slightly above `.999`, yet its scaled product truncates to the valid maximum native value. The decision is now made on the same truncated scaled-native value the conversion actually stores, with values at or beyond `2^63` rejected before the cast. New tests cover `NaN`/`inf`/`-inf` in both `throw` and `saturate` modes and the boundary-fraction values for `DateTime64` and `Time64`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Pushed updates:
Verified locally: The remaining unresolved thread ( |
…ble value With `date_time_overflow_behavior = 'saturate'`, an out-of-range integer cast to `DateTime64`/`Time64` clamped to the whole-second maximum and dropped the sub-second fraction, while the float path clamped to the maximum representable native value. So `CAST(toUInt64(10413791999), 'DateTime64(9, 'UTC'))` with `saturate` returned `2262-04-11 23:47:16.000000000` instead of the true type maximum `2262-04-11 23:47:16.854775807`, and the integer and float saturate results disagreed at every scale. The four integer transforms now clamp `saturate` to `maxRepresentableDateTime64Native` / `maxRepresentableTime64Native` (matching the float transforms), so saturation reaches the exact target-type boundary at every scale and the integer and float results agree. The legacy whole-second clamp is kept for `ignore` only, as before. Updated `04050_time64_cast_uint64_no_wrap` accordingly: the saturate comparisons now assert that an out-of-range integer cast equals the corresponding out-of-range float cast. Addresses the AI review finding on #101512 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…_overflow_behavior-ignored
|
Pushed Integer
The legacy whole-second clamp is kept for
The |
date_time_overflow_behavior being ignored for DateTime…date_time_overflow_behavior being ignored for integer and float casts to DateTime64/Time64
|
Addressed the remaining AI-review "Request changes" verdict (the
No code change and no re-merge: the head is |
…_overflow_behavior-ignored
LLVM Coverage Report
Changed lines: Changed C/C++ lines covered by tests: 98/107 (91.59%) | Lost baseline coverage (was covered on master, now uncovered in this PR): 2 line(s) · Uncovered code |
…_overflow_behavior-ignored Resolve the conflict in `src/Functions/FunctionsConversion.h` with the `toTime64` overflow handling that landed on master via #108028 ("Clamp `toTime64` to the Time64 range in saturate/ignore overflow modes"). Both changes fix the same bug (overflowing values not being clamped to the Time64 range in `saturate`/`ignore` modes). This PR's transforms are a strict superset of #108028: they compare in the source `FromType` domain so wide integers (`UInt128`/`UInt256`) are not truncated before the bounds check, clamp `saturate` to `maxRepresentableTime64Native` (the maximum including its sub-second fraction) while keeping the legacy whole-second clamp only for `ignore`, and handle `NaN`/`inf` and `Float32` precision in the float transform. The conflicting hunks were the `Int8`/`Int16` fall-through (where #108028's clamp is a no-op, since `Int16` max `32767` is far below `MAX_TIME_TIMESTAMP` `3599999`) and the float return path, so both resolve to this branch's version. #108028's unrelated additions on master (interval-conversion fix, Variant/Dynamic `accurateCastOrNull` column typing) merge in unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
date_time_overflow_behavior now applies to the Float32/Float64 branches for DateTime64/Time64, and the integer branches above do the same for Time64, but the generated setting documentation in src/Core/FormatFactorySettings.h still describes only Date/Date32/DateTime/DateTime64 conversions and integer inputs. Users reading the setting docs will miss both the float sources and the Time64 target this PR now relies on.
Please update that setting description in the same PR, e.g. to include integer/float sources and Time/Time64 targets.

Fixes
date_time_overflow_behaviorbeing silently ignored when casting integer andFloat32/Float64values toDateTime64andTime64. Previously, overflowing values could skip theVALUE_IS_OUT_OF_RANGE_OF_DATA_TYPEexception inthrowmode and skip clamping to the target boundary insaturatemode. The fix routes native and wide integer sources (Int8–Int256,UInt8–UInt256) as well asFloat32/Float64through overflow-aware, scale-aware transforms, and scalesFloat32inputs in theFloat64domain so the result is no longer distorted by source-float precision.Decimal*andBFloat16sources are out of scope here — their scale-aware overflow handling belongs inDataTypesDecimal.cpp, not inFunctionsConversion.h— and are left for a follow-up.Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
Fixes
date_time_overflow_behaviorfor integer andFloat32/Float64casts toDateTime64andTime64: overflowing values now throwVALUE_IS_OUT_OF_RANGE_OF_DATA_TYPEinthrowmode and clamp to the target boundary insaturatemode;Float32inputs are scaled inFloat64precision, correcting previously imprecise results.Documentation entry for user-facing changes