fix: reject decimal promotion that changes the scale (#3613) · bookingcom/iceberg-python@57a4c8e · GitHub
Skip to content

Commit 57a4c8e

Browse files
authored
fix: reject decimal promotion that changes the scale (apache#3613)
* fix: reject decimal promotion that changes the scale The DecimalType handler in promote() guarded scale equality with `file_type.scale == file_type.scale`, which compares the file scale to itself and is always true. As a result any decimal-to-decimal promotion with a widening precision was accepted regardless of the scale. Per the Iceberg spec a decimal may only be promoted when the scale is unchanged and the precision widens (decimal(P, S) to decimal(P2, S) with P2 > P), matching TypeUtil.isPromotionAllowed in the Java implementation. The bug affected both paths that use promote(): on read a differing-scale promotion built a reader at the wrong scale and reinterpreted the stored unscaled integers (silent data corruption), and on write it let a DataFrame column with a different scale pass the compatibility check. Compare the file scale to the read scale instead. The identical tautology in the test oracle masked the defect, so fix it too and add a decimal with a different scale to the promotion matrix, plus explicit regression tests for the schema and read paths. Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com> * test: simplify decimal promotion coverage Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com> --------- Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
1 parent 6285004 commit 57a4c8e

3 files changed

Lines changed: 28 additions & 2 deletions

File tree

‎pyiceberg/schema.py‎

Lines changed: 1 addition & 1 deletion

‎tests/avro/test_resolver.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -256,6 +256,15 @@ def test_resolve_decimal_to_decimal_reduce_precision() -> None:
256256
assert "Cannot reduce precision from decimal(19, 25) to decimal(10, 25)" in str(exc_info.value)
257257

258258

259+
def test_resolve_decimal_to_decimal_change_scale() -> None:
260+
# Changing the scale is not a valid promotion, even when the precision widens.
261+
# Allowing it would reinterpret the file's unscaled integers at the wrong scale.
262+
with pytest.raises(ResolveError) as exc_info:
263+
_ = resolve_reader(DecimalType(9, 2), DecimalType(18, 4))
264+
265+
assert "Cannot reduce precision from decimal(9, 2) to decimal(18, 4)" in str(exc_info.value)
266+
267+
259268
def test_column_assignment() -> None:
260269
int_schema = {
261270
"type": "record",

‎tests/test_schema.py‎

Lines changed: 18 additions & 1 deletion

0 commit comments

Comments
 (0)