Search before asking

• I searched in the issues and found nothing similar.

Motivation

Java Paimon and paimon-cpp write the same logical DECIMAL column to two different Parquet physical types.

Java Paimon (paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetSchemaConverter.java, case DECIMAL):

• precision <= 9 → INT32
• precision <= 18 → INT64
• otherwise → FIXED_LEN_BYTE_ARRAY

paimon-cpp: ParquetWriterBuilder::PrepareWriterProperties (src/paimon/format/parquet/parquet_writer_builder.cpp:54) never enables arrow's store_decimal_as_integer, which defaults to false (cpp/src/parquet/properties.h:239). So arrow writes every decimal as FIXED_LEN_BYTE_ARRAY, regardless of precision (cpp/src/parquet/arrow/schema.cc:362).

Two consequences:

1. Physical-layout divergence. For the same table schema, a file written by Flink/Spark Paimon and a file written by paimon-cpp differ in physical type for every decimal column with precision <= 18. Both are readable by both engines, but the divergence is invisible in the table schema and surfaces only in the footer — awkward for tooling, file-level diffing and any future code that keys off physical type.

2. Extra materialization when reading paimon-cpp-written files. An FLBA decimal goes through TransferDecimal in arrow's cpp/src/parquet/arrow/reader_internal.cc, which materializes an intermediate ::arrow::BinaryArray (offsets + data) from the record reader and then converts value by value via Decimal128::FromBigEndian. The INT32/INT64 path (DecimalIntegerTransfer, same file) reads reader->values() directly as a fixed-width column with no intermediate array.

The read side of paimon-cpp already handles all three physical types, so this is a write-side-only gap: ParquetStatsExtractor extracts DECIMAL min/max for INT32, INT64 and FIXED_LEN_BYTE_ARRAY / BYTE_ARRAY alike (src/paimon/format/parquet/parquet_stats_extractor.cpp:174-201), and value decoding is arrow's job either way.

Solution

1. Enable enable_store_decimal_as_integer() on the ::parquet::WriterProperties::Builder in ParquetWriterBuilder::PrepareWriterProperties. Arrow applies it only for 1 <= precision <= 18 and picks INT32 for precision <= 9, INT64 otherwise (cpp/src/parquet/arrow/schema.cc:362) — which is exactly Java Paimon's is32BitDecimal / is64BitDecimal split, so no additional mapping logic is needed on our side.

2. Add a parquet.write.store-decimal-as-integer option so the layout is selectable either way.

3. Decide the default separately, with measurements, because this is a genuine trade-off rather than a pure win. Arrow sizes an FLBA decimal at DecimalType::DecimalSize(precision) — the minimum number of bytes for that precision (cpp/src/arrow/type.cc:1443) — which is narrower than the corresponding integer for most precisions:

   precision	FLBA bytes	integer bytes	delta per value
   1–2	1	4 (INT32)	+3
   3–4	2	4	+2
   5–6	3	4	+1
   7–9	4	4	0
   10–11	5	8 (INT64)	+3
   12–14	6	8	+2
   15–16	7	8	+1
   17–18	8	8	0

   So for a common type like DECIMAL(10, 2) the raw values grow from 5 to 8 bytes, +60% before encoding. How much of that survives dictionary encoding and compression is exactly what needs to be measured ([Feature] Add format-level Parquet read/write micro-benchmarks #228) before flipping the default. Aligning with Java Paimon argues for defaulting it on; the size cost argues for care. Files already written stay readable under either choice — arrow's reader handles all three physical types, as does parquet-mr.

4. Tests:

   • round-trip DECIMAL(9, 2) / DECIMAL(18, 4) / DECIMAL(38, 10) and assert the physical type in the footer for each;
   • stats extraction over the new layout (the INT32 / INT64 branches at parquet_stats_extractor.cpp:180-191 currently only ever run against Java-written files, so this also gives them first-party coverage);
   • predicate pushdown on decimal columns still prunes row groups;
   • cross-engine check: read a paimon-cpp-written file from Java Paimon and vice versa;
   • report written file size for both settings across a range of precisions.

Anything else?

This came out of evaluating velox#17994 against paimon-cpp. velox#17992 adds a SIMD bulk path for PLAIN INT64 short decimals — the equivalent in arrow is DecimalIntegerTransfer (cpp/src/parquet/arrow/reader_internal.cc:645), which converts one value at a time (Decimal128 decimal(value); decimal.ToBytes(out_ptr);). Vectorizing it would require an addition to cmake_modules/arrow.diff.

That is deliberately not proposed here: the loop is already cheap, the gain is unquantified, and every entry in arrow.diff is maintenance debt at the next arrow upgrade. It should only be revisited if the format-level benchmarks (#228) show the decimal transfer is actually material.

Are you willing to submit a PR?

• I'm willing to submit a PR!