feat: introduce binary row format with reader/writer support - #22
Conversation
leaves12138
left a comment
There was a problem hiding this comment.
Thanks for the binary row implementation. This PR currently does not compile against its base branch because it includes and uses BinaryArray/BinaryMap, but src/paimon/common/data/binary_array.h and src/paimon/common/data/binary_map.h are not present in the PR or in main. Please either add those dependencies in this PR, rebase after the PR that introduces them is merged, or remove the nested array/map support from this change.
leaves12138
left a comment
There was a problem hiding this comment.
Thanks for the update. The previous missing BinaryArray/BinaryMap issue is fixed, but I found one remaining correctness blocker in the inline bytes writer path.
leaves12138
left a comment
There was a problem hiding this comment.
Thanks for the update. The previous inline bytes writer issue is fixed, but there is still one signed left-shift overflow issue in the newly added BinaryRow code.
leaves12138
left a comment
There was a problem hiding this comment.
Thanks for the update. I re-reviewed the current PR head end-to-end and found two remaining blockers that should be fixed before merging.
leaves12138
left a comment
There was a problem hiding this comment.
I re-reviewed the latest head (329611d). The previous inline encoding signed-shift issues have been addressed, and my earlier comment about Arrow DECIMAL vs DECIMAL128 was incorrect and has been corrected separately. I did not find any remaining blocking correctness issues in the current binary row/array reader and writer changes.\n\nI also ran a lightweight whitespace check with git diff --check successfully. I could not run a full local build in this environment because the checkout does not include usable CMake build files and Arrow headers are not installed here.

Purpose
Introduce the binary row data format, compatible with Java Paimon's
BinaryRowserialization format. This module provides efficient in-memory row representation with fixed-length and variable-length parts within a singleMemorySegment.Key components:
BinaryRow, supporting all Paimon data types.MemorySegment. ImplementsInternalRowfor reading all field types, supports null tracking via bit set, hashing, equality comparison, and copy operations.InternalArraybacked by a singleMemorySegment. Supports all Paimon data types. Provides bulk conversion methods (ToIntArray,ToLongArray, etc.) and factory methods (FromIntArray,FromLongArray).BinaryArray, inheriting fromAbstractBinaryWriter.InternalMapbacked by a singleMemorySegment.Tests
BinaryRowTestBinaryRowWriterTestBinaryArrayTestBinaryMapTest