feat: introduce binary row format with reader/writer support by lszskye · Pull Request #22 · apache/paimon-cpp · GitHub
Skip to content

feat: introduce binary row format with reader/writer support - #22

Merged
leaves12138 merged 5 commits into
apache:mainfrom
lszskye:p2-4
May 29, 2026
Merged

feat: introduce binary row format with reader/writer support#22
leaves12138 merged 5 commits into
apache:mainfrom
lszskye:p2-4

Conversation

@lszskye

@lszskye lszskye commented May 26, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Introduce the binary row data format, compatible with Java Paimon's BinaryRow serialization format. This module provides efficient in-memory row representation with fixed-length and variable-length parts within a single MemorySegment.

Key components:

  • BinaryWriter: Abstract interface defining the write protocol.
  • AbstractBinaryWriter: Template-method base class implementing variable-length field writing (strings, bytes, decimals, timestamps, arrays, rows, maps) with auto-growing buffer management.
  • BinaryRowWriter: Concrete writer for BinaryRow, supporting all Paimon data types.
  • BinaryRow: Immutable binary row representation backed by MemorySegment. Implements InternalRow for reading all field types, supports null tracking via bit set, hashing, equality comparison, and copy operations.
  • BinaryDataReadUtils: Static utilities for reading typed fields (Timestamp, Decimal, BinaryString, Array, Row, Map) from memory segments with proper offset/size decoding.
  • BinaryArray: A binary implementation of InternalArray backed by a single MemorySegment. Supports all Paimon data types. Provides bulk conversion methods (ToIntArray, ToLongArray, etc.) and factory methods (FromIntArray, FromLongArray).
  • BinaryArrayWriter: Writer for BinaryArray, inheriting from AbstractBinaryWriter.
  • BinaryMap: A binary implementation of InternalMap backed by a single MemorySegment.

Tests

  • BinaryRowTest
  • BinaryRowWriterTest
  • BinaryArrayTest
  • BinaryMapTest

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the binary row implementation. This PR currently does not compile against its base branch because it includes and uses BinaryArray/BinaryMap, but src/paimon/common/data/binary_array.h and src/paimon/common/data/binary_map.h are not present in the PR or in main. Please either add those dependencies in this PR, rebase after the PR that introduces them is merged, or remove the nested array/map support from this change.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. The previous missing BinaryArray/BinaryMap issue is fixed, but I found one remaining correctness blocker in the inline bytes writer path.

Comment thread src/paimon/common/data/abstract_binary_writer.cpp Outdated

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. The previous inline bytes writer issue is fixed, but there is still one signed left-shift overflow issue in the newly added BinaryRow code.

Comment thread src/paimon/common/data/binary_row.cpp Outdated

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. I re-reviewed the current PR head end-to-end and found two remaining blockers that should be fixed before merging.

Comment thread src/paimon/common/data/binary_row_writer.cpp
Comment thread src/paimon/common/data/binary_row_writer.cpp Outdated
@leaves12138

Copy link
Copy Markdown
Contributor

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I re-reviewed the latest head (329611d). The previous inline encoding signed-shift issues have been addressed, and my earlier comment about Arrow DECIMAL vs DECIMAL128 was incorrect and has been corrected separately. I did not find any remaining blocking correctness issues in the current binary row/array reader and writer changes.\n\nI also ran a lightweight whitespace check with git diff --check successfully. I could not run a full local build in this environment because the checkout does not include usable CMake build files and Arrow headers are not installed here.

@leaves12138
leaves12138 merged commit 356e2f2 into apache:main May 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants