feat: add GenericRow, RecordBatch, MemorySize and TimeDuration by lszskye · Pull Request #28 · apache/paimon-cpp · GitHub
Skip to content

feat: add GenericRow, RecordBatch, MemorySize and TimeDuration - #28

Merged
leaves12138 merged 2 commits into
apache:mainfrom
lszskye:p2-3/5
May 29, 2026
Merged

feat: add GenericRow, RecordBatch, MemorySize and TimeDuration#28
leaves12138 merged 2 commits into
apache:mainfrom
lszskye:p2-3/5

Conversation

@lszskye

@lszskye lszskye commented May 28, 2026

Copy link
Copy Markdown
Collaborator

Purpose

GenericRow: A generic implementation of InternalRow backed by an array of VariantType, supporting all Paimon data types.

RecordBatch: Encapsulates a batch of columnar data with partition info, bucket id, and per-row RowKind (INSERT / UPDATE_BEFORE / UPDATE_AFTER / DELETE).

MemorySize: Parses human-readable memory size strings (e.g., "1024mb", "2g") into byte values.
Supports units: bytes, kibibytes, mebibytes, gibibytes, tebibytes, with case-insensitive matching.

TimeDuration: Parses human-readable time duration strings (e.g., "500ms", "10s", "2h") into millisecond values.
Supports units: nanoseconds, microseconds, milliseconds, seconds, minutes, hours, days.

Tests

GenericRowTest
RecordBatchTest
MemorySizeTest
TimeDurationTest

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. I found several issues that should be fixed before this can be merged.

  1. RecordBatchBuilder::Finish() leaves the builder holding a moved-from ArrowArray pointer until the scope guard runs.
    The RecordBatch constructor calls ArrowArrayMove(impl_->data_, data_), which releases the source ArrowArray. After returning from std::make_unique<RecordBatch>(...), the scope guard calls impl_->Reset(), which calls ArrowArrayRelease(impl_->data_) again. This can double-release or at least invoke the release path on a moved-from C Data Interface struct. The builder should transfer ownership by clearing impl_->data_ after the move, or construct the RecordBatch in a way that consumes and nulls the pointer before Reset() runs.

  2. When row_kinds_ is not explicitly set, Finish() stores an empty vector in the RecordBatch, even though the public API says the default row kind is INSERT. Downstream code calling GetRowKind() will see no row kinds at all and cannot distinguish this from a zero-row batch. Please materialize data->length INSERT entries in Finish() when row_kinds_ is empty.

  3. The bucket default is inconsistent with the public contract. The header says that if the bucket is not set, the default value is -1, but the implementation uses std::numeric_limits<int32_t>::min() as the stored sentinel. As a result, GetBucket() returns an unexpected value for the default case, and a user explicitly calling SetBucket(-1) will make HasSpecifiedBucket() return true. Please use the documented -1 default/sentinel or update the API semantics consistently.

  4. GenericRow::operator== unconditionally dereferences kind_ and other.kind_. Since SetRowKind() accepts a raw pointer and there is no null check, a default-constructed row can be made to crash by calling SetRowKind(nullptr) and then comparing. Please either reject null row kinds or make equality robust to null pointers.

@leaves12138
leaves12138 dismissed their stale review May 28, 2026 07:55

Superseded by a corrected review: the ArrowArrayRelease and empty row_kinds points were rechecked and should not be blockers.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification. I rechecked the previously disputed RecordBatchBuilder/ArrowArray release behavior and the empty row_kinds_ convention, and I agree those two points should not be blockers. I have removed them from the review.

I still think the following issues should be fixed before merge:

  1. The bucket default is inconsistent with the public contract. The header says that if the bucket is not set, the default value is -1, but the implementation uses std::numeric_limits<int32_t>::min() as the stored sentinel. As a result, GetBucket() returns an unexpected value for the default case, and a user explicitly calling SetBucket(-1) will make HasSpecifiedBucket() return true. Please use the documented -1 default/sentinel or update the API semantics consistently.

  2. GenericRow::operator== unconditionally dereferences kind_ and other.kind_. Since SetRowKind() accepts a raw pointer and there is no null check, a row can be made to crash by calling SetRowKind(nullptr) and then comparing. Please either reject null row kinds or make equality robust to null pointers.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. I rechecked the two previously blocking points: the bucket sentinel/API contract is now documented consistently, and GenericRow equality is robust when row kind pointers are null. I do not see remaining blockers.

@leaves12138
leaves12138 merged commit f9008f2 into apache:main May 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants