feat: add GenericRow, RecordBatch, MemorySize and TimeDuration - #28
Conversation
leaves12138
left a comment
There was a problem hiding this comment.
Thanks for the PR. I found several issues that should be fixed before this can be merged.
-
RecordBatchBuilder::Finish()leaves the builder holding a moved-fromArrowArraypointer until the scope guard runs.
TheRecordBatchconstructor callsArrowArrayMove(impl_->data_, data_), which releases the source ArrowArray. After returning fromstd::make_unique<RecordBatch>(...), the scope guard callsimpl_->Reset(), which callsArrowArrayRelease(impl_->data_)again. This can double-release or at least invoke the release path on a moved-from C Data Interface struct. The builder should transfer ownership by clearingimpl_->data_after the move, or construct theRecordBatchin a way that consumes and nulls the pointer beforeReset()runs. -
When
row_kinds_is not explicitly set,Finish()stores an empty vector in theRecordBatch, even though the public API says the default row kind isINSERT. Downstream code callingGetRowKind()will see no row kinds at all and cannot distinguish this from a zero-row batch. Please materializedata->lengthINSERTentries inFinish()whenrow_kinds_is empty. -
The bucket default is inconsistent with the public contract. The header says that if the bucket is not set, the default value is
-1, but the implementation usesstd::numeric_limits<int32_t>::min()as the stored sentinel. As a result,GetBucket()returns an unexpected value for the default case, and a user explicitly callingSetBucket(-1)will makeHasSpecifiedBucket()return true. Please use the documented-1default/sentinel or update the API semantics consistently. -
GenericRow::operator==unconditionally dereferenceskind_andother.kind_. SinceSetRowKind()accepts a raw pointer and there is no null check, a default-constructed row can be made to crash by callingSetRowKind(nullptr)and then comparing. Please either reject null row kinds or make equality robust to null pointers.
Superseded by a corrected review: the ArrowArrayRelease and empty row_kinds points were rechecked and should not be blockers.
leaves12138
left a comment
There was a problem hiding this comment.
Thanks for the clarification. I rechecked the previously disputed RecordBatchBuilder/ArrowArray release behavior and the empty row_kinds_ convention, and I agree those two points should not be blockers. I have removed them from the review.
I still think the following issues should be fixed before merge:
-
The bucket default is inconsistent with the public contract. The header says that if the bucket is not set, the default value is
-1, but the implementation usesstd::numeric_limits<int32_t>::min()as the stored sentinel. As a result,GetBucket()returns an unexpected value for the default case, and a user explicitly callingSetBucket(-1)will makeHasSpecifiedBucket()return true. Please use the documented-1default/sentinel or update the API semantics consistently. -
GenericRow::operator==unconditionally dereferenceskind_andother.kind_. SinceSetRowKind()accepts a raw pointer and there is no null check, a row can be made to crash by callingSetRowKind(nullptr)and then comparing. Please either reject null row kinds or make equality robust to null pointers.
leaves12138
left a comment
There was a problem hiding this comment.
Thanks for the update. I rechecked the two previously blocking points: the bucket sentinel/API contract is now documented consistently, and GenericRow equality is robust when row kind pointers are null. I do not see remaining blockers.

Purpose
GenericRow: A generic implementation of InternalRow backed by an array of VariantType, supporting all Paimon data types.
RecordBatch: Encapsulates a batch of columnar data with partition info, bucket id, and per-row RowKind (INSERT / UPDATE_BEFORE / UPDATE_AFTER / DELETE).
MemorySize: Parses human-readable memory size strings (e.g., "1024mb", "2g") into byte values.
Supports units: bytes, kibibytes, mebibytes, gibibytes, tebibytes, with case-insensitive matching.
TimeDuration: Parses human-readable time duration strings (e.g., "500ms", "10s", "2h") into millisecond values.
Supports units: nanoseconds, microseconds, milliseconds, seconds, minutes, hours, days.
Tests
GenericRowTest
RecordBatchTest
MemorySizeTest
TimeDurationTest