feat: add batch reader infrastructure by lszskye · Pull Request #46 · apache/paimon-cpp · GitHub
Skip to content

feat: add batch reader infrastructure - #46

Merged
leaves12138 merged 1 commit into
apache:mainfrom
lszskye:p5-4
Jun 4, 2026
Merged

feat: add batch reader infrastructure#46
leaves12138 merged 1 commit into
apache:mainfrom
lszskye:p5-4

Conversation

@lszskye

@lszskye lszskye commented Jun 4, 2026

Copy link
Copy Markdown
Collaborator

feat: add batch reader infrastructure

Changes

BatchReader

  • Abstract base class for reading batch data into Arrow arrays. Supports NextBatch() returning ArrowArray + ArrowSchema, and NextBatchWithBitmap() returning batches with a validity bitmap for efficient deletion vector and index filtering. Provides EOF detection and reader metrics.

FileBatchReader

  • Extends BatchReader for single-file reading scenarios. Adds SetReadSchema() for dynamic schema/predicate reconfiguration, GetFileSchema(), GetPreviousBatchFirstRowNumber(), GetNumberOfRows(), and SupportPreciseBitmapSelection() for bitmap-based row selection.

PrefetchFileBatchReader

  • Extends FileBatchReader with prefetch capabilities.

CompleteRowKindBatchReader

  • BatchReader wrapper that prepends a _VALUE_KIND column (int8 array of RowKind values) to each output batch.

ConcatBatchReader

  • Concatenates a list of BatchReader instances and reads them sequentially.

DelegatingPrefetchReader

  • FileBatchReader adapter that wraps a PrefetchFileBatchReaderImpl. Transparently delegates to the prefetch reader or its first underlying reader based on whether prefetching is needed.

PredicateBatchReader

  • BatchReader wrapper that applies a Predicate filter to each batch.

ReaderUtils

  • Static utility class.

Tests

  • CompleteRowKindBatchReaderTest
  • ConcatBatchReaderTest
  • PredicateBatchReaderTest
  • ReaderUtilsTest

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed PR #46 as an allowed intermediate step in the batch reader migration. The missing follow-up dependencies are expected to be added by later PRs, so they are not blockers for this staged split. I did not find other blockers. Approved.

@leaves12138
leaves12138 merged commit 9ce823a into apache:main Jun 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants