chore: add thirdparty toolchain and thirdparty diff by lucasfang · Pull Request #6 · apache/paimon-cpp · GitHub
Skip to content

chore: add thirdparty toolchain and thirdparty diff - #6

Merged
leaves12138 merged 3 commits into
apache:mainfrom
lucasfang:dev
May 25, 2026
Merged

chore: add thirdparty toolchain and thirdparty diff#6
leaves12138 merged 3 commits into
apache:mainfrom
lucasfang:dev

Conversation

@lucasfang

Copy link
Copy Markdown
Collaborator

Purpose

add thirdparty toolchain and thirdparty diff patch

Tests

API and Format

Documentation

Generative AI tooling

Copilot AI review requested due to automatic review settings May 22, 2026 10:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR introduces a bundled third-party toolchain and applies local patches to upstream dependencies (Arrow/Parquet, ORC, cppjieba) to adjust build compatibility and extend a few APIs needed by the project.

Changes:

  • Add a full ThirdpartyToolchain.cmake to build/resolve bundled vs system dependencies via ExternalProject.
  • Patch ORC to support using external buffers in DataBuffer, add preBufferRange, and tweak timestamp handling.
  • Patch Arrow/Parquet and cppjieba to add/adjust small APIs and improve build/runtime behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File Description
cmake_modules/ThirdpartyToolchain.cmake New third-party build/resolution toolchain via ExternalProject + per-dependency build macros
cmake_modules/orc.diff Local patch to ORC (DataBuffer ownership/external buffer support, reader prebuffer range API, timestamp fix, PIC tweak)
cmake_modules/arrow.diff Local patch to Arrow/Parquet (row group size property, writer buffered-size API, reader batching tweak, CMake policy compat)
cmake_modules/jieba.diff Local patch to cppjieba to expose stop-words set via accessor
Comments suppressed due to low confidence (6)

cmake_modules/orc.diff:1

  • Adding this static_assert in the template destructor changes DataBuffer<T> into an ill-formed type for any non-trivially-copyable T as soon as the destructor is instantiated, which is a significant API/behavior change. If DataBuffer is intended to be restricted, enforce the constraint at the class level (and consider removing manual element-destruction loops for trivially-copyable types); otherwise, keep supporting non-trivial T by avoiding raw buffer reallocation/copy patterns that require trivial copyability.
diff --git a/c++/include/orc/MemoryPool.hh b/c++/include/orc/MemoryPool.hh

cmake_modules/orc.diff:1

  • setData computes element counts as bufSize / sizeof(T), but the parameter name bufSize is ambiguous (bytes vs element count). If callers pass an element count (common for T* APIs), currentSize_/currentCapacity_ will be wrong. Consider changing the API to either (a) accept an element count explicitly (e.g., num_elems), or (b) rename to buffer_bytes and document that it must be byte-size; then ensure all callers match that contract.
diff --git a/c++/include/orc/MemoryPool.hh b/c++/include/orc/MemoryPool.hh

cmake_modules/orc.diff:1

  • In the move constructor, the moved-from object is only reset when buffer.ownBuffer_ is true. When ownBuffer_ is false, the moved-from object keeps pointing at the external buffer with the original size/capacity, which is an unusual move semantic and can easily lead to accidental reuse of the moved-from buffer view. Consider resetting buffer.buf_ and sizes unconditionally in the move constructor while keeping ownBuffer_ = false semantics for ownership/freeing.
diff --git a/c++/include/orc/MemoryPool.hh b/c++/include/orc/MemoryPool.hh

cmake_modules/orc.diff:1

  • This adds a new pure-virtual method to the public orc::Reader interface, which is a source/ABI breaking change for any external implementations of Reader. If backward compatibility matters for your ORC integration, prefer providing a non-pure virtual with a default implementation (e.g., returning {}) or introducing this as an optional capability on ReaderImpl rather than a required interface method.
diff --git a/c++/include/orc/MemoryPool.hh b/c++/include/orc/MemoryPool.hh

cmake_modules/orc.diff:1

  • read_ranges is built from ranges without reserving capacity. Since the final size is known (ranges.size()), add read_ranges.reserve(ranges.size()) before the loop to avoid repeated reallocations when many ranges are returned.
diff --git a/c++/include/orc/MemoryPool.hh b/c++/include/orc/MemoryPool.hh

cmake_modules/arrow.diff:1

  • Adding a new pure-virtual method to parquet::arrow::FileWriter is an ABI- and source-breaking change for any downstream custom FileWriter implementations. If compatibility with external implementations is required, consider providing a default non-pure virtual implementation (e.g., returning 0) or adding this via a separate interface/utility rather than changing the base contract.
diff --git a/cpp/src/parquet/arrow/schema.cc b/cpp/src/parquet/arrow/schema.cc

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cmake_modules/ThirdpartyToolchain.cmake
Comment thread cmake_modules/ThirdpartyToolchain.cmake
Comment thread cmake_modules/ThirdpartyToolchain.cmake
Comment thread cmake_modules/orc.diff
set(ORC_SYSTEM_DEPENDENCIES)
set(ORC_INSTALL_INTERFACE_TARGETS)

+set(BUILD_POSITION_INDEPENDENT_LIB ON)
Comment thread cmake_modules/arrow.diff

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for migrating the third-party toolchain and patch files. I checked the imported patch files against the source snapshot, and the diff files themselves match the migration source.

I found two blockers before merge:

  1. cmake_modules/ThirdpartyToolchain.cmake is not a pure migration of the current source snapshot. It changes the Arrow/ORC dependency policy to allow SYSTEM/AUTO to resolve to system Arrow/ORC and removes the guard that forced bundled Arrow/ORC because paimon-cpp applies project-specific patches. Since this PR also imports arrow.diff and orc.diff, using unpatched system Arrow/ORC can silently bypass those required patches. Please either revert this toolchain file to the source snapshot behavior for this migration PR, or split and justify the system-Arrow/ORC policy change with tests in a separate PR.

  2. This PR introduces third-party patch files for Arrow, ORC, and cppjieba, but the Apache repository's LICENSE/NOTICE files are not updated with the corresponding third-party attribution/license entries. Please add the relevant Apache Arrow, Apache ORC, and cppjieba entries when introducing these files.

The rest of the staged migration shape looks fine.

@leaves12138

Copy link
Copy Markdown
Contributor

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the latest update. The Arrow/ORC dependency policy blocker is fixed, and the cppjieba attribution was added.

One licensing blocker remains: the current LICENSE/NOTICE update still does not cover the newly added Apache ORC-derived patch file, and the Apache Arrow section still does not list the newly added Arrow-derived files (cmake_modules/ThirdpartyToolchain.cmake and cmake_modules/arrow.diff). Please update LICENSE/NOTICE to accurately cover these newly distributed files before merge.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the latest update. The patched Arrow/ORC dependency policy is enforced, and LICENSE/NOTICE now cover the added Arrow/ORC/cppjieba files. I do not see further blockers.

@leaves12138
leaves12138 merged commit 56e8808 into apache:main May 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants