chore: add thirdparty toolchain and thirdparty diff - #6
Conversation
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR introduces a bundled third-party toolchain and applies local patches to upstream dependencies (Arrow/Parquet, ORC, cppjieba) to adjust build compatibility and extend a few APIs needed by the project.
Changes:
- Add a full
ThirdpartyToolchain.cmaketo build/resolve bundled vs system dependencies viaExternalProject. - Patch ORC to support using external buffers in
DataBuffer, addpreBufferRange, and tweak timestamp handling. - Patch Arrow/Parquet and cppjieba to add/adjust small APIs and improve build/runtime behavior.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.
Comments suppressed due to low confidence (6)
cmake_modules/orc.diff:1
- Adding this
static_assertin the template destructor changesDataBuffer<T>into an ill-formed type for any non-trivially-copyableTas soon as the destructor is instantiated, which is a significant API/behavior change. IfDataBufferis intended to be restricted, enforce the constraint at the class level (and consider removing manual element-destruction loops for trivially-copyable types); otherwise, keep supporting non-trivialTby avoiding raw buffer reallocation/copy patterns that require trivial copyability.
diff --git a/c++/include/orc/MemoryPool.hh b/c++/include/orc/MemoryPool.hh
cmake_modules/orc.diff:1
setDatacomputes element counts asbufSize / sizeof(T), but the parameter namebufSizeis ambiguous (bytes vs element count). If callers pass an element count (common forT*APIs),currentSize_/currentCapacity_will be wrong. Consider changing the API to either (a) accept an element count explicitly (e.g.,num_elems), or (b) rename tobuffer_bytesand document that it must be byte-size; then ensure all callers match that contract.
diff --git a/c++/include/orc/MemoryPool.hh b/c++/include/orc/MemoryPool.hh
cmake_modules/orc.diff:1
- In the move constructor, the moved-from object is only reset when
buffer.ownBuffer_is true. WhenownBuffer_is false, the moved-from object keeps pointing at the external buffer with the original size/capacity, which is an unusual move semantic and can easily lead to accidental reuse of the moved-from buffer view. Consider resettingbuffer.buf_and sizes unconditionally in the move constructor while keepingownBuffer_ = falsesemantics for ownership/freeing.
diff --git a/c++/include/orc/MemoryPool.hh b/c++/include/orc/MemoryPool.hh
cmake_modules/orc.diff:1
- This adds a new pure-virtual method to the public
orc::Readerinterface, which is a source/ABI breaking change for any external implementations ofReader. If backward compatibility matters for your ORC integration, prefer providing a non-pure virtual with a default implementation (e.g., returning{}) or introducing this as an optional capability onReaderImplrather than a required interface method.
diff --git a/c++/include/orc/MemoryPool.hh b/c++/include/orc/MemoryPool.hh
cmake_modules/orc.diff:1
read_rangesis built fromrangeswithout reserving capacity. Since the final size is known (ranges.size()), addread_ranges.reserve(ranges.size())before the loop to avoid repeated reallocations when many ranges are returned.
diff --git a/c++/include/orc/MemoryPool.hh b/c++/include/orc/MemoryPool.hh
cmake_modules/arrow.diff:1
- Adding a new pure-virtual method to
parquet::arrow::FileWriteris an ABI- and source-breaking change for any downstream customFileWriterimplementations. If compatibility with external implementations is required, consider providing a default non-pure virtual implementation (e.g., returning 0) or adding this via a separate interface/utility rather than changing the base contract.
diff --git a/cpp/src/parquet/arrow/schema.cc b/cpp/src/parquet/arrow/schema.cc
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| set(ORC_SYSTEM_DEPENDENCIES) | ||
| set(ORC_INSTALL_INTERFACE_TARGETS) | ||
|
|
||
| +set(BUILD_POSITION_INDEPENDENT_LIB ON) |
leaves12138
left a comment
There was a problem hiding this comment.
Thanks for migrating the third-party toolchain and patch files. I checked the imported patch files against the source snapshot, and the diff files themselves match the migration source.
I found two blockers before merge:
-
cmake_modules/ThirdpartyToolchain.cmakeis not a pure migration of the current source snapshot. It changes the Arrow/ORC dependency policy to allowSYSTEM/AUTOto resolve to system Arrow/ORC and removes the guard that forced bundled Arrow/ORC because paimon-cpp applies project-specific patches. Since this PR also importsarrow.diffandorc.diff, using unpatched system Arrow/ORC can silently bypass those required patches. Please either revert this toolchain file to the source snapshot behavior for this migration PR, or split and justify the system-Arrow/ORC policy change with tests in a separate PR. -
This PR introduces third-party patch files for Arrow, ORC, and cppjieba, but the Apache repository's
LICENSE/NOTICEfiles are not updated with the corresponding third-party attribution/license entries. Please add the relevant Apache Arrow, Apache ORC, and cppjieba entries when introducing these files.
The rest of the staged migration shape looks fine.
leaves12138
left a comment
There was a problem hiding this comment.
Re-reviewed the latest update. The Arrow/ORC dependency policy blocker is fixed, and the cppjieba attribution was added.
One licensing blocker remains: the current LICENSE/NOTICE update still does not cover the newly added Apache ORC-derived patch file, and the Apache Arrow section still does not list the newly added Arrow-derived files (cmake_modules/ThirdpartyToolchain.cmake and cmake_modules/arrow.diff). Please update LICENSE/NOTICE to accurately cover these newly distributed files before merge.
leaves12138
left a comment
There was a problem hiding this comment.
Re-reviewed the latest update. The patched Arrow/ORC dependency policy is enforced, and LICENSE/NOTICE now cover the added Arrow/ORC/cppjieba files. I do not see further blockers.

Purpose
add thirdparty toolchain and thirdparty diff patch
Tests
API and Format
Documentation
Generative AI tooling