Apache Paimon C++ 0.3.0

Apache Paimon C++ 0.3.0 is the first Paimon C++ release published by the Apache Software Foundation.

Paimon C++ provides native, high-performance, and extensible access to the Apache Paimon lake format for C++ engines and services without JVM dependencies.

Supported Features

Core Table Access

Paimon C++ provides native write, commit, scan, and read support for append-only and primary-key tables:

• Batch and streaming scans without changelog
• Primary-key reads with Deletion Vectors and Merge-on-Read
• Arrow-based batch read and write interfaces
• Java-compatible commit messages, data splits, manifests, and storage formats

Append tables support fixed-bucket and unaware-bucket modes. Primary-key tables support fixed-bucket and postpone-bucket modes.

Data Types

Paimon C++ supports boolean, integer, floating-point, string, binary, decimal, date, timestamp, array, map, row, variant, and blob data types.

Notable data capabilities include:

• Variant: Java-compatible encoding, JSON conversion, path-based access, projection, and read/write support with Variant shredding
• Blob: Streaming reads and writes, Blob descriptors, multiple Blob fields, cross-table Blob views, and configurable fallback handling
• Schema evolution: Field addition, removal, rename, reorder, and compatible type changes

Extensible Plugin Architecture

Paimon C++ provides extensible abstractions and factory-based plugin mechanisms for integrating file formats and filesystem implementations. Memory pools, thread pools, and index plugins are also customizable.

Built-in file-format plugins include:

• Apache ORC
• Apache Parquet
• Apache Avro

Built-in filesystem plugins include:

• Local filesystem
• Alibaba Jindo filesystem

Compaction and Write Efficiency

Compaction support includes:

• Append-table automatic and full compaction
• Primary-key universal, level-0, lookup, and Deletion-Vector-aware compaction
• Local and remote lookup files with caching

For memory-constrained primary-key writes, Paimon C++ provides spillable external-sort buffers to limit in-memory data usage.

Indexes and AI-Oriented Features

Paimon C++ supports RowTracking and DataEvolution, including global row IDs, partial-column writes, and reconstruction of rows stored across multiple files.

File-level indexes include Bitmap, Bit-Sliced Index, Bloom Filter, and Range Bitmap.

Global-index capabilities include:

• B-tree indexes for point and range queries
• Range Bitmap indexes for scalar range predicates
• Lumina vector indexes with tag filtering
• Lucene-based full-text indexes
• Experimental Tantivy-based full-text indexes

The global-index framework also supports distributed result serialization, score propagation, filtering, caching, and index-file lifecycle management.

Read Performance

Read-path optimizations include:

• Column pruning, page-level predicate pushdown, Bitmap pushdown, and file-statistics filtering
• File prefetching and multi-threaded row-to-batch conversion
• Optimized COUNT(*) query

System Tables and Table Operations

Table-scoped system tables expose options, schemas, snapshots, manifests, files, audit logs, binlogs, and read-optimized $ro views. Global system tables under the sys database expose catalog-level information about tables, partitions, catalog options, and table options.

Supported table operations include snapshot creation and timestamp-based lookup, truncate, abort, rollback-to-latest, conflict detection, idempotent commits, and cleanup of snapshots, partitions, orphan files, and index files.

Source Release

The official Apache source release is available from:

• Source archive
• SHA-512 checksum
• OpenPGP signature
• Apache Paimon KEYS

Please use the official ASF source archive. The source archives automatically generated by GitHub are not official Apache release artifacts.

Documentation

• Paimon C++ documentation
• Paimon C++ repository

Acknowledgements

Thank you to everyone who contributed code, documentation, testing, reviews, release verification, and project infrastructure to Apache Paimon C++.

What's Changed

• chore: Add development tooling configs, contribution guidelines, and third-party license notices by @lxy-9602 in #1
• chore: add apidoc doc-config files by @lucasfang in #3
• feat(cmake): migrate build utility modules by @zjw1111 in #4
• feat: add common utilities including BinPacking, LinkedHashMap, LongCounter, Math, ScopeGuard, ThreadsafeQueue, and UUID by @dalingmeng in #5
• feat: introduce base utilities including Status, Defs, Executor, and MemoryPool by @lxy-9602 in #2
• feat: introduce basic data types including decimal, blob and timestamp by @lszskye in #10
• feat(cmake): add find package modules by @zjw1111 in #7
• fix(cmake): parse add_paimon_lib link arguments. by @slfan1989 in #9
• chore: add thirdparty toolchain and thirdparty diff by @lucasfang in #6
• feat: introduce data type with JSON serialization by @lszskye in #13
• feat(build): migrate build support scripts by @zjw1111 in #12
• feat: introduce common/memory module by @lxy-9602 in #11
• feat: add Range, RangeHelper, RowRangeIndex, and ByteRangeCombiner ut… by @dalingmeng in #19
• feat: add common data structures (BinarySection, BinaryString, InternalRow) by @lxy-9602 in #18
• Configure squash-only merge button by @leaves12138 in #29
• feat: add GenericLruCache, ConcurrentHashMap, MurmurHash, and Precond… by @dalingmeng in #21
• feat: introduce binary row format with reader/writer support by @lszskye in #22
• feat: add StringUtils, DateTimeUtils, PathUtil, OptionsUtils, RapidJsonUtil, and Jsonizable utilities by @dalingmeng in #23
• feat: migrate common/data/columnar module by @lxy-9602 in #24
• chore: add fix_includes.py for include cleanup workflow by @lucasfang in #25
• feat: add GenericRow, RecordBatch, MemorySize and TimeDuration by @lszskye in #28
• feat: add predicate module with Literal, LeafPredicate, and CompoundPredicate by @lxy-9602 in #27
• fix(types): make RowKind short string parsing case-insensitive by @slfan1989 in #15
• feat(fs): introduce file system abstractions by @zjw1111 in #26
• feat: add binary row serializers and blob implementation by @lszskye in #31
• feat: add projected_array, projected_row, field comparator, partition… by @lszskye in #32
• feat(third-party): introduce roaring bitmap and xxhash by @zjw1111 in #34
• feat: add RoaringBitmap32, RoaringBitmap64, and BitSet utilities by @lxy-9602 in #35
• feat: add predicate leaf functions by @lxy-9602 in #33
• feat: add search predicates and PredicateBuilder and tests by @lxy-9602 in #36
• feat: add IO cache (LRUCache, CacheManager, CacheInputStream) and MemorySlice utilities by @lxy-9602 in #40
• feat(fs): introduce file system tests by @zjw1111 in #42
• feat: add BloomFilter, CRC32C, MurmurHash, varint utilities, and DeltaVarintCompressor by @lxy-9602 in #37
• feat: Migrate format interfaces, column_stats, table special_fields, and lookup store by @lxy-9602 in #43
• feat: Migrate SST block infrastructure and file reader/writer by @lxy-9602 in #44
• feat: add data_converter_utils, field_type_utils, file_type and decim… by @lszskye in #38
• feat: add IO stream infrastructure by @lszskye in #39
• feat: add executor future helpers and metrics utilities by @lucasfang in #47
• feat(format): introduce parquet format components by @zjw1111 in #48
• feat: Migrate file index interfaces, format, reader, result, and factory by @lxy-9602 in #45
• feat: add batch reader infrastructure by @lszskye in #46
• feat: Migrate bitmap file index implementation by @lxy-9602 in #49
• feat(fs): introduce local file system by @zjw1111 in https://github.com/apache/paim...