feat: add IO cache (LRUCache, CacheManager, CacheInputStream) and MemorySlice utilities - #40
Conversation
…orySlice utilities
leaves12138
left a comment
leaves12138
left a comment
There was a problem hiding this comment.
Thanks for the migration. I found one blocker in MemorySliceOutput growth: EnsureSize treats the already-required total size as an increment, which causes excessive growth and can hang forever for a zero-sized initial buffer.
There was a problem hiding this comment.
size is already the required total capacity passed by callers (size_ + bytes_to_write). Adding the current segment size again makes growth much larger than needed; for example, appending one byte to a full power-of-two buffer grows from N to 4N. More importantly, if MemorySliceOutput(0, pool) is used, capacity remains 0 and capacity <<= 1 loops forever. Please use min_capacity = size and handle capacity == 0 explicitly (or reject/normalize zero initial capacity), and add coverage for the zero-capacity case.
leaves12138
left a comment
leaves12138
left a comment
There was a problem hiding this comment.
I re-checked MemorySliceOutput against Java Paimon's MemorySliceOutput.ensureSize(). The C++ implementation intentionally follows the same growth policy, and current callers initialize it with a non-zero capacity. I did another pass over the cache and memory slice code and did not find remaining blockers.
2 participants
Purpose
No Linked issue.
Introduce IO caching infrastructure and memory slice utilities:
IO Cache (
src/paimon/common/io/):Cache— abstract cache interface (cache/cache.h)CacheKey— cache key with file path and offset identity (cache/cache_key.h/cpp)LRUCache— least-recently-used cache implementation (cache/lru_cache.h/cpp)CacheManager— manages cache lifecycle and eviction (cache/cache_manager.h/cpp)CacheInputStream— input stream backed by the page cache (cache_input_stream.h)Memory Slice (
src/paimon/common/memory/):MemorySlice— a view over a contiguous memory region (memory_slice.h/cpp)MemorySliceInput— sequential reader over a MemorySlice (memory_slice_input.h)MemorySliceOutput— growable writer producing a MemorySlice (memory_slice_output.h/cpp)Tests
cache_input_stream_test.cpp— cached read behavior and cache hits/misseslru_cache_test.cpp— LRU eviction order and capacity limitsmemory_slice_test.cpp— slice construction, subslicing, comparisonAPI and Format
Documentation
Generative AI tooling
Migrate-by: Aone Copilot (Claude)