{{ message }}
25.8 Antalya ports: Read optimization using Iceberg metadata#1069
Merged
Enmk merged 10 commits intoOct 12, 2025
Conversation
Enmk
reviewed
Oct 8, 2025
| /// Delta lake related object metadata. | ||
| std::optional<DataLakeObjectMetadata> data_lake_metadata; | ||
| /// Information about columns | ||
| std::optional<DataFileMetaInfoPtr> file_meta_info; |
Member
There was a problem hiding this comment.
why optional shared_ptr? What would set null-value mean in this case and how is it different from unset value?
Enmk
reviewed
Oct 8, 2025
| std::map<size_t, ConstColumnWithValue> constant_columns_with_values; | ||
| std::unordered_set<String> constant_columns; | ||
|
|
||
| NamesAndTypesList requested_columns_copy = read_from_format_info.requested_columns; |
Member
There was a problem hiding this comment.
Can we avoid making a copy here, and use original columns for building requested_columns_list ? I'm Ok with moving that copying down, right before we actually modify it
Enmk
reviewed
Oct 8, 2025
| requested_columns_list[column.getNameInStorage()] = std::make_pair(column_index++, column); | ||
| } | ||
|
|
||
| if (context_->getSettingsRef()[Setting::allow_experimental_iceberg_read_optimization]) |
Member
There was a problem hiding this comment.
It would be nice to have a general description of this optimization, as a comment to the whole section.
Enmk
reviewed
Oct 8, 2025
| if (context_->getSettingsRef()[Setting::allow_experimental_iceberg_read_optimization]) | ||
| { | ||
| auto file_meta_data = object_info->getFileMetaInfo(); | ||
| if (file_meta_data.has_value()) |
Member
There was a problem hiding this comment.
again, this is an excellent example: file_meta_data.value() can be null here. I know that there are precautions now, but this is fragile, IMO simple std::shared_ptr would be more robust here.
Suggested change
| if (file_meta_data.has_value()) | |
| if (file_meta_data) |
and
file_meta_data.value()->columns_info
would be replaced by just
file_meta_data->columns_info
Antalya-specific exception: allow_experimental_iceberg_read_optimization
Enmk
force-pushed
the
frontport/antalya-25.8/optimize_count_in_datalake
branch
from
6c8fbaf to
6777073
Compare
25 tasks
zvonand
reviewed
Oct 14, 2025
Member
There was a problem hiding this comment.
Why do we need this? Why cannot we use Iceberg::ColumnInfo? This is ambiguity with no obvious reason. We have local ColumnInfo (with almost the same subset of fields), and forward-declared another ColumnInfo, and then use them all together. Why?
zvonand
added a commit
that referenced
this pull request
Oct 14, 2025
…wipes_my_snot 25.8 Antalya: Fixes for #1069
1 task
ianton-ru
pushed a commit
that referenced
this pull request
Feb 17, 2026
…e_count_in_datalake 25.8 Antalya ports: Read optimization using Iceberg metadata
ianton-ru
pushed a commit
that referenced
this pull request
Feb 17, 2026
…wipes_my_snot 25.8 Antalya: Fixes for #1069
25 tasks
25 tasks
28 tasks
27 tasks
25 tasks
Closed
25 tasks
26 tasks
27 tasks
zvonand
added a commit
that referenced
this pull request
Jun 30, 2026
When an Iceberg manifest carries no per-column statistics, the parsed `DataFileMetaInfo::columns_info` is empty. The read optimization in `StorageObjectStorageSource::createReader` misread this as "every requested column is absent from the file": it replaced each nullable column with a constant `NULL` and set `need_only_count`, so the reader returned correct row counts but all-NULL values — silent data loss. Gate the absent-column-to-NULL loop on a non-empty `columns_info` so that stats-less manifests fall through to the regular reader, which reads present columns normally and resolves schema-evolution-absent columns via `IcebergMetadata::getInitialSchemaByPath`. Affects `icebergLocal`, `icebergS3`, `icebergAzure`, `icebergHDFS` and their `*Cluster` variants. Antalya-only, introduced by #1069. Add stateless test `04302_iceberg_read_optimization_no_column_stats` with a checked-in stats-less Iceberg fixture and a `generate.py` that reproduces it. C++ change taken from #1895 Closes #1545 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Read optimization using Iceberg metadata
Port of #1019 with many changes
Documentation entry for user-facing changes
Diff with #1019
Exclude tests: