Support per-element aggregation of `Tuple` columns in `SummingMergeTree`, `AggregatingMergeTree` and `CoalescingMergeTree` by JingYanchao · Pull Request #98039 · ClickHouse/ClickHouse · GitHub
Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
bf9176f
support tuple element aggregation and summing
JingYanchao Feb 12, 2026
71792f7
fix review issue
JingYanchao Mar 12, 2026
62f19c9
Merge branch 'master' into johnjing/subcolumn-aggregate-tuple
nihalzp Mar 19, 2026
4472dfe
Merge branch 'master' into johnjing/subcolumn-aggregate-tuple
nihalzp Apr 3, 2026
09343c1
Move setting to 26.4
nihalzp Apr 3, 2026
896a2f7
Merge branch 'master' into johnjing/subcolumn-aggregate-tuple
alexey-milovidov Apr 7, 2026
1d1341b
fix summing merge tree aggregate bug
JingYanchao Apr 7, 2026
63b5306
fix code spaces
JingYanchao Apr 7, 2026
7673c6e
fix review
JingYanchao Apr 9, 2026
1da139e
Merge branch 'master'
JingYanchao Apr 15, 2026
e4bfd12
validate flatten columns for summing sorted algorithm
JingYanchao Apr 15, 2026
2cc77c5
Merge branch 'master' into johnjing/subcolumn-aggregate-tuple
nihalzp Apr 23, 2026
9c29926
Move setting to 26.5
nihalzp Apr 23, 2026
83408eb
fix review bug & set allow_tuple_element_aggregation = true
JingYanchao Apr 28, 2026
32d0f7d
fix review test cases
JingYanchao May 9, 2026
a00aa41
Merge branch 'master'
JingYanchao May 9, 2026
33ae316
fix test cases bug
JingYanchao May 9, 2026
b10b600
fix test cases
JingYanchao May 9, 2026
1c52a4b
fix test cases
JingYanchao May 11, 2026
8fdbac7
Merge branch 'master' into johnjing/subcolumn-aggregate-tuple
JingYanchao May 22, 2026
d4f296a
fix test cases
JingYanchao May 22, 2026
eb00361
Merge branch 'master' into johnjing/subcolumn-aggregate-tuple
nihalzp Jun 8, 2026
3bd2c93
Move setting to 26.6
nihalzp Jun 8, 2026
292cd82
Remove from 26.5
nihalzp Jun 8, 2026
0c67143
Turn off the setting by default
nihalzp Jun 8, 2026
5af94d4
Make the setting false by default
nihalzp Jun 9, 2026
ced0505
Skip `Nullable(Tuple)`
nihalzp Jun 9, 2026
4ccca2a
Refactor utils in NestedUtils
nihalzp Jun 10, 2026
084e117
Improve `isColumnOrAncestorInNames`
nihalzp Jun 10, 2026
79e6b90
Fix *Map being incorrectly summed
nihalzp Jun 10, 2026
4591855
Remove special handling of *Map in `CoalescingMergeTree` that caused …
nihalzp Jun 10, 2026
790212e
Add test for the crash
nihalzp Jun 10, 2026
0aac4af
Use helper in `containsNullableFlattenableTuple`
nihalzp Jun 10, 2026
6fdf422
Add test to make sure setting is disabled by default
nihalzp Jun 10, 2026
6a4da46
Remove `countFlattenedColumnsRecursive`
nihalzp Jun 10, 2026
856e96f
Add test to reject Nullable(Tuple)
nihalzp Jun 10, 2026
052a4db
Add test for subcolumns names with dot
nihalzp Jun 10, 2026
9f3d2e9
Add test for name collision
nihalzp Jun 10, 2026
adebe6b
Increase test coverage
nihalzp Jun 10, 2026
3f31619
Improve `03802_tuple_element_aggregation_settings_validation.sh`
nihalzp Jun 10, 2026
0d54cca
Update example formatting
nihalzp Jun 10, 2026
4a11e18
Fix edge case
nihalzp Jun 10, 2026
93a4f34
Add test for the fix
nihalzp Jun 10, 2026
99bd8b7
Skip sub columns that are in the sorting or partition key
nihalzp Jun 10, 2026
830990b
Add test for skipping sorting and partition key
nihalzp Jun 10, 2026
9ddb333
Add utility to reject columns with same leaf name `flattenTupleLeafNa…
nihalzp Jun 10, 2026
30b04dc
Add test for the case
nihalzp Jun 10, 2026
3613962
Reject duplicate subcolumn name
nihalzp Jun 10, 2026
988f4e6
Check constraint during ALTER as well
nihalzp Jun 10, 2026
696d52a
Add test for ALTER validation
nihalzp Jun 10, 2026
dbc7757
Update test
nihalzp Jun 10, 2026
7570ee6
Rerun CI
nihalzp Jun 10, 2026
fe61ffb
Merge branch 'master' into johnjing/subcolumn-aggregate-tuple
nihalzp Jun 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -133,3 +133,43 @@ SELECT key, last_value(value_int), last_value(value_string), last_value(value_da
```

:::

## Tuple element aggregation {#tuple-element-aggregation}

When the `allow_tuple_element_aggregation` setting is enabled, `Tuple` columns are recursively flattened so that each leaf element participates in coalescing independently. This allows you to store multiple fields in a single `Tuple` column and have them coalesced element-wise during merges — each `Nullable` sub-column retains the latest non-NULL value independently.

The same rules apply to the flattened sub-columns as to regular columns:

- Sub-columns that belong to a `Tuple` in the sorting key or partition key are excluded from coalescing.
- If `columns` is specified, only sub-columns of the listed `Tuple` columns are coalesced.

:::note
This setting is immutable and must be specified at table creation time.
:::

```sql
CREATE TABLE coalescing_tuples
(
key UInt64,
data Tuple(
value_a Nullable(UInt64),
value_b Nullable(String),
nested Tuple(
value_c Nullable(UInt64)
)
)
) ENGINE = CoalescingMergeTree()
ORDER BY key
SETTINGS allow_tuple_element_aggregation = 1;

INSERT INTO coalescing_tuples VALUES (1, (100, NULL, (NULL)));
INSERT INTO coalescing_tuples VALUES (1, (NULL, 'hello', (42)));

SELECT key, data.value_a, data.value_b, data.nested.value_c FROM coalescing_tuples FINAL;
```

```text
┌─key─┬─data.value_a─┬─data.value_b─┬─data.nested.value_c─┐
│ 1 │ 100 │ hello │ 42 │
└─────┴──────────────┴──────────────┴─────────────────────┘
```
44 changes: 44 additions & 0 deletions docs/en/engines/table-engines/mergetree-family/summingmergetree.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,50 @@ When requesting data, use the [sumMap(key, value)](../../../sql-reference/aggreg

For nested data structure, you do not need to specify its columns in the tuple of columns for summation.

### Tuple element aggregation {#tuple-element-aggregation}

When the `allow_tuple_element_aggregation` setting is enabled, `Tuple` columns are recursively flattened so that each leaf element participates in summation independently. This allows you to store multiple metrics in a single `Tuple` column and have them summed element-wise during merges.

The same rules apply to the flattened sub-columns as to regular columns:

- Only numeric sub-columns are summed.
- Sub-columns that belong to a `Tuple` in the sorting key or partition key are excluded from summation.
- If `columns` is specified, only sub-columns of the listed `Tuple` columns are summed.
- If all numeric sub-columns of a row are zero after summation, the row is deleted.

:::note
This setting is immutable and must be specified at table creation time.
:::

```sql
CREATE TABLE summing_tuples
(
key UInt32,
metrics Tuple(
impressions UInt64,
clicks UInt64,
nested Tuple(
conversions UInt64
)
)
) ENGINE = SummingMergeTree()
ORDER BY key
SETTINGS allow_tuple_element_aggregation = 1;

INSERT INTO summing_tuples VALUES (1, (100, 10, (1)));
INSERT INTO summing_tuples VALUES (1, (200, 20, (3)));

OPTIMIZE TABLE summing_tuples FINAL;

SELECT key, metrics.impressions, metrics.clicks, metrics.nested.conversions FROM summing_tuples;
```

```text
┌─key─┬─metrics.impressions─┬─metrics.clicks─┬─metrics.nested.conversions─┐
│ 1 │ 300 │ 30 │ 4 │
└─────┴─────────────────────┴────────────────┴────────────────────────────┘
```

## Related content {#related-content}

- Blog: [Using Aggregate Combinators in ClickHouse](https://clickhouse.com/blog/aggregate-functions-combinators-in-clickhouse-for-arrays-maps-and-states)
4 changes: 3 additions & 1 deletion src/Columns/ColumnAggregateFunction.h
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,9 @@ class ColumnAggregateFunction final : public COWHelper<IColumnHelper<ColumnAggre

bool isDefaultAt(size_t) const override
Comment thread
JingYanchao marked this conversation as resolved.
{
throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Method isDefaultAt is not supported for ColumnAggregateFunction");
/// Aggregate function states have no meaningful default representation, so they are never considered default.
/// This is consistent with getNumberOfDefaultRows returning 0.
return false;
Comment thread
JingYanchao marked this conversation as resolved.
Comment thread
nihalzp marked this conversation as resolved.
}

std::string_view getDataAt(size_t n) const override;
Expand Down
1 change: 1 addition & 0 deletions src/Core/SettingsChangesHistory.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1233,6 +1233,7 @@ const VersionToSettingsChangesMap & getMergeTreeSettingsChangesHistory()
{
addSettingsChanges(merge_tree_settings_changes_history, "26.6",
{
{"allow_tuple_element_aggregation", false, false, "New setting"},
{"shared_merge_tree_try_fetch_part_in_memory_data_from_replicas_on_startup", false, false, "New setting which allows SMT download parts data from replicas instead of S3 on startup"},
});
addSettingsChanges(merge_tree_settings_changes_history, "26.5",
Expand Down
178 changes: 178 additions & 0 deletions src/DataTypes/NestedUtils.cpp
Loading