iframe-proxy

vitlibar · 2026-05-11T07:37:44Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Add functions arrayTopK and arrayBottomK:

arrayTopK(k, array) returns the K largest elements in descending order
arrayBottomK(k, array) returns the K smallest elements in ascending order

For example arrayBottomK(3, [1, 5, 2, 7, 3]) returns [1,2,3]

Likewise arraySort() an optional lambda is supported, so
arrayBottomK((x, y) -> y, 2, ['a', 'b', 'c'], [3, 1, 2]) returns ['b', 'c']

Version info

Merged into: 26.6.1.320

clickhouse-gh · 2026-05-11T07:38:23Z

Copilot

Pull request overview

This PR introduces new higher-order array functions arrayTopK and arrayBottomK to return the top/bottom K elements of an array (with optional lambda sort keys and NULL skipping). It also updates the Prometheus Query-to-SQL converter to leverage these functions for topk/bottomk/limitk, enabling support for non-constant k aligned to the time grid, and adds test coverage for the new behavior.

Changes:

Add arrayTopK / arrayBottomK implementations and function registration.
Refactor Prometheus applyLimitAggregationOperator to use the new functions and support scalar-grid k, enabling an integration test previously disabled.
Add stateless SQL tests + references, and update aspell dictionary for the new function names.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/queries/0_stateless/04105_array_top_k_bottom_k.sql	Adds stateless queries covering top/bottom K behavior, NULL handling, lambdas, and errors.
tests/queries/0_stateless/04105_array_top_k_bottom_k.reference	Expected outputs for the new stateless tests.
tests/integration/test_prometheus_protocols/test_evaluation.py	Re-enables a PromQL `topk` test with time-varying `k` now that it’s supported.
src/Storages/TimeSeries/PrometheusQueryToSQL/applyLimitAggregationOperator.cpp	Uses `arrayTopK/arrayBottomK` for index selection and implements scalar-grid `k` handling.
src/Functions/array/arrayTopK.h	Declares `arrayTopK/arrayBottomK` via `FunctionArrayMapped` and defines return type / argument checks.
src/Functions/array/arrayTopK.cpp	Implements the core top/bottom K selection logic and registers both functions + docs.
ci/jobs/scripts/check_style/aspell-ignore/en/aspell-dict.txt	Adds `arrayTopK` and `arrayBottomK` to spelling ignore list.

vitlibar · 2026-05-11T07:53:50Z

The closest existing alternative is arrayPartialSort(k, arr), but it returns unsorted elements k+1..N in unspecified order and also doesn't skip NULLs. So arrayBottomK(k, arr) is roughly the same as

arraySlice(k, arrayPartialSort(k, arrayFilter(x -> x IS NOT NULL, arr)), 1, k)

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

vitlibar · 2026-05-11T16:20:10Z

@groeneai Investigate CI failures

groeneai · 2026-05-11T17:19:34Z

@vitlibar — investigated CI failures on aaf640d (AST fuzzer ran at 10:35 UTC; perf at later times today). All chronic / infra, none PR-caused — detail below.

1. `AST fuzzer (amd_debug, targeted, old_compatibility)` — STID 3520-1e53 (Bad cast from type A to B) — CHRONIC family

The specific STID variant is new today (3520-1e53 = 1 hit / this PR), but it's a member of the chronic STID 3520-* family which is already known: sibling variants 3520-3e11 (13 hits / 6 PRs), 3520-4237 (8 hits / 8 PRs), 3520-5bd5 (6 hits / 1 PR), etc. The broader "Bad cast from type A to B" LOGICAL_ERROR has 86 hits / 47 PRs / 4 master hits in 30 days. AST fuzzer mutations regularly produce new STID hashes for variants of the same underlying engine bug; this is one such variant.

The crash path is in PlannerActionsVisitor::visitFunction → IExecutableFunction::executeWithoutLowCardinalityColumns (defaultImplementationForConstantArguments), not in your arrayTopK/arrayBottomK code.

2. `Performance Comparison (arm_release, master_head, 1/6 / 5/6 / 6/6)` — `Insert raw query metrics data` / `Insert historical data` — CIDB INFRASTRUCTURE

All 6 shards failed identically with Code: 202. DB::Exception: Too many simultaneous queries. Maximum: 1000. (TOO_MANY_SIMULTANEOUS_QUERIES) or HTTPSConnectionPool(host='play.clickhouse.com', port=443): Read timed out. Unrelated to PR diff — global play.clickhouse.com throughput limit hit (also visible on PR #104564 today).

3. `Performance Comparison Check Results/*::old/new` — NOISE

grace_hash_join, norm_distance, apply_patch_parts_join, function_calculation_after_sorting_and_limit, join_convert_outer_to_inner, set_index — all marked unstable or slower on queries outside arrayTopK/arrayBottomK code paths. Amplified by the CIDB write failures (no historical baseline). Should clear up on a rebase + retry.

Recommendation

Rebase against current master and re-run CI; the perf comparison + AST fuzzer should clear.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (1)

src/Functions/array/arrayTopK.cpp:276

The implementation and tests define signed negative K values as clamped to 0, but the public function documentation for arrayBottomK does not mention that behavior. Please document this edge case alongside the K argument description.

    FunctionDocumentation::Syntax syntax = "arrayBottomK([f,] K, arr [, arr1, ... ,arrN])";
    FunctionDocumentation::Arguments arguments = {
        {"f(arr[, arr1, ... ,arrN])", "Optional. A lambda function to compute the sort key for each element.", {"Lambda function"}},
        {"K", "The number of smallest elements to return.", {"(U)Int*"}},
        {"arr", "An array.", {"Array(T)"}},

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

vitlibar · 2026-06-01T08:13:50Z

@groeneai Investigate CI failures

groeneai · 2026-06-01T11:04:59Z

@vitlibar Triage on head cbb8e0e3:

Check	Status	Verdict
Stateless tests (amd_msan, WasmEdge, parallel, 2/2)	FAILURE	Chronic WasmEdge MSAN flake (not PR-caused)
All other 143 checks	SUCCESS	clean

Failing test: 03469_json_read_subcolumns_combined_2_compact_merge_tree — timeout. CIDB 30-day cross-PR window shows 10 distinct unrelated PRs + 2 master hits failing on the same amd_msan, WasmEdge job (12 of 13 hits), 1 on amd_tsan. Failure spread across 11 days (2026-05-08 to 2026-05-31), well before this PR's latest commit. --diagnose-random-settings confirmed reproducible without minimizing within budget — settings-induced timeout under MSAN+WasmEdge's combined overhead.

Root cause family: the amd_msan, WasmEdge build is a known broken sanitizer/runtime combination — non-deterministic timeouts on slow MSAN-instrumented JSON/MergeTree paths plus the WASM runtime. Fix is being prepared in #105019 (change default x86 target from x86-64-v2 to x86-64-v3) which addresses the underlying __msan_chain_origin + vzeroupper interaction.

Recommendation: rebase + retry on the amd_msan WasmEdge job. If it persists, treat as infra and proceed — this PR has the assignee approval and is functionally clean.

Co-authored-by: Nihal Z. Miaji <81457724+nihalzp@users.noreply.github.com>

clickhouse-gh · 2026-06-01T17:35:15Z

LLVM Coverage Report

Changed lines: Changed C/C++ lines covered by tests: 244/255 (95.69%) | Lost baseline coverage: none · Uncovered code

Full report · Diff report

`CoalescingMergeTree` delegates to `SummingSortedAlgorithm` with `aggregate_all_columns = true`. The dedicated branch for that case in `defineColumns` (`src/Processors/Merges/Algorithms/SummingSortedAlgorithm.cpp:267`) wrapped its work in `if (map_name == column.name || !endsWith(map_name, "Map"))` and then fell through to an unconditional `continue`. For any nested column whose parent table name ends with `Map` (e.g. `testMap.key`, `legacy_features_Map.id`), the inner branch was skipped while the `continue` still fired, so the column was added to neither `columns_to_aggregate` nor `column_numbers_not_to_aggregate`. `postprocessChunk` then constructed `res_columns` of size `num_result_columns` and left those slots `nullptr`. `Chunk::setColumns` -> `Chunk::checkNumRowsIsConsistent` dereferenced the null `ColumnPtr`, producing a SIGSEGV in release builds, a `px != 0` assertion in debug builds, and an UBSan `member call on null pointer` report in `arm_asan_ubsan` (STID 1003-326e on the AST fuzzer for PR ClickHouse#104281 and PR ClickHouse#104563). CoalescingMergeTree performs per-column `last_value` coalescing, so nested `Map` arrays must be coalesced element-wise like any other array column. The legacy `nestedName.suffix` routing into `discovered_maps` for `sumMap`-by-key merging is a SummingMergeTree concept that does not apply here, so the Map-suffix exclusion is dropped from the `aggregate_all_columns = true` branch. SummingMergeTree code paths are untouched. Closes ClickHouse#101937 CI report: https://s3.amazonaws.com/clickhouse-test-reports/PRs/104563/c2e0f0fe3b92d48b5a47fa90555f7d4aca54c553/ast_fuzzer_arm_asan_ubsan/ Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

clickhouse-gh Bot added the pr-feature Pull request with new product feature label May 11, 2026

vitlibar requested a review from Copilot May 11, 2026 07:44

Copilot started reviewing on behalf of vitlibar May 11, 2026 07:45 View session

Copilot AI reviewed May 11, 2026

View reviewed changes

Comment thread src/Functions/array/arrayTopK.h

Comment thread src/Functions/array/arrayTopK.cpp Outdated

Comment thread src/Functions/array/arrayTopK.cpp Outdated

vitlibar marked this pull request as draft May 11, 2026 08:26

vitlibar force-pushed the add-array-top-k-bottom-k branch from d61b373 to 3afdcfe Compare May 11, 2026 09:39

vitlibar requested a review from Copilot May 11, 2026 09:40

Copilot started reviewing on behalf of vitlibar May 11, 2026 09:40 View session

Copilot AI reviewed May 11, 2026

View reviewed changes

Comment thread src/Functions/array/arrayTopK.h

Comment thread src/Functions/array/arrayTopK.cpp Outdated

vitlibar force-pushed the add-array-top-k-bottom-k branch from 3afdcfe to aaf640d Compare May 11, 2026 10:03

vitlibar marked this pull request as ready for review May 11, 2026 10:35

nihalzp self-assigned this May 11, 2026

groeneai mentioned this pull request May 11, 2026

PromQL: Add functions label_replace and label_join #104564

Merged

vitlibar force-pushed the add-array-top-k-bottom-k branch from aaf640d to cfbaa0d Compare May 15, 2026 22:06

vitlibar requested a review from Copilot May 15, 2026 22:06

Copilot started reviewing on behalf of vitlibar May 15, 2026 22:07 View session

clickhouse-gh Bot reviewed May 15, 2026

View reviewed changes

Comment thread src/Functions/array/arrayTopK.cpp

Copilot AI reviewed May 15, 2026

View reviewed changes

Comment thread tests/queries/0_stateless/04105_array_top_k_bottom_k.sql

Comment thread src/Functions/array/arrayTopK.cpp

Comment thread tests/integration/test_prometheus_protocols/test_evaluation.py Outdated

vitlibar force-pushed the add-array-top-k-bottom-k branch from cfbaa0d to 7c9b2ab Compare May 16, 2026 17:29

vitlibar requested a review from Copilot May 16, 2026 17:30

Copilot started reviewing on behalf of vitlibar May 16, 2026 17:30 View session

Copilot AI reviewed May 16, 2026

View reviewed changes

Comment thread tests/queries/0_stateless/04105_array_top_k_bottom_k.sql

Comment thread src/Functions/array/arrayTopK.cpp

Comment thread src/Storages/TimeSeries/PrometheusQueryToSQL/applyLimitAggregationOperator.cpp

Comment thread src/Functions/array/arrayTopK.cpp Outdated

vitlibar force-pushed the add-array-top-k-bottom-k branch from 7c9b2ab to c4d82eb Compare May 16, 2026 17:55

clickhouse-gh Bot reviewed May 16, 2026

View reviewed changes

Comment thread src/Functions/array/arrayTopK.cpp Outdated

vitlibar force-pushed the add-array-top-k-bottom-k branch from c4d82eb to 06d75cf Compare May 16, 2026 18:13

nihalzp reviewed May 17, 2026

View reviewed changes

vitlibar requested a review from Copilot May 28, 2026 09:44

Copilot started reviewing on behalf of vitlibar May 28, 2026 09:44 View session

Copilot AI reviewed May 28, 2026

View reviewed changes

Comment thread src/Storages/TimeSeries/PrometheusQueryToSQL/applyLimitAggregationOperator.cpp Outdated

Comment thread src/Functions/array/arrayTopK.h Outdated

vitlibar force-pushed the add-array-top-k-bottom-k branch from c192177 to 5728e8f Compare May 28, 2026 10:10

vitlibar requested a review from Copilot May 28, 2026 10:12

Copilot started reviewing on behalf of vitlibar May 28, 2026 10:12 View session

Copilot AI reviewed May 28, 2026

View reviewed changes

Comment thread src/Functions/array/arrayTopK.h Outdated

vitlibar force-pushed the add-array-top-k-bottom-k branch from 5728e8f to 8ddb9dd Compare May 28, 2026 10:22

vitlibar requested a review from Copilot May 28, 2026 10:23

Copilot started reviewing on behalf of vitlibar May 28, 2026 10:23 View session

Copilot AI reviewed May 28, 2026

View reviewed changes

nihalzp approved these changes May 28, 2026

View reviewed changes

Comment thread src/Functions/array/arrayTopK.cpp Outdated

nihalzp reviewed May 29, 2026

View reviewed changes

Comment thread src/Functions/array/arrayTopK.cpp Outdated

groeneai mentioned this pull request May 29, 2026

Fix CoalescingMergeTree crash on INSERT with nested Map columns #106100

Closed

1 task

vitlibar and others added 5 commits June 1, 2026 16:18

Add functions arrayTopK and arrayBottomK

27d8d5f

Fixes after review

655c973

Replace toUInt64 with accurateCast

c5f3635

Update src/Functions/array/arrayTopK.cpp

cbb14fb

Co-authored-by: Nihal Z. Miaji <81457724+nihalzp@users.noreply.github.com>

Update src/Functions/array/arrayTopK.cpp

afa1c99

vitlibar force-pushed the add-array-top-k-bottom-k branch from cbb8e0e to afa1c99 Compare June 1, 2026 14:18

vitlibar added this pull request to the merge queue Jun 2, 2026

Merged via the queue into ClickHouse:master with commit cb1182c Jun 2, 2026
166 checks passed

vitlibar deleted the add-array-top-k-bottom-k branch June 2, 2026 09:40

robot-ch-test-poll3 added the pr-synced-to-cloud The PR is synced to the cloud repo label Jun 2, 2026

groeneai mentioned this pull request Jun 4, 2026

SipHash::update: short-circuit on empty input #106469

Merged

Sunbelt Computer Software

PL/B Language Development and Support

Uh oh!

Conversation

vitlibar commented May 11, 2026 • edited by robot-clickhouse Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Version info

Uh oh!

clickhouse-gh Bot commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

AI Review

Summary

Final Verdict

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vitlibar commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

vitlibar commented May 11, 2026

Uh oh!

groeneai commented May 11, 2026

1. AST fuzzer (amd_debug, targeted, old_compatibility) — STID 3520-1e53 (Bad cast from type A to B) — CHRONIC family

2. Performance Comparison (arm_release, master_head, 1/6 / 5/6 / 6/6) — Insert raw query metrics data / Insert historical data — CIDB INFRASTRUCTURE

3. Performance Comparison Check Results/*::old/new — NOISE

Recommendation

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

vitlibar commented May 11, 2026 •

edited by robot-clickhouse

Loading

clickhouse-gh Bot commented May 11, 2026 •

edited

Loading

vitlibar commented May 11, 2026 •

edited

Loading

1. `AST fuzzer (amd_debug, targeted, old_compatibility)` — STID 3520-1e53 (Bad cast from type A to B) — CHRONIC family

2. `Performance Comparison (arm_release, master_head, 1/6 / 5/6 / 6/6)` — `Insert raw query metrics data` / `Insert historical data` — CIDB INFRASTRUCTURE

3. `Performance Comparison Check Results/*::old/new` — NOISE