Convert to FixedHashTable for hash join with small int key range #99275
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds an optimization for hash joins on single Int32/Int64 keys with a small value range by converting the build-side hash map into a direct-indexed (range) table, controlled by new settings and validated with new tests/benchmarks.
Changes:
- Implement FixedRangeHashTable/FixedRangeHashMap and add new HashJoin variants (
range_key32/range_key64) plus post-build conversion logic. - Introduce settings
enable_fixed_range_hash_tableandfixed_range_hash_table_max_size(including query plan serialization plumbing and a new profile event). - Add stateless and performance tests for correctness and performance characterization.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/queries/0_stateless/04029_join_fixed_range_hash_table.sql | New stateless test for conversion trigger + join result correctness. |
| tests/queries/0_stateless/04029_join_fixed_range_hash_table.reference | Expected output for the new stateless test. |
| tests/performance/join_with_fixed_range_hash_table.xml | New perf test workload for comparing fixed-range vs normal hash join. |
| src/Processors/Transforms/JoiningTransform.cpp | Trigger fixed-range conversion at end of build phase for HashJoin. |
| src/Processors/QueryPlan/QueryPlanSerializationSettings.cpp | Add new join-related settings to plan serialization settings. |
| src/Interpreters/TableJoin.h | Store new fixed-range settings in TableJoin and expose accessors. |
| src/Interpreters/TableJoin.cpp | Read/cap new settings from Settings/JoinSettings into TableJoin. |
| src/Interpreters/JoinOperator.h | Add new settings fields to JoinSettings. |
| src/Interpreters/JoinOperator.cpp | Plumb new settings from Settings ↔ QueryPlanSerializationSettings. |
| src/Interpreters/HashJoin/KeyGetter.h | Add key getter mappings for new range key types. |
| src/Interpreters/HashJoin/HashJoin.h | Add range key variants and map storage for FixedRangeHashMap. |
| src/Interpreters/HashJoin/HashJoin.cpp | Implement conversion logic + new profile event + debug log. |
| src/Core/SettingsChangesHistory.cpp | Record compatibility history entries for the new settings. |
| src/Core/Settings.cpp | Define the new Settings entries and defaults. |
| src/Common/ProfileEvents.cpp | Register BuiltJoinRangeHashMapMicroseconds profile event. |
| src/Common/HashTable/FixedRangeHashTable.h | New direct-index table implementation used by the optimization. |
| src/Common/HashTable/FixedRangeHashMap.h | New map wrapper over FixedRangeHashTable for HashJoin use. |
| src/Common/HashTable/FixedHashMapCell.h | Extract FixedHashMapCell into a separate header for reuse. |
| src/Common/HashTable/FixedHashMap.h | Include the extracted FixedHashMapCell header. |
|
The |
|
Azure tests are fixed in #100980, you can update the branch to include the fixes. |
|
The flaky check failure is fixed in #102148, let's update the branch. |
|
Besides the |
LLVM Coverage ReportChanged lines: 94.51% (258/273) · Uncovered code |
533498e

relates to #41398 ( #97882 )
In case the join keys of a hash join are integers with a small range, the hash table can be replaced with an array indexed by
key - min_key.This should have:Capping the optimization at build size 2^18 makes the overhead of converting the hash table below 15 ms. In a follow up we can extract the min/max from statistics and directly build the direct lookup hash table to allow larger build sides.
Performance test:
LEFT JOIN with low selectivity:
INNER JOIN with low selectivity:
LEFT JOIN with high selectivity:
INNER JOIN with high selectivity:
LEFT JOIN with auto algorithm:
LEFT JOIN with grace_hash algorithm:
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
Speedup hash join on int32 and int64 keys with small range by using a direct-index hash table.
Documentation entry for user-facing changes
Note
Medium Risk
Changes hash join’s in-memory data structure for single Int32/Int64 key joins by optionally converting to a direct-indexed range table, which can affect join performance/memory and introduces new edge-case paths (range bounds, signed/unsigned handling). Scope is limited by size/range guards, a hard allocation cap, and added tests.
Overview
Adds a new hash-join optimization for single integer keys with a small value range: after building a
HashJoinright-side map (key32/key64), the join can convert it into a flatFixedRangeHashMapindexed bykey - min_key(skipping ASOF and multi-map cases, with signed min/max handling).Introduces
FixedRangeHashTable/FixedRangeHashMapimplementations, new join variantsrange_key32/range_key64, and instruments conversion time viaProfileEvents::BuiltJoinRangeHashMapMicrosecondsplus a debug log.Adds settings
enable_fixed_range_hash_tableandfixed_range_hash_table_max_size(propagated through query plan serialization and join settings, with a safety cap inTableJoin), and includes performance + stateless tests to validate triggering and result correctness.Written by Cursor Bugbot for commit fb869f2. This will update automatically on new commits. Configure here.