[Draft] Not That Trivial Resharding (basic infrastructure)#58132
Conversation
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Before this patch it leads to this exception:
Code: 10. DB::Exception: Received from localhost:9000. DB::Exception: Not found column key in block. There are only columns: _partition_id: While executing Remote. (NOT_FOUND_COLUMN_IN_BLOCK)
The problem is that intiial query required {"key"} columns, while the
queries on replicas has {"_partition_id"}, and so it thinks that it does
not need any column, while actually it is a virtual column, so it still
needs another column, that will be added in ReadFromMergeTree
And anyway from the remote node it will got only "_partition_id" column
(due to lack of projections), but initiator expects "key".
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
…edColumns() Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
This was an attempt to fix _partition_id filter for replicated-cluster, but this is not enough still, since analyzer adds idnex to the columns and so if the order will differs, then the header will not match, and this is what happens now. Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
This patch contains: ==================== - Distributed SELECT from the ReplicatedMergeTree with cluster=1 - Distributed INSERT from the ReplicatedMergeTree with cluster=1 Implementaiton details ====================== TBD Query Settings ============== - cluster_query_shards MergeTree Settings ================== - cluster=false - cluster_replication_factor=1 ZooKeeper ========= - /replicas/$/host -- information about node (disk space, labels) (*in future*) - /block_numbers/$ -- now contains replicas list Changelog: - Fix ALTER for replicated cluster mode (ALTER_METADATA should be executed on all replicas) Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
This is just a set of "C++ scripts" as the initial re-sharding implementation. Sadly but, as usual in big changes, it is pretty hard to avoid changing something else, and this patch is not an exception from this... (some logging, interfaces, tests). Changelog: - simple implementation with fetchPart() + DROP_RANGE - deferred DROP_RANGE - Distributor with FSM - restore from coordinator - clonePartition and friends - cleaner API for ReplicatedMergeTreeClusterPartition - REVERT_MIGRATE_PARTITION - allow multiple partitions at once - check version of the cluster/distributor node to avoid parallel re-sharding assignments - proper wait of distributor on shutdown to avoid UAF - tests with parallel replica restarts and fixes for it (uneven distribution after re-sharding) - Remove ReplicatedMergeTreeClusterDistributor::migratePartition() - Even distribution on INSERT - Introduce ReplicatedMergeTreeClusterPartitionSelector - ReplicatedMergeTreeClusterPartitionSelector::allocateParitition() Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Likely should fix:
Cannot process partition 3 (all_replicas: [1, 2, 4], active_replicas: [1, 2], MIGRATING 2 -> 4, version: 1), will revert: Code: 233. DB::Exception: Unexpected part name:
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
So that we do not need to look at is_dropped/is_shutdown storage flags inside balancer. Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Triggered by 03011_replicated_cluster_readonly_re_sharding Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
…YNC REPLICA) Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Fixes: 03017_replicated_cluster_mutations_stress Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
…d use active replicas
Otherwise it is possible to go into a sitatuion when mutation entry will
be created for the partition that will be removed during migration, and
never finished:
│ 2023-09-29 17:43:54.778175 │ 8939 │ default.data_r2 (Cluster) │ Loading partition from coordinator: 9 (all_replicas: [3, 4, 2], active_replicas: [3, 4], MIGRATING 3 -> 2, version: 7)
│ 2023-09-29 17:43:54.778266 │ 9111 │ default.data_r2 (ClusterBalancer) │ Source replica does not have part 9_0_0_0_3. Removing it from ZooKeeper.
│ 2023-09-29 17:43:54.778271 │ 9111 │ default.data_r2 (ClusterBalancer) │ Source replica does not have part 9_0_0_0_2. Removing it from ZooKeeper.
│ 2023-09-29 17:43:54.778275 │ 9111 │ default.data_r2 (ClusterBalancer) │ Source replica does not have part 9_0_0_0_1. Removing it from ZooKeeper.
│ 2023-09-29 17:43:54.778280 │ 9111 │ default.data_r2 (ClusterBalancer) │ Source replica does not have part 9_0_0_0_4. Removing it from ZooKeeper.
│ 2023-09-29 17:43:54.778284 │ 9111 │ default.data_r2 (ClusterBalancer) │ Source replica does not have part 9_0_0_0_5. Removing it from ZooKeeper.
│ 2023-09-29 17:43:54.778410 │ 8939 │ default.data_r2 (ReplicatedMergeTreeQueue) │ Will apply 20 mutations and mutate part 9_0_0_0_5 to version 25 (the last version is 25)
│ 2023-09-29 17:43:54.778684 │ 8939 │ default.data_r2 (d966de30-d7b7-404e-ba59-2b5e63072f44) │ Created log entry for mutation 9_0_0_0_25
│ 2023-09-29 17:43:54.778694 │ 9111 │ default.data_r2 (ClusterBalancer) │ Source replica does not have part 9_0_0_0_5. Removing it from working set.
│ 2023-09-29 17:43:54.778701 │ 9111 │ default.data_r2 (ClusterBalancer) │ Detaching 9_0_0_0_5
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
…d is broken) Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Onelinears:
$ for i in tests/queries/0_stateless/*replicated_cluster*.sql; do { echo -e "SET allow_experimental_analyzer=0; -- FIXME: analyzer is not supported yet\n"; cat $i; } | sponge $i; done
$ for i in tests/queries/0_stateless/*replicated_cluster*.sh; do { echo -e '#!/usr/bin/env bash\n\nCLICKHOUSE_CLIENT_OPT+="--allow_experimental_analyzer=0" # FIXME: analyzer is not supported yet\n'; grep -v '#!/usr/bin/env bash' $i; } | sponge $i; done
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
…ePartition() Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Conflicts:
src/Core/SettingsChangesHistory.h
src/Interpreters/ClusterProxy/executeQuery.cpp
src/Interpreters/ClusterProxy/executeQuery.h
src/Storages/StorageReplicatedMergeTree.cpp
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Conflicts:
src/Core/SettingsChangesHistory.h
|
Dear @azat, this PR hasn't been updated for a while. Will you continue working on it? If not, please close it. Otherwise, ignore this message. |
alexey-milovidov
added
the
close in a month if not active
|
@azat Sorry for disturb you. May I ask what the progress in this Resharding implements? I want to contribute this feature together to make this go faster. Maybe setup some subtask 🤔 |
|
I would love to continue, and indeed, the idea is to provide some basic infrastructure for this, and after this, I could really use some help (there are lots of things would need to be done), but, there are couple of problems - I started this PR almost year ago and lots of things since then had been changed (i.e. I need to implement support for analyzer case) but one of the things that is not depends from me is the review, since I want to make sure that this is the best way to going forward. I'm not saying that I'm going to abandon it, but I guess I need push more "energy" into this. |
Thanks for your kindly quick reply 👍 , As i am newbee in clickhouse community, will try my best to involve in. |
|
I need this feature, as described here: #45766 |
|
As If i have only one cluster each has 4 replica: what the difference with two cluster each has 2 replica |
|
@azat I look through this pr carefully, nice design and clean code 👍 As the name == "cluster" setting on mergeTree is readonly. I think one big scenarios of this feature is: |
|
Dear @azat, this PR hasn't been updated for a while. Will you continue working on it? If not, please close it. Otherwise, ignore this message. |
5 similar comments
|
Dear @azat, this PR hasn't been updated for a while. Will you continue working on it? If not, please close it. Otherwise, ignore this message. |
|
Dear @azat, this PR hasn't been updated for a while. Will you continue working on it? If not, please close it. Otherwise, ignore this message. |
|
Dear @azat, this PR hasn't been updated for a while. Will you continue working on it? If not, please close it. Otherwise, ignore this message. |
|
Dear @azat, this PR hasn't been updated for a while. Will you continue working on it? If not, please close it. Otherwise, ignore this message. |
6 participants
This is the draft of the re-sharding via ReplicatedMergeTree (as described in #45766).
Plan
I've splitted implementation into smaller peaces:
I have checklist in my notes, that I can share later, once we will agree on the overall idea
Basic infrastructure
The main goal of this PR is to prepare some basic infrastructure with everything working in general (i.e. basic SELECT/INSERT, replication protocol, ALTERs, re-sharding) so that it can be merged as a first step and later the work can be done on top. This will allow to:
Known tech debt:
I think that this bits can be done on top, not spent time on them.
Implementation notes
I've tried to keep as less changes as possible in the common code (hence sometimes you may find "odd" code, or I avoided introducing some new SYSTEM queries in this PR, that's how
SYSTEM SYNC REPLICA CLUSTERappeared), i.e. not modifyingStorageReplicatedMergeTreea lot, but keep code in a separate classes.Changes should be 100% backward compatible if the cluster mode is not used.
So this PR includes:
cluster_replication_factorreplicasCoordinator changes
It introduces concept of "cluster partition", it is stored in the
zookeeper_path/block_numbers/X.Also there is some meta information that is required, and it is stored under
zookeeper_path/cluster/.And now each replication entry has list of replicas that should execute this log entry (it is filtered only on the moving it form log to queue).
Settings
clustercluster_replicaton_factorcluster_query_shardsQueries
SYSTEM SYNC REPLICA CLUSTERSYSTEM DROP CLUSTER REPLICAIntrospection
system.cluster_partitionsNotes for reviewers
Since this is a draft, please do not look thoroughly, I'm sharing this to get some feedback on the overall implementation and also coordinator communication. I will share some of mine concerns (but maybe you will find something else):
DROP_RANGEcould be scheduled (and also now since DROP_RANGE is deferred you can get a race between removing the range of old parts during cloning partition and excecuting DROP_RANGE of the already migrated data from source replica, not a race, but assert that the state modified twice)zookeeper_path/block_numbers/Xto store information about cluster partition), and I'm not sure that this is a good ideaChangelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Not That Trivial Resharding
Documentation entry for user-facing changes
TBD
CI Settings (Only check the boxes if you know what you are doing):
Cc: @tavplubix