Workload scheduling: Memory reservations#82414
Conversation
|
Dear @serxa, this PR hasn't been updated for a while. Will you continue working on it? If not, please close it. Otherwise, ignore this message. |
|
Dear @serxa, this PR hasn't been updated for a while. Will you continue working on it? If not, please close it. Otherwise, ignore this message. |
|
@groeneai Take a look at this sanitizer report instead https://pastila.nl/?00062084/8f590dc0fb9d680da6000067dfaa5f03#g8N75wFB3bJ0yP06nI3wLg==GCM |
…irness under sanitizers
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
@serxa Not caused by this PR, and not yet fixed. It is a pre-existing data race in the MVCC transaction code that the server-side AST fuzzer surfaces; it is unrelated to the scheduler changes here. What the report shows: a TSan race on
Root cause: asymmetric locking. The copy path takes Why it is not this PR:
Fix direction (for the transaction owners, not this PR): make access to |
Base max_memory_ratio on total_memory_tracker.getHardLimit() so that max_server_memory_usage (and its derived RAM ratio) is respected, falling back to physical memory only when no server hard limit is configured. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
|
||
| // Update increase pointer in case the removed allocation was the current one | ||
| if (setIncrease() && parent) | ||
| propagate(Update().setIncrease(increase)); |
There was a problem hiding this comment.
updateMinMaxAllocated still holds AllocationQueue::mutex when it propagates the new increase upward. That can deadlock the scheduler thread if the new current increase also needs eviction: e.g. one running allocation of 10, pending P1=30, pending P2=20, then max_memory is lowered to 25. This method rejects P1, makes P2 current, and calls propagate under the lock; AllocationLimit::setIncrease sees 10 + 20 > 25, calls selectAllocationToKill, and recurses back into AllocationQueue::selectAllocationToKill, which tries to take the same mutex.
Please collect whether the increase pointer changed while holding the lock, release the mutex, and only then propagate the update, or otherwise ensure victim selection cannot re-enter the same queue while its mutex is held.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
|
||
| // Update increase pointer in case the removed allocation was the current one | ||
| if (setIncrease() && parent) | ||
| propagate(Update().setIncrease(increase)); |
There was a problem hiding this comment.
updateQueueLimit can re-enter this same queue while AllocationQueue::mutex is still held. A concrete path is: lowering max_waiting_queries rejects the front pending allocation, setIncrease exposes the next pending/increasing request, then this propagate call reaches AllocationLimit::setIncrease; if the new request is still over max_memory, victim selection calls back into AllocationQueue::selectAllocationToKill, which tries to lock mutex again and deadlocks the scheduler thread.
Please follow the same pattern needed for updateMinMaxAllocated: decide whether the increase pointer changed while holding the lock, release mutex, and only then propagate the update upward.
Probably indeed some server-side experimental settings, not sure does it make sense to add some basic just for workflows (non-experimental) |
LLVM Coverage ReportChanged lines: Changed C/C++ lines covered by tests: 3491/3740 (93.34%) | Lost baseline coverage (was covered on master, now uncovered in this PR): 9 line(s) · Uncovered code |
Cherry pick #82414 to 26.6: Workload scheduling: Memory reservations
Backport #82414 to 26.6: Workload scheduling: Memory reservations

Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Introduce a memory reservation feature for workloads. More details https://clickhouse.com/docs/operations/workload-scheduling
Documentation entry for user-facing changes
Details
To enable memory reservations for workloads create MEMORY RESERVATION resource and set at least one limit for the total memory reserved using workload settings:
ClickHouse tracks memory allocations of all queries and background activities. The number of allocated bytes is aggregated through the scheduling hierarchy up to the root. Every query has an associated allocation in the leaf workload it belongs to. If a query has the
reserve_memorysetting greater than zero, then the allocation is created in a pending state. Pending allocation reserves requested amount of memory in the workload hierarchy. If there is not enough memory available, the allocation remains pending until enough memory is freed or other allocations are evicted (killed). When allocation is admitted, it becomes running. Running allocation could increase or decrease its size dynamically according to memory consumption of the query. Allocation life-cycle can be depicted with the following state diagram:stateDiagram-v2 [*] --> Pending: init [reserve_memory > 0] [*] --> Running: init [reserve_memory == 0] Pending --> Running: admit state Running { %% Region 1: increase flow NotIncreasing --> Increasing: request Increasing --> NotIncreasing: approve -- %% Region 2: decrease flow NotDecreasing --> Decreasing: request Decreasing --> NotDecreasing: approve } Running --> Killed: evict Running --> Released: finishPending allocations of a leaf workload are admitted according to FIFO order. When multiple workloads have pending allocations, they are admitted according to precedence and weight settings. Higher precedence workloads are served first. Sibling workloads with the same precedence share memory according to weights in a max-min fair manner, which means that workload with lower normalized memory usage (current usage plus requested increase divided by weight) is served first. The reverse logic is applied during eviction. When memory needs to be freed, workloads with lower precedence and higher normalized memory usage are evicted first.
Note that time-shared resources use priority, while space-shared resources use precedence. They are independent settings and could be set to different values. Higher priority implies non-destructive preemption (delay or throttling), while higher precedence may imply destructive eviction (stops with an error). A workload could have high priority for CPU scheduling, but the same precedence for memory reservation to avoid evicting other workloads and losing work that was already done by them.
Every workload with a
max_memorylimit ensures that the total memory allocated in its subtree does not exceed the limit. If a pending or increasing allocation would exceed the limit, eviction procedure is initiated to free memory. Eviction procedure selects a victim to be killed. The least common ancestor workload of killer and victim prevents eviction in the following situations:If eviction is prevented or does not free enough memory, the new allocation is blocked until enough memory is freed. These rules allow queueing of excessive queries based on memory pressure and provide a convenient way to avoid MEMORY_LIMIT_EXCEEDED errors.
NOTE: Workload limits are independent from other ways to limit memory consumption like
max_memory_usagequery setting. They could be used together to achieve better control over memory consumption. It is possible to set independent memory limits based on users (not workloads). This is less flexible and does not provide features like memory reservation and queueing of pending queries. See Memory overcommitWorkload setting
max_waiting_querieslimits the number of pending allocations for the workload. When the limit is reached, the server returns an errorSERVER_OVERLOADED.Memory reservation scheduling is not supported for merges and mutations yet.
Only queries with the
reserve_memorysetting greater than zero are subject to blocking while waiting for memory reservation. However, queries with zeroreserve_memoryare also accounted for in their workload memory footprint, and they can be evicted if necessary to free memory for other pending or increasing allocations. Queries without proper workload markup are not subject to memory reservation scheduling and cannot be evicted by the scheduler.To provide non-elastic memory reservation for a query, set both
reserve_memoryandmax_memory_usagequery settings to the same value. In this case, the query will reserve fixed amount of memory and will not be able to increase its allocation dynamically.Let's consider an example of configuration:
In this example, the total memory reserved by all queries and background activities cannot exceed 10 GiB. The system workload has a guarantee of at least 1 GiB (10% of 10 GiB), while the user workload has a guarantee of at least 9 GiB (90% of 10 GiB). Inside the user workload, production and staging workloads share memory according to weights (3 to 1) with equal precedence of 1. Testing workload has precedence 2, which is lower than production and staging. Therefore, testing workload can only use memory that is not used by production and staging.
If memory pressure arises, testing workload allocations will be evicted first. Then, if more memory needs to be freed, staging workload allocations will be evicted before production workload allocations if they exceed their guarantees. Note that pending queries in production and staging can evict running allocations in testing workload to free memory, but they cannot evict each other because they have the same precedence. In case of memory pressure, they will wait in queues, which allows the system to avoid MEMORY_LIMIT_EXCEEDED errors due to too many concurrently executing queries.
Note that system workload has precedence 0 (default), which is higher than production, staging and testing workloads, but they are not sibling workload. The least common ancestor is workload all, both children of which has equal precedence. So pending system workload cannot evict any of them, and vice versa. This ensures that system activities cannot easily be evicted.
Version info
26.6.1.119026.6.2.5