Adjust dynamic memory hard limit only on significant change#107645
Conversation
The background `MemoryWorker` recomputes the total memory hard limit on
every tick (every ~50ms) from `resident` and `MemAvailable`, both of
which jitter by a few MiB on each read. The previous guard re-applied and
logged the new limit whenever it differed from the current one by even a
single byte, so on an otherwise idle server the trace log was flooded:
<Trace> MemoryWorker: Adjusting total memory hard limit from 89.98 GiB to 89.98 GiB (resident: 383.83 MiB, available: 99.60 GiB, ceiling: 112.88 GiB, ratio: 0.9)
repeated dozens of times per second, and `setHardLimit` was called every
tick to no practical effect.
Adjust only when the change exceeds 1% of the current limit. A genuine
memory-pressure shift moves the limit by far more than this, so the
dynamic adjustment still reacts promptly when it actually needs to.
Discussion: #104964 (comment)
Related: #104964
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
clickhouse-gh
Bot
added
the
pr-not-for-changelog
![clickhouse-gh[bot]](https://avatars.githubusercontent.com/in/928874?s=60&v=4){kind=small}
| /// memory-pressure shift moves the limit by far more than this, so the dynamic | ||
| /// adjustment still reacts promptly when it actually needs to. | ||
| static constexpr double min_change_ratio = 0.01; /// 1% | ||
| const Int64 change_threshold = static_cast<Int64>( |
There was a problem hiding this comment.
The 1% deadband is measured against current_hard_limit, so it can be as large as the entire safety headroom that max_server_memory_usage_to_ram_ratio is supposed to preserve. For example, with ratio = 0.99 and a 99 GiB current limit, change_threshold is about 1 GiB; if another process consumes 1 GiB, the recomputed limit drops to about 98.01 GiB but this condition skips the update, so ClickHouse can still allocate up to 99 GiB and erase the configured 1% headroom until more pressure arrives. That weakens the dynamic-limit contract exactly when the configured headroom is small.
Please either keep applying the exact setHardLimit value and only rate-limit/log significant changes, or cap the hysteresis by an absolute jitter allowance / a fraction of the configured headroom rather than by 1% of the full hard limit.
Algunenano
left a comment
Algunenano
left a comment
There was a problem hiding this comment.
Looks good. Let's see, although I think that for local machines (not servers) disabling this is needed as we have many other background processes. But the general case is positive
…hange Refresh CI on top of current master. The only prior red was performance-comparison shard noise (6 faster / 6 slower, balanced, no significant change); master did not touch src/Common/MemoryWorker.cpp since the merge-base, so the merge is a clean fast re-base of unrelated history. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
LLVM Coverage ReportChanged lines: Changed C/C++ lines covered by tests: 9/9 (100.00%) | Lost baseline coverage: none · Uncovered code |
alexey-milovidov
deleted the
memory-worker-adjust-only-on-significant-change
branch
3 participants
A follow-up to #104964 addressing a review comment: the dynamic total-memory hard limit was re-applied and logged on almost every
MemoryWorkertick.MemoryWorkerrecomputes the hard limit every tick (~50ms on cgroups, ~100ms on jemalloc) fromresident(RSS) andMemAvailable, both of which jitter by a few MiB on each read. The previous guard re-applied and logged the new limit whenever it differed from the current one by even a single byte, so on an otherwise idle server the trace log was flooded:repeated dozens of times per second — a few-byte difference out of ~90 GiB that rounds to the same human-readable value — and
setHardLimitwas called every tick to no practical effect.This change applies (and logs) the new limit only when it differs from the current one by more than 1% of the current limit. A genuine memory-pressure shift moves the limit by far more than that, so the dynamic adjustment still reacts promptly when it actually needs to; sub-1% jitter is ignored.
Related: #104964
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
Follow-up to a not-yet-released feature; no separate changelog entry.
Documentation entry for user-facing changes
Version info
26.6.1.1001