`MemoryWorker` now periodically reads `MemFree + Cached` from
`/proc/meminfo` and updates the global hard memory limit to
`(tracked + free + cached) * max_server_memory_usage_to_ram_ratio`.
This shrinks ClickHouse's effective limit when other processes on
the host consume memory, reducing the risk of OOM-killing.

The ratio is the existing `max_server_memory_usage_to_ram_ratio`
server setting (default 0.9 in Server and Local); setting it to 0
disables the dynamic adjustment.

Keeper does not expose the ratio, so dynamic adjustment is off there.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The dynamic hard-limit adjustment added in the previous commit read
`MemFree + Cached` from `/proc/meminfo`. That source describes the
**host**, not the cgroup, so when ClickHouse runs in a small cgroup
on a big host (8 GiB cgroup on a 192 GiB host) the formula
`(tracked + free + cached) * ratio` inflates the global hard limit
to ~145 GiB on every tick, overriding the cgroup-derived 7.2 GiB
that `Server.cpp` set at startup. `MemoryTracker` then never throws
`MEMORY_LIMIT_EXCEEDED` and the kernel OOM-killer fires at 8 GiB
RSS, killing the server.

The same `setHardLimit` call also overrode an explicit
`max_server_memory_usage` from the config, since the dynamic
adjustment ran every 50 ms regardless of the value `Server.cpp`
had just installed.

Two changes:

* When `MemoryWorker` has a cgroup reader, the dynamic adjustment
  reads the cgroup's `memory.max` paired with the cgroup-aware
  usage from `cgroups_reader->readMemoryUsage` -- the same sources
  `AsynchronousMetrics` reports as `CGroupMemoryTotal` and
  `CGroupMemoryUsed`. The available-memory term becomes
  `memory.max - cgroup_used`. The `/proc/meminfo` path is kept as
  a fallback for the no-cgroup case.

* `Server.cpp` and `LocalServer.cpp` now hand the configured
  `max_server_memory_usage` to `MemoryWorker` via the new
  `setExternalHardLimit` setter. The dynamic adjustment caps its
  computed value at this ceiling, so it can only shrink the budget
  below the user's setting, never raise it above. The setter is
  re-called from the config-reload callback whenever the setting
  changes. Before it is called for the first time, the dynamic
  adjustment is suppressed, so the worker cannot inflate the limit
  during the brief window between `MemoryWorker::start` and the
  first reload.

Reproduced and verified with an 8 GiB systemd-run scope on a 192
GiB host: before the fix, five memory-heavy queries all triggered
the cgroup OOM-killer; after the fix, all five are caught cleanly
by `MemoryTracker` and the server stays alive across the battery.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…er` dynamic limit

Two correctness issues flagged by the AI review on
#104964:

1. In cgroup v2, when the leaf cgroup has `memory.max = max` but an
   ancestor cgroup has a finite limit, the previous code parsed only the
   leaf's `memory.max`, fell back to `/proc/meminfo`, and computed
   `available` from host-scoped memory. That could inflate the dynamic
   hard limit above the effective cgroup budget. Fix: walk all ancestors
   up to the cgroup mount root, open every `memory.max`, and on each
   tick take the minimum finite value. Only fall back to `/proc/meminfo`
   when no ancestor has a finite limit (i.e. the cgroup truly imposes
   no memory cap).

2. Reload race: a worker tick that read a positive `ratio` could compute
   `new_hard_limit` and write it via `setHardLimit` after a concurrent
   reload set the ratio to `0` (and reapplied its own hard limit from
   `total_memory_tracker.setHardLimit(max_server_memory_usage)`),
   leaving a stale positive limit. Fix: add a `settings_generation`
   counter bumped by `setDynamicHardLimitSettings` after writing the
   new ratio/ceiling. The worker captures the generation when it starts
   a dynamic update tick and re-checks just before `setHardLimit`,
   skipping the write if a reload happened in flight.

PR: #104964

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Three feedback items on `MemoryWorker`'s dynamic limit:

  * Use `MemAvailable` from `/proc/meminfo` instead of `MemFree + Cached`.
    `MemAvailable` is the kernel's own estimate of memory available for
    new allocations; it already accounts for the *reclaimable* portion of
    the page cache and slab, so we don't have to add `Cached` ourselves
    and we don't claim memory the kernel considers pinned (dirty pages,
    low watermarks, mlocked).

  * Handle the cgroup-v2 `memory.max` value `"max"` explicitly. The
    previous code relied on `tryReadIntText` returning false for the
    non-numeric string; this is the same effect but doesn't rely on
    parse-failure semantics and is easier to read.

  * Drop the redundant `> 0` check on a `uint64_t` value.

The dynamic-limit formula is now
`(resident + MemAvailable) * max_server_memory_usage_to_ram_ratio`
in the no-cgroup-limit fallback path, instead of
`(resident + MemFree + Cached) * ratio`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Address the unresolved review thread on `src/Common/MemoryWorker.cpp:727`:
returning `0` from `readAvailableForDynamicLimit` was overloaded — it meant
both "we could not read the metric" (legitimate to skip this tick) and
"there is genuinely no memory available" (the highest-pressure case, where
the dynamic limit must still shrink). The previous `if (available)` guard
collapsed those into a single "skip" branch, so under maximum pressure the
worker would leave the previous (often larger) hard limit in place — exactly
when ClickHouse should be tightening its budget the most.

Switch `readAvailableForDynamicLimit` and `readSystemAvailableMemory` to
`std::optional<uint64_t>` so `std::nullopt` signals "no source could be
read" and a returned `0` signals "no memory available right now". The
caller now treats `0` as valid input: the existing clamp-to-`used +
safety_margin` path will shrink the hard limit at the high-pressure boundary
instead of skipping the adjustment.

Addresses the unresolved bot thread on `MemoryWorker.cpp:556`. When
running in cgroup mode (`cgroups_reader && !cgroup_memory_max_bufs.empty()`)
and every `memory.max` read throws, the old code fell through to
host-wide `/proc/meminfo`. On a containerized deployment with a finite
effective cgroup limit, that is fail-open: the host's `MemAvailable` can
be far above the cgroup budget, and using it as headroom can let
`total_memory_tracker` exceed the cgroup limit and trigger a cgroup
OOM kill.

Distinguish "cgroup has no finite limit" (safe to use host meminfo)
from "failed to read cgroup limits" (skip this tick): track a
`any_read_failure` flag in the read loop; when no limit was parsed
finite and at least one read threw, return `std::nullopt` so the
worker leaves the hard limit alone and retries on the next tick.

The "all reads succeeded but every value was `max`/sentinel" case
continues to fall through to host `MemAvailable` as before, since
that is the genuinely-unbounded cgroup case.

…'t raise shrunk limit on reload, add unit tests

Three fixes for the dynamic `total_memory_tracker` hard-limit adjustment in
`MemoryWorker`, addressing review feedback:

- Constructor-time cgroup open failures no longer drop a level permanently.
  `CgroupMemoryLevel` now retains the `memory.max`/`memory.current` paths, and
  `readAvailableForDynamicLimit` lazily (re)opens any level whose buffers are
  null and fails the whole tick closed on a persistent open/read failure. This
  prevents computing the headroom minimum from an incomplete ancestor set and
  overestimating the budget when the dropped ancestor was the tighter one.

- `setDynamicHardLimitSettings` no longer raises an already-shrunk dynamic limit
  back to the static ceiling on an unrelated config reload. When dynamic
  adjustment is enabled and the worker has previously shrunk the limit under
  memory pressure, the reload only lowers the limit (never raises it); the next
  worker tick recomputes from live headroom. `0` is treated as "unlimited" on
  both sides of the comparison.

- Factored the per-level decision into the pure helper
  `MemoryWorkerHelpers::decideCgroupLevelAvailability` and added focused unit
  tests covering finite `memory.max` with same-level usage, the `"max"` token,
  the cgroup v1 "no limit" sentinel, at/over-limit (zero headroom), and
  zero/unparseable values.

…ic-limit

…ic-limit

…e limit adjustment

Address review: a positive max_server_memory_usage_to_ram_ratio used to
opt the server into runtime hard-limit shrinking with no way to keep the
previous static cap, while setting the ratio to 0 removed the cap
entirely. The new memory_worker_dynamic_hard_limit server setting
(enabled by default) gates only the runtime adjustment: when disabled,
max_server_memory_usage_to_ram_ratio caps the hard memory limit
statically at startup and on configuration reload, exactly as in
previous versions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>