iframe-proxy

murphy-4o · 2026-03-24T14:22:37Z

Expose jemalloc heap profiling and statistics through Keeper's HTTP control interface, matching the functionality already available in the ClickHouse Server web UI.

Closes #100544

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Added a jemalloc profiling web UI for ClickHouse Keeper, available at /jemalloc on the HTTP control port.

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

Version info

Merged into: 26.4.1.954

Expose jemalloc heap profiling and statistics through Keeper's HTTP control interface, matching the functionality already available in the ClickHouse Server web UI. New REST API endpoints on the Keeper HTTP control port: - GET /jemalloc — serves the interactive web UI (shared with server via `isKeeper` mode flag injected at serve time) - GET /jemalloc/stats — returns raw `malloc_stats_print` output - GET /jemalloc/status — returns profiling state as JSON - GET /jemalloc/profile?format={collapsed|raw} — flushes a heap profile, symbolizes it, and returns collapsed stacks for flame graph rendering The web UI reuses the server's `jemalloc.html` with a thin `isKeeper` abstraction that swaps SQL-based data fetching for REST API calls and hides server-specific features (Query Profiler, connection params, output format options). This avoids duplicating ~3000 lines of HTML/CSS/JS. New shared helpers in `Jemalloc::heapProfileToCollapsedStacks` (parses raw jemalloc heap dump, symbolizes addresses via `StackTrace::forEachFrame`, aggregates into FlameGraph-compatible collapsed stacks) and `Jemalloc::getStats` (captures `malloc_stats_print` output as a string). Closes ClickHouse#100544 Made-with: Cursor

clickhouse-gh · 2026-03-24T14:23:23Z

Add Keeper tab to the jemalloc web UI section showing the HTTP control port URL and the REST API endpoints for programmatic access. Made-with: Cursor

…oc UI Add config example showing that `keeper_server.http_control.port` must be explicitly enabled, and a security warning noting the lack of application-level auth on the Keeper HTTP control port with recommended network-level mitigations. Made-with: Cursor

… duplicate comment - `handleStatus` no longer silently swallows `mallctl` exceptions. Failures are logged via `tryLogCurrentException` and surfaced in the JSON response as an `errors` array so operators can distinguish "profiling disabled" from "status unknown". - Remove triple-duplicated "Substitute user name" comment in `jemalloc.html`. Made-with: Cursor

…d integration tests - Inject the Keeper config `<script>` after `<head>` instead of before `<!DOCTYPE html>` to avoid triggering quirks mode in browsers. - Redirect `/jemalloc/` (trailing slash) to `/jemalloc` with HTTP 301 so the UI is reachable from both paths. - Add integration tests for all Keeper jemalloc HTTP endpoints: web UI, trailing-slash redirect, /stats, /status, bad profile format (400), and unknown API path (404). Made-with: Cursor

- Avoid `bugprone-suspicious-stringview-data-usage` by building the full HTML string with the injected config script before writing. - Use const reference for `path` to avoid unnecessary copy (`performance-unnecessary-copy-initialization`). Made-with: Cursor

- Call `setResponseDefaultHeaders` on 400/404/301 responses for consistent CORS/cache headers across all paths. - Use `data()` instead of `c_str()` for consistency with the rest of the file and the codebase. Made-with: Cursor

- Register `KeeperJemallocAPIHandler` unconditionally so it works in sanitizer builds where `USE_JEMALLOC` is off. API routes return HTTP 501 when jemalloc is not linked; the `/jemalloc/` trailing-slash redirect works in all builds. - Short-circuit HEAD on `/jemalloc/profile` before `flushProfile` to avoid expensive side effects from lightweight probes. - Fix wrong error key `thread.prof.active` → `prof.thread_active_init` in `handleStatus` JSON errors array. - Move `#include <Server/KeeperJemallocHandler.h>` from the header to `KeeperHTTPHandlerFactory.cpp` to reduce transitive compile cost. - Rewrite integration tests: always validate correct behavior for both jemalloc and non-jemalloc builds (no skips in sanitizer CI), add HEAD-method coverage, add positive-path profile tests, parametrize to eliminate duplication. https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=100606&sha=6ef7cf2a5d0ff347554cf731107fc69d631dc38b&name_0=PR Made-with: Cursor

…h routing Replace the monolithic `KeeperJemallocAPIHandler` with dedicated handler classes (`KeeperJemallocProfileHandler`, `KeeperJemallocStatsHandler`, `KeeperJemallocStatusHandler`, `KeeperJemallocRedirectHandler`, `KeeperJemallocNotAvailableHandler`) each registered on a strict path via the factory. This eliminates manual URL dispatching inside `handleRequest` and follows the same pattern used by other Keeper HTTP handlers. Also fixes `#include` indentation inside `#if USE_JEMALLOC`. Made-with: Cursor

- Fix `attachStrictPath` not matching URIs with query parameters: replace with a custom filter that matches the path portion, ignoring `?...` suffixes. This caused `/jemalloc/profile?format=bad` to 404 in non-jemalloc builds instead of returning 501. - Fix HEAD test assertions: HEAD responses carry no body, so checking `response.text` for "not available" fails on 501. Only assert the status code for HEAD. - Add HEAD short-circuit to `KeeperJemallocStatsHandler` to avoid the expensive `je_malloc_stats_print` call when no body is needed. - Handle `status.errors` in the web UI's `checkProfilingEnabled`: when the `/jemalloc/status` response includes errors, treat profiling state as unknown rather than silently defaulting to disabled. Made-with: Cursor

When `std::filesystem::remove` fails in the `SCOPE_EXIT` of `KeeperJemallocProfileHandler`, log a `LOG_WARNING` with the file path and error message instead of silently dropping the `std::error_code`. https: //github.com/ClickHouse/pull/100606#discussion_r2999333464 Made-with: Cursor

`KeeperJemallocStatusHandler` previously substituted default values (false/0) when `mallctl` reads failed, making it impossible for consumers to distinguish "profiling is disabled" from "status unknown". Replace bare values with `std::optional<T>` and serialize failed reads as JSON `null` instead of silent defaults. When `opt.prof` itself fails, all dependent fields (`prof_active`, `thread_active_init`, `lg_sample`) are also `null`. When `opt.prof` is definitively `false`, dependent fields carry authoritative false/0 values. The `errors` array is still present for failed keys, providing machine-readable diagnostics alongside the per-field tri-state. https: //github.com/ClickHouse/pull/100606#discussion_r2999333464 Made-with: Cursor

alexey-milovidov · 2026-04-07T00:27:02Z

The Stress test (arm_msan) failure is fixed by #101239, which should be merged first. After it is merged, please update the branch to include the fix.

antonio2368

Check src/CMakeLists.txt and how we use OBJECT_DEPENDS with html files

antonio2368 · 2026-04-08T14:42:17Z

 }


+std::string heapProfileToCollapsedStacks(const std::string & input_filename)


look symbolizeJemallocHeapProfile. Maybe we can generalize it more to use it here.

antonio2368 · 2026-04-08T14:43:15Z

            const statusColor = isEnabled ? 'var(--success-color)' : 'var(--error-color)';
            const statusText = isEnabled ? 'Enabled' : 'Disabled';
            const statusIcon = isEnabled ? '✓' : '✗';
+            const instanceName = isKeeper ? 'Keeper' : 'server';


where is this used?

alexey-milovidov · 2026-04-09T03:27:31Z

The MSan stress test failure (MemorySanitizer: use-of-uninitialized-value, STID 4179-5154 or 4148-3044) is a known pre-existing issue unrelated to this PR. Fix: #102158

alexey-milovidov · 2026-04-12T12:08:41Z

Azure was broken, I'm trying to untie our tests from it: #102354 (comment)

…build deps, remove dead code - Add `OBJECT_DEPENDS` for `KeeperJemallocHandler.cpp` in `src/CMakeLists.txt` so that changes to the `#embed`-ed `jemalloc.html` trigger recompilation, matching the existing pattern for other handlers. - Replace the 130-line `heapProfileToCollapsedStacks` in `Common/Jemalloc.cpp` with a call to the existing `JemallocProfileSource` collapsed-stack pipeline via the new `symbolizeJemallocHeapProfileToString` helper. This eliminates duplicated heap-dump parsing and address symbolization logic. - Remove the unused `instanceName` JavaScript variable from `jemalloc.html` (assigned but never referenced in any template). https: //github.com/ClickHouse/pull/100606 Made-with: Cursor

clickhouse-gh · 2026-04-15T10:01:06Z

LLVM Coverage Report

Changed lines: 4.76% (9/189) · Uncovered code

Full report · Diff report

clickhouse-gh Bot added the pr-feature Pull request with new product feature label Mar 24, 2026

Document Keeper jemalloc web UI in allocation-profiling.md

7512db8

Add Keeper tab to the jemalloc web UI section showing the HTTP control port URL and the REST API endpoints for programmatic access. Made-with: Cursor

clickhouse-gh Bot reviewed Mar 24, 2026

View reviewed changes

Comment thread src/Server/KeeperJemallocHandler.cpp Outdated

clickhouse-gh Bot reviewed Mar 24, 2026

View reviewed changes

Comment thread programs/server/jemalloc.html Outdated

murphy-4o added 2 commits March 24, 2026 14:35

clickhouse-gh Bot reviewed Mar 24, 2026

View reviewed changes

Comment thread src/Server/KeeperJemallocHandler.cpp Outdated

clickhouse-gh Bot reviewed Mar 24, 2026

View reviewed changes

Comment thread tests/integration/test_keeper_http_jemalloc/test.py Outdated

clickhouse-gh Bot reviewed Mar 25, 2026

View reviewed changes

Comment thread src/Server/KeeperJemallocHandler.cpp

clickhouse-gh Bot reviewed Mar 25, 2026

View reviewed changes

Comment thread src/Server/KeeperHTTPHandlerFactory.h Outdated

clickhouse-gh Bot reviewed Mar 26, 2026

View reviewed changes

Comment thread programs/server/jemalloc.html

clickhouse-gh Bot reviewed Mar 26, 2026

View reviewed changes

Comment thread src/Server/KeeperJemallocHandler.cpp

clickhouse-gh Bot reviewed Mar 27, 2026

View reviewed changes

Comment thread src/Server/KeeperJemallocHandler.cpp

murphy-4o force-pushed the keeper-jemalloc-webui branch from 69cb391 to ede64f4 Compare March 27, 2026 06:44

clickhouse-gh Bot reviewed Mar 27, 2026

View reviewed changes

Comment thread src/Server/KeeperJemallocHandler.cpp Outdated

groeneai mentioned this pull request Mar 30, 2026

Fix parallel replicas crash with lazy remote source #101154

Merged

murphy-4o added 2 commits March 31, 2026 02:20

Merge branch 'master' into keeper-jemalloc-webui

2bded78

murphy-4o marked this pull request as ready for review April 1, 2026 03:27

murphy-4o requested a review from antonio2368 April 1, 2026 03:27

antonio2368 self-assigned this Apr 1, 2026

Merge branch 'master' into keeper-jemalloc-webui

333142d

Merge branch 'master' into keeper-jemalloc-webui

ef919d4

antonio2368 reviewed Apr 8, 2026

View reviewed changes

Merge remote-tracking branch 'origin/master' into update-pr-100606

d0c1b5f

murphy-4o added 2 commits April 15, 2026 06:54

Merge branch 'master' into keeper-jemalloc-webui

c1ff404

antonio2368 approved these changes Apr 15, 2026

View reviewed changes

murphy-4o enabled auto-merge April 15, 2026 10:25

murphy-4o added this pull request to the merge queue Apr 15, 2026

Merged via the queue into ClickHouse:master with commit 7ce6ac7 Apr 15, 2026
161 checks passed

murphy-4o deleted the keeper-jemalloc-webui branch April 15, 2026 10:53

robot-clickhouse added the pr-synced-to-cloud The PR is synced to the cloud repo label Apr 15, 2026

robot-clickhouse-ci-1 mentioned this pull request Jun 27, 2026

Add jemalloc profiling web UI for ClickHouse Keeper #100544

Closed

		}


		std::string heapProfileToCollapsedStacks(const std::string & input_filename)

Sunbelt Computer Software

PL/B Language Development and Support

Uh oh!

Conversation

murphy-4o commented Mar 24, 2026 • edited by robot-clickhouse Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Documentation entry for user-facing changes

Version info

Uh oh!

clickhouse-gh Bot commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

AI Review

Summary

ClickHouse Rules

Final Verdict

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexey-milovidov commented Apr 7, 2026

Uh oh!

antonio2368 left a comment

Choose a reason for hiding this comment

Uh oh!

antonio2368 Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

murphy-4o Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

antonio2368 Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

murphy-4o Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov commented Apr 9, 2026

Uh oh!

alexey-milovidov commented Apr 12, 2026

Uh oh!

clickhouse-gh Bot commented Apr 15, 2026

LLVM Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

murphy-4o commented Mar 24, 2026 •

edited by robot-clickhouse

Loading

clickhouse-gh Bot commented Mar 24, 2026 •

edited

Loading