Build `musl` from sources by thevar1able · Pull Request #97452 · ClickHouse/ClickHouse · GitHub
Skip to content

Build musl from sources#97452

Open
thevar1able wants to merge 64 commits into
masterfrom
use-musl
Open

Build musl from sources#97452
thevar1able wants to merge 64 commits into
masterfrom
use-musl

Conversation

@thevar1able

@thevar1able thevar1able commented Feb 20, 2026

Copy link
Copy Markdown
Member

Changelog category (leave one):

  • Build/Testing/Packaging Improvement

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Build musl from sources instead of using vendored binaries.

Add contrib/musl submodule and a CMake build system for it.
Modify toolchains, default_libs, and contrib CMake files to support
USE_MUSL flag for building ClickHouse with musl libc.

Assisted-by: Claude Sonnet 4.5 via GitHub Copilot
Add complete musl source file coverage and move native tool builds
to build-time custom commands so protoc/grpc_cpp_plugin are built
without musl's -nostartfiles flag.

Assisted-by: Claude Sonnet 4.5 via GitHub Copilot
Add USE_MUSL=1 to AMD_RELEASE, ARM_RELEASE, and AMD_MUSL CI build
configurations using the standard toolchain files instead of the
standalone musl toolchain. Add USE_MUSL support to the aarch64
toolchain, mirroring the x86_64 implementation.

Assisted-by: Claude Opus 4.6 via GitHub Copilot
@clickhouse-gh

clickhouse-gh Bot commented Feb 20, 2026

Copy link
Copy Markdown
Contributor

@clickhouse-gh clickhouse-gh Bot added pr-build Pull request with build/testing/packaging improvement submodule changed At least one submodule changed in this PR. labels Feb 20, 2026
Reword a comment in contrib/llvm-project-cmake/CMakeLists.txt to not
mention 'check_symbol_exists', which triggers the cpp style check that
bans dynamic compiler checks in contrib cmake files.

Assisted-by: Claude Opus 4.6 via GitHub Copilot
Add toolchain file and -DUSE_MUSL=1 to FastTest cmake configuration,
and add contrib/musl to the submodule list. This replaces the old
commented-out reference to toolchain-x86_64-musl.cmake.

Assisted-by: Claude Opus 4.6 via GitHub Copilot
…mand

Revert the native tool build from build-time (add_custom_command) back to
configure-time (execute_process) to match the upstream approach. The
add_custom_command approach only declared a stamp file as OUTPUT but not the
actual native binary paths, causing ninja to fail with 'missing and no known
rule to make it' for any cross-compilation build (not just musl).

The condition is extended to also trigger native builds when USE_MUSL is set,
since musl's -nostartfiles breaks standalone executables like protoc.

Remove the now-unnecessary NATIVE_TOOLS_TARGET target-level dependencies from
all proto generator files, since with execute_process the native binaries
already exist on disk before ninja starts.

Assisted-by: Claude Opus 4.6 via GitHub Copilot
Add 14 math functions (log, log2, exp, exp2, expm1, pow, floor, fabs,
scalbn, logf, fabsl, scalbnl, copysignl, fmodl) that were previously
provided by glibc but are needed when building with musl. Also add
explogxf.cpp as a shared dependency for exp/exp2/expm1.

Includes both the source files in the SRCS list and corresponding
x86_64 runtime dispatch entries for CPU-optimized variant selection.

Assisted-by: Claude Opus 4.6 via GitHub Copilot
lexer_test uses plain add_executable instead of clickhouse_add_executable,
so it was missing musl CRT objects (crt1, crti, crtn) while the global
-nostartfiles flag prevented linking glibc's CRT. This caused the binary
to have no _start entry point, resulting in a segfault.

Assisted-by: Claude Opus 4.6 via GitHub Copilot
Include all 232 musl math source files instead of a partial subset.
Previously 129 math files were excluded, assuming LLVM-libc would
provide them, but LLVM-libc only covers ~45 functions. This left 84
math functions (mostly float-precision variants like sinf, cosf, expf)
with no implementation, causing 'undefined symbol: nan' linker errors.

LLVM-libc's optimized versions still take precedence via link ordering
(linked before musl in global-libs).

Also fix ksigaction.h include-override for aarch64: only x86_64 has an
arch-specific ksigaction.h in musl; all other architectures should use
the generic version from src/internal/ksigaction.h.

Assisted-by: Claude Opus 4.6 via GitHub Copilot
@thevar1able thevar1able marked this pull request as draft February 20, 2026 06:07
Create include_linux_aarch64_musl with JEMALLOC_STRERROR_R_RETURNS_CHAR_WITH_GNU_SOURCE
undef'd and JEMALLOC_HAVE_PTHREAD_GETNAME_NP disabled. Without this, the aarch64
build uses the glibc header which enables GNU strerror_r (returns char*), but musl
always provides POSIX strerror_r (returns int), causing -Wint-conversion error in
malloc_io.c.

Assisted-by: Claude Opus 4.6 via GitHub Copilot
With USE_MUSL, -nostartfiles is set globally but only clickhouse_add_executable
adds musl CRT objects (crt1, crti, crtn). Fix decompressor, parquet tools, and
perf_checksum which use plain add_executable and would fail to link with
'cannot find entry symbol _start'.

Assisted-by: Claude Opus 4.6 via GitHub Copilot
With USE_MUSL, the target-built compressor binary cannot run on the host
because it's statically linked against musl. Extend the cross-compilation
check to also trigger when USE_MUSL is set, matching the native build
condition in the root CMakeLists.txt. This ensures the native-built
pre_compressor (without musl) is used on the host, while the target-built
decompressor (with musl) is passed separately for embedding.

Assisted-by: Claude Opus 4.6 via GitHub Copilot
Assisted-by: Claude Opus 4.6 via GitHub Copilot
Assisted-by: Claude Opus 4.6 via GitHub Copilot
# Conflicts:
#	ci/jobs/build_clickhouse.py
Assisted-by: Claude Opus 4.6 via GitHub Copilot
Assisted-by: Claude Opus 4.6 via GitHub Copilot
thevar1able added a commit that referenced this pull request Mar 9, 2026
Add 18 additional math functions from LLVM-libc: `exp`, `exp2`,
`expm1`, `fabs`, `fabsl`, `floor`, `fmodl`, `log`, `log2`,
`logf`, `pow`, `scalbn`, `scalbnl`, `copysignl`, `nan`, `nanf`,
`nanl`, and the `explogxf` shared constants. Also add corresponding
x86_64 runtime dispatch entries.

Taken from #97452

Assisted-by: Claude Opus 4.6 via GitHub Copilot
Comment thread cmake/build_clang_builtin.cmake Outdated
Comment thread base/base/getL2CacheSize.h Outdated
@alexey-milovidov alexey-milovidov mentioned this pull request Jun 11, 2026
1 task
@clickhouse-gh

clickhouse-gh Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

LLVM Coverage Report

Metric Baseline Current Δ
Lines 85.10% 85.00% -0.10%
Functions 92.40% 92.30% -0.10%
Branches 77.30% 77.30% +0.00%

Changed lines: Changed C/C++ lines covered by tests: 42/52 (80.77%) | Lost baseline coverage: none · Uncovered code

Full report · Diff report

# Conflicts:
#	base/base/getL2CacheSize.cpp
#	base/base/getL2CacheSize.h
#	base/base/getPageSize.cpp
#	ci/jobs/compatibility_check.py
#	ci/jobs/fast_test.py
#	src/Common/ColumnsHashing.h
#	src/Interpreters/Aggregator.cpp
#	src/Interpreters/HashJoin/HashJoin.cpp
@clickhouse-gh

This comment was marked as outdated.

pull Bot pushed a commit to AKJUS/ClickHouse that referenced this pull request Jul 2, 2026
Build `memcmp`, `memcpy`, `memmove`, `memset`, `bcmp` and `memmem` from
LLVM-libc into the existing `libllvmlibc` target (which already supplies the
math functions), replacing the previous hand-rolled `memcmp`.

The wrapper TU `contrib/libllvmlibc-cmake/string/mem_functions.cpp` includes
the LLVM-libc implementation files and marks each exported C symbol `weak`
via the `LLVM_LIBC_FUNCTION_ATTR_<name>` hook, so sanitizer interceptors can
override them and so the existing strong `memcpy` from
`base/glibc-compatibility/memcpy` keeps precedence. `memmem` is a strong
symbol.

On x86_64, `x86_64_mem_functions.cpp` provides `memcpy`/`memmove`/`memset`
with runtime AVX-512 dispatch (a lazily-resolved function pointer, the same
shape as a glibc IFUNC) plus an AVX-512/AVX2/scalar `memmem`. On aarch64,
`aarch64_mem_functions.cpp` adds a NEON `memmove` and `memmem`.

The library is built with `-ffreestanding -fno-builtin` so clang's loop-idiom
recognition does not rewrite the inner loops into calls to the very functions
being defined. SIMD tiers are selected from `X86_ARCH_LEVEL` (SSE2 / v3 /
v4 with `-mprefer-vector-width=512`).

Ported from the `use-musl` branch (PR ClickHouse#97452).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
thevar1able and others added 2 commits July 2, 2026 23:04
- Revert CI job changes that switched release/fast-test builds to musl and
  rewrote the compatibility check: that is later "musl as main build" work.
- Revert LLVM-libc mem functions (split to PR #107566); keep musl's own
  mem*/bcmp, restoring the x86_64 asm memcpy/memmove/memset.
- Remove the llvm-tblgen -march pin and protoc DEPENDS additions that are
  not needed for this change.
- Restyle contrib/musl-cmake and cmake/musl.cmake closer to the usual
  contrib cmake style: drop the unused musl-Scrt1 target, banner comments
  and redundant arch detection; MUSL_ARCH now comes from the toolchain files.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@thevar1able thevar1able marked this pull request as ready for review July 2, 2026 21:48
@clickhouse-gh clickhouse-gh Bot added the pr-build Pull request with build/testing/packaging improvement label Jul 2, 2026
Comment thread cmake/musl.cmake Outdated
Comment thread base/harmful/harmful.c Outdated
thevar1able and others added 3 commits July 3, 2026 01:38
The AI reviewer correctly noted that the startup files must be linked as
crt1, crti, [program objects], crtn: crti/crtn provide the prologue and
epilogue of the .init/.fini sections and must wrap every other
contribution to them. Reordering source lists cannot express this:
CMake's Ninja generator always emits $<TARGET_OBJECTS:...> before the
target's own object files, so crtn.o preceded the program objects
regardless of its position in add_executable or target_sources.

Wire the startup files through the link line instead: the objects are
copied to stable paths in contrib/musl-cmake, crt1.o and crti.o are
passed via CMAKE_EXE_LINKER_FLAGS (placed before all object files) and
crtn.o at the end of DEFAULT_LIBS (placed after all libraries). This
covers every executable uniformly, so the per-target special cases in
clickhouse_add_executable, contrib and test CMake files are removed.

The order is not hypothetical: OpenSSL contributes a .init fragment
calling OPENSSL_cpuid_setup, which previously landed after crtn's
epilogue (unreachable, so it never ran at startup); a binary whose .init
lacks the crtn epilogue altogether segfaults in __libc_start_init,
because musl calls _init. With this change, .init of the clickhouse
binary disassembles to the complete push/call/pop/ret sequence.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Disabling the trap unconditionally weakened the harmful-function guard
on glibc builds, where mbsrtowcs is not needed by libc internals. Only
musl uses it internally (in getaddrinfo), so limit the exemption to
USE_MUSL as suggested by the AI reviewer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The glibc userspace headers from the sysroot were reachable in every
compilation through clang's builtin sysroot search path, so any header
musl does not provide (or does not win in the include order) silently
fell back to glibc declarations without a link-time signal; the
execinfo.h stub was one manually-shadowed example. The AI reviewer
suggested narrowing the fallback to the kernel headers only.

Compile everything with -nostdlibinc and pass the musl include
directories explicitly in the toolchain flags. The kernel (uapi)
headers, the only part of the sysroot that must stay reachable, are
whitelisted through a directory of symlinks created at configure time
in cmake/musl.cmake. A header musl does not provide is now a compile
error: for example, #include <gnu/libc-version.h> fails with "file not
found", while execinfo.h resolves to the stub and linux/asm headers to
the whitelist.

The two target_include_directories(global-libs ...) blocks removed here
were inert: only INTERFACE_LINK_LIBRARIES of global-libs is forwarded
to global-group, so these directories never appeared on any compile
line; the fallback came from the builtin sysroot path, which
-nostdlibinc disables.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@thevar1able thevar1able removed the pr-performance Pull request with some performance improvements label Jul 2, 2026
Comment thread contrib/compiler-rt-cmake/CMakeLists.txt Outdated
The compiler-rt build resets CMAKE_<LANG>_FLAGS to a whitelist, which
kept the musl -isystem header paths but dropped -nostdlibinc, so the
runtime support objects — linked into every musl executable — could
still silently fall back to the sysroot's glibc userspace headers.
Keep -nostdlibinc through the strip step, as noted by the AI reviewer,
so compiler-rt honors the same "musl headers only" contract as the
rest of the build.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
set(MUSL_GENERATED_DIR "${ClickHouse_BINARY_DIR}/contrib/musl-cmake/include")
set(MUSL_INCLUDE_DIR "${ClickHouse_SOURCE_DIR}/contrib/musl/include")

set(MUSL_C_INCLUDES "-isystem ${MUSL_GENERATED_DIR} -isystem ${MUSL_ARCH_DIR} -isystem ${MUSL_GENERIC_DIR} -isystem ${MUSL_INCLUDE_DIR}")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MUSL_C_INCLUDES no longer reconstructs the full header set that the musl toolchain started with. The regex above strips every -isystem ...musl... entry out of CMAKE_*_FLAGS, which removes both contrib/musl-cmake/include (the new execinfo.h stub) and contrib/musl-cmake/kernel-headers.

That means Cargo cc/cxx builds under USE_MUSL compile against a narrower header surface than the rest of the tree. A source that includes <execinfo.h> already fails here even though this PR intentionally adds that header for musl builds. Please re-add the stub and kernel-header directories when rebuilding MUSL_C_INCLUDES.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This else () hard-codes RelWithDebInfo for every non-DEBUG musl build. As written, -DCMAKE_BUILD_TYPE=Release and MinSizeRel still feed Cargo cc/cxx shims the CMAKE_*_FLAGS_RELWITHDEBINFO profile.

That breaks the build-type contract specifically for musl artifacts and can leave release builds with the wrong optimisation/debug settings. Please mirror the non-musl per-build-type cases here, or select CMAKE_<LANG>_FLAGS_${CMAKE_BUILD_TYPE_UC} dynamically instead of forcing RelWithDebInfo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-build Pull request with build/testing/packaging improvement submodule changed At least one submodule changed in this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant