Repository navigation
macOS / Apple Silicon: setup_env.py build fails (ggml-base can't find i2_s symbols); after working around it, Metal crashes and CPU output is garbage #641
Description
Environment
Apple Silicon MacBook Pro, macOS 15 (Darwin 24.6.0), Apple clang
BitNet main @ 0b341e5; submodule 3rdparty/llama.cpp @ 390c3077 (release-bitnet-embedding-0.6b-270m)
Model: microsoft/BitNet-b1.58-2B-4T-gguf (ggml-model-i2_s.gguf)
Repro
hf download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4T
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s
- Linker error (same as setup_env.py on MacBook M2 failed #611, root cause below)
[ 2%] Linking CXX shared library ../../../../bin/libggml-base.dylib
Undefined symbols for architecture arm64:
"_dequantize_row_i2_s", referenced from: _type_traits in ggml.c.o
"_quantize_i2_s", referenced from: _ggml_quantize_chunk in ggml.c.o
ld: symbol(s) not found for architecture arm64
Cause: ggml/src/ggml.c (target ggml-base) refers to quantize_i2_s and dequantize_row_i2_s, but both are defined in ggml/src/ggml-cpu/quants.c (target ggml-cpu), which ggml-base doesn't link against. setup_env.py builds shared libraries by default. On Linux an ELF shared library can have undefined symbols that are resolved at load time. macOS (ld64) won't allow that, and Windows/MSVC won't either (probably #595).
Workaround: build static libraries:
cmake -B build -DBITNET_ARM_TL1=OFF -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
-DLLAMA_BUILD_TOOLS=ON -DLLAMA_BUILD_EXAMPLES=ON -DLLAMA_BUILD_COMMON=ON -DLLAMA_BUILD_SERVER=ON
-DBUILD_SHARED_LIBS=OFF -DGGML_BACKEND_DL=OFF
Suggested fixes: move the i2_s quantize/dequantize code into ggml-quants.c (which is part of ggml-base), or have setup_env.py pass -DBUILD_SHARED_LIBS=OFF.
-
src1_cont undeclared (same as ggml-cpu.c:
src1_contundeclared → build fails on ARM CPUs with i8mm (Apple M2/M3/M4, Graviton3, ...) #618)
Once linking works, ggml-cpu.c:1495 fails with use of undeclared identifier 'src1_cont': the I2_S GEMV/GEMM fast path uses a variable that is only declared in another scope. Replacing it with ggml_is_contiguous(src1) fixes the build. -
Metal backend crashes on I2_S
With the default GPU offload, llama-cli aborts inside ggml_metal_op_encode during the first llama_decode. Passing -ngl 0 avoids the crash. Either the Metal backend should refuse I2_S tensors, or run_inference.py should pass -ngl 0 by default. -
CPU output is garbage (same as BitNet-b1.58-2B-4T produces garbage output on ARM64/NEON (no AVX2) — scalar fallback uses wrong I2_S unpacking scheme #600, [Bug]: BitNet FFN uses SILU instead of ReLU² — perplexity 99.8 vs 17.1 on every CPU backend #588)
$ build/bin/llama-completion -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf
-p "The capital of France is" -n 24 --temp 0 -ngl 0 -no-cnv
The capital of France is@@@@@@@@@@@@@@@@@@@@@@@@
The output is the same every run, with -t 1 and with -t 8.
Speed is about 1.3 tok/s with 1 thread and about 10 tok/s with 8 threads. That's far below what this model normally runs at.
This matches #600 (wrong I2_S unpacking and scaling in the ARM path). #588 (SiLU used instead of squared ReLU in the FFN) also affects every backend.
Activity
shifulegend commentedon Oct 11, 2026
The repeated @ tokens are basically the ReLU2 mismatch - BitNet b1.58 relies on squared ReLU, but if your FFN config defaults to SiLU, logits explode after the first decode step and the model collapses into repeating high-frequency tokens. Also, Metal doesn't have ternary kernels yet. Any tensor offloaded to GPU will hit an unhandled op and segfault in ggml_metal_op_encode unless you pass -ngl 0 to keep it on CPU.
If you just want a bare-bones reference to verify your weights and I2_S unpacking without wading through bitnet.cpp's dynamic library splits and CMake setup, we wrote a small C99 engine (https://github.com/shifulegend/project-zero). Compiles with a plain make on macOS and runs the b1.58 ReLU2 path directly on CPU.