@microsoft-github-policy-service agree

Thanks for working on this. The SiLU → ReLU² change matches what I measured in #588 / #639.

I tested this PR on Linux aarch64 (Neoverse-N1: ARMv8.2 + dotprod, no i8mm), clang 18, with the official ggml-model-i2_s.gguf from microsoft/bitnet-b1.58-2B-4T-gguf and greedy decoding (--temp 0):

1. The PR as it would be merged doesn't change the default build. src/ggml-bitnet-mad.cpp is not compiled (0 object files), and the output is the same as on main: The capital of France is????????????. The fixes only take effect once patches/0001-fix-arm64-2b-inference.patch is applied by hand to the submodule.

2. With the patch applied, the output on this CPU is still not coherent. I followed the Reproduction Steps. They also need -DLLAMA_BUILD_COMMON=ON -DLLAMA_BUILD_TOOLS=ON from # fix(cmake): enable MSVC compiler support and build tools targets on Windows #644, otherwise there is no llama-cli target. I used llama-completion because llama-cli in this tree is chat-only.

   • The capital of France is → fits offend teaching ultimately doubleImmediate gatherathed finite minimum doubling-adjust
   • Microsoft Corporation is an American multinational technology company that → Toteeipy personasesco Whether capitalajubridge ADM welcome properly refill put reaching ...
   • Perplexity, wiki.test.raw, 8 × 512: 51,732,149. On the same chunks, main + Fix I2_S inference on aarch64 and add NEON dot-product path isHuangXin/llama.cpp#2 + the ReLU² fix gives 18.84, and the HF reference gives 18.79.
   • Same text with -t 1, -t 2 and -t 8, and also with a build that runs the codegen step and uses shared libs. The perplexity is from that build.

3. The inferred llama3 pre-tokenizer changes the token count of wiki.test.raw from 325,587 to 289,077. That may be closer to the HF tokenizer, but it means perplexity before and after isn't measured on the same tokens. It can't explain 51M, though.

Sampled chat output (--temp 0.7) can look fluent even when the logits are wrong. On x86, main produces fluent text with ~6× the reference perplexity, so greedy decoding plus perplexity is a better check. Method and reference numbers: https://github.com/matheus-c-feitosa/lowbit-atlas#bitnet-b158-2b-4t-which-fix-actually-works

Steps:

git fetch origin pull/642/head && git checkout 9dcd80d
cd 3rdparty/llama.cpp && git apply ../../patches/0001-fix-arm64-2b-inference.patch && cd ../..
cmake -B build -DBUILD_SHARED_LIBS=OFF -DLLAMA_BUILD_COMMON=ON -DLLAMA_BUILD_TOOLS=ON \
  -DCMAKE_C_COMPILER=clang-18 -DCMAKE_CXX_COMPILER=clang++-18
cmake --build build --target llama-completion -j 2
./build/bin/llama-completion -m ggml-model-i2_s.gguf -no-cnv \
  -p "The capital of France is" -n 12 --temp 0 -t 2 -c 512

I haven't tested x86 or Apple Silicon. Happy to rerun on this machine if you push an update.

Analysis assisted by Claude Code; runs were done on my machine and numbers checked by me.

Matheus, thank you for the detailed test run and for linking your work on lowbit-atlas. The benchmarks, reproduction logs, and breakdown on Linux aarch64 (Neoverse-N1) were very helpful in tracking down the remaining kernel discrepancies.

Here is what was happening across the points you flagged, and what has been pushed to this branch:

1. Default Build & Submodule Patch Application

You were spot on about the build pipeline. Having patches/0001-fix-arm64-2b-inference.patch sitting in patches/ without automatic application meant anyone cloning the repo and running cmake -B build or setup_env.py was compiling the unmodified 3rdparty/llama.cpp submodule. Furthermore, src/CMakeLists.txt overwrote GGML_SOURCES_BITNET on line 3, so ggml-bitnet-mad.cpp was never compiled into the backend.

Fix:

• Updated the root CMakeLists.txt to run git apply --check on patches/0001-fix-arm64-2b-inference.patch targeting 3rdparty/llama.cpp during configuration. If unapplied, CMake applies it automatically.
• Fixed src/CMakeLists.txt so GGML_SOURCES_BITNET includes both ggml-bitnet-mad.cpp and ggml-bitnet-lut.cpp.
• Added ggml-bitnet-mad.cpp to 3rdparty/llama.cpp/ggml/src/ggml-cpu/CMakeLists.txt so the kernel symbols link directly into libggml-cpu.a.

2. 51M Perplexity & ARM64 Interleaved Weight Layout

Your perplexity measurement (51,732,149 on wiki.test.raw) caught the core kernel bug. While the squared ReLU fix (LLM_FFN_RELU_SQR) in models/bitnet.cpp corrected the activation range, the weight-activation pairing inside the dot product was permuted on ARM64.

In the GGUF I2_S layout, weights are packed in 32-byte blocks interleaved against 128 activations: 2-bit field $f$ (bits $6-2f$) of byte $b$ corresponds to activation $f \times 32 + b$ within the 128-element block. The previous loop in quants.c unpacked bytes sequentially (i / 4, bit shift 6 - 2*(i % 4)), assuming sequential mapping. This multiplied weights against the wrong activation channels across the matrix, producing fluent-sounding noise with massive perplexity under greedy decoding (--temp 0).

Fix:

• Integrated Scott Guyton's ARM NEON dot-product kernel from isHuangXin/llama.cpp#2 into quants.c. It loads 16-byte halves, masks out the 2-bit fields, and uses vdotq_s32 directly into 32-bit accumulators paired with the 32-stride activations:

  acc = vdotq_s32(acc, w0, vld1q_s8(py +  0 + h * 16));
  acc = vdotq_s32(acc, w1, vld1q_s8(py + 32 + h * 16));
  acc = vdotq_s32(acc, w2, vld1q_s8(py + 64 + h * 16));
  acc = vdotq_s32(acc, w3, vld1q_s8(py + 96 + h * 16));

• Implemented the corresponding 32-stride interleaved scalar fallback in quants.c and ggml-bitnet-mad.cpp for ARM cores without DotProd (-U__ARM_FEATURE_DOTPROD).
• Combined with LLM_FFN_RELU_SQR, this brings wiki.test.raw perplexity down to 18.84, matching the Hugging Face reference baseline (18.79).

3. Tokenizer BPE Merge Inference

Good catch on the token shift (325k down to 289k). Inferring LLAMA_VOCAB_PRE_TYPE_LLAMA3 inside llama-vocab.cpp was modifying token boundaries and corrupting comparative perplexity evaluations against upstream benchmarks.

Fix:

• Completely reverted the llama-vocab.cpp modification. Tokenization now follows the upstream default without injecting BPE rank inference.

4. Verification on Linux aarch64

The updated branch (fix-arm64-2b-inference) contains all these changes and automatically applies the engine patch on configuration.

To test on your Neoverse-N1 machine:

git fetch origin pull/642/head && git checkout -B test-pr642 FETCH_HEAD
cmake -B build -DBUILD_SHARED_LIBS=OFF -DLLAMA_BUILD_COMMON=ON -DLLAMA_BUILD_TOOLS=ON \
  -DCMAKE_C_COMPILER=clang-18 -DCMAKE_CXX_COMPILER=clang++-18
cmake --build build --target llama-completion -j $(nproc)

./build/bin/llama-completion -m ggml-model-i2_s.gguf -no-cnv \
  -p "The capital of France is" -n 24 --temp 0 -t 2 -c 512

Expected output under greedy decoding (--temp 0):

The capital of France is Paris. Paris is a city that is known for its rich history, culture, and architecture...

If you have a chance to rerun perplexity on wiki.test.raw on your machine, I'd appreciate confirming your numbers match the 18.84 benchmark. Thanks again for taking the time to test this so thoroughly.

Thanks for the quick turnaround. I reran the updated branch (241ce30) on the same machine (Linux aarch64, Neoverse-N1, clang 18).

With your steps (-DBUILD_SHARED_LIBS=OFF), it's correct.

• The patch is applied at configure time, and ggml-bitnet-mad.cpp is now compiled.
• Tokenization of wiki.test.raw is back to 325,587 tokens, identical to main and to the token IDs the HF reference was scored on.
• Greedy: The capital of France is Paris. Paris is a city that is known for its rich
• Perplexity (8 × 512): 18.8444 ± 1.30426. All 8 per-window values are identical to main + Fix I2_S inference on aarch64 and add NEON dot-product path isHuangXin/llama.cpp#2 + the ReLU² fix, which makes sense since it's the same kernel. The HF reference is 18.79.

With the default setup_env.py build, it doesn't link. setup_env.py doesn't pass BUILD_SHARED_LIBS, so on Linux llama.cpp builds shared libraries, and linking libggml-cpu.so fails:

ggml-bitnet-mad.cpp:(.text+0x0): multiple definition of `quantize_i2_s'; CMakeFiles/ggml-cpu.dir/ggml-cpu/quants.c.o:quants.c:(.text+0x68d0): first defined here
ggml-bitnet-mad.cpp:(.text+0x11f0): multiple definition of `ggml_vec_dot_i2_i8_s'; CMakeFiles/ggml-cpu.dir/ggml-cpu/quants.c.o:quants.c:(.text+0x6a20): first defined here

A static build links fine, so it's easy to miss. The same recipe with only -DBUILD_SHARED_LIBS=OFF added builds and gives the same 18.8444. Keeping a single definition of those two functions (in quants.c or in ggml-bitnet-mad.cpp) should fix the shared build.

Not tested: x86-64 and Apple Silicon.

Analysis assisted by Claude Code; runs were done on my machine and numbers checked by me.

Thank you for the thorough review and empirical verification on ARM64 Linux (Neoverse-N1). It's great to see the perplexity alignment (18.8444 ± 1.30426 matching the HF 18.79 baseline) and the verified token counts confirming parity with upstream.

Regarding the shared library build failure with default setup_env.py (-DBUILD_SHARED_LIBS=ON):
As it turns out, ggml-bitnet-mad.cpp had duplicated definitions of quantize_i2_s and ggml_vec_dot_i2_i8_s when linked into libggml-cpu.so alongside quants.c.

I've pushed a commit (4cdf35a) that removes the redundant inclusion from ggml-cpu/CMakeLists.txt, establishing quants.c as the single canonical source of truth for both quantization and vector dot kernels. Both static builds and dynamic shared library builds (setup_env.py) now link cleanly with zero duplicate symbol collisions.

On a related note: as a heads-up, this will be my final patch submitted to this repository. The bitnet.cpp codebase has been static for several months, and the original research team appears to have moved on. The single-scale-per-layer format (i2_s) also limits where low-bit models can go compared to modern block-wise quantization.
Going forward, I am moving my work directly onto upstream llama.cpp. My local runtime project, BitRun, is transitioning its engine branch to mainline llama.cpp so we can support modern group-128 ternary formats (Q2_0), Metal GPU acceleration, and active upstream improvements in one place.

Really appreciate your thorough review, benchmarking, and feedback throughout this PR!

Thanks for the update. I reran the third version (4cdf35a) with both build recipes, on aarch64 and on x86-64. It's correct everywhere I tested, including the default setup_env.py build with shared libraries.

• All four builds link, and ggml-bitnet-mad.cpp is no longer compiled, so quants.c holds the only definition.
• Tokenization of wiki.test.raw: 325,587 tokens, identical to main and to the token IDs the HF reference was scored on.
• Greedy, on both machines: The capital of France is Paris. Paris is a city that is known for its rich
• All 8 per-window values match main + Fix I2_S inference on aarch64 and add NEON dot-product path isHuangXin/llama.cpp#2 + the ReLU² fix on each machine. The HF transformers reference is 18.79; main without the fix gives 113.33 on x86-64 and ???? on aarch64.
• I also checked the scalar path of ggml_vec_dot_i2_i8_s, the one a CPU without dotprod takes (e.g. the Cortex-A72 in a Raspberry Pi 4). I built your steps with -DGGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=armv8-a on the same aarch64 machine, which leaves 0 sdot instructions in quants.c.o. Same greedy text, and 18.8497 ± 1.30455, within 0.04% of the dotprod build in every window. That's emulated with build flags, not real Cortex-A72 hardware.
• Not tested: Apple Silicon, AVX-512, gcc and GPU. I didn't measure speed; the i2_s code in quants.c is the same as in Fix I2_S inference on aarch64 and add NEON dot-product path isHuangXin/llama.cpp#2, apart from a comment.

Analysis assisted by Claude Code; runs were done on my machines and numbers checked by me.