Repository navigation
I2_S produces incorrect output on aarch64 #598
Description
I2_S inference is broken on aarch64. The model loads and runs, but output is nonsense — it does not crash, so it looks like a bad model rather than a broken kernel.
On an Orange Pi 5 Plus (RK3588, Debian 12, GCC 12.2) with microsoft/BitNet-b1.58-2B-4T's official ggml-model-i2_s.gguf:
$ llama-cli -m ggml-model-i2_s.gguf -p "The capital of France is" -n 12 --temp 0
> The capital of France is ????????????????
[ Prompt: 0.7 t/s | Generation: 0.7 t/s ]
The same file on x86-64 gives The capital of France is Paris. at ~40 t/s. Model md5 verified identical on both machines.
Three separate bugs, all in 3rdparty/llama.cpp code paths that x86 never compiles:
QK_I2_Sis 128 under AVX2 but 64 under__ARM_NEON, in bothquants.candggml-cpu-i2s.c. It is the on-disk block size, so it must match the file format on every architecture.- The scalar
vec_dotfallback decodes the block-interleaved weight layout sequentially. ggml_gemm_i2_i8_s's ACT_PARALLEL branch invertsggml_vec_dot_i2_i8_s'snrcsemantics, corrupting prefill.
There is also no NEON path for I2_S at all — aarch64 unpacks one 2-bit weight at a time.
Fixes in isHuangXin/llama.cpp#2, against the pinned 3rdparty/llama.cpp submodule. After them, perplexity on aarch64 matches x86 to 0.161% (74.0952 vs 73.9758, same model and corpus), and kernel output is bit-identical for both GEMV and GEMM.
This may be the same root cause as #55.
Activity
matheus-c-feitosa commentedon Oct 7, 2026
Independent repro on Neoverse-N1 (2 cores, NEON + dotprod, no i8mm), Ubuntu 24.04, clang 18.1.3 (you used GCC 12.2), same GGUF (
ggml-model-i2_s.gguf, sha2564221b252…2c162, matches HF LFS):0b341e5/ llama.cpp submodule390c307:????????????at ~0.8 tok/s, bothllama-cliandllama-completion -no-cnv.8000be4): coherent output, ~13 tok/s gen / ~24 tok/s prompt, same peak RSS (1.54 GB). Chat mode answers "Paris"; raw completion gives "a small city, and the capital of France is a small" — I can't tell yet whether that is the model or residual numeric error (no x86/BF16 reference on my side yet).missing pre-tokenizer type … GENERATION QUALITY WILL BE DEGRADEDand the header printsftype: Q1_0for this I2_S file, with and without the PR.Happy to run a PPL/KL comparison against the BF16 weights if useful.
matheus-c-feitosa commentedon Oct 9, 2026
Follow-up on my question above about "a small city": it isn't the model. On the same tokens, the HF transformers implementation and ik_llama.cpp both give WikiText-2 PPL ~18.8 and continue "The capital of France is" with " Paris. Paris is a city…", while bitnet.cpp (x86, and aarch64 with the fix) gives PPL ~113 and " a small city…". Details: #639
stevie101 commentedon Oct 12, 2026
Tested the fixes from isHuangXin/llama.cpp#2 on Apple Silicon. They fix the
garbage output but quality still doesn't match a reference run.
Hardware: Apple M5 Max, macOS [26.5.2], Apple clang 21.0.0
BitNet: main, llama.cpp submodule at 8000be462 (PR branch)
Model: official microsoft/BitNet-b1.58-2B-4T-gguf ggml-model-i2_s.gguf
Build notes:
src1_contundeclared → build fails on ARM CPUs with i8mm (Apple M2/M3/M4, Graviton3, ...) #618 (i8mm target, llamafile disabled)llama.cpp results (custom chat template matching the GGUF format):
even with fresh context, --temp 0.3 and repeat penalty
Reference, same machine, microsoft/bitnet-b1.58-2B-4T-bf16 via transformers
5.19 on MPS, greedy:
pour, steep 3-5 minutes)
So the remaining gap looks like accumulated numerical error in the aarch64
I2_S path rather than the model. Speed: ~120 t/s prompt, ~75 t/s generation,
8 threads. Happy to test further patches.
kiljoy001 commentedon Oct 13, 2026
I will look into this and see if I can improve it.
matheus-c-feitosa commentedon Oct 13, 2026
mclaren (@stevie101) before digging into the aarch64 kernel, it's worth applying the activation fix from #588. isHuangXin/llama.cpp#2 fixes the layout, but the graph still uses SiLU where BitNet b1.58 uses ReLU², and that alone wrecks the output quality. The loops without an end of turn you describe match what #602 reports with SiLU: the model never emits
<|eot_id|>; with ReLU² it stops.On my aarch64 machine (Neoverse-N1, clang 18), with the same official GGUF, WikiText-2 8 x 512, scored on the same token IDs as an HF transformers reference (PPL 18.79):
LLM_FFN_SILU->LLM_FFN_RELU_SQR(src/models/bitnet.cpp)(*)
--override-kv bitnet-b1.58.hidden_activation=str:relu2. #605 keeps SiLU for files without the key, so by design it does not break the 1bitLLM models that use SiLU (not tested here).With either fix, greedy text matches the reference ("The capital of France is Paris. Paris is a city..."), so on this CPU the
remaining gap after llama.cpp#2 was the activation. Your M5 takes the i8mm path (#618), which I could not test.
Two more things from #602 that matter for chat: the GGUF has no pre-tokenizer and an outdated chat template. Put all keys in one comma-separated flag (repeating
--override-kvprints a deprecation warning), e.g.:Scott Guyton (@kiljoy001) this may save you some digging. Tables, method and scripts to reproduce:
https://github.com/matheus-c-feitosa/lowbit-atlas
Analysis assisted by Claude Code; runs were done on my machines and all numbers checked by me.