Garbage output on ARMv8.0 (Cortex-A53/A73) — NEON-only fallback path produces incorrect results · Issue #411 · microsoft/BitNet · GitHub
Skip to content

Garbage output on ARMv8.0 (Cortex-A53/A73) — NEON-only fallback path produces incorrect results #411

Open
Open
Garbage output on ARMv8.0 (Cortex-A53/A73) — NEON-only fallback path produces incorrect results#411

Description

@RKAiCodes05
Ram (RKAiCodes05)
opened on Feb 20, 2026

Environment
Hardware:Amlogic A311D SoC
CPU: 4× Cortex-A73 + 2× Cortex-A53 (ARMv8.0, NEON only, no dotprod)
RAM: 4GB
OS: Ubuntu 20.04 aarch64
Compiler: Clang 18.1.8
BitNet commit: 1f86f058
Model: BitNet-b1.58-2B-4T (official GGUF from microsoft/BitNet-b1.58-2B-4T-gguf)

Problem:-
The model loads correctly but produces garbage output on ARMv8.0 CPUs that lack the dotprod extension. The #else (non-dotprod) NEON path in src/ggml-bitnet-mad.cpp (ggml_vec_dot_i2_i8_s_1x1 using vmlal_s8) appears to compute incorrect results.

###Steps to Reproduce:-

git clone --recursive https://github.com/microsoft/BitNet.git
cd BitNet

setup_env.py fails with NotImplementedError on this hardware

Manual build:

cmake -B build
-DCMAKE_BUILD_TYPE=Release
-DCMAKE_C_COMPILER=clang
-DCMAKE_CXX_COMPILER=clang++
-DCMAKE_C_FLAGS="-mcpu=cortex-a53 -march=armv8-a+crc"
-DCMAKE_CXX_FLAGS="-mcpu=cortex-a53 -march=armv8-a+crc"
-DGGML_NATIVE=OFF
-DBITNET_ARM_TL1=OFF

cmake --build build --config Release -j4

./build/bin/llama-cli
-m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf
-p "Hello, tell me about yourself."
-n 128 -t 4

Expected Output:-
Coherent English text.

Actual Output:-
"Dank spraw/sl-ingimum avReactSM may ad bUISapp.ed,reuschPIrcMarketcryptampo
reco initially CoronaMISS speculate441 Geme Eduardo untilidor.wordpresstor
narrowln halband fast standuce Best Diveamma occasionally Mines choppeddates
raspbreaking-NEL inst de(def caves ¬slaBU platformsapiro uncert stumpilla..."

Issues Found During Investigation:-
1.setup_env.py fails with NotImplementedError in gen_code() — the TL1 kernel generator doesn't support the 2B model dimensions (2560/6912), only 3B (3200/8640).

2.BITNET_ARM_TL1=ON has no effect for the 2B model because ggml_preprocessor() and ggml_qgemm_lut() in the generated bitnet-lut-kernels.h only handle dimensions 3200/8640/3200, not 2560/6912.

3.Hardcoded dotprod inline assembly in ggml-aarch64.c (168 .inst encoded sdot instructions in Q4_0 kernels) — these cause SIGILL if the code path is reached on ARMv8.0. Currently dead code for i2_s, but risky.

4.he official GGUF (microsoft/BitNet-b1.58-2B-4T-gguf) uses general.architecture = bitnet-b1.58, but the fork's llama.cpp expects bitnet — causing tokenizer warnings: GENERATION QUALITY WILL BE DEGRADED.

5.The non-dotprod NEON fallback in ggml_vec_dot_i2_i8_s_1x1 (lines ~340-400 of ggml-bitnet-mad.cpp) uses vmlal_s8 accumulation into int16x8_t, which may overflow for large inner dimensions (2560/6912) since int16 saturates at ±32767 and each accumulation adds values up to ±127×2=±254.

6.Suspected Root Cause
The vmlal_s8 accumulation in the non-dotprod path uses int16x8_t accu32 which accumulates 32 iterations × 8 multiply-adds per iteration = 256 signed int8 products before widening to int32. Each product can be up to 127×2 = 254. Sum of 256 such values = up to 65,024, which overflows int16 (max 32,767). The dotprod path (vdotq_s32) accumulates directly into int32 and doesn't have this problem.

7.Suggested Fix
In the non-dotprod path, reduce the inner loop count before widening to int32, or accumulate directly into int32:
// Instead of accumulating 32 iterations in int16:
// Widen to int32 every 8 iterations instead of every 32

Additional Notes:-
Output is deterministic (same garbage every run)
Single-thread (-t 1) produces identical garbage
Both QK_I2_S=64 and QK_I2_S=128 produce garbage
The model metadata loads correctly (210 i2_s tensors, 121 f32, 1 f16)

Activity

  1. Mindev27 commented on Mar 13, 2026

    @Mindev27
    LEE GANGMIN (Mindev27)
    on Mar 13, 2026

    Hi, I looked into this and the int16 overflow (5, 6) looks like something I can fix. Mind if I open a PR for it?

  2. cbuchner1 commented on Mar 14, 2026

    @cbuchner1
    Christian Buchner (cbuchner1)
    on Mar 14, 2026

    Note: I am seeing the same garbled output on Mac Mini with M4Pro. Also experiencing the tokenizer warnings there.
    I have previously tried to switch to tl1 (from i2_s) but also ran into the NotImplementedError problem.

    EDIT: I have applied Mindev27's fix, but the bad results remain on my platform - so either my platform does not use that code or the overflow is not the root cause for the inference bug.

  3. Mindev27 commented on Mar 14, 2026

    @Mindev27
    LEE GANGMIN (Mindev27)
    on Mar 14, 2026

    Christian Buchner (@cbuchner1) Thanks for testing :)
    In that case it sounds like the M4 Pro might be taking the dotprod path, so my fix wouldn't affect it. The tokenizer warnings might be a separate issue.

  4. betovildoza commented on Mar 15, 2026

    @betovildoza
    betovildoza
    on Mar 15, 2026

    Confirming this on Oracle Cloud Compute A1 Flex (ARM Ampere Altra, armv8.2-a, asimddp present), Ubuntu 22.04, clang 18. Same model: microsoft/BitNet-b1.58-2B-4T-gguf.

    I want to add several findings not mentioned in this issue:

    Additional Bug: to_float callback — type signature mismatch

    In 3rdparty/llama.cpp/ggml/src/ggml.c, the type table for GGML_TYPE_I2_S (line ~1185):

    // ggml_to_float_t typedef (3 params):
typedef void (*ggml_to_float_t)(const void * x, float * y, int64_t n);

// dequantize_row_i2_s actual signature (4 params):
void dequantize_row_i2_s(const uint8_t * x, float * y, int64_t n, const float i2_scale);

// Bug — scale is never passed, ARM reads garbage from register:
.to_float = (ggml_to_float_t) dequantize_row_i2_s,

    On ARM, the 4th argument (i2_scale) is read from whatever value happens to be in the x3 register → undefined behavior → to_float path produces garbage.

    Fix (wrapper in same TU):

    static void dequantize_row_i2_s_wrapper(const void * vx, float * y, int64_t n) {
    const uint8_t * x = (const uint8_t *) vx;
    const float i2_scale = ((const float *)(x + (n / 4)))[0];
    dequantize_row_i2_s(x, y, n, i2_scale);
}
// In type table:
.to_float = dequantize_row_i2_s_wrapper,

    Without this fix: GGGG on all batch sizes. With fix: real words with -b 1 (but still incoherent — see below).

    Confirming the dotprod issue (#411 point 6)

    On ARM Ampere Altra, asimddp is present in /proc/cpuinfo but the default CMakeLists generates -march=armv8.2-a without +dotprod. Adding +dotprod activates vdotq_s32 instead of vmlal_s8, but does not resolve incoherence — it's a necessary but not sufficient fix.

    cmake ... -DCMAKE_C_FLAGS="-march=armv8.2-a+fp16+dotprod"

    Additional finding: gemm path (ne11 > 3) completely broken

    With batch ≥ 4 tokens in the same call (e.g., -b 32), the code uses ggml_gemm_i2_i8_s which produces GGGG on ARM, while the gemv path (-b 1) produces incoherent but real words:

    # gemv (-b 1): words, incoherent
./build/bin/llama-cli -m ggml-model-i2_s.gguf -p "The capital of France is" -n 30 -b 1
# → The capital of France is Freedom reduce Dispatch linear Dahl...

# gemm (-b 32): GGGG
./build/bin/llama-cli -m ggml-model-i2_s.gguf -p "The capital of France is" -n 30 -b 32
# → The capital of France isGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG

    Architecture mismatch confirms incoherence root cause

    The GGUF uses general.architecture = "bitnet-b1.58" with no tokenizer.ggml.pre field. Runtime logs:

    llm_load_vocab: missing pre-tokenizer type, using: 'default'
llm_load_vocab: GENERATION QUALITY WILL BE DEGRADED!

    Adding --override-kv tokenizer.ggml.pre=str:llama-bpe suppresses the warning but does not resolve incoherence, suggesting the architecture mismatch also causes wrong model hyperparameters (attention head counts, etc.) to be loaded.

    Full details, additional bugs, and steps to reproduce in #468.

    Platform: Oracle Cloud A1 Flex, Ampere Altra, armv8.2-a, 4 cores, 24GB RAM, Ubuntu 22.04
    Speed: ~18 tok/s with -b 1 (correct for this hardware)
    Model loads without errors, tokenizer warning is the clearest symptom.

  5. Mindev27 commented on Mar 15, 2026

    @Mindev27
    LEE GANGMIN (Mindev27)
    on Mar 15, 2026

    Oh, thanks for the analysis. It looks like my PR only fixes a very small part of the problem...
    There seem to be quite a few issues on the ARM side. Is it better to split these into separate PRs?
    If so, would it be okay if I try to work on a few of them? Or would it make more sense to add those fixes to my current PR?

  6. betovildoza commented on Mar 15, 2026

    @betovildoza
    betovildoza
    on Mar 15, 2026

    Thanks LEE GANGMIN (@Mindev27)! Great that you're working on this.

    We ran into similar issues on Ampere Altra and documented what we found in #468. Your PR fixes the int16 overflow which is definitely part of the puzzle.

    On your question absolutely, go for it! Separate PRs would probably be easier to review since the bugs are in different files, but whatever works best for you. The fixes for Bug 1 and Bug 2 from #468 are ready to use if they help as a starting point.

    We'd really appreciate it , ARM support for BitNet needs people who actually dig into the code like you're doing. Thanks!

  7. aqn96 commented on Mar 15, 2026

    @aqn96
    An (aqn96)
    on Mar 15, 2026

    Confirming on Raspberry Pi 5 (Cortex-A76, aarch64, dotprod present).

    Filed #470 before finding this thread — same root cause.

    Bisected the regression: commit 404980e produces correct output (9.68 tok/s, 4 threads).
    Commit 112f853 (CPU Optimization update) and all subsequent commits produce garbage.

    Build: clang 18.1.8, cmake 3.25.1, Debian 12 Bookworm, manual build per Adafruit guide.
    Model: microsoft/BitNet-b1.58-2B-4T-gguf (ggml-model-i2_s.gguf)

  8. matheus-c-feitosa commented on Oct 8, 2026

    @matheus-c-feitosa
    Matheus C. Feitosa (matheus-c-feitosa)
    on Oct 8, 2026

    For anyone on ARM still hitting ???? with I2_S: isHuangXin/llama.cpp#2 (open, not merged) fixed it in my tests, including the no-dotprod scalar fallback (built with -DGGML_CPU_ARM_ARCH=armv8-a to mimic A53/A72-class cores). With the official ggml-model-i2_s.gguf, the PR's aarch64 output matched an x86 reference (same greedy text; WikiText-2 PPL within 0.06%). Details: isHuangXin/llama.cpp#2 (comment) . To try it: in 3rdparty/llama.cpp, git fetch origin pull/2/head && git checkout FETCH_HEAD, then rebuild.

  9. matheus-c-feitosa commented on Oct 9, 2026

    @matheus-c-feitosa
    Matheus C. Feitosa (matheus-c-feitosa)
    on Oct 9, 2026

    Correction to my comment above: "fixed it in my tests" was too strong. isHuangXin/llama.cpp#2 removes the ???? output on ARM and makes it match bitnet.cpp's x86 output, but both are ~6x worse in perplexity than the reference implementation on the same tokens. Details: #639

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions