Environment

• Platform: Oracle Cloud Compute A1 Flex (Ampere Altra, armv8.2-a, asimddp present in /proc/cpuinfo)
• OS: Ubuntu 22.04 LTS
• Compiler: clang 14.0
• Commit: main branch, March 2026
• Build flags:

  -DBITNET_ARM_TL1=ON
  -DCMAKE_C_COMPILER=clang
  -DCMAKE_CXX_COMPILER=clang++
  -DCMAKE_C_FLAGS="-march=armv8.2-a+fp16+dotprod"
  

• Model: microsoft/BitNet-b1.58-2B-4T-gguf (ggml-model-i2_s.gguf, 1.2GB)
• Python: 3.12 (venv)

System info (from llama-cli output)

NEON = 1 | ARM_FMA = 1 | SVE = 0 | MATMUL_INT8 = 0 | (all x86 = 0)

Problem

With -b 1 (gemv path): model produces real words but completely incoherent output.
With -b 32+ (gemm path for prompt): model produces GGGG garbage for all generated tokens.

# gemv path (-b 1, default in run_inference.py)
$ python run_inference.py -m models/BitNet-b1.58-2B-4T-gguf/ggml-model-i2_s.gguf \
    -p "The capital of France is" -n 30 -t 4

The capital of France is offline officially.ai happy collective lower sl ruled m unit proven-for...

# gemm path (-b 32)
$ ./build/bin/llama-cli -m models/BitNet-b1.58-2B-4T-gguf/ggml-model-i2_s.gguf \
    -p "Hello, my name is" -n 50 -t 4 -b 32

Hello, my name isGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG

Speed is correct (~18 tok/s on 4 ARM cores), model loads without errors.

Bug 1: to_float callback — type mismatch (confirmed, patched)

In 3rdparty/llama.cpp/ggml/src/ggml.c, the type table for GGML_TYPE_I2_S registers dequantize_row_i2_s as a ggml_to_float_t callback, but the signatures do not match:

// Typedef (3 params):
typedef void (*ggml_to_float_t)(const void * x, float * y, int64_t n);

// Actual function (4 params — 4th is the i2 scale):
void dequantize_row_i2_s(const uint8_t * x, float * y, int64_t n, const float i2_scale);

// Bug in type table (ggml.c ~line 1185):
.to_float = (ggml_to_float_t) dequantize_row_i2_s,  // scale never passed → UB

The cast silences the compiler but causes undefined behavior — on ARM, the 4th argument register is garbage → i2_scale is random → dequantized weights are garbage.

Applied fix — wrapper in the same translation unit:

// At top of ggml.c:
static void dequantize_row_i2_s_wrapper(const void * vx, float * y, int64_t n) {
    const uint8_t * x = (const uint8_t *) vx;
    const float i2_scale = ((const float *)(x + (n / 4)))[0];
    dequantize_row_i2_s(x, y, n, i2_scale);
}

// In type table:
.to_float = dequantize_row_i2_s_wrapper,

Effect: without this fix → GGGG for all configs. With fix → real words with -b 1 (but still incoherent, see Bug 3).

Bug 2: +dotprod not enabled in CMakeLists for ARM (build issue)

The build system generates -march=armv8.2-a without +dotprod. On ARM Ampere Altra:

• CPU has asimddp (dotprod SDOT/UDOT) in /proc/cpuinfo
• Without +dotprod, __ARM_FEATURE_DOTPROD is not defined by clang
• ggml-bitnet-mad.cpp falls to the #else path using vmlal_s8 (int16 accumulator)
• For vectors of 2560+ elements, int16 accumulator overflows → incorrect results (related to Garbage output on ARMv8.0 (Cortex-A53/A73) — NEON-only fallback path produces incorrect results #411)

Verification:

clang -march=armv8.2-a+fp16+dotprod -dM -E - < /dev/null | grep DOTPROD
# → #define __ARM_FEATURE_DOTPROD 1

clang -march=armv8.2-a+fp16 -dM -E - < /dev/null | grep DOTPROD
# → (nothing)

Suggested fix: add +dotprod to CMakeLists.txt for AArch64 targets when CPU supports it, or document it as a required flag.

Bug 3: gemm path (ne11 > 3) produces GGGG on ARM — root cause unknown

When the batch has more than 3 tokens (prompt evaluation with -b 32), the code takes the ggml_gemm_i2_i8_s path instead of ggml_gemv_i2_i8_s. On ARM, this path produces completely wrong output:

# Relevant code path in ggml.c (~line 13268):
if (gemm && (ne11 > 3)) {
    if (src0->type == GGML_TYPE_I2_S) {
        // gemm path — produces GGGG on ARM
        const float * scale = (float *)((uint8_t*)(src0->data) + (ne00 * ne01 / 4));
        gemm(...);
        for (...) {
            tmp[...] = (tmp[...] - act_sums[col]) / (act_scales[col]) * (*scale);
        }
    }
}

Since prompt tokens are processed via gemm and corrupt the KV cache, all subsequent generated tokens are also garbage.

Workaround: use -b 1 to force all computation through the gemv path (avoids the bug but is slower and still produces incoherent text).

Bug 4: incoherent output even with gemv path (remaining issue)

Even with -b 1 + Bug 1 fix + dotprod enabled, output is semantically incoherent. The model generates real vocabulary tokens but with no semantic coherence:

"The capital of France is offline officially.ai happy collective..."
"Hello, my name is Freedom reduce Dispatch linear Dahl tent corps..."

This persists even with temperature=0 (greedy decoding), suggesting the issue is in the logit distribution rather than sampling. Possible causes:

• act_sums (activation zero-point correction) computed incorrectly for ARM
• Scale offset wrong for GQA tensors (K/V: [2560, 640] with 5 KV heads vs Q: [2560, 2560])
• Activation quantization (quantize_row_i8_s) producing incorrect scale values

Steps to reproduce

# 1. Clone and build (Ubuntu 22.04, ARM Ampere)
git clone https://github.com/microsoft/BitNet.git && cd BitNet
git submodule update --init --depth=1
python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt

# 2. Build (add +dotprod — not in default CMakeLists)
cmake -S . -B build \
  -DBITNET_ARM_TL1=ON \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_C_FLAGS="-march=armv8.2-a+fp16+dotprod" \
  -DCMAKE_CXX_FLAGS="-march=armv8.2-a+fp16+dotprod" \
  -DCMAKE_BUILD_TYPE=Release
make -C build -j4

# 3. Download model
huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf \
  --local-dir models/BitNet-b1.58-2B-4T-gguf

# 4. Test (without Bug 1 fix — GGGG for all batch sizes)
python run_inference.py -m models/BitNet-b1.58-2B-4T-gguf/ggml-model-i2_s.gguf \
  -p "The capital of France is" -n 30

# 5. Apply Bug 1 fix to 3rdparty/llama.cpp/ggml/src/ggml.c (see above)
# Rebuild and test — now produces words with -b 1 but still incoherent

Related issues

• Garbage output on ARMv8.0 (Cortex-A53/A73) — NEON-only fallback path produces incorrect results #411 — int16 overflow in NEON 1x1 kernel (same root area, different manifestation)
• Model only outputs G repeatedly in interactive mode with ggml-model-i2_s.gguf #195 — ARM inference issues (similar platform)

Hardware confirmation

Tested exclusively on Oracle Cloud Compute A1 Flex (ARM Ampere Altra).
Reports from Reddit/r/LocalLLaMA indicate M1/M2/M3 Mac users achieve coherent output — suggesting the issue may be ARM Ampere-specific or related to the dotprod/vmlal path difference.