[Bug]: i2_s quantization produces garbage output on x86 CPUs without AVX2 (Missing fallback) · Issue #547 · microsoft/BitNet · GitHub
Skip to content

[Bug]: i2_s quantization produces garbage output on x86 CPUs without AVX2 (Missing fallback) #547

Open
#633
#580
Open
[Bug]: i2_s quantization produces garbage output on x86 CPUs without AVX2 (Missing fallback)#547
#633
#580

Description

@ViniciusSchererDeOliveira
Vinicius Scherer De Oliveira (ViniciusSchererDeOliveira)
opened on Apr 18, 2026

Description

When running inference using the i2_s quantization format on an x86 CPU that lacks AVX2 support (e.g., Ivy Bridge architecture), the model produces garbage output, specifically an infinite loop of the space token (e.g., GGGGGG...).

This issue does not occur with standard quantizations like Q2_K on the same exact build and hardware, which successfully outputs coherent text at ~8 tokens/s using AVX1.

Environment

  • OS: Void Linux
  • CPU: Intel Core i5-3317U (Ivy Bridge)
  • Instruction Set: AVX (AVX1), SSE4.2 (Strictly NO AVX2, AVX-512, or FMA)
  • Build Flags: cmake .. -DGGML_AVX2=OFF -DGGML_FMA=OFF -DCMAKE_BUILD_TYPE=Release

Steps to Reproduce

  1. Clone the main branch of microsoft/BitNet.
  2. Build the project forcing AVX2 off (simulating/matching older x86 hardware).
  3. Run inference with an i2_s model (e.g., bitnet_b1_58-large or 2B):
    ./build/bin/llama-cli -m ggml-model-i2_s.gguf -p "Hello" -n 50
  4. The system info correctly reports: AVX = 1 | AVX2 = 0 | MATMUL_INT8 = 0.
  5. The output gets stuck in an infinite loop of G (or the equivalent space token for the loaded vocabulary).

Root Cause Analysis

After investigating the codebase, it appears the i2_s format is missing a functional fallback for x86 architectures without AVX2:

  1. In ggml.c, the function ggml_cpu_has_matmul_int8() returns 0 for all non-ARM architectures.
  2. In src/ggml-bitnet-mad.cpp (e.g., around ggml_vec_dot_i2_i8_s_1x1), the i2_s dot product kernels are wrapped entirely in #if defined(__AVX2__).
  3. There is no #elif defined(__AVX__) fallback using 128-bit intrinsics, nor a standard #else scalar fallback that correctly handles the ternary bit-unpacking for x86.
  4. Testing note: Manually injecting a standard C++ scalar fallback loop into ggml-bitnet-mad.cpp resulted in the exact same garbage output (just significantly slower), suggesting the baseline ternary unpacking logic for non-AVX2 x86 paths might be entirely missing or failing silently.

Suggested Fix

To ensure compatibility with legacy x86 hardware (similar to how ggml gracefully handles Q2_K), the i2_s kernels in src/ggml-bitnet-mad.cpp need:

  1. An #elif defined(__AVX__) path utilizing 128-bit intrinsics (similar to the pattern used for Q2_K in ggml-quants.c).
  2. A robust #else scalar fallback to ensure correct bit-unpacking when no SIMD instructions are available.

Thank you for your incredible work on the 1-bit LLM architecture! I'm available to test any potential patches on my hardware if needed.

Activity

  1. stackblogger commented on Apr 18, 2026

    @stackblogger
    Jameer Khan (stackblogger)
    on Apr 18, 2026

    Checkout this commit 404980e. Till that time it was working fine but after that it started producing garbage data.

    You can checkout this BitNet Stack repo also that uses the BitNet and built a web interface to chat with the LLM directly. It keeps the context, stores all chat history in browser, starts a conversation and many more feature.

  2. ViniciusSchererDeOliveira commented on Apr 23, 2026

    @ViniciusSchererDeOliveira
    Vinicius Scherer De Oliveira (ViniciusSchererDeOliveira)
    on Apr 23, 2026
    Author

    Hey Jameer Khan (@stackblogger), thanks for the suggestion!

    I just ran a clean build and tested this on my end, but unfortunately, rolling back didn't solve the issue for my specific hardware. I tested both the commit you pointed out (404980e) and the latest main branch, explicitly building both with -DGGML_AVX2=OFF.

    In both builds, I got the exact same result: the model gets stuck in an infinite GGGGGG... loop when using the i2_s format.

    While that commit might have been the breaking point for other architectures (like ARM), this confirms that for legacy x86 CPUs without AVX2, the standard scalar fallback for i2_s has either always been missing or was fundamentally broken even before those recent updates.

    Also, thanks for sharing the BitNet Stack repo! I'll definitely take a look at it!

  3. stackblogger commented on Apr 23, 2026

    @stackblogger
    Jameer Khan (stackblogger)
    on Apr 23, 2026

    Yeah someone has already created a bug for it - #470

  4. shifulegend commented on Jun 6, 2026

    @shifulegend
    ssshhh (shifulegend)
    on Jun 6, 2026

    This is a real bug and your root cause analysis is correct. The i2_s kernels are gated entirely on #if defined(__AVX2__) with no scalar fallback, so on AVX1-only hardware the unpacking silently produces zeros/garbage rather than falling back to correct scalar math.

    The fix pattern that works: a runtime dispatch table selected at startup based on cpuid output, not compile-time defines. This way the same binary runs correctly on AVX1, AVX2, and AVX-512 hardware — each codepath is compiled separately and selected at runtime.

    Something like:

    typedef void (*dot_fn)(float*, const void*, const int8_t*, int);
static dot_fn dispatch_ternary_dot(void) {
    if (__builtin_cpu_supports("avx512vnni")) return ternary_dot_avx512;
    if (__builtin_cpu_supports("avx2"))       return ternary_dot_avx2;
    return ternary_dot_scalar;  // always-correct fallback
}

    We use this pattern in Project Zero (https://github.com/shifulegend/project-zero) — it handles AVX-512F, VNNI, VBMI, AVX2, and scalar paths in the same binary. The scalar fallback is verified to produce bit-identical output to the SIMD paths, which is the only way to catch regressions like this. The AVX2-only compile-time gating without a scalar fallback is a silent correctness trap on older hardware.

  5. amitpatole commented on Jul 7, 2026

    @amitpatole
    AMIT SAMSON PATOLE (amitpatole)
    on Jul 7, 2026

    I opened #580 to fix this.

    Root cause: the four ggml_vec_dot_i2_i8_s_* kernels in src/ggml-bitnet-mad.cpp are gated #if defined(__AVX2__) … #elif defined(__ARM_NEON) with no #else. On x86 without AVX2 neither branch compiles, so each kernel has an empty body and never writes its output s — the uninitialized dot products feed softmax and generation collapses to a constant token. That also explains why your hand-injected scalar loop still produced garbage: the four dispatched kernels were the ones left empty, not the path you patched.

    Fix: a portable scalar #else fallback for all four kernels via a shared ggml_i2_s_block_dot() that reproduces the AVX2 ternary unpack bit-for-bit (2-bit codes at bit offsets 6/4/2/0 paired with activation sub-blocks 0/1/2/3).

    Verified on a build with -DGGML_AVX2=OFF (AVX = 1 | AVX2 = 0 | MATMUL_INT8 = 0, matching your environment), running BitNet-b1.58-2B-4T i2_s:

    • before: The capital of France is!!!!!!!!!!!!!…
    • after: The capital of France is Paris. Paris is a city that is known for its rich history, culture, and architecture…

    A regression test (tests/test-i2s-scalar-fallback.cpp) cross-checks the scalar fallback against the exact AVX2 kernels over 7163 random cases — bit-exact. Since you offered to test on your Ivy Bridge hardware, a confirmation there would be very welcome.

  6. puddintaim1975-dev commented on Aug 26, 2026

    @puddintaim1975-dev
    puddintaim1975-dev
    on Aug 26, 2026

    Verified on Ivy Bridge (AVX-only, no AVX2)

    Confirmed that the diagnosis in this issue is correct: the four ggml_vec_dot_i2_i8_s_* kernels in src/ggml-bitnet-mad.cpp have empty bodies on AVX-only builds (neither __AVX2__ nor __ARM_NEON defined), causing SIGILL or garbage output.

    I independently implemented scalar fallbacks for these kernels and verified coherent output on:

    • CPU: Intel Xeon E5-2690 v2 (Ivy Bridge, AVX only)
    • Build: Clang 18.1.3, -DBITNET_ARM_TL1=OFF -DBITNET_X86_TL2=OFF
    • Model: BitNet-b1.58-2B-4T I2_S

    Results after fix:

    • "The capital of France is" → "Paris. Paris is a city that is known for its rich history, culture, and architecture..." ✓
    • "What is 2+2?" → "2+2 equals 4." ✓

    Note on signed vs unsigned 2-bit values

    My quants.c scalar fallback uses unsigned 2-bit values (0, 1, 2) rather than signed ternary (-1, 0, +1). The GEMM post-processing formula (int_dot - act_sum) * (weight_scale / act_scale) expects unsigned values — act_sum = sum(int8_activation_values) compensates for the unsigned encoding.

    PR #580 uses (val >> shift) & 3) - 1 (signed ternary), while my implementation uses (val >> shift) & 3 (unsigned). Both approaches work in their respective code paths, but the unsigned convention matches the AVX2 GEMM kernel semantics.

    Separate codepaths need separate fixes

    The ggml_vec_dot_i2_i8_s_1x1 in quants.c (ggml codepath) and the kernels in ggml-bitnet-mad.cpp (BitNet MAD codepath) are separate implementations that both need fixes. PR #580 addresses the BitNet MAD path; the quants.c GEMV path also needs its scalar fallback fixed.

    I2_S GEMM bug

    Even after fixing the scalar fallbacks, the I2_S GEMM fast path (ggml_gemm_i2_i8_s in ggml-cpu.c:1492) produces wrong results for multi-token prompts. See issue #617 for details.

    Also required

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions