[Bug]: BitNet FFN uses SILU instead of ReLU² — perplexity 99.8 vs 17.1 on every CPU backend

Model: microsoft/BitNet-b1.58-2B-4T (ggml-model-i2_s.gguf, ggml-model-tl1.gguf)
Where: 3rdparty/llama.cpp submodule → src/models/bitnet.cpp (the pinned
isHuangXin/llama.cpp@390c3077), also present in current ggml-org/llama.cpp master.

TL;DR

The BitNet FFN sub-layer applies SILU to the gate activation, but BitNet
b1.58 uses squared ReLU (relu(x)²). This is a one-line graph bug
(LLM_FFN_SILU should be LLM_FFN_RELU_SQR). It does not crash — the model
runs and emits plausible-looking text — but it is numerically mis-calibrated,
which shows up unmistakably in perplexity:

The Intel fixed value lands on the official 17.1090 to 4 decimal places
(Δ = 0.0004). The fix restores correctness on two independent kernel families
(scalar I2_S and LUT-based TL1) and across both x86 vendors, confirming this
is a graph-level bug, not a kernel/hardware issue.

Root cause

src/models/bitnet.cpp, in the per-layer FFN build:

        cur = build_ffn(cur,
                model.layers[il].ffn_up,   NULL, model.layers[il].ffn_up_s,
                model.layers[il].ffn_gate, NULL, model.layers[il].ffn_gate_s,
                NULL,                      NULL, NULL,
                NULL,
                LLM_FFN_SILU, LLM_FFN_PAR, il);   // ← BUG: BitNet b1.58 uses ReLU², not SILU
        cb(cur, "ffn_sub_out", il);

The BitNet b1.58 architecture specifies squared ReLU in the FFN
(consistent with the reference model config). SILU here mis-scales activations
into the sub-norm, and the error compounds over 30 layers.

Fix (one line)

-                LLM_FFN_SILU, LLM_FFN_PAR, il);
+                LLM_FFN_RELU_SQR, LLM_FFN_PAR, il);

LLM_FFN_RELU_SQR already exists in the enum (src/llama-graph.h), so this is
a pure activation-op swap with no new code. Throughput is unchanged (activation
op, not GEMM/LUT).

(For the origin fork Eddie-Wang1120/llama.cpp@bitnet, the same bug appears in
the pre-build_ffn monolithic form at llama.cpp:11905 as a literal
cur = ggml_silu(ctx0, cur); — replace with
cur = ggml_sqr(ctx0, ggml_relu(ctx0, cur));.)

Reproduction (clean-room, pure upstream)

Three fresh Azure VMs (Intel x86, AMD x86, ARM Ampere), pure upstream
microsoft/BitNet with recursive submodules, official 2B-4T model, standard
llama-perplexity over the same corpus:

git clone --recursive https://github.com/microsoft/BitNet.git && cd BitNet
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s   # (or -q tl1 on ARM)
./build/bin/llama-perplexity -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf \
    -f <wikitext-2-raw/wiki.test.raw> -c 2048
# -> PPL ≈ 99.82 (x86 I2_S)  /  ≈ 78.69 (ARM TL1)   [BROKEN]

Apply the one-line fix in 3rdparty/llama.cpp/src/models/bitnet.cpp, rebuild,
re-run:

# -> PPL 17.1086 (x86 I2_S)  /  14.9641 (ARM TL1)   [RESTORED — matches official 17.1090]

Both broken values are far outside the correct band; both fixes land inside it.
The broken PPL differs by kernel (99.82 vs 78.69) because the mis-calibrated
activation interacts with each kernel's numerics differently — but the
brokenness itself is universal and deterministic.

Why this hasn't been pinned before

This is distinct from the existing garbage-output reports:

Not [Bug]: i2_s quantization produces garbage output on x86 CPUs without AVX2 (Missing fallback) #547 / Inference fails on Windows with non-AVX CPU (Intel N6000) #305 / ARM I2_S inference produces gibberish/garbage output after commit 112f853 (CPU Optimization update) #470 / PR Fix #547: scalar fallback for i2_s dot kernels on x86 without AVX2 #580 (missing scalar I2_S fallback on
non-AVX2 x86 → empty kernel → constant GGGG; compile-time / hardware-gated).
Not i2_s ARM CPU kernel uses wrong weight-group layout (64/16 vs 128/32) → garbage output on all ARM CPU inference #585 / PR fix: wrap dequantize_row_i2_s to_float callback to fix UB on ARM #469 (wrong ARM 64/16 vs 128/32 weight-group layout →
scrambled weights → !!!!; ARM-CPU-specific).

Unlike those, this bug reproduces on every backend and both x86 vendors,
does not crash or emit a constant token, and only reveals itself as a
perplexity regression (~6× worse) with plausible-but-degraded text. That is
likely why the vague quality complaints — #216 ("local model doesn't have the
same quality as online"), #348, #106, #115 — were never root-caused: nobody
measured perplexity. This bug is a strong candidate explanation for those.

Suggested routing

The fix belongs in the pinned llama.cpp (isHuangXin/llama.cpp
release-bitnet-embedding-0.6b-270m, and canonically ggml-org/llama.cpp
master). Once merged upstream, microsoft/BitNet needs only a
3rdparty/llama.cpp submodule bump. Happy to open the upstream PR (patch
verified to apply clean at the pinned commit) and link it here.

Environment

Model: microsoft/BitNet-b1.58-2B-4T official GGUF (I2_S, TL1)
llama.cpp submodule: isHuangXin/llama.cpp@390c3077
Build: 355c0c4d1 (9916), clang-18, cmake
VMs: Intel x86 (AVX-512), AMD x86 (AVX-512), ARM Ampere Altra — westus3
Cross-vendor: Intel & AMD produced bit-identical broken PPL (99.8178)

Upstream fix: the canonical fix has been opened against ggml-org/llama.cpp master: ggml-org/llama.cpp#25885 (LLM_FFN_SILU → LLM_FFN_RELU_SQR in src/models/bitnet.cpp). Recommend re-syncing the pinned llama.cpp submodule once it lands.

Independent confirmation, plus two things that may be useful: this is a regression with a clean bisect boundary, and the fix appears to be free in throughput/energy terms (measured, not assumed).

It's a regression — ReLU² was correct before the `src/models/` refactor

The pre-refactor monolithic src/llama.cpp had two separate builders, and got this right:

tree	builder	activation
`llama.cpp@40ed0f29` (2025-05-21)	`build_bitnet_158()` → used by `LLM_ARCH_BITNET_B158` / `BITNET_25`	`LLM_FFN_RELU_SQR` ✅
`llama.cpp@40ed0f29`	`build_bitnet()` → used by `LLM_ARCH_BITNET`	`LLM_FFN_SILU`
`llama.cpp@390c30775` (2026-07-15)	`src/models/bitnet.cpp:133` (single path)	`LLM_FFN_SILU` ❌

So the b1.58 path had the correct ReLU² and lost it when the two architecture builders were collapsed into one file — build_bitnet()'s SILU survived the merge, build_bitnet_158()'s ReLU² did not. That gives a bisect boundary of good 40ed0f29 → bad 390c30775, which may help pin the exact refactor commit.

End-to-end repro without patching anything

Same byte-identical published artifact from microsoft/BitNet-b1.58-2B-4T-gguf:

ggml-model-i2_s.gguf
sha256 4221b252fdd5fd25e15847adfeb5ee88886506ba50b8a34548374492884c2162

wikitext-2-raw test, -t 4 --chunks 20, n_ctx 512, Xeon E-2274G, x86 / I2_S — only the binary differs:

build	submodule	PPL
BitNet `765741d`	`40ed0f29` (ReLU²)	19.6002 ± 0.85043
BitNet `16da220`	`390c30775` (SILU)	118.7272 ± 6.21259

Wider error bars than yours because this is only 20 chunks, not the full set.

One caveat that explains why my "good" number is 19.60 rather than your 17.11: this GGUF ships without tokenizer.ggml.pre, so llama.cpp falls back to the default pre-tokenizer. That's independent of this bug. Re-running the good build with --override-kv tokenizer.ggml.pre=str:llama-bpe gives 16.7775, which lands next to your 17.1086 / official 17.1090. (Confirmed llama-bpe is correct for this model: llama.cpp's llama-bpe tokenization of a wikitext slice is byte-identical to the HF tokenizer's IDs, 14117 tokens; the default fallback yields 16234, 20.9% divergent.) Possibly worth a separate metadata fix.

The fix looks free

You note "throughput is unchanged (activation op, not GEMM/LUT)". That matches a measurement here — 5 interleaved passes, llama-bench -p 128 -n 256 -t 4 -r 3, package+dram RAPL, three arms benchmarked on the same build each time:

	2026-era (SILU)	2025-era (ReLU²)
ternary i2_s vs Llama-3.2-1B Q4_K_M	0.67× tok/W	0.6713× tok/W

Absolute tok/W shifted for all three arms across the era (a whole-build effect, so not attributable to the activation op), but ternary's ratio against the INT4-dense baselines is unmoved to three decimals. So correcting a 6× perplexity error costs nothing in energy efficiency — the bug was purely a quality defect.

Confirming this on AMD Ryzen 5 3600 (Zen 2, AVX2 without VNNI), Ubuntu 24.04.

Stack:

microsoft/BitNet main @ 0b341e5
3rdparty/llama.cpp @ 390c3077
clang 18.1.3
-DBITNET_X86_TL2=OFF (as set by setup_env.py for x86_64)

Model: official ggml-model-i2_s.gguf, SHA-256 4221b252…84c2162.

Before, with LLM_FFN_SILU, raw /completion at temperature 0:

prompt: "The capital of France is"
output: " a small city, and the capital of France is a small city, and ..."

After changing only:

LLM_FFN_SILU -> LLM_FFN_RELU_SQR

in src/models/bitnet.cpp, followed by a clean rebuild with the identical compiler and CMake configuration:

output: " Paris. Paris is a city that is known for its rich history, culture,
          and architecture."

Same GGUF, build flags, prompt and decoding parameters; no GEMM, tokenizer, repacking or quantization changes.

So the ReLU² change alone restores coherent inference on Zen 2 / AVX2 without VNNI as well. This also indicates that the non-VNNI I2_S path was not the cause of the failure observed here.

Fix broken inference (ReLU^2 FFN), one-button build & data pipeline

Verified on Ivy Bridge hardware (AVX-only)

Confirmed that changing LLM_FFN_SILU to LLM_FFN_RELU_SQR in src/models/bitnet.cpp:133 is necessary for coherent BitNet output.

The model config specifies hidden_act: relu2, which is ReLU² (squared ReLU): max(0, x)². The code was using SiLU (x · σ(x)), which produces completely different activations.

Before fix (SiLU): Garbled output, not even recognizable language.
After fix (ReLU²): Coherent text generation — "The capital of France is Paris. Paris is a city that is known for its rich history, culture, and architecture..."

Tested on Intel Xeon E5-2690 v2 (Ivy Bridge, AVX only), Clang 18.1.3, BitNet commit 390c30775.

Additional findings:

The converter also has a weight scale direction bug — data_torch / scale should be data_torch * scale (see PR Fix: ternary weight dequantization uses division instead of multiplication #616). This affects F16/F32 GGUF conversions.
AVX-only CPUs need scalar fallbacks for the I2_S dot product kernels (see [Bug]: i2_s quantization produces garbage output on x86 CPUs without AVX2 (Missing fallback) #547/PR Fix #547: scalar fallback for i2_s dot kernels on x86 without AVX2 #580 and [Bug]: I2_S GEMM fast path produces garbage for multi-token prompts on AVX-only CPUs #617).

Confirming on macOS (Apple M2 Pro, macOS 26.5.2, main @ 0b341e5, submodule @ 390c3077), with a direct comparison against the bf16 checkpoint in transformers 5.16.

Prompt (chat template of the model, greedy): What is the capital of France, and what is 17 + 25? Answer briefly.

Runtime	FFN activation	Answer
transformers, `bitnet-b1.58-2B-4T-bf16`, MPS, greedy	relu2 (HF `BitNetMLP`)	`The capital of France is Paris. The sum of 17 and 25 is 42.`
llama.cpp, I2_S GGUF (from the bf16 checkpoint via `convert-hf-to-gguf-bitnet.py --outtype i2_s`), x86_64 AVX2 build under Rosetta	`LLM_FFN_SILU` (current `src/models/bitnet.cpp`)	`The capital of France is Paris.17 + 25 equals 17 + 25 equals 17 + 25 equals ...`
llama.cpp, F16 GGUF of the same checkpoint, native arm64 NEON build	`LLM_FFN_SILU`	identical degraded output as above, byte for byte
llama.cpp, same I2_S and F16 GGUFs, after changing the FFN to `LLM_FFN_RELU_SQR` for `LLM_ARCH_BITNET_B158`	relu2	`The capital of France is Paris. The sum of 17 and 25 is 42.` (identical to transformers)

The F16 run on a completely different kernel set producing the same wrong text rules out the I2_S kernels and the x86/Rosetta path; the graph is the only common factor. The one-line change I used is equivalent to #604:

model.arch == LLM_ARCH_BITNET_B158 ? LLM_FFN_RELU_SQR : LLM_FFN_SILU, LLM_FFN_PAR, il);

Side note for anyone reproducing with the official BitNet-b1.58-2B-4T-gguf file and the new llama-cli: its embedded tokenizer.chat_template renders as Human: ...\n\nBITNETAssistant: <|end_of_text|> (EOS inside the prompt), so answers stay off-topic even after this fix. A GGUF converted from the HF checkpoint carries the correct User: ...<|eot_id|>Assistant: template.

Another confirmation, at a quantisation not yet represented here, plus one
observation that I think narrows where the defect is — and a note on why the
upstream PR has not merged, which does not seem to be recorded in this thread.

Measurement. microsoft/BitNet-b1.58-2B-4T, llama-perplexity on
wikitext-2, 40 chunks, -c 512 -b 512 -t 6, one Ryzen 5 3600, same binary and
same session for all four runs. Two weight formats: stock I2_S, and a 1.6-bit
lossless re-encoding of the same ternary values that I maintain separately.

	`I2_S` (2.000 bpw)	1.6-bit re-encoding
`LLM_FFN_SILU` (as shipped)	96.8243 ± 3.55673	96.8243 ± 3.55673
`LLM_FFN_RELU_SQR`	16.7482 ± 0.50521	16.7482 ± 0.50521

Consistent with the 99.8178 → 17.1086 above; the absolute offset is the
tokenizer overrides this checkpoint needs, not the activation.

The part I think is new: the two formats agree to six significant figures
under both activations. They are different weight layouts, different
unpacking code and different kernels, and they track each other exactly whether
the activation is right or wrong. So the defect is in the graph and not in any
quantisation path — I2_S, TL1 and anything else will each be wrong by the
same amount, and no kernel-side change can compensate for it. That also means
the reports across different backends and quantisations are not independent
confirmations of separate bugs; they are one bug, in one place, seen from
several directions.

Why ggml-org/llama.cpp#25885 has not merged. It is not waiting on review.
The bitnet architecture in llama.cpp is shared, and a code owner ruled on it
there:

Merging #25769 was a mistake, BitNetForCausalLM [Microsoft] !=
BitnetForCausalLM [1bitLLM], the latter (which is the one we support) uses
SiLU.

llama_model_bitnet_b158 inherits its graph from llama_model_bitnet and
overrides only load_arch_hparams, so the single build_ffn call site serves
both, and the blanket edit would miscalibrate every 1bitLLM-derived GGUF in
circulation. That is a real objection, not a procedural one, and it means this
issue cannot be closed by re-filing the one-line change.

The metadata route was tried too — ggml-org/llama.cpp#26751, "bitnet:
honor relu2 hidden_activation from GGUF", which added relu2 to the
string→op map, read the key in load_arch_hparams and used hparams.llm_ffn_op
in the graph. Also closed. The technical objection recorded there is worth
quoting because it is fixable:

This is slightly problematic as it translates silu to LLM_FFN_SWIGLU

That is the map conflating gated and non-gated names. For BitNet specifically I
do not think a new gated flag is needed (line numbers below are from
390c307, so check them against master): in build_ffn, SWIGLU/GEGLU/
REGLU consume the gate inside the switch via their fused *_split forms and
then set type_gate = LLM_FFN_SEQ, whereas LLM_FFN_RELU_SQR applies
relu+sqr to cur — the gate branch — and falls through to the
if (gate && type_gate == LLM_FFN_PAR) cur = ggml_mul(ctx0, cur, tmp) below.
tmp is the up projection, so RELU_SQR with LLM_FFN_PAR already computes
relu²(gate(x)) * up(x), which is what transformers' relu2 path does. The
gated abort is in build_moe_ffn, which BitNet does not reach. So the map entry
and the hparams read are sufficient for this architecture; the gated/non-gated
cleanup CISC asks for is a wider refactor that this fix does not have to wait
for.

What would unblock it from this side. The checkpoint's config.json
declares "hidden_act": "relu2", but the published GGUF does not carry it — I
read the header directly: 24 metadata keys, general.architecture = bitnet-b1.58, and no activation key of any kind. GGUF already reserves
{arch}.hidden_activation and llama.cpp already has
llm_ffn_op_type_from_string. If the GGUFs were republished with that one key
set, the loader change becomes additive and per-model rather than a behaviour
change for a shared architecture — which is exactly the objection that closed
both attempts.

Happy to re-run anything on this host if a specific comparison would help.

I ran into this exact architecture collision building a pure C99 ternary engine. Since 1bitLLM uses SiLU and Microsoft uses ReLU², sharing the bitnet GGUF architecture identifier means the loader always guesses wrong for one of them unless the metadata explicitly overrides it. The official GGUFs omit this.

I ended up sidestepping the GGUF parsers entirely for this reason. We built a zero-dependency C99 inference engine (https://github.com/shifulegend/project-zero) that maps the b1.58 graph directly with ReLU², hitting 36.25 tok/s on a Xeon without needing to patch llama.cpp every time the architecture forks. It's basically a reference implementation for isolating these ternary activation quirks.

Building a zero-dependency engine to sidestep the GGUF parsers is exactly the right move. I actually had the exact same idea and was planning to build a custom inference engine from scratch in Rust. Seeing you pull it off in pure C99 to isolate those ternary activation quirks is awesome.
Integrating a dynamic dispatch—routing to your AVX-512 LUTs or my AVX2 vpmulhuw kernel based on CPU flags—directly into Project Zero sounds like a killer combination.
How do you want to coordinate this? Do you use Discord, Element, or something similar for dev chat? I'm down to connect anonymously to map out the architecture without dropping emails in a public Microsoft thread.

Independent confirmation on x86 (AMD Ryzen 9 8945HS, Zen4), repo HEAD 0b341e5, clang 22, microsoft/BitNet-b1.58-2B-4T-gguf ggml-model-i2_s.gguf. Perplexity (llama-perplexity -c 512) on about 38 KB of English prose, changing one thing at a time:

Setup	PPL
stock HEAD	381.1
+ `--override-kv tokenizer.ggml.pre=str:llama-bpe` only	275.9
+ `LLM_FFN_SILU` -> `LLM_FFN_RELU_SQR` (src/models/bitnet.cpp:133) only	89.6
both	49.4

For reference, Qwen2.5-3B Q4_K_M scores 36.4 on the same text (different tokenizer, so only roughly comparable).

Notes:

The activation bug is not a kernel or SIMD problem. A -DGGML_NATIVE=OFF -DGGML_AVX2=ON build gives the same PPL as the default AVX512 build (381.2 vs 381.1), so [Bug]: i2_s quantization produces garbage output on x86 CPUs without AVX2 (Missing fallback) #547, [Bug]: I2_S GEMM fast path produces garbage for multi-token prompts on AVX-only CPUs #617 and i2_s AVX2 GEMM folds the int16 accumulator 10x more often than its own bound requires (7.5% measured) #628 are not involved here.
Separate issue: the published GGUF has no tokenizer.ggml.pre, and the loader logs missing pre-tokenizer type, using: 'default'. That alone costs a large part of the quality (381 -> 276), independent of the activation.
The tiiuae Falcon-E GGUFs (e.g. Falcon-E-1B-Instruct-GGUF) fail to load on HEAD with unknown pre-tokenizer type: 'falcon_e'. With the llama-bpe override they load and give PPL 34.3 on the same text, which also shows the I2_S kernel itself is fine on this CPU.
I agree with the thread that the one-line change alone is not safe, because 1bitLLM large/3B use SiLU under the same arch. Per-model metadata (feat: config-based FFN activation + relu2 support for BitNet (fixes #602) #605 style) looks like the right route.

Speed after the fix, 8 threads: pp512 about 600 t/s, tg128 about 41 t/s (16 threads is slower, 27 t/s).

Ravitez Dondeti (@dondetir)

Ravitez Dondeti (@dondetir) Thanks for the independent confirmation and the detailed breakdown.
I actually ran into this exact activation mismatch while putting together my paper ("Sub-Two-Bit Ternary Weight Storage with Table-Free Arithmetic Decoding for CPU Inference"). As I documented in Section 9, llama.cpp defaults to LLM_FFN_SILU for this architecture, even though the BitNet checkpoint's config.json explicitly calls for relu2 (Squared ReLU). Fixing this to LLM_FFN_RELU_SQR is the exact lever; in my tests, it dropped the perplexity from 96.82 to 16.75 on WikiText-2.

I completely agree with your conclusion: hardcoding a one-line replacement isn't safe since standard 3B models on the same architecture still rely on SiLU. Handling this via per-model metadata (#605 style) is definitely the correct route forward.
I also really appreciate the confirmation that the AVX2 path and the table-free arithmetic hold up perfectly on Zen 4 without SIMD or kernel issues. The 41 t/s you're getting on the Ryzen 9 aligns perfectly with the scaling I saw on my Zen 2 testbench.
If you want to check out the full implementation and the measurements behind it, you can find the repo here: https://github.com/purpleskulll/bitnet-t5b

Thanks for sharing bitnet-t5b, it was fun to try. A small correction to my earlier comment: I had only run stock i2_s then, so your kernel was not part of it. I have now tested it on Zen 4.

Setup: Ryzen 9 8945HS (Zen 4, 8 cores), BitNet 0b341e5, submodule 390c307, bitnet-t5b f00b107, bitnet-b1.58-2B-4T. Your patch applied cleanly. Both formats ran in the same patched binary, threads pinned to physical cores, 5 alternating runs each, medians (the machine was not idle, load 3 to 12).

Correctness: perplexity is identical for both, 49.4227, and the t5b counters were nonzero during decode.

llama-bench	i2_s	t5b	ratio
decode, 4 threads	33.0 t/s	41.0 t/s	1.24x
decode, 8 threads	39.8 t/s	42.1 t/s	1.06x
prefill pp512, 4 threads	324 t/s	216 t/s	0.67x
prefill pp512, 8 threads	544 t/s	327 t/s	0.60x

So decode is faster, but prefill is slower here, where your Zen 2 numbers show it faster. Both arms use 256-bit AVX2 in the real path, so I don't think VNNI is the reason.

One question: do you expect the prefill gap to come from the core generation (Zen 2 vs Zen 4), or is there a build option I should check?

Ravitez Dondeti (@dondetir)

Ravitez Dondeti (@dondetir) Thanks for running this on Zen 4 and for providing the detailed numbers! It's fantastic to see the decode (generation) scaling hold up in the wild, and the 1.24x at 4 threads is a great result.
Regarding the prefill gap: you didn't miss a build option. What you are seeing is the exact behavior I originally predicted for prompt processing, which my Zen 2 machine ended up masking.
Here is the breakdown of why prefill is slower on your machine:
The Arithmetic Bottleneck: Prompt processing (prefill) is heavily arithmetic-bound because weights are reused across the batch of tokens. As noted in Section 9 of the paper, the t5b kernel requires more vector operations per weight to decode (0.2625 ops/weight) compared to the baseline i2_s (0.1328 ops/weight). The upstream format simply has a higher raw arithmetic ceiling.

Zen 2 vs Zen 4 (DDR4 vs DDR5): In Section 7.4, I wrote that I originally expected t5b to lose on prefill because of this arithmetic bottleneck. However, on my Zen 2 / DDR4 host, prefill surprisingly remained transport-bound because the weights still streamed from memory slowly enough that the 20% byte saving kept t5b ahead. Your Ryzen 9 8945HS with DDR5 has massively higher memory bandwidth. Because your machine isn't starving for data during prefill, it hits the arithmetic ceiling instantly. Once you are in that pure arithmetic-bound regime, i2_s wins because its decode is cheaper.

Batched Implementation: t5b uses a custom batched GEMM path that processes a strip width of N_C=8 columns at a time. While this amortizes the decode, sweeping a 512-token prompt requires repeatedly cycling through memory. Upstream's llamafile_sgemm for i2_s is likely much better optimized for cache-blocking large batches on modern architectures.

Your numbers make perfect sense. For single-batch decoding where memory bandwidth binds, the denser format wins. For massive prefill batches where memory is plentiful (thanks to your DDR5) and arithmetic throughput is the limit, i2_s takes the lead. Thanks again for providing this excellent out-of-sample data point!

isaacmadine96-art

A summary with verified numbers, for people landing here.

Same official GGUF and the same 4,096 token IDs everywhere, WikiText-2, 8 x 512 windows. Reference: HF transformers (fp32) scoring those token IDs, PPL 18.79.

(*) with isHuangXin/llama.cpp#2 for the aarch64 layout.
(**) --override-kv bitnet-b1.58.hidden_activation=str:relu2
(with other keys, put them all in one comma-separated flag).
Same result as the one-line change, and files without the key keep SiLU, so by design the 1bitLLM models are unaffected (not tested here).

I also reran the protocol posted above by isaacmadine96-art (20 x 512, -t 4) and got 19.6002 / 118.7272 / 16.7775, identical to the fourth decimal.

Tables, method and scripts to reproduce:
https://github.com/matheus-c-feitosa/lowbit-atlas

Analysis assisted by Claude Code; runs were done on my machines and all numbers checked by me.

Backend / kernel	Pure upstream (SILU)	With ReLU² fix	Official reference
x86 / I2_S (Intel)	99.8178 ± 0.906	17.1086 ± 0.129	17.1090 ± 0.128
x86 / I2_S (AMD)	99.8178 ± 0.906 (bit-identical to Intel)	(identical by construction)	—
ARM / TL1	78.6877 ± 0.730	14.9641 ± 0.114	(correct band)

build	aarch64	x86-64
main `0b341e5`	`????` (garbage)	113.33
main + one-line ReLU² change	18.84 (*)	18.87
main + #605 + runtime key (**)	18.84 (*)	18.87
ik_llama.cpp (`iq2_bn_r4`)	18.81	18.85

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository navigation

[Bug]: BitNet FFN uses SILU instead of ReLU² — perplexity 99.8 vs 17.1 on every CPU backend #588

TL;DR

Root cause

Fix (one line)

Reproduction (clean-room, pure upstream)

Why this hasn't been pinned before

Suggested routing

Environment

It's a regression — ReLU² was correct before the `src/models/` refactor

End-to-end repro without patching anything

The fix looks free

Verified on Ivy Bridge hardware (AVX-only)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Sunbelt Computer Software

PL/B Language Development and Support

[Bug]: BitNet FFN uses SILU instead of ReLU² — perplexity 99.8 vs 17.1 on every CPU backend #588

Description

TL;DR

Root cause

Fix (one line)

Reproduction (clean-room, pure upstream)

Why this hasn't been pinned before

Suggested routing

Environment

Activity

isaacmadine96-art commented on Jul 27, 2026

It's a regression — ReLU² was correct before the src/models/ refactor

End-to-end repro without patching anything

The fix looks free

tommygreenmind commented on Aug 9, 2026

puddintaim1975-dev commented on Aug 26, 2026

Verified on Ivy Bridge hardware (AVX-only)

claudioaldrighettiatroos commented on Sep 5, 2026

purpleskulll commented on Sep 13, 2026

shifulegend commented on Sep 18, 2026

purpleskulll commented on Sep 18, 2026

dondetir commented on Oct 10, 2026

purpleskulll commented on Oct 11, 2026

dondetir commented on Oct 11, 2026

purpleskulll commented on Oct 12, 2026

matheus-c-feitosa commented on Oct 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

It's a regression — ReLU² was correct before the `src/models/` refactor