Repository navigation
Latent int16 overflow in ggml_vec_dot_i2_i8_s_1x1: 128 maddubs results per lane against a 32767 ceiling #629
Description
Summary
ggml_vec_dot_i2_i8_s_1x1 accumulates 128 vpmaddubsw results into one
int16 lane before folding. The worst case is 128 * 508 = 65024 against a
32767 ceiling. It is correct in practice only because signed activations
cancel, and it is latent on a default build because the sgemm path takes
the work instead.
I am filing it as hardening, not as a wrong-output bug, and the distinction is
deliberate — see Severity below.
The arithmetic
ggml/src/ggml-cpu/quants.c:
const int group32_num = nb / 32; // :1396
...
__m256i accu32 = _mm256_setzero_si256(); // :1413 (int16 lanes)
for (int j = 0; j < 32; j++) { // :1414
...
accu32 = _mm256_add_epi16(accu32, _mm256_add_epi16(xq8_0, xq8_1)); // :1439
accu32 = _mm256_add_epi16(accu32, _mm256_add_epi16(xq8_2, xq8_3)); // :1440
}
accu = _mm256_add_epi32(_mm256_madd_epi16(accu32, one16), accu); // :144532 iterations × 4 planes = 128 vpmaddubsw results per int16 lane. Each is at
most 2 * 2 * 127 = 508, so the lane can reach 65024, roughly twice the
ceiling. The name accu32 suggests 32-bit; the type is __m256i used with
_mm256_add_epi16.
What actually happens
With all activations the same sign the sum does not cancel and the lane wraps.
Measured on the real weights of BitNet-b1.58-2B-4T — 30 tensors at
K = 6912, driven against int8 activations pinned to +127 — 11,998 of
12,000 rows disagree with exact int64 arithmetic, and every error is a
multiple of 65536, the signature of a wrapped int16.
With ordinary int8 activations, 0 of 12,000 rows wrap. That is why it has
never been seen.
Severity: low, and I want to be accurate about why
- On a build where
llamafile_sgemmhandles these shapes this function is not
called at all. The kernel's own counters on my build read
calls=0 sgemm=52068. - It becomes reachable when that path is disabled, which is a supported switch
rather than an exotic one. - Even then it needs same-signed activations across a whole group of 32 blocks.
So: a latent overflow in a fallback, not a correctness bug users are hitting.
I would not have filed it on its own; I am filing it because it is the same
missing bound-derivation as the fold issue I opened alongside this one (#628), in the
same file family, and a fix for one is naturally a fix for both.
Not a duplicate of the ARM reports
#411 and the audit in #525 report an int16 overflow, but it is the NEON
vmlal_s8 accumulation in src/ggml-bitnet-mad.cpp, which does produce visible
garbage. This is a different function, a different instruction set and a
different failure mode — this one is invisible precisely because it cancels.
Suggested fix
Fold every 8 groups rather than every 32, or accumulate in int32 directly.
Upstream ggml's equivalent kernels derive the bound explicitly and stay inside
it — TQ2_0 in arch/x86/quants.c carries // 16-bit sums, because 256*127 still fits — which is the pattern worth copying here.
Reproducible from https://github.com/purpleskulll/bitnet-t5b.
Activity
purpleskulll commentedon Sep 13, 2026
Correction to my own arithmetic above, before anyone spends time on it.
I wrote that each${-1,0,+1}$ as 2-bit codes
vpmaddubswresult is "at most2 * 2 * 127 = 508". That iswhat the encoder emits — i2_s packs ternary
0, 1, 2 and never uses 3 — but it is not the bound the kernel carries. Its
own mask is
_mm256_set1_epi8(0x03)(quants.c:1400), so the operand range itguarantees is 0..3:
Either figure is at least twice the 32767 ceiling, so the conclusion is
unchanged and the 11,998-of-12,000 measurement stands — it was taken on the real
encoder output, hence on the 508 basis.
I am correcting it because it was inconsistent with #628, which derives the
same kernel family's bound from the mask and gets 762. Two issues from the same
reporter should not quote two different operand ranges for the same
representation, and the mask is the right basis for a bound.
Apologies for the noise.
shifulegend commentedon Sep 15, 2026
Just ran into this. When building Project Zero (a pure C99/CPU engine that runs BitNet), we basically bypassed the vpmaddubsw accumulator overflow entirely by using a 5-trit lookup table approach in our experimental AVX-512BW branch to just precalculate the dot products. That avoids the deep int16 accumulation chain in the first place by packing the ternary weights. Not a drop-in fix for ggml obviously, but the LUT path sidesteps the precision ceiling completely. FWIW, folding every 8 groups like you suggested sounds like the right call for the current layout.
purpleskulll commentedon Sep 15, 2026
Thanks for confirming it independently — good to know it is not just my machine.
Your LUT route interests me, because I arrived at the same five-trit grouping from the other direction. I decode the group arithmetically rather than looking it up: four independent multiply-high operations recover the digit planes. Different mechanism, same effect on the problem at hand — the deep int16 accumulation never forms in the first place.
I would like to see how yours behaves. Is Project Zero public anywhere? And do you have numbers from the AVX-512BW branch? Mine come from one Zen 2 host and a two-core Xeon, which is too little to say much about wider vector machines, so anything you have would tell me more than my own data does.
There may also be something worth combining. The bound I use for the fold interval comes from enumerating all 243 byte values exhaustively, so it is a small standalone check rather than anything specific to my kernel — it should apply to a table-based path just as well, and it is in the repo if it is of any use: https://github.com/purpleskulll/bitnet-t5b. And the choice between the two routes looks like a property of the target rather than of the format: where AVX-512BW is there, your table has room the arithmetic path does not; where it is not, the arithmetic decode is what is left. Something that picks per target does not sound far-fetched.
Agreed on the fold interval for the current layout.
shifulegend commentedon Sep 17, 2026
Yeah, it's public. Repo is here: https://github.com/shifulegend/project-zero
For the AVX-512BW branch specifically, on a Xeon 8259CL (PGO+LTO built), we're seeing 36.25 tok/s peak and 34.75 avg for BitNet-b1.58-2B-4T. That's roughly 1.83x what we measured on bitnet.cpp on the exact same hardware, hitting about 95% of the DRAM bandwidth ceiling on that box.
Really interesting that you went the arithmetic decode route with multiply-high. I actually think you're spot on that it's a target-dependent choice - LUTs win hard when you have the register space in AVX-512, but arithmetic decode scales down better to AVX2/NEON. I'll take a look at your bitnet-t5b repo. We could probably just dynamically dispatch between the two approaches based on CPU flags. Makes a lot of sense.
purpleskulll commentedon Sep 17, 2026
Awesome insights on the AVX-512 register space. You're completely right—LUTs are a beast when you have the room for them, while the arithmetic decode via vpmulhuw keeps things fast and tight for AVX2/NEON where we lack that space.
I'm 100% down for the dynamic dispatch approach. Merging both based on CPU flags makes absolute sense and would be the ultimate drop-in solution.
Let's connect and figure out how to wire this together