test: add model-free I2_S numerical regressions by shashankswe2020-ux · Pull Request #643 · microsoft/BitNet · GitHub
Skip to content

test: add model-free I2_S numerical regressions - #643

Open
Shashank Mishra (shashankswe2020-ux) wants to merge 1 commit into
microsoft:mainfrom
shashankswe2020-ux:test/i2s-kernel-regressions
Open

Shashank Mishra (shashankswe2020-ux) wants to merge 1 commit into
microsoft:mainfrom
shashankswe2020-ux:test/i2s-kernel-regressions

Conversation

@shashankswe2020-ux

Copy link
Copy Markdown

Summary

Add opt-in, model-free I2_S numerical regression tests derived from debugging real
BitNet chat inference on Apple Silicon. This is a test-only draft, not another
competing kernel fix.

  • Link the actual ggml-cpu implementation.
  • Independently pack 128-element / 32-byte blocks and compare with integer
    reference dot products.
  • Cover unsigned-code semantics, activation/output strides, output canaries,
    non-square GEMM and row tails, saturated inputs, and zero-activation scaling.
  • Add 228 cases in three CTest groups; no model downloads, servers, new packages,
    or network access during testing.
  • Keep normal builds unchanged with BITNET_BUILD_TESTS=OFF by default.

Why draft

The tests intentionally fail on current main. There are no WILL_FAIL, disabled
assertions, or inverted success results. Please treat this as reproducible evidence
and regression coverage to accompany the upstream fixes, not as ready-to-merge
green CI.

Related: #585, #600,
#633, #642.
This PR does not close those reports.

Observed results

Apple M4 Max, arm64 macOS, Apple clang 21.0.0, CMake 3.31.10, static CPU build,
GGML_NATIVE=OFF, GGML_CPU_ARM_ARCH=armv8-a, BLAS/Metal/OpenMP/TL1/TL2 disabled.

Source Dot failures GEMM failures Zero-scale failures
Main 0b341e582afbf9e1011f24744b554c96a3477eb5 112/112 112/112 4/4
#642 at 9dcd80d20a5125b61eeea70472db9553dba378d9, with its supplied submodule patch applied 84/112 84/112 4/4
Local diagnostic scalar correction, not submitted here 0/112 0/112 0/4

Both upstream revisions pin llama.cpp
390c307752ab78fd8189f359d6954c9ba1be74af.
The comparison with #642 includes both its MAD source changes and
its patches/0001-fix-arm64-2b-inference.patch applied inside that submodule.
It is not an assessment of its end-to-end chat behavior or other CPU configurations.

The first mixed dot case returns 461 on main and -1039 with
#642, versus an independently calculated -1090.
Zero activations return scale 0, which is unsuitable for the caller's division
by the activation scale.

On current main's ARM build, the active dot entry point is the scalar fallback in
ggml/src/ggml-cpu/quants.c, rather than the top-level MAD source. Linking the
actual backend is important here.

Verification

  • Test target builds; the test source passes -Wall -Wextra -Werror.
  • All three CTest groups run; failures and exit status are preserved on main.
  • Final main fixtures also reproduce the same numerical failures with ASan/UBSan
    and no sanitizer diagnostics.
  • The local diagnostic control fixes scalar packing, unsigned codes,
    activation-column addressing, output stride, and zero scaling. All 228 cases
    pass under ASan/UBSan. This control is not a performance-qualified fix and is
    intentionally excluded from this test-only PR.
  • Invalid test selector reports usage and exits 2.
  • Whitespace checks pass; no production kernel, submodule pointer, generated
    header, model artifact, or machine-local benchmark data is included.

See tests/README.md for complete reproduction commands,
contracts, and limitations. This is numerical correctness coverage, not a model
intelligence benchmark or a throughput claim.

Add opt-in tests for packed layout, unsigned dots, activation/output strides and zero scaling. Preserve current upstream failures as draft regression evidence rather than bundling a competing kernel fix.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@shashankswe2020-ux

Shashank Mishra (shashankswe2020-ux) commented Oct 12, 2026 •

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree company="Microsoft"

@shashankswe2020-ux
Shashank Mishra (shashankswe2020-ux) marked this pull request as ready for review October 12, 2026 08:55
@shashankswe2020-ux

Copy link
Copy Markdown
Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant