feat: config-based FFN activation + relu2 support for BitNet (fixes #602) by praneshnikhar · Pull Request #605 · microsoft/BitNet · GitHub
Skip to content

feat: config-based FFN activation + relu2 support for BitNet (fixes #602) - #605

Open
Pranesh Nikhar (praneshnikhar) wants to merge 1 commit into
microsoft:mainfrom
praneshnikhar:fix/602-ffn-relu2-config
Open

Pranesh Nikhar (praneshnikhar) wants to merge 1 commit into
microsoft:mainfrom
praneshnikhar:fix/602-ffn-relu2-config

Conversation

@praneshnikhar

Copy link
Copy Markdown

BitNet b1.58 uses squared ReLU (relu²) in the FFN, but the graph builder was hardcoded to SiLU and the GGUF converters never wrote a hidden_activation key.

This PR makes the activation config-driven:

llama.cpp submodule changes

  1. src/llama-model.cpp: Add "relu2" → LLM_FFN_RELU_SQR mapping
  2. src/models/bitnet.cpp: Read <arch>.hidden_activation from GGUF, default to SiLU for backward compat, use hparams.llm_ffn_op in graph

Main repo changes

  1. utils/convert-hf-to-gguf-bitnet.py: Write hidden_act("relu2") for BitNet models
  2. utils/convert-ms-to-gguf-bitnet.py: Write hidden_act("relu2") for BitNet models

Behavior

  • New GGUF files (converted with updated converters): FFN uses ReLU² automatically
  • Existing GGUF files (no hidden_activation key): FFN uses SiLU (unchanged behavior, no regression)

Depends on: isHuangXin/llama.cpp#4

Closes #602

BitNet b1.58 architecture uses squared ReLU (relu²) in the FFN, but the
graph builder was hardcoded to SiLU and the GGUF files didn't include the
hidden_activation key. This caused ~6x perplexity regression on every backend.

Changes:
- Update HF/MS converters to write hidden_activation='relu2' for BitNet models
- Bump llama.cpp submodule to include config-based activation support
  (reads <arch>.hidden_activation, defaults to SiLU for backward compat,
   uses ReLU² when key is present)

When models are re-converted with these changes, the runtime will
automatically use the correct ReLU² activation. Existing GGUF files
without the key continue to work (behavior unchanged).

Fixes microsoft#602
Submodule PR: https://github.com/praneshnikhar/llama.cpp/pull/2
@microsoft-github-policy-service

Copy link
Copy Markdown

@praneshnikhar

Copy link
Copy Markdown
Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bitnet-b1.58-2B-4T runs FFN with SiLU instead of relu2 - wrong logits on every backend

1 participant