Environment

Where are you using Unsloth?

• Unsloth desktop application
• Unsloth web UI (unsloth studio)
• Unsloth CLI
• Python package or notebook
• Colab or Kaggle

Operating system and version: Kaggle notebook (Linux), Python 3.13

GPU model(s) and accelerator backend: 2x Tesla T4, NVIDIA CUDA 12.8

Versions: unsloth 2026.10.7 (git), unsloth_zoo (git), transformers 5.2.0, torch 2.11.0+cu128, trl 0.22.2

What happened?

Steps to reproduce:

1. FastDecisionModel.from_pretrained("google/gemma-3-270m", decision_head="clef", max_seq_length=4096, load_in_4bit=False)
2. FastDecisionModel.get_peft_model(model, r=64, lora_alpha=64, target_modules="all-linear")
3. Train with DecisionTrainer on ~40k decision rows (lr 1e-4, head lr 3e-4, batch 4 x accum 4, 1 epoch)

Expected behavior: loss decreases and accuracy rises, as it does for Qwen3.5-0.8B, LFM2-700M, LFM2.5-350M and MiniCPM5-1B with the identical script.

Actual behavior: loss stays flat for the whole epoch and accuracy is at chance (BANKING77 1.4%, CLINC150 0.4–0.6%).

Model and operation involved: google/gemma-3-270m and google/gemma-3-270m-it, FastDecisionModel + Clef decision head, LoRA training.

Details

Training a decision model on google/gemma-3-270m (and -it) with FastDecisionModel + decision_head="clef" gives a flat loss and chance-level accuracy. The same script trains Qwen3.5-0.8B, LFM2-700M, LFM2.5-350M and MiniCPM5-1B normally. The same Clef head and data on plain transformers + PEFT (no Unsloth import) trains Gemma fine, so the issue looks specific to the Unsloth Gemma3 training path. Gemma3 is forced to float32 ("Using float16 precision for gemma3 won't work! Using float32.").

Repro

model, tok = FastDecisionModel.from_pretrained("google/gemma-3-270m", decision_head="clef",
                                               max_seq_length=4096, load_in_4bit=False)
items, _ = FastDecisionModel.build_dataset(rows, tok, model)   # ~40k decision rows
train, holdout = FastDecisionModel.split_holdout(items, 3407)
model = FastDecisionModel.get_peft_model(model, r=64, lora_alpha=64, target_modules="all-linear")
DecisionTrainer(model=model, tokenizer=tok, train_dataset=train, head_learning_rate=3e-4,
                args=TrainingArguments(per_device_train_batch_size=4, gradient_accumulation_steps=4,
                                       learning_rate=1e-4, num_train_epochs=1, lr_scheduler_type="cosine")).train()

What happens

Base and -it produce near-identical loss curves (to 3 decimals), i.e. the head's output barely depends on the backbone. Grad norm is finite and decays (0.57 → 0.03); no NaN/inf.

Ruled out (400 steps each, all still flat, BANKING77 1–3%)

• Prepending BOS to every Clef record (encode_record builds inputs without <bos>)
• Widening the sliding window to full attention
• Rescaling the final norm to remove outlier dims (rms max/median 19.5 → 1.0)
• use_gradient_checkpointing=False + batch size 1 (no padding)

Head-only training on a frozen Gemma backbone does learn a little (BANKING77 4.6%, CLINC150 18.8% at 300 steps), so the hidden states are usable; it's the LoRA/backbone training through the patched path that fails.

Control: same head + data without Unsloth

Vendored joint_schema_model.py (JointSchemaHead + ClefModel), AutoModelForCausalLM fp32 + PEFT LoRA r=64 all-linear, BOS prepended, no Unsloth import:

• 300 steps: gemma-3-270m BANKING77 51.7% (LFM2.5-350M same script: 52.3%), loss 1.40 → 0.91
• full epoch: gemma-3-270m-it BANKING77 73.8%, CLINC150 65.8%, typed-decisions 60.4%; JevBench public 52.4%

Suspects

• Gemma3 float32 patch path in training mode (fp16 clamp/cast inside the compiled attention prepare) interacting with LoRA backward
• encode_record omitting BOS for Gemma (didn't fix it on its own under Unsloth, but the working plain run includes BOS)

Logs / configs

https://huggingface.co/datasets/Manitchahar/decision-bakeoff — folders round2/, gemma_fix/, gemma_fix2/, gemma_plain/, gemma_full/