iframe-proxy

LeoBorcherding · 2026-04-21T05:42:39Z

Companion branch: https://github.com/LeoBorcherding/unsloth-zoo/tree/fix/cpu-no-torch-import

Problem

The Windows installer advertises CPU-only / GGUF mode but import unsloth
fails on machines without a GPU via two separate paths:

Path 1 (CPU PyTorch, no GPU): unsloth_zoo.device_type.get_device_type()
raises NotImplementedError at import time
Path 2 (--no-torch): torchvision_compatibility_check() raises
ImportError: torch not found

Changes

unsloth:

import_fixes.py: return early in torchvision_compatibility_check() when torch is not installed instead of raising
__init__.py: auto-detect no-torch mode, respect UNSLOTH_NO_TORCH flag even when torch is installed, gate GPU-only imports, distinct warnings for each CPU/no-torch scenario, clean up module-level variables after import
device_type.py: return "cpu" instead of raising when no GPU accelerator found

unsloth-zoo (companion branch: LeoBorcherding/unsloth-zoo@fix/cpu-no-torch-import):

device_type.py: return "cpu" instead of raising when no GPU accelerator found
__init__.py: gate all torch-dependent imports and allocator config behind _HAS_TORCH, add CPU stubs for device type functions
temporary_patches/gpt_oss.py: guard CUDA memory query behind device type check to avoid crash on CPU

Validation

Full validation log posted in #5008.

…t installed refs unslothai#5008 Fixes --no-torch mode crashing with ImportError on import. The check is a version compatibility guard, not a hard dependency enforcer.

…efs unslothai#5008 Fixes CPU-only installs (issue unslothai#5008 path 1): when DEVICE_TYPE is "cpu", skip triton import (not available on CPU) and set SUPPORTS_BFLOAT16 = False to avoid NameError.

…or check refs unslothai#5008

…refs unslothai#5008

LeoBorcherding · 2026-04-21T05:43:00Z

for more information, see https://pre-commit.ci

gemini-code-assist

Code Review

This pull request introduces a 'no-torch' mode and enhances CPU support, allowing the library to initialize when PyTorch is missing or when a GPU is unavailable. Key changes include environment-based toggling of Torch dependencies and fallback mechanisms for device detection. Review feedback indicates that several import fix functions remain unguarded and may crash in the absence of PyTorch. Additionally, the current guards on model and trainer imports are noted as being too restrictive, potentially preventing GGUF inference on CPU-only systems where PyTorch is present.

gemini-code-assist · 2026-04-21T05:45:40Z

-        "Unsloth: Pytorch is not installed. Go to https://pytorch.org/.\n"
-        "We have some installation instructions on our Github page."
+if _NO_TORCH_MODE:
+    torch = None


The torch = None assignment correctly handles the missing dependency for the main import flow. However, the subsequent calls to import_fixes functions (lines 193-212 in the full file) are not guarded and will crash if torch is None.

Specifically, functions like check_vllm_torch_sm100_compatibility, fix_vllm_guided_decoding_params, fix_vllm_pdl_blackwell, and patch_enable_input_require_grads attempt to import torch, vllm, or transformers internally, which will fail if PyTorch is not installed. These calls should be wrapped in an if torch is not None: block to ensure the "no-torch" mode works as intended.

gemini-code-assist · 2026-04-21T05:45:40Z

-elif DEVICE_TYPE == "xpu":
+elif torch is not None and DEVICE_TYPE == "xpu":
    import bitsandbytes as bnb



The guard DEVICE_TYPE != "cpu" is too restrictive here. It prevents unsloth from loading models, tokenizers, and trainers even when PyTorch is installed on a CPU-only machine.

This contradicts the warning issued at line 262, which states that "Only GGUF inference is supported" in CPU mode. To actually support GGUF inference (which requires FastLanguageModel and the saving utilities), the models and tokenizers must be imported. The guard should be simplified to only check if torch is present.

Suggested change

if torch is not None:

nidhishgajjar

Review of PR #5119: fix(cpu): support CPU-only and no-torch import on Windows

Summary

This PR fixes import failures on Windows machines without a GPU by supporting CPU-only and no-torch modes. The changes address two failure paths:

CPU PyTorch with no GPU: get_device_type() raises NotImplementedError
--no-torch mode: torchvision_compatibility_check() raises ImportError

Changes Analysis

1. `unsloth/init.py` ✅

Added environment detection:

_NO_TORCH_MODE: Respects UNSLOTH_NO_TORCH environment variable
_HAS_TORCH: Auto-detects if torch is installed using find_spec()
Sets _NO_TORCH_MODE = True if torch is not installed

Conditional imports:

Torch imports are now gated behind if not _NO_TORCH_MODE
Device type imports are gated behind if torch is not None
Provides CPU stubs for device type functions when torch is unavailable

GPU-only imports gated:

triton import: gated by torch is not None and DEVICE_TYPE != "cpu"
Models, trainer, dataprep: all gated by torch is not None and DEVICE_TYPE != "cpu"
ROCm and XPU-specific imports: properly guarded

User feedback:

Distinct warnings for each scenario:
- No-torch mode with torch installed
- No-torch mode without torch
- CPU mode with GPU detected

Cleanup:

del _HAS_TORCH, _NO_TORCH_MODE at the end
Good practice to avoid polluting module namespace

2. `unsloth/device_type.py` ✅

Graceful degradation:

Changed from raising NotImplementedError to returning "cpu"
Allows CPU-only mode to work without errors
Maintains backward compatibility for GPU scenarios

Error handling preserved:

Still raises RuntimeError for weird accelerator states
Still checks for cuda, xpu, hip availability

3. `unsloth/import_fixes.py` ✅

Early return:

Changed from raising ImportError to returning early
Allows no-torch mode to work
Simple and effective fix

Strengths

Comprehensive approach: Addresses both failure paths mentioned in the issue
Graceful degradation: Falls back to CPU mode instead of failing
Clear user feedback: Distinct warnings for different scenarios
Proper gating: GPU-only imports are correctly guarded
Clean code: Well-structured and maintainable
Draft status: Appropriate for such significant changes

Minor Suggestions

1. Documentation

Consider adding a README section or docstring explaining:

How to use CPU-only mode
How to use no-torch mode
What features are available in each mode
The UNSLOTH_NO_TORCH environment variable

2. Testing

The PR mentions "Full validation log posted in #5008" - please ensure:

Tests are added for CPU-only mode
Tests are added for no-torch mode
Tests cover both Windows and Linux scenarios
Tests verify the warnings are displayed correctly

3. Feature Documentation

For the CPU stub functions:

def is_hip():
    return False

def get_device_type():
    return "cpu"

Consider adding docstrings:

def is_hip():
    """Stub function for no-torch mode. Always returns False."""
    return False

def get_device_type():
    """Stub function for no-torch mode. Always returns cpu."""
    return "cpu"

4. Error Messages

The error message for the "weird accelerator" case could be clearer:

raise RuntimeError(
    f"Unsloth: Weirdly `torch.cuda.is_available()`, `torch.xpu.is_available()` and `is_hip` all failed.\n"
    f"But `torch.accelerator.current_accelerator()` works with it being = `{accelerator}`\n"
    f"Please reinstall torch - it"s most likely broken :("
)

Consider adding the detected accelerator type to the message for better debugging.

Potential Concerns

1. Breaking Changes

This PR changes behavior from failing to gracefully degrading:

Before: Import fails on CPU-only machines
After: Import succeeds but with limited functionality

This is generally a positive change, but ensure:

Users are aware of the limitations via warnings
Documentation clearly states what features are unavailable
The change is mentioned in release notes

2. Companion Branch

The PR mentions a companion branch in unsloth-zoo. Please ensure:

The companion PR is linked in this PR
Both PRs are reviewed and merged together
The changes are compatible

Approval Criteria

I would approve this PR if:

Changes are well-structured and maintainable ✅
Error handling is graceful ✅
User feedback is clear ✅
Documentation is added (suggestion)
Tests are added (mentioned in issue, verify they"re in companion PR)
Companion PR is linked and reviewed

Overall Assessment

✅ Strong approach to fixing the issue
✅ Well-implemented with proper error handling
✅ Good user communication via warnings
⚠️ Consider adding documentation and tests

This is a solid fix that addresses the core issue while maintaining backward compatibility. The graceful degradation approach is appropriate for this use case.

…backs

for more information, see https://pre-commit.ci

LeoBorcherding · 2026-04-21T20:04:51Z

Hi @nidhishgajjar, quick status update addressing your review points.

Completed

Path 1 fixed: CPU torch + no GPU no longer raises NotImplementedError.
Path 2 fixed: --no-torch path no longer raises ImportError.
Follow-up fixed: FastLanguageModel is now importable on CPU via safe kernel/AMP fallbacks.
Distinct warnings are in place for CPU mode vs no-torch mode.
README updated with CPU-only/no-torch usage, limitations, and UNSLOTH_NO_TORCH behavior.
Companion PR ready for review: unsloth-zoo#604 (including grouped-gemm probe fallback hardening).

Validated so far (Windows, CPU-only)

CPU torch import: PASS (exit 0)
No-torch import: PASS (exit 0)
FastLanguageModel CPU import: PASS (exit 0)

Still pending

Linux validation is still pending and planned before final merge.

Not included in this pass
Tests: Adding CPU-only, no-torch, Windows/Linux, and warning-check coverage as requested. Held out of this pass but will include in a follow-up.

Optional suggestion not included: the torch.accelerator.current_accelerator() RuntimeError wording tweak (plus CPU stub docstrings).

If you want, I can do a small follow-up PR for the optional polish items after this fix lands.

LeoBorcherding · 2026-04-22T06:11:53Z

Retested today against the latest commits. Full before/after validation below.

Environment

OS: Windows 11
Python: 3.13.13
Torch: 2.11.0+cpu (CPU-only wheel, no CUDA)
GPU: None

Commits tested

unsloth fix/cpu-no-torch-import @ ce35b46
unsloth-zoo fix/cpu-no-torch-import @ 7b3c412

Part 1 — Repro on `main` (both original failures confirmed)

Path 1 — CPU torch, no GPU:

NotImplementedError: Unsloth cannot find any torch accelerator? You need a GPU.
exit: 1

Path 2 — no-torch (UNSLOTH_NO_TORCH=1):

NotImplementedError: Unsloth cannot find any torch accelerator? You need a GPU.
exit: 1

Note: on main, UNSLOTH_NO_TORCH is never reached — unsloth_zoo raises at module level before unsloth even starts processing the flag.

Part 2 — Validation on `fix/cpu-no-torch-import` (all pass)

Test 1 — CPU torch, no GPU: import unsloth

<string>:1: UserWarning: Unsloth: No GPU detected — running in CPU mode. Only GGUF inference is supported.
Unsloth: Will patch your computer to enable 2x faster free finetuning.
Unsloth Zoo will now patch everything to make training faster!
exit: 0 ✅

Test 2 — CPU torch, no GPU: from unsloth import FastLanguageModel

Unsloth: Will patch your computer to enable 2x faster free finetuning.
Unsloth Zoo will now patch everything to make training faster!
FastLanguageModel import OK
exit: 0 ✅

Test 3 — No-torch mode: UNSLOTH_NO_TORCH=1 import unsloth

<string>:1: UserWarning: Unsloth: UNSLOTH_NO_TORCH is set — running in no-torch mode
  even though torch is installed. Training and GPU features are disabled.
Unsloth: Will patch your computer to enable 2x faster free finetuning.
import OK
exit: 0 ✅

Results

Test	Branch	Exit	Result
`import unsloth` (CPU torch, no GPU)	`main`	1	❌ FAIL — repro confirmed
`import unsloth` (`UNSLOTH_NO_TORCH=1`)	`main`	1	❌ FAIL — repro confirmed
`import unsloth` (CPU torch, no GPU)	fix	0	✅ PASS
`from unsloth import FastLanguageModel` (CPU)	fix	0	✅ PASS
`import unsloth` (`UNSLOTH_NO_TORCH=1`)	fix	0	✅ PASS

3/3 fix-branch tests pass. Both original failure paths reproduced and fixed.

Also confirmed from code review (Gemini flagged these): the import_fixes functions are safe in no-torch mode (they guard themselves internally), and FastLanguageModel is importable on CPU as shown by Test 2 — the models/trainer block uses if torch is not None: with no CPU exclusion.

Still pending: Linux validation before final merge.

for more information, see https://pre-commit.ci

LeoBorcherding · 2026-04-22T08:14:30Z

Linux validation done: 3/3 pass on Ubuntu 24.04 (WSL2, Python 3.12, torch==2.11.0+cpu, no GPU):

=== Test 1: CPU import ===
🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
/root/t1.py:1: UserWarning: Unsloth: No GPU detected: running in CPU mode. Only GGUF inference is supported.
  import unsloth
🦥 Unsloth Zoo will now patch everything to make training faster!
TEST 1 PASS

=== Test 2: FastLanguageModel import ===
🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
/root/t2.py:1: UserWarning: Unsloth: No GPU detected: running in CPU mode. Only GGUF inference is supported.
  from unsloth import FastLanguageModel
🦥 Unsloth Zoo will now patch everything to make training faster!
TEST 2 PASS

=== Test 3: UNSLOTH_NO_TORCH=1 ===
🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
/root/t3.py:1: UserWarning: Unsloth: UNSLOTH_NO_TORCH is set: running in no-torch mode even though torch is installed. Training and GPU features are disabled.
  import unsloth
TEST 3 PASS

Both platforms confirmed ✅. Also added pytests covering these three paths (ae5c271d - tests/test_cpu_no_torch_import.py): all pass locally. Should keep regressions from sneaking back in.

rolandtannous · 2026-04-22T08:29:54Z

@LeoBorcherding the issue i see with that , is that if someone wants to use unsloth core (not studio) , they shouldn't be able to do so without GPU and we need to raise those errors for that specific case. Can you please test that scenario. Install unsloth from this branch, take any of the notebooks and try to run it , what happens?

…ts refs unslothai#5008

for more information, see https://pre-commit.ci

LeoBorcherding · 2026-04-22T09:20:36Z

@rolandtannous, I tested fix/cpu-no-torch-import in a simple custom notebook and from_pretrained returns:

UnboundLocalError: cannot access local variable 'SUPPORTS_BFLOAT16' where it is not associated with a value

this comes from inside gradient_checkpointing.py. i added a check at the top of FastLanguageModel.from_pretrained and FastModel.from_pretrained so now they return the following instead:

RuntimeError: Unsloth: FastLanguageModel requires a GPU for model loading and training.
No GPU was detected on this machine.
For GGUF inference on CPU, use llama.cpp or Unsloth Studio directly.

import unsloth still works without a gpu since studio needs that for gguf stuff, but from_pretrained stops right away with a clear message. also added regression tests for both cases.

here is the test notebook:

test_cpu_notebook.ipynb

rolandtannous · 2026-04-22T09:26:23Z

@LeoBorcherding show me screenshots of the behavior in a notebook before your fix and after your fix. For the after your fix scenario, install unsloth as usual then re-install both unsloth-zoo and unsloth from the fix branches using --force-reinstall --no-deps git+https://github.com/unsloth-zoo@your-branch (same thing for unsloth)

LeoBorcherding · 2026-04-22T10:32:52Z

@rolandtannous here are the screenshots of before and after the cpu guard fix in loader.py in a colab, installed using the commit hash prior vs the latest commit,

Before:

After:

as you can see from_pretrained stops and returns an informative error message right away rather than erroring inside gradient_checkpointing.py with an error that users wouldnt understand.

Datta0 · 2026-04-22T12:07:34Z

-        "Unsloth: Pytorch is not installed. Go to https://pytorch.org/.\n"
-        "We have some installation instructions on our Github page."
+if _NO_TORCH_MODE:
+    torch = None


Im not sure if this works if torch is imported before unsloth

torch = None only affects the name inside init.py, sys.modules['torch'] stays untouched so any other module's import torch still works normally. All our code below that line is gated by if torch is not None: so nothing downstream ever sees the None.

Datta0 · 2026-04-22T12:20:52Z

one thing I want you to check is we have a torchvision comparbility check when importing unsloth
whether we respect HAS_TORCH there too?

I'm just concerned about the ripple effects of this change

rolandtannous · 2026-04-22T12:29:34Z

@rolandtannous here are the screenshots of before and after the cpu guard fix in loader.py in a colab, installed using the commit hash prior vs the latest commit,

Before:

After:

as you can see from_pretrained stops and returns an informative error message right away rather than erroring inside gradient_checkpointing.py with an error that users wouldnt understand.

it seems from teh screenshots that both tests seem to actually install the fix PRs? while only the after case should.

LeoBorcherding · 2026-04-22T12:33:39Z

@rolandtannous the before is using unsloth with the commit hash prior to the CPU guard fix, the after uses the latest commit

before:

"git+https://github.com/LeoBorcherding/unsloth-zoo@fix/cpu-no-torch-import"
"git+https://github.com/LeoBorcherding/unsloth.git@1ddef3e1"

after:

"git+https://github.com/LeoBorcherding/unsloth-zoo@fix/cpu-no-torch-import"
"git+https://github.com/LeoBorcherding/unsloth@fix/cpu-no-torch-import"

…rella, clarify README CPU-only note

for more information, see https://pre-commit.ci

LeoBorcherding · 2026-04-22T13:49:26Z

LeoBorcherding added 12 commits April 20, 2026 03:23

fix: return early in torchvision_compatibility_check when torch is no…

aa55601

…t installed refs unslothai#5008 Fixes --no-torch mode crashing with ImportError on import. The check is a version compatibility guard, not a hard dependency enforcer.

fix: gate triton import and define SUPPORTS_BFLOAT16 for cpu device r…

b4b35ed

…efs unslothai#5008 Fixes CPU-only installs (issue unslothai#5008 path 1): when DEVICE_TYPE is "cpu", skip triton import (not available on CPU) and set SUPPORTS_BFLOAT16 = False to avoid NameError.

fix: print CPU mode message when no GPU detected refs unslothai#5008

70dff56

fix: allow no-torch import path during setup refs unslothai#5008

f401341

fix: allow CPU and no-torch import initialization refs unslothai#5008

8502cc5

fix(cpu): remove redundant early return that bypassed torch.accelerat…

a6fff19

…or check refs unslothai#5008

fix(cpu): use warnings.warn for CPU mode message for consistency refs u…

9d6c024

…nslothai#5008

fix: remove env var mutation for no-torch mode detection refs unsloth…

1e41b7f

…ai#5008

fix: clean up _HAS_TORCH and _NO_TORCH_MODE module-level variables refs

33625ed

unslothai#5008

fix: respect UNSLOTH_NO_TORCH flag even when torch is installed refs u…

dbccc44

…nslothai#5008

fix: only warn 'no GPU' for CPU torch path, not no-torch path refs un…

c52a214

…slothai#5008

fix: distinct warning when UNSLOTH_NO_TORCH set with torch installed …

0c1145e

…refs unslothai#5008

[pre-commit.ci] auto fixes from pre-commit.com hooks

280f69a

for more information, see https://pre-commit.ci

gemini-code-assist Bot reviewed Apr 21, 2026

View reviewed changes

nidhishgajjar reviewed Apr 21, 2026

View reviewed changes

LeoBorcherding and others added 3 commits April 21, 2026 08:52

fix(cpu): allow FastLanguageModel import on CPU with safe kernel fall…

eaff11a

…backs

[pre-commit.ci] auto fixes from pre-commit.com hooks

926520e

for more information, see https://pre-commit.ci

Merge branch 'unslothai:main' into fix/cpu-no-torch-import

de308d8

LeoBorcherding force-pushed the fix/cpu-no-torch-import branch from 498f19b to 9952164 Compare April 21, 2026 17:36

docs: add CPU-only and no-torch mode documentation to README

4b34462

LeoBorcherding mentioned this pull request Apr 21, 2026

fix(cpu): return "cpu" from get_device_type() instead of raising when no GPU found unslothai/unsloth-zoo#604

Draft

LeoBorcherding added 2 commits April 21, 2026 23:33

docs: fix CPU-only and no-torch mode README section

62532ac

docs: clarify no-torch mode disables inference too

ce35b46

LeoBorcherding force-pushed the fix/cpu-no-torch-import branch from 80b0b52 to ce35b46 Compare April 22, 2026 07:58

LeoBorcherding and others added 2 commits April 22, 2026 01:09

test: add CPU and no-torch import tests refs unslothai#5008

ae5c271

[pre-commit.ci] auto fixes from pre-commit.com hooks

1ddef3e

for more information, see https://pre-commit.ci

LeoBorcherding and others added 2 commits April 22, 2026 01:44

fix(cpu): raise RuntimeError in from_pretrained on CPU, add guard tes…

c0574fa

…ts refs unslothai#5008

[pre-commit.ci] auto fixes from pre-commit.com hooks

5ce469f

for more information, see https://pre-commit.ci

Merge branch 'main' into fix/cpu-no-torch-import

7a7cdde

Datta0 reviewed Apr 22, 2026

View reviewed changes

LeoBorcherding and others added 2 commits April 22, 2026 06:11

Address Datta review: simplify _NO_TORCH_MODE, single torch guard umb…

e7d129e

…rella, clarify README CPU-only note

[pre-commit.ci] auto fixes from pre-commit.com hooks

c7d51d3

for more information, see https://pre-commit.ci

Merge branch 'main' into fix/cpu-no-torch-import

36646b1

LeoBorcherding marked this pull request as ready for review April 24, 2026 02:25

LeoBorcherding requested review from danielhanchen, mmathew23 and rolandtannous as code owners April 24, 2026 02:25

Sunbelt Computer Software

PL/B Language Development and Support

Uh oh!

Conversation

LeoBorcherding commented Apr 21, 2026

Problem

Changes

Validation

Uh oh!

LeoBorcherding commented Apr 21, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

nidhishgajjar left a comment

Choose a reason for hiding this comment

Review of PR #5119: fix(cpu): support CPU-only and no-torch import on Windows

Summary

Changes Analysis

1. unsloth/__init__.py ✅

2. unsloth/device_type.py ✅

3. unsloth/import_fixes.py ✅

Strengths

Minor Suggestions

1. Documentation

2. Testing

3. Feature Documentation

4. Error Messages

Potential Concerns

1. Breaking Changes

2. Companion Branch

Approval Criteria

Overall Assessment

Uh oh!

LeoBorcherding commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LeoBorcherding commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Part 1 — Repro on main (both original failures confirmed)

Part 2 — Validation on fix/cpu-no-torch-import (all pass)

Results

Uh oh!

LeoBorcherding commented Apr 22, 2026

Uh oh!

rolandtannous commented Apr 22, 2026

Uh oh!

LeoBorcherding commented Apr 22, 2026

Uh oh!

rolandtannous commented Apr 22, 2026

Uh oh!

LeoBorcherding commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before:

After:

Uh oh!

Datta0 Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

LeoBorcherding Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Datta0 commented Apr 22, 2026

Uh oh!

rolandtannous commented Apr 22, 2026

Before:

After:

Uh oh!

LeoBorcherding commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. `unsloth/init.py` ✅

2. `unsloth/device_type.py` ✅

3. `unsloth/import_fixes.py` ✅

LeoBorcherding commented Apr 21, 2026 •

edited

Loading

LeoBorcherding commented Apr 22, 2026 •

edited

Loading

Part 1 — Repro on `main` (both original failures confirmed)

Part 2 — Validation on `fix/cpu-no-torch-import` (all pass)

LeoBorcherding commented Apr 22, 2026 •

edited

Loading

LeoBorcherding commented Apr 22, 2026 •

edited

Loading