iframe-proxy

shimmyshimmer · 2026-08-17T02:06:14Z

Addresses #9032. Deliberately not "Fixes": the defect below is proven and deterministic, but I have not reproduced the reporter's spike on their hardware, so I would rather the issue stay open until they confirm.

Problem

RAM spikes to 2-3x while a model downloads alongside a running model, and the machine locks up.

hf_xet's reconstruction buffers are sized in unsloth_zoo.hf_xet_tuning.xet_env_overrides, and every step of that sizing reads total_ram_bytes:

total = profile.total_ram_bytes or 8 * _GB
limit = _clamp(total // _RAM_FRACTION, _MIN_BUFFER_LIMIT, _MAX_BUFFER_LIMIT)
size  = min(max(limit // 4, min(_STOCK_BUFFER_SIZE, limit // 2)), max(256 * _MB, total // 6))
affordable = max(2, (min(limit, total // 3) - size) // perfile)

SystemProfile.available_ram_bytes is computed right beside it and read by nothing. So the budget is identical whether the box is idle or nearly full:

identical output despite 30x difference in free RAM: True
  30GB free -> buffer limit 4.29GB
   1GB free -> buffer limit 4.29GB

Those buffers are the worker's RSS, not reclaimable page cache, so on the reporter's WSL2 VM the download's allocation lands on top of a resident 27B UD-Q4_K_XL and the two together are the swap. Measured budgets for a single worker with the disk clamp idle:

Two things make it worse. hf_xet_health applies MIN_XET_RAM_BYTES to total RAM too, so a 32 GB host with 2 GB free still starts on the heavier transport. And _preflight_disk_space fails a download that will not fit on disk, but nothing in hub/ reads memory at all.

Fix

Both changes sit in the Studio layer. The zoo stays the one place that decides how big a download should be, and this only ever hands it a smaller machine to decide about, so the two cannot drift the way Studio's old hand-rolled copy did.

clamp_to_available_ram in utils/hf_xet_fallback.py post-processes what the zoo sized. If the budget exceeds a quarter of free RAM, it re-asks xet_env_overrides about a proportionally smaller machine so limit, shared buffer, per-file size and concurrent-file count all scale together, and bottoms out at the zoo's own _MIN_BUFFER_LIMIT.

_memory_pressure_reason in hub/services/download_lifecycle.py applies the zoo's existing MIN_XET_RAM_BYTES threshold to free RAM. Below the floor the answer is a different transport, not a tinier buffer. It only ever demotes, so a machine the zoo already sent to HTTP keeps the zoo's reason.

This does not slow any download down

The clamp is unreachable with headroom by construction: the zoo's budget is an eighth of TOTAL, the clamp triggers at a quarter of AVAILABLE, and total is never below available. Measured on a 68.7 GB host, disk-clamped to 3.60 GB in-flight:

   60.0GB free  |  before  3.60GB   after  3.60GB   unchanged
   20.0GB free  |  before  3.60GB   after  3.60GB   unchanged
   12.0GB free  |  before  3.60GB   after  2.91GB   CLAMPED
    8.0GB free  |  before  3.60GB   after  1.90GB   CLAMPED
    4.0GB free  |  before  3.60GB   after  0.88GB   CLAMPED (floor)

Three more properties keep it free:

Cost is 22 us per download spawn (one psutil read plus one statvfs), against 23.5 us for the zoo sizing it wraps. Nothing runs per chunk.
A user-set HF_XET_HIGH_PERFORMANCE still stands the caps down. That falls out rather than being special-cased: the zoo drops its cap keys, so no budget key reaches the clamp. The same mechanism protects any explicitly set variable, since the zoo's apply is setdefault and a user value never lands in what it reports writing.
Unmeasurable RAM reads as unclamped. No psutil, no profile, an older zoo, or a raising probe leaves the download exactly as it was.

Tests

9 new tests across test_hf_xet_fallback.py and test_hub_download_transport_auto.py: no-op with headroom, clamp engages under pressure, floor, high-performance stand-down, user-set keys untouched, every degradation path, the transport gate in all four states, and one test pinning the arithmetic against the real zoo formulas so a future sizing change cannot quietly reintroduce a budget bigger than free RAM.

57 pass in those two files, plus 755 download and transport tests and 555 hub tests. ruff check clean.

Pre-existing failures on my macOS box are unchanged by this branch, each confirmed by stashing and re-running: test_hf_cache_settings.py and test_video_routes.py fail 8 of 71 identically with and without the patch, and test_unresumable_partial_purge.py has one case-insensitive-filesystem failure either way.

What this does and does not claim

Proven: the sizing reads total RAM only, available_ram_bytes is dead, and the budget is identical at 30x different free RAM. That is a real defect regardless of #9032.

Not proven: that it is the whole cause of the reporter's 2-3x spike. I have not reproduced it, and I cannot tell from the report whether they were on the Xet transport at all. If their download ran over HTTP, this changes nothing for them. The reporter can check which transport the download used, and confirm whether the spike is smaller on this branch.

Not covered here

The page-cache half. Writing roughly 18 GB through the cache evicts the running model's mmap'd pages, since llama.cpp gets no --mlock or --no-mmap unless "reserve system RAM" is on, and WSL2 releases guest page cache lazily. So some apparent spike remains even with buffers clamped. The fix is a posix_fadvise(DONTNEED) sweep trailing the writer in the download worker, which is Linux-only and touches the resume-critical .incomplete path, so it belongs in its own PR.

hf_xet's reconstruction buffers are sized from the machine's TOTAL RAM, so a download started while a model is loaded asks for the same multi-GB budget it would ask for on an idle box. Those buffers are the worker's RSS, not reclaimable page cache, so the request and the resident weights add up and the machine swaps. Clamp the zoo-sized budget to a quarter of free RAM by re-asking the zoo about a smaller machine, so buffer, per-file and file count scale together. A quarter of AVAILABLE always exceeds the zoo's eighth of TOTAL on an idle host, so the clamp is unreachable unless RAM is genuinely held: sizing is byte-identical when there is headroom. Also apply the zoo's existing MIN_XET_RAM_BYTES floor to free RAM when picking a transport, so a host too tight for even the clamped floor downloads over HTTP. Fixes #9032

for more information, see https://pre-commit.ci

The recompute calls xet_env_overrides directly, without the throttled flag apply_xet_env threads through after a 429, so an un-throttled recompute could hand back a stream ceiling that backoff had lowered. Take the smaller of the two per key. Every derived value is monotonic in total RAM, so the result stays coherent.

for more information, see https://pre-commit.ci

The Studio UI never sends transport_mode=auto. effectiveTransportMode() resolves Auto through get_download_transport_capabilities(probe=true) and submits the answer as an explicit xet/http, which resolve_requested_use_xet honours without calling resolve_auto_use_xet. The gate therefore never ran on the primary flow. Move the verdict into a shared free_ram_pressure_reason() and call it from the probe as well, so the UI path and an API caller that sends auto agree. Probe only, so an ordinary browse poll stays read-only and still does not load Zoo.

for more information, see https://pre-commit.ci

Both early returns in resolve_auto_use_xet skipped the gate: a zoo without hf_xet_health, and a health probe that raises. Free RAM is read from hf_xet_tuning, a different module, so neither says anything about whether the machine can afford Xet right now. Fold both into one optimistic path that still consults free RAM, and move the registry probe's RAM read outside the health try for the same reason. A health verdict that already demoted keeps its own reason.

for more information, see https://pre-commit.ci

A worker allocates inside the child, after Popen returns, so free RAM does not move until well after sizing. Four downloads starting together each read the same untouched value and each took a quarter of it, promising the whole machine. Sizing now subtracts what live siblings were already promised, and the reservation is bound to the worker's pid so it frees when that worker exits. The transport gate subtracts it too: the clamp bottoms out at Xet's floor, so enough simultaneous workers would still add up past free RAM, and the next download is better served by HTTP than by Xet at its minimum.

for more information, see https://pre-commit.ci

The ledger read and the reserve sat in separate critical sections, so concurrent sizings could all read the same total before any of them wrote, which is the overcommit the ledger exists to stop. Hold the lock across the whole decide-and-reserve region; the recompute inside is pure arithmetic on a frozen profile, and the RAM/disk reading stays outside it. The earlier reservation tests started workers sequentially, which never entered that window. The new test races four threads through it.

…FORCE_XET override the free-RAM gate

danielhanchen · 2026-08-18T11:58:09Z

Review summary

Verdict: useful, merge it. It fixes a real, deterministic defect, and the scoping is honest.

Before / after

Before, unsloth_zoo.hf_xet_tuning sized hf_xet reconstruction buffers from total_ram_bytes only, and available_ram_bytes was computed and read by nothing. A 32 GB box got the same ~4.29 GB buffer whether 30 GB or 1 GB was free. Those buffers are worker RSS, not reclaimable cache, so a download starting next to a resident 27B GGUF swapped the machine (#9032). The Xet-vs-HTTP floor was applied to total too, so a 32 GB host with 2 GB free still started on the heavier transport.

After, the zoo still decides and Studio just hands it a smaller machine. Measured against the real zoo:

  60.0GB free | zoo 64.00GB -> env 15.00GB CLAMPED   files 24->24
  12.0GB free | zoo 64.00GB -> env  3.00GB CLAMPED   files 24->11
   4.0GB free | zoo 64.00GB -> env  1.00GB CLAMPED   files 24->3
   0.0GB free | zoo 64.00GB -> env 64.00GB unchanged  (unmeasurable -> no-op)

Does it break anything

No, and the strong form of that holds: this PR can never refuse a download. download_transport_unavailable_reason(TRANSPORT_HTTP) returns None unconditionally, an explicit "xet" request bypasses the gate, and the clamp is reduce-only, flooring at the zoo's own _MIN_BUFFER_LIMIT. Worst case is Xet to HTTP (slower) or a smaller buffer (slower). Measured boundary: 4.1GB free to Xet, 3.9GB to HTTP, 0.0GB unmeasurable and left alone.

Old installs: a 12-combination missing-attribute matrix (no system_profile, no xet_env_overrides, no _MIN_BUFFER_LIMIT/_RAM_FRACTION/MIN_XET_RAM_BYTES, no hf_xet_tuning at all) x idle/loaded. Every degradation is a no-op. An existing Xet cache is untouched. Full backend suite 25,885 passed, with the 173 unrelated failures identical at the merge base.

Two fixes pushed (`b132624`)

1. Windows liveness probe (P0, data corruption). os.kill(pid, 0) is not a probe on Windows: CPython maps every signal other than CTRL_C_EVENT/CTRL_BREAK_EVENT onto TerminateProcess(handle, sig), so the ledger's liveness check killed the download worker it was asking about. TerminateProcess(handle, 0) sets exit code 0, and download_lifecycle.py maps rc == 0 to "complete" - so a partially downloaded model was registered complete and served. This repo already documents the hazard at utils/process_lifetime.py:877, so the fix delegates to that module's handle-based probe rather than growing a second copy. The no-platform-probe fallback assumes alive, so the worst case is a reservation held too long.

2. UNSLOTH_FORCE_XET=1 was silently overridden by the new gate. The PR honoured the OFF switches but not the ON switch, making the escape hatch one-directional while the zoo still logs "set UNSLOTH_FORCE_XET=1 to override". Fixed symmetrically in the auto path and the probe, with buffers still clamped.

All four new tests fail on unfixed source and pass with the fix; 77 passed, 2 skipped; ruff clean.

Simulation

428 new simulations plus 109 repo tests, all passing. Platform x accelerator matrix run over all 24 cells of [Windows, Linux, WSL, macOS] x [NVIDIA, AMD, CPU-only, no torch] - simulated at the sys.platform and psutil seams, not measured; only Linux + NVIDIA is real hardware here, and I am not claiming otherwise. The verdict is accelerator-independent by construction. Also covered: integer overflow at 2^70, division by zero, pathological _RAM_FRACTION, unit-suffixed values, 429-throttled ceilings, 512MB/1GB/3.5GB hosts, 8 threads racing the clamp, and a real child bound then killed with the reservation freed.

Limits worth stating rather than hiding

Inside a container the gate will not fire. system_profile() is cgroup-aware but computes available = min(host_available, cg_mem) and never reads memory.current, so a 12 GB container with an 11 GB model resident reports 12 GB available. Not a regression - pre-PR used the same number as total - but it means [Bug] RAM usage spike while downloading a new model #9032 in Docker or Colab is not caught. The fix belongs in unsloth_zoo.
The admission side (free_ram_pressure_reason) is still read-then-release and structurally cannot be atomic, since probes run in FastAPI's threadpool and admission and spawn are separated by awaits. Bounded to N x 1 GB by the floor.
The frontend's 30s capabilities cache defeats the ledger for concurrent downloads via the UI (cacheUsable accepts a previously probed entry). Still a large net improvement; one-line follow-up is { probe: true, force: true }.
/v1 auto-download never consults either gate (openai_auto_download.py:857 sends no transport_mode), and it is the highest-RAM-pressure caller. Pre-existing; one-line follow-up is transport_mode = "auto".
11 in-process loader call sites still size from total RAM, bypassing spawn_worker. Worth a line in the description as intentional scope.

CI

The one red check, Unsloth install + inference without Visual Studio, is a ~1% infra flake, not PR-caused: over 400 runs of that workflow on 2026-08-17 there were exactly 4 failures, including main itself and an unrelated branch 16 seconds after this PR. That job also primes models via hf-download-with-retry.sh, not Studio's download manager, so this code never executes on it.

The probe answer was cached for 30 seconds, so several Auto downloads begun inside that window were all admitted on the same pre-reservation verdict and each submitted an explicit transport_mode=xet, which the start path honours without re-reading free RAM. Measured on a 32 GB box with 5 GB free: four starts promised 4.25 GB of Xet buffer pools, one per worker process, where a live gate admits the first and sends the rest over HTTP. Render polls still answer from the cache; only the probe itself has to be live, and it costs no network probe because the health verdict is memoized upstream.

The ledger charged a bound worker's whole buffer promise for its whole lifetime, but the free-RAM reading it is subtracted from has already dropped by whatever that worker allocated, so the same bytes went out twice. Measured with a real child holding real touched RSS: 4.04 GB genuinely free, reported as 2.0 GB, and the next Auto download demoted to HTTP for room that existed. The reservation exists to bridge the window between sizing and allocation, which is what the comment above it says, so credit what has materialised and keep the full promise for a worker that has not bound yet. The xet buffer is drawn through a permit semaphore rather than preallocated, which is why resident memory is the right measure of how much of the promise is already spent.

for more information, see https://pre-commit.ci

danielhanchen · 2026-08-18T16:58:13Z

@codex review

…://github.com/unslothai/unsloth into r9052

danielhanchen · 2026-08-18T18:40:27Z

@codex review

danielhanchen · 2026-08-18T20:54:24Z

@codex review

danielhanchen · 2026-08-19T01:26:04Z

@codex review

danielhanchen · 2026-08-19T07:53:17Z

Confirmed the sizing in unsloth_zoo/hf_xet_tuning.py reads total_ram_bytes only and available_ram_bytes is unused, so the clamp is addressing a real defect.

danielhanchen · 2026-08-19T11:50:23Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cc30a1efa5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-19T11:54:07Z

+    pressure = _memory_pressure_reason()
+    if pressure is not None:
+        return (False, pressure)


Recheck RAM when admitting each Xet worker

When multiple Auto downloads start concurrently, every request can pass this pressure check before any request reaches spawn_worker and records a reservation; model starts widen that window further by awaiting GGUF metadata resolution before spawning. All requests then retain use_xet=True, while the later clamp bottoms out at _MIN_BUFFER_LIMIT, so enough concurrently admitted workers can collectively reserve more RAM than is available and recreate the host-lockup condition this change targets. Make the pressure decision and reservation atomic at worker admission, or recheck immediately before each spawn.

Useful? React with 👍 / 👎.

The ledger already closes the case that actually happens, which is one download starting while others run: free_ram_pressure_reason reads the reservations, so the second request sees the first. What is left needs several Auto downloads issued inside the same metadata-resolution window on a host sitting right on the 4GB threshold. Rechecking inside spawn_worker cannot fix that either, since the registry row, the UI and the partial-file writer are already committed to xet by then and a late downgrade would break resume rather than protect RAM.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 468bf1412c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-19T12:08:22Z

+        if now - stamp > _BOUND_RESERVATION_TTL or not _pid_alive(pid):
+            _budget_reservations.pop(token, None)
+            continue
+        total += max(0, nbytes - _worker_rss(pid))


Credit only Xet buffers against the reservation

When concurrent Xet workers are live, _worker_rss(pid) includes the Python interpreter, imported Hugging Face libraries, metadata, and other allocations—not just reconstruction buffers. Because that unrelated RSS is already absent from available_ram_bytes, subtracting it from nbytes effectively ignores it entirely: for baseline RSS B and promised buffer P, the code reports (available - B) - (P - B) = available - P rather than available - B - P. Multiple workers can therefore appear to have more unclaimed RAM than they actually do and be admitted with buffer promises that recreate the host-overcommit condition this ledger is intended to prevent; track a buffer-attributable delta or otherwise preserve the non-buffer portion of each worker's reservation.

Useful? React with 👍 / 👎.

The buffer-attributable delta is not observable here. bind_worker_budget runs in the finally block immediately after Popen, so the only baseline we could snapshot is a freshly exec'd interpreter of a few MB, and a delta from that would count the worker's own huggingface_hub and hf_xet imports as drawn buffer, landing back on today's number. At steady state, when the promise is fully drawn, the current arithmetic is exact; the optimism is one interpreter baseline per worker during the transient and is already bounded by the cap at the promise, so I would rather keep the simple accounting than add a constant we cannot measure.

danielhanchen · 2026-08-19T12:21:32Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 468bf1412c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-19T12:24:11Z

+    overrides = getattr(module, "xet_env_overrides", None)
+    profile_of = getattr(module, "system_profile", None)
+    if overrides is None or profile_of is None or _BUFFER_LIMIT_KEY not in sized:
+        return sized


Reserve user-pinned Xet buffer limits

When HF_XET_RECONSTRUCTION_DOWNLOAD_BUFFER_LIMIT is explicitly set, the zoo's setdefault behavior omits that key from sized, so this early return also skips _reserve_worker_budget. Unlike the earlier general admission race, this fresh bypass persists even after the ledger was added: concurrent workers using a user-pinned limit remain invisible to free_ram_pressure_reason() until their buffers become resident, allowing each worker to allocate the same explicit cap. Preserve the user's value, but reserve the effective limit read from env.

Useful? React with 👍 / 👎.

A pinned HF_XET_RECONSTRUCTION_DOWNLOAD_BUFFER_LIMIT is the same deliberate stand-down as a user-set HF_XET_HIGH_PERFORMANCE: the zoo leaves the value alone, so we neither clamp it nor account for it. Reserving a number we never wrote would only push a sibling download to HTTP, and it cannot prevent the pressure the pin itself creates, since the very first worker already takes the full pinned buffer with no clamp.

shimmyshimmer requested a review from danielhanchen as a code owner August 17, 2026 02:06

pre-commit-ci Bot and others added 3 commits August 17, 2026 02:09

[pre-commit.ci] auto fixes from pre-commit.com hooks

9a24ec6

for more information, see https://pre-commit.ci

[pre-commit.ci] auto fixes from pre-commit.com hooks

857ea2f

for more information, see https://pre-commit.ci

This comment was marked as resolved.

Sign in to view

danielhanchen and others added 2 commits August 16, 2026 20:10

[pre-commit.ci] auto fixes from pre-commit.com hooks

519cf92

for more information, see https://pre-commit.ci

This comment was marked as resolved.

Sign in to view

danielhanchen and others added 2 commits August 16, 2026 20:34

[pre-commit.ci] auto fixes from pre-commit.com hooks

3d59f96

for more information, see https://pre-commit.ci

This comment was marked as resolved.

Sign in to view

danielhanchen and others added 2 commits August 17, 2026 00:09

[pre-commit.ci] auto fixes from pre-commit.com hooks

8a43d86

for more information, see https://pre-commit.ci

This comment was marked as resolved.

Sign in to view

Probe worker liveness without signalling on Windows, and let UNSLOTH_…

b132624

…FORCE_XET override the free-RAM gate

This comment was marked as resolved.

Sign in to view

danielhanchen and others added 2 commits August 18, 2026 15:56

[pre-commit.ci] auto fixes from pre-commit.com hooks

fe6417a

for more information, see https://pre-commit.ci