Give each xdist worker its own torch.compile cache directory by danielhanchen · Pull Request #9134 · unslothai/unsloth · GitHub
Skip to content

Give each xdist worker its own torch.compile cache directory - #9134

Merged
danielhanchen merged 1 commit into
mainfrom
per-worker-compile-caches
Aug 18, 2026
Merged

Give each xdist worker its own torch.compile cache directory#9134
danielhanchen merged 1 commit into
mainfrom
per-worker-compile-caches

Conversation

@danielhanchen

Copy link
Copy Markdown
Member

Inductor's on-disk caches default to one directory per user, not per process, so the four workers under pytest -n 4 share /tmp/torchinductor_<user> and its fxgraph, aotautograd and Triton subtrees. The upstream recipe is explicit that a common TORCHINDUCTOR_CACHE_DIR is what makes processes share compiled artifacts, so a different value per worker is how they are kept apart.

Sharing is not obviously wrong — the entries are content-addressed. It is still a write-write interaction between processes that nothing here controls, and a cache is exactly the sort of thing that turns a deterministic suite into an intermittent one. This removes the interaction for the price of some recompilation.

What it buys, measured rather than assumed

A full unsloth_zoo run against an empty, dedicated cache directory:

COLD  578s   cache entries=0   size=0
WARM  503s   cache entries=0   size=0

That suite never populates the on-disk cache at all, so for the largest step in the Core legs this is neither a saving nor a cost — and the 75s cold/warm gap is page cache, not compilation, since there is nothing in the directory either time. Worth recording, because it also rules out persisting TORCHINDUCTOR_CACHE_DIR across CI runs for that suite: there is nothing to persist.

So this is insurance for the suites that do compile, applied wherever tests run in parallel rather than only where a problem has already been seen.

Details that matter

  • It runs as a module-level import in the two conftests, not a fixture, because it must take effect before torch is imported.
  • TRITON_CACHE_DIR is set explicitly rather than left to follow: it only derives from the inductor directory when unset, and an environment that exports it would keep all four workers on one Triton cache.
  • An explicit TORCHINDUCTOR_CACHE_DIR is split underneath rather than replaced, so a CI path chosen on purpose is respected.
  • A single-process run is left alone entirely — it already has the default to itself.

Verification

Under real xdist, not only unit tests: four workers report four distinct directories, and the parallel suites still pass. The guard also asserts that both conftests still reach the helper, because a helper nobody imports is the silent failure here.

Inductor's on-disk caches default to one directory per USER, not per process, so
the four workers under pytest -n 4 share /tmp/torchinductor_<user> and its
fxgraph, aotautograd and Triton subtrees. The upstream recipe is explicit that a
common TORCHINDUCTOR_CACHE_DIR is what makes processes SHARE compiled artifacts,
so a different value per worker is how they are kept apart.

Sharing is not obviously wrong, since the entries are content-addressed. It is
still a write-write interaction between processes that nothing here controls, and
a cache is exactly the sort of thing that turns a deterministic suite into an
intermittent one. This removes the interaction for the price of some
recompilation.

What it buys, measured rather than assumed, so nobody has to guess later: a full
unsloth_zoo run against an empty dedicated cache directory took 578s and left
ZERO entries in it. That suite never populates the on-disk cache, so for that
step this is neither a saving nor a cost. It is insurance for the suites that do
compile, applied wherever tests run in parallel rather than only where a problem
has already been seen.

It has to run before torch is imported, so it is a module-level import in the two
conftests rather than a fixture. TRITON_CACHE_DIR is set explicitly rather than
left to follow, because it only derives from the inductor directory when unset
and an environment exporting it would keep all four workers on one Triton cache.
An explicit TORCHINDUCTOR_CACHE_DIR is split underneath rather than replaced, so
a CI path chosen on purpose is respected. A single-process run is left alone
entirely.

Verified under real xdist, not only in unit tests: four workers report four
distinct directories, and the parallel suites still pass.
@danielhanchen
danielhanchen merged commit af2fa43 into main Aug 18, 2026
28 of 37 checks passed
@danielhanchen
danielhanchen deleted the per-worker-compile-caches branch August 18, 2026 01:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant