Repository navigation
[tests] skip parallelism tests for some models. - #14505
Conversation
|
❌ |
|
/diffusers-bot pytest tests/models/autoencoders/test_models_autoencoder_kl_ltx2_audio.py::TestAutoencoderKLLTX2AudioMemory::test_layerwise_casting_memory |
|
✅ |
|
/diffusers-bot pytest tests/models/autoencoders/test_models_autoencoder_kl_ltx2_audio.py::TestAutoencoderKLLTX2AudioMemory::test_layerwise_casting_memory |
|
✅ |
| @pytest.mark.skip( | ||
| "`forward` runs through the `apply_forward_hook`-decorated `encode` and `decode`, and that decorator's " | ||
| "`pre_forward` call clears the input device accelerate's `AlignDevicesHook` recorded for the caller, so the " | ||
| "output comes back on the last device of the split rather than the input device and the comparison raises. " | ||
| "`test_cpu_offload` covers split placement instead — there every submodule executes on the same device." | ||
| ) |
There was a problem hiding this comment.
My understanding (with help from Claude) is that the @apply_forward_hook / AlignDevicesHook interaction bug described above happens in general and is not limited to the Minimax-H3 video VAE. Claude flags several other autoencoders that use @apply_forward_hook on forward and therefore test_model_parallelism also fails for them:
pytest tests/models/autoencoders/test_models_autoencoder_kl.py::TestAutoencoderKL::test_model_parallelismpytest tests/models/autoencoders/test_models_autoencoder_kl_cogvideox.py::TestAutoencoderKLCogVideoX::test_model_parallelismpytest tests/models/autoencoders/test_models_ltx2_diffusion_decoder.py::TestLTX2VideoDiffusionDecoderModel::test_model_parallelism
I have confirmed these tests fail locally with the same error (base_model_output on cuda:0, new_output on cuda:1). So I think we can also consider skipping these tests for now for the same reason.
There was a problem hiding this comment.
Yeah makes sense. Addressed.
There was a problem hiding this comment.
Since we're increasing num_frames from 8 to 512 here, are the TestAutoencoderKLLTX2AudioMemory tests still reasonably light?
dg845
left a comment
dg845
left a comment
There was a problem hiding this comment.
Thanks for the PR! Left some comments.
github-actions
Bot
added
size/M
2 participants
Fixes #14485. I know that fixing by skipping sounds meh but I am following what many other model tests are already doing. I think it's better treat this case by case as the issues arise. Skipping them is still transparent (as long as the reason is well documented).
Additionally, fixing
tests/models/autoencoders/test_models_autoencoder_kl_ltx2_audio.py::TestAutoencod erKLLTX2AudioMemory::test_layerwise_casting_memory. There's a failing path that's never reached on a T4, hence this passed.