Repository navigation
Qwen Image: txt_seq_lens is redundant and not used #12344
Description
Describe the bug
Here
the parametertxt_seq_lens is calculated.
It is set to the length of the text attention mask for each batch sample - but both the attention mask and txt_seq_lens are passed to the transformer:
https://github.com/huggingface/diffusers/blob/efb7a299af46d739dec6a57a5d2814165fba24b5/src/diffusers/pipelines/qwenimage/pipeline_qwenimage.py#L695C38-L695C39
This is redundant in principle, but not only that: both of them are not used
We have already learned that the attention mask passed to the transformer is ignored: #12294
The only point in code where txt_seq_lens is used is here:
Therefore only the maximum value in the 1D array is used, which is the longest mask. The longest mask when using the entire Qwen Image pipeline is by definition the shape of the text embeddings, because they are encoded using the longest padding strategy here:
[
True is equivalent to longest]
Using this padding strategy, you always have at least 1 batch sample in which the attention mask is completely True, therefore max(txt_seq_lens) is the token length.
If you do not use the entire pipeline but only the transformer, you could pass txt_seq_lens where max(txt_seq_lens) is not equal to the shape of the embeddings. However, then there is a shape mismatch here:
RuntimeError: The size of tensor a (512) must match the size of tensor b (6) at non-singleton dimension 1max(txt_seq_lens) in this example was 6, while the embeddings shape was [..., 512, ...]
Conclusion:
txt_seq_len is redundant, basically unused, and can only be set to 1 valid value
What could be done about this depends on how this was intended. txt_seq_lens is in principle a comfortable and nice way to set an attention mask, if the attention mask is always a sequence of [True, True, True, ... False, False False]. But then we don't need to pass an attention mask, and txt_seq_lens has to accept all valid values, not only the shape of the embeddings.
If the attention mask will be used (see #12294) then there is no need for txt_seq_lens.
Reproduction
Let me know if this needs reproduction, and which part.
Logs
System Info
diffusers HEAD
Activity
dxqb commentedon Sep 23, 2025
workaround here:
https://github.com/Nerogar/OneTrainer/blob/b09ef110d34232ff3abeade787457bea39d8da5f/modules/modelSetup/BaseQwenSetup.py#L133C1-L149C100
harshaljanjani commentedon Nov 11, 2025
Hello @sayakpaul!
Would it be alright if I try to tackle this issue as part of the Diffusers MVP program? It’d be my first contribution to Diffusers, though I’ve raised PRs to Transformers before. From the issue description, it looks like
txt_seq_lensmight be redundant and a bit confusing -- I wanted to confirm if it’s safe to remove it completely, or if there’s a specific reason to keep it around.Secondly, I wanted to ask where can I find info on the hardware requirements for testing the QwenImage pipeline; and in regard to testing, should I run tests with the full model, or are there lightweight test fixtures already available in the repo?
Thanks a lot for your time and guidance!
sayakpaul commentedon Nov 11, 2025
Thanks for your interest here!
Please use your understanding to determine if it's truly redundant first.
Cc @naykun who contributed the pipeline in the first place.
dxqb commentedon Nov 11, 2025
Without input from Alibaba we can only speculate, but I think I know the intention of passing the sequence lengths now:
In their original training code they must have used large batch sizes, that are probably bucketed by image sequence length, but they cannot be bucketed by text sequence lengths because each caption has a different length. Attention masks can be used for that, and are used for that in diffusers right now in some pipelines, but this means you cannot use efficient flash attention: Flash attention neither supports attention masks, nor different sequence lengths in the same batch - but "varlen" flash attention does support different sequence lengths in the same batch.
To use varlen flash attention, you need to know the sequence lengths on CPU: Converting an attention mask (which lives on GPU) back to CPU sequence lengths, as is currently done by the diffusers attention dispatch here,
diffusers/src/diffusers/models/attention_dispatch.py
Line 466 in 66e6a02
Therefore I speculate that they have not used attention masks in their training code, but they achieved the same goal with passing sequence lengths to varlen attention. This would also match that attention masks are ignored in their diffusers code, see #12294)
If you want to make a small project out of this @harshaljanjani , my suggestion would be to
However, some input from Alibaba @naykun would be really appreciated here, to reduce the amount of speculation.
Note that I am in no way associated with diffusers. This is just my personal opinion.
Also see here for a more detailled explanation why "varlen" attention isn't possible efficiently with the current attention dispatch: Nerogar/OneTrainer#1110
dxqb commentedon Nov 12, 2025
My speculation in the previous comment has been confirmed here: #12294 (comment)
sayakpaul commentedon Nov 12, 2025
I am also guessing a couple issues you have opened are related? 👀
dxqb commentedon Nov 12, 2025
Now they are. I didn't know at the time. My proposed solution above adresses both this issue and #12294
github-actions commentedon Jan 9, 2026
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.