add sdxl to prompt weighting by patrickvonplaten · Pull Request #4439 · huggingface/diffusers · GitHub
Skip to content

add sdxl to prompt weighting - #4439

Merged
patrickvonplaten merged 9 commits into
mainfrom
add_sd_xl_to_compel
Aug 3, 2023
Merged

patrickvonplaten merged 9 commits into
mainfrom
add_sd_xl_to_compel

Conversation

@patrickvonplaten

@patrickvonplaten patrickvonplaten commented Aug 2, 2023 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Since 2.0.1 the compel library supports SDXL - this PR updates the docs to give the user a nice example.

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Comment thread docs/source/en/using-diffusers/weighted_prompts.md Outdated
Comment thread docs/source/en/using-diffusers/weighted_prompts.md Outdated
@patrickvonplaten

Copy link
Copy Markdown
Contributor Author

@HuggingFaceDocBuilderDev

HuggingFaceDocBuilderDev commented Aug 2, 2023 •

Copy link
Copy Markdown

The documentation is not available anymore as the PR was closed or merged.

Comment on lines +121 to +127
Let's first make sure we have the newest version of compel installed:

```
pip install --upgrade compel
```

Now we can run an example.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like this should go before the StableDiffusionPipeline example as repeating it becomes kind of redundant.

Comment on lines +133 to +135
pipeline = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", variant="fp16", use_safetensors=True, torch_dtype=torch.float16).to("cuda")

compel = Compel(tokenizer=[pipeline.tokenizer, pipeline.tokenizer_2] , text_encoder=[pipeline.text_encoder, pipeline.text_encoder_2], returned_embeddings_type=ReturnedEmbeddingsType.PENULTIMATE_HIDDEN_STATES_NON_NORMALIZED, requires_pooled=[False, True])

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we indent it please?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, let's leave a link to what ReturnedEmbeddingsType.PENULTIMATE_HIDDEN_STATES_NON_NORMALIZED means otherwise it might get confusing for the readers.

@sayakpaul sayakpaul left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet!

@stevhliu stevhliu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool! 👍

Comment thread docs/source/en/using-diffusers/weighted_prompts.md Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it'd look nicer to compare the images side-by-side (you can copy the code here).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot

Comment thread docs/source/en/using-diffusers/weighted_prompts.md Outdated
Comment thread docs/source/en/using-diffusers/weighted_prompts.md Outdated
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Comment thread docs/source/en/using-diffusers/weighted_prompts.md Outdated
Comment thread docs/source/en/using-diffusers/weighted_prompts.md Outdated
Comment thread docs/source/en/using-diffusers/weighted_prompts.md
@patrickvonplaten
patrickvonplaten merged commit 1a8843f into main Aug 3, 2023
@patrickvonplaten
patrickvonplaten deleted the add_sd_xl_to_compel branch August 3, 2023 19:41
AmericanPresidentJimmyCarter pushed a commit to AmericanPresidentJimmyCarter/diffusers that referenced this pull request Apr 26, 2024
* add sdxl to prompt weighting

* Update docs/source/en/using-diffusers/weighted_prompts.md

* Update docs/source/en/using-diffusers/weighted_prompts.md

* add sdxl to prompt weighting

* Apply suggestions from code review

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Apply suggestions from code review

* Update docs/source/en/using-diffusers/weighted_prompts.md

* Apply suggestions from code review

* correct

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants