Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Studio 1.0): Merge LoRA Selection Boxes, add a LoRA Strength parameter, and LoRA alpha calculation fixes #2015

Conversation

one-lithe-rune
Copy link
Contributor

@one-lithe-rune one-lithe-rune commented Dec 5, 2023

Motivation

When I was doing the A1111 API compatibility work, I noticed that we were taking the use_lora argument, splitting it into a use_lora and hf_lora_id, that was reflected in the UI, and then when it came to do the generation we combined these back together to end up with what use_lora had originally.

So since gradio now allows dropdowns allow entry of non-list value and has for a while, I embarked changing the UI to remove this silliness. But whilst doing that, and working on the LoRA tags display PR, I ran a bunch of txt2img with LoRA applied, and noticed that some LoRAs just gave noise, and other looked like the were being applied much to strongly compared with the images with a similar strength on Civitai.

Since I thought this was down to the hardcoded strength/alpha value in the code, I decided to add setting the strength in the UI too.

And then once I did that, I then compared the output to ComfyUI, and though I don't expect to the exact same image in Shark, the LoRAs giving just noise in Shark worked fine there, and Shark still looked like overcooking a lot of the others.

TL;DR lots of stuff in LoRA-land was bad and broken. This hopefully much less broken even though I think things are still fairly ugly.

Changes (so far)

  • Merges LoRA selection in the UI into a single selection, rather than one for LoRAs under ./models and another for Hugging Face Id.
  • Add LoRA strength to UI, and pipeline parameters.
  • Use LoRA embedded alpha values and up tensor dimension * LoRA strength for final alpha when applying LoRA weights rather than a hardcoded value of 0.75. This seems to be how the Koyha scripts are doing it, and this cleanups the pure noise LoRAs for me and brings the look much more inline with ComfyUI.

ToDo

  • Fix scheduler selection (broken by rebase on current main, I think, although main itself is fine).
  • Add command line parameter for LoRA strength.
  • Attempt switch to Diffusers to load the LoRA rather than going through our custom ProcessLoRA function. (may not work, existing codepath that tries this doesn't seem to).
  • Include LoRA weight in metadata info and PNG import.
  • [ ] Manual smoke test of generation with LoRA on each non-SDXL UI tab. Needs merge of UI: Fixes for Gradio 4.7.1/4.8.0 update  #2024 for this to be practical
  • Allow selection LoRA strength above 1.0. Though this is officially out of range, the previous implementation was effectively setting strength higher than this for some LoRAs, so if you want the same level of effect now for those, you'll need to push higher.

Problems/Concerns

  • LoRA strength is baked into the .vmfb currently, and as such I've reflected that in the .vmfb naming. Having to build a set of .vmfbs when you change the LoRA strength to a value not used before is an obvious pain-point.
  • We seem to be running through the codepath updating the model with the LoRA weights even when we're loading up an existing .vmfb. This seems strange, but I'm pretty sure it's not something I've introduced.
  • I'm still not 100% sure I have alpha calculation correct, the results look much much more similar to what ComfyUI is giving me, but...
  • Everything is still ending up in ProcessLoRA which is the exception path for when one style of Diffusers loading breaks.

Obvious Future Work, that I won't be doing on this PR

  • Multiple-LoRA
  • Any changes needed for SDXL LoRA
  • Lycoris

@one-lithe-rune one-lithe-rune changed the title SD/UI: Merge LoRA Selection Boxes, add a LoRA Strength parameter, and LoRA alpha fixes calculation SD/UI: Merge LoRA Selection Boxes, add a LoRA Strength parameter, and LoRA alpha calculation fixes Dec 5, 2023
one-lithe-rune referenced this pull request Dec 10, 2023
- Initializers for webui/API launch
- Schedulers file for SD scheduling utilities
- Additions to API-level utilities
- Added embeddings module for LoRA, Lycoris, yada yada
- Added image_processing module for resamplers, resize tools,
  transforms, and any image annotation (PNG metadata)
- shared_cmd_opts module -- sorry, this is stable_args.py. It lives on.
  We still want to have some global control over the app exclusively
  from the command-line. At least we will be free from shark_args.
- Moving around some utility pieces.
- Try to make api+webui concurrency possible in index.py
- SD UI -- this is just img2imgUI but hopefully a little better.
- UI utilities for your nod logos and your gradio temps.
one-lithe-rune added a commit to one-lithe-rune/SHARK that referenced this pull request Dec 12, 2023
* Updates ProcessLoRA to use both embedded LoRA alpha, and lora_strength
optional parameter (default 1.0) when applying LoRA weights.
* Updates ProcessLoRA to cover more dim cases.
* This bring ProcessLoRA into line with PR nod-ai#2015 against Studio1
monorimet pushed a commit that referenced this pull request Dec 12, 2023
* Updates ProcessLoRA to use both embedded LoRA alpha, and lora_strength
optional parameter (default 1.0) when applying LoRA weights.
* Updates ProcessLoRA to cover more dim cases.
* This bring ProcessLoRA into line with PR #2015 against Studio1
monorimet pushed a commit that referenced this pull request Dec 12, 2023
* Updates ProcessLoRA to use both embedded LoRA alpha, and lora_strength
optional parameter (default 1.0) when applying LoRA weights.
* Updates ProcessLoRA to cover more dim cases.
* This bring ProcessLoRA into line with PR #2015 against Studio1
monorimet pushed a commit that referenced this pull request Dec 12, 2023
* Updates ProcessLoRA to use both embedded LoRA alpha, and lora_strength
optional parameter (default 1.0) when applying LoRA weights.
* Updates ProcessLoRA to cover more dim cases.
* This bring ProcessLoRA into line with PR #2015 against Studio1
monorimet pushed a commit that referenced this pull request Dec 15, 2023
* Updates ProcessLoRA to use both embedded LoRA alpha, and lora_strength
optional parameter (default 1.0) when applying LoRA weights.
* Updates ProcessLoRA to cover more dim cases.
* This bring ProcessLoRA into line with PR #2015 against Studio1
@one-lithe-rune one-lithe-rune changed the base branch from main to SHARK-1.0 December 15, 2023 12:56
@one-lithe-rune
Copy link
Contributor Author

Re-targeted PR to SHARK-1.0 branch.

@one-lithe-rune one-lithe-rune changed the title SD/UI: Merge LoRA Selection Boxes, add a LoRA Strength parameter, and LoRA alpha calculation fixes (Studio 1.0): Merge LoRA Selection Boxes, add a LoRA Strength parameter, and LoRA alpha calculation fixes Dec 15, 2023
* Merges LoRA selection in the UI into a single selection, rather than
one for LoRAs under ./models and another for Hugging Face Id
* Add LoRA strength to UI, and pipeline parameters.
* Use LoRA embedded alpha values and up tensor dimension *
LoRA strength for final alpha when applying LoRA weights rather
than a hardcoded value of 0.75
* Add a `--lora_strength` command line argument
* Include lora strength when reading and writing png metadata
* Allow lora_strength to be set above 1.0 in the UI
* Remove LoRA analysis json files mistakenly commited
* Fix deletion of StableDiffusionPipeline.fromPretrained return by previous commit
@one-lithe-rune one-lithe-rune deleted the ui_lora_selection_and_strength branch December 30, 2023 14:54
@one-lithe-rune one-lithe-rune restored the ui_lora_selection_and_strength branch December 30, 2023 16:20
monorimet pushed a commit that referenced this pull request Jan 17, 2024
* Updates ProcessLoRA to use both embedded LoRA alpha, and lora_strength
optional parameter (default 1.0) when applying LoRA weights.
* Updates ProcessLoRA to cover more dim cases.
* This bring ProcessLoRA into line with PR #2015 against Studio1
gpetters94 pushed a commit to gpetters-amd/SHARK that referenced this pull request Feb 8, 2024
* Updates ProcessLoRA to use both embedded LoRA alpha, and lora_strength
optional parameter (default 1.0) when applying LoRA weights.
* Updates ProcessLoRA to cover more dim cases.
* This bring ProcessLoRA into line with PR nod-ai#2015 against Studio1
monorimet added a commit that referenced this pull request Mar 29, 2024
* (WIP): Studio2 app infra and SD API

UI/app structure and utility implementation.

- Initializers for webui/API launch
- Schedulers file for SD scheduling utilities
- Additions to API-level utilities
- Added embeddings module for LoRA, Lycoris, yada yada
- Added image_processing module for resamplers, resize tools,
  transforms, and any image annotation (PNG metadata)
- shared_cmd_opts module -- sorry, this is stable_args.py. It lives on.
  We still want to have some global control over the app exclusively
  from the command-line. At least we will be free from shark_args.
- Moving around some utility pieces.
- Try to make api+webui concurrency possible in index.py
- SD UI -- this is just img2imgUI but hopefully a little better.
- UI utilities for your nod logos and your gradio temps.

Enable UI / bugfixes / tweaks

* Studio2/SD: Use more correct LoRA alpha calculation (#2034)

* Updates ProcessLoRA to use both embedded LoRA alpha, and lora_strength
optional parameter (default 1.0) when applying LoRA weights.
* Updates ProcessLoRA to cover more dim cases.
* This bring ProcessLoRA into line with PR #2015 against Studio1

* Studio2: Remove duplications from api/utils.py (#2035)

* Remove duplicate os import
* Remove duplicate parse_seed_input function

Migrating to JSON requests in SD UI

More UI and app flow improvements, logging, shared device cache

Model loading

Complete SD pipeline.

Tweaks to VAE, pipeline states

Pipeline tweaks, add cmd_opts parsing to sd api

* Add test for SD

* Small cleanup

* Shark2/SD/UI: Respect ckpt_dir, share and server_port args (#2070)

* Takes whether to generate a gradio live link from the existing --share command
line parameter, rather than hardcoding as True.
* Takes server port from existing --server_port command line parameter, rather than
hardcoding as 11911.
* Default --ckpt_dir parameter to '../models'
* Use --ckpt_dir rather than hardcoding ../models as the base directory for
checkpoints, vae, and lora, etc
* Add a 'checkpoints' directory below --ckpt_dir to match ComfyUI folder structure.
Read custom_weights choices from there, and/or subfolders below there matching
the selected base model.
* Fix --ckpt_dir possibly not working correctly when an absolute rather than relative path
is specified.
* Relabel "Custom Weights" to "Custom Weights Checkpoint" in the UI

* Add StreamingLLM support to studio2 chat (#2060)

* Streaming LLM

* Update precision and add gpu support

* (studio2) Separate weights generation for quantization support

* Adapt prompt changes to studio flow

* Remove outdated flag from llm compile flags.

* (studio2) use turbine vmfbRunner

* tweaks to prompts

* Update CPU path and llm api test.

* Change device in test to cpu.

* Fixes to runner, device names, vmfb mgmt

* Use small test without external weights.

* HF-Reference LLM mode + Update test result to match latest Turbine. (#2080)

* HF-Reference LLM mode.

* Fixup test to match current output from Turbine.

* lint

* Fix test error message + Only initialize HF torch model when used.

* Remove redundant format_out change.

* Add rest API endpoint from LanguageModel API

* Add StreamingLLM support to studio2 chat (#2060)

* Streaming LLM

* Update precision and add gpu support

* (studio2) Separate weights generation for quantization support

* Adapt prompt changes to studio flow

* Remove outdated flag from llm compile flags.

* (studio2) use turbine vmfbRunner

* tweaks to prompts

* Update CPU path and llm api test.

* Change device in test to cpu.

* Fixes to runner, device names, vmfb mgmt

* Use small test without external weights.

* Formatting and init files.

* Remove unused import.

* Small fixes

* Studio2/SD/UI: Improve various parts of the UI for Stable Diffusion (#2074)

* Studio2/SD/UI: Improve various parts of the UI of Shark 2

* Update Gradio pin to 4.15.0.
* Port workarounds for Gradio >4.8.0 main container sizing from Shark 1.0.
* Move nod Logo out of the SD tab and onto the top right of the main tab bar.
* Set nod logo icon as the favicon (as current Shark 1.0).
* Create a tabbed right hand panel within the SD UI sized to the viewport height.
* Make Input Image tab 1 in the right hand panel.
* Make output images, generation log, and  generation buttons, tab 2 in the
right hand panel
* Make config JSON display, with config load, save and clear, tab 3 in the
right hand panel
* Make gallery  area of the Output tab take up all vertical space the other controls
on the tab do not.
* Tidy up the controls on the Config tab somewhat.

* Studio2/SD/UI: Reorganise inputs on Left Panel of SD tab

* Rename previously added Right Panel Output tab to 'Generate'.
* Move Batch Count, Batch Size, and Repeatable Seeds, off of Left Panel and onto 'Generate' Tab.
* On 'Generate' tab, rename 'Generate Image(s)' button to 'Start', and 'Stop Batch' button to 'Stop'. They are now below the Batch inputs on a Generate tab so don't need the specificity.
* Move Device, Low VRAM, and Precision inputs into their own 'Device Settings' Accordion control. (starts closed)
* Rename 'Custom Weights Checkpoint' to 'Checkpoint Weights'
* Move Checkpoint Weights, VAE Model, Standalone Lora Weights, and Embeddings Options controls, into their own 'Model Weights' Accordion control.  (starts closed)
* Move Denoising Strength, and Resample Type controls into their own 'Input Image Processing' Accordion. (starts closed)
* Move any remaining controls in the 'Advanced Options' Accorion directly onto the left panel, and remove then Accordion.
* Enable the copy button for all text boxes on the SD tab.
* Add emoji/unicode glphs to all top level controls and Accordions on the SD Left Panel.
* Start with the 'Generate' as the initially selected tab in the SD Right Panel, working around Gradio issue #7805
* Tweaks to SD Right Tab Panel vertical height.

* Studio2/SD/UI: Sizing tweaks for Right Panel, and >1920 width

* Set height of right panel using vmin rather than vh, with explicit affordances
for fixed areas above and below.
* Port >1920 width Gradio >4.8 CSS workaround from Shark 1.0.

* Studio2/SD: Fix sd pipeline up to "Windows not supported" (#2082)

* Studio2/SD: Fix sd pipeline up to "Windows not supported"

A number of fixes to the SD pipeline as run from the UI, up until the point that dynamo
complains "Windows not yet supported for torch.compile".

* Remove separate install of iree-runtime and iree-compile in setup_venv.ps1, and rely on the
versions installed via the Turbine requirements.txt. Fixes #2063 for me.
* Replace any "None" strings with python None when pulling the config in the UI.
* Add 'hf_auth_token' param to api StableDiffusion class, defaulting to None, and then pass
that in to the various Models where it is required and wasn't already being done before.
* Fix clip custom_weight_params being passed to export_clip_model as "external_weight_file"
rather than "external_weights"
* Don't pass non-existing "custom_vae" parameter to the Turbine Vae Model, instead
pass custom_vae as the "hf_model_id" if it is set. (this may be wrong in the custom vae
cast, but stops the code *always* breaking).

* Studio2/SD/UI: Improve UI config None handling

* When populating the UI from a JSON Config set controls to "None" for null/None
values.
* When generating a JSON Config from the UI set props to null/None for controls
set to "None".
* Use null rather string 'None' in the default config

---------

Co-authored-by: Ean Garvey <[email protected]>

* Studio2/SD/UI: Further sd ui pipeline fixes (#2091)

On Windows, this gets us all the way failing in iree compile of the with SD 2.1 base.

- Fix merge errors with sd right pane config UI tab.
- Remove non-requirement.txt install/build of torch/mlir/iree/SRT in setup_venv.ps1, fixing "torch.compile not supported on Windows" error.
- Fix gradio deprecation warning for `root=` FileExplorer kwarg.
- Comment out `precision` and `max_length` kwargs being passed to unet, as not yet supported on main Turbine branch. Avoids keyword argument error.

* Tweak compile-time flags for SD submodels.

* Small fixes to sd, pin mpmath

* Add pyinstaller spec and imports script.

* Fix the .exe (#2101)

* Fix _IREE_TARGET_MAP (#2103) (#2108)

- Change target passed to iree for vulkan from 'vulkan'
to 'vulkan-spriv', as 'vulkan' is not a valid value for
--iree-hal-target-backends with the current iree compiler.

Co-authored-by: Stefan Kapusniak <[email protected]>

* Cleanup sd model map.

* Update dependencies.

* Studio2/SD/UI: Update gradio to 4.19.2 (sd-studio2) (#2097)

- Move pin for gradio from 4.15 -> 4.19.2 on the sd-studio2 branch

* fix formatting and disable explicit vulkan env settings.

---------

Co-authored-by: Stefan Kapusniak <[email protected]>
Co-authored-by: Stanley Winata <[email protected]>
Co-authored-by: gpetters-amd <[email protected]>
Co-authored-by: gpetters94 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant