KeyError: 'model.diffusion_model.middle_block.0.in_layers.0.weight'

[Luke2642]

Great extension. Ran into a little bug, managed to work around it by selecting only a range of layers. It seemed anything higher than IN09... so IN10, 11, 12 and MID made it fall over. It's probably a trivial little regex or something that needs fixing.

*** Error running before_process: /home/stable-diffusion-webui/extensions/sd-webui-model-mixer/scripts/model_mixer.py
Traceback (most recent call last):
File "/home/stable-diffusion-webui/modules/scripts.py", line 611, in before_process
script.before_process(p, *script_args)
File "/home/stable-diffusion-webui/extensions/sd-webui-model-mixer/scripts/model_mixer.py", line 2191, in before_process
first_permutation, y = weight_matching(permutation_spec, models["model_a"], theta_0, usefp16=usefp16, device=device)
File "/home/stable-diffusion-webui/extensions/sd-webui-model-mixer/scripts/weight_matching.py", line 811, in weight_matching
w_b = get_permuted_param(ps, perm, wk, params_b, except_axis=axis)
File "/home/stable-diffusion-webui/extensions/sd-webui-model-mixer/scripts/weight_matching.py", line 771, in get_permuted_param
w = params[k]
KeyError: 'model.diffusion_model.middle_block.0.in_layers.0.weight'

I hope you find a way to simplify the extension. I've been using https://github.com/ashen-sensored/sd-webui-runtime-block-merge for a long time, and it's a little bit simpler, less powerful, and doesn't have git rebasin, but neither has a great interface. My suggestions would be:

• Improve labels with "text encoder", "text decoder" and "unet" terminology. Maybe one section for each?
• Make the sliders update the textbox in real time.
• A graphical interface - spline nodes that could be dragged, or so you could drag the mouse across a box and set a load of nodes in a streak - would be amazing, although, I realise not easy to implement.

[wkpark]

Thank you for your bug report and suggestions!

[Luke2642]

No problem, just wanted to say thanks again! I realise it's completely off topic putting more ideas in a bug report, but if I could brain dump here, maybe it's helpful, if not, just ignore it :-)

My theory is that all block merge extensions are hamstrung by the drop down box of options, gradV, gradA, wrap, cos, etc, yours is slightly better because it has the additional model b weight slider, but using it is confusing as 0.5 is a 1:1 multiplier on the block weights, and you have to 'apply' the from the slider to the textbox, which I personally hate.

As a work around, I've made up a huge excel sheet of weight patterns and formatted them in notepad++. Examples are:

• 'gradV mild' which start at 0.5 and descend by 0.05 on both in and out
• 'grad V very mild' 0.25 descending.
• 'zero in 0.1 pyramid out' which has 0's on IN and rises by 0.1 then falls again over OUT.
• 'gradA rising 0.1 delay 6' which has zeros for 6 blocks but then rises by 0.1.
• 'Wrap 4 0.5' where the first 4 blocks on in and out are set to 0.5
• etc

In other words, this list is very inefficient, but, it seems to have helped me narrow down the two strategies that can get move features across models without losing coherent structure or a blurry mess:

1. Symmetrical changes across in and out, which I assume is due to the importance of skip connections,
2. Softened bumps, which seems to make a more coherent change than just moving one slider without changing the blocks either side. Like cos/smoothstep. I very rarely find smoothstep^2/3/4 or cubic hermite useful on their own, they're far too strong.

So, to implement this quickly, couple of line buttons could help the user build up the merge weights in a lego brick modular way. I've also made the buttons the graph too, but it could be just two lines of buttons and vertical sliders or something.

Underneath you'd have labelled in sliders to modify how the click action impacts the weights:

1. Step change size, from 0 to 1, which changes how big the change is when you click on a button. Default 0.3.
2. Smoothness slider, from 1 to 13 in steps, which changes how the change tapers off. Default 5, so the immediate above/below blocks go up 4/5 of the value, and the next adjacent blocks go up 3/5, and so on.
3. Symmetry radio buttons: on, off, inverse... so the user clicking e.g. IN04 also clicks OUT07.

The other quality of life feature missing during merging is using multiple prompts in one batch, so you can see how the change affects landscapes, architecture, close up faces, full body, etc. There are extensions that implement different prompts across a batch, so it is possible, and much faster than the xyz script which generates one image at at time.

[wkpark]

this issue automatically closed by merging #37
thank you very much for sharing your idea! will look at it carefully later on

• PR dynamic update merge block weights using js #40 - sync - "Make the sliders update the textbox in real time."
• support fast partial UNet update
• support partial textencoder(BASE) update with workaround
• block level weights helper (as lego brick way or something to easily control merging block weights)
• xyz + prompt batch