Error when attempting to merge SDXL Inpaint .1 Beta model - Perhaps impossible? (RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1)

[CCpt5]

The developer branch of A1111 allows for use of SDXL inpaint models. SAI quietly released a very early beta inpaint model for SDXL months ago in diffuser format, and I found a safetensor conversion of it hosted here:

Card: https://huggingface.co/wangqyqq/sd_xl_base_1.0_inpainting_0.1.safetensors
Safetensor: https://huggingface.co/wangqyqq/sd_xl_base_1.0_inpainting_0.1.safetensors/tree/main

I've been using it w/ this simple outpainting extension today (https://github.com/Haoming02/sd-webui-mosaic-outpaint) and it's worked pretty well. I was hoping to try merging it w/ some of my dreambooth (kohya) trained models but am getting the following error: RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1 - (See full console in details)

Is there a fundamental issue w/ this beta SDXL inpainting model that would prevent it from being compatible for merging w/ other SDXL models? Or is there a way the model could be mixed w/ "normal" SDXL models (a la how you can do that w/ v1.5's inpainting model) to allow for better inpainting.

Thanks for any insight!!

Details

venv "D:\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: v1.7.0-437-gce168ab5
Commit hash: ce168ab5dbc8b54b7245f352a2eaa55a37019b91
Launching Web UI with arguments: --opt-sdp-attention --no-half-vae --opt-channelslast --skip-torch-cuda-test --skip-version-check --ckpt-dir e:\Stable Diffusion Checkpoints
No module 'xformers'. Proceeding without it.
*** Extension "sd-webui-lama-cleaner-masked-content" requires "sd-webui-controlnet" which is not installed.
ControlNet preprocessor location: D:\stable-diffusion-webui\extensions\3sd-webui-controlnet\annotator\downloads
2024-02-01 14:36:46,452 - ControlNet - INFO - ControlNet v1.1.440
2024-02-01 14:36:46,552 - ControlNet - INFO - ControlNet v1.1.440
[-] ADetailer initialized. version: 24.1.2, num models: 9
Tag Autocomplete: Could not locate model-keyword extension, Lora trigger word completion will be limited to those added through the extra networks menu.
Loading weights [fe1b97fe65] from e:\Stable Diffusion Checkpoints\SDXL\sd_xl_base_1.0_inpainting_0.1.safetensors
Creating model from config: D:\stable-diffusion-webui\configs\sd_xl_inpaint.yaml
Loading VAE weights specified in settings: D:\stable-diffusion-webui\models\VAE\sdxl_vae.safetensors
Applying attention optimization: sdp-no-mem... done.
Model loaded in 2.7s (create model: 0.2s, apply weights to model: 2.0s).
2024-02-01 14:36:51,003 - ControlNet - INFO - ControlNet UI callback registered.
Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch().
Startup time: 11.6s (prepare environment: 0.6s, import torch: 2.6s, import gradio: 0.7s, setup paths: 0.7s, initialize shared: 0.2s, other imports: 0.5s, list SD models: 0.2s, load scripts: 2.1s, refresh VAE: 0.1s, create ui: 3.4s, gradio launch: 0.4s).

img2img: topnotch artstyle
debugs = []
use_extra_elements = True

• mm_max_models = 7
  config hash = 2733d0e96220606eb11a46bc48f171b0c6c24c67b4b6e2cdcad8e7bc11cbf64d
• mm_use [True, False, False, False, False, False, False]
• model_a SDXL\sd_xl_base_1.0_inpainting_0.1.safetensors [fe1b97fe65]
• base_model None
• max_models 7
• models ['SDXL\2023-11-28 - Topnotch Artstyle (Mostly Ruler-Math) - RealVision Try - 17img (20repeats) - 900reg - b2 - 4000max-step00005400.safetensors']
• modes ['Sum']
• calcmodes ['Normal']
• usembws [[]]
• weights ['0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5']
• alpha [0.5]
• adjust
• use elemental [False]
• elementals ['']
  model_a = SDXL_sd_xl_base_1.0_inpainting_0.1
  Loading SDXL\sd_xl_base_1.0_inpainting_0.1.safetensors [fe1b97fe65] from loaded model...
• loading script.patches...
• base lora_patch
  Applying attention optimization: sdp-no-mem... done.
  isxl = True , sd2 = False
  compact_mode = False
• check possible UNet partial update...
• partial changed blocks = ['BASE', 'IN00', 'IN01', 'IN02', 'IN03', 'IN04', 'IN05', 'IN06', 'IN07', 'IN08', 'M00', 'OUT00', 'OUT01', 'OUT02', 'OUT03', 'OUT04', 'OUT05', 'OUT06', 'OUT07', 'OUT08']
• UNet partial update mode
  Loading model SDXL_2023-11-28 - Topnotch Artstyle (Mostly Ruler-Math) - RealVision Try - 17img (20repeats) - 900reg - b2 - 4000max-step00005400...
  Loading from file e:\Stable Diffusion Checkpoints\SDXL\2023-11-28 - Topnotch Artstyle (Mostly Ruler-Math) - RealVision Try - 17img (20repeats) - 900reg - b2 - 4000max-step00005400.safetensors...
  Calculating sha256 for e:\Stable Diffusion Checkpoints\SDXL\2023-11-28 - Topnotch Artstyle (Mostly Ruler-Math) - RealVision Try - 17img (20repeats) - 900reg - b2 - 4000max-step00005400.safetensors: 3183e182eb1a9439f63712afe40c9933a3f20da78655647392b1e337199e278f
  mode = Sum, alpha = 0.5
  Stage Issue where model in merged but not save leads to "Cannot copy out of meta tensor; no data!" - persists through restart - had to disable app to continue #1/2: 0%| | 0/2255 [00:00<?, ?it/s]
  *** Error running before_process: D:\stable-diffusion-webui\extensions\sd-webui-model-mixer\scripts\model_mixer.py
  Traceback (most recent call last):
  File "D:\stable-diffusion-webui\modules\scripts.py", line 776, in before_process
  script.before_process(p, *script_args)
  File "D:\stable-diffusion-webui\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 3740, in before_process
  theta_0[key] = weighted_sum(theta_0[key], theta_1[key], alpha)
  File "D:\stable-diffusion-webui\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 3435, in _torch_lerp return torch.lerp(theta0.to(torch.float32), theta1.to(torch.float32), alpha).to(theta0.dtype)
  RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1

---

[-] ADetailer: img2img inpainting detected. adetailer disabled.
100%|██████████████████████████████████████████████████████████████████████████████████| 16/16 [00:01<00:00, 13.97it/s]

img2img: topnotch artstyle
debugs = []
use_extra_elements = True

• mm_max_models = 7
  config hash = 2733d0e96220606eb11a46bc48f171b0c6c24c67b4b6e2cdcad8e7bc11cbf64d
• mm_use [True, False, False, False, False, False, False]
• model_a SDXL\sd_xl_base_1.0_inpainting_0.1.safetensors [fe1b97fe65]
• base_model None
• max_models 7
• models ['SDXL\2023-11-28 - Topnotch Artstyle (Mostly Ruler-Math) - RealVision Try - 17img (20repeats) - 900reg - b2 - 4000max-step00005400.safetensors']
• modes ['Sum']
• calcmodes ['Normal']
• usembws [[]]
• weights ['0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5']
• alpha [0.5]
• adjust
• use elemental [False]
• elementals ['']
  model_a = SDXL_sd_xl_base_1.0_inpainting_0.1
  Loading SDXL\sd_xl_base_1.0_inpainting_0.1.safetensors [fe1b97fe65] from loaded model...
• base lora_patch
  Applying attention optimization: sdp-no-mem... done.
  isxl = True , sd2 = False
  compact_mode = False
• check possible UNet partial update...
• partial changed blocks = ['BASE', 'IN00', 'IN01', 'IN02', 'IN03', 'IN04', 'IN05', 'IN06', 'IN07', 'IN08', 'M00', 'OUT00', 'OUT01', 'OUT02', 'OUT03', 'OUT04', 'OUT05', 'OUT06', 'OUT07', 'OUT08']
• UNet partial update mode
  Loading model SDXL_2023-11-28 - Topnotch Artstyle (Mostly Ruler-Math) - RealVision Try - 17img (20repeats) - 900reg - b2 - 4000max-step00005400...
  Loading from file e:\Stable Diffusion Checkpoints\SDXL\2023-11-28 - Topnotch Artstyle (Mostly Ruler-Math) - RealVision Try - 17img (20repeats) - 900reg - b2 - 4000max-step00005400.safetensors...
  mode = Sum, alpha = 0.5
  Stage Issue where model in merged but not save leads to "Cannot copy out of meta tensor; no data!" - persists through restart - had to disable app to continue #1/2: 0%| | 0/2255 [00:00<?, ?it/s]
  *** Error running before_process: D:\stable-diffusion-webui\extensions\sd-webui-model-mixer\scripts\model_mixer.py
  Traceback (most recent call last):
  File "D:\stable-diffusion-webui\modules\scripts.py", line 776, in before_process
  script.before_process(p, *script_args)
  File "D:\stable-diffusion-webui\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 3740, in before_process
  theta_0[key] = weighted_sum(theta_0[key], theta_1[key], alpha)
  File "D:\stable-diffusion-webui\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 3435, in _torch_lerp
  return torch.lerp(theta0.to(torch.float32), theta1.to(torch.float32), alpha).to(theta0.dtype)
  RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1

---

[-] ADetailer: img2img inpainting detected. adetailer disabled.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:04<00:00, 3.27it/s]

img2img: topnotch artstyle
debugs = []
use_extra_elements = True

• mm_max_models = 7
  config hash = 571332e681bd53d1c5055e1cf25b8077992c865dcf08f321cc7e9f1b8e457634
• mm_use [True, False, False, False, False, False, False]
• model_a SDXL\sd_xl_base_1.0_inpainting_0.1.safetensors [fe1b97fe65]
• base_model None
• max_models 7
• models ['SDXL\2024-01-17 - Topnotch Artstyle - 20img - Chips and tree landscapes- 16x9-step00002100.safetensors [a711139531]']
• modes ['Sum']
• calcmodes ['Normal']
• usembws [[]]
• weights ['0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5']
• alpha [0.5]
• adjust
• use elemental [False]
• elementals ['']
  model_a = SDXL_sd_xl_base_1.0_inpainting_0.1
  Loading SDXL\sd_xl_base_1.0_inpainting_0.1.safetensors [fe1b97fe65] from loaded model...
• base lora_patch
  Applying attention optimization: sdp-no-mem... done.
  isxl = True , sd2 = False
  compact_mode = False
• check possible UNet partial update...
• partial changed blocks = ['BASE', 'IN00', 'IN01', 'IN02', 'IN03', 'IN04', 'IN05', 'IN06', 'IN07', 'IN08', 'M00', 'OUT00', 'OUT01', 'OUT02', 'OUT03', 'OUT04', 'OUT05', 'OUT06', 'OUT07', 'OUT08']
• UNet partial update mode
  Loading model SDXL_2024-01-17 - Topnotch Artstyle - 20img - Chips and tree landscapes- 16x9-step00002100...
  Loading from file e:\Stable Diffusion Checkpoints\SDXL\2024-01-17 - Topnotch Artstyle - 20img - Chips and tree landscapes- 16x9-step00002100.safetensors...
  mode = Sum, alpha = 0.5
  Stage Issue where model in merged but not save leads to "Cannot copy out of meta tensor; no data!" - persists through restart - had to disable app to continue #1/2: 0%| | 0/2255 [00:00<?, ?it/s]
  *** Error running before_process: D:\stable-diffusion-webui\extensions\sd-webui-model-mixer\scripts\model_mixer.py
  Traceback (most recent call last):
  File "D:\stable-diffusion-webui\modules\scripts.py", line 776, in before_process
  script.before_process(p, *script_args)
  File "D:\stable-diffusion-webui\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 3740, in before_process
  theta_0[key] = weighted_sum(theta_0[key], theta_1[key], alpha)
  File "D:\stable-diffusion-webui\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 3435, in _torch_lerp
  return torch.lerp(theta0.to(torch.float32), theta1.to(torch.float32), alpha).to(theta0.dtype)
  RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1

[CCpt5]

There's a blurb at the bottom of this model card (by the guy who converted the SDXL inpainting beta model to safetensors) which mentions how to merge it to make any models inpaint models. I guess a base SDXL model w/ the VAE baked in needs to be used: https://huggingface.co/benjamin-paine/sd-xl-alternative-bases

[wkpark]

now, inpainting model is supported by PR #119