Video output is shaking. Link to video and merge settings in description

[oo92]

Hi.

These are my merging settings:

Loading new_SAEHD model...

Initializing models: 100%|#############################################################################################################################################################################################################################| 4/4 [00:02<00:00,  1.43it/s]
======================== Model Summary ========================
==                                                           ==
==            Model name: new_SAEHD                          ==
==                                                           ==
==     Current iteration: 2422811                            ==
==                                                           ==
==---------------------- Model Options ----------------------==
==                                                           ==
==              use_fp16: False                              ==
==            resolution: 320                                ==
==             face_type: wf                                 ==
==     models_opt_on_gpu: True                               ==
==                 archi: liae-ut                            ==
==               ae_dims: 448                                ==
==                e_dims: 102                                ==
==                d_dims: 102                                ==
==           d_mask_dims: 22                                 ==
==       masked_training: True                               ==
==    retraining_samples: False                              ==
==            mouth_prio: False                              ==
==           uniform_yaw: False                              ==
==         blur_out_mask: True                               ==
==             adabelief: True                               ==
==            lr_dropout: y                                  ==
==         loss_function: SSIM                               ==
==           random_warp: False                              ==
==      random_hsv_power: 0.0                                ==
==     random_downsample: False                              ==
==          random_noise: False                              ==
==           random_blur: False                              ==
==           random_jpeg: False                              ==
==         random_shadow: none                               ==
==      background_power: 0.0                                ==
==       true_face_power: 0.0                                ==
==      face_style_power: 0.0                                ==
==        bg_style_power: 0.0                                ==
==               ct_mode: none                               ==
==          random_color: False                              ==
==              clipgrad: True                               ==
==              pretrain: False                              ==
==               cpu_cap: 8                                  ==
==       preview_samples: 4                                  ==
==    force_full_preview: False                              ==
==                    lr: 5e-05                              ==
==          session_name: lrd on                             ==
==       autobackup_hour: 12                                 ==
==     maximum_n_backups: 6                                  ==
== write_preview_history: False                              ==
==           target_iter: 0                                  ==
==       random_src_flip: False                              ==
==       random_dst_flip: True                               ==
==            batch_size: 8                                  ==
==             gan_power: 0.0                                ==
==        gan_patch_size: 44                                 ==
==              gan_dims: 16                                 ==
==         gan_smoothing: 0.1                                ==
==             gan_noise: 0.0                                ==
==       eyes_mouth_prio: False                              ==
==                                                           ==
==----------------------- Running On ------------------------==
==                                                           ==
==          Device index: 0                                  ==
==                  Name: NVIDIA GeForce RTX 3060 Laptop GPU ==
==                  VRAM: 4.18GB                             ==
==                                                           ==
===============================================================
[y] Use interactive merger? ( y/n ) : n
Choose mode: 
(0) original
(1) overlay
(2) hist-match
(3) seamless
(4) seamless-hist-match
(5) raw-rgb
(6) raw-predict
[1] : 4
[255] Hist match threshold ( 0..255 ) : 255

Choose mask mode: 
(0) full
(1) dst
(2) learned-prd
(3) learned-dst
(4) learned-prd*learned-dst
(5) learned-prd+learned-dst
(6) XSeg-prd
(7) XSeg-dst
(8) XSeg-prd*XSeg-dst
(9) learned-prd*learned-dst*XSeg-prd*XSeg-dst
[1] : 4

[0] Choose erode mask modifier ( -400..400 ) : 2020
[0] Choose blur mask modifier ( 0..400 ) : 100100
[0] Choose motion blur power ( 0..100 ) : 0
[0] Choose output face scale modifier ( -50..50 ) : 0
Color transfer to predicted face ( rct/lct/mkl/mkl-m/idt/idt-m/sot-m/mix-m ) : 

Choose sharpen mode: 
(0) None
(1) box
(2) gaussian
[0] ( ?:help ) : 0

[0] Choose super resolution power ( 0..100 ?:help ) : 0
[0] Choose image degrade by denoise power ( 0..500 ) : 0
[0] Choose image degrade by bicubic rescale power ( 0..100 ) : 0
[0] Degrade color power of final image ( 0..100 ) : 0

[16] Number of workers? ( 1-16 ?:help ) : 16

This is my video: https://drive.google.com/file/d/1GuJ9ONTKuLqCSfThHW8DqI8-2bgHZzJ1/view?usp=sharing

I've trained the model until loss stopped decreasing and this is what I got.

[Elsaam2y]

I see the color is not consistent. Did you try setting the color transfer parameter? Can you also share the ist and src videos to better tell where is the problem?

[oo92]

@Elsaam2y

I select rct for color transfer.

This is data_src: https://drive.google.com/drive/folders/1NUsGGAmE0vSZdiDy23c3DUhALqpuGE9P?usp=sharing

This is data_dst: https://drive.google.com/file/d/1HMm5-wDgr7qvtj0Pqvdd7_5lTxYyfWJq/view?usp=sharing

[Elsaam2y]

Okay I See. I think it works fine considering the technology limitation especially with inconsistent lightning, like in your dst video. This could be the problem why you are getting this inconsistent color distribution in the final output. To be more sure, I would encourage you to train using a simpler dst video with fixed lightning.