Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Best iteration is always 0 #133

Open
Enferlain opened this issue Mar 22, 2024 · 2 comments
Open

Best iteration is always 0 #133

Enferlain opened this issue Mar 22, 2024 · 2 comments
Assignees
Labels
documentation Improvements or additions to documentation

Comments

@Enferlain
Copy link

I've ran around 10 auto merges so far, and for some reason at the end I always end up with it saying "best iteration: 0"

Also the entire time seems to be spent on evaluation.

Thoughts?

Results: 'hyper_score'
   Best score: 0.5331370521490586
   Best parameter set:
      'model_b.BASE'  : 0.26
      'model_b.IN00'  : 0.18
      'model_b.IN01'  : 0.47
      'model_b.IN02'  : 0.01
      'model_b.IN03'  : 0.15
      'model_b.IN04'  : 0.71
      'model_b.IN05'  : 0.11
      'model_b.IN06'  : 0.64
      'model_b.IN07'  : 0.49
      'model_b.IN08'  : 0.91
      'model_b.M00'   : 0.26
      'model_b.OUT00' : 0.7
      'model_b.OUT01' : 0.47
      'model_b.OUT02' : 0.67
      'model_b.OUT03' : 0.13
      'model_b.OUT04' : 0.87
      'model_b.OUT05' : 0.76
      'model_b.OUT06' : 0.56
      'model_b.OUT07' : 0.51
      'model_b.OUT08' : 0.36
      'model_c.BASE'  : 0.66
      'model_c.IN00'  : 0.85
      'model_c.IN01'  : 0.27
      'model_c.IN02'  : 0.26
      'model_c.IN03'  : 0.18
      'model_c.IN04'  : 0.48
      'model_c.IN05'  : 0.82
      'model_c.IN06'  : 0.03
      'model_c.IN07'  : 0.65
      'model_c.IN08'  : 0.36
      'model_c.M00'   : 0.73
      'model_c.OUT00' : 0.59
      'model_c.OUT01' : 0.37
      'model_c.OUT02' : 0.52
      'model_c.OUT03' : 0.05
      'model_c.OUT04' : 0.89
      'model_c.OUT05' : 0.0
      'model_c.OUT06' : 0.16
      'model_c.OUT07' : 0.7
      'model_c.OUT08' : 0.86
   Best iteration: 0

   Random seed: 3908740

   Evaluation time   : 26290.356142520905 sec    [100.0 %]
   Optimization time : 0.13603782653808594 sec    [0.0 %]
   Iteration time    : 26290.492180347443 sec    [120.6 sec/iter]

 - Best weights para =  ['0.26,0.18,0.47,0.01,0.15,0.71,0.11,0.64,0.49,0.91,0.26,0.7,0.47,0.67,0.13,0.87,0.76,0.56,0.51,0.36', '0.66,0.85,0.27,0.26,0.18,0.48,0.82,0.03,0.65,0.36,0.73,0.59,0.37,0.52,0.05,0.89,0,0.16,0.7,0.86'] [True, True, False]
 - Best alpha para =  []
debugs =  ['elemental merge']
use_extra_elements =  True
 - mm_max_models =  3
config hash =  ffccadf55e18e3362e614c3d1a24e39b5abf545d5c8d10d7af27c1ece98694e6
  - mm_use [True, True, False]
  - model_a umbra_mecha.fp16.safetensors [80da973b09]
  - base_model sd_xl_base_1.0.safetensors [31e35c80fc]
  - max_models 3
  - models ['tpn34pdfv10js2ts05tensoradjust.fp16.safetensors [cf4f62151c]', '4thtail3fix.fp16.safetensors [bdc6379d5b]']
  - modes ['DARE', 'Add-Diff']
  - calcmodes ['Normal', 'Normal']
  - usembws [['ALL'], ['ALL']]
  - weights ['0.26,0.18,0.47,0.01,0.15,0.71,0.11,0.64,0.49,0.91,0.26,0.7,0.47,0.67,0.13,0.87,0.76,0.56,0.51,0.36', '0.66,0.85,0.27,0.26,0.18,0.48,0.82,0.03,0.65,0.36,0.73,0.59,0.37,0.52,0.05,0.89,0,0.16,0.7,0.86']
  - alpha [0.5, 0.5]
  - adjust
  - use elemental [False, False]
  - elementals ['', '']
  - Parse elemental merge...
model_a = umbra_mecha.fp16
Loading from file D:\stable-diffusion-webui\models\Stable-diffusion\umbra_mecha.fp16.safetensors...
isxl = True , sd2 = False
compact_mode =  True
 - check possible UNet partial update...
 - partial changed blocks =  ['BASE', 'IN00', 'IN01', 'IN02', 'IN03', 'IN04', 'IN05', 'IN06', 'IN07', 'IN08', 'M00', 'OUT00', 'OUT01', 'OUT02', 'OUT03', 'OUT04', 'OUT05', 'OUT06', 'OUT07', 'OUT08']
 - UNet partial update mode
Open state_dict from file D:\stable-diffusion-webui\models\Stable-diffusion\tpn34pdfv10js2ts05tensoradjust.fp16.safetensors...
mode = DARE, mbw mode, alpha = [0.26, 0.18, 0.47, 0.01, 0.15, 0.71, 0.11, 0.64, 0.49, 0.91, 0.26, 0.7, 0.47, 0.67, 0.13, 0.87, 0.76, 0.56, 0.51, 0.36]
Stage #1/4: 100%|██████████████████████████████████████████████████████████████████| 2263/2263 [00:57<00:00, 39.42it/s]
Check uninitialized #2/4: 100%|████████████████████████████████████████████████| 2263/2263 [00:00<00:00, 215202.24it/s]
Open state_dict from file D:\stable-diffusion-webui\models\Stable-diffusion\4thtail3fix.fp16.safetensors...
mode = Add-Diff, mbw mode, alpha = [0.66, 0.85, 0.27, 0.26, 0.18, 0.48, 0.82, 0.03, 0.65, 0.36, 0.73, 0.59, 0.37, 0.52, 0.05, 0.89, 0.0, 0.16, 0.7, 0.86]
Stage #3/4: 100%|█████████████████████████████████████████████████████████████████| 2263/2263 [00:20<00:00, 109.25it/s]
Save unchanged weights #4/4: 100%|███████████████████████████████████████████████████████████| 253/253 [00:00<?, ?it/s]
 - merge processing in 92.6s (prepare: 14.4s, merging: 78.2s).
 - loading scripts.patches...
 - lora patch
 - Textencoder(BASE) has been successfully updated
 - update UNet block input_blocks.0.
 - update UNet block input_blocks.1.
 - update UNet block input_blocks.2.
 - update UNet block input_blocks.3.
 - update UNet block input_blocks.4.
 - update UNet block input_blocks.5.
 - update UNet block input_blocks.6.
 - update UNet block input_blocks.7.
 - update UNet block input_blocks.8.
 - update UNet block middle_block.
 - update UNet block output_blocks.0.
 - update UNet block output_blocks.1.
 - update UNet block output_blocks.2.
 - update UNet block output_blocks.3.
 - update UNet block output_blocks.4.
 - update UNet block output_blocks.5.
 - update UNet block output_blocks.6.
 - update UNet block output_blocks.7.
 - update UNet block output_blocks.8.
 - update UNet block time_embed.
 - update UNet block out.
 - UNet partial blocks have been successfully updated
 - Reload full state_dict...
 - remove old checkpointinfo
Unloading model 4 over the limit of 3...
 - model 3: umbra_mecha.fp16 + dare_weights(diff tpn34pdfv10js2ts05tensoradjust.fp16) x alpha_0 + (4thtail3fix.fp16 - sd_xl_base_1.0.safetensors [31e35c80fc]) x alpha_1(0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5),(0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5).safetensors [44e1eaaa09]
Creating model from config: D:\stable-diffusion-webui\repositories\generative-models\configs\inference\sd_xl_base.yaml
Loading VAE weights specified in settings: D:\stable-diffusion-webui\models\VAE\sdxl_vae.safetensors
Applying attention optimization: xformers... done.
Model loaded in 3.4s (create model: 0.6s, apply weights to model: 1.8s, apply half(): 0.1s, load VAE: 0.5s, calculate empty prompt: 0.1s).
100%|██████████████████████████████████████████████████████████████████████████████████| 28/28 [00:09<00:00,  3.07it/s]
Total progress: 28it [00:09,  2.92it/s]
 > score_origin = -0.967432594299316/s]
 - Result score = 0.2753925362860547
@wkpark wkpark self-assigned this Mar 24, 2024
@wkpark wkpark added the bug Something isn't working label Mar 24, 2024
@wkpark
Copy link
Owner

wkpark commented Mar 31, 2024

what's your optimizing method?
you can attach your particial screenshot for example:
image

  • Normally, I select specific model and blocks to optimize,
  • Normally, I select Hill Climbing optimizer. I did not test all optimze method. some optimizer will not work as expected or work with specific condition.

@wkpark
Copy link
Owner

wkpark commented Mar 31, 2024

and this is an example hyperactive app (modified slightly. from https://github.com/SimonBlanke/Hyperactive/blob/master/examples/optimization_applications/hyperpara_optimize.py )

"""
This example shows the original purpose of Hyperactive.
You can search for any number of hyperparameters and Hyperactive
will return the best one after the optimization run.

"""

import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.datasets import load_wine
from hyperactive import Hyperactive

data = load_wine()
X, y = data.data, data.target


def model(opt):
    gbr = GradientBoostingClassifier(
        n_estimators=opt["n_estimators"],
        max_depth=opt["max_depth"],
        min_samples_split=opt["min_samples_split"],
        min_samples_leaf=opt["min_samples_leaf"],
        criterion=opt["criterion"],
    )
    scores = cross_val_score(gbr, X, y, cv=4)

    return scores.mean()


search_space = {
    "n_estimators": list(range(10, 150, 5)),
    "max_depth": list(range(2, 12)),
    "min_samples_split": list(range(2, 25)),
    "min_samples_leaf": list(range(1, 25)),
    "criterion": ["friedman_mse", "squared_error"],#, "absolute_error"],
    "subsample": list(np.arange(0.1, 3, 0.1)),
}

if __name__ == '__main__':
    hyper = Hyperactive()
    hyper.add_search(model, search_space, n_iter=40)
    hyper.run()

and its result:

[0] model (Random Search): 100%|───────────────────────────────────────────────────────────────────────| 40/40 [01:17<00:00,  1.94s/it, best_iter=0, best_pos=[17  5 12 18  0 21], best_score=0.9498737373737374]


Results: 'model'
   Best score: 0.9776515151515152
   Best parameter set:
      'n_estimators'      : 130
      'max_depth'         : 3
      'min_samples_split' : 15
      'min_samples_leaf'  : 22
      'criterion'         : friedman_mse
      'subsample'         : 0.7000000000000001
   Best iteration: 0

   Random seed: 416581482

   Evaluation time   : 77.45525693893433 sec    [99.99 %]
   Optimization time : 0.005005359649658203 sec    [0.01 %]
   Iteration time    : 77.46026229858398 sec    [1.94 sec/iter]

you can see "Best iteration: 0" in this case
(I guess this is a mitake/bug of the hyperactive)

@wkpark wkpark added documentation Improvements or additions to documentation and removed bug Something isn't working labels Mar 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

No branches or pull requests

2 participants