hyperparameter optimization

[geoHeil]

Are there any recommended best practices to optimize the hyperparameters of TSAI models? Given the fact that TSAI is built on fasatai I would think https://optuna.readthedocs.io/en/stable/reference/generated/optuna.integration.FastAIV2PruningCallback.html#optuna.integration.FastAIV2PruningCallback is suitable but are not sure if it would be a recommended option.

[vrodriguezf]

Interesting! Is there any example of that callback working in practice?

[geoHeil]

https://github.com/optuna/optuna-examples/blob/main/fastai/fastaiv2_simple.py

[oguiza]

Hi @geoHeil,
Thanks for your question.
Hyperparameter optimization is a complex topic as there is a myriad of possible combinations you can try.
There's currently no recommended approach in tsai in terms of hyperparameter optimization.
I usually follow one that is based on a mix of fast.ai and Full Stack Deep Learning approaches. This is, at a high level, the process I follow:

1. Establish a simple baseline (most frequent class, median, etc)
2. Set aside a small train set (<=10k samples)
3. Architecture selection: try several architectures with default parameters.
4. Use learn.lr_find to select a learning rate (sometimes with different batch sizes, although I tend to use the largest possible bs)
5. Try different loss functions
6. Test different preprocessing approaches (no scaling, TSStandardize with different by_sample and by_var values)
7. Test several data augmentation techniques (including CutMix1d, MixUp1d, etc)
8. Test several key hyperparameters (width, depth, weight initialization, etc)
9. Test other hyperparameters: batch size, optimizer, learning rate schedule (fit_one_cycle vs fit_flat_cos), activation, etc.

It might be interesting to build a tsai tutorial on Hyperparameter Optimization combining the approach above with optuna
and/or wandb.
Ideally, it would work on both notebooks and python scripts.

Would any of you @geoHeil, @vrodriguezf, @williamsdoug, @dnth be interested in creating such a tutorial? I do believe that the best way to learn something is by teaching others.

[dnth]

I happen to use Optuna with TSAI before! Its incredibly easy to use. All we need to do is define the objective function, define search space and run the search (called 'Study' in Optuna). They can all be done in short snippets below

import optuna
from optuna.integration import FastAIPruningCallback

num_epochs = 5
num_optuna_trials = 100
    
def objective(trial:optuna.Trial):
    # Define search space here. 
    num_filters = trial.suggest_categorical('num_filters', [32, 64, 96, 128]) 
    dropout_rate = trial.suggest_float("dropout_rate", 0.0, 1.0)

    model = InceptionTimePlus(dls.vars, dls.c, nf=num_filters, fc_dropout=dropout_rate)
    learn = Learner(dls, model, metrics=accuracy, cbs=FastAIPruningCallback(trial))
    learn.fit_one_cycle(num_epochs, lr_max=1e-3)
            
    # return the valid_loss or any value that is to be optimized - learn.recorder.values returns [[train_loss, valid_loss, acc]]
    return learn.recorder.values[-1][1]

And run the study!

study = optuna.create_study(direction='minimize', pruner=optuna.pruners.HyperbandPruner())
study.optimize(objective, n_trials=num_optuna_trials, show_progress_bar=True)

After the study is done print the best combination

print("Study statistics: ")
print("  Number of finished trials: ", len(study.trials))

print("Best trial:")
trial = study.best_trial

print("  Value: ", trial.value)

print("  Params: ")
for key, value in trial.params.items():
    print("    {}: {}".format(key, value))

In this case the combination that yields the lowest objective value is

Study statistics: 
  Number of finished trials:  100
Best trial:
  Value:  0.7740388512611389
  Params: 
    num_filters: 128
    dropout_rate: 0.98735655761383

Because the search is random, so every run will result in slightly different outcome, unless you seed the run.

Full colab notebook below. The notebook is modified from the Intro to Time Series Classification notebook.
https://colab.research.google.com/drive/1XMXR9oLaHmsJ7ubS4eSA2a3_vz_W9NrA?usp=sharing

[Waterkin]

learn.recorder.values returns [[train_loss, valid_loss, acc]]

Hi dnth, is the acc in learn.recorder.values valid accuracy, or train accuracy?

[dnth]

Valid :)

[Waterkin]

Valid :)

Thank for your reply, your answer really helps!

[dnth]

I'm glad it helped!

[vrodriguezf]

I use weights & biases for the hyperparameter tuning, I'm not sure how better/worse this is with respect to something like Optuna though. Wandb has a part of the api called sweeps, in which you can define, in a separate yaml file from the code, the configuration of the hyperparameter search, for example:

method: grid
metric:
  goal: maximize
  name: test_acc
name: architecture_comparison
parameters:
  arch_name:
    distribution: categorical
    values:
    - InceptionTime
    - ResNet
    - XCM
command:
  - papermill
  - ${program}
  - null
program: /home/victor/work/nbs/motion_classification_time_series.ipynb

What I do there is to change override the default command to run each trial of the sweep (which is by default python3) with papermill, a library to execute notebooks as if they were scripts. Then in program, instead of a .py file I put the notebook in which I am working on. That way I can do hyperparameter tuning without leaving the confort and the flexibility of playing with a Jupyter Notebook and without having to write extra code for the tuning.

[strakehyr]

Any news on this?
I am currently working on a WandB script to run a Bayesian Hyperparameter Optimization (way more efficient than a Grid search) via Sweeps. I believe I am close to making it work, but when passing the agent command, the agent doesn't seem to generate a value for each of the hyperparameters on the first sweep. The code is as follows:

from tsai.all import *
from fastai.callback.wandb import *
from sklearn.metrics import mean_squared_error, mean_absolute_error
import torch
import wandb

tfms  = [None, [TSRegression()]]
batch_tfms = TSStandardize(by_var=True)
dls = get_ts_dls(X, y, splits=splits, tfms=tfms, batch_tfms=batch_tfms, device = 'cuda')

sweep_config = {
    'method':'bayes',
    'parameters': {
        'epochs': {
            'value': 100
            },
        'n_layers': {
            'values':[1, 2, 3, 4, 5]
            },
        'd_model': {
            'values': [64, 96, 128, 160, 192]
            },
        'd_ff':{
            'values': [128, 192, 256, 320, 384]
            },
        'dropout': { 
            'values': [0, 0.3, 0.6]
            },
        'fc_dropout': { 
            'values': [0, 0.3, 0.6]
            }
        },
    'metric': {
        'name': 'val_loss',
        'goal': 'minimize'
               } 
    }
sweep_id = wandb.sweep(sweep_config, project="TSTPlus")
def train(dls, config=None):
    with wandb.init(config=config):
        config = wandb.config
        print(wandb.config)
        learn = ts_learner(dls, TSTPlus, n_layers = config.parameters['n_layers'], d_model = config.parameters['d_model'], 
                           n_heads = config.parameters['d_model']/8, d_ff = config.parameters['d_ff'], 
                           dropout = config.parameters['dropout'], fc_dropout = config.parameters['fc_dropout'], 
                           opt_func = SGD, loss_func = mse, metrics=[mae, mse], cbs=WandbCallback())
        lr = learn.lr_find()[0]
        learn.fit_one_cycle(config.parameters['epochs'], lr)

wandb.agent(sweep_id, train(config=config, dls = dls), count=5)

I get stuck at:

KeyError: 'parameters'

I have tried several ways to access parameters, such as config.n_layers for instance or config['n_layers'] .

[vrodriguezf]

config should already points to the paameters of the sweep, so there's no need to do config.parameters