Why does plot_confusion_matrix() give different results the 2nd time it's called?

[ebagjones]

I've noticed that if I call the learn.plot_confusion_matrix() 3 times, the 1st time is different from the 2nd and 3rd times.
Is this expected?

[oguiza]

I'm not sure what's causing that issue.
I've tested it with both the pip release and the GitHub code and it works well.
Here's the code I've used:

from tsai.all import *
X, y, splits = get_UCR_data('MoteStrain', split_data=False)
tfms = [None, TSClassification()]
batch_tfms = TSStandardize(by_sample=True)
dls = get_ts_dls(X, y, splits=splits, tfms=tfms, batch_tfms=batch_tfms)
learn = ts_learner(dls, InceptionTimePlus, metrics=[accuracy, RocAucBinary(), APScoreBinary()], cbs=[ShowGraph()])
learn.fit_one_cycle(10, 1e-2)
learn.plot_confusion_matrix()
learn.plot_confusion_matrix()

[ebagjones]

Thanks for checking @oguiza .
administrator:~$ pip3 install tsai -U Requirement already up-to-date: tsai in ./.local/lib/python3.8/site-packages (0.3.0)

Interestingly, when I run it on the MoteStrain dataset, I get consistent results, but when I run it with my own data (the only change is where X, y, splits come from) then I have similar but inconsistent confusion matrices. Is there some randomness/non-determinism to how this is calculated?

[oguiza]

The only cause for randomness would be if you pass set learn.plot_confusion_matrix(ds_idx=0) or pass a dl using learn.plot_confusion_matrix(dl=train_dk). If the dataloader has shuffle set to True, it would create different batches per run.
Otherwise, it should always deliver the same output.