Understanding Feature Importance/Engineering

[ebagjones]

Are there any tools or recommended approaches to determining feature importance with tsai? I'm particularly interested in a method that would work with (mini)ROCKET but I'm all ears if anyone knows of a good approach with the other algorithms. Thanks and Regards.

[vrodriguezf]

Do you mean the importance of the features generated by ROCKET or the the importance with respect to the original time series variables? In case it's the first, it's simple (any tabular approach with good interpretability such as xgb can do it), but I'm not sure if it is useful at all since the features are random kernels.

If you are interested in interpretability I wouldn't go with ROCKET. Instead, you can try NN architectures like XCM in which you can create activation maps in both the time and the variable dimensions, which allows for a sort of heatmap that could give you the feature importance.

[ebagjones]

@vrodriguezf thanks for the reply. I'm interested in knowing the importance of the original time-series signals. I would assume that the time series signals may be ranked differently importance or signal contribution vs a stationary snapshot of a single point like I'd get with a RF or xgboost classification. Any guidance on this would be much appreciated.

[vrodriguezf]

Have a look at get_attribution_map, there you can pass data to a trained NN, and get a "heatmap" of how each point in the sequence contributes to the activations and gradients of one specific layer. Normally you would use it for the last module of the backbone of the NN, in Plus models that have backbone and head.

[ebagjones]

@vrodriguezf this is very cool, thanks for pointing me in the right direction!

[oguiza]

Hi @jefferyanderson and @vrodriguezf,
I've just released some new code to allow you to plot feature importance using permutation feature importance (issue #286). This shows the drop in the model's performance when a single feature is shuffled.
It's important to note that it doesn't show the importance of the feature along the time axis.
To use this new functionality all you need to do is this:

X, y, splits = get_UCR_data('LSST', split_data=False)
tfms = [None, TSClassification()]
batch_tfms = TSStandardize(by_sample=True)
dls = get_ts_dls(X, y, splits=splits, tfms=tfms, batch_tfms=batch_tfms)
learn = ts_learner(dls, InceptionTimePlus, metrics=accuracy, cbs=[ShowGraph()])
learn.fit_one_cycle(10, 1e-2)
learn.feature_importance()

If you try it please, let me know if it works well.

[ebagjones]

@oguiza this is an awesome feature, thanks again for adding it.

[ebagjones]

@oguiza the new feature importance looks awesome! Thank you.

[oguiza]

Thanks a lot for your feedback @jefferyanderson ! It's much appreciated 🤩

[strakehyr]

Just wanted to give my kudos to the team for the step_importance() feature that I recently found out. Really nice.

[oguiza]

Thanks, @strakehyr! 🙂