Ask users if they'd like to have default taints added to the user and worker node groups of Nebari for potential cost savings

[Adam-D-Lewis]

Context

#2605 adds taints to user nodes to fix issues in the past where node groups won't scale down. This is a nice benefit and would result in cost savings for most users. However most prior users won't know about the change unless we prompt them about it during the upgrade process.

Value and/or benefit

good defaults for existing users resulting in cost savings

Anything else?

No response

[Adam-D-Lewis]

Keep in mind that other user node groups (e.g. gpus, argo workflows using jupyterflow-override) should add the user taint to other user node profiles in order to prevent the same issue this PR prevents.

[viniciusdc]

@Adam-D-Lewis, will #2605 address this consideration?

[Adam-D-Lewis]

This comment addresses this need since taints will be applied by default. We should add a note in the upgrade step to saying nodes will be restarted during the upgrade though.