Rollouts Upgrade from v0.10.2 to v1.0.2 causes new pods to be created due to preserveUnknownFields

[nextrevision]

When I upgrade from v0.10.2 to v1.0.2 using kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/download/v1.0.2/install.yaml, all rollouts (blue/green) immediately become out of sync and new pods are created using the same image. This causes quite the thrashing in our environments since autoPromotionEnabled is set to false and essentially container count gets doubled.

Per the release notes

To avoid the Argo CD OutOfSync conditions, you can remove spec.preserveUnknownFields from the manifests
entirely after upgrading to v1.0.

We are using the Rollout CRD (not Application) without ArgoCD. If I upgrade, then immediately "remove spec.preserveUnknownFields from the [CRD] manifests", the same thing still happens.

What is the right way to upgrade to v1.0.2 and avoid creating duplicate pods without ArgoCD?

[jessesuen]

@nextrevision if your blue-green rollouts deployed an unsolicited update, I think you may have been hit by this bug #1367. Were you using service account in the affected rollouts? The fix for this is in v1.0.3

[nextrevision]

Very well could have been. We ended up just taking the hit in an after hours session. That sounds like the issue I was seeing.