GCP: InstanceTemplates are no longer global resources

[4wdonny]

Which component are you using?:
Cluster Autoscaler

cluster-autoscaler
What version of the component are you using?: 1.29.4

Component version: 1.29.4

What k8s version are you using (kubectl version)?:

v1.29.8-gke.1096000

kubectl version Output

$ kubectl version

What environment is this in?: Google

GCP has changed the location of instance templates...they used to be Global and now they are regional. Is this configurable in the Autoscaler?

[4wdonny]

Last night a node pool was upgraded to 1.29.8-gke.1096000 and the underlying instance template for the MIG changed from a global one to a regional one.

[4wdonny]

I've tried master, and even creating a new node spot node pool, and it looks like this is broken. The instanceTemplate regional logic is not being called correctly.

[Shubham82]

/area provider/gce
/area cluster-autoscaler

[Shubham82]

cc @BigDarkClown @x13n

[aaktaev]

Same for k8s=1.29.9-gke.1177000 and ClusterAutoscaler appVersion=v1.29.3:

Unable to build proper template node for https://www.googleapis.com/compute/v1/projects/$PROJECT/zones/europe-west1-b/instanceGroups/gke-k8s-system-grp: googleapi: Error 404: The resource 'projects/$PROJECT/global/instanceTemplates/gke-k8s-system-7777777' was not found, notFound

[x13n]

This has been fixed in CA 1.29 (#7231), but not 1.28. I just created1.28 cherrypick here: #7409

[4wdonny]

@x13n I attempted using CA 1.29 and encountered the same issue.

[x13n]

It was backported but not released yet. The release is happening right now - new CA patch versions for both 1.28 and 1.29 should work correctly with regional templates.