Add job for graceful node shutdown feature

[torredil]

This PR introduces a new job to validate the graceful node shutdown feature:

• Configured to run only tests labeled with [Feature:GracefulNodeShutdown].
• The job uses the SHUTDOWN_GRACE_PERIOD & SHUTDOWN_GRACE_PERIOD_CRITICAL_PODS environment variables to enable the graceful shutdown feature, see Conditionally add the graceful shutdown Kubelet parameters kubernetes#125413 (comment)
• optional and does not always run by default.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: torredil
Once this PR has been reviewed and has the lgtm label, please assign cheftako for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• config/jobs/kubernetes/sig-cloud-provider/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[torredil]

The regex change made in pull-kubernetes-e2e-gce ensures that the e2e test implemented in PR 125070 is skipped (graceful node shutdown is not enabled for that job)

[dims]

/assign @mrunalp

[pacoxu]

/cc @wzshiming @bobbypage

[aojea]

/hold

is not clear to me the direction of these changes, the feature does not seem to have e2e tests (only e2e_node) and makes some existing tests to be skipped

[torredil]

@aojea

the feature does not seem to have e2e tests (only e2e_node)

That is correct - today, NodeFeature:GracefulNodeShutdown does not have any e2e tests, only e2e_node tests. The purpose of this PR is to enable writing full cluster e2e tests for that feature - there are no jobs which currently create a test cluster with the Kubelet graceful shutdown feature enabled.

see this PR for more context: kubernetes/kubernetes#125070. I synced up with @msau42 offline and we agreed to implement the e2e test written in that PR under e2e. That test assumes that graceful shutdown is enabled on the node. It cannot be part of e2e_node because it relies on setting up CSI drivers, provisioning volumes, and so on, which is not possible to do in the limited test environment created by e2e_node.

[bobbypage]

That is correct - today, NodeFeature:GracefulNodeShutdown does not have any e2e tests, only e2e_node tests. The purpose of this PR is to enable writing full cluster e2e tests for that feature - there are no jobs which currently create a test cluster with the Kubelet graceful shutdown feature enabled.

Sorry, I might be missing some context here but what is the purpose of having an e2e test for this vs relying on the existing node e2e tests? All of the logic for graceful shutdown is node specific so I'm trying to understand what additional signal a cluster e2e would provide.

[torredil]

@bobbypage, the node specific logic is already covered by e2e_node tests as you mention, but we need to validate how the shutdown process interacts with other cluster components like the A/D controller and CSI drivers.

For example, users often observe 6 minute delays for stateful pods to enter a running state after a node is gracefully terminated, due to having to wait for the A/D controller to issue a force detach if volumes were not unmounted in time.

^ kubernetes/kubernetes#125070 addresses that delay by taking volume status into account before proceeding with termination and it includes an e2e test to validate that stateful pods enter a running state in a timely manner (as expected when nodes are gracefully terminated). My understanding is that this level of validation is not possible with e2e_node tests alone.

[torredil]

cc: @bobbypage for review.