feat: change pod mutating webhook to fail open

[mikutas]

Issue

closes #2711

Description

Checklist

• Added tests that cover your change (if possible)
• Added/modified documentation as required (such as the README.md, or the docs directory)
• Manually tested
• Made sure the title of the PR is a good description that can go into the release notes

BONUS POINTS checklist: complete for good vibes and maybe prizes?! 🤯

• Backfilled missing tests for code in same general area 🎉
• Refactored something and made the world a better place 🌟

[k8s-ci-robot]

Hi @mikutas. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[johngmyers]

The linked issue #2711 is specific to the PodReadinessGate mutating webhook for Pods. It makes sense to have that webhook fail open, as failing closed can easily cause persistent lack of availability and failing open can only cause an intermittent lack of availability.

This PR, however, creates a single configuration option for the failure policy of all webhooks, not just the PodReadinessGate. Other webhooks, especially the validating ones, are much more critical. So that's a NACK from me.

The Helm chart should just configure the PodReadinessGate mutating webhook to fail open. There is no need for it to be configurable.

[johngmyers]

Why does this have to be configurable? The chart should just configure failurePolicy: Ignore.

[johngmyers]

/ok-to-test
/lgtm
/assign @kishorj

[johngmyers]

/retest

[mikutas]

/retest

[johngmyers]

/lgtm

[johngmyers]

@mikutas please update the comment at https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/webhooks/core/pod_mutator.go#L46 and run make crds.

/lgtm cancel

[johngmyers]

/retest

[johngmyers]

/lgtm

[kawai-kobold]

@johngmyers It looks like the approval label wasn't applied, can you check? We'd love to have this fix in our environment, the webhook failures make most of our GitOps deploys into EKS a real pain.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[mikutas]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[kawai-kobold]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[kawai-kobold]

/remove-lifecycle stale

[seanturner026]

I am truly sorry to be a pest here, but, this is a small PR and was approved previously per the below comment. Is there anything needed to get this out? This issue just caused an outage

#3261 (comment)

@johngmyers @kishorj @M00nF1sh

[seanturner026]

Apologies, looks like this is already implemented #3653

[k8s-ci-robot]

New changes are detected. LGTM label has been removed.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mikutas
Once this PR has been reviewed and has the lgtm label, please ask for approval from johngmyers. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mikutas]

This issue is about pod mutating webhook
#3653 is about service mutating webhook