Exposing the webhook receiver using GCE Ingress

[valorl]

Since Flux notification-controller serves the webhook and health/readiness endpoints on different ports (9292 and 9440 respectively), GCE Ingress requires a custom configuration.

BackendConfig

An Ingress rule handled by the GKE-native ingress-gce controller translates into a load balancer backend service. Each of these needs to have a health check (i.e. a GCP-side health check, separate from standard K8S probes) configured.

By default there's no need to configure this explicitly, the health path will be inferred from the K8S native readiness probe in the serving Pod's spec (assuming it has been defined). However, this is only inferred if the probe is defined with the same port as the port being served. Since this is not the case for the notification controller, the resulting backend service health check will be defaulted to / on the serving port (9292 in this case). Since the webhook port does not respond 200 on /, the backend will never become healthy.

To fix this, we can apply a BackendConfig specifying the health check port:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: webhook-receiver
  namespace: flux-system
spec:
  healthCheck:
    requestPath: /readyz
    type: HTTP
    port: 9440

The BackendConfig needs to be explicitly tied to the Service being exposed on the Ingress, e.g. via a kustomize patch:

patches:
- target:
    kind: Service
    name: webhook-receiver
  patch: |-
    apiVersion: v1
    kind: Service
    metadata:
      name: webhook-receiver
      annotations:
        cloud.google.com/neg: '{"ingress": true}'
        beta.cloud.google.com/backend-config: '{"default": "webhook-receiver"}'

Ingress definition, for completeness:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: webhook-receiver
  namespace: flux-system
  annotations:
    kubernetes.io/ingress.class: gce
spec:
  rules:
  - host: flux-webhooks.example.com
    http:
      paths:
      - pathType: Prefix
        path: /hook/
        backend:
          service:
            name: webhook-receiver
            port:
              number: 80

Network Policy

In a default flux installation, the network policy (allow-webhooks) to allow ingress to the notification controller only allows traffic from other Pods but not from outside the cluster. This needs to be changed to allow traffic from Google LB probe ranges.