Annotation: Make `proxy-ssl-secret` optional. #11490

g1franc · 2024-06-20T06:45:15Z

Update main.go to make proxy-ssl-secret annotation optional usage of other proxy-ssl-* annotations.

What this PR does / why we need it:

Currently, if the annotation proxy-ssl-secret is not provided, it is not possible to use other proxy-ssl-* annotations. For example, it prevent to set only proxy-ssl-protocols to specify the TLS protocol to connect to the upstream backend.
This change aim to make the annotation proxy-ssl-secret optional by preventing an exit from the method in case of it not being specified. I don't alter the current behavior in the regard that if that an invalid value if specified, no other proxy-ssl-* annotations will be processed.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
CVE Report (Scanner found CVE and adding report)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation only

Which issue/s this PR fixes

fixes #10264

How Has This Been Tested?

Tested in local dev cluster and also with tested regression with FOCUS='proxyssl' make kind-e2e-test

Checklist:

My change requires a change to the documentation.
I have updated the documentation accordingly.
I've read the CONTRIBUTION guide
I have added unit and/or e2e tests to cover my changes.
All new and existing tests passed.

… usage of other proxy-ssl-* annotations

k8s-ci-robot · 2024-06-20T06:45:24Z

Hi @g1franc. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

netlify · 2024-06-20T06:45:37Z

✅ Deploy Preview for kubernetes-ingress-nginx ready!

Name	Link
🔨 Latest commit	`fcae7d2`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-ingress-nginx/deploys/6673cffd0bbda10008dba8a0
😎 Deploy Preview	https://deploy-preview-11490--kubernetes-ingress-nginx.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

k8s-ci-robot · 2024-06-20T06:45:40Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: g1franc
Once this PR has been reviewed and has the lgtm label, please assign puerco for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Gacko

/retitle Annotation: Make proxy-ssl-secret optional.
/triage accepted
/kind bug
/priority backlog

Assuming that you only changed the indentation of most of the lines: Could you please follow the linter recommendation and revert the change in indentation?

https://github.com/kubernetes/ingress-nginx/pull/11490/files#diff-4755dd545c8c3c3407d1172ed7f03457a5e462d974f9b0b9701238a9cacb1944R195

Apart from that your change looks good.

Gacko · 2024-07-01T10:01:14Z

/cherry-pick release-1.10

k8s-infra-cherrypick-robot · 2024-07-01T10:01:17Z

@Gacko: once the present PR merges, I will cherry-pick it on top of release-1.10 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Gacko · 2024-07-01T10:02:11Z

Oh and please also rebase your code on top of main - thank you!

Gacko · 2024-07-01T10:04:11Z

The failing CI / Build step is on our behalf. Do not worry about that.

Gacko · 2024-07-01T10:05:04Z

/ok-to-test

longwuyuan · 2024-07-01T10:47:20Z

@g1franc why would you set proxy-ssl-protocols if you are not configuring a secrert ?

@Gacko why would we accept setting TLS-Protocols optionally and that too without a secret for a certificate to begin with ?

Its possible I am completely lost here. @g1franc so is there any chance that you can actually create a kind/minikube cluster and do the setting of the proxy-protocol and then send TLS traffic all the way through from a client outside the cluster to the upstream inside clusrter. If you copy paste all the kubectl describe command outputs for every single related resource in the kubernetes cluster, and show some kind of packet capture etc, this MAJOR change suggested would be review of possible impacts and security etc etc

g1franc · 2024-07-01T11:07:43Z

@longwuyuan I'm configuring proxy-ssl-protocol because the backend is using TLS (HTTPS) but I don't need to provide a secret for the TLS client certificate as the backend don't use one to authentificate the client.
Here the backend is only listening in TLSv1.3 but it seems that nginx is trying to connect using TLSv1.2 and when it failed because the backend don't use TLSv1.2 at all, it doesn't fallback to trying TLSv1.3.
Therefore, the only way for nginx to connect to the backend successfully, is to provide proxy-ssl-protocol: TLSv1.3

Error when trying to use TLSv1.2 instead of 1.3 is

[error] 45203#45203: *387489720 SSL_do_handshake() failed (SSL: error:0A00042E:SSL routines::tlsv1 alert protocol version:SSL alert number 70) while SSL handshaking to upstream, client: 10.134.53.41, server: xxxxxx.xxxxxxxxx.xxxxxxxxxx, request: "GET / HTTP/1.1", upstream: "https://100.74.47.6:8081/", host: "xxxxxx.xxxxxxxxx.xxxxxxxxxx", referrer: "https://my.refferer.com/"

longwuyuan · 2024-07-01T11:17:20Z

@g1franc thank you for explaining. Makes sense now.

But controller does not try TLS v1.3 then should we not check why controller does not try TLS v1.3 ? The choice to make the existence of the TLS certificate optional, seems completely NOT related to the version of the protocol being used for TLS.

I am not a developer so I am taking this at face value. I will check the code changes now and see if I can make any sense of it. But my concern comes from arbitrarily deciding to change something that is not at all broken.

longwuyuan · 2024-07-01T12:21:17Z

@g1franc thank for sharing screen on meet.jit.si and showing the problem and the use-case. I have thought about it and I see that the CA related information is also needed in that secret (this data below screenshot)

so I am very confused about this because you said you don't want to use a client-certificate but only want to use "backend-protocol: HTTPS" and so you should provide the secret with the self-signed CA or the HTTPS certificate of the backend-service/pod should be signed by a well-known CA like Letsencrypt/Geotrust etc. , who have a better chance of being known, when compared to a self-signed CA.

So is this potentially making it easy to use "backend-protocol" HTTPS" without the controller needing to validate the CA of the HTTPS cert, that the backend pod's HTTPS server is presenting.

staizen-stephen · 2024-07-01T13:02:33Z

I think part of the confusion here is mixing several different behaviours, specifically in the proxy case (i.e. when nginx is acting as a client to an upstream):

Client certificate authentication is increasingly used with service meshes, but it is by no means the only or most normal case for for TLS communication. Largely, it represents authentication. And in this case there is a secret (my identity proof).
TLS communication is always used (irrespective of (1)) for encryption in-transit of messages. This has value irrespective of further enhancements, but in most cases it won't be a sufficient level of security
Almost always you want some form of channel verification from a CLIENT's perspective. This takes on several increasingly secure cases:
1. Verification of certificate non-expiry (just a valid certificate in isolation)
2. Verification of certificate SNI (domain or IP address)
3. Verification of certificate issuer (trusted CA)
4. Verification of certificate issuer and checking revocation

(3)(i) and (3)(ii) do not require any trust information about a CA. (3)(iii) and (3)(iv) require at least that the CA can be verified. Public CAs aren't an option for many private clusters, especially given the cost of public CA certificates and no external route for automated verification.

(3)(iii) and (3)(iv) can also use the installed host-level CA verification as a baseline. Adding additional configuration should only be necessary if those cannot be used or if further restriction is needed.

But I think the biggest factor here is that (1) and (3) have been conflated. To be very precise: a certificate does not need to be a secret. It's only a secret if it contains a private key. And that case only occurs with client authentication.

longwuyuan · 2024-07-03T19:07:26Z

@g1franc I did a test and I can't reproduce the problem you mentioned

Created a docker image with nginx, self-signed ssl cert, and only TLS1.3 in nginx.conf

--------------------------------
FileName: 40-create-ssl-cert.sh
--------------------------------
#!/bin/sh
# vim:sw=4:ts=4:et
#

openssl req -x509 -newkey rsa:4096 -sha256 -days 365 \
	-nodes -keyout /etc/nginx/dev.mydomain.com.key -out /etc/nginx/dev.mydomain.com.crt -subj "/CN=dev.mydomain.com" \
	-addext "subjectAltName=DNS:dev.mydomain.com,DNS:*.dev.mydomain.com"

-----------------------
FileName: default.conf
-----------------------
server {
    listen       443 ssl;
    http2 on;
    server_name  dev.mydomain.com;

    ssl_certificate /etc/nginx/dev.mydomain.com.crt;
    ssl_certificate_key /etc/nginx/dev.mydomain.com.key;
    ssl_protocols TLSv1.3;

    #access_log  /var/log/nginx/host.access.log  main;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    # proxy the PHP scripts to Apache listening on 127.0.0.1:80
    #
    #location ~ \.php$ {
    #    proxy_pass   http://127.0.0.1;
    #}

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
    #location ~ \.php$ {
    #    root           html;
    #    fastcgi_pass   127.0.0.1:9000;
    #    fastcgi_index  index.php;
    #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
    #    include        fastcgi_params;
    #}

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    #location ~ /\.ht {
    #    deny  all;
    #}
}

---------------------
FileName: Dockerfile
---------------------
FROM nginx:alpine

RUN apk add -u openssl
COPY 40-create-ssl-cert.sh /docker-entrypoint.d/40-create-ssl-cert.sh 
COPY default.conf /etc/nginx/conf.d/default.conf
COPY index.html /usr/share/nginx/html/index.html

EXPOSE 443

---------------------
FileName: index.html
---------------------
index.html of nginx with only TLSv1.3 for *.dev.mydomain.com

Created pod with that image

Name:             nginx-with-tlsv13-only-66bbf76df8-8fhdh
Namespace:        default
Priority:         0
Service Account:  default
Node:             kind-control-plane/172.18.0.2
Start Time:       Wed, 03 Jul 2024 23:16:41 +0530
Labels:           app=nginx-with-tlsv13-only
                  pod-template-hash=66bbf76df8
Annotations:      <none>
Status:           Running
IP:               10.244.0.24
IPs:
  IP:           10.244.0.24
Controlled By:  ReplicaSet/nginx-with-tlsv13-only-66bbf76df8
Containers:
  4c71ac42eac46:
    Container ID:   containerd://769068992e358121e07e2320d9720771a9a899886a5bf545ca3ca4089bbbf734
    Image:          docker.io/library/nginx-with-tlsv13-only:latest
    Image ID:       docker.io/library/import-2024-07-03@sha256:f57151fa8a5650c5ca65278481a21cbd01dec97612123cc59e506e1aeeab94a5
    Port:           443/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Wed, 03 Jul 2024 23:16:41 +0530
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mwl25 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  kube-api-access-mwl25:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  43m   default-scheduler  Successfully assigned default/nginx-with-tlsv13-only-66bbf76df8-8fhdh to kind-control-plane
  Normal  Pulled     43m   kubelet            Container image "docker.io/library/nginx-with-tlsv13-only:latest" already present on machine
  Normal  Created    43m   kubelet            Created container 4c71ac42eac46
  Normal  Started    43m   kubelet            Started container 4c71ac42eac46

Created service for that pod

Name:              nginx-with-tlsv13-only
Namespace:         default
Labels:            app=nginx-with-tlsv13-only
Annotations:       <none>
Selector:          app=nginx-with-tlsv13-only
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.96.34.64
IPs:               10.96.34.64
Port:              <unset>  443/TCP
TargetPort:        443/TCP
Endpoints:         10.244.0.24:443
Session Affinity:  None
Events:            <none>

Created ingress for that svc
% k get ing
NAME CLASS HOSTS ADDRESS PORTS AGE
nginx-with-tlsv13-only nginx nginx-tlv13.dev.enjoydevops.com 172.18.0.2 80, 443 48m

Name:             nginx-with-tlsv13-only
Labels:           <none>
Namespace:        default
Address:          172.18.0.2
Ingress Class:    nginx
Default backend:  <default>
TLS:
  SNI routes nginx-tlsv13.dev.enjoydevops.com
Rules:
  Host                             Path  Backends
  ----                             ----  --------
  nginx-tlv13.dev.enjoydevops.com  
                                   /   nginx-with-tlsv13-only:443 (10.244.0.24:443)
Annotations:                       nginx.ingress.kubernetes.io/backend-protocol: HTTPS
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    34m (x4 over 40m)  nginx-ingress-controller  Scheduled for sync

Note that the backend-protocol annotation is set to a value HTTPS
HTTP request gets 308 response because TLS section in ingress has my letsencrypt cert and so there is forced redirection to HTTPS

% curl nginx-tlv13.dev.enjoydevops.com
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>
[~/Documents/ingressnginxwork/nginx-only-tls13]

Inside pod, its only listening on 443 TLS

% k exec -ti nginx-with-tlsv13-only-66bbf76df8-8fhdh -- sh
/ # netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      1/nginx: master pro
netstat: /proc/net/tcp6: No such file or directory
/ #

HTTPS request makes controler use TLS v1.3 for HTTPS to backend

% curl nginx-tlv13.dev.enjoydevops.com -L
index.html of nginx with only TLSv1.3 for *.dev.mydomain.com
[~]

curl with -v

*   Trying 172.18.0.2:80...
* Connected to nginx-tlv13.dev.enjoydevops.com (172.18.0.2) port 80 (#0)
> GET / HTTP/1.1
> Host: nginx-tlv13.dev.enjoydevops.com
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Date: Wed, 03 Jul 2024 18:51:17 GMT
< Content-Type: text/html
< Content-Length: 164
< Connection: keep-alive
< Location: https://nginx-tlv13.dev.enjoydevops.com
< 
* Ignoring the response-body
* Connection #0 to host nginx-tlv13.dev.enjoydevops.com left intact
* Clear auth, redirects to port from 80 to 443
* Issue another request to this URL: 'https://nginx-tlv13.dev.enjoydevops.com/'
*   Trying 172.18.0.2:443...
* Connected to nginx-tlv13.dev.enjoydevops.com (172.18.0.2) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.dev.enjoydevops.com
*  start date: May 11 02:24:00 2024 GMT
*  expire date: Aug  9 02:23:59 2024 GMT
*  subjectAltName: host "nginx-tlv13.dev.enjoydevops.com" matched cert's "*.dev.enjoydevops.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x63a95e6edeb0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: nginx-tlv13.dev.enjoydevops.com
> user-agent: curl/7.81.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200 
< date: Wed, 03 Jul 2024 18:51:17 GMT
< content-type: text/html
< content-length: 61
< last-modified: Wed, 03 Jul 2024 17:35:04 GMT
< etag: "66858bc8-3d"
< accept-ranges: bytes
< strict-transport-security: max-age=31536000; includeSubDomains
< 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #1 to host nginx-tlv13.dev.enjoydevops.com left intact

tcpdump inside the backend pod

/ # tcpdump -n not arp                                                                                                                                                                  
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode                                                                                                               
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes                                                                                                            
18:55:22.877727 IP 10.244.0.17.59692 > 10.244.0.24.443: Flags [S], seq 140612690, win 64240, options [mss 1460,sackOK,TS val 1816073344 ecr 0,nop,wscale 7], length 0                   
18:55:22.877743 IP 10.244.0.24.443 > 10.244.0.17.59692: Flags [S.], seq 4176706456, ack 140612691, win 65160, options [mss 1460,sackOK,TS val 3712965341 ecr 1816073344,nop,wscale 7], l
ength 0                                                                                                                                                                                 
18:55:22.877766 IP 10.244.0.17.59692 > 10.244.0.24.443: Flags [.], ack 1, win 502, options [nop,nop,TS val 1816073344 ecr 3712965341], length 0                                         
18:55:22.878091 IP 10.244.0.17.59692 > 10.244.0.24.443: Flags [P.], seq 1:294, ack 1, win 502, options [nop,nop,TS val 1816073344 ecr 3712965341], length 293
18:55:22.878100 IP 10.244.0.24.443 > 10.244.0.17.59692: Flags [.], ack 294, win 507, options [nop,nop,TS val 3712965341 ecr 1816073344], length 0
18:55:22.885336 IP 10.244.0.24.443 > 10.244.0.17.59692: Flags [P.], seq 1:2171, ack 294, win 507, options [nop,nop,TS val 3712965349 ecr 1816073344], length 2170
18:55:22.885382 IP 10.244.0.17.59692 > 10.244.0.24.443: Flags [.], ack 2171, win 501, options [nop,nop,TS val 1816073352 ecr 3712965349], length 0
18:55:22.886481 IP 10.244.0.17.59692 > 10.244.0.24.443: Flags [P.], seq 294:374, ack 2171, win 501, options [nop,nop,TS val 1816073353 ecr 3712965349], length 80
18:55:22.886594 IP 10.244.0.17.59692 > 10.244.0.24.443: Flags [P.], seq 374:735, ack 2171, win 501, options [nop,nop,TS val 1816073353 ecr 3712965349], length 361
18:55:22.886744 IP 10.244.0.24.443 > 10.244.0.17.59692: Flags [P.], seq 2171:2442, ack 735, win 504, options [nop,nop,TS val 3712965350 ecr 1816073353], length 271
18:55:22.886855 IP 10.244.0.24.443 > 10.244.0.17.59692: Flags [P.], seq 2442:2713, ack 735, win 504, options [nop,nop,TS val 3712965350 ecr 1816073353], length 271
18:55:22.886894 IP 10.244.0.17.59692 > 10.244.0.24.443: Flags [.], ack 2713, win 501, options [nop,nop,TS val 1816073353 ecr 3712965350], length 0
18:55:22.887040 IP 10.244.0.24.443 > 10.244.0.17.59692: Flags [P.], seq 2713:3032, ack 735, win 504, options [nop,nop,TS val 3712965350 ecr 1816073353], length 319
18:55:22.927297 IP 10.244.0.17.59692 > 10.244.0.24.443: Flags [.], ack 3032, win 501, options [nop,nop,TS val 1816073394 ecr 3712965350], length 0

This URL says default protocol is TLS 1.3 since Nginx version 1.23.4 https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_protocols and not TLS v1.2 but you are experiencing TLS default version of v1.2 as per the jitsi meeting info. We have Nginx v1.2.5.3 in the controller so default TLS protocol version is v1.3 and not v1.2

So what am I missing. Please help me understand why you think that controller is using TLS v1.2

longwuyuan · 2024-07-03T20:21:19Z

If you are interested, I can join another meeting on jitsi to do tpdump of traffic from controller to backend.

g1franc · 2024-07-04T06:56:20Z

Hello @longwuyuan,

Maybe I misunderstood your test, but it seems that you tested NGINX serving TLSv1.3 while the issue is about the Operator configuring NGINX to proxify to an TLSv1.3 backend.

So to make the test relevant, you can keep your container to act as backend TLSv1.3 only + you need a second nginx to proxify to this first container, using proxy-ssl-protocol property.

longwuyuan · 2024-07-04T08:32:57Z

@g1franc thank you very much for your comment because it adds more light to the use case and proves that my assumption of the use case is not accurate.

I need your help to get the description of your use-case in much much more clear and very accurate details. I am going to write some description below. Please correct it in your response.

client laptop <--HTTPS --> LB <--HTTPS--> ingress-controller <-- HTTPS --> backend

strongjz · 2024-07-04T16:52:12Z

I think part of the confusion here is mixing several different behaviours, specifically in the proxy case (i.e. when nginx is acting as a client to an upstream):
1. Client certificate authentication is increasingly used with service meshes, but it is by no means the only or most normal case for for TLS communication. Largely, it represents authentication. And in this case there is a secret (my identity proof).

2. TLS communication is always used (irrespective of (1)) for encryption in-transit of messages. This has value irrespective of further enhancements, but in most cases it won't be a sufficient level of security

3. Almost always you want some form of channel verification from a CLIENT's perspective. This takes on several increasingly secure cases:
   
   1. Verification of certificate non-expiry (just a valid certificate in isolation)
   2. Verification of certificate SNI (domain or IP address)
   3. Verification of certificate issuer (trusted CA)
   4. Verification of certificate issuer and checking revocation
(3)(i) and (3)(ii) do not require any trust information about a CA. (3)(iii) and (3)(iv) require at least that the CA can be verified. Public CAs aren't an option for many private clusters, especially given the cost of public CA certificates and no external route for automated verification.

(3)(iii) and (3)(iv) can also use the installed host-level CA verification as a baseline. Adding additional configuration should only be necessary if those cannot be used or if further restriction is needed.

But I think the biggest factor here is that (1) and (3) have been conflated. To be very precise: a certificate does not need to be a secret. It's only a secret if it contains a private key. And that case only occurs with client authentication.

Something like this needs to be documented on why or not a proxy ssl secret is needed.

longwuyuan · 2024-07-05T17:46:47Z

Hello @longwuyuan,

Maybe I misunderstood your test, but it seems that you tested NGINX serving TLSv1.3 while the issue is about the Operator configuring NGINX to proxify to an TLSv1.3 backend.

So to make the test relevant, you can keep your container to act as backend TLSv1.3 only + you need a second nginx to proxify to this first container, using proxy-ssl-protocol property.

Hi @g1franc ,

Could you kindly review my test one more time.

If you read my backend image Dockerfile, you will see the image only exposes HTTPS on port 443
If you read my backend image default.conf file, the nginx directive is set for only TLSv1.3
If you see the backend-protocol from controller to backend is HTTPS via annotation
So I have tried to configure my test for controller-to-backend traffic using only TLSv1.3

If my config is wrong, please let me know and I will change and retest.

Also note the nginx docs link that says default TLS protocol version is v1.3 since Nginx v1.24.x and not v1.2. The controler uses Nginx v1.25.x internally. So do you think we need to check your cluster & controller to understand why its using default TLS protocol as v1.2.

longwuyuan · 2024-07-06T01:23:55Z

@g1franc I am also adding some proof that my backend can only serve TLSv1.3 and not TLSv1.2

% k get po -o wide
NAME                       READY   STATUS    RESTARTS   AGE     IP            NODE       NOMINATED NODE   READINESS GATES
mynginx-7f6d59d464-f4csm   1/1     Running   0          7h48m   10.244.0.16   minikube   <none>           <none>
[~] 
% 

/etc/nginx $ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 62:A0:6C:16:64:14  
          inet addr:10.244.0.12  Bcast:10.244.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:675912 errors:0 dropped:0 overruns:0 frame:0
          TX packets:574701 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:214686598 (204.7 MiB)  TX bytes:335023099 (319.5 MiB)

/etc/nginx $ curl https://10.244.0.16 -k --tls-max 1.0
curl: (35) OpenSSL/3.1.4: error:0A0000BF:SSL routines::no protocols available
/etc/nginx $ curl https://10.244.0.16 -k --tls-max 1.1
curl: (35) OpenSSL/3.1.4: error:0A0000BF:SSL routines::no protocols available
/etc/nginx $ curl https://10.244.0.16 -k --tls-max 1.2
curl: (35) OpenSSL/3.1.4: error:0A00042E:SSL routines::tlsv1 alert protocol version
/etc/nginx $ curl https://10.244.0.16 -k --tls-max 1.3
You reached nginx using TLSv1.3
/etc/nginx $

Update main.go to make proxy-ssl-secret annotation optional regarding…

fcae7d2

… usage of other proxy-ssl-* annotations

k8s-ci-robot added needs-priority size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 20, 2024

k8s-ci-robot requested review from Gacko and strongjz June 20, 2024 06:45

Gacko requested changes Jul 1, 2024

View reviewed changes

k8s-ci-robot changed the title ~~Update main.go to make proxy-ssl-secret annotation optional~~ Annotation: Make proxy-ssl-secret optional. Jul 1, 2024

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 1, 2024

longwuyuan mentioned this pull request Jul 3, 2024

Allow proxy-ssl-* annotations without proxy-ssl-secret #10557

Open

10 tasks

longwuyuan mentioned this pull request Jul 3, 2024

Annotations: Consume proxy-ssl-name and proxy-ssl-server-name even without proxy-ssl-secret. #6728

Open

Gacko self-assigned this Jul 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Annotation: Make `proxy-ssl-secret` optional. #11490

Annotation: Make `proxy-ssl-secret` optional. #11490

g1franc commented Jun 20, 2024

k8s-ci-robot commented Jun 20, 2024

netlify bot commented Jun 20, 2024 •

edited

Loading

k8s-ci-robot commented Jun 20, 2024

Gacko left a comment

Gacko commented Jul 1, 2024

k8s-infra-cherrypick-robot commented Jul 1, 2024

Gacko commented Jul 1, 2024

Gacko commented Jul 1, 2024

Gacko commented Jul 1, 2024

longwuyuan commented Jul 1, 2024

g1franc commented Jul 1, 2024

longwuyuan commented Jul 1, 2024

longwuyuan commented Jul 1, 2024

staizen-stephen commented Jul 1, 2024

longwuyuan commented Jul 3, 2024

longwuyuan commented Jul 3, 2024

g1franc commented Jul 4, 2024

longwuyuan commented Jul 4, 2024

strongjz commented Jul 4, 2024

longwuyuan commented Jul 5, 2024 •

edited

Loading

longwuyuan commented Jul 6, 2024

Annotation: Make proxy-ssl-secret optional. #11490

Are you sure you want to change the base?

Annotation: Make proxy-ssl-secret optional. #11490

Conversation

g1franc commented Jun 20, 2024

What this PR does / why we need it:

Types of changes

Which issue/s this PR fixes

How Has This Been Tested?

Checklist:

k8s-ci-robot commented Jun 20, 2024

netlify bot commented Jun 20, 2024 • edited Loading

✅ Deploy Preview for kubernetes-ingress-nginx ready!

k8s-ci-robot commented Jun 20, 2024

Gacko left a comment

Choose a reason for hiding this comment

Gacko commented Jul 1, 2024

k8s-infra-cherrypick-robot commented Jul 1, 2024

Gacko commented Jul 1, 2024

Gacko commented Jul 1, 2024

Gacko commented Jul 1, 2024

longwuyuan commented Jul 1, 2024

g1franc commented Jul 1, 2024

longwuyuan commented Jul 1, 2024

longwuyuan commented Jul 1, 2024

staizen-stephen commented Jul 1, 2024

longwuyuan commented Jul 3, 2024

longwuyuan commented Jul 3, 2024

g1franc commented Jul 4, 2024

longwuyuan commented Jul 4, 2024

strongjz commented Jul 4, 2024

longwuyuan commented Jul 5, 2024 • edited Loading

longwuyuan commented Jul 6, 2024

Annotation: Make `proxy-ssl-secret` optional. #11490

Annotation: Make `proxy-ssl-secret` optional. #11490

netlify bot commented Jun 20, 2024 •

edited

Loading

longwuyuan commented Jul 5, 2024 •

edited

Loading