Additional Traefik Helm chart options not passed to Traefik pod

[jozefs]

Environmental Info:
K3s Version:

k3s version v1.21.3+k3s1 (1d1f220f)
go version go1.16.6

Node(s) CPU architecture, OS, and Version:
Linux testnode 5.4.0-80-generic #90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
Single node deployed onto a Ubuntu 20.04 VM.

Describe the bug:
Additional values defined in /var/lib/rancher/k3s/server/manifests/traefik.yaml
are not passed to Traefik. This makes it not possible to enable access logs or
set the log level to DEBUG.

Steps To Reproduce:

• Installed K3s: curl -sfL https://get.k3s.io | sh - on a Ubuntu 20.04 VM.
• Edit /var/lib/rancher/k3s/server/manifests/traefik.yaml and add the following:

    accessLogs:
      enabled: true
    debug:
      enabled: true

The parameters above have been obtained from the Traefik for Helm Configuration Parameters linked from the Networking docs.

• Observe that the above change is reflected in the chart-values-traefik ConfigMap:

  kubectl describe cm chart-values-traefik -n kube-system
  

  ...
    
  Data
  ====
  values-01_HelmChart.yaml:
  ----
  accessLogs:
    enabled: true
  debug:
    enabled: true
  
  ...

• Follow the logs of the Traefik pod:

  kubectl logs -f -n kube-system traefik-abc-def
  

There should be new debug and/or access log entries generated continuously but they aren't.

Expected behavior:
When viewing the traefik Pod logs in the kube-system namespace (using kubectl logs traefik-abc-def), they should include access and/or debug log entries. They should look similar to this, produced every few seconds:

10.42.0.1 - - [11/Aug/2021:16:50:30 +0000] "GET /ping HTTP/1.1" 200 2 "-" "-" 1637 "ping@internal" "-" 0ms
10.42.0.1 - - [11/Aug/2021:16:50:30 +0000] "GET /ping HTTP/1.1" 200 2 "-" "-" 1638 "ping@internal" "-" 0ms
time="2021-08-11T16:50:30Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2021-08-11T16:50:30Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetes

Actual behavior:
No access or debug log entries present in the Traefik logs.

Additional context / logs:
Looking at the environment variables on the Traefik pod, there are no log related ones, and in the Deployment there are no additional log related parameters either.

As a workaround, the only way I was able to see the access logs and debug log entries was to manually modify the traefik Deployment in kube-system and pass additional arguments to the traefik executable:

Name: traefik
Namespace: kube-system
Pod Template:
  Containers:
   traefik:
    Args:
      ...
      --accesslog
      --log.level=DEBUG

#1001 may be related.

Backporting

• Needs backporting to older releases

[brandond]

The best way to do what you want is to add a HelmChartConfig to customize the Traefik chart deployment, otherwise you'll find that (like the rest of the packaged components) the manifest is rewritten on startup, discarding your changes.

https://rancher.com/docs/k3s/latest/en/helm/#customizing-packaged-components-with-helmchartconfig

[brandond]

Which version of Traefik are you using? There are different values for the v1 and v2 charts; if you're not seeing things get updated when changing the chart values its likely you're not using the right values.

It's also possible there's a syntax error in your yaml, in which case you should see an error in the k3s service logs when it attempts to apply your changes.

[brandond]

The v2 values structure is available here: https://github.com/traefik/traefik-helm-chart/blob/master/traefik/values.yaml#L147

[jozefs]

I'm using Traefik 2. I used the values from the latest docs, the link where it says "Helm Configuration Parameters". Could that be incorrect?

[brandond]

Yes, those are for the v1 chart. The docs could use an update to also include a link for the v2 chart values. We still support releases that include the v1 chart so it should have both.

[jozefs]

Thank you very much - that makes sense.

I looked through the issues again and found 3766 which captures the documentation gap.