Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cloudflared deployment restarts endlessly when using FluxCD #111

Open
maxpain opened this issue Oct 29, 2024 · 7 comments
Open

cloudflared deployment restarts endlessly when using FluxCD #111

maxpain opened this issue Oct 29, 2024 · 7 comments

Comments

@maxpain
Copy link

maxpain commented Oct 29, 2024

After a couple of changes in TunnelBinding, the cloudflared deployment started to restarting endlessly.

2024-10-29T11:02:48Z	INFO	Selected protocol	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"prerender","namespace":"fastcup"}, "namespace": "fastcup", "name": "prerender", "reconcileID": "ece5d62e-0a70-42c5-8f9e-b4c5805eb9e0", "protocol": "http"}
2024-10-29T11:02:48Z	INFO	generated cloudflare config	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"prerender","namespace":"fastcup"}, "namespace": "fastcup", "name": "prerender", "reconcileID": "ece5d62e-0a70-42c5-8f9e-b4c5805eb9e0", "hostname": "prerender.example.com", "target": "http://prerender.fastcup.svc:80"}
2024-10-29T11:02:48Z	INFO	Tunnel status is set	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"prerender","namespace":"fastcup"}, "namespace": "fastcup", "name": "prerender", "reconcileID": "ece5d62e-0a70-42c5-8f9e-b4c5805eb9e0", "status": {"hostnames":"prerender.example.com","services":[{"hostname":"prerender.example.com","target":"http://prerender.fastcup.svc:80"}]}}
2024-10-29T11:02:48Z	DEBUG	events	Configuring ConfigMap	{"type": "Normal", "object": {"kind":"TunnelBinding","namespace":"fastcup","name":"prerender","uid":"2605617a-7e99-4378-9e07-2195130e04e2","apiVersion":"networking.cfargotunnel.com/v1alpha1","resourceVersion":"6077115"}, "reason": "Configuring"}
2024-10-29T11:02:48Z	DEBUG	events	Applying ConfigMap to Deployment	{"type": "Normal", "object": {"kind":"TunnelBinding","namespace":"fastcup","name":"prerender","uid":"2605617a-7e99-4378-9e07-2195130e04e2","apiVersion":"networking.cfargotunnel.com/v1alpha1","resourceVersion":"6077115"}, "reason": "ApplyingConfig"}
2024-10-29T11:02:48Z	DEBUG	events	Applying ConfigMap to Deployment	{"type": "Normal", "object": {"kind":"Deployment","namespace":"cloudflare-operator-system","name":"fastcup","uid":"814b343c-65a3-4220-bb18-1c409761549c","apiVersion":"apps/v1","resourceVersion":"6077156"}, "reason": "ApplyingConfig"}
2024-10-29T11:02:48Z	INFO	Restarted deployment	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"prerender","namespace":"fastcup"}, "namespace": "fastcup", "name": "prerender", "reconcileID": "ece5d62e-0a70-42c5-8f9e-b4c5805eb9e0"}
2024-10-29T11:02:48Z	DEBUG	events	ConfigMap applied to Deployment	{"type": "Normal", "object": {"kind":"TunnelBinding","namespace":"fastcup","name":"prerender","uid":"2605617a-7e99-4378-9e07-2195130e04e2","apiVersion":"networking.cfargotunnel.com/v1alpha1","resourceVersion":"6077115"}, "reason": "AppliedConfig"}
2024-10-29T11:02:48Z	DEBUG	events	ConfigMap applied to Deployment	{"type": "Normal", "object": {"kind":"Deployment","namespace":"cloudflare-operator-system","name":"fastcup","uid":"814b343c-65a3-4220-bb18-1c409761549c","apiVersion":"apps/v1","resourceVersion":"6077176"}, "reason": "AppliedConfig"}
2024-10-29T11:02:48Z	DEBUG	events	Configured Cloudflare Tunnel	{"type": "Normal", "object": {"kind":"TunnelBinding","namespace":"fastcup","name":"prerender","uid":"2605617a-7e99-4378-9e07-2195130e04e2","apiVersion":"networking.cfargotunnel.com/v1alpha1","resourceVersion":"6077115"}, "reason": "Configured"}
2024-10-29T11:02:48Z	INFO	In validation	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "08a32c38-8008-4be1-8496-c7e47f6f629b"}
2024-10-29T11:02:48Z	INFO	Validation successful	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "08a32c38-8008-4be1-8496-c7e47f6f629b"}
2024-10-29T11:02:48Z	DEBUG	events	TunnelBinding Finalizer and Labels added	{"type": "Normal", "object": {"kind":"TunnelBinding","namespace":"fastcup","name":"prerender","uid":"2605617a-7e99-4378-9e07-2195130e04e2","apiVersion":"networking.cfargotunnel.com/v1alpha1","resourceVersion":"6077115"}, "reason": "MetaSet"}
2024-10-29T11:02:48Z	INFO	Tunnel status is set	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "08a32c38-8008-4be1-8496-c7e47f6f629b", "status": {"tunnelId":"redacted","tunnelName":"fc-hetzner-k8s","accountId":"redacted","zoneId":"redacted"}}
2024-10-29T11:02:48Z	INFO	In validation	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "cb1f0236-13f2-481c-a39d-96e1f0f9806e"}
2024-10-29T11:02:48Z	INFO	Validation successful	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "cb1f0236-13f2-481c-a39d-96e1f0f9806e"}
2024-10-29T11:02:48Z	INFO	Tunnel status is set	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "cb1f0236-13f2-481c-a39d-96e1f0f9806e", "status": {"tunnelId":"redacted","tunnelName":"fc-hetzner-k8s","accountId":"redacted","zoneId":"redacted"}}
2024-10-29T11:02:48Z	INFO	In validation	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "6cf62f1c-ea6d-4003-a8e1-19acb561fb8f"}
2024-10-29T11:02:48Z	INFO	Validation successful	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "6cf62f1c-ea6d-4003-a8e1-19acb561fb8f"}
2024-10-29T11:02:48Z	INFO	Tunnel status is set	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "6cf62f1c-ea6d-4003-a8e1-19acb561fb8f", "status": {"tunnelId":"redacted","tunnelName":"fc-hetzner-k8s","accountId":"redacted","zoneId":"redacted"}}
2024-10-29T11:02:48Z	INFO	In validation	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "c80cf0d7-0403-4380-979f-b846a36d48ae"}
2024-10-29T11:02:48Z	INFO	Validation successful	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "c80cf0d7-0403-4380-979f-b846a36d48ae"}
2024-10-29T11:02:48Z	INFO	Tunnel status is set	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "c80cf0d7-0403-4380-979f-b846a36d48ae", "status": {"tunnelId":"redacted","tunnelName":"fc-hetzner-k8s","accountId":"redacted","zoneId":"redacted"}}
2024-10-29T11:02:48Z	INFO	In validation	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "7b1bac63-e396-479f-91f0-a89fe93e27e8"}
2024-10-29T11:02:48Z	INFO	Validation successful	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "7b1bac63-e396-479f-91f0-a89fe93e27e8"}
2024-10-29T11:02:48Z	INFO	Tunnel status is set	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "7b1bac63-e396-479f-91f0-a89fe93e27e8", "status": {"tunnelId":"redacted","tunnelName":"fc-hetzner-k8s","accountId":"redacted","zoneId":"redacted"}}
2024-10-29T11:02:50Z	INFO	Updating existing record	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"prerender","namespace":"fastcup"}, "namespace": "fastcup", "name": "prerender", "reconcileID": "ece5d62e-0a70-42c5-8f9e-b4c5805eb9e0", "fqdn": "prerender.example.com", "dnsId": "redacted"}
2024-10-29T11:02:50Z	INFO	In validation	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "97105764-b203-4b10-a27e-17ba540c4c7c"}
2024-10-29T11:02:50Z	INFO	Validation successful	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "97105764-b203-4b10-a27e-17ba540c4c7c"}
2024-10-29T11:02:50Z	INFO	Tunnel status is set	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "97105764-b203-4b10-a27e-17ba540c4c7c", "status": {"tunnelId":"redacted","tunnelName":"fc-hetzner-k8s","accountId":"redacted","zoneId":"redacted"}}
2024-10-29T11:02:50Z	INFO	In validation	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "1263b978-ca6a-43fb-b7a1-ed4b93896418"}
2024-10-29T11:02:50Z	INFO	Validation successful	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "1263b978-ca6a-43fb-b7a1-ed4b93896418"}
2024-10-29T11:02:50Z	INFO	Tunnel status is set	{"controller": "clustertunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "ClusterTunnel", "ClusterTunnel": {"name":"fastcup"}, "namespace": "", "name": "fastcup", "reconcileID": "1263b978-ca6a-43fb-b7a1-ed4b93896418", "status": {"tunnelId":"redacted","tunnelName":"fc-hetzner-k8s","accountId":"redacted","zoneId":"redacted"}}
2024-10-29T11:02:50Z	INFO	DNS record updated successfully	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"prerender","namespace":"fastcup"}, "namespace": "fastcup", "name": "prerender", "reconcileID": "ece5d62e-0a70-42c5-8f9e-b4c5805eb9e0", "fqdn": "prerender.example.com"}
2024-10-29T11:02:50Z	INFO	Updating existing TXT record	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"prerender","namespace":"fastcup"}, "namespace": "fastcup", "name": "prerender", "reconcileID": "ece5d62e-0a70-42c5-8f9e-b4c5805eb9e0", "fqdn": "prerender.example.com", "dnsId": "redacted", "txtId": "redacted"}
2024-10-29T11:02:51Z	INFO	DNS record updated successfully	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"prerender","namespace":"fastcup"}, "namespace": "fastcup", "name": "prerender", "reconcileID": "ece5d62e-0a70-42c5-8f9e-b4c5805eb9e0", "fqdn": "prerender.example.com"}
2024-10-29T11:02:51Z	INFO	Inserted/Updated DNS/TXT entry	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"prerender","namespace":"fastcup"}, "namespace": "fastcup", "name": "prerender", "reconcileID": "ece5d62e-0a70-42c5-8f9e-b4c5805eb9e0"}
2024-10-29T11:02:51Z	DEBUG	events	Inserted/Updated DNS/TXT entry	{"type": "Normal", "object": {"kind":"TunnelBinding","namespace":"fastcup","name":"prerender","uid":"2605617a-7e99-4378-9e07-2195130e04e2","apiVersion":"networking.cfargotunnel.com/v1alpha1","resourceVersion":"6077115"}, "reason": "CreatedDns"}
@maxpain
Copy link
Author

maxpain commented Oct 29, 2024

The problem only exists if using Flux CD.
I have this TunnelBinding config:

apiVersion: networking.cfargotunnel.com/v1alpha1
kind: TunnelBinding
metadata:
  name: prerender
subjects:
  - name: prerender
tunnelRef:
  kind: ClusterTunnel
  name: fastcup

And cloudflare-operator changes subjects, adding spec:

subjects:
  - kind: Service
    name: prerender
    spec:
      noTlsVerify: false
      proxyAddress: 127.0.0.1
      proxyPort: 0
      proxyType: ''

After this, Flux CD reconciles this Custom Resource again.

@maxpain
Copy link
Author

maxpain commented Oct 29, 2024

Why does cloudflare-operator change this CR at all? Is it necessary?

@adyanth
Copy link
Owner

adyanth commented Oct 30, 2024

The controller adds labels and sets the status on the tunnel binding, which is required to track it. I use this with ArgoCD and it does not seem to have problems with diffs, so this seems to be something you could let Flux ignore?

cloudflare-operator/controllers/tunnelbinding_controller.go

Line 250 in d6135dd

func (r *TunnelBindingReconciler) creationLogic() error {

@maxpain
Copy link
Author

maxpain commented Oct 30, 2024 •
edited
Loading

@adyanth FluxCD ignores changes in status and labels, but doesn't ignore spec changes.
I think changing spec is an antipattern.

@adyanth
Copy link
Owner

adyanth commented Oct 30, 2024 •
edited
Loading

The controller does not explicitly change the values in the spec. I believe those are defaults getting serialized, which might be either a JSON tag I'm missing or I might need to be using pointers for it to be nullable to not be serialized. I am not fully sure since my instance only added the boolean for noTlsVerify and not the rest.

@benperove
Copy link

I had the same issue with ArgoCD and was able to work around it by adding to the Application spec:

  ignoreDifferences:
  - group: networking.cfargotunnel.com
    kind: TunnelBinding
    jsonPointers:
    - /subjects/spec

@maxpain maxpain changed the title cloudflared deployment restarts endlessly cloudflared deployment restarts endlessly when using FluxCD Nov 2, 2024
@adyanth
Copy link
Owner

adyanth commented Nov 5, 2024

@benperove you should not need to ignore the whole spec, that is surprising.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@benperove @maxpain @adyanth