posts/traefik-portainer-ssl/

[giscus[bot]]

posts/traefik-portainer-ssl/

Documentation for Techno Tim YouTube videos for all your copy pasta needs!

https://technotim.live/posts/traefik-portainer-ssl/

[jhoski]

Hi Tim, enjoying your content and ran across this as a solution for my certificates, however there is something in the compose files that prevents the containers from running with the following showing in the log: "exec /entrypoint.sh: operation not permitted". This happens for both the traefik and portainer images.
I tested a simple traefik compose file I found elsewhere and was able to get it running, but without the cloudflare integration. Also it is unclear to me what the auth credentials are for. Are they a login to traefik or something else? This is on a ubuntu 22.04 server Linux 5.15 running in a proxmox vm. Any clues would be helpful, thanks -Jay

[jhoski]

I have worked around getting traefik up and running with the wildcard certificates. I had to adjust docker-compose no-new-priviledges to false and then the container would spin up and log better info to work through the rest of it. I determined the authentication was for the traefik dashboard which was mentioned in the video. I'm a bit of a proxy, dns, ssl noob and it wasn't mentioned that you need a TXT DNS record for the _acme-challenge, but the error messages were good enough to work through it. Looking forward to getting rid of all my not private connections. Thanks! - Jay

[tchubaba]

I also ran into the TXT DNS _acme-challenge issue, but I can't figure out how to solve this one... I keep getting this error message when I spin up the docker container:
DNS problem: NXDOMAIN looking up TXT for _acme-challenge.local.mydomain.com - check that a DNS record exists for this domain
I see that a subdomain named _acme-challenge.local.mydomain.com briefly shows up in the cloudfare DNS for my domain, and then gets deleted, so I'm wondering if there's some sort of timing issue here.

Anyways I've been trying to get this to work for half a day now with no luck... I'm a noob when it comes to DNS so any help here would be appreciated.

P.S.: I've replaced my real domain name with mydomain.com in the messages above for privacy.

[tchubaba]

Well, shortly after that, I ended up answering my own question. It was indeed a timing issue. And the solution to that is to add the following parameter to the traefik.yml file: delayBeforeCheck: 5. This would go under dnsChallenge.

After I added this, it worked!

[smarty81]

could someone please explain the TXT DNS setup to me for cloudflare ?
I've been trying to get this working and googling for days and getting nowhere. I have Cloudflare access, I have a domain purchased through them. I have a API token for my domain. All of which works fine with nginx proxy manager which can auth wildcard certs. But npm keeps bugging out with other issues so I really want to get traefik working but I'm getting nowhere.
under DNS management on CF I have A and CNAME records for sub domains including a wildcard. I also created a TXT record with the name of "@" and content field of "_acme-challenge.local.mydom.uk"
But, with all the above, Traefik does not get an SSL cert as per Tims video. I must be missing something. Please help.

[jdppettit]

A point I was stuck on that might help somebody - recreating the container does not refresh the docker-compose environment variables, you need to actually rm it for that. So if you change anything in the docker-compose file you will need to rm and then docker-compose up again.

[kaylak4520]

Hey Tim,

Great doc! Worked well for me. Is there any modifications you can suggest to serve something like Home Assistant, via the traefik proxy, for both the internal domain which you explain here (homeassistant.local.domain.com) and an external domain (homeassistant.domain.com)?? Note: replace home assistant with anything else one would want to serve publicly.

Traefik is about to become my Swiss army knife of proxies!

[sowcrayts]

Thank you for the tutorials! I have learned a ton over the years from your content.

I am trying to get this set up on my network, which has services hosted on separate VLANs. I've noticed that in pihole, unless I specify t he url and port for the traefik dashboard, I can't get to it. Further, none of the other services hosts on other VLANs are accessible via their URLs. The traefik dashboard shows their routers and certs as working, and the config looks correct.

Do you think it's a firewall issue? Something else?

I appreciate the help!

[zeraussiul]

I think I'm having the same issue. No luck getting to the dashboard. Unsure wha to change within pihole though.

[jrs-anderson]

This is a really good guide and it's helped me get my homelab services nicely organised with SSL certs on all the self hosted web services. Thanks Tim!

The only thing I'm struggling with is that I can't get any shell up and running on Proxmox. Has anyone else found this? At first I thought that I was trying to be too clever and it was the load-balancing between the Proxmox nodes, but the behaviour persists with a modified config.yml (and a rebuild) pointing to just a single node.

This makes it impossible to use Proxmox fully. Anyone else experienced this? I even copied the config.yml directly from the guide just to check for typos but no joy.

I'll keep plugging away at this but my googling has only found a few mentions of this problem. I guess I should be using something like Ansible for updates but for the moment I want to have the GUI fully functioning.

All thoughts and pointers are most welcome, this is really puzzling me.

[jrs-anderson]

Hey everyone - I solved it so I wanted to follow up with a reply to myself just in case someone else hits the same issue as me.

It was NOTHING to do with traefik or Tim's suggested config.yml but it was a legacy of me having replaced a node of my proxmox cluster and have some old entries in ssh known_hosts files. The ssh errors were here, and not related to the traefik setup.

This guide lets me clear out the old entries and get everything running again. Fantastic result.

[sailhobie]

I am wondering what the best approach is to have some services stay local (with certificate and hostname) and others be accessible over WAN.

If I only port forward my router to the Traefik server (port 443, 80), won't that expose everything? Should I adjust the whitelist section in the config.yml and apply to specific routes? could I use a two sub-domains, ie. local.example.com and external.example.com and differentiate them in some way?

Thanks in advance

[sailhobie]

I found this thread which is the same setup I am working on. Any thoughts on the approach used here?

https://www.reddit.com/r/Traefik/comments/qi2435/traefik_v2_mixed_and_both_internal_and_external/

[sailhobie]

for those interested I have been using the default-whitelist as listed in data/config.yml to control what networks have access. This effectively blocks access from external access.

for routers/services entered via the config.yml add 'default-whitelist' as a middleware similar to 'default-headers'

for dockers, call out the middleware in the traefik labels section of the docker-compose.yml
#prevent external access beyond whitelist IPs in traefik/data/config.yml default-whitelist section
- "traefik.http.routers.portainer-secure.middlewares=default-whitelist@file"
- "traefik.http.routers.portainer.middlewares=default-whitelist@file"

When I try to access from external WAN I get "Forbidden" message; success!

Let me know if I am missing anything

[TimLinton]

Is this possible with docker swarm?

[slade208]

I just wanted to say thank you for making this content. The official docs from traefik aren't exactly the most newbie friendly and this is exactly what I needed. A few comments below

1. It would have made the video a bit longer but explaining how traefik works with the static and dymanic config would be nice

2. I had problems getting my wildcard cert to work (cloudflare) but I found the section in the traefik.yml that said uncomment this if it doesn't work. I uncommented it and voila it worked

3. If you are using Cloudflare proxy you could get the message "too many redirects" after googling I found that if you disable cloudflare proxy it will work

4. If you are testing a lot be sure to test browser incognito mode also. I found that I needed to do this

[YubinShin]

You are my hero

[andye230]

Another great video from Techno Tim!
Anyway some pointers for anyone who follows since I'm a noob at all this. DNS is the main issue that I found for this not working. If you are using pihole then you need to create an A record pointing to your server then CNAME pointing to your services i.e. portainer.mydomain at 192.168.30.35 and then traefik.mydomain pointing at portainer.mydomain. The yaml files are perfect (when adapted to your domain.com). My advice is get portainer running then you can use the logs to find out why traefik isn't working. It could be because your password isn't in the correct format. I found attaching traefik container to same bridge as portainer fixed the different IPs that portainer and traefik were on in portainer. I attached traefik to the same bridge then they both ended on the same subnet following container reboot.

[radhakrisri]

Thanks for a detailed video and these instructions, Tim!

Can you help me with this setup: I have a node, lets call it N1, that runs proxmox, which hosts pihole as a container.

Can I run traefik on N1? and have a domain - proxmox.local.example.com point to the proxmox URL hosted on N1? Is this OK? Or would you recommend that the traefik be hosted on a separate server, say N2, and not N1?

[andye230]

Yes you can run it on N1. This is my setup except I run Pihole in an Ubuntu VM

[Scrawf1966]

Hi! I'm having real problems with this! I've followed both the video and the instructions to the latter. I think I have everything set up including a Cloudflare domain and account etc. The problem is that when I spin up the docker container it just constantly restarts. It spins up for a for seconds and then restarts over and over. Can anyone explain why??

[andye230]

If the container keeps crashing then it maybe the image you've pulled. You could work this out by running the container with the YAML file on the documentation without making any changes then stop and start making changes as you go. If you have portainer running you could look at the files there or go via the CLI

[remio1209]

I've had similar issues and it was the ports that was my problem. Make sure you are not using the same ports for any container.

[userpasta]

Hi folks! Do you have details on how to setup Clouflare part? Does this whole setup work when there temporarily no Internet access? Thanks!

[MatijaTerzic]

In the video the domain local.technotim.live is it configured on cloudflare DNS (pointing to .... what ) or internally on the piHole ? And also when you add a service like "proxmox.local.tehcnotim.live", the DNS entry for this is configured where ?

[timothystewart6]

everything on the local subdomain is always local dns, pihole in this case

[Scubatime]

HI there! I had this setup working previously, but due to a Proxmox crash I am rebuilding from scratch.

This time however, I CANNOT get to the Portainer GUI at all... neither the sub-sub domain name or by IP address. I have reinstalled everything via the tutorial 3 times, and no luck. Traefik loads, but there is no certificate. I re-generated a fresh CF token as well, to make sure that is working (it is via their test script).

Any clue where to start? No certs and no Portainer GUI.

[baz-snow-ss]

I tried to install and reinstall this for 3 days, I have a Cloudflare account, a domain, I did the DNS records,
I got an API Token zone:zone:read zone:zonesettings:read zone:dns:edit all.
Saved it modified the necessary files, created all file required.
I keep getting this error:
command traefik error: yaml: line 27: did not find expected key
looking at line 27 in the traefik.yml -----> dnsChallenge:
Could I get some help please I have tried every thing I would greatly appreciate it, I'm also willing to donate for your time and effort..

[killerb77]

@timothystewart6 I am having some trouble getting this working with vs code server. I get "bad gateway" message when going through the local domain. When I go through the ip/port directly it works. I have it working with portainer and homer so I know traefik is setup correct. I fear it's some grouping of labels might be causing the issue.

---

services:
code-server:
image: lscr.io/linuxserver/code-server:latest
container_name: code-server
environment:
- PUID=1000
- PGID=1000
- TZ=America/New_York
- PASSWORD= #optional
- HASHED_PASSWORD= #optional
- SUDO_PASSWORD= #optional
- SUDO_PASSWORD_HASH= #optional
- PROXY_DOMAIN= #optional
- DEFAULT_WORKSPACE=/config/workspace #optional
security_opt:
- no-new-privileges:true
networks:
- proxy
volumes:
- /home/user/code-server/config:/config
ports:
- 7443:8443
restart: unless-stopped
labels:
- "traefik.enable=true"
- "traefik.http.routers.code-server.entrypoints=http"
- "traefik.http.routers.code-server.rule=Host(code-server.home.mydomain.com)"
- "traefik.http.middlewares.code-server-https-redirect.redirectscheme.scheme=https"
- "traefik.http.routers.code-server.middlewares=code-server-https-redirect"
- "traefik.http.routers.code-server-secure.entrypoints=https"
- "traefik.http.routers.code-server-secure.rule=Host(code-server.home.mydomain.com)"
- "traefik.http.routers.code-server-secure.tls=true"
- "traefik.http.routers.code-server-secure.service=code-server"
- "traefik.http.services.code-server.loadbalancer.server.port=7443"
- "traefik.docker.network=proxy"

networks:
proxy:
external: true

---

log from docker

[migrations] started
[migrations] no migrations found
───────────────────────────────────────
██╗ ███████╗██╗ ██████╗
██║ ██╔════╝██║██╔═══██╗
██║ ███████╗██║██║ ██║
██║ ╚════██║██║██║ ██║
███████╗███████║██║╚██████╔╝
╚══════╝╚══════╝╚═╝ ╚═════╝
Brought to you by linuxserver.io
───────────────────────────────────────
To support LSIO projects visit:
https://www.linuxserver.io/donate/
───────────────────────────────────────
GID/UID
───────────────────────────────────────
User UID: 1000
User GID: 1000
───────────────────────────────────────
[custom-init] No custom files found, skipping...
starting with no password
[2024-01-31T04:06:41.146Z] info code-server 4.20.1 e76afa4a2bf4667a3c9f71bf56ef34b8ad365fbe
[2024-01-31T04:06:41.147Z] info Using user-data-dir /config/data
[2024-01-31T04:06:41.157Z] info Using config file /config/.config/code-server/config.yaml
[2024-01-31T04:06:41.157Z] info HTTP server listening on http://0.0.0.0:8443/
[2024-01-31T04:06:41.157Z] info - Authentication is disabled
[2024-01-31T04:06:41.157Z] info - Not serving HTTPS
[2024-01-31T04:06:41.157Z] info Session server listening on /config/data/code-server-ipc.sock
[ls.io-init] done.

[hanley-development]

I have been playing around with a 3 node pihole setup and for some reason Unifi doesn't really load balance it's traffic on dns servers and just blew up my primary pihole. I went down a rabbit-hole of haproxy (doesn't do UDP), nginx, LVS, DNSdist.....so many more and finally landed back to traefik. Interesting enough the UDP load balancing is amazing. YMMV but for my journey I ended up doing the following:

1. Fired up a LXC with ubuntu
2. Disabled and stopped systemd-resolved.service

systemctl disable systemd-resolved.service
systemctl stop systemd-resolved

3. added tcp and udp port 53 to my docker compose:

    ports:
      - 80:80
      - 443:443
      - 53:53/tcp
      - 53:53/udp

4. then in my static config (traefik.yml) added the entrypoint

entryPoints:
  dnstcp:
    address: ":53"
  dnsudp:
    address: ":53/udp"

5. Lastly added the dynamic config (config.yml)

udp:
  routers:
    dnsudp_router:
      # does not listen on "other" entry point
      entryPoints:
        - dnsudp
      service: dnsudp_svc

  services:
    dnsudp_svc:
      loadBalancer:
        servers:
          - address: 192.168.1.7:53
          - address: 192.168.1.8:53

tcp:
  routers:
    dnstcp_router:
      entryPoints:
        - dnstcp
      rule: "HostSNI(`*`)"
      service: dnstcp_svc

  services:
    dnstcp_svc:
      loadBalancer:
        servers:
          - address: 192.168.1.7:53
          - address: 192.168.1.8:53

Planning on putting keepalived on the LB and third so that I have an active passive LB

[rao8sandhya]

hi Tim,
basically I followed above steps and video, JUST to launch traefik with my domain.com and cloudflare all working good.
And this comment did help with hurdle https://community.traefik.io/t/cloudflare-failed-to-find-zone-domain/19884/11
I do not see any errors in traefik docker container logs of cloudflare

Now when I try to access https://traefik-dashboard.mydomain.com/ It says DNS error in browser
And when I try https://localhost/ I get "404 page not found" and says not-secure at chrome location.

Am I supposed to run Portainer and PiHole to get end-end working like your youtube video, shows !?

Please help, Thanks
Rajesh

[rajeshpv]

hi Tim,
basically I followed above steps and video, JUST to launch traefik with my domain.com and cloudflare all working good.
And this comment did help with hurdle https://community.traefik.io/t/cloudflare-failed-to-find-zone-domain/19884/11
I do not see any errors in traefik docker container logs of cloudflare

Now when I try to access https://traefik-dashboard.mydomain.com/ It says DNS error in browser
And when I try https://localhost/ I get "404 page not found" and says not-secure at chrome location.

Am I supposed to run Portainer and PiHole to get end-end working like your youtube video, shows !?

Please help, Thanks
Rajesh

[rajeshpv]

I did start portainer, even then same behavior !,
Any help is greatly appreciated :) Thanks, Rajesh

[rajeshpv]

I updated my logs here https://gist.github.com/rajeshpv/1129f2465bb96348a7e6251465002214
I cannot access my portainer UI even at 8000 or 9000

[DHeinrichs]

So foremost: Thank you for the great tutorial — this is the first time I was able to really understand the depths of Traefik. I am currently stumbling upon a weird error — perhaps you or the community can help… I have set up the Traefik container, checked mappings via your supposed troubleshooting-guide (great idea!) and the certificates appear to be pulled correctly. Nonetheless, I am experiencing an SSL-Mismatch Error, does anyone know why?

https://share.cleanshot.com/QnZFzVlk+ (Anonymized Certificate Settings)
https://share.cleanshot.com/Vy02SzMF+ (The Error)

[timothystewart6]

Thank you! Hey, not sure off hand but can you embed these images rather than provide them as a download? You can do that by dragging them or using the image icon below.

[DHeinrichs]

For sure! While having a walk, I had the “revelation” - perhaps that's helpful for some people stumbling across the same error: I was setting up my DNS-Server whilst setting up Traefik, but I simply forgot to change my local network settings to use my new DNS. A clear case of "think before you deploy"... Thanks for the quick try to help!