Reverse proxy getting 502 Bad Gateway errors #2811

git-n-pissed · 2024-03-05T04:03:14Z

git-n-pissed
Mar 5, 2024

I've got a separate container running an Nginx reverse proxy. It is using a user defined network and I've got several subdomains (all in containers) running behind it with no issues. I've added a / to the end of the proxy URL as described in the docs and I've disabled UFW on the host. I've looked at the docker logs output and it isn't helpful. I don't see anywhere in the docs where the location of any other log files are.

Here is my docker-compose.yml:

version: '3.6'
networks:
  my_net:
    external: true
services:
  transmission:
    image: haugene/transmission-openvpn-proxy:5.3.1
    container_name: transmission
    cap_add:
      - NET_ADMIN
    volumes:
      - './data/:/data/:rw'
      - './config/:/config/:rw'
    environment:
      - OPENVPN_PROVIDER=NORDVPN
      - OPENVPN_USERNAME=username
      - OPENVPN_PASSWORD=password
      - OPENVPN_CONFIG=canada
      - NORDVPN_COUNTRY=CA
      - NORDVPN_CATEGORY=legacy_p2p
      - NORDVPN_PROTOCOL=tcp
      - NORDVPN_SERVER=ca111.nordvpn.com
      #- LOCAL_NETWORK=192.168.0.0/16
    logging:
      driver: json-file
      options:
        max-size: 10m
    networks:
      - my_net
    ports:
      - '9091:9091'

I don't know if the container is doing what it is suppoed to be doing or not. The documentation is... confusing... The Supported providers page makes it seem like if IPVANISH, NORDVPN, PIA, or VYPRVPN are your provider, then the config will be downloaded for you or something? Best I can tell that hasn't happend. The config and data directories mapped in volumes are both empty. Any pointers as to what might be happening would be appreciated.

Update 1:

I entered the container and ran curl localhost:9091 as described here and I got this error curl: (7) Failed to connect to localhost port 9091 after 0 ms: Couldn't connect to server, so apparently transmission isn't starting with the container. I get the same error when I remove my custom network configuration and make the docker-compose.yml file "stock".

Update 2:

I've been performing the steps listed here and all of them had the expected output until I performed the steps in the Running with a valid configuration section with my NORDVPN credentials. Here is the output:

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2024/03/05 04:33:48 [emerg] 1#1: host not found in upstream "transmission" in /etc/nginx/nginx.conf:14
nginx: [emerg] host not found in upstream "transmission" in /etc/nginx/nginx.conf:14

Update 3

I was using the haugene/transmission-openvpn-proxy:5.3.1 image, but I changed to the haugene/transmission-openvpn:5.3.1 image and now I'm getting a new set of errors:

Starting container with revision: 07f5a2b9aea5028c9bb75438c1552708e91dde71
TRANSMISSION_HOME is currently set to: /config/transmission-home
Creating TUN device /dev/net/tun
Using OpenVPN provider: NORDVPN
Running with VPN_CONFIG_SOURCE auto
Provider NORDVPN has a bundled setup script. Defaulting to internal config
Executing setup script for NORDVPN
/etc/openvpn/nordvpn/..
INFO: OVPN: Checking curl installation
INFO: OVPN: DNS resolution ok
ERROR: ERROR: OVPN: cannot ping https://downloads.nordcdn.com/configs/files, network or internet unavailable. Cannot download NORDVPN configuration files

pkishino · 2024-03-05T06:12:39Z

pkishino
Mar 5, 2024
Collaborator

Why are you using the proxy image ?
Why is your local _network commented out?
Why are you using relative paths for mounts?
If you search issues and PR you will find info regarding nordvpn.. try dev branch

4 replies

git-n-pissed Mar 5, 2024
Author

Why are you using the proxy image ?

I was using the proxy image because I didn't see the non-proxy one on Docker hub initially

Why is your local _network commented out?

I've tried it both ways, but I figured it needed to be commented out based on the How to fix this section of the docs where it talks about LOCAL_NETWORK and says "Alternatively you can reverse proxy the traffic through another container". The word "Alternatively" makes me think that LOCAL_NETWORK wasn't needed if I was using a reverse proxy.

Why are you using relative paths for mounts?

Why not? Never had a problem with it with any of my other containers. FWIW, the config directory is being populated with some files on some of my attempts at getting this working. More on that in a minute.

If you search issues and PR you will find info regarding nordvpn.. try dev branch

:/ Been searching for about 6 hours now.

Currently I can get the container to start if I remove my user defined network from the docker-compose.yml, but that is the network my reverse proxy is on, so I think I need it to let my reverse proxy container communicate with the transmission container. Regardless, even when I remove my user defined network config, the transmission container will start up, but eventually gets "unhealthy". The last thing shown in the logs is this:

STARTING TRANSMISSION
Transmission startup script complete.
2024-03-05 06:06:15 Initialization Sequence Completed

pkishino Mar 5, 2024
Collaborator

#2806

git-n-pissed Mar 5, 2024
Author

Thanks, I'll take a look at that tomorrow morning

git-n-pissed Mar 5, 2024
Author

I went to #2806 and followed the trail and found this comment where it was said switching to the edge image solved the problem, but it didn't for me. I get the same result I was getting in my previous post, regardless if I'm trying to use my user defined network or not.

Small tangent, but it is really hard to figure out what is going on without good logs. The docker compose log transmission command shows a log, but it always ends with:

STARTING TRANSMISSION
Transmission startup script complete.
2024-03-05 18:22:43 Initialization Sequence Completed

When I docker exec -it transmission bash and go to the var/log directory, there are no logs for transmission, no logs for openvpn, no helpful logs what-so-ever. If there are any additional log files, I can't find them and they aren't mentioned in the documentation.

edgd1er · 2024-03-06T07:22:55Z

edgd1er
Mar 6, 2024

Nordvpn process is to contact nordvpn's api to get the best server available (lower usage, fit settings) and download its config file. without dns or internet access the connection cannot be established. As you defined a server, the file path is given by the api, and the file downloaded.
When using NORDVPN_SERVER, it may fail as the server could be down. Letting the api select the best server in a country and/or in a group is the way to be sure to have an up and running server.

As logged, dns resolution is failing in the container when using the user defined networked (my_net) and nothing mentioned earlier will occur. dns resolution is the first check.
how was the network created (bridge, ipvlan, macvlan, ....) ? Are others containers connected to that network have access to internet ? Do you have a local dns container like pihole running ?

You may set DEBUG env var to see commands running.

I have traefik as reverse proxy with a user defined network. I had hard to configure it, but now it works as expected.

transmission logs are located in config/transmission-home.

3 replies

git-n-pissed Mar 6, 2024
Author

how was the network created (bridge, ipvlan, macvlan, ....) ?

The network my_net was created with the comamnd docker network create --ipv6 --subnet 2222:BEEF::/112 my_net. When comparing the outputs generated from the commands docker inspect my_net and docker inspect transmission_default, there are no appreciable differerences in the networks other than the fact that my_net has IPv6 enabled. To be clear to others, transmission_default is the network created when I commented out the network configuration in docker-compose.yml and let the container create its own network.

Are others containers connected to that network have access to internet ?

Yes. All of my containers are behind a reverse proxy and working as expected in regard to their Internet connection. All port 80 and 443 traffic is funneled through the reverse proxy to the respective containers.

Do you have a local dns container like pihole running ?

No. My reverse proxy container uses the Docker DNS 172.0.0.11 as its resolver. Just to be sure this wasn't the issue, I disabled my reverse proxy and tried transmision again with the same results. I also have a PiHole instance running on a different server on my network which should not be effecting this, but just to be sure I checked its logs and it had not blocked anything in the timeframe of my test. To be double sure I disabled PiHole and tried transmission again with the same result.

You may set DEBUG env var to see commands running.

This is helpful. When viewing the logs after running the container with DEBUG enabled I see the following right before the last line of the error message I included in "Update 3" of my OP. I also get this same output if I run the same ping command inside any of my other containers:

+++ ping -c2 downloads.nordcdn.com
++ ret='PING downloads.nordcdn.com(2606:4700::6811:d0ed (2606:4700::6811:d0ed)) 56 data bytes
From server2 (2222:BEEF::1) icmp_seq=1 Destination unreachable: No route
From server2 (2222:BEEF:1) icmp_seq=2 Destination unreachable: No route

--- downloads.nordcdn.com ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1001ms'
++ true
++ [[ PING downloads.nordcdn.com(2606:4700::6811:d0ed (2606:4700::6811:d0ed)) 56 data bytes
From server2 (2222:BEEF::1) icmp_seq=1 Destination unreachable: No route
From server2 (2222:BEEF:1) icmp_seq=2 Destination unreachable: No route

What is interesting about this is is that it is resolving to an IPv6 address. When I run the same ping command on the host or any other machine I get this output:

PING downloads.nordcdn.com (104.17.207.237) 56(84) bytes of data.
64 bytes from 104.17.207.237 (104.17.207.237): icmp_seq=1 ttl=57 time=2.78 ms
64 bytes from 104.17.207.237 (104.17.207.237): icmp_seq=2 ttl=57 time=2.11 ms

--- downloads.nordcdn.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 2.112/2.446/2.781/0.334 ms

So when the ping command is issued from any container on my_net which has IPv6 enabled, the IP address the ping is being sent from is one in the address space of the my_net IPv6 subnet instead of an IPv4 address.

edgd1er Mar 6, 2024

Using https://dnschecker.org/all-dns-records-of-domain.php?query=downloads.nordvpn.com&rtype=AAAA&dns=cloudflare I could not find an AAAA record , I found only an A record. this could be the cause of your problem.

Is your network ipv4 enabled ? I've never used an ipv6 enabled network.

git-n-pissed Mar 6, 2024
Author

You typo'ed the FQDN. It is downloads.nordcdn.com, not downloads.nordvpn.com, and it does seem to be listed:

https://dnschecker.org/all-dns-records-of-domain.php?query=downloads.nordcdn.com&rtype=AAAA&dns=cloudflare

My network has both IPv4 and IPv6 enabled. I've been trying to find a way to force a container to use IPv4 when connected to a network with both protocols, but so far my attempts have been unsuccessful. Apparently Docker favors IPv6 if it is available.

As a test I re-created my_net, but without enabling IPv6. Now I'm able to access the transmission web interface and everything is working as expected, however the container still show unhealthy status in the output of docker ps. I had enabled IPv6 on my docker network for some reason, I'll have to see how disabling it effects my other containers.

But short story here, there is definitely a problem with this image's use of IPv6.

sj64 · 2024-03-19T15:46:01Z

sj64
Mar 19, 2024

I'm also struggling with 504 timeout errors using the bundled reverse proxy. Don't think it's all ipv6 related but this does show up in the log:

transmission-openvpn-1 | 2024-03-19 14:48:54 WARNING: OpenVPN was configured to add an IPv6 route. However, no IPv6 has been configured for tun0, therefore the route installation may fail or may not work as expected.
transmission-openvpn-1 | 2024-03-19 14:48:54 add_route_ipv6(2000::/3 -> :: metric -1) dev tun0
transmission-openvpn-1 | 2024-03-19 14:48:54 net_route_v6_add: 2000::/3 via :: dev tun0 table 0 metric -1
transmission-openvpn-1 | 2024-03-19 14:48:54 sitnl_send: rtnl: generic error (-13): Permission denied
transmission-openvpn-1 | 2024-03-19 14:48:54 ERROR: Linux IPv6 route can't be added

0 replies

sj64 · 2024-03-19T15:48:58Z

sj64
Mar 19, 2024

PS had to use 8081, as already got jitsi on 8080

3 replies

git-n-pissed Mar 19, 2024
Author

Are you using a reverse proxy?

sj64 Mar 19, 2024

Aye nginx in a separate container. Following the debug tests I get as far as curl localhost:9091 inside the container and am getting 401 unauthorized - still to figure out where that's coming from.

sj64 Mar 20, 2024

OK, got this working finally. I spotted the container_name env var in your compose.yml, added it to mine (not in default config), set it to transmission, to match the proxy url in nginx.conf i.e. proxy_pass http://transmission:9091; Also added ipv6 listen [::]:8081 to server block, cos the docker log for the proxy had lots of stuff like:

proxy-1 | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf

still get the tun0 misconfigure entry, but for now, seems to be working!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reverse proxy getting 502 Bad Gateway errors #2811

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 10 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Reverse proxy getting 502 Bad Gateway errors #2811

git-n-pissed Mar 5, 2024

Replies: 4 comments · 10 replies

pkishino Mar 5, 2024 Collaborator

git-n-pissed Mar 5, 2024 Author

pkishino Mar 5, 2024 Collaborator

git-n-pissed Mar 5, 2024 Author

git-n-pissed Mar 5, 2024 Author

edgd1er Mar 6, 2024

git-n-pissed Mar 6, 2024 Author

edgd1er Mar 6, 2024

git-n-pissed Mar 6, 2024 Author

sj64 Mar 19, 2024

sj64 Mar 19, 2024

git-n-pissed Mar 19, 2024 Author

sj64 Mar 19, 2024

sj64 Mar 20, 2024

git-n-pissed
Mar 5, 2024

Replies: 4 comments 10 replies

pkishino
Mar 5, 2024
Collaborator

git-n-pissed Mar 5, 2024
Author

pkishino Mar 5, 2024
Collaborator

git-n-pissed Mar 5, 2024
Author

git-n-pissed Mar 5, 2024
Author

edgd1er
Mar 6, 2024

git-n-pissed Mar 6, 2024
Author

git-n-pissed Mar 6, 2024
Author

sj64
Mar 19, 2024

sj64
Mar 19, 2024

git-n-pissed Mar 19, 2024
Author