An error ocurred when trying to query database settings for the config parameters

[djohnson-mulytic]

Ok, I am trying to update my Open Balena from 1 device to another.
Can you see why the old one is working and new one is not?

2 servers in same AWS region

• Same Open Balena Admin setup (both open-balena-admin/compose/services.yml v2.0)

Old one seems to connect and open-balena-admin comes up, new one does not.

They should not be different, yet they are. I am not sure if this is in the Open Balena config, the Open Balena Admin config or somewhere else.
Why are the ports different? New (not working) is 5432, Old (working) is 3000, as shown below. Again should be the same

Like when the interface names changed, I suspect it has something to do with the new versions of Open Balena, but no way of knowing. I went on the new one and changed the network name from openbalena_default to open-balena_default everywhere, maybe that was a mistake?

NEW BALENA
open-balena/open-balena-admin/compose/services.yml:open-balena_default
open-balena/open-balena-admin/compose/services.yml:open-balena_default
open-balena/open-balena-admin/compose/services.yml:open-balena_default
open-balena/open-balena-admin/compose/services.yml:open-balena_default
open-balena/src/haproxy-sidecar/balena.sh:open-balena_default
open-balena-admin/compose/services.yml:open-balena_default
open-balena-admin/compose/services.yml:open-balena_default
open-balena-admin/compose/services.yml:open-balena_default
open-balena-admin/compose/services.yml:open-balena_default

OLD BALENA
open-balena-admin/open-balena-admin/compose/services.yml:openbalena_default
open-balena-admin/open-balena-admin/compose/services.yml:openbalena_default
open-balena-admin/open-balena-admin/compose/services.yml:openbalena_default
open-balena-admin/open-balena-admin/compose/services.yml:openbalena_default
open-balena-admin/compose/services.yml:openbalena_default
open-balena-admin/compose/services.yml:openbalena_default
open-balena-admin/compose/services.yml:openbalena_default
open-balena-admin/compose/services.yml:openbalena_default
open-balena-admin/compose/services.yml:openbalena_default
open-balena-admin/compose/services.yml:openbalena_default
open-balena-admin/compose/services.yml:openbalena_default

---

NEW BALENA: Site is not fine

balena@ip-172-31-27-192:~$ open-balena-admin/scripts/compose up
WARN[0000] /home/balena/open-balena-admin/compose/services.yml: version is obsolete
WARN[0000] /home/balena/open-balena-admin/config/docker-compose.yml: version is obsolete
[+] Running 3/0
✔ Container openbalena-admin-postgrest-1 Created 0.0s
✔ Container openbalena-admin-remote-1 Created 0.0s
✔ Container openbalena-admin-ui-1 Created 0.0s
Attaching to postgrest-1, remote-1, ui-1
postgrest-1 | 22/Jun/2024:05:57:46 +0000: An error ocurred when trying to query database settings for the config parameters
postgrest-1 | 22/Jun/2024:05:57:46 +0000: ConnectionError (Just "connection to server at "db.openbalena.name.de" (172.18.0.4), port 5432 failed: Connection refused\n\tIs the server running on that host and accepting TCP/IP connections?\n")
postgrest-1 | 22/Jun/2024:05:57:46 +0000: Attempting to connect to the database...
postgrest-1 | 22/Jun/2024:05:57:46 +0000: {"details":"connection to server at "db.openbalena.name.de" (172.18.0.4), port 5432 failed: Connection refused\n\tIs the server running on that host and accepting TCP/IP connections?\n","code":"","message":"Database connection error. Retrying the connection."}
postgrest-1 | 22/Jun/2024:05:57:46 +0000: Attempting to reconnect to the database in 0 seconds...
remote-1 | [2024/06/22 05:57:46:9162] N: ttyd 1.7.3-a2312cb (libwebsockets 4.0.20)
remote-1 | [2024/06/22 05:57:46:9180] N: tty configuration:
remote-1 | [2024/06/22 05:57:46:9181] N: start command: /usr/src/app/scripts/connect-ssh.sh
remote-1 | [2024/06/22 05:57:46:9181] N: close signal: SIGHUP (1)
remote-1 | [2024/06/22 05:57:46:9181] N: terminal type: xterm-256color
remote-1 | [2024/06/22 05:57:46:9181] N: endpoints:
remote-1 | [2024/06/22 05:57:46:9181] N: base-path: /ttyd
remote-1 | [2024/06/22 05:57:46:9181] N: index : /ttyd/
remote-1 | [2024/06/22 05:57:46:9181] N: token : /ttyd/token
remote-1 | [2024/06/22 05:57:46:9181] N: websocket: /ttyd/ws
remote-1 | [2024/06/22 05:57:46:9182] N: allow url arg: true
remote-1 | [2024/06/22 05:57:46:9811] N: Using foreign event loop...
remote-1 | [2024/06/22 05:57:46:9858] N: Listening on port: 7681
postgrest-1 |
postgrest-1 | > [email protected] start
postgrest-1 | > node postgrest-proxy.js
postgrest-1 |
postgrest-1 | 22/Jun/2024:05:57:47 +0000: {"details":"connection to server at "db.openbalena.name.de" (172.18.0.4), port 5432 failed: Connection refused\n\tIs the server running on that host and accepting TCP/IP connections?\n","code":"","message":"Database connection error. Retrying the connection."}
postgrest-1 | 22/Jun/2024:05:57:47 +0000: Attempting to reconnect to the database in 1 seconds...
remote-1 | [HPM] Proxy created: / ->
remote-1 | Waiting for 127.0.0.1:10000.[HPM] Proxy created: / ->
remote-1 | Waiting for 127.0.0.1:10001.[HPM] Proxy created: / ->
remote-1 | Waiting for 127.0.0.1:10002.[HPM] Proxy created: / ->
remote-1 | Waiting for 127.0.0.1:10003.[HPM] Proxy created: / ->
remote-1 | Waiting for 127.0.0.1:10004.[HPM] Proxy created: / ->
remote-1 | Waiting for 127.0.0.1:10005.[HPM] Proxy created: / ->
remote-1 | Waiting for 127.0.0.1:10006.[HPM] Proxy created: / ->
remote-1 | Waiting for 127.0.0.1:10007.[HPM] Proxy created: / ->
remote-1 | Waiting for 127.0.0.1:10008.[HPM] Proxy created: / ->
remote-1 | Waiting for 127.0.0.1:10009.
remote-1 | Connected!
remote-1 |
remote-1 | Connected!
remote-1 |
remote-1 | Connected!
remote-1 |
remote-1 | Connected!
remote-1 |
remote-1 | Connected!
remote-1 |
remote-1 | Connected!
remote-1 |
remote-1 | Connected!
remote-1 |
remote-1 | Connected!
remote-1 |
remote-1 | Connected!
remote-1 |
remote-1 | Connected!
postgrest-1 | [HPM] Proxy created: / -> http://127.0.0.1:3000
ui-1 |
ui-1 | > [email protected] start
ui-1 | > node server
ui-1 |
ui-1 | isDevelopment: false
ui-1 | Running open-balena-ui on http://0.0.0.0:8080

---

OLD BALENA: open-balena-admin site works (it has other issues, but not login)

balena@ip-172-31-39-247:~$ open-balena-admin/scripts/compose up
Creating openbalena-admin_postgrest_1 ... done
Creating openbalena-admin_remote_1 ... done
Creating openbalena-admin_ui_1 ... done
Attaching to openbalena-admin_postgrest_1, openbalena-admin_remote_1, openbalena-admin_ui_1
postgrest_1 | 22/Jun/2024:05:45:26 +0000: Attempting to connect to the database...
postgrest_1 | 22/Jun/2024:05:45:26 +0000: Connection successful
postgrest_1 | 22/Jun/2024:05:45:26 +0000: Listening on port 3000
remote_1 | [2024/06/22 05:45:26:5732] N: ttyd 1.6.3-3e37e33 (libwebsockets 4.0.20)
remote_1 | [2024/06/22 05:45:26:5751] N: tty configuration:
remote_1 | [2024/06/22 05:45:26:5751] N: start command: /usr/src/app/scripts/connect-ssh.sh
remote_1 | [2024/06/22 05:45:26:5751] N: close signal: SIGHUP (1)
remote_1 | [2024/06/22 05:45:26:5751] N: terminal type: xterm-256color
remote_1 | [2024/06/22 05:45:26:5751] N: endpoints:
remote_1 | [2024/06/22 05:45:26:5751] N: base-path: /ttyd
remote_1 | [2024/06/22 05:45:26:5751] N: index : /ttyd/
remote_1 | [2024/06/22 05:45:26:5751] N: token : /ttyd/token
remote_1 | [2024/06/22 05:45:26:5751] N: websocket: /ttyd/ws
remote_1 | [2024/06/22 05:45:26:5751] N: allow url arg: true
postgrest_1 | 22/Jun/2024:05:45:26 +0000: Config re-loaded
remote_1 | [2024/06/22 05:45:26:6049] N: Using foreign event loop...
remote_1 | [2024/06/22 05:45:26:6050] N: Listening on port: 7681
postgrest_1 | 22/Jun/2024:05:45:26 +0000: Listening for notifications on the pgrst channel
postgrest_1 | 22/Jun/2024:05:45:26 +0000: Schema cache loaded
remote_1 | [HPM] Proxy created: / ->
remote_1 | Waiting for 127.0.0.1:10000.[HPM] Proxy created: / ->
remote_1 | Waiting for 127.0.0.1:10001.[HPM] Proxy created: / ->
remote_1 | Waiting for 127.0.0.1:10002.[HPM] Proxy created: / ->
remote_1 | Waiting for 127.0.0.1:10003.[HPM] Proxy created: / ->
remote_1 | Waiting for 127.0.0.1:10004.[HPM] Proxy created: / ->
remote_1 | Waiting for 127.0.0.1:10005.[HPM] Proxy created: / ->
remote_1 | Waiting for 127.0.0.1:10006.[HPM] Proxy created: / ->
remote_1 | Waiting for 127.0.0.1:10007.[HPM] Proxy created: / ->
remote_1 | Waiting for 127.0.0.1:10008.[HPM] Proxy created: / ->
remote_1 | Waiting for 127.0.0.1:10009.
remote_1 | Connected!
remote_1 |
remote_1 | Connected!
remote_1 |
remote_1 | Connected!
remote_1 |
remote_1 | Connected!
remote_1 |
remote_1 | Connected!
remote_1 |
remote_1 | Connected!
remote_1 |
remote_1 | Connected!
remote_1 |
remote_1 | Connected!
remote_1 |
remote_1 | Connected!
remote_1 |
remote_1 | Connected!
ui_1 | yarn run v1.22.19

Many thanks for your time, please let me know what i can do to help you all out
Cheers, David

[robbe-haesendonck]

Hey again :)

So it seems that the postgrest container can't access your openbalena postgres database.
One thing I also noticed is that in the new version haproxy sidecar is present, in the old one it's not.

Might be that by changing the haproxy container, the requests aren't proxied anymore.
One small note that you should take into account is what postgrest actually does, which is two-fold:

1. It sets up a connection to your openbalena database (using port 5432)
2. It sets up a socket on which it listens to requests (HTTP REST calls) on port 3000

So, you're experiencing a connection failure to the database as mentioned earlier.
I think this is because you're probably outside of that docker network again, causing it to refuse the connections (because nothing's listening on that port anyways).

I would advise you to test it in a local lab environment using the new config, and start changing things to see what works.
The only thing you better not do (except in a lab environment) is opening up the database port externally, if need be at least use an internal network (docker or by your infra provider) to make sure your database connection isn't routed over the public internet.

[djohnson-mulytic]

Thanks so much, I will do that! And we have to stop meeting like this, people are starting to talk... LOL
I'll test for awhile and follow up, interesting how much the code has changed...

[robbe-haesendonck]

@djohnson-mulytic Are you still experiencing the issues with the setup?

[djohnson-mulytic]

unfortunately yes, still havent found the right fix...

[robbe-haesendonck]

If you've got the time I'm always willing to go on a call to troubleshoot with you.
Let me know :)