Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add supavisor to supabase #2712

Open
wants to merge 7 commits into
base: next
Choose a base branch
from
Open

Add supavisor to supabase #2712

wants to merge 7 commits into from

Conversation

Geczy
Copy link
Sponsor Contributor

@Geczy Geczy commented Jun 28, 2024

@RobertHH-IS
Copy link

Can you post the directions here? Discord link does not lead anywhere.

@Geczy
Copy link
Sponsor Contributor Author

Geczy commented Jun 28, 2024

Sure. Here it is

and also need to run this in supavisor

curl  -X PUT \
  'http://172.28.0.7:4000/api/tenants/dev_tenant' \
  --header 'Accept: */*' \
  --header 'User-Agent: Thunder Client (https://www.thunderclient.com)' \
  --header 'Authorization: Bearer YOUR_SUPABASE_SERVICE_KEY_HERE' \
  --header 'Content-Type: application/json' \
  --data-raw '{
  "tenant": {
    "db_host": "supabase-db",
    "db_port": 5432,
    "db_database": "postgres",
    "ip_version": "auto",
    "enforce_ssl": false,
    "require_user": false,
    "auth_query": "SELECT rolname, rolpassword FROM pg_authid WHERE rolname=$1;",
    "users": [
      {
        "db_user": "postgres",
        "db_password": "YOUR_DB_PASSWORD_HERE",
        "pool_size": 20,
        "mode_type": "transaction",
        "is_manager": true
      }
    ]
  }
}'

there might be a way to set this up without running this, like with an init script tho

and my url looks like this

:6543/postgres?pgbouncer=true&connection_limit=1&options=reference%3Ddev_tenant

- PROXY_PORT_TRANSACTION=6543
- DATABASE_URL=ecto://postgres:${SERVICE_PASSWORD_POSTGRES}@${POSTGRES_HOST:-supabase-db}:${POSTGRES_PORT:-5432}/${POSTGRES_DB:-postgres}
- CLUSTER_POSTGRES=true
- SECRET_KEY_BASE=12345678901234567890121234567890123456789012345678903212345678901234567890123456789032123456789012345678901234567890323456789032

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These probably shouldn't be hard coded

Copy link
Sponsor Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you recommend? They need to specifically be this length

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally there's a way to generate a random string either through docker-compose or coolify, I couldn't find one though.

Considering this should be unique to every user, and I don't imagine many people are changing the defaults while using these, I'd either wait for the functionality to be added into coolify, or force the user to enter the information.

You can get docker-compose to error if an environment variable is missing using ${VAR:?error} so that would mean changing it to: SECRET_KEY_BASE=${SECRET_KEY_BASE:?error}.

Supabase is a fairly popular template, so people would need to be told that they need to put in a variable that contains a string that is X character long, but I don't know where a good place for the instruction to go would be.

Failing all that, maybe setting it to SECRET_KEY_BASE=${SECRET_KEY_BASE} and finding where the default values for Dashboard User and Dashboard Password are generated and ensure a new one is generated for the 2 variables that need them.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Appears there is some functionality to do this:

case 'PASSWORD':
$generatedValue = Str::password(symbols: false);
break;
case 'PASSWORD_64':
$generatedValue = Str::password(length: 64, symbols: false);
break;
// This is not base64, it's just a random string
case 'BASE64_64':
$generatedValue = Str::random(64);
break;
case 'BASE64_128':
$generatedValue = Str::random(128);
break;
case 'BASE64':
case 'BASE64_32':
$generatedValue = Str::random(32);
break;
// This is base64,
case 'REALBASE64_64':
$generatedValue = base64_encode(Str::random(64));
break;
case 'REALBASE64_128':
$generatedValue = base64_encode(Str::random(128));
break;
case 'REALBASE64':
case 'REALBASE64_32':
$generatedValue = base64_encode(Str::random(32));
break;
case 'USER':
$generatedValue = Str::random(16);
break;
case 'SUPABASEANON':
$signingKey = $service->environment_variables()->where('key', 'SERVICE_PASSWORD_JWT')->first();
if (is_null($signingKey)) {
return;
} else {
$signingKey = $signingKey->value;
}
$key = InMemory::plainText($signingKey);
$algorithm = new Sha256();
$tokenBuilder = (new Builder(new JoseEncoder(), ChainedFormatter::default()));
$now = new DateTimeImmutable();
$now = $now->setTime($now->format('H'), $now->format('i'));
$token = $tokenBuilder
->issuedBy('supabase')
->issuedAt($now)
->expiresAt($now->modify('+100 year'))
->withClaim('role', 'anon')
->getToken($algorithm, $key);
$generatedValue = $token->toString();
break;
case 'SUPABASESERVICE':
$signingKey = $service->environment_variables()->where('key', 'SERVICE_PASSWORD_JWT')->first();
if (is_null($signingKey)) {
return;
} else {
$signingKey = $signingKey->value;
}
$key = InMemory::plainText($signingKey);
$algorithm = new Sha256();
$tokenBuilder = (new Builder(new JoseEncoder(), ChainedFormatter::default()));
$now = new DateTimeImmutable();
$now = $now->setTime($now->format('H'), $now->format('i'));
$token = $tokenBuilder
->issuedBy('supabase')
->issuedAt($now)
->expiresAt($now->modify('+100 year'))
->withClaim('role', 'service_role')
->getToken($algorithm, $key);
$generatedValue = $token->toString();
break;
default:
$generatedValue = Str::random(16);
break;
}
return $generatedValue;
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this will work.

- SECRET_KEY_BASE=${SERVICE_BASE64_128_SUPAVISOR}
- VAULT_ENC_KEY=${SERVICE_PASSWORD_64_SUPAVISOR}

Doesn't actually need to be 128 characters I don't think, fly deployment docs suggest some secure value: https://supabase.github.io/supavisor/deployment/fly/

Might be worth setting them both to PASSWORD_64 if only for better naming.

@Mortalife
Copy link

Thinking on the issue of running the dev_tenant curl.

Sure. Here it is

and also need to run this in supavisor


there might be a way to set this up without running this, like with an init script tho

and my url looks like this

:6543/postgres?pgbouncer=true&connection_limit=1&options=reference%3Ddev_tenant

Maybe this could also be a run once container like the minio create-bucket, using a curl image and overriding the entrypoint.

@Geczy
Copy link
Sponsor Contributor Author

Geczy commented Jul 1, 2024

the vault secret has to be 32 characters:

As a general note, if you are not using the Makefile you will have to set a
VAULT_ENC_KEY which should be at least 32 bytes long.

and this tenant business might actually be even easier than that. i think we can just add this to the sql file where i create the _supavisor schema. but we have to encrypt the password on the fly -- which we probably do need a cron image for, unless sql can do it for us?

  config :supavisor, Supavisor.Vault,
    ciphers: [
      default: {
        Cloak.Ciphers.AES.GCM,
        tag: "AES.GCM.V1", key: System.get_env("VAULT_ENC_KEY")
      }
    ]
SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = ON;

SELECT pg_catalog.Set_config('search_path', '', false);

SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = OFF;

--
-- Data for Name: tenants; Type: TABLE DATA; Schema: _supavisor; Owner: postgres
--
INSERT INTO _supavisor.tenants
            (id,
             external_id,
             db_host,
             db_port,
             db_database,
             inserted_at,
             updated_at,
             default_parameter_status,
             ip_version,
             upstream_ssl,
             upstream_verify,
             upstream_tls_ca,
             enforce_ssl,
             require_user,
             auth_query,
             default_pool_size,
             sni_hostname,
             default_max_clients,
             client_idle_timeout,
             default_pool_strategy,
             client_heartbeat_interval,
             allow_list)
VALUES      ('7453afc5-d04c-45f7-b0ed-11edd51a3279',
             'dev_tenant',
             'supabase-db',
             5432,
             'postgres',
             '2024-06-28 20:14:29',
             '2024-06-28 20:14:29',
             '{"server_version": "15.1 (Ubuntu 15.1-1.pgdg20.04+1)"}',
             'auto',
             false,
             NULL,
             NULL,
             false,
             false,
             'SELECT rolname, rolpassword FROM pg_authid WHERE rolname=$1;',
             15,
             NULL,
             1000,
             0,
             'fifo',
             60,
             '{0.0.0.0/0,::/0}');

--
-- Data for Name: users; Type: TABLE DATA; Schema: _supavisor; Owner: postgres
--
INSERT INTO _supavisor.users
            (id,
             db_user_alias,
             db_user,
             db_pass_encrypted,
             pool_size,
             mode_type,
             is_manager,
             tenant_external_id,
             inserted_at,
             updated_at,
             pool_checkout_timeout,
             max_clients)
VALUES      ('524e2da5-b4cb-4917-88ed-ad6a304649e8',
             'postgres',
             'postgres',
             '-the-encrypted-password-here-',
             20,
             'transaction',
             true,
             'dev_tenant',
             '2024-06-28 20:14:29',
             '2024-06-28 20:14:29',
             60000,
             NULL); 

the tricky part is to get the encrypted password there

@Mortalife
Copy link

Mortalife commented Jul 1, 2024

  config :supavisor, Supavisor.Vault,
    ciphers: [
      default: {
        Cloak.Ciphers.AES.GCM,
        tag: "AES.GCM.V1", key: System.get_env("VAULT_ENC_KEY")
      }
    ]

Chatgpt gave me a confidently incorrect answer, but it doesn't look like pgcrypto can be used with AES-GCM, it also suggested some code using the v8 extension to run js, but honestly at this point just sending the curl is probably best option.

Shame pg_net doesn't support PUT otherwise we could've used that in sql to send the request.

@Geczy
Copy link
Sponsor Contributor Author

Geczy commented Jul 1, 2024

so something like this could work?

services:
  init-curl:
    image: curlimages/curl:8.8.0
    command: >
      sh -c "curl -X PUT 'http://supabase-supavisor:4000/api/tenants/dev_tenant' \
      --header 'Accept: */*' \
      --header 'User-Agent: Thunder Client (https://www.thunderclient.com)' \
      --header 'Authorization: Bearer ${SERVICE_SUPABASEANON_KEY}' \
      --header 'Content-Type: application/json' \
      --data-raw '{
        \"tenant\": {
          \"db_host\": \"${POSTGRES_HOST:-supabase-db}\",
          \"db_port\": ${POSTGRES_PORT:-5432},
          \"db_database\": \"${POSTGRES_DB:-postgres}\",
          \"ip_version\": \"auto\",
          \"enforce_ssl\": false,
          \"require_user\": false,
          \"auth_query\": \"SELECT rolname, rolpassword FROM pg_authid WHERE rolname=$1;\",
          \"users\": [
            {
              \"db_user\": \"postgres\",
              \"db_password\": \"${SERVICE_PASSWORD_POSTGRES}\",
              \"pool_size\": 20,
              \"mode_type\": \"transaction\",
              \"is_manager\": true
            }
          ]
        }
      }'"
    depends_on:
      - supabase-supavisor
    entrypoint: [ "sh", "-c" ]

  supabase-supavisor:
    ...
    depends_on:
      supabase-db:
        condition: service_healthy
      init-curl:
        condition: service_completed_successfully

@Mortalife
Copy link

Mortalife commented Jul 1, 2024

init-curl:
    image: curlimages/curl:8.8.0

I think the only thing potentially missing is: restart: "no" inside the init-curl service, and I'd probably name the service something like supavisor-setup but yeah. Looks like it'll probably work to me.

One last thing, I wonder if those string replacements will work. If not you may need to link the environment variables to the curl container and pass them through using bash.

https://github.com/coollabsio/coolify/blob/main/templates/compose/supabase.yaml#L1053-L1071

The minio gives an example of how this is done with a docker-compose defined file.

@Geczy
Copy link
Sponsor Contributor Author

Geczy commented Jul 1, 2024

i updated the pr, but needs testing

@Geczy
Copy link
Sponsor Contributor Author

Geczy commented Jul 1, 2024

you can check with

psql -c "SELECT * from _supavisor.tenants;" "postgres://postgres:[email protected]:5432"

Comment on lines +619 to +652
entrypoint: [ "sh", "-c" ]
command: >
#!/bin/sh

while ! curl -sSfL --head -o /dev/null -H "Authorization: Bearer $SETUP_SERVICE_SUPABASEANON_KEY" http://supabase-supavisor:4000/api/health; do
sleep 2;
done;

curl -X PUT 'http://supabase-supavisor:4000/api/tenants/dev_tenant' \
--header 'Accept: */*' \
--header 'User-Agent: Thunder Client (https://www.thunderclient.com)' \
--header 'Authorization: Bearer $SETUP_SERVICE_SUPABASEANON_KEY' \
--header 'Content-Type: application/json' \
--data-raw '{
"tenant": {
"db_host": "${SETUP_POSTGRES_HOST}",
"db_port": ${SETUP_POSTGRES_PORT},
"db_database": "${SETUP_POSTGRES_DB}",
"ip_version": "auto",
"enforce_ssl": false,
"require_user": false,
"auth_query": "SELECT rolname, rolpassword FROM pg_authid WHERE rolname=$1;",
"users": [
{
"db_user": "postgres",
"db_password": "${SETUP_SERVICE_PASSWORD_POSTGRES}",
"pool_size": 20,
"mode_type": "transaction",
"is_manager": true
}
]
}
}'
exit 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

entrypoint: sh
command: /docker-entrypoint-initdb.d/supavisor.sh
volumes:
  -
    type: bind
    source: ./volumes/supavisor.sh
    target: /docker-entrypoint-initdb.d/supavisor.sh
    content: |
        while ! curl -v -H "Authorization: Bearer ${SETUP_SERVICE_SUPABASEANON_KEY}" http://supabase-supavisor:4000/api/health; 
        do sleep 2;
        done;
        
        curl  -X PUT -v \
          'http://supabase-supavisor:4000/api/tenants/dev_tenant' \
          --header 'Accept: */*' \
          --header 'User-Agent: Thunder Client (https://www.thunderclient.com)' \
          --header "Authorization: Bearer ${SETUP_SERVICE_SUPABASEANON_KEY}" \
          --header 'Content-Type: application/json' \
          --data '
        {
          "tenant": {
            "db_host": "'"${SETUP_POSTGRES_HOST}"'",
            "db_port": '$SETUP_POSTGRES_PORT',
            "db_database": "postgres",
            "ip_version": "auto",
            "enforce_ssl": false,
            "require_user": false,
            "auth_query": "SELECT rolname, rolpassword FROM pg_authid WHERE rolname=$1;",
            "users": [
              {
                "db_user": "postgres",
                "db_password": "'"$SETUP_SERVICE_PASSWORD_POSTGRES"'",
                "pool_size": 20,
                "mode_type": "transaction",
                "is_manager": true
              },
            ]
          }
        }'
        
        exit 0

fixes the formatting issues in the compose and vars in single quotes
also makes it much easier to add extra users, since you prob dont want to be using postgres user account you can just edit storage file

@andrasbacsai andrasbacsai added the ⚙️ New Service Issues requesting or PRs adding new service templates. label Aug 27, 2024
@Mortalife
Copy link

Mortalife commented Aug 30, 2024

I have a much simpler way working:

supabase-supavisor:
    image: 'supabase/supavisor:1.1.56'
    healthcheck:
      test:
        - CMD
        - curl
        - '-sSfL'
        - '-o'
        - /dev/null
        - 'http://127.0.0.1:4000/api/health'
      timeout: 5s
      interval: 5s
      retries: 10
    restart: unless-stopped
    ports:
        - "5432:5432"
        - "6543:6543"
    depends_on:
      supabase-db:
        condition: service_healthy
    environment:
      - PORT=4000
      - PROXY_PORT_SESSION=5432
      - PROXY_PORT_TRANSACTION=6543
      - 'DATABASE_URL=ecto://supabase_admin:${SERVICE_PASSWORD_POSTGRES}@${POSTGRES_HOSTNAME:-supabase-db}:${POSTGRES_PORT:-5432}/${POSTGRES_DB:-postgres}'
      - CLUSTER_POSTGRES=true
      - 'SECRET_KEY_BASE=${SERVICE_PASSWORD_SUPAVISORSECRET}'
      - 'VAULT_ENC_KEY=${SERVICE_PASSWORD_VAULTENC}'
      - 'API_JWT_SECRET=${SERVICE_PASSWORD_JWT}'
      - 'METRICS_JWT_SECRET=${SERVICE_PASSWORD_JWT}'
      - REGION=local
      - 'ERL_AFLAGS=-proto_dist inet_tcp'
    command:
      - sh
      - '-c'
      - "/app/bin/migrate && /app/bin/supavisor eval '{:ok, _} = Application.ensure_all_started(:supavisor)\n{:ok, version} =\n  case Supavisor.Repo.query!(\"select version()\") do\n    %{rows: [[ver]]} -> Supavisor.Helpers.parse_pg_version(ver)\n    _ -> nil\n  end\nparams = %{\n  \"external_id\" => \"${SUPAVISOR_TENANT:-dev_tenant}\",\n  \"db_host\" => \"${POSTGRES_HOSTNAME}\",\n  \"db_port\" => ${POSTGRES_PORT},\n  \"db_database\" => \"${POSTGRES_DB}\",\n  \"require_user\" => false,\n  \"auth_query\" => \"SELECT * FROM pgbouncer.get_auth($1)\",\n  \"default_max_clients\" => 100,\n  \"default_pool_size\" => 20,\n  \"default_parameter_status\" => %{\"server_version\" => version},\n  \"users\" => [%{\n    \"db_user\" => \"pgbouncer\",\n    \"db_password\" => \"${SERVICE_PASSWORD_POSTGRES}\",\n    \"mode_type\" => \"transaction\",\n    \"pool_size\" => 20,\n    \"is_manager\" => true\n  }]\n}\nif !Supavisor.Tenants.get_tenant_by_external_id(params[\"external_id\"]) do\n  {:ok, _} = Supavisor.Tenants.create_tenant(params)\nend' && /app/bin/server"

This is based off what it does within the cli when running it locally.

Requires the change made to the db volumes:

      - type: bind
        source: ./volumes/db/supavisor.sql
        target: /docker-entrypoint-initdb.d/migrations/99-supavisor.sql
        content: |
          \set pguser `echo "supabase_admin"`
          create schema if not exists _supavisor;
          alter schema _supavisor owner to :pguser;

@djsisson
Copy link
Contributor

@Mortalife that's great, can you modify it so that the script is in a file and use a volume bind, just so its much easier to read and edit

i guess you could also add a psql line to create if not exists the schema as well, since it wont run if the database already exists.

@Geczy
Copy link
Sponsor Contributor Author

Geczy commented Sep 10, 2024

supavisor

has anyone seen this in the supavisor container? i get hundreds of these

ClientHandler: User requested SSL connection but no downstream cert/key found

minio

and for this container
supabase-minio-aggc400

i get hundreds of these every day:
Error: Storage resources are insufficient for the write operation .minio.sys/buckets/.bloomcycle.bin, Object buckets/.bloomcycle.bin (*fmt.wrapError)

minio/minio#19622
minio/minio#17040
minio/minio#17875

for minio, but idk how coolify comes into play here. i have over 200gb free space

Copy link

gitguardian bot commented Sep 19, 2024

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
- - GitHub App Keys ccbbfd8 database/seeders/GithubAppSeeder.php View secret
- - Generic Password e1bcae7 templates/compose/resend.yaml View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secrets safely. Learn here the best practices.
  3. Revoke and rotate these secrets.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@rajaiswal
Copy link

@Geczy Thanks for all the efforts towards this! Do we have a list of all things blocking this from getting merged?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
⚙️ New Service Issues requesting or PRs adding new service templates.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants