Add support for CIFS (and S3) volumes

[CC007]

The issue

To my knowledge, right now the one-click database resources (like PostgreSQL) only support standard docker volumes or bind-mounts, so to install a postgres database using a CIFS volume, you'd have to create a docker-compose file and specify the CIFS volume there, to be used by a postgres service.

This takes away from the one-click nature of creating a database, since you'd have to define the service, along with all the environment, port and image configuration

Alternatives

My server runs on a raspberry pi with only some sd-card storage, so running my postgres on a local volume is no option for me.

Solution

It would be nice if you could check a "Use network storage" checkmark, which gives you the option to provide a CIFS domain/ip, along with the share name and the folder to put the data in (or a single url field with the whole path) and the credentials.

Optionally, since you already support s3 buckets for backups, it would also be nice to use an s3 bucket as a volume. This might not be optimal for the random reads and writes of a database, but it might be useful for other file storage purposes. This could either be an Amazon s3 bucket or a self-hosted s3-compatible bucket hosted on something like a NAS.

[CC007]

Correction

Even docker-compose doesn't seem to work, since it modifies the docker-compose to point to a different volume, without any ability to edit it, so CIFS isn't working either through the direct Volume dialog or through a docker-compose file.

[CC007]

Here's my docker-compose.yml file:

services:
  db:
    image: postgres:latest
    volumes:
      - pg_data:/var/lib/postgresql/data
      - ./init-scripts:/docker-entrypoint-initdb.d
    environment:
      POSTGRES_USER: "postgres"
      POSTGRES_PASSWORD: "{{ environment.POSTGRES_PASSWORD }}"
      ADMIN_USER: "{{ project.ADMIN_USER }}"
      ADMIN_PASSWORD: "{{ environment.ADMIN_PASSWORD }}"
      APP_USER: "{{ project.APP_USER }}"
      APP_PASSWORD: "{{ environment.APP_PASSWORD }}"
    ports:
      - "5432:5432"

volumes:
  pg_data:
    driver: local
    driver_opts:
      type: cifs
      device: "//{{ CIFS_IP }}/{{ CIFS_SHARE }}/{{ CIFS_SUBFOLDER }}"
      username: "{{ CIFS_USERNAME }}"
      password: "{{ CIFS_PASSWORD }}"

And here is how Coolify transformed it:

services:
  db:
    image: 'postgres:latest'
    volumes:
      - 'qkscc8480wws44kc4wkcskwg-pg_data:/var/lib/postgresql/data'
      - '/data/coolify/applications/qkscc8480wws44kc4wkcskwg/init-scripts:/docker-entrypoint-initdb.d'
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: '{{ environment.POSTGRES_PASSWORD }}'
      ADMIN_USER: '{{ project.ADMIN_USER }}'
      ADMIN_PASSWORD: '{{ environment.ADMIN_PASSWORD }}'
      APP_USER: '{{ project.APP_USER }}'
      APP_PASSWORD: '{{ environment.APP_PASSWORD }}'
    ports:
      - '5432:5432'
    networks:
      qkscc8480wws44kc4wkcskwg: null
    labels:
      - coolify.managed=true
      - coolify.version=4.0.0-beta.323
      - coolify.applicationId=3
      - coolify.type=application
      - coolify.name=db-qkscc8480wws44kc4wkcskwg-135516574471
      - coolify.pullRequestId=0
    restart: unless-stopped
    container_name: db-qkscc8480wws44kc4wkcskwg-135516574471
volumes:
  pg_data:
    driver: local
    driver_opts:
      type: cifs
      device: '//{{ CIFS_IP }}/{{ CIFS_SHARE }}/{{ CIFS_SUBFOLDER }}'
      username: '{{ CIFS_USERNAME }}'
      password: '{{ CIFS_PASSWORD }}'
  qkscc8480wws44kc4wkcskwg-pg_data:
    name: qkscc8480wws44kc4wkcskwg-pg_data
networks:
  qkscc8480wws44kc4wkcskwg:
    name: qkscc8480wws44kc4wkcskwg
    external: true
configs: {  }
secrets: {  }

As you can see, the service no longer points to the CIFS volume, but to an internal volume. Yes, I could probably hack it by mounting a CIFS on the host machine and creating a sym link to the correct folder at /var/lib/docker/volumes/qkscc8480wws44kc4wkcskwg-pg_data, but I shouldn't need to have to go through all that hassle.

[LEstradioto]

maybe you could try to change the volume name after deployed in here?

do you think PR #2710 would help in this case? setting up coolify base path different from the default?

[CC007]

I noticed that coolify doesn't take the git repo as the working directory, so the relative path ./init-scripts didn't work for me.
I changed my docker-compose file to this:

services:
  db:
    image: postgres:latest
    volumes:
      - db-pg-data:/var/lib/postgresql/data
      - db-init-scripts:/docker-entrypoint-initdb.d
    environment:
      POSTGRES_USER: "postgres"
      POSTGRES_PASSWORD: "{{ environment.POSTGRES_PASSWORD }}"
      ADMIN_USER: "{{ project.ADMIN_USER }}"
      ADMIN_PASSWORD: "{{ environment.ADMIN_PASSWORD }}"
      APP_USER: "{{ project.APP_USER }}"
      APP_PASSWORD: "{{ environment.APP_PASSWORD }}"
    ports:
      - "5432:5432"

volumes:
  db-pg-data:
    driver: local
    driver_opts:
      type: cifs
      device: "//{{ CIFS_IP }}/{{ CIFS_SHARE }}/{{ CIFS_DATAFOLDER }}"
      username: "{{ CIFS_USERNAME }}"
      password: "{{ CIFS_PASSWORD }}"
  db-init-scripts:
    driver: local
    driver_opts:
      type: cifs
      device: "//{{ CIFS_IP }}/{{ CIFS_SHARE }}/{{ CIFS_INITFOLDER }}"
      username: "{{ CIFS_USERNAME }}"
      password: "{{ CIFS_PASSWORD }}"

When I look at the Storages page though, I see the following message:

So I'm unable to change the volume name.

As for PR #2710 I don't need or want Coolify itself to be running off of network storage. I just need network storage to be an option when creating volumes for my applications

[LEstradioto]

take a look into this https://github.com/coollabsio/coolify-examples/blob/main/docker-compose-test/docker-compose-volumes.yaml
i think only the auto-generated appears on the storages (or should tweak something)

[CC007]

Ah, I didn't realize I needed the o option, instead of username/password field. It probably also explains the file permission issues in my other tests. I'm quite new to useing CIFS/SMB 😅

I'll try it out and see if that works.

My feature request still stands though, to be able to configure this from a dialog in the PostgreSQL resource (or any other resource that requires a volume), rather than having to create a docker-compose resource.

[CC007]

I tried a couple of things:

First, I tried to replicate the example as close as possible, including the cap_add and privileged and also with quoted volumes (I read somewhere that could be impactful):

services:
  db:
    image: postgres:latest
    volumes:
      - 'db-pg-data:/var/lib/postgresql/data'
      - 'db-init-scripts:/docker-entrypoint-initdb.d'
    cap_add:
      - SYS_ADMIN
      - DAC_READ_SEARCH
    privileged: true
    environment:
      POSTGRES_USER: "postgres"
      POSTGRES_PASSWORD: "{{ environment.POSTGRES_PASSWORD }}"
      ADMIN_USER: "{{ project.ADMIN_USER }}"
      ADMIN_PASSWORD: "{{ environment.ADMIN_PASSWORD }}"
      APP_USER: "{{ project.APP_USER }}"
      APP_PASSWORD: "{{ environment.APP_PASSWORD }}"
    ports:
      - "5432:5432"

volumes:
  db-pg-data:
    driver: local
    driver_opts:
      type: cifs
      device: '//{{ CIFS_IP }}/{{ CIFS_SHARE }}/{{ CIFS_DATAFOLDER }}'
      o: 'username={{ CIFS_USERNAME }},password={{ CIFS_PASSWORD }},vers=3.0,uid=0,gid=0,dir_mode=0777,file_mode=0777'
  db-init-scripts:
    driver: local
    driver_opts:
      type: cifs
      device: '//{{ CIFS_IP }}/{{ CIFS_SHARE }}/{{ CIFS_INITFOLDER }}'
      o: 'username={{ CIFS_USERNAME }},password={{ CIFS_PASSWORD }},vers=3.0,uid=0,gid=0,dir_mode=0777,file_mode=0777'

But that still gave an extra volume that wasn't on the CIFS drive and was used by postgres:

services:
  db:
    image: 'postgres:latest'
    volumes:
      - 'tss4sksg4c4oowkow4cwo4wk-db-pg-data:/var/lib/postgresql/data'
      - 'tss4sksg4c4oowkow4cwo4wk-db-init-scripts:/docker-entrypoint-initdb.d'
    cap_add:
      - SYS_ADMIN
      - DAC_READ_SEARCH
    privileged: true
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: '{{ environment.POSTGRES_PASSWORD }}'
      ADMIN_USER: '{{ project.ADMIN_USER }}'
      ADMIN_PASSWORD: '{{ environment.ADMIN_PASSWORD }}'
      APP_USER: '{{ project.APP_USER }}'
      APP_PASSWORD: '{{ environment.APP_PASSWORD }}'
    ports:
      - '5432:5432'
    networks:
      tss4sksg4c4oowkow4cwo4wk: null
    labels:
      - coolify.managed=true
      - coolify.version=4.0.0-beta.323
      - coolify.applicationId=6
      - coolify.type=application
      - coolify.name=db-tss4sksg4c4oowkow4cwo4wk-222838882067
      - coolify.pullRequestId=0
    restart: unless-stopped
    container_name: db-tss4sksg4c4oowkow4cwo4wk-222838882067
volumes:
  db-pg-data:
    driver: local
    driver_opts:
      type: cifs
      device: '//{{ CIFS_IP }}/{{ CIFS_SHARE }}/{{ CIFS_DATAFOLDER }}'
      o: 'username={{ CIFS_USERNAME }},password={{ CIFS_PASSWORD }},vers=3.0,uid=0,gid=0,dir_mode=0777,file_mode=0777'
  db-init-scripts:
    driver: local
    driver_opts:
      type: cifs
      device: '//{{ CIFS_IP }}/{{ CIFS_SHARE }}/{{ CIFS_INITFOLDER }}'
      o: 'username={{ CIFS_USERNAME }},password={{ CIFS_PASSWORD }},vers=3.0,uid=0,gid=0,dir_mode=0777,file_mode=0777'
  tss4sksg4c4oowkow4cwo4wk-db-pg-data:
    name: tss4sksg4c4oowkow4cwo4wk-db-pg-data
  tss4sksg4c4oowkow4cwo4wk-db-init-scripts:
    name: tss4sksg4c4oowkow4cwo4wk-db-init-scripts
networks:
  tss4sksg4c4oowkow4cwo4wk:
    name: tss4sksg4c4oowkow4cwo4wk
    external: true
configs: {  }
secrets: {  }

I tried to enable Raw Compose Deployment, but I got this error:

[2024-Aug-16 22:41:00.895343]                                          Oops something is not okay, are you okay? 😢

[2024-Aug-16 22:41:00.901534]                                          Container tss4sksg4c4oowkow4cwo4wk-db-1  Creating
Error response from daemon: failed to mount local volume: mount //{{ CIFS_IP }}/{{ CIFS_SHARE }}/{{ CIFS_DATAFOLDER }}:/var/lib/docker/volumes/tss4sksg4c4oowkow4cwo4wk_db-pg-data/_data, data: username={{ CIFS_USERNAME }},password=********,vers=3.0,uid=0,gid=0,dir_mode=0777,file_mode=0777: no route to host

I tried enabling network_mode: host, but I still got the same error.

As a sanity check, I mounted the CIFS share on the host manually and that connected without any issues.

[Rustypredator]

@CC007 If you havent found a good way to integrate your cifs share yet, i found this works pretty well.
I would only use this method coolify supports cifs volumes, this is obviously just a hacky workaround, and i dont know how stable it is.

You can just create a volume manually that connects to the share and tell your compose or image to use that volume.

1. Create a Service/Stack/image
2. Stop Service, Remove the volume (if you ran the container already)
3. Re-Create the volume using exactly the same name as coolify generated, but use the cifs driver (see below)
4. Deploy the service. It should now use the volume you created.
   • It may give you a warning about volumes being created outside of docker compose, but i havent found anything wrong with the volumes yet

To create a new docker volume that connects to cifs:

docker volume create \
            --driver local \
            --opt type=cifs \
            --opt device=//Your_SMB_Host_here/Share_Path_here \
            --name insert_coolifys_volume_name_here

If needed, you can also add user and group ids like this:

docker volume create \
            --driver local \
            --opt type=cifs \
            --opt device=//Your_SMB_Host_here/Share_Path_here \
            --opt uid=xxxx,gid=xxxx,filemode=xxxx,dirmode=xxxx # all of these are optional. i think every option that cifs allows works here
            --name insert_coolifys_volume_name_here