...

[ghost]

,,,

[manics]

Dropbox has some specific requirements related to how filesystem notifications are obtained: https://docs.openmicroscopy.org/omero/5.4.10/sysadmins/dropbox.html
It may be the case that the /OMERO docker volume is incompatible with Dropbox.

[manics]

We haven't done much testing with DropBox on Docker. My guess is the same as yours, there's probably an issue with notifications passing from the filesystem on the host through to the container. It may also depend on the filesystem, kernel version and docker version.

I'll see if anyone else can suggest how to verify whether the problem is with notifications, or with something else.

[joshmoore]

Hi @G-9999 . Could you try running run.sh from https://gist.github.com/joshmoore/9768d02568d8a166f2babfa7ef5875fe on your system and let us know if you see any results?

In my case, in terminal 1:

$ ./run.sh
Press Enter to quit...
[('/dropbox/a', Create)]
[('/dropbox/a', Modify)]
[('/dropbox/a', Modify)]
[('/dropbox/a', Delete)]

In terminal 2:

$ touch a
$ date >> a
$ rm a

[joshmoore]

Hi @G-9999 . Ok, trying to figure out where we are:

• The main problem you listed seems to be gone, making it difficult to debug. Can I propose we assume something else was wrong and focus on the usability as it currently stands?
• The primary usability issue is the time it takes to import? If so, we'll need to look into the precise timings. There are definitely several wait configurations so that groups of files will be detected together. If you know you will always be importing only single-file filesets, then this could perhaps be lowered.
• Which leaves "until after I delete the .jpg files in the DropBox folder" which I find concerning. Can you explain what happened?

[joshmoore]

Wow. Thanks for the doc & the scripts, @G-9999. This makes it clear what you're trying to achieve. I don't have much experience with portainer or rclone though, so can I suggest we whittle down some of the steps from your side for me to reproduce?

Starting at the minimum of just spinning up OMERO with this:

version: "3.2"

services:
  database:
    image: "postgres:9.6"
    environment:
      - POSTGRES_USER=omero
      - POSTGRES_DB=omero
      - POSTGRES_PASSWORD=omero
    networks:
      - omero
    volumes:
      - "database:/var/lib/postgresql/data"

  omero:
    image: openmicroscopy/omero-server:5.4
    environment:
      - CONFIG_omero_db_host=database
      - CONFIG_omero_db_user=omero
      - CONFIG_omero_db_pass=omero
      - CONFIG_omero_db_name=omero
      - ROOTPASS=omero
    networks:
      - omero
    ports:
      - "4063:4063"
      - "4064:4064"
    volumes:
      - "omero:/OMERO"

  web:
    image: openmicroscopy/omero-web-standalone:5.4
    environment:
      - OMEROHOST=omero
    networks:
      - omero
    ports:
      - "4080:4080"

networks:
  omero:
    driver: overlay
    attachable: true

volumes:
  database:
  omero:

If I run this script:

docker-compose exec omero mkdir /OMERO/DropBox/root
docker-compose exec omero touch /OMERO/DropBox/root/a.fake
docker-compose exec omero tail -f /opt/omero/server/OMERO.server/var/log/DropBox.log

I see:

[jamoore@idr1-slot2 g-9999]$ bash test.sh
2019-04-09 09:20:34,047 INFO  [                        fsclient.DropBox] (MainThread) Created monitor with id = ba653d5a-5aa8-11e9-969b-0242c0a88004
2019-04-09 09:20:34,049 INFO  [                        fsclient.DropBox] (MainThread) Started OMERO.fs DropBox client
2019-04-09 09:20:47,573 INFO  [                            omero.remote] (Dummy-4   )  Meth: MonitorClientI.fsEventHappened
2019-04-09 09:20:47,574 INFO  [         fsclient.fsDropBoxMonitorClient] (Dummy-4   ) EVENT_RECORD::Cookie::1554801647.57::Batch::1
2019-04-09 09:20:47,574 INFO  [         fsclient.fsDropBoxMonitorClient] (Dummy-4   ) EVENT_RECORD::Cookie::1554801647.57::Create::/OMERO/DropBox/root/a.fake
2019-04-09 09:20:47,711 INFO  [                            omero.remote] (Dummy-4   )  Rslt: None
2019-04-09 09:20:49,907 INFO  [         fsclient.fsDropBoxMonitorClient] (Thread-3  ) Processing 1 events (1 ids). 0 remaining
2019-04-09 09:20:49,907 INFO  [         fsclient.fsDropBoxMonitorClient] (Thread-3  ) Getting filesets on : set(['/OMERO/DropBox/root/a.fake'])
2019-04-09 09:20:53,338 INFO  [         fsclient.fsDropBoxMonitorClient] (Thread-3  ) EVENT_RECORD::Cookie::1554801653.34::Filesets::{'/OMERO/DropBox/root/a.fake': ['/OMERO/DropBox/root/a.fake']}
2019-04-09 09:20:53,339 INFO  [         fsclient.fsDropBoxMonitorClient] (Thread-3  ) New entry /OMERO/DropBox/root/a.fake contains 1 file(s). Files=1 Timers=1
[jamoore@idr1-slot2 g-9999]$ vi test.sh
[jamoore@idr1-slot2 g-9999]$ bash test.sh
2019-04-09 09:20:34,047 INFO  [                        fsclient.DropBox] (MainThread) Created monitor with id = ba653d5a-5aa8-11e9-969b-0242c0a88004
2019-04-09 09:20:34,049 INFO  [                        fsclient.DropBox] (MainThread) Started OMERO.fs DropBox client
2019-04-09 09:20:47,573 INFO  [                            omero.remote] (Dummy-4   )  Meth: MonitorClientI.fsEventHappened
2019-04-09 09:20:47,574 INFO  [         fsclient.fsDropBoxMonitorClient] (Dummy-4   ) EVENT_RECORD::Cookie::1554801647.57::Batch::1
2019-04-09 09:20:47,574 INFO  [         fsclient.fsDropBoxMonitorClient] (Dummy-4   ) EVENT_RECORD::Cookie::1554801647.57::Create::/OMERO/DropBox/root/a.fake
2019-04-09 09:20:47,711 INFO  [                            omero.remote] (Dummy-4   )  Rslt: None
2019-04-09 09:20:49,907 INFO  [         fsclient.fsDropBoxMonitorClient] (Thread-3  ) Processing 1 events (1 ids). 0 remaining
2019-04-09 09:20:49,907 INFO  [         fsclient.fsDropBoxMonitorClient] (Thread-3  ) Getting filesets on : set(['/OMERO/DropBox/root/a.fake'])
2019-04-09 09:20:53,338 INFO  [         fsclient.fsDropBoxMonitorClient] (Thread-3  ) EVENT_RECORD::Cookie::1554801653.34::Filesets::{'/OMERO/DropBox/root/a.fake': ['/OMERO/DropBox/root/a.fake']}
2019-04-09 09:20:53,339 INFO  [         fsclient.fsDropBoxMonitorClient] (Thread-3  ) New entry /OMERO/DropBox/root/a.fake contains 1 file(s). Files=1 Timers=1
2019-04-09 09:21:53,339 INFO  [         fsclient.fsDropBoxMonitorClient] (Thread-5  ) Removed key /OMERO/DropBox/root/a.fake
2019-04-09 09:21:53,410 INFO  [         fsclient.fsDropBoxMonitorClient] (Thread-5  ) Importing /OMERO/DropBox/root/a.fake (session=56e8544d-219b-49ac-97f6-bbfb90ab826b)
2019-04-09 09:21:53,722 INFO  [                    omero.util.Resources] (Thread-6  ) Starting
2019-04-09 09:21:53,737 WARNI [                                  stderr] (Thread-5  ) Joined session for root@localhost:4064. Idle timeout: 10 min. Current group: system
2019-04-09 09:22:01,385 INFO  [         fsclient.fsDropBoxMonitorClient] (Thread-5  ) Import of /OMERO/DropBox/root/a.fake completed (session=56e8544d-219b-49ac-97f6-bbfb90ab826b)
2019-04-09 09:22:01,408 INFO  [                    omero.util.Resources] (Thread-6  ) Halted

Do you get that far or does this already require restarts? A potential worry that we're trying to track down is the particular interaction with your filesystem. Can you perhaps tell us about what /var/lib/docker and more importantly /var/lib/docker/volumes is stored on? (Or similar if you've installed differently.) In my case, I'm testing on a LVM volume with ext4. If the above doesn't work, what happens if you configure DropBox to read from /opt/omero/dropbox (i.e. from the same mount as the server)?

[joshmoore]

Hey @G-9999. Sorry to have left you with a headache; I didn't realize you were trying to use a stack. I've updated the docker-compose file above (version: "3.2"; attachable: true) so that you can connect to the stack once it's started. e.g.:

$ docker run --net omero0000_omero -ti --rm --entrypoint=python openmicroscopy/omero-server /opt/omero/server/OMERO.server/bin/omero login root@omero

[joshmoore]

forgive the learning curve :)

No worries. I'm learning in the process!

I had to lookup the omero-server container's network name

I would hope you never need to do this. But if it's working, let's go with it.

also pass the -w option

Makes sense. It's likely possible to also achieve login with the environment variable OMERO_PASSWORD.

post the results here later on.

Looking forward to it.

[manics]

Thanks for the update!

[joshmoore]

Nice. Thanks @funqtion. If I understand correctly, the tar method was leading to the error you were seeing? (KeyError: path(u'/OMERO')) If so, this points to something that could use fixing in the Python code, but I'm inclined to say that this issue can be closed. I'm copying your scanner code here lest anything happen to your gist. If others find this useful, perhaps something similar could be added to OMERO core.

All the best, ~Josh.

# requires python 3.5+
import os
import os.path
from pathlib import Path
from shlex import quote

count = 0
DIR = '/opt/dropbox-mount/NLM_Malaria_Screener'
DOCKER = '/usr/local/bin/docker'
OMERO = 'omero_omero_1'
users = [f.name for f in os.scandir(DIR) if f.is_dir() ]    
for user in users:
    csv = Path(DIR+'/'+user+'/MalariaScreenerDB.csv')
    if csv.is_file():
        print(user+' ✔ csv!')
        subdirs = [f.name for f in os.scandir(DIR+'/'+user) if f.is_dir() ] 
        for subdir in subdirs:
            subdirq = quote(subdir)
            print('- '+subdirq)
            images = [f.name for f in os.scandir(DIR+'/'+user+'/'+subdir) if f.is_file() ]
            for image in images:
                if image.endswith('.png'):
                    imageq = quote(image)
                    omero = Path(DIR+'/'+user+'/'+subdir+'/'+image+'.omero')
                    if omero.is_file():
                        print('- - '+image+' ✔')
                    elif count < 5:
                        print('- - '+image+' 🔜')
                        Path(DIR+'/'+user+'/'+subdir+'/'+image+'.omero').touch()
                        os.system(DOCKER+' exec '+OMERO+' mkdir -p /OMERO/DropBox/'+user)
                        os.system(DOCKER+' cp '+DIR+'/'+user+'/'+subdirq+'/'+imageq+' '+OMERO+':/OMERO/DropBox/'+user+'/'+subdirq+'_'+imageq)
                        count += 1
                    else:
                        print('- - '+image+' 🐄')
    else:
        print(user+' ✖ no csv')

[joshmoore]

may/may not have been the problem with the tar import method I attempted.

Noted.

It's been working great for a few weeks!

Sounds like I should go on vacation more often! Thanks again, @funqtion.