Update aspera-client-docker image to use the new download workflow #5

sbesson · 2023-12-19T09:00:55Z

This updates the source of the https://hub.docker.com/r/imagedata/download Docker image to use the new Aspera download workflow. This assumes the IDR team wants to keep supporting this image and download workflow. Otherwise, it should probably be archived/deprecated

Included in this PR are the minimal change to make the command functional again

Drops the study input in favor of the generic fasp-public user
Update the README with the new command format
Update the GitHub workflow to use recent GitHub actions and Docker labels
Upgrade the base image to Rocky Linux 9

Drops the study input in favor of the generic fasp-public user Update the README with the new command format

joshmoore · 2023-12-19T09:43:33Z

README.md

-| idr0034 | 20170421-original/Nathalie/HipSci Experiment 10 Plate 1__2014-08-21T17_57_35-Measurement1                   | experiment_10                               | 5.1G |
+        docker run -ti --rm -v /tmp/data:/data imagedata/download SOURCE TARGET
+
+where SOURCE can be any subfolder under https://ftp.ebi.ac.uk/pub/databases/IDR/. For instance


pwalczysko · 2023-12-20T10:42:38Z

Test 1 on Mac M1 - build and run the docker as-is in this PR, use the docker run... cmd from README and building an image test-seb:

aspera-client-docker %  docker build -f Dockerfile --rm --no-cache -t test-seb . 
aspera-client-docker %  docker run -ti --rm -v /Users/pwalczysko:/data test-seb idr0001-graml-sysgro/20151116-verified/JL_120731_S6A /data/JL_120731_S6A 
/home/data/.aspera/connect/bin/ascp -P33001 -l 40m -i /home/data/.aspera/connect/etc/asperaweb_id_dsa.openssh -k 0 [email protected]:/pub/databases/IDR/idr0001-graml-sysgro/20151116-verified/JL_120731_S6A /data/JL_120731_S6A
qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

As per https://stackoverflow.com/questions/71040681/qemu-x86-64-could-not-open-lib64-ld-linux-x86-64-so-2-no-such-file-or-direc/71611002#71611002, this ^^^ is caused by me being on M1. Went for a solution where I add the line into the Dockerfile FROM --platform=linux/amd64 centos:centos7 as per line 3 in sbesson@ab405c3, saved the file as DockerfileM1 and build a new image test-sebm1. This works fine, see below.

aspera-client-docker % docker build -f DockerfileM1 --rm --no-cache -t test-sebm1 .
aspera-client-docker % docker run -ti --rm -v /Users/pwalczysko:/data test-sebm1 idr0001-graml-sysgro/20151116-verified/JL_120731_S6A /data/JL_120731_S6A
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
/home/data/.aspera/connect/bin/ascp -P33001 -l 40m -i /home/data/.aspera/connect/etc/asperaweb_id_dsa.openssh -k 0 [email protected]:/pub/databases/IDR/idr0001-graml-sysgro/20151116-verified/JL_120731_S6A /data/JL_120731_S6A
001005001.flex                                                     53%   47MB 38.9Mb/s    00:09 ETA

Note: Adding a --platform flag to the docker run... cmd was not successful in my hands.
e.g.

docker run -ti --rm -v --platform linux/amd64 /Users/pwalczysko:/data test-seb2 idr0001-graml-sysgro/20151116-verified/JL_120731_S6A /data/JL_120731_S6A
Unable to find image 'linux/amd64:latest' locally
docker: Error response from daemon: pull access denied for linux/amd64, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.

Only the Dockerfile edit worked.

pwalczysko · 2023-12-20T10:56:39Z

Is it reasonable to keep using centos7 images for this workflow ?

pwalczysko

Two issues:

Please comment on the M1 support - imho, this type of platform is more and more prevalent and we could offer a solution in this repo.
centos7 ?

DockerfileM1

sbesson · 2023-12-20T11:59:41Z

Last commit should allow to build the Linux/amd64 image independently on whether the build machine is arm64 or amd64. Thanks @pwalczysko

On the base image, I am of the opinion we should change it to either Ubuntu 22.04 or Rocky Linux 9 but am not opinionated on either of them. Happy to follow the guidance of the IDR team

jburel · 2023-12-26T22:24:20Z

since the initial image was centOS 7, it might be more natural to switch to Rocky Linux 9

Copy workflow from omero-server-docker

pwalczysko · 2024-01-03T10:45:12Z

Tested last 2 commits (Rocky Linux 9 based image).

Works fine on

Mac Apple M1 Pro
Mac amd64 (old laptop)

joshmoore

👍 (Here's to an AND vs OR option for the requested reviewers 😉)

sbesson · 2024-01-29T13:20:07Z

Unless someone wants to take the lead on this, I'll merge this PR by EOB and tag 0.4.0, hoping the deployment logic is still functional

sbesson · 2024-01-29T16:05:45Z

.github/workflows/main.yml

        with:
          githubToken: ${{ secrets.GITHUB_TOKEN }}
-          prefix: "${{ steps.getprefix.outputs.prefix }}"
+          prefix: "${{ env.name }}"


Looks like I missed there was a breaking change and a : need to be appended as in ome/omero-server-docker@6f31737.

Unfortunately, this has created quite a few unwanted images at https://hub.docker.com/u/imagedata. @jburel @joshmoore do you have the credentials to delete those? I can open a PR for a follow-up 0.4.1 release (unless you'd prefer me to delete the tag).

Update aspera-client-docker image to use the new download workflow

7a0b8aa

Drops the study input in favor of the generic fasp-public user Update the README with the new command format

sbesson requested review from jburel and pwalczysko December 19, 2023 09:00

joshmoore reviewed Dec 19, 2023

View reviewed changes

DockerfileM1

ab405c3

pwalczysko suggested changes Dec 20, 2023

View reviewed changes

sbesson mentioned this pull request Dec 20, 2023

DockerfileM1 sbesson/aspera-client-docker#1

Merged

pwalczysko and others added 3 commits December 20, 2023 11:39

Modify Dockerfile cf. Seb's comment

9cc1c02

Remove DockerfileM1 as now Dockerfile works for both M1 and intel

39b8bf4

Merge pull request #1 from pwalczysko/m1

4b40389

DockerfileM1

sbesson added 2 commits January 3, 2024 09:23

Use Rocky Linux 9 as the base image

bdb8f26

Bump GitHub Actions version

455b9cc

Copy workflow from omero-server-docker

sbesson requested a review from pwalczysko January 3, 2024 09:29

pwalczysko approved these changes Jan 3, 2024

View reviewed changes

jburel approved these changes Jan 29, 2024

View reviewed changes

joshmoore approved these changes Jan 29, 2024

View reviewed changes

sbesson merged commit 3414623 into IDR:master Jan 29, 2024
2 checks passed

sbesson deleted the aspera_2023_update branch January 29, 2024 15:57

sbesson commented Jan 29, 2024

View reviewed changes

sbesson mentioned this pull request Jan 29, 2024

Suffix prefix with : for image upload #7

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update aspera-client-docker image to use the new download workflow #5

Update aspera-client-docker image to use the new download workflow #5

sbesson commented Dec 19, 2023 •

edited

Loading

joshmoore Dec 19, 2023

pwalczysko commented Dec 20, 2023 •

edited

Loading

pwalczysko commented Dec 20, 2023

pwalczysko left a comment

sbesson commented Dec 20, 2023

jburel commented Dec 26, 2023

pwalczysko commented Jan 3, 2024

joshmoore left a comment

sbesson commented Jan 29, 2024

sbesson Jan 29, 2024

Update aspera-client-docker image to use the new download workflow #5

Update aspera-client-docker image to use the new download workflow #5

Conversation

sbesson commented Dec 19, 2023 • edited Loading

joshmoore Dec 19, 2023

Choose a reason for hiding this comment

pwalczysko commented Dec 20, 2023 • edited Loading

pwalczysko commented Dec 20, 2023

pwalczysko left a comment

Choose a reason for hiding this comment

sbesson commented Dec 20, 2023

jburel commented Dec 26, 2023

pwalczysko commented Jan 3, 2024

joshmoore left a comment

Choose a reason for hiding this comment

sbesson commented Jan 29, 2024

sbesson Jan 29, 2024

Choose a reason for hiding this comment

sbesson commented Dec 19, 2023 •

edited

Loading

pwalczysko commented Dec 20, 2023 •

edited

Loading