Add ARM builds

[realies]

Currently, ElasticMQ builds and works fine when using a Dockerfile on an ARM64 systems. Would it be possible for SoftwareMill to add the architecture to their Docker Hub releases?

[adamw]

Sure, though I would have to think how to do it exactly :) If you'd have some other projects which implement this, please provide a link, could be easier by getting inspired by them :)

[avaz]

I'm trying to run this project on a M1 Mac (ARM64) and I'm unable to connect to the queue. I received this error from AWS Java SDK:

Caused by: org.apache.http.NoHttpResponseException: The target server failed to respond

In the docker logs I see only this and nothing else:
23:09:06.084 [main] INFO org.elasticmq.server.Main$ - Starting ElasticMQ server (1.1.0) ...

Can I assume that this is because the lack of support for ARM64 on this project? If not any clue of what can be wrong?

[adamw]

@avaz yes, that's probably the cause. If you hava java though, you can try running it in embedded mode, by adding the elasticmq dependency and starting the server in-tests.

[olegakbarov]

@adamw could you please get into more details how to do it?
Also would appreciate ARM builds!

[adamw]

@olegakbarov You should add the following dependency in the test scope: https://mvnrepository.com/artifact/org.elasticmq/elasticmq-server_2.13/1.1.0

Then you should be able to start the server:

val config = ConfigFactory.load("elasticmq.conf")
val server = new ElasticMQServer(new ElasticMQServerConfig(config))
server.start()

As described here: https://github.com/softwaremill/elasticmq#starting-an-embedded-elasticmq-server-with-an-sqs-interface

Regarding ARM builds - PRs welcome :)

[avaz]

@adamw thanks for the reply.

Just to let you and everyone know what worked for me. I switched to the java native docker image (softwaremill/elasticmq-native) and things worked. It boots properly and I'm being able to send and received messages.

Also recently I had stumbled on issue 123 and I had to the /queue to the connection url, now using M1 however with the native image I had to remove the /queue/ to actually make things works.

[adamw]

@avaz good to hear this worked :) can you add a comment in #123 specifying what path you used before, and what path you are using now?

[realies]

@avaz, I can see softwaremill/elasticmq-native only lists linux/amd64 for available architectures. Does that mean that Docker emulates linux/amd64 on the M1 or you build softwaremill/elasticmq-native with a Docker file on ARM64?

[realies]

@adamw, would elasticmq-native get an arm image using the same codebase?

[adamw]

@realies I believe @rojekp is working on it

[adamw]

I'll reopen until this is done :)

[olegakbarov]

I followed these steps, but the container hangs without logs in docker

1. git clone [email protected]:softwaremill/elasticmq.git
2. navigated to server/docker
3. created a Dockerfile there:

FROM openjdk:8-jre-alpine
ARG ELASTICMQ_VERSION
ENV ELASTICMQ_VERSION ${ELASTICMQ_VERSION:-1.1.0}
RUN apk add --no-cache curl ca-certificates
RUN mkdir -p /opt/elasticmq/log /opt/elasticmq/lib /opt/elasticmq/config
RUN curl -sfLo /opt/elasticmq/lib/elasticmq.jar https://s3-eu-west-1.amazonaws.com/softwaremill-public/elasticmq-server-${ELASTICMQ_VERSION}.jar
COPY ${PWD}/elasticmq.conf /opt/elasticmq/config/elasticmq.conf
WORKDIR /opt/elasticmq
EXPOSE 9324
ENTRYPOINT [ "/usr/bin/java", "-Dconfig.file=/opt/elasticmq/conf/elasticmq.conf", "-jar", "/opt/elasticmq/lib/elasticmq.jar" ]

4. built it with docker buildx build --platform=linux/amd64 --load -t softwaremill/elasticmq .

[rojekp]

@olegakbarov, after following your steps I run that image with command
docker % docker run -p 9324:9324 -p 9325:9325 softwaremill/elasticmq
And I have following output:

[main] INFO  org.elasticmq.server.Main$ - Starting ElasticMQ server (1.1.0) ...
[main] ERROR org.elasticmq.server.Main$ - Uncaught exception in thread: main
com.typesafe.config.ConfigException$IO: /opt/elasticmq/conf/elasticmq.conf: java.io.FileNotFoundException: /opt/elasticmq/conf/elasticmq.conf (No such file or directory)

There's probably something wrong with Dockerfile. I tried to build and run image generated from Dockerfile generated by set docker:stage. Please check it - https://github.com/softwaremill/elasticmq#building-multi-architecture-image and let us know if that works.

[realies]

Thanks for the great work @rojekp! Should we expect the ARM image for elasticmq-native to appear in Docker Hub @adamw?

[rojekp]

After merging above PR and releasing new version - the ARM image for elasticmq-native will be available in Docker Hub.

[rojekp]

You can check out Docker Hub - the ARM image for elasticmq-native is available.

[elruwen]

I gave it a try but it doesn't work for me:

docker run -it softwaremill/elasticmq-native
[FATAL tini (7)] exec /opt/docker/bin/elasticmq-native-server failed: Exec format error

docker inspect softwaremill/elasticmq-native
[
    {
        "Id": "sha256:76aade4ab12c713d5901580a2d38a7edf75d316f5482b4a6cd7b771d8b94bb19",
        "RepoTags": [
            "softwaremill/elasticmq-native:latest"
        ],
        "RepoDigests": [
            "softwaremill/elasticmq-native@sha256:605b5af1e89c34968196e020a0ba93b72634b798876df226ba9f0052ea94a7fd"
        ],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2021-06-01T09:50:50.268786721Z",
        "Container": "",
        "ContainerConfig": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": null,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
        "DockerVersion": "",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "1001:0",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "9324/tcp": {},
                "9325/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": null,
            "ArgsEscaped": true,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "/opt/docker",
            "Entrypoint": [
                "/sbin/tini",
                "--",
                "/opt/docker/bin/elasticmq-native-server",
                "-Dconfig.file=/opt/elasticmq.conf"
            ],
            "OnBuild": null,
            "Labels": null
        },
        "Architecture": "arm64",
        "Os": "linux",
        "Size": 63180458,
        "VirtualSize": 63180458,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/2000.1001/overlay2/62a2f59c3165378c435b24f916101823f043d30a12b40f87cdd948481324e71a/diff:/var/lib/docker/2000.1001/overlay2/ce6d693acfec43d860a586db6430dbc1fb6faf29b1e744f0b88b0e447cadb42c/diff:/var/lib/docker/2000.1001/overlay2/f9444129e6c51648b16c3ad1a566c64c24f1b697fb53d056d1fbda48a8aa3d74/diff:/var/lib/docker/2000.1001/overlay2/05a13566c379bea74a55147aa7dbe60eba6991af8e07ae228a46f99483294c69/diff:/var/lib/docker/2000.1001/overlay2/939f0bcb477ac2d6b0b30c0f8eb2746e76a38051e8aae0ee1459c623be46a936/diff",
                "MergedDir": "/var/lib/docker/2000.1001/overlay2/747cb2e7471832068376d6c4485eb94a183d3f5de52a58fb4780e237efa96104/merged",
                "UpperDir": "/var/lib/docker/2000.1001/overlay2/747cb2e7471832068376d6c4485eb94a183d3f5de52a58fb4780e237efa96104/diff",
                "WorkDir": "/var/lib/docker/2000.1001/overlay2/747cb2e7471832068376d6c4485eb94a183d3f5de52a58fb4780e237efa96104/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:f645bd8028b1c79524b4e3485daa20ca8f3d2f105bd7c0e4da70230e487c7fde",
                "sha256:239a4597705a6c8b316f2582e9190b5b6451270ad979bfc39fa61fdd00cba6ec",
                "sha256:b048d66263a8713a7b5a1ad0323b5e574566996436eb0e3eb48b2655dd437806",
                "sha256:975a1533a55a0a536c0a66af481d5960ad72f665a11f31bbd0a7791125ec13af",
                "sha256:b32330e6d30de5c0f41123a36dcb4d89b7ebc7330d3bf74d8173e60436463b06",
                "sha256:aa6e60edfbdb9f9210aeeb5294ca0c9b893ba476387810494ef669655acdec5d"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

I can run the parent image, so it looks like something is wrong in the build process:

docker run ghcr.io/graalvm/graalvm-ce:java11-21.1.0
Unable to find image 'ghcr.io/graalvm/graalvm-ce:java11-21.1.0' locally
java11-21.1.0: Pulling from graalvm/graalvm-ce
4b26d50a9215: Pull complete 
51cceed84e73: Pull complete 
4c9d5902abc6: Pull complete 
a4a7a9efc36d: Pull complete 
194143661dfb: Pull complete 
Digest: sha256:04bd41711712997222a9d16c9e5feeaf27c6550e6e6bd084fd7b59519d1bcdf6
Status: Downloaded newer image for ghcr.io/graalvm/graalvm-ce:java11-21.1.0
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment GraalVM CE 21.1.0 (build 11.0.11+8-jvmci-21.1-b05)
OpenJDK 64-Bit Server VM GraalVM CE 21.1.0 (build 11.0.11+8-jvmci-21.1-b05, mixed mode)

The non-native version works. And the native version works on amd64.

[ghost]

@elruwen Could you run docker run --privileged --rm tonistiigi/binfmt --install amd64, before trying to run elasticmq docker image and let us know if it is working.
It seems to be a problem with compilation of GraalVM native image. Which currently is being done on machine with AMD arch so even while docker image arch seems to be ARM compliant the native image itself is not.

[elruwen]

$ docker run -it softwaremill/elasticmq-native
Unable to find image 'softwaremill/elasticmq-native:latest' locally
latest: Pulling from softwaremill/elasticmq-native
535256e01bcb: Pull complete 
bf68952f9927: Pull complete 
2c746076e194: Pull complete 
b32bcd4023f4: Pull complete 
5a9c73f6a36d: Pull complete 
2963c4b32118: Pull complete 
Digest: sha256:2a585a33fde4e89be2ea50fece640369908328134eeab7645c93029436d06a70
Status: Downloaded newer image for softwaremill/elasticmq-native:latest
[FATAL tini (8)] exec /opt/docker/bin/elasticmq-native-server failed: Exec format error

$ docker run --privileged --rm tonistiigi/binfmt --install amd64
docker: Error response from daemon: privileged mode is incompatible with user namespaces.  You must run the container in the host namespace when running privileged mode.
See 'docker run --help'.

$ docker run --userns=host --privileged --rm tonistiigi/binfmt --install amd64
Unable to find image 'tonistiigi/binfmt:latest' locally
latest: Pulling from tonistiigi/binfmt
d707a541bd5e: Pull complete 
e9ec2a575abb: Pull complete 
Digest: sha256:ce4d5a2a6ac4a189047fca2d71cbd901cc7beebacf538be95fccb3aca87cb2ec
Status: Downloaded newer image for tonistiigi/binfmt:latest
installing: amd64 OK
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "qemu-x86_64"
  ]
}

$ docker run -it softwaremill/elasticmq-native
06:07:08.762 [main] INFO  org.elasticmq.server.Main$ - Starting ElasticMQ server (1.2.0) ...
06:07:09.919 [elasticmq-akka.actor.default-dispatcher-5] INFO  akka.event.slf4j.Slf4jLogger - Slf4jLogger started
06:07:10.697 [elasticmq-akka.actor.default-dispatcher-5] INFO  o.e.rest.sqs.TheSQSRestServerBuilder - Started SQS rest server, bind address 0.0.0.0:9324, visible server address http://localhost:9324
06:07:11.010 [elasticmq-akka.actor.default-dispatcher-5] INFO  o.e.r.s.TheStatisticsRestServerBuilder - Started statistics rest server, bind address 0.0.0.0:9325
06:07:11.035 [main] INFO  org.elasticmq.server.Main$ - === ElasticMQ server (1.2.0) started in 2289 ms ===

I saw that you run it via sbt, I am not familiar with building docker images like that. It looks like you are using somewhere build instead of buildx, because I can run the parent image on my ARM machine.

[ghost]

You are right we are using sbt-native-packager to build Graal native image. Unfortunately native packager pull docker image for host arch that's why we end up with ARM compatible docker image and AMD compatible Graal native image. I have opened issue sbt/sbt-native-packager#1443 to get some more information regarding this topic.

[realies]

@adamw, it seems like there's darwin aarch64 support over at https://github.com/graalvm/graalvm-ce-dev-builds/releases now, could m1 arm native elasticmq-native images be a thing with it?