Enable multi-arch docker builds

[dcarbone]

Hi there, me again :)

Another thing that would help adoption is to remove the hard-coded requirement of amd64 (and linux), when building the binary either on-host or in-container.

As a quick example, I removed the GOOS and GOARCH vars from the dockerfile, then subsequently built and pushed this image to my public dockerhub here: https://hub.docker.com/r/dancarbone/danielchalef-mrfparse/tags

You can see that there are two images present for this tag, one arm64, one amd64. This was done using the modified Dockerfile in this PR and the following command:

docker buildx build --platform linux/arm64,linux/amd64 -t dancarbone/danielchalef-mrfparse --push .

Proof of function:

$ docker run -it --entrypoint bash --platform linux/arm64 dancarbone/danielchalef-mrfparse -c 'uname -m'
aarch64
$
$ docker run -it --platform linux/arm64 dancarbone/danielchalef-mrfparse help  
MRFParse is a Go parser for Transparency in Coverage Machine Readable Format (MRF) files. 
        
The parser is designed to be memory and CPU efficient, and easily containerized. It will run on any modern cloud container platform (and potentially cloud function infrastructure).

Input and Output paths can be local filesytem paths or AWS S3 and Google Cloud Storage paths.

Usage:
  mrfparse [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  parse       Parse in-network MRF files. Expects split NDJSON files as input.
  pipeline    Parse in-network MRF files. Input is a single MRF JSON file. Output is a parquet fileset.
  split       Split JSON files.

Flags:
      --config string       config file (default config.yaml)
      --cpuprofile string   Write CPU profile to this file
  -h, --help                help for mrfparse
      --memprofile string   Write memory profile to this file

Use "mrfparse [command] --help" for more information about a command. 

Enabling multi-arch builds would let the ARM users on my team run this locally without the need to build it themselves, let alone have golang installed :)

[danielchalef]

Thanks for the suggestion. I'd love to add ARM64 support. Unfortunately I've made extensive use of simdjson which does not yet support ARM64 NEON intrinsics. Hence the requirement for a CPU with AVX2 and CLMUL instructions.

[dcarbone]

Hmm, so I'm not super familiar with the simdjson project, but according to their homepage ARM NEON is supported.

I will hopefully have time this weekend to dig in a smidge, see if I can test this properly :)

[danielchalef]

The reference C implementation does indeed support ARM64 NEON. Unfortunately, the Go port does not: https://github.com/minio/simdjson-go There’s an open issue regarding this: minio/simdjson-go#51

…

On Fri, Jan 13 2023 at 8:47 PM, Daniel Carbone < ***@***.*** > wrote: Hmm, so I'm not super familiar with the simdjson project, but according to their homepage ( https://simdjson.org/about/ ) ARM NEON is supported. I will hopefully have time this weekend to dig in a smidge, see if I can test this properly :) — Reply to this email directly, view it on GitHub ( #2 (comment) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AABAAZ53KUZ54JS6HQH2UCLWSIVXXANCNFSM6AAAAAAT25FI4E ). You are receiving this because you commented. Message ID: <danielchalef/mrfparse/pull/2/c1382660758 @ github. com>

[dcarbone]

Ah, I see! They're using an assembly generator. Unfortunately looks like the generator itself has been archived :(

Hmm...that's unfortunate. Would you be willing to accept a PR that allows for the use of the admittedly slower internal go json handling? I haven't had a look to check the feasibility of this, but it would be a stop-gap until a more efficient solution could be found.

[danielchalef]

@dcarbone How about we implement this for development / test purposes? https://github.com/kiwicom/fakesimdjson

Happy to accept a PR.

[dcarbone]

Just as an update, i've not forgotten about this PR. I added ParseND functionality to the upstream fakesimdjson lib, and will be working on the actual pr for this repo soon :)

Relevant PR here: kiwicom/fakesimdjson#2

[danielchalef]

Just as an update, i've not forgotten about this PR. I added ParseND functionality to the upstream fakesimdjson lib, and will be working on the actual pr for this repo soon :)

Relevant PR here: kiwicom/fakesimdjson#2

Great to hear!

[dcarbone]

closing in favor of #10