Considerations For Using MAPs In Research

[ericspod]

As discussed recently there are a few specific details and points of discussion relating to how MAPs are used in research, specifically using them as means of distributing research networks not meant for front line use to users with a range of environments and skill levels. I would suggest that a few of the points here should be integrated into our guideline documents in some way but also just stand as a set of use cases that I would encounter as a researcher, but I imagine a lot of this is present in our current design and documentation.

Considering a researcher who has developed a MAP there are different audience categories for that software:

• Students studying a biomedical degree which may or may not include research activities
• Fellow collaborator researcher doing similar activities
• Other researcher doing similar activities
• Competition/challenge/workshop manager receiving software submissions from participants
• Reviewer participating in a peer review process

What's important to consider with these audiences is what they would want to get from the MAP, their technical capability, how they would host the MAP, and what requirements they might have the MAP must satisfy. Students probably would use MAPs just to get results using a dataset they've been given as an exercise, or part of research where the network is not the subject of study directly but just a tool. Other collaborators would have a deeper understanding of the network and would have likely contributed to its development. Other researchers not involved in developing the MAP would want to use it for testing, comparison against their own software, or as part of their own research. Competition managers would want to run the MAP as part of a testing phase using hidden data participants can't access. Reviewers would want to run the MAP to verify the claimed results in submissions are true.

For technical capability, knowledge of Docker is essential but otherwise no knowledge of deep learning or other expertise would be required, a student for example may need to run a MAP to get results for data they have been given so need only know how to run it. Other collaborators or external researchers would want to use the MAP for professional work rather than educational exercises, but their expertise would vary as well but could be reasonably expected to understand what the packaged network is and how it functions in general terms. Reviewers similarly may not be familiar with MONAI Deploy but would be gathering a deep understanding of the network itself in the process of reviewing submitted work. Many challenges and competitions require participants to submit their software as Docker or Singularity images because inference must be run on hidden test data, these will specify how the image is meant to behave so that testing can be automated. How MAPs actually function would require adjustment most likely to meet such requirements but the expectation is that the managers themselves would have significant understanding of Docker and the networks the images host.

How a MAP is hosted or run would for the most part involving running the image on the command line. Larger infrastructure like Kubernetes likely wouldn't be used and wouldn't be needed, so consideration must be given to how a user would invoke the MAP directly with or without the Deploy App SDK. Users would expect to provide input files on the command line and get results in an output folder, all these require mount points through Docker. For students especially this interface has to be quite straight forward with input and output done through common file types (eg. Dicom or Nifti for image data). Extending MAPs to include in-built modes for apply inference to batches of files or to run as a web service would provide very useful options as well.

Actually distributing a MAP would be through a Docker registry but could be easiest as a saved export file despite its size. Within an institutional network this wouldn't be a problem but a compelling alternative is running the MAP as a web service on one's own compuiter and made accessible across this network. This is much more lightweight than setting up a Kubernetes cluster or other infrastructure since it can be done using workstation level computers as described in the guidelines. It's important for scientific reproducibility to make ones code and data available whenever possible, MAPs can help with this task most of all by being straight forward to use with a minimal amount of understanding of MONAI Deploy itself.