Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test applications for validating systems that use MONAI Deploy components #63

Open
laurencejackson opened this issue Apr 13, 2022 · 4 comments
Assignees
Labels
enhancement New feature or request feature MONAI Deploy code implementation help wanted Extra attention is needed

Comments

@laurencejackson
Copy link
Collaborator

Hi all, as discussed in the MONAI deploy WG meeting 7-Apr-22 I think there would be value in creating a suite of MONAI application packages for testing the capabilities of a system that uses MONAI deploy components. These applications would serve the same purpose for MONAI deploy platforms as phantoms do for QA/acceptance testing in imaging.

To give a bit of background: at GSTT we will be deploying the AIDE platform, which is built using the MONAI informatics gateway and MONAI workflow manager components, we will also use the MONAI application runner plugin for the WFM. When AIDE is deployed at GSTT we will need to perform some kind of acceptance testing to ensure that the entire system is fit and safe to use in a clinical setting. When I refer to the system here, I mean the specific combination of software, hardware, network and other local factors that make up the local environment. The testing would aim to verify that this system as a whole can operate reliably under pressure. The testing we perform will aim to explore (non-exhaustive):

  • System loading/stress tests: max number of applications that the system (local software + hardware combination) can support. This would probably use many instances of several different test MAPs (small, medium, large resource use)
  • Performance metrics of applications when system is running at low-high loads. Metrics including things like startup time, processing time delta, etc
  • understand modes of failure

I'm not sure exactly what operations the MAPs themselves would perform, but we would need to test the CPU and GPU resources in isolation and in combination.

I'd like to start a discussion here to gauge whether the community feels these applications would be valuable as part of MONAI and whether there is anything missing from the concept. Please leave your thoughts below!

@dbericat
Copy link
Member

Potential datasets talked about today during weekly WG call:

https://datacommons.cancer.gov/data#key-datasets

@dbericat dbericat added enhancement New feature or request help wanted Extra attention is needed feature MONAI Deploy code implementation labels Apr 14, 2022
@dbericat
Copy link
Member

I love the idea. Let's take it to the next level and define those small, medium and large MAPs.

We can use the Operating environments guideline to for each of the envs type we are targetting, come up with some numbers.

What do others think? @CPBridge @GreySeaWolf @aylward @ericspod @evberrypi @Danulf @nakfour

@dbericat
Copy link
Member

@JoeBatt1989 please add thoughts to this from your performance plan and add it to a roadmap we can track and act upon.

@JoeBatt1989
Copy link
Contributor

@dbericat thanks David, I have added a new ticket in the MWM repo for refinement Project-MONAI/monai-deploy-workflow-manager#118. The sizes in this table are indicative of what we plan to create, although the system specifications and run times should be configurable so we can update as we see fit.

Name CPU Cores RAM GPU Disk Space
Dummy (mean pixel calc)        
Small 2 1GB 1GB 2GB
Medium 8 10GB 6GB 15GB
Large 12 16GB 12GB 25GB

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request feature MONAI Deploy code implementation help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

6 participants