Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a Survey of Institutional Workflows and Use Cases for MONAI Deploy #49

Open
slbryson opened this issue Jan 15, 2022 · 31 comments
Open
Assignees
Labels
documentation Improvements or additions to documentation

Comments

@slbryson
Copy link
Collaborator

MONAI Deploy working group should produce a generalized survey that existing institutional members and new members contribute to that collects information related to the MONAI Workloads for (non-exhaustive)

  • Use Cases with priority of deployment
  • Types of imaging studies modalities
  • Statistics on data characteristics
  • other key AI data characteristics

The goal of the effort is to begin building a useful profile and catalog of best practices and state of the art descriptions for MONAI Deploy related workflows where this information can be further utilized with regular reports on MONAI Deploy use case and workflows and their characteristics in the community.

@slbryson slbryson added the documentation Improvements or additions to documentation label Jan 15, 2022
@hshuaib90
Copy link
Collaborator

See draft here: https://docs.google.com/forms/d/1zHSS9exLLfivQlh9HxRUI3S8FLnznhlmShr5IIvARWo/edit

What's the best way to develop this? Happy for people to put comments here and I can make changes or can try convert to markdown (somehow?) and we can work on that until finalised and then convert back to Google Form.

@ristoh
Copy link
Collaborator

ristoh commented Jan 20, 2022

I like the survey. Do we have any members of the working group who haven't yet provided this info, who could dry run the survey and see if it fulfills the purpose?

@hshuaib90
Copy link
Collaborator

It's possible since we've had a few new members. We can ask during this Thursday's meeting.

@hshuaib90 hshuaib90 changed the title Create a Survey of Intuitional Workflows and Use Cases for MONAI Deploy Create a Survey of Institutional Workflows and Use Cases for MONAI Deploy Jan 20, 2022
@vikashg
Copy link
Contributor

vikashg commented Jan 28, 2022

Is it possible to add a question to institutions that, If you wanted to deploy AI based solutions at you institutions and you have not deployed it, what was the reason ?
Some of the options are

  1. Lack of Resources (Financial and Man Power)
  2. Lack of acceptance because of ethical issues
  3. Lack of knowhow.
  4. Not enough financial benefits from AI based deployment.
  5. Lack of trust.

@JHancox
Copy link
Contributor

JHancox commented Jan 28, 2022

Some of the questions are asking about information that is probably publicly available (e.g. Number of studies per year etc.) We may want to consider sourcing this information independently, which will also make it consistent from different folks in the same institution, who also may not know all those figures.

@laurencejackson
Copy link
Collaborator

To add to @vikashg's comment, how would people feel if the answers to those questions could be on a scale (1-5 + N/A) of how large an obstacle to deployment each factor is, then we could monitor improvements in each category in the case of this becoming an annual survey. e.g. could have evidence that in 2023 lack of trust was less of an obstacle to deployment than in 2022.

1 = very difficult obstacle
5 = not an obstacle
e.g. Lack of resources | 1 2 3 4 5 N/A

@JHancox
Copy link
Contributor

JHancox commented Jan 28, 2022

Is the Radiology focus deliberate? AI can obviously be a part of other MONAI Deploy pipelines such as pathology, endoscopy etc. I appreciate that Radiology is probably the simplest option - just don't want to miss a trick unless it's deliberate.

@JHancox
Copy link
Contributor

JHancox commented Feb 1, 2022

I would be interested to know whether the respondents anticipate that AI will be provided as predominantly as:
a) Cloud Services
b) On Premises
c) Hybrid

@hshuaib90
Copy link
Collaborator

I think it is deliberate. It would be a much bigger endeavour to try and map things like pathology , endoscopy etc. It might be worth adding to next year's audit.

I think on-premise question is a good one. Should we allow some fuzzyness in options (a) and (b) by adding something like (>90% of deployments)? Let me know and I'll add it to the questionnaire.

@hshuaib90
Copy link
Collaborator

Some of the questions are asking about information that is probably publicly available (e.g. Number of studies per year etc.) We may want to consider sourcing this information independently, which will also make it consistent from different folks in the same institution, who also may not know all those figures.

I agree we should do this where available.

@hshuaib90
Copy link
Collaborator

To add to @vikashg's comment, how would people feel if the answers to those questions could be on a scale (1-5 + N/A) of how large an obstacle to deployment each factor is, then we could monitor improvements in each category in the case of this becoming an annual survey. e.g. could have evidence that in 2023 lack of trust was less of an obstacle to deployment than in 2022.

1 = very difficult obstacle 5 = not an obstacle e.g. Lack of resources | 1 2 3 4 5 N/A

I think this is a good idea, and seeing as it has some thumbs up, i'll add it

@hshuaib90
Copy link
Collaborator

Is it possible to add a question to institutions that, If you wanted to deploy AI based solutions at you institutions and you have not deployed it, what was the reason ? Some of the options are

  1. Lack of Resources (Financial and Man Power)
  2. Lack of acceptance because of ethical issues
  3. Lack of knowhow.
  4. Not enough financial benefits from AI based deployment.
  5. Lack of trust.

Can we have some thoughts about the best set of 5 options we want to offer? @ristoh @laurencejackson @JHancox @dbericat @slbryson

I think the question should be something like :
A major challenge in the routine deployment of AI is:

  1. funding (1 | 2 | 3 |4 | 5)
  2. technical infrastructure (integration, hardware, software) (1 | 2 | 3 |4 | 5)
  3. viable products (return on investment, performance) (1 | 2 | 3 |4 | 5)
  4. local expertise (1 | 2 | 3 |4 | 5)
  5. business process change (1 | 2 | 3 |4 | 5)

@JHancox
Copy link
Contributor

JHancox commented Feb 2, 2022

Another interesting response might be: Insufficient Clinical Demand, but I think what you have @hshuaib90 is good

@laurencejackson
Copy link
Collaborator

I think what you've got here is great. The only thing I'd add is how do we capture general mistrust/resistance to AI from institutions? I think safety and liability concerns could be one of the primary obstacles to deployment, but not immediately clear which category this falls under? I think probably local expertise - in which case we could use some bracketed examples e.g.

  1. local expertise (technical, clinical and regulatory) (1 | 2 | 3 |4 | 5)

@hshuaib90
Copy link
Collaborator

I think what you've got here is great. The only thing I'd add is how do we capture general mistrust/resistance to AI from institutions? I think safety and liability concerns could be one of the primary obstacles to deployment, but not immediately clear which category this falls under? I think probably local expertise - in which case we could use some bracketed examples e.g.

  1. local expertise (technical, clinical and regulatory) (1 | 2 | 3 |4 | 5)

Thanks, Laurence. I think we don't want to allow a general answer/option because it is of limited use when interpreting the results. I think what you're describing could be either local expertise or it could be viable products (i.e. there are no products that allay my fears around AI). I'll amend option 4 with your suggestion whilst awaiting other comments

@JHancox
Copy link
Contributor

JHancox commented Feb 2, 2022

I think on-premise question is a good one. Should we allow some fuzzyness in options (a) and (b) by adding something like (>90% of deployments)? Let me know and I'll add it to the questionnaire.

Yes, I think that would be fine @hshuaib90

@slbryson
Copy link
Collaborator Author

slbryson commented Feb 2, 2022

I like the set of questions posed. Definitely keep the format of allowing the response to comment on "All" the choices with 1|2|3|... adding that 5 is strongly impacted or whatever the desired meaning of the range.

@ristoh
Copy link
Collaborator

ristoh commented Feb 2, 2022

Survey looks good!

@hshuaib90 previously you mentioned, we could incentivize different ways to contribute to this effort. Should that be included in the survey?

i.e. Would you like to contribute more in the AI survey and MONAI Deploy WG effort?

  • Yes, I could contribute as a reviewers of research [your name gets listed in the acknowledgement on the published paper]
  • Yes, I could contribute a response and my opinion and insights on how my institution uses AI [your institution gets acknowlement on the published paper]
  • Yes, I would like to join the MONAI Deploy WG as an active participant to define the future of AI deployment across institutions and industry
  • Yes, other: (suggest ways to contribute)
  • No, I'd rather focus on other things

@hshuaib90 if you think this is diluting the message, feel free to leave out.

@hshuaib90
Copy link
Collaborator

Survey looks good!

@hshuaib90 previously you mentioned, we could incentivize different ways to contribute to this effort. Should that be included in the survey?

i.e. Would you like to contribute more in the AI survey and MONAI Deploy WG effort?

  • Yes, I could contribute as a reviewers of research [your name gets listed in the acknowledgement on the published paper]
  • Yes, I could contribute a response and my opinion and insights on how my institution uses AI [your institution gets acknowlement on the published paper]
  • Yes, I would like to join the MONAI Deploy WG as an active participant to define the future of AI deployment across institutions and industry
  • Yes, other: (suggest ways to contribute)
  • No, I'd rather focus on other things

@hshuaib90 if you think this is diluting the message, feel free to leave out.

I think this is a great idea. I think it would be good to follow up with respondents with this ask as opposed to putting it in the initial survey, just to keep the survey as focussed as possible.

@msheller
Copy link

msheller commented Feb 3, 2022

This is really cool effort! Thank you for this work! If I may, I have a few questions:

  • Do you have aspirations for future surveys?
  • How are you recruiting respondents? Wondering if we could help at all, or if you've already covered the folks we might bring along (e.g. the MICCAI crowd).
  • Would you have interest in similar surveys aimed at infosec, IT, or IRBs?

@slbryson
Copy link
Collaborator Author

slbryson commented Feb 3, 2022

Ralf Floca can elaborate here and on the issue #49 that we should be careful to clarify (if not already) on the survey they we are interested in responses for research workflows as well as clinical workflows. In some regions, when mentioning clinical workflows it may limit the response to Federally approved AI workflows (like FDA algorithms in the US) vs a broader category of research applied workflows please let me know if I captured correctly.
And this would be research hospitals.

@hshuaib90
Copy link
Collaborator

This is really cool effort! Thank you for this work! If I may, I have a few questions:

  • Do you have aspirations for future surveys?
  • How are you recruiting respondents? Wondering if we could help at all, or if you've already covered the folks we might bring along (e.g. the MICCAI crowd).
  • Would you have interest in similar surveys aimed at infosec, IT, or IRBs?

Thanks @msheller ! Yes the plan is to update it annually, potentially expanding the scope.
There's a list in the appendix of the Google doc , please add to that table! Currently just went through list from International Societies of Radiology.

Yes, I think it would definitely be worthwhile exploring other dimensions around AI deployment where we can either set a standard or report an analysis of the current state.

@rfloca
Copy link

rfloca commented Feb 3, 2022

Ralf Floca can elaborate here and on the issue #49 that we should be careful to clarify (if not already) on the survey they we are interested in responses for research workflows as well as clinical workflows. In some regions, when mentioning clinical workflows it may limit the response to Federally approved AI workflows (like FDA algorithms in the US) vs a broader category of research applied workflows please let me know if I captured correctly. And this would be research hospitals.

Yes, in the end it revolves around the question which kind of AI workload you want to cover with the survey. If you also want to
get things that are done with AI, but not in prospective manner in the regulatory sense (e.g. because it's the research and not yet in a regulatory compliant product), you should stated more explicitly in the introduction.
And you should ask yourself about the granularity you want for those use cases. Currently you only have "retrospective study" in your list, which automatically could imply that everything else is prospectively. But then you could not differentiate between different types of projects in a retrospective setting (not sure if this is of interest for you though); e.g. in context of the research project you also could run "emergency diagnostic" in a "retrospective" setting (thus not impacting clinical decision) to see how it performs.

Hope that I got the point across.

@rfloca
Copy link

rfloca commented Feb 3, 2022

one more thing. I would say you're missing the use case of image-based therapy planning (e.g. for image guided surgery or radiotherapy). I would this be out of scope for the survey?

@hshuaib90
Copy link
Collaborator

one more thing. I would say you're missing the use case of image-based therapy planning (e.g. for image guided surgery or radiotherapy). I would this be out of scope for the survey?

Yes I think this would be more in the MONAI Stream WG remit. @ristoh @dbericat may be able to confirm. I think for future survey we might want to think about including that scope

@rfloca
Copy link

rfloca commented Feb 3, 2022

one more thing. I would say you're missing the use case of image-based therapy planning (e.g. for image guided surgery or radiotherapy). I would this be out of scope for the survey?

Yes I think this would be more in the MONAI Stream WG remit. @ristoh @dbericat may be able to confirm. I think for future survey we might want to think about including that scope

For ig surgery that might be the case. But I think typical RT planning use cases do not fall into that regime. Only MR guided RT or gating/tracking use cases based on fluroscopy, US or alike are about "streaming". In these are in numbers not the majority compared to "classical" planning use cases. Even things like "plan of the day" is closer to "emergency diagnostic" than to streaming.
My 2 cents.

@hshuaib90
Copy link
Collaborator

one more thing. I would say you're missing the use case of image-based therapy planning (e.g. for image guided surgery or radiotherapy). I would this be out of scope for the survey?

Yes I think this would be more in the MONAI Stream WG remit. @ristoh @dbericat may be able to confirm. I think for future survey we might want to think about including that scope

For ig surgery that might be the case. But I think typical RT planning use cases do not fall into that regime. Only MR guided RT or gating/tracking use cases based on fluroscopy, US or alike are about "streaming". In these are in numbers not the majority compared to "classical" planning use cases. Even things like "plan of the day" is closer to "emergency diagnostic" than to streaming. My 2 cents.

Sorry @rfloca of course you are correct, I would consider RT planning in scope - i guess i see that as a medical imaging workflow (even though it does not happen in Radiology). Can you think of a good general way to describe that work so we can add as an option?

@slbryson
Copy link
Collaborator Author

slbryson commented Feb 3, 2022

one more thing. I would say you're missing the use case of image-based therapy planning (e.g. for image guided surgery or radiotherapy). I would this be out of scope for the survey?

Yes I think this would be more in the MONAI Stream WG remit. @ristoh @dbericat may be able to confirm. I think for future survey we might want to think about including that scope

For ig surgery that might be the case. But I think typical RT planning use cases do not fall into that regime. Only MR guided RT or gating/tracking use cases based on fluroscopy, US or alike are about "streaming". In these are in numbers not the majority compared to "classical" planning use cases. Even things like "plan of the day" is closer to "emergency diagnostic" than to streaming. My 2 cents.

Sorry @rfloca of course you are correct, I would consider RT planning in scope - i guess i see that as a medical imaging workflow (even though it does not happen in Radiology). Can you think of a good general way to describe that work so we can add as an option?

@rfloca We can add it within the Workloads document itself and we already have an open issue to add an example "streaming" use case to the Workloads document. Do you think it fits in that category?
#37
I can help with where we want to add it into the document if you can draft and initial description.

@carolinarsm
Copy link

Just wanted to share this article about information-blocking rules:
https://www.sciencedirect.com/science/article/pii/S0363018822000123

In particular, and concerning MONAI, some related articles have suggested that in order for providers to handle the volume of requests and comply with a reduced embargo period, AI might assist in generating patient-facing reports (from an AI perspective, the target being patients probably adds an extra layer of complexity, beyond the ethical issues mentioned in the article)
This goes beyond the diagnostics stages mentioned in the survey, but maybe is a good idea to keep track of it, since providers could want to add this functionality to their clinical workflows in the future.

@dbericat
Copy link
Member

dbericat commented Feb 4, 2022

Approved from my end! Great team work everyone! :D

@rfloca
Copy link

rfloca commented Feb 4, 2022

Can you think of a good general way to describe that work so we can add as an option?

It might be a bit naive, but in the survey I would have just added the option "Image based intervention/therapie planning (e.g. for radiation therapy treatment)". Assuming that people who are using such kind of techniques know what it means and the other would not care anyways.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

No branches or pull requests

10 participants