Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a Model Moniter [FEA] #383

Open
vikashg opened this issue Oct 31, 2022 · 2 comments
Open

Create a Model Moniter [FEA] #383

vikashg opened this issue Oct 31, 2022 · 2 comments
Assignees
Labels
architectural story Work related to architecture Contribution wanted If you are looking for an issue to resolve that will help Project MONAI App SDK enhancement New feature or request integration

Comments

@vikashg
Copy link
Collaborator

vikashg commented Oct 31, 2022

Is your feature request related to a problem? Please describe.
Often we would like to know how the model is performing on the data over a period of time and maintain overall statistics of the model performance. These statistics can be further granularized to specific centers and demographics. If several models are deployed we can use this performance to track the performance of each separate model.

Describe the solution you'd like
A web-based dashboard where the model performance can be displayed. Something like Tensorboard.

Describe alternatives you've considered
None so far.

Additional context
These statistics can be further used for design model updating rules in a continuous fashion.

@vikashg vikashg added the enhancement New feature or request label Oct 31, 2022
@Leengit
Copy link
Contributor

Leengit commented Oct 31, 2022

I use torch.utils.tensorboard.SummaryWriter and TensorBoard. Would that do what you want? If not, what are you looking for?

@MMelQin
Copy link
Collaborator

MMelQin commented Nov 1, 2022

This is more than a feature request, and in fact this is an area that requires a suite of software.

  • Model themselves, specifically the one App SDK support, do not change over time
  • Sample models from Model Zoo, and any other source, need to be retrained with local data, and could be biased or specialized for certain demographics
  • How a model performs in poduction, can not be from the inference application alone. The results have to be analyzed/evaluated either with expert and/or set of corresponding ground truth data, in a way, like in a double blind read, and the AI results compared with clinicians's result
  • Is there any benefit for a realtime/rolling stats monitoring? Questionable, but
    • technically, the application and/or orchestrator needs to track the clinical data of the input images, e.g. demographics and more important symptom and lab data
    • technically, the corresponding ground truth data, i,e, a manual liver tumor seg of a given CT exam, can be extracted the application and associated with the AI results, for comparison
    • all collected data then need to go through data analytics, AI based? Big data? Who'd be the users of the dashboard, and how sophisticated and interactive will the analytics be?
  • More realistic and more feasible, is to have periodic retro study
    • Rely on the PACS or image archive/data lake
    • collect the AI results and the corresponding ground truth data
    • for clinical use, both the AI and ground truth data are most likely DICOM
    • Utilize/enhance the App SDK DICOM parsing capabilities, combined with image processing algorithms in MONAI and others, do batch evaluation

It is a area of active research, and I have had in discussion with some with regard to the production approach. Would be good that we continue the discussion here. As more and more models get into clinical (IRB'ed) use, we'd be able to access data and do some PoC work.

@MMelQin MMelQin added architectural story Work related to architecture Contribution wanted If you are looking for an issue to resolve that will help Project MONAI App SDK integration labels Nov 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
architectural story Work related to architecture Contribution wanted If you are looking for an issue to resolve that will help Project MONAI App SDK enhancement New feature or request integration
Projects
None yet
Development

No branches or pull requests

4 participants