Label_image in view function is not working.

[ManooshSamiei]

Hello,

I think the label_image is not able to show the segmentation in view function in itk 3.5.0. Is there a working example of view with label_image that I can run in colab? I am trying to show a tumor segmentation in a 3d mri image but have not been successful. I appreciate your assistance

[aylward]

@thewtex / @bnmajor - any ideas?

[bnmajor]

I can't think of a reason that changing the ITK version would affect just the label image...

@ManooshSamiei I will check this out. Couple of quick questions:

• Looks like you were testing with the Model Zoo notebook, is that correct?
• Am I understanding you correctly that you are seeing the image, just not the label image?
• Are you using version 1.0a23 that is pinned in the notebook or another version of ITKWidgets?

[bnmajor]

@ManooshSamiei I just ran the Model Zoo on colab after bumping the itkwidgets version to the latest v1.0a48 and everything looks correct to me!

@thewtex @aylward Is there a reason that we pinned to 1.0a23 other than that being the latest version at the time? I cannot remember a reason but I wanted to double-check before I open a PR to bump the version in the TCIA Notebooks. It should fix the bug, not to mention we've added a lot of improvements over the versions that may be useful to users.

[ManooshSamiei]

@ManooshSamiei I just ran the Model Zoo on colab after bumping the itkwidgets version to the latest v1.0a48 and everything looks correct to me!

@thewtex @aylward Is there a reason that we pinned to 1.0a23 other than that being the latest version at the time? I cannot remember a reason but I wanted to double-check before I open a PR to bump the version in the TCIA Notebooks. It should fix the bug, not to mention we've added a lot of improvements over the versions that may be useful to users.

Thank you the Monai notebook shows the tumor with this version upgrade. However, I still have difficulty with showing tumor in the ReMIND collection. Is the tumor origin in the dicom file misaligned? Do you suggest any way that I can show the glioma tumor? Here is a link to my notebook: https://colab.research.google.com/drive/1lxy4VZUt7w8Veko8DuzZw1eaagyBym2n?usp=sharing

[ManooshSamiei]

I can't think of a reason that changing the ITK version would affect just the label image...

@ManooshSamiei I will check this out. Couple of quick questions:

• Looks like you were testing with the Model Zoo notebook, is that correct?
• Am I understanding you correctly that you are seeing the image, just not the label image?
• Are you using version 1.0a23 that is pinned in the notebook or another version of ITKWidgets?

I actually tried with 1.0a30 but it still did not work 1.0a48 fixed it!

[bnmajor]

Thank you the Monai notebook shows the tumor with this version upgrade. However, I still have difficulty with showing tumor in the ReMIND collection. Is the tumor origin in the dicom file misaligned? Do you suggest any way that I can show the glioma tumor? Here is a link to my notebook: https://colab.research.google.com/drive/1lxy4VZUt7w8Veko8DuzZw1eaagyBym2n?usp=sharing

Thanks for the example notebook! It does seem to be misaligned:

Note the bounding box buttons which can be very helpful in tracking down these kinds of issues! :) (And another good reason to get example notebooks on the latest version!)

[kirbyju]

@fedorov Checking to see if you all already knew about this potential alignment issue with the ReMIND segmentation data and whether we may need to investigate a fix.

[fedorov]

I would not necessarily blame the data. It shows fine in 3D Slicer, which is using dcmqi to convert DICOM SEG into ITK representation.

Note that the segmentations in this specific collection are formatted as orthogonal to the orientation of slices in MR.

CC: @pieper.

[pieper]

It's not uncommon for multimodality datasets to define segmentations in different sampling grids from any particular image. It would be great for itkwidgets to handle this situation or at least to detect it and flag that it's not supported.

[fedorov]

For the full disclosure, those SEGs were generated with dcmqi, and the result was tested in Slicer by the submitters, so it is not surprising that it is working. I see from your notebook that you are using pydicom-seg. I do not have experience with that package, and I do not know how robust it is. It is also unfortunate that there very few tools supporting DICOM SEG, so it is difficult to test this. OHIF has its own limitations, and we can't even try to test if this works there due to another deficiency of how those SEGs were encoded (see https://projectweek.na-mic.org/PW40_2024_GranCanaria/Projects/LinkingSegmentationAndImagingDataWithDifferentGeometriesUsingDcmqi/ for details if interested).

BUT we do have VolView, where I believe @floryst implemented DICOM SEG support independently. It does not work there as expected, but at least the alignment makes sense. The jageddness of the contours is a typical artifact when the implementation does not account for the byte boundary in frame encoding, which is very often not done correctly (see exceedingly long discussion on this topic with the references here: QIICR/dcmqi#341).

Link to open ReMIND-001 case form IDC in VolView: https://volview.kitware.app/?urls=[s3://idc-open-data/f6e255ba-3024-42c6-b303-f87215c8cf33,s3://idc-open-data/73da2952-15b7-4bd2-8af4-797d5e197fc1,s3://idc-open-data/4471944a-6aa8-425a-8a9e-0f0cec694b3f,s3://idc-open-data/b298b0f9-e419-4058-8170-24195ade5077,s3://idc-open-data/a41915ef-d0ae-4c55-8ce1-d93182c30382,s3://idc-open-data/65c9d832-2e88-42b1-a5a7-2e07a7f2ea36]

As a very quick fix to the underlying issue @kirbyju you could just switch your notebook to using dcmqi segimage2itkimage. It is a command line tool, but (to the best of my knowledge) it works. Absolute majority of TCIA DICOM SEG were created using dcmqi.

[fedorov]

The jageddness of the contours is a typical artifact when the implementation does not account for the byte boundary in frame encoding

Sorry, I forgot to clarify this for the specific dataset at hand. Such implementation issue will only lead to problems when the number of pixels in the segmentation frame is not divisible by 8, which is not very common. But it is exactly the case in the current situation, since those segmentations are cropped: frame size is 61x63.

[thewtex]

It's not uncommon for multimodality datasets to define segmentations in different sampling grids from any particular image. It would be great for itkwidgets to handle this situation or at least to detect it and flag that it's not supported.

Different sampling grids between the intensity and label image are already supported. This is handled via ITK-Wasm during render.

It looks like we are encountering a spatial metadata issue here.

As a very quick fix to the underlying issue @kirbyju you could just switch your notebook to using dcmqi segimage2itkimage. It is a command line tool, but (to the best of my knowledge) it works.

Kudos to @fedorov, who is helping to provide resources so this will be available in python packages over the coming months 🌟

[pieper]

Different sampling grids between the intensity and label image are already supported. This is handled via ITK-Wasm during render.

Great! Then maybe the issue is with the package being used to read the dicom seg files. It would be good to see that fixed upstream.

[ManooshSamiei]

For the full disclosure, those SEGs were generated with dcmqi, and the result was tested in Slicer by the submitters, so it is not surprising that it is working. I see from your notebook that you are using pydicom-seg. I do not have experience with that package, and I do not know how robust it is. It is also unfortunate that there very few tools supporting DICOM SEG, so it is difficult to test this. OHIF has its own limitations, and we can't even try to test if this works there due to another deficiency of how those SEGs were encoded (see https://projectweek.na-mic.org/PW40_2024_GranCanaria/Projects/LinkingSegmentationAndImagingDataWithDifferentGeometriesUsingDcmqi/ for details if interested).

BUT we do have VolView, where I believe @floryst implemented DICOM SEG support independently. It does not work there as expected, but at least the alignment makes sense. The jageddness of the contours is a typical artifact when the implementation does not account for the byte boundary in frame encoding, which is very often not done correctly (see exceedingly long discussion on this topic with the references here: QIICR/dcmqi#341).

Link to open ReMIND-001 case form IDC in VolView: https://volview.kitware.app/?urls=[s3://idc-open-data/f6e255ba-3024-42c6-b303-f87215c8cf33,s3://idc-open-data/73da2952-15b7-4bd2-8af4-797d5e197fc1,s3://idc-open-data/4471944a-6aa8-425a-8a9e-0f0cec694b3f,s3://idc-open-data/b298b0f9-e419-4058-8170-24195ade5077,s3://idc-open-data/a41915ef-d0ae-4c55-8ce1-d93182c30382,s3://idc-open-data/65c9d832-2e88-42b1-a5a7-2e07a7f2ea36]

As a very quick fix to the underlying issue @kirbyju you could just switch your notebook to using dcmqi segimage2itkimage. It is a command line tool, but (to the best of my knowledge) it works. Absolute majority of TCIA DICOM SEG were created using dcmqi.

Thank you for your suggestion. But I modified my notebook to use dcmqi segimage2itkimage, and the tumor is still misaligned with the same origin and location as before. I am reading the segmentation with dcmqi, and i read mri dicoms using itk. Here is a link to my updated notebook: https://colab.research.google.com/drive/1lxy4VZUt7w8Veko8DuzZw1eaagyBym2n?usp=sharing
I appreciate your help.

[fedorov]

I ran your notebook, and I did not see tumor segmentation at all in itkwidgets, and could not figure out how to toggle it. I think it would be best if someone knowledgeable about itkwidgets looked into this.

[kirbyju]

Hi all, just wanted to check back to see if there was any resolution to this. Do we need to update any of the itkwidget notebooks in https://github.com/kirbyju/TCIA_Notebooks?

[bnmajor]

@kirbyju I've opened a PR to bump itkwidgets in the TCIA Notebooks repo because that was certainly a piece of the problem.

As to the apparent spatial metadata issue I am not sure if @ManooshSamiei has tried again using dcmqi (as suggested by @fedorov and @thewtex) and has had success with segimage2itkimage or if there is further investigation needed there.

[thewtex]

@bnmajor thank you!

@kirbyju @ManooshSamiei We are working on making dcmqi more accessible in Python -- I will follow up once we can use it here; looking forward to helping with your use case.

[ManooshSamiei]

@kirbyju I've opened a PR to bump itkwidgets in the TCIA Notebooks repo because that was certainly a piece of the problem.

As to the apparent spatial metadata issue I am not sure if @ManooshSamiei has tried again using dcmqi (as suggested by @fedorov and @thewtex) and has had success with segimage2itkimage or if there is further investigation needed there.

Hello,

Thanks for your follow up. Yes, I did try with dcmqi but had no luck.. finally I switched to VTK and was able to show tumor and the brain segmentations overlaid.

[ManooshSamiei]

@bnmajor thank you!

@kirbyju @ManooshSamiei We are working on making dcmqi more accessible in Python -- I will follow up once we can use it here; looking forward to helping with your use case.

Thank you! Looking forward to it

[fedorov]

Yes, I did try with dcmqi but had no luck..

@ManooshSamiei could you elaborate on what you did to try it, and what exactly did not work?

[kirbyju]

Hi all, I am a bit lost here and would like to confirm nobody is waiting on me for something. In response to the @fedorov Apr 1 comment, it seems like @ManooshSamiei modified https://colab.research.google.com/drive/1lxy4VZUt7w8Veko8DuzZw1eaagyBym2n?usp=sharing to use segimage2itkimage but it didn't work.

I also see you mentioned this issue in InsightSoftwareConsortium/ITK-Wasm#1205. I'm assuming there is still nothing you need me to do beyond wait for an itkWidget fix. Please let me know if that's not the case.

[thewtex]

@kirbyju thanks for the ping. Yes, there are metadata issues that are being resolved.