add class mapping for multi class segmentation

[chaymaaBOUSNAH]

No description provided.

[Mattwmaster58]

This doesn't seem to be working for me, at least using 3 channel pngs as masks

[chaymaaBOUSNAH]

This doesn't seem to be working for me, at least using 3 channel pngs as masks

how many classes do you have? you must ensure that you have mapped all classes, can you share the mapping dict you have set?

[Mattwmaster58]

Upon closer investigation, the issue appears to be related to my mask image format as its in palette mode already. As I understand it, no further conversion should be necessary then (ie, no need to use mask_to_class). Although even after converting with convert("RGB"), other things break that I'm not sure are related. Here's one mask, I'm not sure that GH won't re-encode.

Here's the code I added in __getitem__:

        mask_mode = mask.mode

        assert img.size == mask.size, \
            f'Image and mask {name} should be the same size, but are {img.size} and {mask.size}'

        img = self.preprocess(img, self.scale, is_mask=False)
        mask = self.preprocess(mask, self.scale, is_mask=True)

        if mask_mode != "P":  # if pallete mode, no need to convert a RGB color because its already correct at this point
            # mapping the class colors
            mask = self.mask_to_class(mask, self.mapping)

The error I get when trying to convert to RGB first, then using mask_to_class:

Traceback (most recent call last):
  File "I unet-multi-seg/train.py", line 213, in <module>
    train_net(net=net,
  File "unet-multi-seg/train.py", line 91, in train_net
    for batch in train_loader:
  File ".venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File ".venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 671, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File ".venv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File ".venv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "unet-multi-seg/utils/data_loading.py", line 90, in __getitem__
    mask = self.mask_to_class(mask, self.mapping)
  File "unet-multi-seg/utils/data_loading.py", line 67, in mask_to_class
    validx = (idx.sum(0) == 3)
AttributeError: 'bool' object has no attribute 'sum'
python-BaseException

I don't think PIL is handling the conversion to RGB correctly, the mask np.array has absurdly large values in it

[chaymaaBOUSNAH]

I have removed /255 in preprocessing, I think it must be the cause of your issue, can you retry now?
if it doesn't work check your mask shape and values after conversion to RGB !

[sevimcengiz]

how do we need to arrange for mapping values if we have more than 3 classess?

[chaymaaBOUSNAH]

how do we need to arrange for mapping values if we have more than 3 classess?

You must map each class color code to an index, for example: if you have 5 classes with the following colors: Black, blue, red, aqua, and magenta. the mapping dict must be: mapping = {(0, 0, 0): 0, (0, 0, 255): 1, (255, 0, 0): 2, (0, 255, 255): 3, (255, 0, 255): 4}

[sevimcengiz]

Thank u so much for the quick reply. Mapping values worked for me for my custom dataset.

I have another question when I load any mask from Carvana datasets. It shows 1280x1918 uint8. It is not 1280x1918x3. The intensity value of the car is 1 and the background is 0. In multi-class segmentation, the mask should has w,h,c. for ex: 1280x1918x3 uint8, and a blue label which represents car should have intensity value 0 in r, 0, g, 255 in blue channel. or for magenta channel red 255, green 0 and blue 255.

Question2: why does data_loading.py only take the transpose of the images instead of both mask and images? Because code says "if it is not mask, it takes transpose ((2,0,1))"?,

it gives me an error. Error:

File "train.py", line 200, in
amp=args.amp)
File "train.py", line 98, in train_net
loss = criterion(masks_pred, true_masks)
File "/home/sevim/anaconda3/envs/pytorch37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/sevim/anaconda3/envs/pytorch37/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 1165, in forward
label_smoothing=self.label_smoothing)
File "/home/sevim/anaconda3/envs/pytorch37/lib/python3.7/site-packages/torch/nn/functional.py", line 2996, in cross_entropy
return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
RuntimeError: input and target batch or spatial sizes don't match: target [1, 959, 3], input [1, 5, 640, 959]
"

Question 3: why did you remove /255? what will happen to the normalization?

sorry for the long comment. I'm trying to understand.

[chaymaaBOUSNAH]

I didn't test the Carnava dataset, but if you are talking about the mask shape after mapping, it must be (h, w) and there will be no channel dim.
we do the transpose to have the shape (channels, height, width) instead of (height, width, channels) because most Pytorch functions must have this input format, like the Cross-Entropy loss ...
I removed /255 because mapping do already the job (after mapping masks contain only indexes of each class color).
I hope my answer is clear!

[onway666]

how do we need to arrange for mapping values if we have more than 3 classess?

how do we need to arrange for mapping values if we have more than 3 classess?

You must map each class color code to an index, for example: if you have 5 classes with the following colors: Black, blue, red, aqua, and magenta. the mapping dict must be: mapping = {(0, 0, 0): 0, (0, 0, 255): 1, (255, 0, 0): 2, (0, 255, 255): 3, (255, 0, 255): 4}

Doesn't this operation create some kind of association between different classes? What if my classes have no relationship with each other?

[onway666]

What should I do if my original mask is a four-channel image where each channel is a binary image?

[BenjaminSchaar]

I am not 100 sure how to use more than 2 classes:
Should I do more than changing the parameter classes ?