false merged segmentation mask

[UnholyCarrot]

Hi there,
My YOLOv8-n model for segmentation trained on a custom dataset where there is only one class. Eventhough it was trained on a large data set, it still produces false possitives which are merged masks for two objects. I want to consider them as negative samples and train them to model but cannot find a solution except creating a "non-class" and label them so. My question is that how can i solve such problem?

[UltralyticsAssistant]

👋 Hello @UnholyCarrot, thank you for reaching out and sharing your question with the Ultralytics community 🚀!

We highly recommend starting with the Docs to explore key resources, including Python and CLI examples. These resources include insights that might help refine your segmentation task setup. If you're confronting a persistent problem, the docs are often a great place to find potential solutions!

If this is a 🐛 Bug Report, please help us debug it faster by providing a minimum reproducible example, including code, dataset sample examples, and outputs.

If this is a custom training ❓ Question, it is helpful to share additional information such as what preprocessing steps you’ve applied, specifics about your dataset, or any logs of the training process. Additionally, you might find insights in our Tips for Best Training Results, which cover best practices for improving YOLOv8 performance. Since you are dealing with merged masks and considering negative samples, exploring data augmentation strategies might also help enhance your dataset.

You can also join the Ultralytics community to collaborate with others. Head to Discord 🎧 for live discussions, visit Discourse for in-depth Q&A, or connect with peers on Reddit for additional community support.

Upgrade

Ensure your environment is up-to-date by upgrading to the latest ultralytics release along with all requirements. Simply run the following command in a clean Python>=3.8 environment with PyTorch>=1.8:

pip install -U ultralytics

Environments

For a smooth experience, you might consider running YOLO in any of the following verified environments equipped with all dependencies like CUDA, Python, and PyTorch preinstalled:

• Notebooks with free GPU access:
• Google Cloud Deep Learning VM - see GCP Quickstart Guide
• Amazon Deep Learning AMI - see AWS Quickstart Guide
• Docker Image - see Docker Quickstart Guide

Status

When the CI badge is green, it confirms that all Ultralytics CI tests are passing successfully across macOS, Windows, and Ubuntu. These tests include all YOLO Modes and Tasks.

This is an automated response 🤖, but don’t worry—an Ultralytics engineer will review your issue and provide further assistance shortly!

[glenn-jocher]

@UnholyCarrot thank you for your question. False positives like merged masks can occur due to overlapping objects or insufficient training diversity. While creating a "non-class" is one approach, you could also improve the dataset by including more examples of challenging or overlapping scenarios. Additionally, fine-tuning the model's hyperparameters or using post-processing techniques like confidence threshold adjustments may help reduce false positives. If the issue persists, consider experimenting with a larger model size or data augmentation strategies to enhance model learning.

[Y-T-G]

You can add background images to the dataset by just not providing any labels for those images. They're treated as negative samples.