zh/tasks/segment/

[giscus[bot]]

zh/tasks/segment/

了解如何通过Ultralytics YOLO 使用实例分割模型。有关训练、验证、图像预测和模型导出的说明。

https://docs.ultralytics.com/zh/tasks/segment/

[gaotaogaotao]

Hello, may I ask how to count and print segmented objects of different categories?

[pderrenger]

@gaotaogaotao hello! To count and print segmented objects of different categories using a YOLOv8 segmentation model, you can iterate through the results after prediction and keep a tally for each category. Here's a quick example using Python:

from ultralytics import YOLO
from collections import Counter

# Load your segmentation model
model = YOLO('yolov8n-seg.pt')  # Assuming you're using the YOLOv8n-seg model

# Predict on an image
results = model('path/to/your/image.jpg')

# Count objects by category
object_counts = Counter([model.names[int(cls)] for cls in results[0].masks.cls])

# Print the counts
for category, count in object_counts.items():
    print(f"{category}: {count}")

This code loads the segmentation model, predicts on an image, and then uses a Counter from Python's collections module to tally the occurrences of each category in the segmented objects. Finally, it prints out the counts for each category. Hope this helps! 😊

[gaotaogaotao]

Hello, may I ask how to count and print segmented objects of different categories?

[pderrenger]

Hello! To count and print segmented objects of different categories using a YOLOv8 segmentation model, you can iterate through the results after prediction and keep a tally of each category. Here's a quick example using Python:

from ultralytics import YOLO
from collections import Counter

# Load a pretrained segmentation model
model = YOLO('yolov8n-seg.pt')

# Predict on an image
results = model('https://ultralytics.com/images/bus.jpg')

# Count objects by category
object_counts = Counter([model.names[int(cls)] for cls in results[0].masks.cls])

# Print counts
for category, count in object_counts.items():
    print(f"{category}: {count}")

This code loads a pretrained segmentation model, runs a prediction on an image, and then uses a Counter from Python's collections module to tally the occurrences of each object category based on the class IDs in the results. Finally, it prints the count of objects for each category. 📊

[Boaruzhanchik]

Hi, can you tell me how I can get the detected car plates in separate windows (as many as I have detected and output), my code outputs only one window.

while True:
ret, im0 = cap.read()
if not ret:
break
results = model(im0,verbose=False,conf=0.4,retina_masks=True)
boxes = results[0].boxes.xyxy.tolist()
classes = results[0].boxes.cls.tolist()
confidences = results[0].boxes.conf.tolist()
masks = results[0].masks
if results[0].masks is not None:
clss = results[0].boxes.cls.cpu().tolist()
masks = results[0].masks.xy
coordinates_mask = masks[0]
contour = coordinates_mask.reshape((-1, 1, 2)).astype(np.int32)
epsilon = 0.04 * cv2.arcLength(contour, True)
approx = cv2.approxPolyDP(contour, epsilon, True)
auto_transform = four_point_transform(im0, approx.reshape(-2, 2))
cv2.imshow('automated', auto_transform)

[pderrenger]

@Boaruzhanchik hey there! 👋 To display each detected car plate in separate windows, you can loop through the detected boxes and create a window for each. Here's a quick example to get you started:

for i, box in enumerate(boxes):
    x1, y1, x2, y2 = map(int, box[:4])
    plate = im0[y1:y2, x1:x2]
    cv2.imshow(f'Plate {i}', plate)
cv2.waitKey(0)
cv2.destroyAllWindows()

This code snippet extracts each detected plate based on its bounding box coordinates and displays it in a new window. Make sure to replace im0 with your frame variable if it's named differently. Hope this helps! 😊

[Boaruzhanchik]

Thanks a lot for the help, I did it this way, it seems to work too

if results[0].masks is not None:
    clss = results[0].boxes.cls.cpu().tolist()
    masks = results[0].masks.xy
    if len(masks) == 1:
         coordinates_mask = masks[0]
         auto_transformations_first(coordinates_mask)
    elif len(masks) <= 2:
          auto_transformations_first(coordinates_mask)
          coordinates_mask_2 = masks[1]
          auto_transformations_second_screen(coordinates_mask_2)
def auto_transformations_first(coordinates_mask):
     contour = coordinates_mask.reshape((-1, 1, 2)).astype(np.int32)
     epsilon = 0.04 * cv2.arcLength(contour, True)
     approx = cv2.approxPolyDP(contour, epsilon, True)
     auto_transform = four_point_transform(im0, approx.reshape(-2, 2))
     cv2.imshow('automated_first', auto_transform)

[pderrenger]

@Boaruzhanchik hey there! 👋 It looks like you've found a way to handle different numbers of masks with your approach. Nicely done! 🎉 Just a quick tip: you might want to ensure coordinates_mask is defined before using it in auto_transformations_first(coordinates_mask) for the elif len(masks) <= 2: case, to avoid any potential errors. Other than that, your method seems solid. Keep up the great work, and if you have any more questions or need further assistance, feel free to ask. Happy coding! 😊

[yingchaoAo]

My segmentation dataset is VOC format, can I use the yolo v8-seg model to train it? My label file is in png format.

[glenn-jocher]

Yes, you can train the YOLOv8-seg model with your VOC format dataset. For the label files in PNG format, ensure they are properly annotated with instance segmentation masks. Here's a quick example to get you started with training:

from ultralytics import YOLO

# Load the YOLOv8 segmentation model
model = YOLO('yolov8n-seg.pt')  # Load a pretrained segmentation model

# Train the model on your dataset
results = model.train(data='your_dataset_config.yaml', epochs=100, imgsz=640)

Make sure your dataset config file (your_dataset_config.yaml) is correctly set up for your VOC dataset. If you need to convert your dataset to a compatible format, consider using tools like Roboflow to ease the process. Happy training! 🚀

[Bjyh22]

My question is same as "how to count and print segmented objects of different categories". But if my source is a frame, it will report an error like ‘object_counts = Counter([model.names[int(cls)] for cls in results[0].masks.cls])
AttributeError: 'NoneType' object has no attribute 'cls'’

[Bjyh22]

What can i do? Hope for help.

[Bjyh22]

My problem has been solved, and after reading the Results documentation, I see that cls is now in boxes instead of masks. But thank you for developing such a great yolov8

[glenn-jocher]

@Bjyh22 hello! It looks like you're trying to count segmented objects using a detection model rather than a segmentation model. The error you encountered typically arises when attempting to access segmentation-specific attributes on results from a detection model. For instance segmentation, ensure you're working with a segmentation model (yolov8n-seg.pt, for example).

Here's a quick fix if you're aiming for instance segmentation:

1. Confirm you're using a segmentation model, which will include masks in the results.
2. Ensure your source frame is compatible and correctly processed by the model.

For instance segmentation with yolov8n-seg.pt:

from ultralytics import YOLO
from collections import Counter

# Load the segmentation model
model = YOLO('yolov8n-seg.pt')

# Run prediction on your frame (assuming 'frame' is your image)
results = model(frame)

# Assuming segmentation model, we should check for presence of 'masks'
if results[0].masks:
    object_counts = Counter([model.names[int(cls)] for cls in results[0].masks.cls])
    # Printing the counted objects
    print(object_counts)
else:
    print("No segmented objects found or incorrect model type.")

This code snippet will count and print the number of segmented objects per category if you're using an instance segmentation model and have segmented results. If results[0].masks is not available (None), it likely means no objects were detected or the model loaded is not for segmentation.

Hope this helps! If you have more questions, feel free to ask. 😊

[apsthe]

hello, may i ask how to use this for semantic segmentation?

[pderrenger]

Hello! 😊 For semantic segmentation with YOLOv8, you'll want to use our segmentation models, such as yolov8n-seg.pt. Here's how you can do it:

1. To load a pretrained segmentation model:

   from ultralytics import YOLO
   model = YOLO('yolov8n-seg.pt')

2. To run predictions on an image:

   results = model('path/to/your/image.jpg')
   results.show()

3. To train a model on your segmentation dataset:

   results = model.train(data='your_dataset.yaml', epochs=100, imgsz=640)

Make sure your dataset is in the correct format, detailed in our docs. For more customization options and commands, our documentation provides extensive guides. Enjoy experimenting with your segmentation tasks! 🚀

[741opamp]

Hello, I encountered the following error when I used pyside6 to put the segmentation result on the UI

Traceback (most recent call last):
  File "d:/ultralytics-main/base_ui.py", line 35, in open_img
    qimage = self.img_pred(file_path)
  File "d:/ultralytics-main/base_ui.py", line 26, in img_pred
    return convert2QImage(img)
  File "d:/ultralytics-main/base_ui.py", line 10, in convert2QImage
    height, width, channel = img.shape
ValueError: not enough values to unpack (expected 3, got 2)

part of my code is below

def convert2QImage(img):
    height, width, channel = img.shape
    return QImage(img, width, height, width * channel, QImage.Format_RGB888)

 def img_pred(self,file_path):
    results = self.model(file_path)
       
    img = results[0].boxes.xywh.cpu().numpy()
    print(img)
    return convert2QImage(img)

 def open_img(self):
     print("img opened")
     file_path = QFileDialog.getOpenFileName(self, dir="./dataset/images/train", filter="*.jpg;*.png")
     if file_path[0]:
         file_path = file_path[0]
         qimage = self.img_pred(file_path)
         self.input.setPixmap(QPixmap(file_path))
         self.output.setPixmap(QPixmap.fromImage(qimage))

Could this be caused when the result is converted to an array with decimals in it or something else in the code?

[glenn-jocher]

Hey there! 👋 It seems the issue you're encountering is due to the shape of the img variable returned by the img_pred() method. The error suggests that img is returning a shape with only two dimensions (likely because you're working with bounding boxes that don't include channel information), but convert2QImage() expects an image with three dimensions: height, width, and channels.

A simple workaround is to modify the convert2QImage() function to check the number of dimensions in img and adjust accordingly. For example:

def convert2QImage(img):
    if img.ndim == 2:  # If img is grayscale (height, width)
        img = np.stack((img,)*3, axis=-1)  # Convert to RGB
    height, width, channel = img.shape
    bytesPerLine = 3 * width
    return QImage(img.data, width, height, bytesPerLine, QImage.Format_RGB888)

This checks if img is a 2D array (grayscale image) and converts it to a 3-channel RGB image by stacking the grayscale values across three channels. This should solve the issue by ensuring the img passed to convert2QImage() always has three dimensions.

Regarding the segmentation results, since you're working with a segmentation model, the output would not directly be in a format suitable for convert2QImage() without additional processing like creating a mask overlay or converting segmentation maps to visual formats. Make sure you're processing the segmentation results (e.g., masks) correctly before attempting to convert them to QImage.

Hope this helps! Let me know if you have any more questions. 😊

[wkywwds]

Hello, can you give the frame diagrammodule of YOLOv8 segmentation model

[pderrenger]

Hey there! 😄 It seems like you're looking for the framework on how the YOLOv8 segmentation model is structured, right?

The YOLOv8 segmentation models, including yolov8n-seg, go beyond just identifying objects by segmenting each identified object from the rest of the image. This is particularly useful when not only the location but the precise shape of objects in images is required.

For instance, to train, validate or predict with the YOLOv8 segmentation models in Python you can use the following examples:

🔧 To Train:

from ultralytics import YOLO

# Load and train a model
model = YOLO('yolov8n-seg.pt')  # This loads the pretrained segmentation model
results = model.train(data='coco128-seg.yaml', epochs=100, imgsz=640)

✅ To Validate:

# Validate the trained model
metrics = model.val()

👁 To Predict:

# Predict on new images
results = model('path/to/your/image.jpg')

And to export a trained segmentation model to say ONNX format, it’s as simple as:

# Export the model
model.export(format='onnx')

For a detailed dive, our documentation on instance segmentation is a great next stop: 实例分割

As of now, we don't provide a frame diagram of the segmentation model's architecture directly. However, the documentation and code snippets should give a clear understanding of how the model operates on tasks.

If you're looking for something more detailed or specific, don't hesitate to let me know! 😊

[wkywwds]

Is there any difference in the framework of the segmentation model compared to the detection model

[pderrenger]

@wkywwds hey there! 👋 The segmentation model, such as yolov8n-seg, does indeed differ from the standard detection models like yolov8n. While both are adept at identifying objects, the segmentation model takes it a step further by precisely outlining the shape of each identified object. This is achieved through the generation of masks or contours around objects, allowing for a detailed separation from the image's background.

For instance, while a detection model might simply indicate a bounding box around a person, the segmentation model outlines the exact boundaries, even capturing subtle outlines around complex shapes. This makes segmentation models incredibly useful for tasks requiring a nuanced understanding of object shapes and their spatial layouts within images.

Here's a quick peek into using a YOLOv8 segmentation model in Python:

from ultralytics import YOLO

# Load the segmentation model
model = YOLO('yolov8n-seg.pt')  # A pretrained segmentation model

# Predict and segment objects in an image
results = model('path/to/your/image.jpg')

# The results contain both the bounding boxes and the segmentation masks

This model is especially great for scenarios where precise object shapes, beyond just their locations, are crucial. 🎨

[CoinBin27]

Hello! I've been working with the segmentation model and encountered an issue I hope you can help with.
After training the model on a railway segmentation dataset with YOLOv8m-seg and using it for prediction, I noticed a discrepancy: the output txt file outlines fewer polygons than expected compared to the predicted image results. Could anyone explain why this might be happening or suggest a solution? Thanks in advance for your help!

[CoinBin27]

This is my CLI:

yolo segment predict model=/mnt/data/rhl/ultralytics-main/runs/segment/train5/weights/best.pt source=/mnt/data/rhl/ultralytics-main/testPics/railway save_txt=True show_boxes=False conf=0.2 save_conf=True retina_masks=True

And i also checked the attribute mask.xy of result. The output was the same.

[pderrenger]

Hi there! 👋 Thanks for reaching out and sharing the details of your CLI command. It seems you're facing an issue where the exported .txt files from your predictions with YOLOv8m-seg model are showing fewer polygons compared to what's visible in the prediction images. This could be due to the difference in how the segmentations are being saved versus how they're visualized.

For segmentation tasks, the .txt files save the class and confidence values, while the mask details are often stored separately or visualized directly on the images. If you're looking to have detailed segmentation outputs (like polygons or mask data) saved, you'll want to ensure that retina_masks=True is effectively generating and saving those mask details as expected.

A quick thing you can check is to ensure the format and pathway for saving these detailed segmentation outputs are correctly specified and accessible. Moreover, verifying the mask.xy attribute indicates that the mask details are available but might not be saved in the format you're expecting.

For debugging, you might try explicitly specifying the path and format for saving segmentation outputs if not already. Here’s a code snippet that might help guide you on how to save these outputs:

from ultralytics import YOLO

# Assume your model and source are correctly loaded
results = model.predict(source)

# For each result, save or process segmentation masks
for i, result in enumerate(results):
    # Check and process result.masks here
    # E.g., save mask details to a file in your preferred format
    pass

Please make sure your environment has the necessary permissions to write files to the specified locations, and review the output directory for any generated mask details.

If the issue persists, please share more details about the output files and any errors/logs you might see. That would help in diagnosing the problem further! 🛠️

[CoinBin27]

I have already set retina_masks=True, but it seems that this parameter does not affect the output results. Let me further explain the issue I encountered: there is a continuous railway, and a utility pole blocks a part of the railway in the image. We consider this railway as divided into three parts: A, B, and C, where part B is the portion blocked by the utility pole. When I use the trained model to perform inference on this image, in the output image, parts A and C are detected by the model and highlighted in red. However, the model combines parts A and C together as one object, and the output txt file and masks.xy only contain the polygon for part A.

[pderrenger]

Hey there! It looks like retina_masks=True is correctly set for exporting the masks, which is great. However, the issue you're encountering with part A and C being recognized as one object instead of separate entities is not uncommon in object segmentation tasks, especially when these parts are closely located or visually similar.

One possible reason for the combining of parts could be how the model interprets the visual continuity of the railway, particularly when it's partially obscured. If the model's perception of object boundaries is primarily based on visual cues, the utility pole might not provide a strong enough signal to indicate a segmentation between parts A and C.

A potential solution to improve segment differentiation in such scenarios could involve fine-tuning your model with more varied examples that specifically include cases of obscured or segmented objects of interest, like railways interrupted by other items. This way, your model can learn to recognize and segment objects more accurately even when partially blocked.

It might also be helpful to adjust the model's confidence threshold (conf=0.2) to see if it impacts the segmentation positively. Sometimes, tweaking this value helps in catching more subtle distinctions between object segments.

Here's a small piece of code for fine-tuning with custom data, assuming you've already prepared a dataset with the mentioned characteristics:

yolo segment train data=your_custom_dataset.yaml model=yolov8n-seg.pt epochs=50 imgsz=640

Remember to replace your_custom_dataset.yaml with your dataset configuration file. Fine-tuning not only tailors the model more closely to your specific requirements but can also help address unique challenges like the one you're facing. 🚂🛤️

Let me know if there's anything else I can do to assist!

[Chen-s-l]

Hello, may I ask how to output the area of each mask in the image, and count the number of masks corresponding to different categories？

[glenn-jocher]

Hello! To output the area of each mask and count the number of masks for different categories, you can use the results object after running inference with your segmentation model. Here's a quick example on how you can achieve that with Python:

from ultralytics import YOLO

# Load your segmentation model
model = YOLO('yolov8n-seg.pt')  # Use the appropriate model for your task

# Predict on your image
results = model('path/to/your/image.jpg')

# Calculate areas and counts
areas = [mask.area.cpu().numpy() for mask in results.masks]
categories = [results.names[i] for i in results.masks.label.cpu().numpy()]

# Count masks per category
category_counts = {category: categories.count(category) for category in set(categories)}

print("Areas of masks:", areas)
print("Number of masks per category:", category_counts)

This will give you the areas of each detected object's mask and the count of masks per category. Hope this helps! 😊

[LeaderOfFuture]

Hello, I am doing an instance segmentation task on the COCO dataset, how to output its COCO evaluation metric when verifying. For example, map, map50, map75, map50-95, map(large, medium, small)...

[glenn-jocher]

Hello! 😊 For evaluating your instance segmentation tasks on the COCO dataset with metrics such as mAP, mAP50, mAP75, mAP50-95, and mAP for different object sizes, you can use the val mode after training your model. When you run validation with yolo segment val model=<your_model>.pt, the output will include these metrics automatically if your dataset is properly configured with COCO-style annotations.

Here's a quick example of how you could validate a trained model on the COCO dataset:

yolo segment val model=yolov8n-seg.pt data=coco.yaml

This command assumes you have a COCO dataset correctly set up and referenced by coco.yaml, and your model (yolov8n-seg.pt) trained for the segmentation task. The validation output will give you detailed metrics including mAP across different thresholds and object sizes.

Let us know if you need further assistance!

[LeaderOfFuture]

Thanks for your reply. In fact, I do instance segmentation tasks on coco dataset. After I improve some modules of yolov8 for training, I verify the training model effect and want to obtain its coco type evaluation indexes, such as mAP, mAP75, mAP(large), mAP(medium), mAP(small). I generate JSON files to verify the training effect of the model, and then follow the pycocotools package in the Linux environment to try to obtain its coco type instance segmentation evaluation indexes. But the results are not ideal. Here is the code for the get_COCO_metric.py file.

import argparse
from pycocotools.coco import COCO
from pycocotools.cocoeval import COCOeval

def parse_opt():
parser = argparse.ArgumentParser()
parser.add_argument('--anno_json', type=str, default='/public/home/ultralytics-main/datasets/coco/labels/instances_val2017.json', help='training model path')
parser.add_argument('--pred_json', type=str, default='/public/home/ultralytics-main1/runs/segment/val2/predictions.json', help='data yaml path')

return parser.parse_known_args()[0]

if name == 'main':
opt = parse_opt()
anno_json = opt.anno_json
pred_json = opt.pred_json

anno = COCO(anno_json)  # init annotations api
pred = anno.loadRes(pred_json)  # init predictions api
eval = COCOeval(anno, pred, 'segm') # segm  bbox  keypoints
eval.evaluate()
eval.accumulate()
eval.summarize()

============================================================
Its output is as follows.
loading annotations into memory...
Done (t=0.67s)
creating index...
index created!
Loading and preparing results...
DONE (t=8.28s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type segm
DONE (t=25.78s).
Accumulating evaluation results...
DONE (t=11.63s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.001
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.001
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.001
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.001
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.001
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.024
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.041
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.043
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.032
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.047
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.056

I don't know what go wrong.
The output in val mode is：
Class Images Instances Box(P R mAP50 mAP50-95) Mask(P R mAP50 mAP50-95)
all 5000 36335 0.639 0.473 0.52 0.372 0.631 0.459 0.495 0.311

[glenn-jocher]

Hi there! Thanks for sharing your process and the output you're getting during COCO evaluation after making modifications to YOLOv8. Based on the output you've shared, it looks like the model is not performing as expected given the very low mAP values across the board.

Given the significant difference in metrics between your custom val output and what's expected from a successfully trained YOLOv8 segment model (as seen in the Ultralytics documentation), a few things could be happening:

1. Data Preparation: Ensure your COCO dataset is correctly prepared and annotated for instance segmentation tasks. Any discrepancy in annotation format or missing labels could significantly affect performance.

2. Model Training: Consider reviewing the modifications made to the model. It's possible that certain changes might not be contributing positively to learning. One way to diagnose this is to incrementally introduce modifications and validate their impact on a smaller dataset.

3. Prediction JSON: The predictions.json you're generating should closely match the format COCO evaluation expects. This includes ensuring that segmentation masks are correctly formatted and associated with the right category IDs.

4. Evaluation Script: Your get_COCO_metric.py script seems fine at first glance. To double-check, you might want to compare its output using predictions from an unmodified, pretrained model directly from Ultralytics. This could serve as a baseline to confirm the script works as expected before using it with modified model outputs.

For better diagnostic insights:

• When running your validation in the Linux environment with pycocotools, ensure your environment is correctly set up, and verify that other models (perhaps an unmodified version) give expected results using the same evaluation script.
• Also, closely inspect your predictions.json for any anomalies like unexpected bounding box sizes, incorrect categories, or badly formed segmentation masks.

Here’s a small adjustment tip: If you're not already using it, consider visualizing some predictions using the Ultralytics inference code to manually inspect if the segmentation masks and categories look reasonable.

Hope this helps! If the issue persists, feel free to share more details about the modifications and perhaps some sample outputs for further assistance. 🤝

[Daniel-lucifer]

How to use the segmentation function of YOLOv8 to segment a panoramic image and a target image simultaneously, and then use feature point matching to perform feature point matching on the segmented object in both images. Match feature points only between segmented objects in two images

[glenn-jocher]

@Daniel-lucifer to perform instance segmentation on both a panoramic image and a target image simultaneously using YOLOv8, you can follow these steps:

1. Load the Pretrained Model: Use a segmentation model like yolov8n-seg.pt which is trained for instance segmentation tasks.
2. Run Segmentation: Segment both images to get the masks and class labels for each detected object.
3. Feature Point Matching: Extract feature points from the segmented areas of both images. You can use OpenCV or similar libraries for feature extraction and matching.

Here's a basic example in Python using Ultralytics YOLO:

from ultralytics import YOLO
import cv2

# Load the segmentation model
model = YOLO('yolov8n-seg.pt')

# Load images
img1 = cv2.imread('panoramic.jpg')
img2 = cv2.imread('target.jpg')

# Perform segmentation
results1 = model(img1)
results2 = model(img2)

# Assuming you have a function to extract and match features
matches = match_features(results1, results2)

def match_features(results1, results2):
    # Extract features and perform matching
    # This is a placeholder for your feature matching logic
    pass

For feature point matching, you might consider using SIFT, SURF, or ORB algorithms provided by OpenCV, focusing only on the regions defined by the segmentation masks.

This approach ensures that feature matching is constrained to the segmented objects, potentially improving the relevance and accuracy of the matches.

[Daniel-lucifer]

from ultralytics import YOLO
import cv2
import numpy as np

初始化模型

model = YOLO('F:/ultralytics-main/yolov8n-seg.pt')

读取图像

img1 = cv2.imread('C:/Users/801/Desktop/lab.jpg')
img2 = cv2.imread('C:/Users/801/Desktop/test/image3.jpg')

进行分割

results1 = model(img1)
results2 = model(img2)

def extract_features_and_keypoints(segment):
# 使用SIFT提取特征
sift = cv2.SIFT_create()
# 假设segment是灰度图
if len(segment.shape) == 3:
segment = cv2.cvtColor(segment, cv2.COLOR_BGR2GRAY)
keypoints, descriptors = sift.detectAndCompute(segment, None)
return keypoints, descriptors

def match_features(descriptors1, descriptors2):
# 使用FLANN匹配器进行特征匹配
index_params = dict(algorithm=1, trees=5)
search_params = dict(checks=50)
flann = cv2.FlannBasedMatcher(index_params, search_params)
matches = flann.knnMatch(descriptors1, descriptors2, k=2)
# 应用Lowe's ratio test
good_matches = []
for m, n in matches:
if m.distance < 0.7 * n.distance:
good_matches.append(m)
return good_matches

def process_and_match_segments(results1, results2):
# 提取分割的结果，通常这将是第一个层级包含的分割对象
segment1 = results1.xyxy[0] # 得到分割结果的bounding box
segment2 = results2.xyxy[0] # 得到分割结果的bounding box

# 通过bounding box裁剪图像获取关注区域
cropped1 = img1[int(segment1[1]):int(segment1[3]), int(segment1[0]):int(segment1[2])]
cropped2 = img2[int(segment2[1]):int(segment2[3]), int(segment2[0]):int(segment2[2])]

keypoints1, descriptors1 = extract_features_and_keypoints(cropped1)
keypoints2, descriptors2 = extract_features_and_keypoints(cropped2)

matches = match_features(descriptors1, descriptors2)
return keypoints1, keypoints2, matches, cropped1, cropped2

处理分割结果并匹配特征

keypoints1, keypoints2, matches, cropped1, cropped2 = process_and_match_segments(results1, results2)

可视化匹配结果

img_matches = cv2.drawMatches(cropped1, keypoints1, cropped2, keypoints2, matches, None,
flags=cv2.DrawMatchesFlags_NOT_DRAW_SINGLE_POINTS)
cv2.imshow('Matches', img_matches)
cv2.waitKey(0)
cv2.destroyAllWindows()
Please help me check this code so that it can call the segmented result and perform feature point matching between the segmented objects on two images，thanks

[glenn-jocher]

@Daniel-lucifer hello! It looks like you've set up your segmentation and feature matching pipeline quite well. Here are a few suggestions to ensure everything works as expected:

1. Segmentation Results: Ensure that results1.xyxy[0] and results2.xyxy[0] are indeed the bounding boxes you want to use. These should correspond to the objects of interest in both images. If there are multiple objects, you might need to iterate through results or select them based on class IDs or confidence scores.

2. Feature Extraction: Your feature extraction and matching logic seem solid. Just make sure that the cropped regions from the segmentation are correctly converted to grayscale (if they aren't already) before passing them to the SIFT detector.

3. Visualization: The visualization step with cv2.drawMatches is a great way to see the matched features. Ensure that the keypoints and matches are correctly passed to this function.

If you encounter any specific errors or unexpected results, feel free to share more details, and I'd be happy to help further! Good luck with your feature matching! 🚀

[Daniel-lucifer]

The result after I ran it was that only two images were segmented, and there was no result of matching feature points. Can you help me check which part can be modified，thanks

[glenn-jocher]

@Daniel-lucifer hi there! It sounds like you might be facing an issue with the feature matching step after successfully segmenting your images. Here are a couple of things you might want to check:

1. Feature Extraction: Ensure that the feature extraction is being applied correctly to the segmented areas. Verify that the keypoints and descriptors are being computed only within the segmented regions.

2. Matching Logic: Double-check your feature matching logic. Ensure that the matcher is set up correctly and that you're filtering matches based on a reliable criterion, such as Lowe's ratio test.

If you're still facing issues, it might be helpful to visualize the keypoints on the segmented areas to ensure they are being detected correctly. If you can provide any specific errors or more details, I'd be happy to help further! 😊

[Daniel-lucifer]

How to extract feature points on segmented objects

[glenn-jocher]

@Daniel-lucifer to extract feature points on segmented objects using a YOLOv8-seg model, you can follow these steps:

1. Load the Model: Load your segmentation model.

   from ultralytics import YOLO
   model = YOLO('yolov8n-seg.pt')  # Load a pretrained segmentation model

2. Run Prediction: Perform segmentation on your image.

   results = model('path/to/your/image.jpg')

3. Extract Masks: Access the segmentation masks from the results.

   masks = results.masks  # This will give you the segmentation masks

4. Feature Points Extraction: You can then apply feature detection techniques (like Harris Corner Detection, SIFT, etc.) on these masks to extract feature points.

Here’s a quick example of how you might apply a simple feature detection on the masks using OpenCV:

import cv2
import numpy as np

# Assuming 'masks' is a numpy array from the YOLO segmentation output
for i, mask in enumerate(masks):
    mask = (mask * 255).astype(np.uint8)  # Convert mask to 8-bit image
    corners = cv2.goodFeaturesToTrack(mask, maxCorners=100, qualityLevel=0.01, minDistance=10)
    # corners will contain the coordinates of the feature points

This will give you the feature points for each segmented object in your image. For more detailed processing or different feature extraction techniques, adjust the methods accordingly.

[HouNAiL]

Hello, I would like to know how to add other evaluation metrics, such as Dice and accuracy, which are commonly used in semantic segmentation tasks.

[glenn-jocher]

Hello,

Thank you for your question! Adding additional evaluation metrics such as Dice and accuracy to your YOLOv8 segmentation tasks is a great way to enhance the analysis of your model's performance.

To integrate these metrics, you can modify the validation script to include the calculations for Dice coefficient and accuracy. Below is an example of how you can extend the existing validation code to include these metrics:

Python Example

from ultralytics import YOLO
import numpy as np

# Load a model
model = YOLO("yolov8n-seg.pt")  # load an official model
# or
# model = YOLO("path/to/best.pt")  # load a custom model

# Define a function to calculate Dice coefficient
def dice_coefficient(y_true, y_pred, smooth=1):
    intersection = np.sum(y_true * y_pred)
    return (2. * intersection + smooth) / (np.sum(y_true) + np.sum(y_pred) + smooth)

# Define a function to calculate accuracy
def accuracy(y_true, y_pred):
    return np.sum(y_true == y_pred) / y_true.size

# Validate the model
metrics = model.val()

# Calculate additional metrics
for result in metrics:
    y_true = result.masks  # Ground truth masks
    y_pred = result.pred  # Predicted masks

    dice = dice_coefficient(y_true, y_pred)
    acc = accuracy(y_true, y_pred)

    print(f"Dice Coefficient: {dice}")
    print(f"Accuracy: {acc}")

CLI Example

If you prefer using the CLI, you can export the predictions and then calculate the metrics using a custom script:

# Run validation and save predictions
yolo segment val model=yolov8n-seg.pt save_txt=True

# Custom script to calculate Dice and accuracy
python calculate_metrics.py

In calculate_metrics.py, you can read the saved predictions and ground truth masks, then compute the Dice coefficient and accuracy as shown in the Python example above.

For more detailed information on instance segmentation and how to work with YOLOv8 models, you can refer to the Ultralytics documentation on segmentation.

If you encounter any issues or need further assistance, please provide a minimum reproducible example of your code and ensure you are using the latest versions of torch and ultralytics. This will help us investigate and provide a more accurate solution. You can find more details on creating a minimum reproducible example here.

Happy coding! 😊

[pderrenger]

@HouNAiL hello,

Thank you for your question! Adding additional evaluation metrics such as Dice and accuracy to your YOLOv8 segmentation tasks can provide more insights into your model's performance, especially for semantic segmentation tasks.

To incorporate these metrics, you can modify the validation script to include custom metric calculations. Here's a basic example of how you can add Dice coefficient and accuracy to your evaluation:

Python Example

from ultralytics import YOLO
import torch

def dice_coefficient(pred, target, smooth=1e-6):
    intersection = (pred * target).sum()
    union = pred.sum() + target.sum()
    dice = (2. * intersection + smooth) / (union + smooth)
    return dice

def accuracy(pred, target):
    correct = (pred == target).sum().item()
    total = target.numel()
    return correct / total

# Load a model
model = YOLO("yolov8n-seg.pt")  # load an official model
model = YOLO("path/to/best.pt")  # load a custom model

# Validate the model
metrics = model.val()  # no arguments needed, dataset and settings remembered

# Calculate additional metrics
for result in metrics:
    pred_mask = result.masks  # predicted masks
    true_mask = result.labels  # true masks

    dice = dice_coefficient(pred_mask, true_mask)
    acc = accuracy(pred_mask, true_mask)

    print(f"Dice Coefficient: {dice:.4f}")
    print(f"Accuracy: {acc:.4f}")

CLI Example

Currently, the CLI does not directly support adding custom metrics. However, you can export your predictions and then calculate the metrics using a separate script.

# Predict and save results
yolo segment predict model=yolov8n-seg.pt source='path/to/your/images' save=True

After running the above command, you can write a script to load the saved predictions and calculate the Dice coefficient and accuracy.

For more detailed information on segmentation tasks, you can refer to our Instance Segmentation Documentation.

If you encounter any issues or need further assistance, please ensure you are using the latest versions of torch and ultralytics. If the problem persists, providing a minimum reproducible example will help us investigate further. You can find more details on creating a reproducible example here.

[ZHIYUAN816]

Could you provide the architecture diagram of yolov8-seg?

[glenn-jocher]

@ZHIYUAN816 hello!

Thank you for your interest in YOLOv8-seg! Currently, we don't have a specific architecture diagram for YOLOv8-seg available in our documentation. However, the architecture of YOLOv8-seg builds upon the core YOLOv8 model with additional layers and modifications to support instance segmentation tasks.

For a deeper understanding of the YOLOv8 architecture and its segmentation capabilities, I recommend checking out the Ultralytics YOLOv8 documentation. This resource provides comprehensive details on how the model works, including training, validation, prediction, and export functionalities.

If you have any specific questions or need further assistance, feel free to ask! 😊

[LwpZoe]

Hello, here's a question:
I want to use YOLO11 to train my dataset from scratch and make predictions, and don't want to use prepared pre-trained weights, so how do I write the training commands?
Is this code below ok:
yolo segment train data=datasets/yolo_seg/data.yaml model=yolo11n-seg.yaml epochs=300 batch=16 resume=True imgsz=640 device=0,1

[glenn-jocher]

@LwpZoe your command is almost correct for training from scratch without pretrained weights. Ensure the resume=True argument is removed, as it is only used to resume interrupted training. The corrected command would be:

yolo segment train data=datasets/yolo_seg/data.yaml model=yolo11n-seg.yaml epochs=300 batch=16 imgsz=640 device=0,1

This will train your YOLO11 segmentation model from scratch using your custom dataset. For more details, visit the Segment Task Documentation.

[ZHIYUAN816]

correct

…

---- Replied Message ---- | From | ***@***.***> | | Date | 12/12/2024 15:06 | | To | ***@***.***> | | Cc | ***@***.***>***@***.***> | | Subject | Re: [ultralytics/ultralytics] zh/tasks/segment/ (Discussion #8508) | Hello, here's a question: I want to use YOLO11 to train my dataset from scratch and make predictions, and don't want to use prepared pre-trained weights, so how do I write the training commands? Is this code below ok: yolo segment train data=datasets/yolo_seg/data.yaml model=yolo11n-seg.yaml epochs=300 batch=16 resume=True imgsz=640 device=0,1 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[glenn-jocher]

@ZHIYUAN816 yes, your command is correct for training from scratch without pre-trained weights. Ensure your dataset and configuration files are properly formatted. If you encounter issues, feel free to ask. For more details, refer to the Train Documentation.

[LwpZoe]

Hello, I'd like to ask a question: I'd like to add a CBAM Attention Mechanism module to the yolo11-seg network model to improve my segmentation accuracy, as the current precision is only about 65. I don't know in which folder to import this python file, i.e. I don't find the network neck in which folder at the moment.

[glenn-jocher]

@LwpZoe to add a CBAM module to the YOLO11-seg architecture, you will need to modify the model's configuration files and code. The network's architecture is defined in the ultralytics/cfg/models/11 folder. Look for the specific YAML file (e.g., yolo11n-seg.yaml) corresponding to your model, as this defines the backbone, neck, and head. You can integrate the CBAM module in the neck by editing the relevant sections and ensuring the Python implementation of CBAM is correctly imported and utilized. For further details, consult the Ultralytics YOLO11 models documentation.