YOLOv8: How to calculate map on test set

Question

Let's say I have a folder called 'test' with folders inside, 'images' and 'labels'. I also have a YOLOv8 model which I've trained called 'best.pt. My labels are polygons (yolo-obb .txt files).

I want to find the mean average precision (MAP) of my YOLOv8 model on this test set.

I've read both the documentation for predicting and benchmarking, however, I'm struggling to find an example of calculating map from some test images.

https://docs.ultralytics.com/modes/predict/

https://docs.ultralytics.com/modes/benchmark/

from ultralytics import YOLO

# Load a pretrained YOLOv8n model
model = YOLO('best.pt')

# Run inference on an image
results = model(['test/images/bus.jpg', 'test/images/zidane.jpg'])  # list of 2 Results objects

I imagine I have to put the list of images in the above, then write code to calculate map for everything in the test folder and average it. Are there packages that have already done this?

What's the code to achieve this task?

Answer 1

To validate YOLOv8 model on a test set do the following:

1. In the data.yaml file specify the test folder path as a val argument:

path: ../dataset  # dataset root dir
train: train  
val: test  # test directory path for validation

names:
  0: person
  1: bicycle

2. Validate the model:

from ultralytics import YOLO

# Load a pretrained YOLOv8 model
model = YOLO('best.pt')

# Run validation on a set specified as 'val' argument
metrics = model.val(data='data.yaml')

3. Get the relevant metrics:

metrics.results_dict

# output example

{'metrics/precision(B)': 0.8539022762405677,
 'metrics/recall(B)': 0.8012653662055587,
 'metrics/mAP50(B)': 0.8833459433957722,
 'metrics/mAP50-95(B)': 0.6608424795290528,
 'fitness': 0.6830928259157248}

More about the validation mode: https://docs.ultralytics.com/modes/val/