I have my own pre-trained YOLO model, and I want to have different colors of LEDs light up on Arduino Uno breadbroad when different labels are being detected from my webcam.
So, I was thinking of first assigning the command for different LEDs to light up. ("G" for green led, "R" for red led, "Y" for yellow led, "A" for turning both red and yellow led at the same time, and "0" to turn off all led. ) I started with my Arduino code as below:
char command;
void setup() {
Serial.begin(9600);
pinMode(2, OUTPUT); // Green LED pin
pinMode(3, OUTPUT); // Red LED pin
pinMode(4, OUTPUT); // Yellow LED pin
}
void loop() {
if (Serial.available() > 0) {
command = Serial.read();
if (command == 'G') {
// Turn on green LED
digitalWrite(2, HIGH);
digitalWrite(3, LOW); // Turn off red LED
digitalWrite(4, LOW); // Turn off yellow LED
} else if (command == 'R') {
// Turn on red LED
digitalWrite(2, LOW); // Turn off green LED
digitalWrite(3, HIGH);
digitalWrite(4, LOW); // Turn off yellow LED
} else if (command == 'Y') {
// Turn on yellow LED
digitalWrite(2, LOW); // Turn off green LED
digitalWrite(3, LOW); // Turn off red LED
digitalWrite(4, HIGH);
} else if (command == 'A') {
// Turn on both red and yellow LEDs
digitalWrite(2, LOW); // Turn off green LED
digitalWrite(3, HIGH);
digitalWrite(4, HIGH);
} else if (command == '0') {
// Turn off all LEDs
digitalWrite(2, LOW);
digitalWrite(3, LOW);
digitalWrite(4, LOW);
}
}
}
In my YOLO model, there are three classes
#Classes
names:
0: aaa
1: bbb
2: ccc
Therefore, for my python code I started with first turning on the webcam and loading then using serial library to connect to arduino.
from ultralytics import YOLO
import serial
import time
import sys
# Load my YOLO model using
model = YOLO("best.pt")
# Open the serial port for communication with Arduino
arduino = serial.Serial('COM3', 9600) # Change 'COM3' to your Arduino port
after that, I want to assign the labels with command to Arduino, and I also try to do a little
# Map the class labels to corresponding Arduino commands
label_commands = {
'aaa': 'G',
'bbb': 'R',
'ccc': 'Y',
'aaa_bbb': 'A',
'aaa_bbb_ccc': 'A',
'none': '0'
}
while True:
try:
results = model.predict(source="0", show=True) # assumes '0' is your source identifier
# Extract the detected labels from results
detected_labels = [item['label'] for item in results.xyxy[0].numpy()]
# Determine the Arduino command based on detected labels
if 'aaa' in detected_labels and 'bbb' in detected_labels and 'ccc' in detected_labels:
command = label_commands['aaa_bbb_ccc']
elif 'aaa' in detected_labels and 'bbb' in detected_labels:
command = label_commands['aaa_bbb']
elif 'aaa' in detected_labels:
command = label_commands['aaa']
elif 'bbb' in detected_labels and 'ccc' in detected_labels:
command = label_commands['aaa_bbb_ccc']
elif 'bbb' in detected_labels:
command = label_commands['bbb']
elif 'ccc' in detected_labels:
command = label_commands['ccc']
else:
command = label_commands['none']
# Print the command for debugging
print(f"Sending command to Arduino: {command}")
sys.stdout.flush() # Flush the standard output
# Send the command to Arduino
arduino.write(command.encode())
except Exception as e:
print(f"Error: {e}")
sys.stdout.flush() # Flush the standard output
# Wait for 5 seconds before sending the next command
time.sleep(5)
BUT, this part of code doesn't work at all. I know my arduino is connected for sure, and I have also tested with from randoming sending the letter command to arduino from python, which worked perfectly.
Also when I run this code, the run window are showing only things like:
1/1: 0... Success ✅ (inf frames of shape 640x480 at 25.00 FPS)
WARNING ⚠️ inference results will accumulate in RAM unless `stream=True` is passed, causing potential out-of-memory
errors for large sources or long-running streams and videos. See https://docs.ultralytics.com/modes/predict/ for help.
Example:
results = model(source=..., stream=True) # generator of Results objects
for r in results:
boxes = r.boxes # Boxes object for bbox outputs
masks = r.masks # Masks object for segment masks outputs
probs = r.probs # Class probabilities for classification outputs
0: 480x640 (no detections), 141.0ms
0: 480x640 1 aaa, 182.2ms
0: 480x640 1 aaa, 134.3ms
0: 480x640 1 aaa, 122.1ms
0: 480x640 (no detections), 121.4ms
0: 480x640 (no detections), 147.5ms
0: 480x640 (no detections), 131.5ms
0: 480x640 (no detections), 140.3ms
0: 480x640 1 aaa, 127.1ms
0: 480x640 1 aaa, 124.1ms
0: 480x640 1 aaa, 123.2ms
0: 480x640 1 aaa, 172.2ms
...
even with my debugging script, it doesn't tell me what command is being sent to the arduino broad
# Print the command for debugging
print(f"Sending command to Arduino: {command}")
sys.stdout.flush() # Flush the standard output
For the YOLO part, this can be helpful: first, as you predict a live stream, use stream=True to get a results generator. Next, to reach the class names of the detected objects, you first need to get their class ids and then refer to the result.names which is a list of model names, in your case it will be ['aaa', 'bbb', 'ccc']. Example for a COCO pre-trained YOLO model:
model = YOLO(yolov8n.pt)
results = model.predict(source="0", stream=True)
# for each frame
for result in results:
class_ids = result.boxes.cls.tolist()
detected_labels = [result.names[int(i)] for i in class_ids]
#print(detected_labels)
#['person', 'person', 'cat']
#(if 2 person, 1 cat detected in a frame)
More about how to use results generator and how to work with the prediction results.