How would I go about detecting brightness in a specific area

I would like to make a robot to automatically play a guitar hero type of game, however all of the notes are the same color. These notes are bright and stand out, so I think that finding the average brightness between two points would be the best way of detecting notes. I am open to new suggestions on how to detect these notes though and any support would be appreciated. I would like to do this project in python, due to how its one of the only coding languages I know. Detecting the color however could work as well, because the notes would be fast, and it shouldn't take more than a millisecond to decide if its a note or not

I already tried googling for my answer, however it seems that there isn't a specific library I should use and there aren't many reputable sources about this topic.

Solution

I set up a little code that should get you started at what you want to do. I tried to keep it light on dependencies, but it does require mss for screenshots, PIL for image manipulations and numpy for some numerics.

In brief:

Take a screenshot and calibrate where the buttons are
Use those positions to draw lines along each coloured button
Each frame read the intensity along each line
Interpret the intensity vector to determine when to issue inputs to play the game

I didn't have rockband, so I looked up a youtube video of someone playing it to get some video, so it should be a pretty close match to your game, credit at the bottom.

Here's the mockup:

from PIL import Image, ImageDraw
import matplotlib.pyplot as plt
import mss
import numpy as np

def get_screenshot(sct: mss.base.MSSBase, monitor_dict: dict) -> Image:
    sct_img = sct.grab(monitor_dict)
    pil_img = Image.frombytes('RGB', sct_img.size, sct_img.bgra, 'raw', 'BGRX')
    return pil_img

def setup(img: Image, num_lines: int, lines: list[tuple[int, int, int, int]] = None):
    # If lines was not already provided, ask the user to provide the info to generate them
    # This is the calibration
    if lines is None:
        plt.imshow(img)
        plt.title('Select corners of bar, starting with bottom left going clockwise')
        points = plt.ginput(4, timeout=-1)
        plt.close()
        lower_line = int((points[0][1] + points[3][1]) / 2)
        upper_line = int((points[1][1] + points[2][1]) / 2)
        lower_space = (points[3][0] - points[0][0]) / num_lines
        upper_space = (points[2][0] - points[1][0]) / num_lines
        lines = [
            (int(points[0][0] + (i + 0.5) * lower_space), lower_line, int(points[1][0] + (i + 0.5) * upper_space), upper_line)
            for i in range(num_lines)
        ]
    # Draw the image with lines on it to let user verify good calibration
    draw = ImageDraw.Draw(img)
    for line in lines:
        draw.line(line, fill=128, width=3)
    plt.imshow(img)
    plt.title('Verify that lines align with screen, abort if not.')
    plt.show()
    return lines


def get_raster_lines(img, lines_expanded):
    # This function is set up to be pretty amenable to a numba improvement if necessary
    raster_lines = np.empty(shape=lines_expanded.shape[:2], dtype=float)
    for line_num in range(lines_expanded.shape[0]):
        for point_num in range(lines_expanded.shape[1]):
            raster_lines[line_num, point_num] = np.linalg.norm(img[lines_expanded[line_num, point_num][1], lines_expanded[line_num, point_num][0]])
    return raster_lines


def _main():
    # How many guitar lines, seems like 5 based on youtube videos
    num_lines = 5
    # The colours to make plots more understandable
    colour_order = ['green', 'red', 'yellow', 'blue', 'orange']
    # Set up the screen capturing library
    sct = mss.mss()
    monitor = sct.monitors[2]
    monitor['mon'] = 1
    img = get_screenshot(sct, monitor)
    # Run this the first time with lines set to None, then after you get the output, you can put it here
    lines = [(540, 799, 675, 470), (639, 799, 707, 470), (738, 799, 739, 470), (838, 799, 771, 470), (937, 799, 803, 470)]
    lines = setup(img, num_lines, lines)
    # lines = setup(img, num_lines)
    # lines follows: [(x0, y0, x1, y1), ...]
    #   [(bottom left), (top left), (top right), (bottom right)]
    print('lines: ', lines) # print so you can put it above and not have to calibrate every time
    # Generate all the points along the lines
    lines_expanded = np.array([
        [(int(np.interp(yval, (line[3], line[1]), (line[2], line[0]))), yval) for yval in range(line[3], line[1])]
        for line in lines
    ])
    # Wait to start so that you can set up the game or any other initial setup
    input('Press Enter when ready to start')
    print('Starting ...')
    # While true so that it will run for the whole game
    while True:
        try:
            # Take the screenshot
            img = get_screenshot(sct, monitor)
            # Pull out the raster lines
            raster_lines = get_raster_lines(np.array(img), lines_expanded)
            # Process raster_lines to generate your commands
            # In this case just plot to show that the information was captured
            for i in range(num_lines):
                plt.plot(raster_lines[i], color=colour_order[i])
            plt.legend([f'bar {colour_order[i]}' for i in range(num_lines)])
            plt.show()
        except KeyboardInterrupt:
            break
        break # comment this out to continue running

if __name__ == '__main__':
    _main()

When I run that I get:

lines:  [(540, 799, 675, 470), (639, 799, 707, 470), (738, 799, 739, 470), (838, 799, 771, 470), (937, 799, 803, 470)]
Press Enter when ready to start
Starting ...

Process finished with exit code 0

Here is the calibration points I used approximately:

And here is the calibration graph, where you can see the red lines run straight up each coloured key lane:

And here is the raster lines it generates:

I've annotated it here to show some features of interest:

And that should be enough to get you going!

Credit to this link for the video I tested with: https://www.youtube.com/watch?v=TIs9x8MfROk