Currently, I'm using grayscaling to compare pixels, which is not really ideal I think since it loses a ton of information. What are some common algorithms for doing this? Is Euclidean distance, for example, generally considered a good metric? I'm sure there are many and I'd like to know of a way to pick the best one for the type of images I work with, railway tracks.
I was thinking of something like sampling a thousand points on two images, one with a train present and one without. Then I could take the difference between them (using any common metric) and then plot the distribution for starters. Ideally, once I have this distribution, I can perform some sort of statistical test to determine which algorithm behaves best.
One of the main advantages of using euclidean distance is that is simple to compute. However, the RGB space is not a good color space in which to compare color differences. Its grayscale projection is even less adapt.
Based on these premises, several color spaces have been proposed in history in order to measure human-perceived color differences with the euclidean distance.
The complexity of the human perception, however, collides with the attempt to use simple formulas for mapping coordinates relative to different color-spaces. This led to the proliferation of a number of solutions that balance the objective to approximate the human perception with the simplicity of the formulas.
The CIELAB is one of the most broadly used color-spaces for evaluating color differences. According to Wikipedia
CIELAB was intended as a perceptually uniform space, where a given numerical change corresponds to similar perceived change in color.
The original definition that uses just the euclidean distance dates back to 1976. Another Wikipedia page explains that various refinements have been published in history in order to correct the non-uniformities that were discovered later.
To summarize, for evaluating the human-perceived difference between two colors, I suggest you to proceed in the following way: