Given vertices of rectangle and triangle, I can't find or figure out an algorithm that would check if a rectangle (2D, x-y axis aligned, not rotated) is inside a triangle (2D).
The only way I see it is to check if all rectangle points are inside the triangle, but I need the algorithm to be as fast as possible, so maybe there is a faster way to do this.
As both a rectangle and a triangle are convex polygons, it suffices to check that the four corners of the rectangle lie inside the triangle. This can be done by plugging the coordinates of the corners into the implicit equations of the sides and checking the signs.
Maybe using this answer: https://stackoverflow.com/a/21510010/1196549