In a 2D game I want to imitate the player's vision using blacked out areas (for now). I figure the easiest way to do this is using a cG shader.
There are two main components to this, I have the wall endpoints and the player position. First, in the c# code, I rotate the wall endpoints around the player position like so (equation, not code):
x = px - (wx * cos(a) - wy * sin(a)) and y = py - (wx * sin(a) + wy * cos(a))
then in the shader I am using basic algebra to find the equation of the line produced by the wall and deciding if the current pixel is behind the wall or not:
y - y1 = m(x - x1) => 0 = m(x - x1) - y + y1 where x1 and y1 are for my test point.
By using that equation for the two test points (the player position and the pixel position on screen) I can test if the pixel is on the opposite side of the line using this test:
0 > (m * (x - x1) - y + y1)(m * (x - x2) - y + y2)
While it seems to work okay(ish), when I rotate the camera the lines from the walls seem to flail around in a controlled but somewhat unpredictable way. Where am I going wrong with my math? If you need code I can supply it, but the problem here really is just with the mathematics.
It seems like you are rotating the wall endpoints around the origin and then translating by the player position, instead of rotating them around the player's position by translating first and then rotating. Try something like this:
x = ((px - wx) * cos(a) - (py - wy) * sin(a)) and y = ((px - wx) * sin(a) + (py - wy) * cos(a))
I'm also not sure why you are subtracting the wall endpoints from the player position instead of vice versa, but the correctness of this might depend on other stuff in your code (depends if you want the vector from the wall to the player or the player to the wall).