I have a transformation matrix (rotation + translation) which was constructed in a right-handed coordinate system where X is right, Y is up and Z is into the screen.
If I want to apply this same transformation but in a left-handed coordinate system where X is right, Y is down and Z is out of the screen (is this left handed?) what alterations should I make to the existing transformation matrix.
Thanks
a right-handed coordinate system where X is right, Y is up and Z is into the screen
Then it's a left-handed coordinate system: X is the thumb, Y is the forefinger, Z is the middle-finger. Your right hand should not be able to do what you describe without breaking a few bones :-p
a left-handed coordinate system where X is right, Y is down and Z is out of the screen
This one is indeed a left-handed coordinate system too.
As for the transformation linking these two, this is a simple 180° rotation around the X axis (beware of the wrist sprain while trying this one), which expresses as the following 3x3 matrix:
(1 0 0)
(0 -1 0)
(0 0 -1)
You can pre-multiply your transformation with this one!
Hope this helps!