There are 3 points in 3D space. There are 2 orthogonal coordinate systems with the same origin. I know coordinates of those 3 points in both coordinate systems. Given a new point with its coordinates in the first coordinate system, how can I find its coordinates in the second coordinate system?
I think it's possible to get a rotation matrix using given points which does this, but I did not succeed doing this.
You can do it using matrix inverses. Three matrix-vector multiplications (e.g. transforming three 3D vectors by a 3x3 matrix) is equivalent to multiplying two 3x3 matrices together.
So, you can put your first set of points in one matrix, call it A:
0 0 1 < vector 1
0 1 0 < vector 2
2 0 0 < vector 3
Then put your second set of points in a second matrix, call it C. As an example, imagine a transform that scales by a factor of 2 around the origin and flips the Y and Z axes:
0 2 0 < vector 1
0 0 2 < vector 2
4 0 0 < vector 3
So, if A x B = C, we need to find the matrix B, which we can find by finding the A-1:
Inverse of A:
0 0 0.5
0 1 0
1 0 0
The multiply A-1 x C (in that order):
2 0 0
0 0 2
0 2 0
This is a transform matrix B that you can apply to new points. Dot-product multiply the vector by the first column to get the transformed X, second column to get the transformed Y, etc.