I have 4 points in space A(x,y,z), B(x,y,z), C(x,y,z) and D(x,y,z). How can I check if these points are the corner points of a rectangle?
You must first determine whether or not the points are all coplanar, since a rectangle is a 2D geometric object, but your points are in 3-space. You can determine they are coplanar by comparing cross products as in:
V1 = (B-A)×(B-C)
V2 = (C-A)×(C-D)
This will give you two vectors which, if A, B, C, and D are coplanar are linearly dependent. By considering what Wolfram has to say on vector dependence, we can test the vectors for linear dependence by using
C = (V1∙V1)(V2∙V2) - (V1∙V2)(V2∙V1)
If C is 0 then the vectors V1 and V2 are linearly dependent and all the points are coplanar.
Next compute the distances between each pair of points. There should be a total of 6 such distances.
D1 = |A-B|
D2 = |A-C|
D3 = |A-D|
D4 = |B-C|
D5 = |B-D|
D6 = |C-D|
Assuming none of these distances are 0, these points form a rectangle if and only if the vertices are coplanar (already verified) and these lengths can be grouped into three pairs where elements of each pair have the same length. If the figure is a square, two sets of the pairs will have be the same length and will be shorter than the remaining pair.
Update: Reading this again, I realize the the above could define a parallelogram, so an additional check is required to check that the square of the longest distance is equal to the sum of the squares of the two shorter distances. Only then will the parallelogram also be a rectangle.
Keep in mind all of this is assuming infinite precision and within a strictly mathematical construct. If you're planning to code this up, you will need to account for rounding and accept a degree of imprecision that's not really a player when speaking in purely mathematical terms.