I have two circles, each defined by a centre point c = (x,y,z) a radius r and a normal vector n = (x,y,z) (perpendicular to the circle's plane).
How to calculate the intersection points of two such circles (not necessarily being in the same plane)?
If there is no intersections, but those circles are not parallel, how to get the point, where those circles would intersect if you increased or decreased the radius until they touch?
If n1 x n2 = 0 then normal vectors are (anti)collinear, and planes are parallel.
They are the same if Dot(c1-c2, n1) = 0, otherwise circle intersection is impossible.
Case of two circles in the same plane. I assume that r2>=r1
cdiff = (c1-c2)
cdifflen = cdiff.Length
if cdifflen > r1 + r2 then no intersection
if cdifflen = r1 + r2 then intersection exists in one point
p = (c1 * r2 + c2 * r1) / (r1 + r2)
if cdifflen < r2 - r1 then no intersection
if cdifflen = r2 - r1 then intersection exists in one point
p = (c1 - c2) * r2 /(r2 - r1)
otherwise there are two intersection points
cdiffnorm = cdiff.Normalized //unit vector
cdiffperp = cdiffnorm * n1.Normalized
q = cdifflen^2 + r2^2 - r1^2
dx = 1/2 * q / cdifflen
dy = 1/2 * Sqrt(4 * cdifflen^2 * r2^2 - q^2) / cdifflen
p1,2 = c1 + cdiffnorm * dx +/- cdiffperp * dy
Some formulas are explained here
If planes are not parallel, their intersection is a line with direction vector
dl = n1 x n2
Base point of this line might be found using Geometric Tools Engine (file GteIntrPlane3Plane3.h). Then check for intersection of both circles with this line and compare intersection points if exist.