Lets assume we have a 2D space (to simplify situation), and layer S and layer C, where C is sublayer of S. The conversion process must affect bounds, position of C, transform of C, sublayersTransform of S, anchorPoint of C. My guess was the next:
CGAffineTransform transformToChild(CALayer *S, CALayer *C) {
CGFloat txa = - C.bounds.origin.x - C.bounds.size.width * C.anchorPoint.x;
CGFloat tya = - C.bounds.origin.y - C.bounds.size.height * C.anchorPoint.y;
CGFloat txb = C.position.x;
CGFloat tyb = C.position.y;
CGAffineTransform sublayerTransform = CATransform3DGetAffineTransform(S.sublayerTransform);
CGAffineTransform fromS = CGAffineTransformTranslate(sublayerTransform, txb, tyb);
fromS = CGAffineTransformConcat(fromS, C.affineTransform);
fromS = CGAffineTransformTranslate(fromS, txa, tya);
return fromS;
}
But this is not working when transform of the child layer is not identity (e.g. in case of rotation to M_PI_2 angle). Whole code with layers:
CALayer *l1 = [CALayer new];
l1.frame = CGRectMake(-40, -40, 80, 80);
l1.bounds = CGRectMake(40, 40, 80, 80);
CALayer *l2 = [CALayer new];
l2.frame = CGRectMake(50, 40, 20, 20);
l2.bounds = CGRectMake(40, 40, 20, 20);
CGAffineTransform t2 = CGAffineTransformMakeRotation(M_PI / 2);
l2.affineTransform = t2;
[l1 addSublayer:l2];
CGAffineTransform toL2 = transformToChild(l1, l2);
CGPoint p = CGPointApplyAffineTransform(CGPointMake(70, 50), toL2);
NSLog(@"Custom Point %@", [NSValue valueWithCGPoint:p]);
p = [l1 convertPoint:CGPointMake(70, 50) toLayer:l2];
NSLog(@"CoreAnimation Point %@", [NSValue valueWithCGPoint:p]);
Comparison to system results:
Custom Point NSPoint: {-50, 80}
CoreAnimation Point NSPoint: {50, 40}
There's an old mailing list thread with some details about this here:
those messages are quite old, so e.g. they don't include the effects of the geometryFlipped property, which was added more recently, but that would just add another term onto the merged matrix.