I can get the bounding box of sheared or rotated rectangle using this formula
shearedW = Abs(Tan(shearX) * Height) + Width
shearedH = Abs(Tan(shearY) * Width) + Height
rotatedW = Abs(Cos(angle) * Width) + Abs(Sin(angle) * Height)
rotatedH = Abs(Sin(angle) * Width) + Abs(Cos(angle) * Height)
But how to combine those? I just need to know the width and height. The transformation is done by shearing then rotating the shape.
Just build affine matrix for combined transformation and apply it to vertices, then get differences for y- and x- coordinates.
Note that first pair of formulas is wrong - it gives additional width and heght. Full width:
shearedW = Width + Abs(Tan(shearX) * Height)