I have multiplied two numbers with imulq. It returns a signed number on rdx:rax. I want to compress this into the rax register. Basically, I am looking for the instruction that reverses cqo.
If the result fits in 64 bits, then rax already contains the 64-bit result and you don't need to do anything to get the result into the rax register.
However, if you only need a 64-bit result, then you should use the two-operand form of the instruction, imul rax, <src>, which is faster because it doesn’t have to compute the full 128-bit result.
If you want to handle error on overflow, then use jo <error-handler> after the multiply to check for a result that doesn't fit in 64 bits. This works with either the single-operand form of imul, which generates a 128-bit result, and with the two- or three-operand forms, which generate a 64-bit result. But if you intend to discard the upper 64 bits even when there's an overflow, then there's no reason to use that form of the instruction.