Search code examples
assemblyfloating-pointx86-64nasmavx512

Unable to get correct rounding mode code for `vrndscalepd`

I am calling vrndscalepd to round a ZMM register. The register to round is:

{3.9304247359239284, 0.85804618270500566, 1.899940908279022,  
 1.5554455222287524, 9.1150061014624413,  9.3562881423859334,  
 1.3206387781690712, 9.0367010342260201}

I want to round each of these to two digits past the decimal, but none of the rounding mode codes I have tried do that:

vrndscalepd zmm9{k1}{z},zmm8,32 ; 00100000
{4, 0.75, 2, 1.5, 9, 9.25, 1.25, 9}

vrndscalepd zmm9{k1}{z},zmm8,33 ; 00100001
{3.75, 0.75, 1.75, 1.5, 9, 9.25, 1.25, 9}

vrndscalepd zmm9{k1}{z},zmm8,34 ; 00100010
{4, 1, 2, 1.75, 9.25, 9.5, 1.5, 9.25}

vrndscalepd zmm9{k1}{z},zmm8,35 ; 00100011
{3.75, 0.75, 1.75, 1.5, 9, 9.25, 1.25, 9}

vrndscalepd zmm9{k1}{z},zmm8,41 ; 00110001
{3.75, 0.75, 1.75, 1.5, 9, 9.25, 1.25, 9}

vrndscalepd zmm9{k1}{z},zmm8,42 ; 00110010
{4, 1, 2, 1.75, 9.25, 9.5, 1.5, 9.25}

None of those does what I want. Is there another rounding mode that I have missed, and that will round to two places past the decimal without rounding to the nearest even multiple of 0.25, on AVX512? Is there another instruction that will do what I want using ZMM registers?

Solution

  • No, AVX-512 uses binary floating point, not decimal.
    (https://en.wikipedia.org/wiki/Double-precision_floating-point_format)

    3.93 isn't exactly representable as a float or double so it's literally impossible for any sequence of instructions to create a bit-pattern that represents it. (https://www.h-schmidt.net/FloatConverter/IEEE754.html)

    You could of course do something that gets a value that's close, like 3.9300000667572021484375f, but it's not as simple as zeroing some low bits of the mantissa like vrndscaleps/pd does. That will necessarily produce a value that's representable as a fraction with a power-of-2 denominator (because all FP values are like that.) A "more round" binary FP value will have a smaller denominator; with 2 bits after the radix point, the denominator will be 4, so .0, .25, .5, or .75 are the only possibilites.

    The normal way to round to 2 decimal digits is I think to multiply by 100.0, round to nearest integer, multiply by 1.0/100. I don't know if there's anything more efficient / clever. (Note that 1.0/100 isn't exactly representable, so dividing by 100.0 could be more precise.)