GNU MP: Cannot allocate memory (size=4294959136)

GMP cannot seem to allocate memory larger than this amount, despite being compiled in 64bit. I have a simple program that you can test this on, and whilst running it only seems to use up to 1500mb memory and no more. This should be able to calculate the number for even 32bit applications, however since it is 64bit, it seems very odd that it cannot calculate the number. Here is a simple program that runs into this issue

#include "gmp.h"

int main()
{
    unsigned int n = 500000000;
    mpz_t output;

    mpz_init(output);
    mpz_fac_ui(output, n);
    mpz_clear(output);

    return 0;
}

GMP 6.2.1 installed from MSYS2, Windows 10 64bit, 20h2, GCC

Solution

This situation is typical for using a primarily UNIX-oriented library on Windows: YMMV.

Sure enough, a quick inspection reveals extensive use of the long data type (example). It's unsurprising that it doesn't work on Win32, where long is 32-bit. Win32 uses the LLP64 model, and MinGW follows that (source).

Even the error reporting code that prints GNU MP: Cannot allocate memory is buggy:

   fprintf (stderr, "GNU MP: Cannot allocate memory (size=%lu)\n", (long) size);

It's actually printing a 64-bit size_t as a 32-bit long. So you don't see the actual size value.

If we fix that:

   fprintf (stderr, "GNU MP: Cannot allocate memory (size=%zu)\n", size);

Now that prints the true size it tried to malloc:

GNU MP: Cannot allocate memory (size=18446744073709250080)

So it's overflowing and/or wrapping around somewhere (probably another long variable). Report it to the libgmp project. For more exposure can also report to MSYS2.