why does dynamic memory allocation fail after 600MB?

Question

i implemented a bloom filter(bit table) using three dimension char array it works well until it reaches at a point where it can no more allocate memory and gives a bad_alloc message. It gives me this error on the next expand request after allocating 600MB.

The bloom filter(the array) is expected to grow as big as 8 to 10GB.

Here is the code i used to allocate(expand) the bit table.

unsigned char ***bit_table_=0;
unsigned int ROWS_old=5;
unsigned int EXPND_SIZE=5;


void expand_bit_table()
     {
         FILE *temp;
         temp=fopen("chunk_temp","w+b");
         //copy old content
         for(int i=0;i<ROWS_old;++i)
             for(int j=0;j<ROWS;++j)
                 fwrite(bit_table_[i][j],COLUMNS,1,temp);
         fclose(temp);
         //delete old table
         chunk_delete_bit_table();
         //create expanded bit table ==> add EXP_SIZE more rows
         bit_table_=new unsigned char**[ROWS_old+EXPND_SIZE];
         for(int i=0;i<ROWS_old+EXPND_SIZE;++i)
            {
                bit_table_[i]=new unsigned char*[ROWS];
                for(int k=0;k<ROWS;++k)
                    bit_table_[i][k]=new unsigned char[COLUMNS];
            }
         //copy back old content

          temp=fopen("chunk_temp","r+b");
         for(int i=0;i<ROWS_old;++i)
         {
            fread(bit_table_[i],COLUMNS*ROWS,1,temp);
         }
          fclose(temp);
         //set remaining content of bit_table_to 0
         for(int i=ROWS_old;i<ROWS_old+EXPND_SIZE;++i)
             for(int j=0;j<ROWS;++j)
                 for(int k=0;k<COLUMNS;++k)
                     bit_table_[i][j][k]=0;

         ROWS_old+=EXPND_SIZE;
     }

What is the maximum allowable size for an array and if this is not the issue what can i do about it.

EDIT: It is developed using a 32 bit platform.

It is run on 64 bit platform(server) with 8GB RAM.

Answer 1

A 32-bit program must allocate memory from the virtual memory address space. Which stores chunks of code and data, memory is allocated from the holes between them. Yes, the maximum you can hope for is around 650 megabytes, the largest available hole. That goes rapidly down from there. You can solve it by making your data structure smarter, like a tree or list instead of one giant array.

You can get more insight in the virtual memory map of your process with the SysInternals' VMMap utility. You might be able to change the base address of a DLL so it doesn't sit plumb in the middle of an otherwise empty region of the address space. Odds that you'll get much beyond 650 MB are however poor.

There's a lot more breathing room on a 64-bit operating system, a 32-bit process has a 4 gigabyte address space since the operating system components run in 64-bit mode. You have to use the /LARGEADDRESSAWARE linker option to allow the process to use it all. Still, that only works on a 64-bit OS, your program is still likely to bomb on a 32-bit OS. When you really need that much VM, the simplest approach is to just make a 64-bit OS a prerequisite and build your program targeting x64.