When defining a global or static array in c++ its memory is not immediately reserved at the start of the programme but only once we write to the array. What I found surprising is, if we only write to a small part of the array it still does not reserve the entire memory. Consider the following small example which writes sparsely to the global array:
#include <cstdio>
#include <cstdlib>
#define MAX_SIZE 250000000
double global[MAX_SIZE];
int main(int argc, char** argv) {
if(argc<2) {
printf("usage: %s <step size>\n", argv[0]);
exit(EXIT_FAILURE);
}
size_t step_size=atoi(argv[1]);
for(size_t i=0; i<MAX_SIZE; i+=step_size) {
global[i]=(double) i;
}
printf("finished\n"); getchar();
return EXIT_SUCCESS;
}
Now executing this for different step sizes and looking at the output of top we get for example:
./a.out 1000000
./a.out 100000
./a.out 10000
./a.out 1000
./a.out 100
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15718 user 20 0 1918m 1868 728 S 0 0.0 0:00.00 a.out
15748 user 20 0 1918m 10m 728 S 0 0.1 0:00.00 a.out
15749 user 20 0 1918m 98m 728 S 1 0.8 0:00.04 a.out
15750 user 20 0 1918m 977m 728 S 0 8.1 0:00.39 a.out
15751 user 20 0 1918m 1.9g 728 S 23 15.9 0:00.80 a.out
The RES column indicates memory is only reserved in small blocks, which also means the array is unlikely to be contiguous in physical memory. Anyone got more insight on the lower level of things?
This also has the negative side effect that I can easily run many programmes where the sum of all VIRT exceeds the physical memory as long the sum of RES is below. However, as soon as they all write to the global arrays the system runs out of physical memory and programmes get sent sigkill or something.
Ideally I'd like to tell the compiler to reserve the memory of global and static variables at the start. Possible?
Edit
@Magnus: The lines are actually in the right order. :) Take the first line for example ./a.out 1000000 means I'm writing every 1 millionth entry in the array and so only 250 in total. This corresponds to a RES of only 1868k. In the last example ./a.out 100 every one hundreds entry is written and then the total memory is also physically allocated RES=VIRT=1.9g. From the numbers it appears that whenever an entry is written to the array something like a full 4k block is reserved on the physical memory.
@Nawaz: The array is contiguous in virtual address space but as I understand the OS might be lazy and only reserve physical memory when it's actually needed. Since this is done in little blocks and not the whole array at once how can it be guaranteed to be contiguous in physical memory?
@Nemo: Thanks for that, indeed when calling multiple instances of a.out which pause at the beginning and then write to the array I got oom-killer messages in /var/log/messages, and indeed your sysctrl command prevents me from starting too many a.out instances in the first place. Thanks!
Jun 1 17:49:16 localhost kernel: [32590.293421] a.out invoked oom-killer: gfp_mask=0x280da, order=0, oomkilladj=0
Jun 1 17:49:18 localhost kernel: [32592.110033] kded4 invoked oom-killer: gfp_mask=0x201da, order=0, oomkilladj=0
Jun 1 17:49:20 localhost kernel: [32594.718757] firefox invoked oom-killer: gfp_mask=0x201da, order=0, oomkilladj=0
The last two lines slightly worrying. :)
@doron: Thanks, great explanation, sorry can't upvote/select.
There are two things in play here viz. virtual memory and physical memory.
The virtual memory for for the static data, just like the instructions for your program are assigned before your program begins execution. By this I mean that the address, for your program is always defined.
The operating system might be lazy, however when it comes to loading both static data and program instructions into physical memory RAM. The way that this works is like this:
This is a total slight of hand which the operating system is allowed to do simply because it is totally undetectable to the running process. Unless of course we are short of memory.