Search code examples
perlmemoryforkcopy-on-write

Avoid read-only forked() RAM allocation on exit in Perl


In Perl, I generate a huge read-only data-structure once, then fork().

This is to take advantage of COW on RSS pages when forking. It works really well, but when a child process exits, it allocates all the RAM from itelf just prior dying.

Is there a way to avoid this useless allocation ?

Here is sample Perl code that shows the issue.

#! /usr/bin/perl
my $a = [];
# Allocate 100 MiB
for my $i (1 .. 100000) {
        push @$a, "x" x 1024;
}
# Fork 10 other process
for my $j (1 .. 10) {
        last unless fork();
}
# Sleep for a while to be able to see the RSS
sleep(5);

In the sample vmstat output, we can see that it first allocates only 100MiB, then after the 1rst sleep it allocates the whole for a short while, and then releases all of it.

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 1329660  80596  86936    0    0    21    18  160   25  0  0 100  0  0
 1  0      0 1328048  80596  86936    0    0     0     0 1013   44  0  0 100  0  0
 0  0      0 1223888  80596  86936    0    0     0     0 1028   76 11  5 84  0  0
 0  0      0 1223888  80596  86936    0    0     0     0 1010   40  0  0 100  0  0
 0  0      0 1223888  80596  86936    0    0     0     0 1026   54  0  0 100  0  0
 0  0      0 1223888  80596  86936    0    0     0     0 1006   39  0  0 100  0  0
13  0      0 741156  80596  86936    0    0     0     0 1012   66 13 58 28  0  0
 0  0      0 1329288  80596  86936    0    0     0     0 1032   60  0  0 100  0  0

Note: it seems it isn't a Perl version specific issue. As I tested 5.8.8, 5.10.1 & 5.14.2 and they all do exhibit this behavior.

Update:

As @choroba asked in comments, I also tried to undef the data-structure, but it seems that it triggers the memory-touching as the RAM is then allocated.

You can add the following snippet at the end of the first script.

# Unallocate $a
undef $a;
# Sleep for a while to be able to see the RSS
sleep(5);

Solution

  • Actually, as I found out myself, this behavior is a feature, and the answer lies in the Perl doc:

    The exit() function does not always exit immediately. Likewise any object destructors that need to be called are called before the real exit. If this is a problem, you can call POSIX::_exit($status) to avoid END and destructor processing.

    And indeed, adding it at the end of the original code sample does avoid the behavior.

    # XXX - To be added just before ending the process
    # Use POSIX::_exit($status) to end without allocating copy-on-write RAM
    use POSIX;
    POSIX::_exit(0);
    

    Note: for this to work, the child has to exit also before the data-structure goes out of scope.