Search code examples
diffpatchlarge-files

memory exhausted : for large files using diff


I am trying to create a patch using two large size folders (~7GB).

Here is how I'm doing it :

$ diff -Naurbw . ../other-folder > file.patch

But maybe due to file sizes, patch is not getting created and giving an error:

diff: memory exhausted

I tried making space more than 15 GB but still the issue persists. Could someone help me out with the flags that I should use?


Solution

  • Recently I came across this too when I needed to diff two large files (>5Gb each).

    I tried to use 'diff' with different options, but even the --speed-large-files had no effect. Other methods like splitting the files into smaller ones, using xdelta or sorting the files as per this suggestion didn't help either. I even got my hands around a very powerful VM (> 72Gb RAM), but still got this memory exhausted error.

    I finally got to work by adding the following parameter to sysctl.conf (sudo vim /etc/sysctl.conf):

    vm.overcommit_memory=1
    

    vm.overcommit_memory has three values (0,1,2) and sets the kernel virtual memory accounting mode. From the proc(5) man page:

    0: heuristic overcommit (this is the default)
    1: always overcommit, never check
    2: always check, never overcommit

    To make sure that the parameter is indeed applied you can run

    sudo sysctl -p
    

    Don't forget to change this parameter back when you finish!