Search code examples
mongodbdockercluster-computinglustre

MongoDB "Unable to establish lock" error on mounted drive using Docker and Lustre


I am trying to use Docker to containerize a MongoDB instance using a drive mounted on host.

Using the mongo:latest image:

[user@dcos-master ~]$ docker run -d --name mongo -v /local/cluster/drive:/data/db mongo:latest

But constantly fails with:

exception in initAndListen: 98 Unable to lock file: /data/db/mongod.lock Function not implemented. Is a mongod instance already running?, terminating

Considered it a permission problem so I have attempted opening up the permissions to 777 on the host directory and also just running the mongod instance as root and I get the same problem.

There definitely is not another mongod instance running and pointing at the same data directory.

Lastly, something worth pointing out: The only common denominator seems to be the filesystem. Mongod consistently fails on Lustre drives but not on any other filesystem. If I the source directory is on an NFS drive or anything else, it runs great.


Solution

  • https://anilmaurya.wordpress.com/2015/05/06/abaqus-with-lustre-file-system/

    By default lustre doesn't have the the flock feature [1], you have to add the "flock" option when mounting [2] the filesystem to make it work [3]

    [1]

    $ touch temp
    $ strace -e flock perl -e 'open(FH, "+< temp"); flock(FH, 2) or exit 1; sleep 1; close(FH);'
    flock(3, LOCK_EX)                       = -1 ENOSYS (Function not implemented)
    +++ exited with 1 +++
    

    [2]

    You have to umount and mount the filesystem as lustre doesn't support the -o remount option:

    $ mount | grep /data
    mgs@tcp:/data on /data type lustre (rw,lazystatfs)
    $ mount -o remount,flock /data
    $ mount | grep /data
    mgs@tcp:/data on /data type lustre (rw,lazystatfs)
    
    # umount /data
    # mount -t lustre -o rw,flock mgs@tcp:/data /data
    $ mount | grep /data
    mgs@tcp:/data on /data type lustre (rw,flock,lazystatfs)
    

    [3]

    $ strace -e flock perl -e 'open(FH, "+< temp"); flock(FH, 2) or exit 1; sleep 1; close(FH);'
    flock(3, LOCK_EX)                       = 0
    +++ exited with 0 +++