Search code examples
pythonmultithreadingwith-statementfile-lockingfilelock

Python filelock module behavior using `with:` statement

I was just wondering about the specifics of python's module filelock and its behavior in a few circumstances.

First, how exactly is the with: statement handled by threads. If multiple threads call with: is it locked on a thread by thread basis? Is it also possible that two threads could acquire the lock at the same time?

Second, when I use with: do I have to clear the lock after its use? Is the lock automatically cleared after the with: statement is done?

Third, I have an instance in my code where I believe a file must be created then locked immediately. Currently I am using this:

channel_file = open(os.path.join('channels', username), 'w+')
with filelock.FileLock(os.path.join('channels', username)):
  channel_file.write(json.dumps({'rate': reobj.group(1),'time': reobj.group(2)}))

If there were a possibility that another thread could read the file since the time it was created, would this protect against that?

This also brings up a forth point. Does filelock lock read access while using with:?

Solution

    1. The FileLock maintains a lock counter, which is shared across all threads in the process, and is protected behind a thread-wise lock. Every call to acquire() will increase the lock counter, and additionally will obtain the OS-level file lock when the counter was zero. Similarly, every call to release() will decrease the lock counter and unlock the file when the counter reaches zero.

      Therefore, if two threads acquires the lock at the same time, the file will be locked once by this process in the OS level, and the lock counter will be increased by 2. The two threads will not block each other.

    2. The point of with: is to automatically acquire and release the lock after its scope exits. See What is the python "with" statement designed for?.

    3. The file lock is used to protect against file access outside of the current process. It is not used for thread-wise locking. Use a regular threading.Lock for thread-wise locking.

      # in __main__ or somewhere before we start the threads.
channel_lock = threading.Lock()

# in the worker thread
with channel_lock:
    with open(...) as channel_file:
        channel_file.write(...)

    For implementation detail you could refer to the source code of py-filelock.