Do I need to handle atomicity myself for LocalTargets?

Question

The Luigi documentation describes potential problems regarding atomicity of writing to targets. They say that this is a common pitfall and that it should be handled using temporary directories that are moved to the target location in the end.

Do I also need to handle this myself if my target is a single-file LocalTarget?

The luigi.local_target module contains an atomic_file class, which seems to indicate that this is done automatically, and the design goals include “atomic file system operations”. But I couldn't find any documentation stating that LocalTarget is safe to be used that way.

Answer 1

If you see the code of Luigi luigi/local_target.py class LocalTarget:

def open(self, mode='r'):
    rwmode = mode.replace('b', '').replace('t', '')
    if rwmode == 'w':
        self.makedirs()
        return self.format.pipe_writer(atomic_file(self.path))

Luigi create a temporary file and when it's closed, changes to the final name.