with open(file, 'a+') as f is not appending

Question

I have a function that gets called repetitively and writes error data to a log file via subprocess, however, whenever new data is written to the file, the old data is cleared. So it is not appending any new data with the old data.

I'm doing following:

error_file = '\\\\some\\unc\\path\\error.log'

class Node:  
    def __init__(self, path, rev):
        self.path = path
        self.rev = rev

    def __hash__(self):
        return hash((self.path, self.rev))

    def __eq__(self, node):
        return (self.path, self.rev) == (node.path, node.rev)

    def __ne__(self, node):
        return not(self == node)

def get_excluded_nodes(excludes_dir):
    nodes = list()
    for root, subdirs, files in os.walk(os.path.dirname(excludes_dir)):
       if 'flagged' in files:
          with open(os.path.join(root, 'flagged')) as f:
             for line in f.readlines():
                 try:
                     comps = line.split(' -a')
                     path = comps[0].strip()
                     rev = comps[1].split(':')[0].strip()
                     Nodes.append(Node(path,rev))
                 except:
                     pass
    return nodes

def export_node(node, path=archive_dir):
    with open(error_file, 'a') as f:
        try:
            comps = node.path.split('/')
            if '.' in comps[len(comps)-1]:
                os.makedirs(os.path.join(archive_dir, '/'.join(comps[:-1])))
            else:
                os.makedirs(os.path.join(archive_dir, node.path))

            subprocess.call(['svn', 'export', os.path.join(some_path, node.path), another_path)], stderr=f)
        except:
            pass


def remove_duplicate_nodes(nodes):
   return set(nodes)

if __name__ == '__main__':
    all_nodes = get_excluded_nodes(os.path.realpath(__file__))
    nodes = remove_duplicate_nodes(all_nodes)
    for node in nodes:
       export_node(node)

Why doesn't this work?

Answer 1

That's still not an MVCE: it isn't minimal (what's Node for, how does it affect appending to a file?) and it isn't complete (where is get_excluded_nodes?). Since it isn't complete, it can't be verifiable either.

This is an MVCE with the minimal code that ought to do the same thing as your example, and it works fine.

import subprocess

def test(filename):
    with open(filename, 'a') as f:
        subprocess.call(['bash', '-c', 'echo $$ >&2'], stderr=f)

if __name__=='__main__':
    for _ in range(2):
        test('stderr.log')

which does exactly what you wanted:

$ python stderr.py 
$ cat stderr.log 
344
345

$ python stderr.py 
$ cat stderr.log 
344
345
366
367

---

edit I see you're on Windows, so strace is probably out. Bad luck, you'll just have to write an actual MVCE instead.

---

Maybe run your original (real, sort-of-working) script under strace, and see what's different. For reference, this script shows:

$ strace -f -e trace=open,dup2,lseek,write python stderr.py

open("stderr.log", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3
lseek(3, 0, SEEK_END)                   = 28
strace: Process 567 attached
...
[pid   567] dup2(3, 2)                  = 2 # stderr=f
... libc, locale stuff ...
[pid   567] dup2(2, 1)                  = 1 # >&2
[pid   567] write(1, "567\n", 4)        = 4