Better way to "copy only the comments from one file" and "prepend it into another file" using python

Basically I want to copy comments from one file and add it to the another data.

The file 'data_with_comments.txt' can be obtained from pastebin: http://pastebin.com/Tixij2yG

And it looks like this:

# coating file for detector A/R
# column 1 is the angle of incidence (degrees)
# column 2 is the wavelength (microns)
# column 3 is the transmission probability
# column 4 is the reflection probability
      14.2000     0.300000  8.00000e-05     0.999920
      14.2000     0.301000  4.00000e-05     0.999960
      14.2000     0.302000  2.00000e-05     0.999980
      14.2000     0.303000  2.00000e-05     0.999980
      14.2000     0.304000  2.00000e-05     0.999980
      14.2000     0.305000  3.00000e-05     0.999970
      14.2000     0.306000  5.00000e-05     0.999950

Now, i have another datafile 'test.txt' which looks like this:

300.0 1.53345164121e-32
300.1 1.53345164121e-32
300.2 1.53345164121e-32
300.3 1.53345164121e-32
300.4 1.53345164121e-32
300.5 1.53345164121e-32

Required output:

# coating file for detector A/R
# column 1 is the angle of incidence (degrees)
# column 2 is the wavelength (microns)
# column 3 is the transmission probability
# column 4 is the reflection probability
300.0 1.53345164121e-32
300.1 1.53345164121e-32
300.2 1.53345164121e-32
300.3 1.53345164121e-32
300.4 1.53345164121e-32

One way to do this is:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# Author    : Bhishan Poudel
# Date      : Jun 18, 2016


# Imports
from __future__ import print_function
import fileinput


# read in comments from the file
infile = 'data_with_comments.txt'
comments = []
with open(infile, 'r') as fi:
    for line in fi.readlines():
        if line.startswith('#'):
            comments.append(line)

# reverse the list
comments = comments[::-1]
print(comments[0])
#==============================================================================


# preprepend a list to a file
filename = 'test.txt'

for i in range(len(comments)):
    with file(filename, 'r') as original: data = original.read()
    with file(filename, 'w') as modified: modified.write(comments[i] + data)

In this method we have to open the file many times and it is not efficient when the data file is very large.

Is there any better way of doing this?

Solution

You already have a great answer using a temporary directory but it is also common to just create a temporary file in the same directory as the target file. On systems where tmp is a separate mount point, you avoid an additional copy of the data when renaming the temporary file. Notice that there is no intermediate list of comments which is significant if the comment list is large.

import os
import shutil

infile = 'data_with_comments.txt'
filename = 'test.txt'

tmpfile = filename + '.tmp'

try:
    # write wanted data to tempfile
    with open(tmpfile, 'w') as out_fp:
        # prepend comments from infle
        with open(infile) as in_fp:
            out_fp.writelines(filter(lambda l: l.startswith('#'), in_fp))
        # then add filename
        with open(filename) as in2_fp:
            shutil.copyfileobj(in2_fp, out_fp)
    # get rid of original data
    os.remove(filename)
    # replace with new data
    os.rename(tmpfile, filename)
finally:
    # cleanup on error
    if os.path.exists(tmpfile):
        os.remove(tmpfile)