How to search the output of a script and save a subsection to a file?

I have a command nipype.interface.afni.Warp which gives me the following python terminal output:

190920-12:22:00,333 nipype.interface INFO:
         stderr 2019-09-20T12:22:00.333467:++ 3dWarp: AFNI version=AFNI_19.2.21 (Aug 29 2019) [64-bit]
190920-12:22:00,334 nipype.interface INFO:
         stderr 2019-09-20T12:22:00.334117:++ Authored by: RW Cox
190920-12:22:00,365 nipype.interface INFO:
         stderr 2019-09-20T12:22:00.365105:++ Using minimum spacing of 1.000000 mm for new grid spacing
190920-12:22:03,252 nipype.interface INFO:
         stderr 2019-09-20T12:22:03.252756:++ Output dataset /media/sf_Ubuntu_files/dicomtest/warp_test.nii.gz
190920-12:22:03,253 nipype.interface INFO:
         stdout 2019-09-20T12:22:03.253083:# mat44 Obliquity Transformation ::
190920-12:22:03,253 nipype.interface INFO:
         stdout 2019-09-20T12:22:03.253083:      1.000000     -0.000000      0.000000       0.000000
190920-12:22:03,253 nipype.interface INFO:
         stdout 2019-09-20T12:22:03.253083:      0.000000      0.999592     -0.028568      -1.842994
190920-12:22:03,253 nipype.interface INFO:
         stdout 2019-09-20T12:22:03.253083:     -0.000000      0.028568      0.999592       3.788057

I want to capture the matrix at the bottom under the "# mat44 Obliquity Transformation ::" line and write that to a file. I've already done this in bash, which looks like this:

3dWarp -flags_and_stuff | \grep  -A 4 '# mat44 Obliquity Transformation ::'  > $filename.1D

However I want to write the above bash command using python instead.

Following the steps of this blog post I tried this:

command = ['python3' ,"nipype.interfaces.afni.Warp('more stuff').run()"]
my_env = os.environ.copy()
my_env["PATH"] = "/usr/sbin:/sbin:" + my_env["PATH"]
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, env=my_env)

But when I type p.communicate() I get:

>>> p.communicate()
(b"python3: can't open file 'nipype.interfaces.afni.Warp(<stuff>).run()': [Errno 2] No such file or directory\n", None)

How do I make this work in python? Or is it better to execute in bash? The script I'm writing will use this line thousands of times so whatever the fastest (which I'm assuming also means the most "pythonic") method is.

Solution

I think your command-line arguments to python3 are incorrect in Python's Popen call. With

command = ['python3' ,"nipype.interfaces.afni.Warp('more stuff').run()"]

the launched python3 process thinks the first argument is a file to execute, but you intended a snippet of Python code to run.

Change your command declaration to the following:

command = ['python3' ,'-c', "nipype.interfaces.afni.Warp('more stuff').run()"]

This should make the spawned python3 process interpret that argument as commands to execute, rather than a filename.

This is of course assuming that you want to do this in the first place. If you are launching a sub-process of Python in Python, why not just run the nipype.interfaces.afni.Warp('more stuff').run() in your script without using Popen?