How can I avoid calling Python interpreter many times from a Bash script?

I'm working on a project involving two main components, one with a Bash command-line interface and one with a Python API. For the purposes of this question, assume that neither of these components can be modified to use a different language for the interface.

I now need to write some scripts that use features of both components. I have two possible approaches in mind: (a) write Bash scripts invoking the Python interpreter for interfacing with the Python component, or, (b) write Python scripts using calls to os.system or something similar, which allows one to run Bash commands.

The problem with writing Bash scripts is that I often need to call a function from the Python component inside a loop, something like this:

while read -r line ; do
    echo "Now handling the next line of the file."
    python -c "from my_module import *; my_function('${line}')"
    # some bash commands
done < some_file.txt

I don't like the overhead of calling the Python interpreter so many times, and also doing this same import so often. Is there any way to start up the interpreter and import the module once, and then dynamically call other functions in that context? Keep in mind that I will be using other Bash commands between Python function calls.

The problem with writing Python scripts is that any time we need to access the component with the command-line interface, we have to use calls to os.system or something similar. Some of the scripts only include calls to the command-line interface, meaning I would have a Python script with every line using os.system. I don't like this either.

These scripts, whether written in Python or Bash, will be used for unit testing, so readability, maintainability, and extensibility are higher priorities than pure speed. Furthermore, all of them should be written in the same language for consistency's sake.

I'm hoping that someone can guide me to the most elegant solution to this problem. Thanks in advance!

Solution

You can do this in bash:

mkfifo in out
python -ic 'from my_module import *'  <in >out & exec 3> in 4< out
while read -r line ; do
    echo "Now handling the next line of the file."
    echo "my_function('$line')" >&3
    # read the result of my_function
    read r <&4; echo $r
    # some bash commands
done < some_file.txt
exec 3>&-
exec 4<&-
rm in out

and you can send commands to python while continue executing bash commands.