I have a few Python files that I want to execute sequentially in order to get an output and now would like to automate this process. So I would like to have a parent script from which I can execute all my child scrips in the right order. Also, I would like to execute one of the files twice but with two different date variables and would like to store the outputs in two different folders. How do I create such a parent script in Python?
For example I want to execute file1.py first and the date (a variable in file1.py) should be date1 and the output should be stored in dir1 and then I want to execute file1.py again but this time the date should be date2 and the output should be in dir2. And the I want to execute file2.py.
How would I do this?
You can easily run python scripts from another python scripts using subprocesses. Something like:
import subprocess
subprocess.Popen("script2.py some_argument")
Problem with using subprocesses - it's quite annoying to get results from it (you can print results from the script and then get them in the parent file, but still).
Better solution - save middle results in some database (like simple SQLite file), so you use your main script to initiate child scripts, but get arguments from the database and write child script results to the database too. It's quite easy and could solve your problems (https://docs.python.org/3/library/sqlite3.html).
For example, to save some variable in the SQLite database, all you need is:
import sqlite3
conn = sqlite3.connect("mydatabase.db")
cursor = conn.cursor()
# Create table (no need if it was created before)
cursor.execute("""CREATE TABLE example_table
(variable_name, variable_value)
""")
# Save changes
conn.commit()
# Insert data
cursor.execute("""INSERT INTO example_table
VALUES ('some_variable', 'some_value')"""
# Save changes
conn.commit()