how to get argparse to read arguments from a file with an option rather than prefix

I would like to know how to use python's argparse module to read arguments both from the command line and possibly from text files. I know of argparse's fromfile_prefix_chars but that's not exactly what I want. I want the behavior, but I don't want the syntax. I want an interface that looks like this:

$ python myprogram.py --foo 1 -A somefile.txt --bar 2

When argparse sees -A, it should stop reading from sys.argv or whatever I give it, and call a function I write that will read somefile.text and return a list of arguments. When the file is exhausted it should resume parsing sys.argv or whatever. It's important that the processing of the arguments in the file happen in order (ie: -foo should be processed, then the arguments in the file, then -bar, so that the arguments in the file may override --foo, and --bar might override what's in the file).

Is such a thing possible? Can I write a custom function that pushes new arguments onto argparse's stack, or something to that effect?

Solution

You can solve this by using a custom argparse.Action that opens the file, parses the file contents and then adds the arguments then.

For example this would be a very simple action:

class LoadFromFile (argparse.Action):
    def __call__ (self, parser, namespace, values, option_string = None):
        with values as f:
            # parse arguments in the file and store them in the target namespace
            parser.parse_args(f.read().split(), namespace)

Which you can the use like this:

parser = argparse.ArgumentParser()
# other arguments
parser.add_argument('--file', type=open, action=LoadFromFile)
args = parser.parse_args()

The resulting namespace in args will then also contain any configuration that was also loaded from the file where the file contained arguments in the same syntax as on the command line (e.g. --foo 1 --bar 2).

If you need a more sophisticated parsing, you can also parse the in-file configuration separately first and then selectively choose which values should be taken over. For example, since the arguments are evalutated in the order they are specified, it might make sense to prevent the configurations in the file from overwriting values that have been explicitly specified ont the command line. This would allow using the configuration file for defaults:

def __call__ (self, parser, namespace, values, option_string=None):
    with values as f:
        contents = f.read()

    # parse arguments in the file and store them in a blank namespace
    data = parser.parse_args(contents.split(), namespace=None)
    for k, v in vars(data).items():
        # set arguments in the target namespace if they have not been set yet
        if getattr(namespace, k, None) is None:
            setattr(namespace, k, v)

Of course, you could also make the file reading a bit more complicated, for example read from JSON first.