Python argparse with optional nargs ath the end of multiple subparsers

I'm trying to achieve the following command definition with argparse, but I can't seem to figure it out: script.py {scan,list} ... [targets [targets...]]

I've gone through the complete documentation and checked multiple different questions which were somewhat related, however, I can't find a resource which seems to address the specific way I want to implement it.

What I want is two different subparsers (scan and list), which both have a shared OPTIONAL and POSITIONAL argument as the LAST ARGUMENT (with nargs=*).

These are the approaches I attempted so far, but as you'll see, each of them has a different issue with them.

Using a parent parser on the subparsers. This works, however, it has the following issues:

The "targets" argument is not shown in the main help section of the script. (so when running script.py -h, I get [-h] {scan,list} ... instead of [-h] {scan,list} ... [targets [targets...]]
The proxy argument will be placed AFTER the targets argument. I want "targets" to be at the end at all times.

    parser = argparse.ArgumentParser()
    parent_parser = argparse.ArgumentParser(add_help=False)
    parent_parser.add_argument('targets', type=str, nargs='*', default="ALL")

    subparsers = parser.add_subparsers()

    parser_scan = subparsers.add_parser('scan', parents = [parent_parser])
    parser_scan.add_argument('proxy', type=str)

    parser_list = subparsers.add_parser('list', parents = [parent_parser])

Adding it manually to all parsers separately:

This works as well, but the "targets" argument is again not shown in the main help section of the script. (so when running script.py -h, I get [-h] {scan,list} ... instead of [-h] {scan,list} ... [targets [targets...]]
The help section can be fixed by uncommenting the 3rd line, however, then the targets argument of the subparser is overwritten by the main parser. As a result args.targets will always return the default value "ALL", regardless of whether you pass targets to the script or not...

    parser = argparse.ArgumentParser()
    subparsers = parser.add_subparsers()
    # parser.add_argument('targets', type=str, nargs='*', default="ALL")

    parser_scan = subparsers.add_parser('scan')
    parser_scan.add_argument('proxy', type=str)
    parser_scan.add_argument('targets', type=str, nargs='*', default="ALL")

    parser_list = subparsers.add_parser('list')
    parser_list.add_argument('targets', type=str, nargs='*', default="ALL")

Only adding the argument to the main parser:

In this case, the help section of the main script correctly shows [-h] {scan,list} ... [targets [targets...]]
However, when running the script like this: script.py list Target1, I get the error message:

usage: target-utils.py [-h] {scan,list} ... [targets [targets ...]]
target-utils.py: error: unrecognized arguments: Target1

    parser = argparse.ArgumentParser()
    subparsers = parser.add_subparsers()
    parser.add_argument('targets', type=str, nargs='*', default="ALL")

    parser_scan = subparsers.add_parser('scan')
    parser_scan.add_argument('proxy', type=str)

    parser_list = subparsers.add_parser('list')

Solution

Subparsers is a relatively simple, and modular, addition to the basic parsing. Most of the work is done by a custom Action subclass. So there isn't a lot of flexibility, in setup, parsing, or help, beyond what's documented.

To clarify how subparsers works, and expand on my comments, let's define a parser:

In [46]: parser = argparse.ArgumentParser()
    ...: parser.add_argument('--foo')
    ...: parser.add_argument('bar')
    ...: sp = parser.add_subparsers(dest='cmd')
    ...: sp1 = sp.add_parser('test')
    ...: sp1.add_argument('xxx');

In [47]: parser.print_help()
usage: ipykernel_launcher.py [-h] [--foo FOO] bar {test} ...

positional arguments:
  bar
  {test}

optional arguments:
  -h, --help  show this help message and exit
  --foo FOO

In [48]: sp1.print_help()
usage: ipykernel_launcher.py bar test [-h] xxx

positional arguments:
  xxx

optional arguments:
  -h, --help  show this help message and exit

Here I define all arguments for the main, followed by the subparsers. The main usage attempts to show that test gets all trailing strings. The main parser does not return to parsing after passing the task to the subparser.

I could add a positional to the main after setting up the subparsers, and that will be reflected in the usage:

In [49]: parser.add_argument('targets',nargs='*');

In [50]: parser.print_help()
usage: ipykernel_launcher.py [-h] [--foo FOO] bar {test} ... [targets ...]

positional arguments:
  bar
  {test}
  targets

optional arguments:
  -h, --help  show this help message and exit
  --foo FOO

But the actual parsing does not see that added positional.

Parsing with the original main arguments goes fine:

In [51]: parser.parse_args('--foo 1 barval test testval'.split())
Out[51]: Namespace(foo='1', bar='barval', cmd='test', targets=[], xxx='testval')

But attempting to add 'targets' at the end fails - because the test parser does not recognize them. It just puts them in the "unrecognized" category, and the main just passes that on to the exit.

In [52]: parser.parse_args('--foo 1 barval test testval 1 2 3'.split())
usage: ipykernel_launcher.py [-h] [--foo FOO] bar {test} ... [targets ...]
ipykernel_launcher.py: error: unrecognized arguments: 1 2 3
An exception has occurred, use %tb to see the full traceback.

SystemExit: 2

C:\Users\paul\anaconda3\lib\site-packages\IPython\core\interactiveshell.py:3377: UserWarning: To exit: use 'exit', 'quit', or Ctrl-D.
  warn("To exit: use 'exit', 'quit', or Ctrl-D.", stacklevel=1)

If I change test xxx to nargs='*', these would have been parsed. (adding a new targets to sp1 would be clearer).

In [57]: sp1._actions[1].nargs='*'
In [58]: parser.parse_args('--foo 1 barval test testval 1 2 3'.split())
Out[58]: Namespace(foo='1', bar='barval', cmd='test', targets=[], xxx=['testval', '1', '2', '3'])

In [59]: sp1.print_help()
usage: ipykernel_launcher.py bar test [-h] [xxx ...]

positional arguments:
  xxx

optional arguments:
  -h, --help  show this help message and exit

The key points are that:

once the parsing is passed to the subparser, the main does not do any more parsing.
trying to add positionals after the subparsers does not work - for the above reason. The help may show them there, but that's not reflecting the parsing
using '*' (or '+?') positionals in the main is not a good idea. To the main, subparsers is just a specialized positional. It is recognized by position, not value.
It is best not use the same dest for the main and the subparsers. The subparsers default will always override the main.
parents is just a convenience tool, saving some typing when adding similar arguments to multiple subparsers. It doesn't do anything special.