How does argparse (and the deprecated optparse) respond to 'tab' keypress after python program name, in bash?

Question

I have tested optcomplete working with the optparse module. Its example is a simple file so I could get that working. I also tested it using the argparse module as the prior one is deprecated. But I really do not understand how and by whom the python program gets called on tab presses. I suspect bash together with the shebang line and the argparse (or optparse) module are involved in some way. I have been trying to figure this out (now gonna read the source code).

I have a little more complex program structure, which includes a wrapper around the piece of code which handles the arguments. Its argparse.ArgumentParser() instantiation and calls to add_argument() - which are superclassed into another intermediate module to avoid duplicating code, and wrapper around that is being called - are inside a function.

I want to understand the way this tab completion works between bash and python (or for that matter any other interpretor like perl).

NOTE: I have a fair understanding of bash completion (which I learned just now), and I think I understand the bash(only) custom completion.

NOTE: I have read other similar SO questions, and none really answer this Q.

Edit: Here is the bash function.
I already understood how the python module gets to know about words typed in the command line, by reading os.environ values of variables

$COMP_WORDS
$COMP_CWORD
$COMP_LINE
$COMP_POINT
$COMPREPLY

These variables have values only on tab press. My question is how does the python module gets triggered?

Answer 1

To understand what's happening here, let's check what that bash function actually does:

COMPREPLY=( $( \
    COMP_LINE=$COMP_LINE  COMP_POINT=$COMP_POINT \
    COMP_WORDS="${COMP_WORDS[*]}"  COMP_CWORD=$COMP_CWORD \
    OPTPARSE_AUTO_COMPLETE=1 $1 ) )

See the $1 at the end? That means that it actually calls the Python file we want to execute with special environment variables set! To trace what's happening, let's prepare a little script to intercept what optcomplete.autocomplete does:

#!/usr/bin/env python2
import os, sys
import optparse, optcomplete
from cStringIO import StringIO

if __name__ == '__main__':    
    parser = optparse.OptionParser()

    parser.add_option('-s', '--simple', action='store_true',
                      help="Simple really simple option without argument.")

    parser.add_option('-o', '--output', action='store',
                      help="Option that requires an argument.")

    opt = parser.add_option('-p', '--script', action='store',
                            help="Option that takes python scripts args only.")
    opt.completer = optcomplete.RegexCompleter('.*\.py')
    
    # debug env variables
    sys.stderr.write("\ncalled with args: %s\n" % repr(sys.argv))
    for k, v in sorted(os.environ.iteritems()):
        sys.stderr.write("  %s: %s\n" % (k, v))

    # setup capturing the actions of `optcomplete.autocomplete`
    def fake_exit(i):
      sys.stderr.write("autocomplete tried to exit with status %d\n" % i)
    sys.stdout = StringIO()
    sys.exit = fake_exit

    # Support completion for the command-line of this script.
    optcomplete.autocomplete(parser, ['.*\.tar.*'])

    sys.stderr.write("autocomplete tried to write to STDOUT:\n")
    sys.stderr.write(sys.stdout.getvalue())
    sys.stderr.write("\n")

    opts, args = parser.parse_args()

This gives us the following when we try to autocomplete it:

$ ./test.py [tab]
called with args: ['./test.py']
  ...
  COMP_CWORD: 1
  COMP_LINE: ./test.py 
  COMP_POINT: 10
  COMP_WORDS: ./test.py 
  ...
  OPTPARSE_AUTO_COMPLETE: 1
  ...
autocomplete tried to exit with status 1
autocomplete tried to write to STDOUT:
-o -h -s -p --script --simple --help --output

So optcomplete.autocomplete just reads the environment, prepares the matches, writes them to STDOUT and exits. The result -o -h -s -p --script --simple --help --output is then put into a bash array (COMPREPLY=( ... )) and returned to bash to present the choices to the user. No magic involved :)