So I used shlex.split() recently to split a command as argument to subprocess.Popen() function. I recalled that long back I also used re.split() function to split a string with a specific delimiter specified. Can someone point out what is the essential difference in between them? In which scenario is each function best suited?
shlex.split() is designed to work like the shell's split mechanism.
This means doing things like respecting quotes, etc.
>>> shlex.split("this is 'my string' that --has=arguments -or=something")
['this', 'is', 'my string', 'that', '--has=arguments', '-or=something']
re.split() will just split on whatever pattern you define.
>>> re.split('\s', "this is 'my string' that --has=arguments -or=something")
['this', 'is', "'my", "string'", 'that', '--has=arguments', '-or=something']
Trying to define your own regex to work like shlex.split is needlessly complicated, if it's even possible.
To really see the differences between the two, you can always Use the Source, Luke:
>>> re.__file__
'/usr/lib/python3.5/re.py'
>>> shlex.__file__
'/usr/lib/python3.5/shlex.py'
Open these files in your favorite editor and start poking around, you'll find that they operate quite differently.