Python subprocess passing args wrong

it seems some change in recent python versions introduced a different behavior of how subprocess.run parses arguments. A line that worked for years stopped working now. Now, my python script wraps LaTex Songs and lilypond to generate chordsheets. After running into problems with python versions and lilypond's bundled python, I set up everything for a clean start. Maybe the fault isn't with python at all but with lilypond, however, I asked at the usually very responsive lilypond support and got no help.

So, here is the problem:

command = "lilypond-book.py --output="+args.output+" --pdf -lERROR --lily-loglevel=ERROR "+f
print(command)

prints:

lilypond-book.py --output=C:\Users\xxx\Desktop\Tools\Songbook\Sample\pyOut --pdf -lERROR --lily-loglevel=ERROR book_general.lytex

Pasting that output back to the shell works fine. However, doing subprocess.run(command,shell=True) results in an error:

lilypond-book.py: error: file not found:  --output=C:\Users\xxx\Desktop\Tools\Songbook\Sample\pyOut --pdf -lERROR --lily-loglevel=ERROR book_general.lytex

It seems that using subprocess messes up the argument parsing for lilypond-book.py, it ignores the arguments and treats the whole command as "file". I also tried the prefered subprocess.run method using comma-separated arguments but got the same error. Now, I'm pretty clueless.

I'm on Python 3.12.2, previous installed version was 3.7, IIRC. Lilypond is 2.24.4. OS is Windows 10.

Thank you in advance for your help!

Solution

As suggested in comments, use a list of arguments directly if you already know how they should be separated and drop shell=True, the shell feature is generally much worse, never needed, and far more brittle (as you're finding out!) than an explicit list

Check out Actual meaning of 'shell=True' in subprocess !

p = subprocess.run(
    [
        "python",  # name in PATH
        "./lilypond-book.py",
        "--output=" + args.output,
        "--pdf",
        "-lERROR",
        "--lily-loglevel=ERROR",
        f,
    ],
)
p.check_returncode()  # check for success

You can use shlex.split() to get a shell-like splitting if you have a complete or a partial string which you for some reason don't want to parse yourself

And if you want to collect the output for other use, pass stdout=subprocess.PIPE, stderr=subprocess.PIPE, which will fill the resulting CompletedProcess's .stderr and .stdout properties respectively

More from the official docs https://docs.python.org/3/library/subprocess.html