Pytest: How can I blindly run any subprocess and capture all output?

Question

I have a Python program that calls print() and/or calls subprocess.Popen() on scripts in several other languages, which can also print, console.log, system.out.println, etc. The actual program easily prints all of this to the terminal, which is the intended behavior.

For integration testing, I want to capture all of this output to check if it matches an expected value. I cannot mock the subprocess. This is not unit testing, I don't care what my Python code is actually doing, only that the output is correct.

All of these options work for capturing the internal Python output:

• Use the capfd fixture to capture and read all output directly with Pytest
• Use context_lib and redirect_stdout
• Reassign sys.stdout manually to an explicit file, then read this file to get output

Here's roughly what the code looks like. For capfd and redirect_stdout you just shift around what you do right before and right after run_string.

# tests are parametrized from several JSON files.
def test_code(test_json):
    with open("/tmp/output_file.txt", "w+") as output_file:
        orig_stdout = sys.stdout
        sys.stdout = output_file
        print("initial")
        run_string(test_json["code"])
        output_file.seek(0)
        sys.stdout = orig_stdout
        output = output_file.read()
        assert output == test_json["expected"]

These all get the internal output perfectly, but all fail to get consistent output from subprocess.Popen. I have tried using all of these methods, setting subprocess.stdout to a file, to PIPE and printing it back out later, shell=True, and a few other things, but they all have the same strange issues. I really have no explanation for any of this behavior, so I'm just hoping someone can help me:

• All of this code works fine when I actually run my code using its CLI.
• The JavaScript subprocess always works, it's output is captured correctly, no matter which method I use.
• The Python subprocess does work when I use VSCode integrated test debugging, but not when I run pytest from the command line. Huh? Why?!?!?!?
• The Lua subprocess hangs entirely. It appears the subprocess never actually starts, although it's very hard to tell what's going on since I can't debug from Python into Lua, and I can't just print things out because Pytest is trying to capture everything.
• I've only added these 3 languages so far, but I intend to add many more (and in theory, all of them) so I'm hoping for a generic solution.

I could also try testing the entire CLI, but I'd like to avoid that, and I'm not sure it would work anyway. If anyone knows a way to force Pytest to run exactly like the native code does without putting subprocesses in some kind of box, that's really what I need.

Answer 1

For those wondering, I eventually settled on the capfd hook, and removed the nested subprocess requirement entirely. The subprocesses my app needs to execute are no longer allowed to output directly to shell, specifically because it is so inconsistent. This is the final code I ended up with.

def test_dits(dit_json, capfd):
    if "long" in dit_json and not pytest.all_val:  # type: ignore
        pytest.skip("Long test")
    run_string(dit_json["dit"], "tests/fail.dit")
    output, err = capfd.readouterr()

    assert output == dit_json["expected"]