Can't output lmod commands to text file using python in hpc environment

Question

I am trying to run this script on an h.p.c environment with the goal of writing all modules available in the environment to a text file. The module system in use is lmod.

This script works as expected for any regular bash command I have tried such as echo or ls, but when I try to use any module command like the one below I get the results output to my terminal and nothing written to the text file.

I have tried using the os module with os.popen and stream.read() but I encountered the same problem.

#!/usr/bin/python

import subprocess

def shell_cmd(command):
    """Executes the given command within terminal and returns the output as a string

    :param command: the command that will be executed in the shell
    :type command: str

    :return: the output of the command
    :rtype: str
    """
    process = subprocess.run([command], stdout=subprocess.PIPE, shell=True, universal_newlines=True)
    
    return process
    



#runs command
cmd = 'module avail'
out = shell_cmd(cmd)
output = out.stdout

# write output to text file
with open('output.txt', 'w+') as file:
    file.write(output)

Answer 1

Lmod (and Environment Modules) send their output to STDERR, not STDOUT so I think you should just modify your script to:

#!/usr/bin/env python

import subprocess

def shell_cmd(command):
    """Executes the given command within terminal and returns the output as a string

    :param command: the command that will be executed in the shell
    :type command: str

    :return: the output of the command
    :rtype: str
    """
    process = subprocess.run([command], stderr=subprocess.PIPE, shell=True, universal_newlines=True)
    
    return process
    



#runs command
cmd = 'module avail'
out = shell_cmd(cmd)
output = out.stderr

# write output to text file
with open('output.txt', 'w+') as file:
    file.write(output)

and it should work (it did on the system I tested).