OpenMDAO: How to use parallel processing for a particular subsystem while others run in serial?

Question

I am carrying out static aeroelastic analysis using an OpenMDAO-based script. I am trying to adapt my script to run it on a supercomputer.

The MDA loop includes several components, including a CFD solver usually run in terminal with mpirun -np 24 python runScript.py and a structural solver usually run in terminal with nast20212 nastran_static_alpha_c.bdf.

In order to run CFD in parallel mode within my main script, I need to run it with mpirun -np 24 python aeroelastic_mda.py.

The issue is that when I do that, the script tries to run all components in parallel mode. However I would only need to run my CFD solver to run on multiple nodes while other components run in serial mode.

I have looked up the OpenMDAO documentation and other posts but I did not find information regarding my issue.

Below is a summarized view of my OpenMDAO script which I run with mpirun -np 24 python aeroelastic_mda.py.

#Importing necessary modules
...  
if __name__ == "__main__":
    #Problem parameters
    Sw = 383.689555/2 #Wing reference surface
    V = 295.0 #Airspeed
    rho_a = 1.17 #Air density
    ... #Some other parameters
   
    root = Problem()

    #Add independent variables
    root.model.add_subsystem('wing_area', IndepVarComp('Sw', Sw), promotes=['*'])
    root.model.add_subsystem('airspeed', IndepVarComp('V', V), promotes=['*'])
    root.model.add_subsystem('air_density', IndepVarComp('rho_a', rho_a), promotes=['*'])
    ... #Some other independent variables

    mda = Group()

    #Add disciplines to the group

    # Displacement transfer: 
    # Inputs -> Displacements at structural nodes
    # Outputs -> Displacements to be applied to aerodynamic nodes 
    # Simple matrix product
    mda.add_subsystem('displacement_transfer', DisplacementTransfer(), promotes=['*'])

    # Aerodynamic solver
    mda.add_subsystem('aerodynamics', Dafoam(), promotes=['*'])

    # Load transfer
    # Inputs -> Forces at aerodynamic nodes
    # Outputs -> Forces to be applied to structural nodes 
    # Simple matrix product
    mda.add_subsystem('load_transfer', LoadTransfer(), promotes=['*'])
   
    # Structural solver
    mda.add_subsystem('structures', NastranStatic(), promotes=['*'])

    root.model.add_subsystem('mda_group', mda, promotes=['*'])

    mda.nonlinear_solver = NonlinearBlockGS()
    mda.nonlinear_solver.options['rtol'] = 1.e-3
    mda.nonlinear_solver.options['use_aitken'] = True

    mda.linear_solver = ScipyKrylov()
    root.linear_solver = ScipyKrylov()

    root.setup()
    root.run_model()

Also here is a simplified view of my aerodynamic component:

#Importing necessary modules
...
class Dafoam(ExplicitComponent):
    ... #Some constants
    
    def initialize(self):
        ... #Declaring options

    def setup(self):
        ... #Declaring inputs
        ... #Declaring outputs

    def compute(self, inputs, outputs):
        #Generate current deformed shape
        self.create_current_geom(inputs, outputs)

        #Generate the input file for Dafoam from current geometry and flow conditions
        self.create_input_file(inputs, outputs)

        # Run DAFoam
        p = Popen(['python', 'runScript.py'])
        p.wait()

        # Parse the output file from the external code    
        ...

    def create_current_geom(self, inputs, outputs):
    ...
    #Method that creates the Dafoam input file runScript.py
    def create_input_file(self, inputs, outputs):
    ...
    def get_forces(self, inputs, outputs):
    ...
    def get_aero_coeff(self, inputs, outputs):     
    ...

Also here is a simplified view of my structural component:

#Importing necessary modules
...  
class NastranStatic(ExplicitComponent):
    ... #Some constants
    
    def initialize(self):
        ... #Declaring options

    def setup(self):
        ... #Declaring inputs
        ... #Declaring outputs

    def compute(self, inputs, outputs):
        # Generate the input file for Nastran
        self.create_input_file(inputs, outputs)

        # Run MSC Nastran
        p = Popen(['nast20212', 'nastran_static_alpha_c.bdf'])
        p.wait()

        # Parse the output file from the external code and set output values
        ...

    def create_input_file(self, inputs, outputs):
    ...
    def get_output_data(self, inputs, outputs):
    ...

Thank you very much in advance for your help!

Answer 1

I have eventually solved my issue by replacing Popen by ShellProc (openmdao.utils.shell_proc) function. For some reason, Popen did not work properly when used with mpirun command.