I used a SLURM manager to submit a bunch of scripts that all run in one Conda environment. I would like to install a new Python package to this environment. Do I need to wait until all of my scripts are done running? Or can I install the package now without messing anything up?
Can you? Sure. Should you? No. It could lead to changing existing packages, which could potentially lead to problems (e.g., missing references, API changes) - really depends on how the scripts are written and the dynamics of library loading throughout the scripts.
However, there is a larger issue of working reproducibly. May not apply here, but most SLURM users are doing scientific computing and scientific users should never mutate environments after using them to produce results. Intact environments are the scientific record and essential for reproducibility. If software requirements change, then create a new environment. Conda uses hardlinks to minimize disk usage, so one should be extremely liberal about creating new environments.