Is there a way in python 3 to log the memory (ram) usage, while some program is running?
Some background info. I run simulations on a hpc cluster using slurm, where I have to reserve some memory before submitting a job. I know that my job require a lot of memory, but I am not sure how much. So I was wondering if there is a simple solution for logging the memory over time.
You can do that with the memory_profiler package. Just with adding a decorator @profile to a function, you will get an output like this:
Line # Mem usage Increment Line Contents
==============================================
3 @profile
4 5.97 MB 0.00 MB def my_func():
5 13.61 MB 7.64 MB a = [1] * (10 ** 6)
6 166.20 MB 152.59 MB b = [2] * (2 * 10 ** 7)
7 13.61 MB -152.59 MB del b
8 13.61 MB 0.00 MB return a
Otherwise, the easiest way to do it is to ask Slurm afterwards with the sacct -l -j <JobId> command (look for the MaxRSS column) so that you can adapt for further jobs.
Also, you can use the top command while running the program to get an idea of its memory consumption. Look for the RES column.