pause python script to save output

Question

I am running a code in python to calculate some metric on adjacency matrices, where I loop through all adjacency matrices, iteratively writing to a numpy array, and then save the output of the array at the end of the loop. As I can visualize the progress of the calculation, I see that the metric calculation on one of the matrices is taking way too long (e.g., instead of the usual 5 seconds, it's been hours). Here is my script:

longmat = scipy.io.loadmat('longmat.mat')
longmat = longmat['longmat']
matlength = longmat.shape
densval = np.arange(0, 1, 0.01)
betti = 0
output_long = np.zeros((264, 168, 12), dtype=float, order='F')

for z in range(matlength[2]): 
    for s in range(matlength[1]):
        temp = longmat[:, s, z] #vector of values
        tempr = scipy.spatial.distance.squareform(temp,checks=True)

        for dens in range(len(densval)):
            G_tda = gd.graph_density(densval[dens], tempr)
            B = bet.Betti_k(G_tda, betti, verbose=False)
            output_long[dens, s, z] = B
            if B < 2:
                break

scipy.io.savemat('longout.mat', {'output_long': output_long})

I'd like to pause the script and save the array output up until this point and was wondering if this is possible. Thank you very much for your help!

Georgette

Answer 1

You need an if statement.

while true: # or a for loop
    #do the calculations here
    if condition: # condition is the metric before the one that takes too long to calculate
        open("test.txt", "a").write(output) # save the output to the file

This is only possible if you know which metric takes a long time to calculate.

UPDATE: Now that you have shared your code, I can see a better option. You need to put the scipy.io.savemat() in the most inner loop. As below:

longmat = scipy.io.loadmat('longmat.mat')
longmat = longmat['longmat']
matlength = longmat.shape
densval = np.arange(0, 1, 0.01)
betti = 0
output_long = np.zeros((264, 168, 12), dtype=float, order='F')

for z in range(matlength[2]): 
    for s in range(matlength[1]):
        temp = longmat[:, s, z] #vector of values
        tempr = scipy.spatial.distance.squareform(temp,checks=True)

        for dens in range(len(densval)):
            G_tda = gd.graph_density(densval[dens], tempr)
            B = bet.Betti_k(G_tda, betti, verbose=False)
            if B < 2:
                break
            output_long[dens, s, z] = B
            scipy.io.savemat('longout.mat', {'output_long': output_long}, appendmat=True)

This way at the end of each loop B is appended to the output file. Note that appendmat=True ensures no overwrite will happen. You can see its documentation here. I didn't use else, but you can use it for beauty. It's not necessary anyway.