Pause a for loop, save progress and resume it another time?

Question

In python(3) I'm coding a script which uses a recursive for loop (the for loop calls it's own function various times) and I know the script will only finish after around 45 hours, so I'm searching for a way to save the progress in a file, close python, turn off and then resume it another day.

Is there a way to save the progress and then resume it in another (new) session?

Example of recursive loop:

i=1
def for_loop(i):
    if i==10:
        print(i)
        #code
    else:
        for x in range(0, 10):
            #code
            for_loop(i+1)

Whole code:

def TheLoop(num_N, num_a, num_A, num_S, i):
    global Key
    global num_Keys
    global num_Declined
    global KeyTogether
    global f

    if i > 8:
        if (num_N > 0) and (num_a > 0) and (num_A > 0) and (num_S > 0):
            num_Keys = num_Keys + 1
            KeyTogether = ''
            for t in range(8):
                try:
                    number = int(Key[t])
                    Key[t] = str(Key[t])
                except ValueError:
                    pass
                KeyTogether += Key[t]
            #Output Key to a file
            f.write(KeyTogether + "\n")
        else:
            # Invalid Key
            num_Declined = num_Declined + 1
    else:
       if (num_N < 4) and ((i < 8) or ((num_a > 0) and (num_A > 0) and (num_S > 0))):
            num_N = num_N + 1
            for x in range(0, 10):
                Key[i-1] = x
                TheLoop(num_N, num_a, num_A, num_S, i+1)
       if (num_a < 4) and ((i < 8) or ((num_N > 0) and (num_A > 0) and (num_S > 0))):
            num_a = num_a +1
            for c in "abcdefghijklmnopqrstuvwxyz":
                Key[i-1] = c
                TheLoop(num_N, num_a, num_A, num_S, i+1)
       if (num_A < 4) and ((i < 8) or ((num_N > 0) and (num_a > 0) and (num_S > 0))):
            num_A = num_A + 1
            for c in "ABCDEFGHIJKLMNOPQRSTUVWXYZ":
                Key[i-1] = c
                TheLoop(num_N, num_a, num_A, num_S, i+1)
       if (num_S < 4) and ((i < 8) or ((num_N > 0) and (num_a > 0) and (num_A > 0))):
            num_S = num_S + 1
            for c in '"'"!#$%&'()*+,-./:;<=>?@[\]^_`{|}~":
                Key[i-1] = c
                TheLoop(num_N, num_a, num_A, num_S, i+1)

Start = time.time()
TheLoop(0,0,0,0,1)
duration = time.time() - Start
print("Duration: " + str(duration))
print("Declined Keys: ", num_Declined)
print("Valid Keys: ", num_Keys)
duracionstr = str(duration)
f.write("Duration: " + durationstr)

Answer 1

I had the same problem with some long running loops in a project. I created a function which runns the loop with some additional merits.

• Progress is saved every N iterations (pickle)
• Continue from where you left off
• If input consists of more than one iterable, I loop over their permutation.
• Progress bar for nice visuals.

Let me know in case of any issues and I'm happy to help.

The function is available on pip:

pip install daves_utilities

Usage:

from daves_utilities.for_long import for_long

letters = ["a","b","c","d"]
numbers = [10,20,40,100,500,1000]
attribute_dict = {"letters": letters, "numbers": numbers, "input_bool":True}

df_out = for_long(iter_fun = my_function, iter_attr = attribute_dict,\
                  save_every_n = 10, path = "./")