I am trying to resume training from a certain checkpoint (Tensorflow) because I'm using Colab and 12 hours aren't enough

Question

This is some part of the code I'm using

checkpoint_dir = 'training_checkpoints1'
checkpoint_prefix = os.path.join(checkpoint_dir, "ckpt")
checkpoint = tf.train.Checkpoint(optimizer=optimizer,
                             encoder=encoder,
                             decoder=decoder)

Now this is the training part

EPOCHS = 900

for epoch in range(EPOCHS):
  start = time.time()

  hidden = encoder.initialize_hidden_state()
  total_loss = 0

  for (batch, (inp, targ)) in enumerate(dataset):
      loss = 0
    
      with tf.GradientTape() as tape:
          enc_output, enc_hidden = encoder(inp, hidden)
        
          dec_hidden = enc_hidden
        
          dec_input = tf.expand_dims([targ_lang.word2idx['<start>']] * batch_size, 1)       
        
          # Teacher forcing - feeding the target as the next input
          for t in range(1, targ.shape[1]):
              # passing enc_output to the decoder
              predictions, dec_hidden, _ = decoder(dec_input, dec_hidden, enc_output)
            
              loss += loss_function(targ[:, t], predictions)
            
              # using teacher forcing
              dec_input = tf.expand_dims(targ[:, t], 1)
    
      batch_loss = (loss / int(targ.shape[1]))
    
      total_loss += batch_loss
    
      variables = encoder.variables + decoder.variables
    
      gradients = tape.gradient(loss, variables)
    
      optimizer.apply_gradients(zip(gradients, variables))
    
      if batch % 100 == 0:
          print('Epoch {} Batch {} Loss {:.4f}'.format(epoch + 1,
                                                     batch,
                                                     batch_loss.numpy()))
  # saving (checkpoint) the model every 2 epochs
  if (epoch + 1) % 2 == 0:
    checkpoint.save(file_prefix = checkpoint_prefix)

  print('Epoch {} Loss {:.4f}'.format(epoch + 1,
                                    total_loss / num_batches))
  print('Time taken for 1 epoch {} sec\n'.format(time.time() - start))

Now I want to restore for exp this checkpoint and start training from there but I don't know how.

path="/content/drive/My Drive/training_checkpoints1/ckpt-9"
checkpoint.restore(path)

Result

<tensorflow.python.training.tracking.util.CheckpointLoadStatus at 0x7f6653263048>

Answer 1

You should create a CheckpointManager at the start as:

checkpoint_path = os.path.abspath('.') + "/checkpoints"   # Put your path here
ckpt = tf.train.Checkpoint(encoder=encoder,
                           decoder=decoder,
                           optimizer = optimizer)
ckpt_manager = tf.train.CheckpointManager(ckpt, checkpoint_path, max_to_keep=5)

Now after running for few epoch, to restore latest checkpoint, you should get the latest checkpoint from the CheckpointManager:

start_epoch = 0
if ckpt_manager.latest_checkpoint:
    start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])
    # restoring the latest checkpoint in checkpoint_path
    ckpt.restore(ckpt_manager.latest_checkpoint)

This will restore your session from the latest epoch.