Search code examples
kerasdrivegoogle-colaboratory

Do programs continue execution after connection is lost on google-colaboratory?


I am using google-colaboratory to train some convnets on image classification with Keras. And I get often disconnected from the notebook which I've found (on some forums) to be a frequent issue with some colaboratory users. And when I try to reconnect it gets stuck on "initializing":

enter image description here

So I decided to use Keras Checkpoint to save the model I'm training on my google drive, after each epoch.

During the training the notebook disconnected after 2 epochs, but when I monitored my google drive I've observed that models are being saved after each epoch, despite my colaboratory being disconnected.

enter image description here

My guess is that programs continue their execution in the background even if my noteboook is disconnected.

My question is, Am I guessing it right? And what do these disconnections really mean? (Are our variables, etc., preserved)?


Solution

  • Normally, notebook connection expires after 12 hours and all the data loaded into colab will also be expired with session.

    When you run a process and the notebook get disconnected after some time but before 12 hours, the code run in background.

    So, if you try to reconnect the notebook, it get stuck on "initializing" until the completion of code execution or session expiry (whichever occurs first). After that, the connection is established again.