Search code examples
pythongoogle-colaboratorytailngrok

In Colab : Reading / Watching logs as they grow

In Python, if you use the logging module, you can configure it to write the logs to stdout/console, a file, etc.

The problem is that you may want to monitor the logs without blocking the execution of other things. You want to see them grow, while you run other things.

In Linux you typically open another console and use # tail -f file.log and watch it grow, even with color adding other tools to the mix.

In Colab ... that is not so easy.

I'll dump here a few ideas so the question can be found and better answers may be added.

You could:

  • run a tiny web server so the file.log can be accessed from a browser. Problem: it doesn't grow/update with new content. There are workarounds for that, but they look ugly.

  • cat the file.log into a netcat connection connected to ngrok, and from your local machine use netcat (nc) to read that pipe

  • open a ssh into Colab and do the tail -f file.log just as you used to do.

  • send them to AWS Cloudwatch and maybe retrieve them on real-time for inspection

QUESTION: What is your preferred way to read the logs produced within Colab ( read inside and/or outside Colab )

Solution

  • open a ssh into Colab and do the tail -f file.log just as you used to do.

    This option is very comfortable as you can resize your local console window and move it around, being it outside Colab.

    Thanks to some people, nowadays is really easy to get there.

    Run this on a cell. Code comes from here.

    # Install colab_ssh
!pip install colab_ssh --upgrade

from colab_ssh import launch_ssh
launch_ssh('YOUR_NGROK_AUTH_TOKEN', 'SOME_PASSWORD')

    This will output something like (as of 2022-02-10)

    Collecting colab_ssh
  Downloading colab_ssh-0.3.27-py3-none-any.whl (26 kB)
Installing collected packages: colab-ssh
Successfully installed colab-ssh-0.3.27
Warning: Due to some issues with ngrok on Google Colab, reported in the issue https://github.com/WassimBenzarti/colab-ssh/issues/45, 
we highly recommend that update your code by following this documentation https://github.com/WassimBenzarti/colab-ssh#getting-started
Successfully running X.tcp.ngrok.io:12345
[Optional] You can also connect with VSCode SSH Remote extension using this configuration:

  Host google_colab_ssh
    HostName X.tcp.ngrok.io
    User root
    Port 12345

    Now, on your local machine (or wherever you want) open a console and type.

    ssh -p 12345 root@X.tcp.ngrok.io

    After that you are INSIDE Colab and you can do.

    tail -f /content/file.log

    And watch it grow.

    enter image description here

    After some inactivity the connection may get closed.

    Because the host changes frequently, using RSA keys to avoid typing the password doesn't seem like a good solution.

    So you may want to install sshpass

    # sudo apt-get install sshpass

    So you can simply reuse the command for connection after getting disconnected

    # sshpass -p <PASSWORD> ssh -p <PORT>  root@X.tcp.ngrok.io