Search code examples
python-3.xdockerpysparkjupyter-notebookmicroservices

Pyspark-Jupyter- Docker : How to rebuild and deploy image/container after defining master and worker node in the jupyter/pyspark-notebook?

How to rebuild the jupyter-pyspark-notebook docker container after add/modify the script in it?

My requirement is as follows after pulled the jupyter/pyspark-notebook

  1. Create a Master and worker node in the cluster as 2 container 1 for master and for worker
  2. Build a image/container as docker containerization application
  3. Deploy this container as microservices application

I am not seeing any suitable article or document specifically related to this..

Appreciate if anybody can help on this?

Thanks

Solution

  • Follow the blog article.

    1. Pull jupyter/pyspark-notebook and run
    2. Do your modification
    3. Commit the custom docker image to the docker hub by 'docker commit'. To do this, you might be needed to log-in to the docker hub.
    4. Check your custom image from the docker hub
    5. Pull your docker image from your application