We need to use an external library which only runs on Python 3.6 or higher. Unfortunately, HDInsight only has Python 3.5 installed by default. Is there a way to upgrade the minor version for Python on HDInsight?
The official Azure documentation does not mention about it.
https://learn.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-python-package-installation
As stated in Microsoft Q&A:
It's not recommended as using non-cluster built-in Python versions are unsupported scenario.
WARNING!: HDInsight cluster depends on the built-in Python environment - Python 3.5. Directly installing custom packages in those default built-in environments may cause unexpected library version changes. And break the cluster further.
If you want to install, you can use Python 3.6, change “python=3.5” in this command to python=3.6, and follow the rest steps in the document works.
sudo /usr/bin/anaconda/bin/conda create --prefix /usr/bin/anaconda/envs/py36new python=3.6 anaconda –yes