Azure Databricks pipeline failing because of dependency

Question

we have a Databricks Pipeline which runs every 6 hours most of the time it runs successfully but sometimes its getting failed by throwing below error

ModuleNotFoundError: No module named '<module-name>'

I am new to Databricks can anyone tell me how to fix it.

Answer 1

Some of the dependent libraries will not be working fine is base environment. It's highly suggestable to create virtual environment.

Follow the link to create virtual environment in Data Bricks: https://learn.microsoft.com/en-us/azure/databricks/runtime/conda

Refer the following link to connect pyodbc with databricks.

https://docs.databricks.com/dev-tools/pyodbc.html