databricksazure-databrickspyodbc
Adding init .sh file for databricks job
I have a python script that's using a cluster that has an init script:
/Volumes/projects/default/admin/odbc.sh
this helps me connect via odbc to a sql server table.
When I run the job and add the cluster that has this ^ I see the error I saw before adding this to my cluster which was:
Can't open lib 'ODBC Driver 17 for SQL Server' : file not found
My question is, how to add this .sh file to my job??
Solution
I have tried the below approach:
The below code downloads the install MS SQL ODBC Driver on Azure Databricks cluster.
First Approach:
%sh
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
sudo apt-get update
sudo ACCEPT_EULA=Y apt-get -q -y install msodbcsql17
Second Approach:
if ! [[ "18.04 20.04 22.04 23.04 24.04" == *"$(lsb_release -rs)"* ]];
then
echo "Ubuntu $(lsb_release -rs) is not currently supported.";
exit;
fi
curl https://packages.microsoft.com/keys/microsoft.asc | sudo tee /etc/apt/trusted.gpg.d/microsoft.asc
curl https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/prod.list | sudo tee /etc/apt/sources.list.d/mssql-release.list
sudo apt-get update
sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18
#optional: for bcp and sqlcmd
sudo ACCEPT_EULA=Y apt-get install -y mssql-tools18
echo 'export PATH="$PATH:/opt/mssql-tools18/bin"' >> ~/.bashrc
source ~/.bashrc
#optional: for unixODBC development headers
sudo apt-get install -y unixodbc-dev
Results:
Reference: