Search code examples
pythondatabricksrustup

How to install Cargo, Rust package manager on Databricks


I'm trying to install Vaex with !pip install vaex on a cluster on Databricks, but the installation is failing because it has a dependency called blake3, which needs Cargo.

Collecting blake3
  Using cached blake3-0.3.1.tar.gz (131 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      
      Cargo, the Rust package manager, is not installed or is not on PATH.
      This package requires Rust and Cargo to compile extensions. Install it through
      the system's package manager or via https://rustup.rs/
      
      Checking for Rust toolchain....
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

Going to the redirected link has a command to install Cargo

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

But running it on a Databricks python notebook doesn't work because it is an interactive shell command which needs an input to be given.

Current installation options:


   default host triple: aarch64-unknown-linux-gnu
     default toolchain: stable (default)
               profile: default
  modify PATH variable: yes

1) Proceed with installation (default)
2) Customize installation
3) Cancel installation

I have tried curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -1 but it throws an illegal option error


Solution

  • rustup-init 1.22.1 (76644d669 2020-07-08)
    The installer for rustup
    
    USAGE:
        rustup-init [FLAGS] [OPTIONS]
    
    FLAGS:
        -v, --verbose           Enable verbose output
        -q, --quiet             Disable progress output
        -y                      Disable confirmation prompt.
            --no-modify-path    Don't configure the PATH environment variable
        -h, --help              Prints help information
        -V, --version           Prints version information
    
    OPTIONS:
            --default-host <default-host>              Choose a default host triple
            --default-toolchain <default-toolchain>    Choose a default toolchain to install
            --default-toolchain none                   Do not install any toolchains
            --profile [minimal|default|complete]       Choose a profile
        -c, --component <components>...                Component name to also install
        -t, --target <targets>...                      Target name to also install
    

    you can use -y to Disable confirmation prompt.

    like this

    curl --proto '=https' --tlsv1.2 https://sh.rustup.rs -sSf > rustup.sh
    sh rustup.sh -y
    sudo bash -c "echo source $HOME/.cargo/env >> /etc/bash.bashrc"