Search code examples
rustrustup

Distributing private Rust compiler builds in an organization, e.g. with `rustup`

I have created a custom rust compiler build that requires a proprietary c++ compiler. How can I make this compiler available to my coworkers, e.g. installable with rustup, be able to refresh/update it, view available versions, etc.

Solution

  • We're building a Rust compiler for QNX which is then published/distributed internally. First step is to build a distribution of the toolchain for your target(s), e.g.:

    #!/bin/sh -e
 
export build_env='
    CC_aarch64-unknown-nto-qnx710=qcc
    CFLAGS_aarch64-unknown-nto-qnx710=-Vgcc_ntoaarch64le_cxx
    CXX_aarch64-unknown-nto-qnx710=qcc
    AR_aarch64_unknown_nto_qnx710=ntoaarch64-ar
    CC_x86_64-pc-nto-qnx710=qcc
    CFLAGS_x86_64-pc-nto-qnx710=-Vgcc_ntox86_64_cxx
    CXX_x86_64-pc-nto-qnx710=qcc
    AR_x86_64_pc_nto_qnx710=ntox86_64-ar'
 
env $build_env \
    ./x.py dist \
        --target \
            x86_64-unknown-linux-gnu,aarch64-unknown-nto-qnx710,x86_64-pc-nto-qnx710

    You need to adapt your environment variables and targets as needed. The important parameter is dist in the x.py command.

    To distribute the files for rustup, you also need some additional "manifest" files. Add your targets to src/tools/build-manifest/src/main.rs, e.g. with this diff:

    diff --git a/src/tools/build-manifest/src/main.rs b/src/tools/build-manifest/src/main.rs
index 21dad9eb74a..2964c2e916c 100644
--- a/src/tools/build-manifest/src/main.rs
+++ b/src/tools/build-manifest/src/main.rs
@@ -61,6 +61,7 @@
     "aarch64-unknown-none-softfloat",
     "aarch64-unknown-redox",
     "aarch64-unknown-uefi",
+    "aarch64-unknown-nto-qnx710",
     "arm-linux-androideabi",
     "arm-unknown-linux-gnueabi",
     "arm-unknown-linux-gnueabihf",
@@ -144,6 +145,7 @@
     "x86_64-pc-windows-msvc",
     "x86_64-sun-solaris",
     "x86_64-pc-solaris",
+    "x86_64-pc-nto-qnx710",
     "x86_64-unknown-freebsd",
     "x86_64-unknown-illumos",
     "x86_64-unknown-linux-gnu",

    Create the manifest (update date and URL as needed!):

    mkdir build/manifest
cargo run --release -p build-manifest build/dist build/manifest 1970-01-01 http://my-rustup-server:8000 stable

    If you don't have a suitable webserver, you can easily create one e.g. with the Rocket framework. You only need to do this once.

    cargo new own-rustup-server
cd own-rustup-server
cargo add rocket@0.5

    Change the content of src/main.rs:

    #[macro_use]
extern crate rocket;
 
use std::fs::File;
 
#[get("/<_something>/<filename>")]
fn dist(_something: &str, filename: &str) -> Option<File> {
    let filename = format!("dist/{filename}");
    File::open(filename).ok()
}
 
#[launch]
fn rocket() -> _ {
    rocket::build().mount("/", routes![dist])
}

    ... and create a Rocket.toml:

    [default]
port = 8000
address = "0.0.0.0"
log_level = "normal"

    Now we can copy the distribution to the rustup server and create required sha256 hash files (adapt the buildserver name and directories):

    mkdir -p ~/own-rustup-server/dist
cd ~/own-rustup-server/dist
scp buildserver:/home/user/rust/build/manifest/* .
scp buildserver:/home/user/rust/build/dist/* .
for f in *.gz *.xz *.txt *.toml ; do [ -e $f.sha256 ] || sha256sum $f > $f.sha256 ; done

    Compile and run the webserver (cargo run --release is sufficient for testing; ensure that it keeps running in the background, e.g. create systemd config files to let it run automatically). It is fine to let the server running while adding new distributions.

    Now your developers can use rustup to install your toolchain. It's best not to mix up the official Rust distribution and your one. It's best to install rustup without installing any toolchain:

    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-toolchain none

    Rustup uses the environment variable RUSTUP_DIST_SERVER to access the server. For simple use-cases set the variable when running rustup; it can make sense to set the variable e.g. in ~/.profile.

    To install e.g. Rust 1.71.0 with stdlib for QNX 7.1 (aarch64) and local host (x86_64 in my case):

    RUSTUP_DIST_SERVER=http://my-rustup-server:8000 rustup install 1.71.0
RUSTUP_DIST_SERVER=http://my-rustup-server:8000 rustup target add aarch64-unknown-nto-qnx710 --toolchain 1.71.0-x86_64-unknown-linux-gnu

    Depending on your needs, either specify which toolchain to use when running cargo (like cargo +1.71.0) or define a default toolchain with rustup default 1.71.0-x86_64-unknown-linux-gnu.