Cross Compiling Rust with Nix

Recently I got to thinking about cross compiling rust again because I need to produce binaries for different platforms for my project pict-rs. I never quite liked how I made it work in CI after migrating from drone to forgejo actions (see my first blog post on here for forgejo actions hate), so I'm always interested in ways to make builing my binaries easier.

Existing Solutions

Unfortunately, cross compiling with Nix has been complicated. Sure, there's a bunch of packages in pkgs.pkgsCross.*, but how do you even use any of them? And why can't I just do cargo build --target=some-other-triple? In other environments, I would simply use rustup and install the right toolchain, set a few environment variables, and produce some statically linked musl binaries. In my current CI setup, I use cargo zigbuild from within a debian environment to compile and link against the musl libc for x86_64, aarch64, and armv7, and that's the simplest I've managed to get things.

Looking online for documentation on rust programs with nix, there's always references to third party overlays like fenix, which I'm sure are very good, but I don't want to use. I want my programs to be easily packaged for Nix and other distributions. The more I complicate my build process, the more work distro maintainers will need to do to keep up with it.

Derivations

In my pict-rs repo, I maintain a simple pict-rs.nix derivation, which is copied almost verbatim from the NixOS pict-rs package. I do this so that I can run nix build every now and then to see if the build process needs changing, and to provide an example to packagers that includes a clean dependency list and build process.

Naively crossPackagesing

Since I have a neat derivation for building pict-rs already, and nix includes a callPackage helper, and supports the targets I care about in pkgs.pkgsCross, would it be possible to just use my existing derivation and cross-compile natively in nix? Yes, kind of, and no.

let
  crossPackages = import nixpkgs {
    inherit system;
    crossSystem = "aarch64-linux";
  };
in
{
  packages = {
    pict-rs-arm64 = crossPackages.callPackage ./pict-rs.nix { };
  };
}

This works! Except it produces an aarch64 binary linked dynamically against glibc, which is not what I want for distributing binaries that should Run Anywhere. I need resulting binaries to be statically linked. When looking into this, I found nix provides a pkgsMusl package set, which is conveniently easy to try!

{
  packages = {
    pict-rs-arm64 = crossPackages.pkgsMusl.callPackage ./pict-rs.nix { };
  };
}

And this works too! Except...

$ file result/bin/.pict-rs-wrapped
result/bin/.pict-rs-wrapped: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/rpqv8iricasr9hbmp1xfwsfhy2q1k9dk-musl-aarch64-unknown-linux-musl-1.2.5/lib/ld-musl-aarch64.so.1, not stripped

What do you mean dynamically linked? I built with musl and rust statically links as much as it can! Well, rust here is coming from nix, and nix configured it to dynamically link, so this also is not what I want.

But wait! What's that I see in the nix repl? pkgsStatic? Is this what I have been looking for the whole-

collect2: error: ld returned 1 exit status

I see. Well that's unfortunate. I guess I can't cross compile pict-rs that easily with nix, then.

But it's actually easy

Let's have a bit of a think on that linking error we got. What failed to link? it was perl-static-aarch64-unknown-linux-musl. Do we care about perl? why is it even here? Well it's a dependency of exiftool, which we need at runtime for pict-rs to function, but not at build time. And we don't need ffmpeg or imagemagick either! What if instead of calling callPackage on the normal pict-rs derivation, I called it on a custom derivation with no runtime dependencies listed?

{
  pict-rs = pkgs.pkgsCross.aarch64-multiplatform.pkgsStatic.callPackage ./pict-rs-unwrapped.nix { };
}

$ file result/bin/pict-rs
result/bin/pict-rs: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, not stripped

Oh!

Well that's good. What about armv7? I still want to support that even though it's old and SBCs these days are aarch64. What if someone is running a Rasbperry Pi 2 Model B?

{
  pict-rs = pkgs.pkgsCross.armv7-hf-multiplatform.pkgsStatic.callPackage ./pict-rs-unwrapped.nix { };
}

$ file result/bin/pict-rs
result/bin/pict-rs: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped

Oh! That was easy. But... it took so long. These static builds' tools aren't cached in hydra, which means I had to compile gcc and rustc on my computer before I could even start compiling pict-rs. If I wanted to do this in CI, I'd need my own nix cache to avoid doing that a lot. Oh well.

I added these packages to my flake anyway, slightly modified to make extending it with other platforms easier.

let
  targets = [
    {
      triple = "x86_64-unknown-linux-musl";
      name = "amd64";
    }
    {
      triple = "aarch64-unknown-linux-musl";
      name = "arm64";
    }
    {
      triple = "armv7l-unknown-linux-musleabihf";
      name = "armv7l";
    }
  ];
in
{
  packages =
    builtins.foldl' (
      acc:
      { triple, name }:
      let
        crossPkgs = import nixpkgs {
          inherit system;
          crossSystem.config = triple;
        };
      in
      acc
      // {
        "pict-rs-${name}" = crossPkgs.pkgsStatic.callPackage ./pict-rs-unwrapped.nix {
          inherit (pkgs.darwin.apple_sdk.frameworks) Security;
        };
      }
    ) { } targets;
}

Configuring crossPkgs with crossSystem.config is notable, since I didn't realize that was possible until I saw another blog post about cross compiling rust with nix. I came to a different conclusion than they did about the ease of compiling, though. Using derivations and callPackage definitely simplifies the build, and adding other targets just means extending the targets list.

Since I don't want to set up a nix cache for myself, I don't think I'll switch my CI to use nix to build these binaries, but it's good to know that it's possible and not even difficult.