Cross Compiling Rust with Nix

2026-03-20T00:00:00+00:00

Recently I got to thinking about cross compiling rust again because I need to produce binaries for different platforms for my project pict-rs</a>. I never quite liked how I made it work in CI after migrating from drone to forgejo actions (see my first blog post on here for forgejo actions hate), so I'm always interested in ways to make builing my binaries easier.</p>

Existing Solutions</h2>
Unfortunately, cross compiling with Nix has been complicated. Sure, there's a bunch of packages in pkgs.pkgsCross.*</code>, but how do you even use any of them? And why can't I just do cargo build --target=some-other-triple</code>? In other environments, I would simply use rustup and install the right toolchain, set a few environment variables, and produce some statically linked musl binaries. In my current CI setup, I use cargo zigbuild</code> from within a debian environment to compile and link against the musl libc for x86_64, aarch64, and armv7, and that's the simplest I've managed to get things.</p>
`Looking online for documentation on rust programs with nix, there's always references to third party overlays likefenix</a>, which I'm sure are very good, but I don't want to use. I want my programs to be easily packaged for Nix and other distributions. The more I complicate my build process, the more work distro maintainers will need to do to keep up with it.</p>`

Derivations</h2>
In my pict-rs repo, I maintain a simple pict-rs.nix derivation, which is copied almost verbatim from
the NixOS pict-rs package. I do this so that I can run nix build</code> every now and then to see if the
build process needs changing, and to provide an example to packagers that includes a clean
dependency list and build process.</p>
Naively crossPackagesing</h2>
Since I have a neat derivation for building pict-rs already, and nix includes a callPackage</code>
helper, and supports the targets I care about in pkgs.pkgsCross</code>, would it be possible to just use
my existing derivation and cross-compile natively in nix? Yes, kind of, and no.</p>
let</span></span>
  crossPackages</span> =</span> import</span> nixpkgs</span> {</span></span>
    inherit</span> system</span>;</span></span>
    crossSystem</span> =</span> "</span>aarch64-linux</span>"</span>;</span></span>
  }</span>;</span></span>
in</span></span>
{</span></span>
  packages</span> =</span> {</span></span>
    pict-rs-arm64</span> =</span> crossPackages</span>.</span>callPackage</span> ./pict-rs.nix</span> {</span> }</span>;</span></span>
  }</span>;</span></span>
}</span></span></code></pre>
This works! Except it produces an aarch64 binary linked dynamically against glibc, which is not what
I want for distributing binaries that should Run Anywhere. I need resulting binaries to be statically
linked. When looking into this, I found nix provides a pkgsMusl</code> package set, which is conveniently
easy to try!</p>
{</span></span>
  packages</span> =</span> {</span></span>
    pict-rs-arm64</span> =</span> crossPackages</span>.</span>pkgsMusl</span>.</span>callPackage</span> ./pict-rs.nix</span> {</span> }</span>;</span></span>
  }</span>;</span></span>
}</span></span></code></pre>
And this works too! Except...</p>
$ file result/bin/.pict-rs-wrapped</span></span>
result/bin/.pict-rs-wrapped: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/rpqv8iricasr9hbmp1xfwsfhy2q1k9dk-musl-aarch64-unknown-linux-musl-1.2.5/lib/ld-musl-aarch64.so.1, not stripped</span></span></code></pre>
What do you mean dynamically linked? I built with musl and rust statically links as much as it
can! Well, rust here is coming from nix, and nix configured it to dynamically link, so this also is
not what I want.</p>
But wait! What's that I see in the nix repl</code>? pkgsStatic</code>? Is this what I have been looking for the whole-</p>
collect2: error: ld returned 1 exit status</span></span></code></pre>
I see. Well that's unfortunate. I guess I can't cross compile pict-rs that easily with nix, then.</p>
But it's actually easy</h2>
Let's have a bit of a think on that linking error we got. What failed to link? it was
perl-static-aarch64-unknown-linux-musl</code>. Do we care about perl? why is it even here? Well it's a
dependency of exiftool, which we need at runtime for pict-rs to function, but not at build time. And
we don't need ffmpeg or imagemagick either! What if instead of calling callPackage</code> on the normal
pict-rs derivation, I called it on a custom derivation with no runtime dependencies listed?</p>
{</span></span>
  pict-rs</span> =</span> pkgs</span>.</span>pkgsCross</span>.</span>aarch64-multiplatform</span>.</span>pkgsStatic</span>.</span>callPackage</span> ./pict-rs-unwrapped.nix</span> {</span> }</span>;</span></span>
}</span></span></code></pre>$ file result/bin/pict-rs</span></span>
result/bin/pict-rs: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, not stripped</span></span></code></pre>
Oh!</p>
Well that's good. What about armv7? I still want to support that even though it's old and SBCs these
days are aarch64. What if someone is running a Rasbperry Pi 2 Model B?</p>
{</span></span>
  pict-rs</span> =</span> pkgs</span>.</span>pkgsCross</span>.</span>armv7-hf-multiplatform</span>.</span>pkgsStatic</span>.</span>callPackage</span> ./pict-rs-unwrapped.nix</span> {</span> }</span>;</span></span>
}</span></span></code></pre>$ file result/bin/pict-rs</span></span>
result/bin/pict-rs: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped</span></span></code></pre>
Oh! That was easy. But... it took so long. These static builds' tools aren't cached in hydra, which
means I had to compile gcc and rustc on my computer before I could even start compiling pict-rs. If
I wanted to do this in CI, I'd need my own nix cache to avoid doing that a lot. Oh well.</p>
I added these packages to my flake anyway, slightly modified to make extending it with other
platforms easier.</p>
let</span></span>
  targets</span> =</span> [</span></span>
    {</span></span>
      triple</span> =</span> "</span>x86_64-unknown-linux-musl</span>"</span>;</span></span>
      name</span> =</span> "</span>amd64</span>"</span>;</span></span>
    }</span></span>
    {</span></span>
      triple</span> =</span> "</span>aarch64-unknown-linux-musl</span>"</span>;</span></span>
      name</span> =</span> "</span>arm64</span>"</span>;</span></span>
    }</span></span>
    {</span></span>
      triple</span> =</span> "</span>armv7l-unknown-linux-musleabihf</span>"</span>;</span></span>
      name</span> =</span> "</span>armv7l</span>"</span>;</span></span>
    }</span></span>
  ]</span>;</span></span>
in</span></span>
{</span></span>
  packages</span> =</span></span>
    builtins</span>.</span>foldl'</span> (</span></span>
      acc</span>:</span></span>
      {</span> triple</span>,</span> name</span> }</span>:</span></span>
      let</span></span>
        crossPkgs</span> =</span> import</span> nixpkgs</span> {</span></span>
          inherit</span> system</span>;</span></span>
          crossSystem</span>.</span>config</span> =</span> triple</span>;</span></span>
        }</span>;</span></span>
      in</span></span>
      acc</span></span>
      //</span> {</span></span>
        "</span>pict-rs-</span>${</span>name</span>}</span>"</span> =</span> crossPkgs</span>.</span>pkgsStatic</span>.</span>callPackage</span> ./pict-rs-unwrapped.nix</span> {</span></span>
          inherit</span> (</span>pkgs</span>.</span>darwin</span>.</span>apple_sdk</span>.</span>frameworks</span>)</span> Security</span>;</span></span>
        }</span>;</span></span>
      }</span></span>
    )</span> {</span> }</span> targets</span>;</span></span>
}</span></span></code></pre>
Configuring crossPkgs with crossSystem.config</code> is notable, since I didn't realize that was possible
until I saw another blog post about cross compiling rust with nix. I came to a different conclusion
than they did about the ease of compiling, though. Using derivations and callPackage definitely
simplifies the build, and adding other targets just means extending the targets</code> list.</p>
Since I don't want to set up a nix cache for myself, I don't think I'll switch my CI to use nix to
build these binaries, but it's good to know that it's possible and not even difficult.</p>


Moving
2024-07-20T00:00:00+00:00
I will be moving in two weeks.</p>
Because of this, all asonix.dog services will be OFFLINE for somewhere between a few hours and a few
days starting Friday August 2nd at 1pm UTC.</p>
If you need to access source code for anything I work on, the important stuff is mirrored on
codeberg: https://codeberg.org/asonix</a>.</p>
I will do my best to bring everything back online in a timely manner but we'll need to see how it
actually goes when we get there.</p>


Help! My function is Crimson!
2024-02-15T00:00:00+00:00
This post will hopefully be shorter than the last two. I just wanted to dig into the fourth function
color brought up in
Which "Red" is your function?</a>
I'm not going to get into the details of function coloring here. If you want to know you can read
the linked article, and the articles the linked article links.</p>

Note: If you want to talk about io-uring or io-ring or IOCP that's an entire other discussion that
plays into this problem but will not be covered here.</p>
</blockquote>
How do you do IO?</h2>
This is really the crux of the issue. How do you do IO. In the standard library, you can have a
TcpStream and issue a .read()</code> on it and get the bytes. This works by passing the TcpStream's File
Descriptor to the read</code> syscall</a>. Calling
this function can have a couple results:</p>

Bytes were read from the provided descriptor and placed into the provided buffer. The function's
return value indicates how many bytes were read.</li>
No bytes were read because there are no more bytes to read. No bytes are placed into the buffer
and the function's return value is 0. This indicates that the read</code> call should not be retried.</li>
No bytes were read because some sort of error occured. No bytes are placed into the buffer and
the function's return value is less than 0, each possible sub-zero value corresponds with a
specific error.</li>
</ol>
This is all well and good, but it doesn't work well in contexts where we aren't supposed to block</code>.
Consider the case where we're talking with a very slow computer, and it is sending us bytes, but we
don't receive them in a reasonable timeframe. Using the read</code> syscall as-is doesn't allow our
program to do anything else while we're waiting on the slow computer.</p>
Luckily there's a very common workaround for this specific issue. We can put our TcpStream in
non-blocking mode, which will slightly alter the behavior of the read</code> syscall. Now instead of
waiting for bytes to put into the provided buffer, if there are no bytes available the read</code> call
will return immediately with a new error value. Our program can notice this new error value and
realize that although we didn't read anything this time, in the future we will be able to read
something.</p>
But how will we know when to try reading again? That is the core problem that creates our Crimson
functions.</p>
How will they know? (They're gonna know)</h2>
In Rust there's a number of async runtimes that you can choose from. tokio is by and large the one
people will use or tell you to use, but there's also smol, async-std, embassy, glommio, monoio,
actix-rt, jive (I wrote this one)</a>, and I'm sure others.
The primary motivating factor for having multiple runtimes like this is to play with different ideas
of IO. I said up front that I'm not going to talk about completion-based IO, embassy is another
special case, actix-rt is just tokio with extra steps, and smol and async-std are basically the same
runtime, so let's pretend for now that the only runtimes that exist are tokio and async-std.</p>
The real distinction between tokio and async-std is how they attempt to figure out when</em> to try
reading more bytes. In both cases, after failing to read bytes from the TcpStream, the runtime will
register the TcpStream's file descriptor with an event mechanism. This mechanism is backed by a
blocking IO operation, just like a blocking read</code>, but unlike a blocking read</code>, the event
mechanism does not itself read any bytes, and it is capable of waiting for available bytes for many
file descriptors simultaneously. This means that if your program has 30 TcpStreams, you can block
waiting for any one of them to become available for reading on a single thread. Since this operation
does block, tokio and async-std need to strike a balance between waiting for events on these
registered file descriptors, and making progress on other asynchronous tasks.</p>
And that's why my crimson functions explode?</h2>
Yeah basically. tokio's TcpStream type knows how to register itself with tokio's event mechanism,
and is designed to play with tokio's scheduling. Underneath, the bytes are still read with the
read</code> syscall we talked about earlier, but all the surrounding bits about deciding when to read
make it unique.</p>
async-std's TcpStream type is the same thing. It knows how to register itself with async-std's event
mechanism, and plays as nicely as it can*</em> with async-std's scheduling. It really comes down to
that registration step for the difference between these two runtimes. It is feasible that the Future
trait's Context</code> argument could be extended with a method such as .register(fd: &BorrowedFd<'_>)</code>.
This could potentially allow any type with access to a File Descriptor to register with any
runtime's event mechanism. But it's not enough to handle every scenario.</p>

*tokio has a cooperation mechanism that its TcpStream integrates with to reduce task starvation
and async-std does not.</p>
</blockquote>
Why not?</h2>
Well for one, not every runtime has an event mechanism like async-std and tokio do. I left out
runtimes using completion-based IO because they work in an entirely different way. I left out
embassy because it's designed for microcontrollers and doesn't have access to an operating system
that could provide read</code>. Building an abstraction around an event mechanism and file descriptors
leaves out other runtime implementations that don't include one or both of these concepts. These
problems are why we don't have a unified API for dealing with runtimes outside of the Future trait.
Unification is hard. It might not be possible. But there's a bunch of smart folks thinking about it,
so maybe things will get better.</p>


What is Public?
2024-02-13T00:00:00+00:00

Foreword: In this post I mention Mastodon by name a couple times because that is the primary
software I use to access the ActivityPub network. Please do not see this as a conflation of
Mastodon with "The Fedi."</p>
</blockquote>
Yesterday I became aware of a project that aims to bridge ActivityPub and ATProto together by acting
as a traditional ActivityPub server, as well as an ATProto PDS. This is a reasonable approach from a
technical perspective. It will behave as expected of a native implementation on each side, while
proxying users and posts through.</p>
But there's a disconnect between what Public means on each side of the bridge, and I'd like to take
some time to dig into what that word means. While this discussion will get somewhat technical, there
are reasons</em> for the distinction after all, I'm not going to go in depth into either protocol.
We'll try to stick to a higher level view of each network.</p>
ATProto</h2>
What the heck is ATProto? It's likely that more folks on Mastodon know about ATProto than folks on
Bluesky, but there's certainly folks on both platforms that don't know about ATProto. ATProto is the
federation protocol that Bluesky will be implementing eventually. It will enable Bluesky to exist
within a wider network of services that operate independently of eachother, but still enable ease of
discovery and "seamless integration." ATProto is the reason why everything on Bluesky is public. The
protocol works by giving every participating service full access to everything a user does,
including Posts, Follows, Followers, Blocks, Likes, etc.</p>
ATProto is designed to be maximally discoverable. This works by having large central services that
are responsible for collecting the data from all known users, and those central services provide
mechanisms to subscribe to all the user data they know about. "The Firehose." This model hasn't been
deployed widely in production because Bluesky itself has not enabled ATProto, and so there's no
concrete data for how this model will pan out.</p>
For those of you on Bluesky surprised by the idea that Bluesky might federate, I want you to know
this was always the plan. From when Jack first mentioned Bluesky when he still owned Twitter,
federation was always the plan.</p>
ActivityPub</h2>
ActivityPub has been around for some years now. It is famously the protocol that Mastodon uses to
federate. In ActivityPub, each participating server is responsible for making their posts and users
known to the wider network. There's no central discovery or publishing service like in ATProto,
although relays</code> exist to fill a similar niche. In ActivityPub, there's no prescribed mechanism to
denote what is public and what is not public, so implementations began relying on convention to
signal to each other how public a given post should be. In Mastodon's model there are four varying
levels of "public-ness" that posts can have.</p>

Direct - This post is only visible to mentioned users</li>
Followers Only - This post is only visible to mentioned users, and users that follow the poster</li>
Unlisted - This post is visible to anyone, but is left out of discovery mechanisms</li>
Public - This post is visible to anyone, and is included in discovery mechanisms</li>
</ol>
In order for this model to work, all participating services must agree on these privacy levels
meaning what they do. It is possible for a malicious or malfunctioning ActivityPub implementation to
improperly treat a post as more public than it was intended to be, resulting in reach beyond the
intended audience. In order for this to happen, however, the post must first reach</em> the bad
implementation for it to be rebroadcast. Direct messages are not likely to be made public, since it
would require one of the parties in the thread to exist on a malicious server. I assume that most
Direct messages happen between parties that know each other and are using compliant impelementations.</p>
Relays</h3>
As I mentioned above, ActivityPub has a concept of "relays." These are opt-in mechanisms to improve
discoverability of posts. ActivityPub servers can "subscribe" to relays of their choosing, and when
they do so, they send all future public posts they host to the relay. The relay in turn broadcasts
all those posts to the other subscribed servers. This provides a similar function to the central
services in ATProto, but in ActivityPub there is no default relay, and relays are generally operated
at small scales. They also don't generally have a "public feed" that anyone can subscribe to,
although that's not a huge barrier to entry for a sufficiently motivated programmer.</p>
I personally maintain an ActivityPub relay implementation called
AodeRelay</a>, and I host an
invite-only version of it</a> for furry and adjacent servers to share their
posts to each other. In order for a server to join my relay, that server's administrator needs to
contact me to request permission to join. I haven't yet said no to a request, but just having that
option is nice. As a relay administrator, I am responsible for the content my relay is forwarding. I
want my relay to remain useful to the servers that subscribe to it, and if my relay starts putting
unwanted content into people's feeds, it is no longer useful.</p>
So What is Public?</h2>
This is where we really get to compare the two models. Public in ATProto is everything. Every post.
Every Like. Every Follow. Every participating ATProto service can be made aware of any activity on
ATProto in real time. All user history is publicly available to anyone who looks. In ActivityPub
only Some Things are public, and even then, there's far less reach for things that are public.
There's no mechanism to subscribe to All Public Things, and often times even public things are only
shared between one or two participating servers. Public means two different things in these two
different networks.</p>
I have mused in the past about direct ATProto integration in Mastodon. I think it's possible, and I
think it's even a good idea to pursue. It would give mastodon users more reach if they want it.</em> My
prefered implementation of this would be the introduction of a fifth privacy setting.</p>

Direct - This post is only visible to mentioned users</li>
Followers Only - This post is only visible to mentioned users, and users that follow the poster</li>
Unlisted - This post is visible to anyone, but is left out of discovery mechanisms</li>
Public - This post is visible to anyone, and is included in discovery mechanisms</li>
Super Public - This post is visible to anyone, and is included in discovery mechanisms, and is
actively broadcast to literally anyone who happens to be listening on ATProto</li>
</ol>
Super Public is inherently a superset if Public. Not only can anyone see the post, and not only is
the post used to help users find each other and sent to relays and able to be forwarded farther in
the network, but it is sent directly to every single service that is listening to ATProto's
firehose. It greatly improves reach over just the Public option that Mastodon currently has, but
that comes at the cost of privacy.</p>
As it stands, Public posts on Mastodon (and other compliant ActivityPub impelementations) are still
somewhat private. Sure anyone can</em> see them, but realistically who will? Someone happening to check
the federated timeline on one of the 100 servers subscribed to my relay when I make the post? My
followers? If nobody boosts my public post then the total number of views it might get is probably
20. And certainly 3rd parties, who scrape and aggregate posts to gain more insight into users, are
far less likely to see even my public post on ActivityPub than any given post on ATProto.</p>
And you don't like the bridge?</h2>
No I don't like the bridge. I've already requested that my accounts be opted out, and provided a
suggestion for a middle ground that enables anyone to opt in at any time for any specific post. From
the ATProto side, bridging posts into ActivityPub comes at no additional disadvantage. All their
data is public already, and trivially accessible to anyone who cares to look. But from the
ActivityPub side, the bridge is introducing a new level of discoverability to a network that thusfar
hasn't had such a concept.</p>
I think it's critically important that developers of bridge software like this recognize</em> that what
they are doing is novel. Developers of bridge software need to acknowledge that they are changing
the experience of ActivityPub users by creating these bridges, even if nobody ever interacts with
them across the bridge. ActivityPub users need to understand as well that their reasonable
expectations of privacy can be violated at any moment by implementations such as this ATProto bridge
and the folks who would build and deploy such services.</p>
Extra Thoughts in no particular order</h2>

I tried not to make this post about Bluesky and Mastodon themselves, but to talk specifically
about the nature of the protocols involved.</li>
I have some experience building ActivityPub software outside of AodeRelay. I experimented with a
request-to-federate model in an image gallery platform in 2021</li>
Mentioning Jack Twitter isn't super important. From what I've gathered he's not very involved in
Bluesky or ATProto anymore.</li>
There exist bridges from ActivityPub to other platforms already (such as Nostr). I have these
bridges blocked from my mastodon server, not for privacy reasons but for moderation reasons.</li>
</ul>


Ugh Forgejo Actions
2024-02-10T00:00:00+00:00
Over the last few days I've figured out how to use Forgejo Actions. I was excited to try it since
it's integrated directly into Forgejo these days, and compatibility with github Actions means
there's already loads of third party actions to take advantage of. Previously I was using Drone CI.
I still am in a number of projects. But I'm hoping to get everything moved over to it over the next
week or so.</p>
But that's not why I'm writing this blog post. I'm writing this blog post because I haven't found
very many useful resources for understanding how Actions works, despite the prevalence github
Actions. All the documentation makes it look so simple to use, and in many cases I'm sure it is, but
when you're running it yourself, you'll find the edges.</p>
Getting Started</h2>
Setting up Forgejo Actions isn't a huge deal. I'm running it via docker-compose, but it can be run
natively or in kubernetes or with lxc (etc). The one crucial thing though is it needs access to a
docker daemon in order to operate. Giving it a Docker in Docker container is fine. Here's my DinD
section:</p>
1</span>s</span>ervices</span>:</span></span>
2</span>  d</span>ocker-in-docker</span>:</span></span>
3</span>    i</span>mage</span>:</span> d</span>ocker:dind</span></span>
4</span>    p</span>rivileged</span>:</span> true</span></span>
5</span>    c</span>ommand</span>:</span> [</span>"</span>dockerd</span>"</span>,</span> "</span>-H</span>"</span>,</span> "</span>tcp://0.0.0.0:2375</span>"</span>,</span> "</span>--tls=false</span>"</span>]</span></span>
6</span>    r</span>estart</span>:</span> a</span>lways</span></span></code></pre>
Simple enough? Now we're going to get a little more complicated with the Runner configuration. The
runner needs to be brought up in 2 steps. The first step registers the runner with Forgejo and
creates the runner configuration file. The second step launches the runner process. This is
unfortunately not as simple as launching the container, but it isn't too bad.</p>
 1</span>s</span>ervices</span>:</span></span>
 2</span>  #</span> docker-in-docker: ...</span></span>
 3</span></span>
 4</span>  F</span>orgejo-runner-1-register</span>:</span></span>
 5</span>    i</span>mage</span>:</span> c</span>ode.Forgejo.org/Forgejo/runner:3.3.0</span></span>
 6</span>    l</span>inks</span>:</span></span>
 7</span>      -</span> d</span>ocker-in-docker</span></span>
 8</span>    e</span>nvironment</span>:</span></span>
 9</span>      D</span>OCKER_HOST</span>:</span> t</span>cp://docker-in-docker:2375</span></span>
10</span>    v</span>olumes</span>:</span></span>
11</span>      -</span> /</span>storage/Forgejo-actions/runner-1:/data</span></span>
12</span>    u</span>ser</span>:</span> 0</span>:0</span></span>
13</span>    c</span>ommand</span>:</span> ></span>-</span></span>
14</span>      bash -ec '</span></span>
15</span>      if [ -f config.yml ]; then</span></span>
16</span>        exit 0 ;</span></span>
17</span>      fi ;</span></span>
18</span>      while : ; do</span></span>
19</span>        Forgejo-runner register --no-interactive --instance https://git.asonix.dog --name bluestar-runner-1 --token TOKEN && break ;</span></span>
20</span>        sleep 1 ;</span></span>
21</span>      done ;</span></span>
22</span>      Forgejo-runner generate-config > config.yml ;</span></span>
23</span>      sed -i -e "s|network: .*|network: host|" config.yml ;</span></span>
24</span>      sed -i -e "s|labels: \[\]|labels: \[\"docker:docker://bash:alpine3.19\"\]|" config.yml ;</span></span>
25</span>      chown -R 1000:1000 /data</span></span>
26</span>      '</span></span></code></pre>
This is the first step. The script loops attempting to register the runner with Forgejo if there
isn't an existing configuration file, and when it succeeds it writes the configuration file and
updates some values.</p>
This can be more-or-less copied verbatim, with the exception of TOKEN</code>, which needs to be copied
from the Forgejo actions admin panel. We'll come back to the config.yml file later. Next up we
actually run the runner.</p>
 1</span>s</span>ervices</span>:</span></span>
 2</span>  #</span> docker-in-docker: ...</span></span>
 3</span>  #</span> Forgejo-runner-1-register: ...</span></span>
 4</span></span>
 5</span>  F</span>orgejo-runner-1-daemon</span>:</span></span>
 6</span>    i</span>mage</span>:</span> c</span>ode.Forgejo.org/Forgejo/runner:3.3.0</span></span>
 7</span>    l</span>inks</span>:</span></span>
 8</span>      -</span> d</span>ocker-in-docker</span></span>
 9</span>    e</span>nvironment</span>:</span></span>
10</span>      D</span>OCKER_HOST</span>:</span> t</span>cp://docker-in-docker:2375</span></span>
11</span>    d</span>epends_on</span>:</span></span>
12</span>      F</span>orgejo-runner-1-register</span>:</span></span>
13</span>        c</span>ondition</span>:</span> s</span>ervice_completed_successfully</span></span>
14</span>    v</span>olumes</span>:</span></span>
15</span>      -</span> /</span>storage/Forgejo-actions/runner-1:/data</span></span>
16</span>    c</span>ommand</span>:</span> "</span>Forgejo-runner --config config.yml daemon</span>"</span></span>
17</span>    r</span>estart</span>:</span> a</span>lways</span></span></code></pre>
A lot less going on. We let the Forgejo runner access our Docker-in-Docker daemon and launch it
after the registration container finishes. Off to a great start.</p>
Using my basic "make sure it works" action that I cobbled together after reading some of the
documentation, we can make sure the runner works:</p>
 1</span>on</span>:</span> </span></span>
 2</span>  p</span>ull_request</span>:</span></span>
 3</span>  p</span>ush</span>:</span></span>
 4</span>    b</span>ranches</span>:</span></span>
 5</span>      -</span> m</span>ain</span></span>
 6</span>    t</span>ags</span>:</span></span>
 7</span>      -</span> "</span>v*.*.*</span>"</span></span>
 8</span></span>
 9</span>e</span>nv</span>:</span></span>
10</span>  B</span>INARY</span>:</span> e</span>xample</span></span>
11</span></span>
12</span>j</span>obs</span>:</span></span>
13</span>  t</span>est</span>:</span></span>
14</span>    r</span>uns-on</span>:</span> d</span>ocker</span></span>
15</span>    s</span>trategy</span>:</span></span>
16</span>      m</span>atrix</span>:</span></span>
17</span>        i</span>nfo</span>:</span></span>
18</span>          -</span> a</span>rch</span>:</span> a</span>md64</span></span>
19</span>          -</span> a</span>rch</span>:</span> a</span>rm64v8</span></span>
20</span>          -</span> a</span>rch</span>:</span> a</span>rm64v7</span></span>
21</span>    s</span>teps</span>:</span></span>
22</span>      #</span> We can run multiple bash commands, and for each item in our matrix!</span></span>
23</span>      -</span> r</span>un</span>:</span> e</span>nv</span></span>
24</span>      -</span> r</span>un</span>:</span> e</span>cho "${{ matrix.info.arch }} Good"</span></span>
25</span></span>
26</span>  t</span>est2</span>:</span></span>
27</span>    r</span>uns-on</span>:</span> d</span>ocker</span></span>
28</span>    c</span>ontainer</span>:</span></span>
29</span>      #</span> we can override the docker image, how fancy</span></span>
30</span>      i</span>mage</span>:</span> d</span>ebian:bookworm-slim</span></span>
31</span>    s</span>teps</span>:</span></span>
32</span>      -</span> r</span>un</span> :</span> e</span>cho "Hello, debian"</span></span>
33</span></span>
34</span>  t</span>est3</span>:</span></span>
35</span>    r</span>uns-on</span>:</span> d</span>ocker</span></span>
36</span>    c</span>ontainer</span>:</span></span>
37</span>      i</span>mage</span>:</span> d</span>ocker.io/asonix/rust-builder:latest-linux-arm32v7</span></span>
38</span>    s</span>teps</span>:</span></span>
39</span>      #</span> We can even compile rust code! It's amazing!</span></span>
40</span>      -</span> r</span>un</span>:</span> c</span>argo init --bin --name $BINARY</span></span>
41</span>      -</span> r</span>un</span>:</span> b</span>uild</span></span></code></pre>
This all runs successfully when a branch or a tag matching v*.*.* is pushed. We did it! We're
done!</del></p>

This is a deliberately simplified version of the setup I actually went through, which involved
bash</code> not existing at first.</p>
</blockquote>
Let's add it to an existing project (say, pict-rs</a>)</p>
 1</span>on</span>:</span> </span></span>
 2</span>  p</span>ush</span>:</span></span>
 3</span>  p</span>ull_request</span>:</span></span>
 4</span>    b</span>ranches</span>:</span></span>
 5</span>      -</span> m</span>ain</span></span>
 6</span></span>
 7</span>j</span>obs</span>:</span></span>
 8</span>  c</span>lippy</span>:</span></span>
 9</span>    r</span>uns-on</span>:</span> d</span>ocker</span></span>
10</span>    c</span>ontainer</span>:</span></span>
11</span>      i</span>mage</span>:</span> d</span>ocker.io/asonix/rust-builder:latest-linux-arm32v7</span></span>
12</span>    s</span>teps</span>:</span></span>
13</span>      -</span></span>
14</span>        n</span>ame</span>:</span> C</span>heckout pict-rs</span></span>
15</span>        u</span>ses</span>:</span> a</span>ctions/checkout@v4</span></span>
16</span>      -</span></span>
17</span>        n</span>ame</span>:</span> C</span>lippy</span></span>
18</span>        r</span>un</span>:</span> |</span></span>
19</span>          cargo clippy --no-default-features -- -D warnings</span></span>
20</span>          cargo clippy --no-default-features --features io-uring -- -D warnings</span></span></code></pre>
And run it!</p>
OCI runtime exec failed: exec failed: unable to start container process: exec: "node":</span></span>
executable file not found in $PATH: unknown</span></span></code></pre>
Oh.</p>
...what?</p>
Hmm...</p>
The Problems</h2>
So actions/checkout@v4 depends on node to run, but my rust builder container doesn't have node in it
so... I can't checkout my code? Well let's just split this up a bit, then.</p>
 1</span>on</span>:</span> </span></span>
 2</span>  p</span>ush</span>:</span></span>
 3</span>  p</span>ull_request</span>:</span></span>
 4</span>    b</span>ranches</span>:</span></span>
 5</span>      -</span> m</span>ain</span></span>
 6</span></span>
 7</span>j</span>obs</span>:</span></span>
 8</span>  c</span>lone</span>:</span></span>
 9</span>    r</span>uns-on</span>:</span> d</span>ocker</span></span>
10</span>    c</span>ontainer</span>:</span></span>
11</span>      i</span>mage</span>:</span> d</span>ocker.io/node:20-bookworm</span></span>
12</span>    s</span>teps</span>:</span></span>
13</span>      -</span></span>
14</span>        n</span>ame</span>:</span> C</span>heckout pict-rs</span></span>
15</span>        u</span>ses</span>:</span> a</span>ctions/checkout@v4</span></span>
16</span></span>
17</span>  c</span>lippy</span>:</span></span>
18</span>    n</span>eeds</span>:</span> [</span>c</span>lone</span>]</span></span>
19</span>    r</span>uns-on</span>:</span> d</span>ocker</span></span>
20</span>    c</span>ontainer</span>:</span></span>
21</span>      i</span>mage</span>:</span> d</span>ocker.io/asonix/rust-builder:latest-linux-amd64</span></span>
22</span>    s</span>teps</span>:</span></span>
23</span>      -</span></span>
24</span>        n</span>ame</span>:</span> C</span>lippy</span></span>
25</span>        r</span>un</span>:</span> |</span></span>
26</span>          cargo clippy --no-default-features -- -D warnings</span></span>
27</span>          cargo clippy --no-default-features --features io-uring -- -D warnings</span></span></code></pre>
Except that doesn't work. The cloned repo doesn't stick around between jobs. If we had more than one
runner the jobs might not even run on the same one! We could try solving this with artifacts but
wait... actions/download-artifact@v4</code> also depends on node, so we can't run it in the rust-builder
container.</p>
So we have Actions but we can't use them. How does github handle this? Well github's answer is to
install anything you could ever need into their default actions containers. Meaning node, go,
python, ruby, docker (oh, docker... we'll need that too) and more are all bundled into a 60GB image.
If you remember when we had a script that ran sed</code> on the config earlier...</p>
sed</span> -</span>i</span> -</span>e</span> "</span>s|labels: \[\]|labels: \[</span>\"</span>docker:docker://bash:alpine3.19</span>\"</span>\]|</span>"</span> config.yml</span> ;</span></span></code></pre>
We were setting a value in the Forgejo runner's config to provide our runners with default
containers. We started with bash</code> on alpine</code>. While that particular container is very small and
doesn't take long to download, it doesn't contain the majority of things that actions expect to
exist.</p>
In order to build pict-rs we need rust, and not just any rust but a rust that is capable of
cross-building (I target armv7, aarch64, and x86_64 with musl libc). In order to clone pict-rs, use
the actions cache, use the actions artifacts, and more we need nodejs. In order to build docker
containers we need docker and qemu. The container image we need to have or make to run our CI is now
nontrivial.</p>
As a proof of concept, I first wrote my actions where I installed everything by hand. I started with
the node:20-bookworm image from dockerhub. I had steps to apt-get install docker, download rustup
and install it, add the proper targets, add clippy, add cargo-binstall,
cargo binstall cargo-zigbuild</code>, and install zig. While this worked, it took a while just in the
setup phase, which isn't what we want for CI.</p>

My previous CI for pict-rs used my rust-builder</code> image, which doesn't actually use
cargo-zigbuild. I opted to use zig's linker for pict-rs in Forgejo Actions because I knew it
would be easier than manually constructing a cross-compile environment. It would also enable me
to use a single container to build for any platform, rather than my previous CI which had a
unique container for each platform I targeted.</p>
</blockquote>
So I wrote a bit of caching. I had used an action to install zig, and that action cached zig on the
runner. I wrote my own caching layer for all the rustup and cargo bits. That sped things up as well,
but it still meant using space in the runner cache, and potentially installing everything again on a
cache miss. In this process, I also hit the github rate limit for downloading cargo-zigbuild via
cargo-binstall, meaning I had to start compiling it from crates.io instead (which doesn't take too
long, but it's still longer than downloading a binary).</p>

As an aside, I had to set DOCKER_HOST: tcp://docker-in-docker:2375</code> as an environment variable in
the runner config.yml</code> file so that my use of docker in the actions container would find my
docker-in-docker daemon.</p>
</blockquote>
Giving In</h2>
I decided I needed a universal base container image to run my CI the way github does it, because
it's the only way that makes sense for their CI system. If all the actions people write are going to
expect me to have things installed, then I better have them installed. You can find the actions
workflow I use to produce my base image in my
actions-base-image repository</a>. I'm sure that in
the future I will encounter more actions that fail to run on this image and I will need to update it
to add more dependencies.</p>
I also wrote another caching action, simpler now than before since all the rust, zig, and docker
bits are baked into the base image. You can find it here</a> in
the cache-rust-dependencies</code> folder. It's extremely basic but saves me a download from crates.io
for each job pict-rs runs.</p>
So?</h2>
Am I happy with Forgejo Actions? Not really. I think the design of Actions is pretty bad for the
Forgejo Actions case where you control the runners yourself. On github it's fine, since github
manages the 60GB image behind the scenes where you never need to think about it. Outside of github
it's less ideal. I'm still going to migrate all my projects to it now that I have everything
working and a not-too-big base image for myself (it's 1GB).</p>
I hope anyone else struggling with Actions (github, gitea, forgejo, or otherwise) gains some insight
from this post. It's not like I built this blog in the first place to be able to put this online or
anything. Let's hope now that I've written all this that I don't need to update my base image to
publish this to garage</a>.</p>

I ended up adding minio-client</code> to my base image in order to publish this</p>
</blockquote>

Cross Compiling Rust with Nix

Moving

Help! My function is Crimson!

What is Public?

Ugh Forgejo Actions

The Problems</h2> So actions/checkout@v4 depends on node to run, but my rust builder container doesn't have node in it so... I can't checkout my code? Well let's just split this up a bit, then.</p>