Advanced package transformations in manifests and Derivations introduction

In the last post we looked at how to create package variants using the package transformation capabilities in Guix's API. We're going to continue that topic, creating package variants by using the projects git repository or it's latest source release. The final transformation (with-patch) alters the way that a package is built, presenting a good opportunity to explore how Guix runs builds and creates intermediary steps in a build (called Derivations). As a reminder we're recreating all the transformations using the API that we previously created (in part 2) using the command line tools - by the end we'll have created each transformation using the CLI and programmatically in a manifest.

This post is one of a series on Guix packaging:

Git transforms

We're going to update fzf, doing the equivalent of the command line transformation from our command line blog post:

guix build fzf --with-git-url=fzf= --no-substitutes --no-grafts

Create a manifest file called fzf-local.scm, with the following contents:

1  (use-modules (guix profiles)
2               (guix transformations)
3               (gnu packages terminals) )
5  (define fzf-transform (options->transformation
6                          (list '(with-git-url . "fzf="))))
8  (packages->manifest (list (fzf-transform fzf)))

Conceptually, this is the same as we used previously, it's just been condensed a bit since we're becoming more familiar with the layout. Starting on line 5 a variable is defined called fzf-transform, this has two forms in it so read it from inside to outside. The inside form creates a list which has a single pair in it, this time we're using with-git-url and we provide the spec as text. The outer form is the normal options->transformation function which accepts the list that we've created as an argument. Line 8 is the package->manifest function which consumes a package in a list, in this case after the transformation's been applied to the fzf variable.

Rather than passing the manifest to guix shell, lets build it this time - as this gives a bit more information about the build:

$ guix shell --container --nesting --development fzf --network --no-grafts nss-certs

[env]$ guix build --manifest=fzf-local.scm --no-substitutes --no-grafts
Starting download of /tmp/guix-file.cwxHbg
The following derivation will be built:
building /gnu/store/kv7s0k4092fps75337d72d0wxnzp6984-fzf-0.41.0.drv...
[... lots of build output ...]
successfully built /gnu/store/k168q8rmx5xxfl46lnhd5g03n7p66bxm-fzf-0.41.0.drv

To use with-branch and with-commit we have to adjust the fzf-transform:

(define fzf-transform (options->transformation
                        (list '(with-git-url . "fzf="
                              '(with-commit . "fzf=2024010")))))

We can have more than one transformation as we're sending a list to options->transformation. A list is a collection, so in this case we have two items in the list: each one of these is a pair.

When we build this in our shell:

successfully built /gnu/store/1qvcj1673i0pxdp7s1zrsg9jv78kps80-fzf-git.2024010.drv

[env]$ guix package --install /gnu/store/34x5w0kn7yg8a7ggcfc6ldalj7iyfy4z-fzf-git.2024010
[env]$ guix package --list-installed
fzf     git.2024010     out     /gnu/store/34x5w0kn7yg8a7ggcfc6ldalj7iyfy4z-fzf-git.2024010

I'm going to skip the input example that we did in the transformations command line tool post - see if you can figure it out!

Latest transformation

We previously used --with-latest from the command line to build the latest version of Weechat, lets see how we can do it through the API in a manifest file.

The manifest file (weechat-latest.scm) is as follows:

(use-modules (guix profiles)
             (guix transformations)
             (gnu packages irc))

(define weechat-transform (options->transformation
                        (list '(with-latest . "weechat"))))

(packages->manifest (list (weechat-transform weechat)))

We've seen previously that when we use the with-latest transformation Guix will attempt to verify the command using gpg which has to be in the users profile. To test that the transformation runs correctly do either:

$ option 1: run the manifest as part of creating a shell
$ guix shell --container --nesting --network --no-grafts --no-substitutes --manifest=weechat-local.scm
[env]$ weechat --version

# option 2: create a shell then build - more detailed logs
guix shell --container --nesting --development weechat --network --no-grafts --share=/home/<user> nss-certs gnupg
[env]$ guix build --manifest=weechat-local.scm --no-substitutes --no-grafts

Derivations detour

The next two transformations to look at alter the way a build is executed, they are with-configure-flags and with-patch. During the transformation the source or build derivations are changed: to get a better sense of what's happening lets look at Derivations.

When we define a package and tell guix build to create it, there's a sequence of steps that create Derivations. The manual says:

"A derivation is a low-level representation of the build actions to be taken, and the environment in which they should occur—derivations are to package definitions what assembly is to C programs. The term “derivation” comes from the fact that build results derive from them." Guix Manual: Programming interface

When a guix build completes successfully we see both the package output and the build's derivation. For example the output of a guix build mutt:

successfully built /gnu/store/j2l3rzaf7r31qqbyjvcv1pfi1piz6c6m-mutt-2.2.12.drv

We can look at the Derivation to see what the output of the build has created, the inputs and information about the build environment. Using cat is the easiest:

$ cat /gnu/store/fw5b8hakgdlwn9qkg5kp3z7kgklv9zhj-mutt-2.2.12

# if we want reformat the derivation then using vim is nice
$ cat /gnu/store/fw5b8hakgdlwn9qkg5kp3z7kgklv9zhj-mutt-2.2.12 | vim -

Derivations are low-level so they are pretty messy to look at - we're looking at Guile Scheme data represented as text. The details of the <derivation> record are in (guix derivations) module. A derivation record has the following fields:

    <derivation file name>

Here's the Mutt example that I've tried to format so it's easier to see:


    [   ;; (1) derivation-outputs field

    [   ;; (2) derivation-inputs field

    [   ;; (3) derivation-sources field

    ;; (4) derivation-system field

    ;; (5) derivation-builder field

    [   ;; (6) derivation-builder-arguments field

    [   ;; (7) derivation-file-name field

As we can see the derivation-outputs field contains a list, which specifies what the build can output. The second field, derivation-inputs, has any input derivations that have to be built as part of building this derivation. The derivation-sources field is a list with inputs that don't require a derivation to be built - this contains the package builder file which has the source paths in it. The fifth field is the derivation-system, commonly x86_64-linux. The next field is the derivation builder, which is the instance of guile that will perform the build. The sixth field derivation-builder-arguments shows all the arguments that were provided to the Guile script: including the source paths from the store wrapped in a file - here the package builder file is called mutt-2.2.12-builder.

Diagram of a Guix packages derivations

Figure 1: Guix build action Derivations

My mental model for how the different inputs and steps in the build come together is in Figure 1. The inputs are the source (from origin in the package definition), and any patches (from the patches field), the rest of the fields in the <package> definition impact things like the builder selected and the builder arguments provided.

We can use the build's output Derivation to inspect different parts of the build. For example, we could look at the derivation inputs field and then go back a step to check one of the input's build.

The main item I often want to look at is the package build's details (in the package builder file), to check any arguments that went to the build, or to look at the source so I can check if a patch was applied. The thing to look for is the <package>-<version>-builder in the derivation-builder-arguments field: in this example it's "/gnu/store/n4c747f2019lf6fvad4ylivd0fydp308-mutt-2.2.12-builder". If we look at this we see all the elements that were passed into the build process.

The source for the build is one of the first items:

(("source" . "/gnu/store/n9sr83y8y3r18a4m3hbb1hajabr9ipw6-mutt-2.2.12.tar.xz")

This is also a Derivation that was created when the source was downloaded and any patches applied. To look at the log we'd do:

$ guix build --log-file /gnu/store/n9sr83y8y3r18a4m3hbb1hajabr9ipw6-mutt-2.2.12.tar.xz

For more detailed information about Derivations start with this fantastic Dissecting Guix blog post.

Patch transformation

Alright, it's time to create the equivalent of a --with-patch transformation. Create a manifest called mutt-local.scm with the following contents:

(use-modules (guix profiles)
             (guix transformations))

(define mutt-transform (options->transformation
                        (list '(with-patch . "mutt=./882690-use_fqdn_from_etc_mailname.patch"))))

(packages->manifest (list (mutt-transform (specification->package "mutt"))))

To build it we create a shell and download the patch - this is exactly the same steps as the previous post that did this using the Guix command line:

$ guix shell --development mutt --container --nesting --network --no-grafts nss-certs coreutils curl

[env]$ curl --remote-name \

[env] guix build --manifest=mutt-local.scm --no-substitutes --no-grafts
successfully built /gnu/store/j2l3rzaf7r31qqbyjvcv1pfi1piz6c6m-mutt-2.2.12.drv

Having built the package we need to check the derivation to make sure our build actually used the patch. Which is why we splunked through Derivations above!

⚠️ WARNING: Look at derivations in a normal shell/split - not in a guix container as for some reason it will not show the local builds if you try to look at them in a container.

First, look in the output derivation to find the mutt-builder file's path:

$ cat /gnu/store/j2l3rzaf7r31qqbyjvcv1pfi1piz6c6m-mutt-2.2.12.drv

At the end of the file is the part where it shows what was sent to the compiler (Guile) to build the binary package. Look towards the end of the output and find the reference to the builder (mutt-builder) derivation:


We can look inside this with:

$ cat /gnu/store/n4c747f2019lf6fvad4ylivd0fydp308-mutt-2.2.12-builder

The builder derivation has all the details of what was sent to the build process, including the source derivation. Look for something like this

gnu-build #:source "/gnu/store/n9sr83y8y3r18a4m3hbb1hajabr9ipw6-mutt-2.2.12.tar.xz"

We can now check the build log for this source derivation:

$ guix build --log-file /gnu/store/n9sr83y8y3r18a4m3hbb1hajabr9ipw6-mutt-2.2.12.tar.xz

$ zless /var/log/guix/drvs/il/5ca6lyl78w3b4s5xv1dgl3hx9mq4ar-mutt-2.2.12.tar.xz.drv.gz

To find the patches being applied grep for applying \'/gnu:

applying '/gnu/store/92w6h5vqzz8cvi2x2d2xpl1y35dbh4c9-882690-use_fqdn_from_etc_mailname.patch'...

A source derivation is created from the pristine source, and any patches that are being applied. As we can see our patch has been applied, so the transformation works!

Multiple packages

We use manifests when we want to install multiple packages into a shell environment or profile. We can create multiple different transformations in a manifest, like so:

 1  (use-modules (guix profiles)
 2               (guix transformations))
 4  (define fzf-transform (options->transformation
 5                          (list '(with-git-url . "fzf=")
 6                                '(with-commit . "fzf=2024010"))))
 8  (define calcurse-transform (options->transformation
 9        '((with-source . "calcurse@20231207="))))
11  (packages->manifest
12    (append
13      (specifications->packages (list "hello" "cbonsai" "gnupg"))
14      (list (fzf-transform (specification->package "fzf"))
15            (calcurse-transform (specification->package "calcurse")))))

We're applying transforms fzf-transform and calcurse-transform to two packages. The only noteworthy difference between the two is that the first (fzf-transform) uses list, while the second creates a list with parentheses and uses a quote to escape it '( ... ).

The packages->manifest function (starting on line 11) uses the Guile function append() to join two different lists together: the first is a specification->packages list that doesn't have any transformations, the second is a list with the two transformed packages.

To test it out do:

$ guix shell --container --nesting --network --no-grafts nss-certs
[env]$ guix build --manifest=multiple-transforms.scm --no-substitutes --no-grafts
[env]$ guix package --manifest=multiple-transforms.scm --no-substitutes --no-grafts
[env]$ guix package --list-installed
calcurse        20231207        out     /gnu/store/5cdy09016ndd9ifw9knx048gc3nb2p0q-calcurse-20231207
fzf             git.2024010     out     /gnu/store/34x5w0kn7yg8a7ggcfc6ldalj7iyfy4z-fzf-git.2024010
gnupg           2.2.39          out     /gnu/store/yr0f3jl7zaqjkibc4x6ymyp4z1mm7qdv-gnupg-2.2.39
cbonsai         1.3.1           out     /gnu/store/mgc2i6yxm2zbqf8yx8x5f4ig4nbii2cv-cbonsai-1.3.1
hello           2.12.1          out     /gnu/store/5mqwac3zshjjn1ig82s12rbi7whqm4n8-hello-2.12.1

That works - we've installed multiple packages using a manifest, some of which have transforms applied.

Multiple manifests

In some situations we want to combine (or re-use) manifests. The most common situation is if we want to create a development environment, so the equivalent of:

guix shell --development mutt --container vim git fzf

We can do this using the concatenate-manifests() and package->development-manifest(). Create a manifest called multiple-manifests.scm with these contents:

 1 (use-modules (guix profiles)
 2              (guix transformations))
 4 (concatenate-manifests
 5   (list
 6     (package->development-manifest
 7       (specification->package "mutt"))
 8     (packages->manifest
 9       (specifications->packages (list "vim" "git")))))

Reading the manifest from the inside to the outside, we have package->development-manifest manifest (line 6) which is created by providing the mutt package. There's also a packages->manifest manifest (line 8) that consists of two packages vim and git. These two manifest records are wrapped in a list (line 5), and fed to concatenate-manifests which accepts a list and returns a single manifest.

To avoid locally building all of Mutt's development inputs (along with Vim and Git), install them into a shell first. Doing this without --no-substitutes will pull them down and put them into the local Guix store:

$ guix shell --development mutt --container --network vim git
[env] exit

Now we're ready to test the manifest with:

$ guix shell --container --nesting --network --no-grafts nss-certs
[env]$ guix build --manifest=multiple-manifests.scm --no-substitutes --no-grafts
[env]$ guix package --manifest=multiple-manifests.scm --no-substitutes --no-grafts

For our last step, we're going to add a transformation - we'll add fzf from git to the packages that are installed:

 1  (use-modules (guix profiles)
 2               (guix transformations))
 4  (define fzf-transform (options->transformation
 5                          (list '(with-git-url . "fzf=")
 6                                '(with-commit . "fzf=2024010"))))
 8  (concatenate-manifests
 9    (list
10      (package->development-manifest
11        (specification->package "mutt"))
12      (packages->manifest
13        (cons
14          (fzf-transform (specification->package "fzf"))
15          (specifications->packages (list "vim" "git"))
16        )
17      )
18    )
19  )

The fzf-transform part is the same as before. Looking at concatenate-manifests it's the same for adding the development manifest and packages manifest. It's the packages->manifest part that's changed (line 12 onwards). It has the normal specifications->packages with our list of vim and git - this function returns a list which is key. One line above it we have the fzf-transform. We use Guile Scheme's cons() function (line 13) to add the output of the fzf-transform to the list. Cons is a commonly use function in Scheme, it can add an item to a list (if the items is in front of the list:

(cons 1 '(2 3))  ;=> '(1 2 3)

Test this manifest and check that it installs Mutt's development inputs, along with Vim, Git and a transformed Fzf.

Renaming packages

I finally managed to work out how to create package variants with a different name when using transformations. I couldn't find any examples of this being done using package transformations, so perhaps it's not useful.

The normal reason for renaming local builds is that it prevents the package manager upgrading the variant (e.g. guix upgrade). It also gives us a way to see which packages we've altered on the system.

Guix's command-line tooling already tracks packages transformations:

"Package transformation options are preserved across upgrades: guix upgrade attempts to apply transformation options initially used when creating the profile to the upgraded packages." (Guix manual transformation options)

This means package renames aren't needed, since Guix will keep transformations through an upgrade.

Even when using manifests package renames aren't strictly necessary. For a start I generally don't do a guix upgrade, but instead reapply the manifest. We can ensure Guix doesn't change a package variant by specifying the version of the package (e.g. <package>@<version>) and using a transformation that supports this: for example, using with-source or with-git-url with a commit. That way we'll only install the version that we've specified.

Nonetheless, I like being able to rename the package to something that clearly tells me I've altered it from the standard build. The first step is to create a manifest file called calcurse-rename.scm, with the following contents:

(use-modules (guix profiles)
             (guix transformations)
             (guix packages)
             (gnu packages calcurse))

(define calcurse-transform
        (list '(with-source . "calcurse=")))

(define calcurse-local (package/inherit
                         (version "4.5.0")))

(packages->manifest (list calcurse-local))

I couldn't quite get this to work as one piece of code - and it would probably be hard to explain anyway. Starting at the bottom, our last step is converting a package to a manifest. The calcurse-local define is new, this uses a function from (guix package) called package/inherit. this allows us to create a new package by inheriting an existing package record: we can alter the new record at creation. In this case, we create the new package by inheriting the output of calcurse-transform and give it a new version.

Lets look at how a locally build package can be upgraded by Guix:

$ guix shell --development calcurse --container --nesting --network --no-grafts nss-certs

[env]$ guix build --manifest=calcurse-rename.scm --no-substitutes --no-grafts
The following derivation will be built:
building /gnu/store/35ab4bznc9pxybzyx267ar81hyf5pxmd-calcurse-4.5.0.drv...

successfully built /gnu/store/35ab4bznc9pxybzyx267ar81hyf5pxmd-calcurse-4.5.0.drv

Check through the logs and we can see it downloading the source of Calcurse 4.5.0 from the upstream's site. Now we install this in the shell, and then ask Guix to upgrade:

[env]$ guix package --install /gnu/store/c369d6ky364jddbkin0i00dnwb8hfblk-calcurse-4.5.0

[env] guix package --list-installed
calcurse        4.5.0   out     /gnu/store/c369d6ky364jddbkin0i00dnwb8hfblk-calcurse-4.5.0

[env]$ /home/steve/.guix-profile/bin/calcurse --version
calcurse 4.5.0 -- text-based organizer

If we ask Guix to upgrade then it will upgrade the package as version 4.5.1 is in it's archive:

[env]$ guix upgrade

The following package will be upgraded:
   calcurse 4.5.0 -> 4.5.1

[env]$ guix package --list-installed
calcurse        4.5.1   out     /gnu/store/sf2dk1y33gls2s00217qxh0kds20k26n-calcurse-4.5.1

[env]$ /home/steve/.guix-profile/bin/calcurse --version
calcurse 4.5.1 -- text-based organizer

We can avoid this problem by calling our altered package by a name that isn't used within the Guix archive. Alter the manifest and add a name:

(use-modules (guix profiles)
             (guix transformations)
             (guix packages)
             (gnu packages calcurse))

(define calcurse-transform
        (list '(with-source . "calcurse=")))

(define calcurse-local (package/inherit
                         (name "calcurse-local")
                         (version "4.5.0")))

(packages->manifest (list calcurse-local))

To test this we first need to exit the guix shell that we used for testing so that calcurse isn't installed in any profile.

Now we can test the new package in a new clean shell environment:

$ guix shell --development calcurse --container --nesting --network --no-grafts nss-certs

[env]$ guix build --manifest=calcurse-rename.scm --no-substitutes --no-grafts
successfully built /gnu/store/1n45hjdgddbc0an5mdqf9xk5v95pwrmj-calcurse-local-4.5.0.drv

[env]$ guix package --install /gnu/store/6s79izjl1c6yc4mik0qfzyp8sba6apf9-calcurse-local-4.5.0

[env] guix package --list-installed
calcurse-local  4.5.0   out     /gnu/store/6s79izjl1c6yc4mik0qfzyp8sba6apf9-calcurse-local-4.5.0

[env]$ /home/steve/.guix-profile/bin/calcurse
calcurse 4.5.0 -- text-based organizer

By inspecting the build it's clear that we're installing 4.5.0, and the package is now called calcurse-local. As Guix doesn't have a package called calcurse-local in the archive it won't accidentally upgrade the package. In fact, if we try to upgrade it Guix tells us it doesn't have a package in the archive called calcurse-local:

$ guix upgrade
guix upgrade: warning: package 'calcurse-local' no longer exists

Transforms summary

That's package transformations done - phew! If you want more (really!?) see defining package variants in the manual and look at the available transformations in the (guix transformations) module. The area we haven't explored is tuning - which looks very cool!

Between using manifests and package renames there's now an easy way to see that transformations are being maintained correctly. The alternative to using manifests with transformations is to start writing our own package definitions - but that's for another day!

Are you using transformations, if so on what packages and why? Are they a hidden gem or too limited - I'd love to hear what you think!

Posted in Tech Friday 08 December 2023
Tagged with tech ubuntu guix