Modifying Guix packages using package transformations

Ever wanted to change a package in Guix to use newer source code? Today we go beyond rebuilding, to altering packages. Examples of altering packages are using different source code, changing the libraries a package uses, or to adding a feature through a patch. It all comes under our second scenario Modifying an existing package. In this post we'll cover how to use the Guix command line tool's capabilities to create package variants.

This post is one of a series on Guix packaging:

A package variant is one where our local build of a package is different from the one in the Guix archive. There are a few ways to create them:

We're going to cover the first of these, modifying existing packages by sending some command line options to guix build. The package transformations options are a great way to start creating package variants as using the command-line is straightforward, and there's a clear scope to what we can do. While they don't give us the full power of creating a package definition, they do allow us to create variants without having to understand package recipes.

The Guix manual has extensive coverage of all the Package Transformation Options with examples. I'm going the cover the main capabilities.

Changing a packages source

The --with-source package transformation is used to build a package using different source code. The source code can either be the source code for the package that we're building, or an input that's used as part of the build (e.g. a library that's needed).

The simplest invocation is:

guix build <package> --with-source=<source location>

This builds the <package> using the source that is located at <source location>. The source location can be a https location or a local file. It infers what package the <source location> is pointing at by looking at the source files: for example if the source location ended with calcurse-4.8.1.tar.gz it will infer that the package the source is associated with is calcurse and that the version is 4.8.1.

As I write this post the version of Calcurse in the Guix archive is 4.5.1, and the upstream version is 4.8.1. We'll use the --with-source package transformation to create a local package variant that contains the newer upstream version.

The first step is to create a build environment which contains the runtime inputs (build dependencies) that Guix already knows about for the Calcurse package: this will give us a build environment that has all the libraries and compilers we need to build the Calcurse software. We include the nss-certs package so that Guix can download the source over HTTPS:

$ guix shell --container --nesting --network --development calcurse --no-grafts nss-certs


[env]$ guix package --profile=$GUIX_ENVIRONMENT --list-installed
[... shows all the development inputs ...]

The last command lists all the packages that were pulled into the shell environment. As we can see all the different inputs (build dependencies) have been installed into the environment for us.

We're ready to build the later version of Calcurse:

[env]$ guix build calcurse --with-source= \
  --no-substitutes --no-grafts
The following derivation will be built:
building /gnu/store/5p8nf0ab5xyw612waj9s1fjlsrpvmx2m-calcurse-4.8.1.drv...
[... lots of output ...]

successfully built /gnu/store/5p8nf0ab5xyw612waj9s1fjlsrpvmx2m-calcurse-4.8.1.drv

As a reminder the --no-substitutes option tells guix build to build the package rather than download a binary substitute; while the --no-grafts option means that it should not graft changes onto an existing binary in the store.

In this form we've told guix build where to get the source from, it then infers the package name (calcurse) and the version (4.8.1) from the source file name.

At the end of the build Guix gives us two pieces of information:

  • the build's derivation (the .drv file). It tells us what was built and the inputs used in the build environment.
  • the actual package that has been placed in Guix's store.

To look at the build log, in a separate split/terminal, do:

$ guix build --log-file /gnu/store/af4nwvbcd8rbix4vcvamblmbf3ns9wsz-calcurse-4.8.1

⚠️ WARNING: The guix build --log-file command has to be done in a normal environment and not in the nested Guix container environment.

Finally, switch back to the build container and install for a quick test:

[env]$ guix package --install /gnu/store/af4nwvbcd8rbix4vcvamblmbf3ns9wsz-calcurse-4.8.1
The following package will be installed:
    calcurse 4.8.1
[lots of output as it installs it]

[env]$ /home/steve/.guix-profile/bin/calcurse --version
calcurse 4.8.1 -- text-based organizer

[env]$ guix package --list-installed
calcurse        4.8.1   out     /gnu/store/af4nwvbcd8rbix4vcvamblmbf3ns9wsz-calcurse-4.8.1

As usual we add the package to the build environment by specifying the location in the Guix Store ensuring that it installs the version we just built. Then we can test it by directly referencing the command, or by sourcing the GUIX_PROFILE.

Great, we can build variants of packages using different source code.

The main thing that we can't do is alter the name of the package, so we always land up with a package from the archive built with some alterations and called the same thing. For example, if we run guix build calcurse --with-source=./calcurse-4.8.1.tar.gz` our output package is calcurse and we can't have calcurse-oops-I-messed-with-it or some other name!

Changing package version

We can alter the version of a package variant like so:

guix build <package> --with-source=<package>@version=<source location>

To provide the package version we also provide the package name: the package version can be anything with numbers separated by dots. The version field is constrained to numbers and dots, so 1.2.3 works, but not 5:1.2.3 or 1.2~something. Lets change our package variant to add an indicator of the number of times we've had to build the package locally:

$ guix shell --container --nesting --network --development calcurse --no-grafts nss-certs

[env]$ guix build calcurse --with-source=calcurse@ \
  --no-substitutes --no-grafts
[... lots of output ...]
successfully built /gnu/store/0q23cbckfffiifzvb2kdy2jd0agv1h9d-calcurse-

[env]$ guix package --install /gnu/store/jg8sz5l60igzxddnqqxd0rh9370h4n3y-calcurse-

[env]$ guix package --list-installed
calcurse        out     /gnu/store/jg8sz5l60igzxddnqqxd0rh9370h4n3y-calcurse-

This is just a silly example, it's probably not useful to show the number of times a build has happened! However, it demonstrates how to change version numbers to whatever we want!

Changing package inputs

All packages have inputs - all the software that is required to build it. A common use-case is that we want to build and test an application using a later version of a library (an input). I'm carefully using the word input as a more precise word than dependency as what we're looking at is software that is needed during the build: in other packaging communities this is referred to as a build dependency. We can use the --with-source package transformation to create a package variant that does this:

guix build <package> --with-source=<input package>@<version>=<input package source location>

In this invocation, one of the inputs to the build of <package> is <input package>, and here we define a different <input package source location>.

For a concrete example, examine the jnettop package with guix edit jnettop. The package definition looks like this:

(define-public jnettop
    (name "jnettop")
    (version "0.13.0")
    (source (origin
              (method url-fetch)
               (string-append ""
                              version ".tar.gz"))
    (build-system gnu-build-system)
     (list pkg-config))
     (list glib ncurses libpcap))
    (synopsis "Visualize network traffic by bandwidth use")
     "Jnettop is a traffic visualiser, which captures traffic going
through the host it is running from and displays streams sorted
by bandwidth they use.")
    (license license:gpl2+)))

There's an inputs form which has (list glib ncurses libpcap) - this tells us the package needs glib, ncurses and libpcap as part of the build process. A good way to see this is:

$ guix shell --container --nesting --development jnettop

[env]$ guix package --profile=$GUIX_ENVIRONMENT --list-installed
[... long list of packages required for the build environment ...]
libpcap                 1.10.1                  out     /gnu/store/lvr4riwkc4qq8l5a38qvlik2mxa9yy4k-libpcap-1.10.1
ncurses                 6.2.20210619            out     /gnu/store/d3ai01pzxwsavn59g6b8glzn4x3sxb4j-ncurses-6.2.20210619
glib                    2.72.3                  out     /gnu/store/9vig86rkgn8ggpwzxcvyxdavbxp0jig2-glib-2.72.3

This demonstrates that these three packages (along with others) are part of the dependency graph and have been placed into the build environment which was created by --development jnettop.

We're going to create a package variant of Jnettop that uses a later version of Libpcap as an input. At the time of writing this the Guix archive has libpcap version 1.10.1, while the upstream version has moved to 1.10.2.

First we check whether we can build the later version of libpcap using the Guix archives current package definition:

$ guix shell --container --nesting --network --development libpcap --no-grafts nss-certs
[env]$ guix build libpcap --with-source= \
  --no-substitutes --no-grafts
[... lots of build output ...]
successfully built /gnu/store/hxnz8k6hp8ccd041vks2hx0hhxqal3wk-libpcap-1.10.2.drv

Success! We can now build jnettop using the later input. To do that we use --with-source, we tell it we want to rebuild libpcap and use it as part of building jnettop:

$ guix shell --container --nesting --network --development jnettop --no-grafts nss-certs

[env]$ guix build jnettop --with-source=libpcap@1.10.2= \
  --no-substitutes --no-grafts
Starting download of /tmp/guix-file.ZW6FkD
The following derivations will be built:
building /gnu/store/ynhj7y4g5cs0jl2jdw5dqdp6zzhyn87r-jnettop-0.13.0.tar.gz.drv...
[... lots of output as it builds libpcap and jnettop]

successfully built /gnu/store/vi2yiirwxj5irx7wgrwxpkxvs58gn4am-jnettop-0.13.0.drv

It's so cool that we can build a dependency and use it in the graph like this! This invocation of --with-source says we want to build libpcap@1.10.2 using the source at <source location>. Then, in the context of building jnettop it knows that libpcap is an input into the jnettop build. Consequently, the result is that it builds the new libcap, and then builds jnettop using the new version as an input.

In a different split or terminal (not a container) it's worth looking at the build logs:

$ guix build --log-file /gnu/store/3b5x5x8w0j0id20n1pbzamyn31s1j1b4-jnettop-0.13.0
$ less /var/log/guix/drvs/vi/2yiirwxj5irx7wgrwxpkxvs58gn4am-jnettop-0.13.0.drv.gz

# we can also see the libpcap build
# first look at the jnet derivative which has a reference to the libpcap derivative
# then we an use this to get the build log
$ cat /gnu/store/vi2yiirwxj5irx7wgrwxpkxvs58gn4am-jnettop-0.13.0.drv
guix build --log-file /gnu/store/hxnz8k6hp8ccd041vks2hx0hhxqal3wk-libpcap-1.10.2.drv
zless /var/log/guix/drvs/hx/nz8k6hp8ccd041vks2hx0hhxqal3wk-libpcap-1.10.2.drv.gz

We can see jnettop and by looking at it's derivation we can find the libpcap derivation to look at it's build log. Without going into detail it's worth knowing that every package build is transformed into a derivation, so when we look at a derivation we're seeing the various outputs of the build process.

As jnettop needs root access to the network interfaces to listen to them, we have to test in a standard shell environment:

# start a shell that's not a 'container'
$ guix shell coreutils

# install the package into a temporary profile so that the main profile is not polluted
[guix-dev]$ guix package --profile=./tmp-profile --install /gnu/store/3b5x5x8w0j0id20n1pbzamyn31s1j1b4-jnettop-0.13.0

# run the program as the root user so that it can access the interfaces
[guix-dev]$ sudo /home/steve/tmp-profile/bin/jnettop

# to clean up exit the shell and remove the profile files
$ rm tmp-profile

An alternative way of doing this would be to use the --with-input package transformation, but this requires a local package which we're not covering in this post.

Transforming using git

Lots of projects don't release source file archives (e.g. .tar.gz), or we may want the cutting edge - by building the source out of the git repository. For this Guix provides the --with-git-url package transformation, which we can think of as a partner to --with-source.

We're going to build fzf, a general-purpose command-line fuzzy finder. We can see the package definition by using guix edit go-github-com-junegunn-fzf:

(define-public go-github-com-junegunn-fzf
    (name "go-github-com-junegunn-fzf")
    (version "0.41.0")
       (method git-fetch)
       (uri (git-reference
             (url "")
             (commit version)))
       (file-name (git-file-name name version))
    (build-system go-build-system)
     `(#:import-path ""))
     (list go-github-com-mattn-go-runewidth
    (home-page "")
    (synopsis "Command-line fuzzy-finder")
    (description "This package provides an interactive command-line filter
usable with any list--including files, command history, processes and more.")
    (license license:expat)))

A few more inputs than Calcurse, but notice how the package definition is structured the same. The source section uses a (method git-fetch), so the --with-git-url is the right package transformation to use. Lets try it:

$ guix shell --container --nesting --development fzf --network --no-grafts nss-certs

[env]$ guix build fzf --with-git-url=fzf= --no-substitutes --no-grafts
updating checkout of ''...
retrieved commit 20240101197348a78db5a1dac945a2dff6a4e654
[... lots of output as it builds ...]
successfully built /gnu/store/wha5jskx2ygv1lbm4yll11r8zl0829ya-fzf-0.41.0.drv

As we can see it starts by updating the commit that it's using, checking this against the one in Github confirms that it's the latest commit. It then builds as usual - while the package is labelled as version 0.41.0 in fact it's using the very latest source code that's available from the master branch. This is pretty cool!

I personally, don't like it being labelled as 0.41.0 - there doesn't appear to be a way with --with-git-url to set the version, so you can't do the equivalent of with-source's <package>@<version> to explicitly set it to what you want.

However, we can also use this transformation in conjunction with --with-branch=<branch> and --with-commit=<commit>. Here's an example building a specific commit of fzf:

$ guix shell --container --nesting --development fzf --network --no-grafts nss-certs

# build a specific label
[env]$ guix build fzf --with-git-url=fzf= --with-commit=fzf=0.44.1 \
    --no-substitutes --no-grafts

# build a particular commit - in this case the tip of the master branch
guix build fzf --with-git-url=fzf= --with-commit=fzf=2024010 \
    --no-substitutes --no-grafts

We can use any commit or tag to build a version of the package and it will give it the correct version. In the second example we use the HEAD of the master branch, this gives it a package version of git and the commit hash.

Just like --with-source we can also use the git transformations to replace an input that's used in the dependency tree. In this example we're going to build ytfzf which is a script for searching Youtube, it uses fzf as one of the inputs:

$ guix shell --container --nesting --development ytfzf --network --no-grafts nss-certs

[env]$ guix build ytfzf --with-git-url=fzf= --with-commit=fzf=2024010 \
    --no-substitutes --no-grafts
updating checkout of ''...
retrieved commit 20240101197348a78db5a1dac945a2dff6a4e654
The following derivations will be built:
[... lots of output as it builds ... ]
successfully built /gnu/store/9k3k1b2g7jd5dzyw2liyn81wbmzmzalk-ytfzf-2.6.0.drv

To check that it's referencing the correct version of fzf we can check the derivation that was built as this shows all the inputs:

$ cat /gnu/store/9k3k1b2g7jd5dzyw2liyn81wbmzmzalk-ytfzf-2.6.0.drv

Amongst the inputs we can see /gnu/store/51csdk93y360g4pag784pzb3zrjay36b-fzf-git.2024010.drv which is the right one for it to use.

If a package recipe has a source method of url-fetch it's most likely that the --with-source package transformation is needed, and that trying to build it from it's git repository will fail. That's because the package recipe will be expecting a package archive (e.g. tar.gz) , and the source from git may not be the same. It's down to whether the package recipe is given source that it can build. However, it can work - here's an example (taken from the help-guix mailing list) of building emacs-eglot from the git archive, even though the package recipe uses url-fetch:

$ guix shell --container --nesting --development emacs-eglot --network --no-grafts nss-certs

[env]$ guix build emacs-eglot \
            --with-git-url=emacs-eglot= \
        [... lots of output ...]
successfully built /gnu/store/nz9r0jm4cjw4m78ssl97az6mpavw32vc-emacs-eglot-git.db91d58.drv

With latest transformation

If you don't want to figure out the version and just want the latest that's available then use the --with-latest package transformation. For this example we'll try and build the latest version of Weechat that's available.

One of the challenges when using --with-latest is that it can't always verify the source archive that it's finding. It will check the OpenPGP signature of the download if there is one, if there isn't one it gives you a warning. As part of the signature verification process it will search for the signatures public key. If it finds a key it will attempt to import it - note that this doesn't work in a guix shell --container unless the gnupg directory has been explicitly shared. A simpler way is to import the public key onto the gnupg public keyring and then start the container. For the Weechat key we do:

$ gpg --keyserver hkps:// --recv-keys A9AB5AB778FA5C3522FD0378F82F4B16DEC408F8

Now we build Weechat:

$ guix shell --container --nesting --development weechat --network --no-grafts nss-certs gnupg

[env]$ guix build weechat --with-latest=weechat --no-substitutes --no-grafts

Starting download of /tmp/guix-file.wx8xwZ

gpgv: Signature made Sun 03 Dec 2023 18:20:02 GMT
gpgv:                using RSA key A9AB5AB778FA5C3522FD0378F82F4B16DEC408F8
gpgv: Good signature from "WeeChat (signing key) <>"

[... lots of build output ...]
successfully built /gnu/store/6jhj8xsv08pg3hfj8cfx5ljsmckmhva4-weechat-4.1.2.drv

The guix build command finds the latest version of Weechat that has been released: the Weechat package definition in the archive uses url-fetch so it examines this URL and looks for later releases. It's found version 4.1.2 and having checked that the signature is good it's able to build the package. The final package that's created is named and versioned correctly so we can install and test it:

[env]$ guix package --install /gnu/store/7yxrqn3wa6qfqvha24b6k9dnfnqchhb2-weechat-4.1.2
[env]$ /home/steve/.guix-profile/bin/weechat --version

Rather than just getting the latest, we can also specify a particular version using the --with-version package transformation. For this example we'll build a specific version of pwclient which is client for Patchwork written in Python: the version in the Guix archive is currently 1.3.0 and we'll install 2.6.2:

$ guix shell --container --nesting --development pwclient --network --no-grafts nss-certs gnupg

[env]$ guix build pwclient --with-version=pwclient=2.6.2 --no-substitutes --no-grafts

guix build: warning: cannot authenticate source of 'pwclient', version 2.6.2
updating checkout of ''...
retrieved commit 1ea2e8dcbfe41c9d6a5c42596e028a33c6b21576
The following derivations will be built:
[... lots of output as it builds ...]

successfully built /gnu/store/w0vhanp05vlakdq8d6b8a776cl5kh724-pwclient-2.6.2.drv

With patch transformation

There are a few package transformations that change how a package is built, giving us the ability to tune the build to the hardware, alter the options provided to the build or add additional features to the source code. The two we'll look at are --with-configure-flag and --with-patch.

Software that is built using configure (Autoconf) and Make is packaged in Guix using the gnu-build-system: this is the most important build system and you'll see it's used by many packages in the archive. For our example, we're going to alter Mutt - a small but very powerful text-based email client.

We can see the package definition in the Guix archive by doing guix edit mutt:

(define-public mutt
    (name "mutt")
    (version "2.2.12")
    (source (origin
             (method url-fetch)
             (uri (list
                    (string-append ""
                                   "mutt-" version ".tar.gz")
                    (string-append ""
                                   version ".tar.gz")))
             (patches (search-patches "mutt-store-references.patch"))))
    (build-system gnu-build-system)
     (list cyrus-sasl
     `(#:configure-flags '("--enable-smtp"
                           "--enable-hcache" ; for header caching
                           "--with-sqlite3" ; required for Autocrypt
                           "--with-idn2" ; recommended for Autocrypt
                           ;; So that mutt does not check whether the path
                           ;; exists, which it does not in the chroot.
    (home-page "")
    (synopsis "Mail client")
     "Mutt is a small but very powerful text-based mail client for Unix
operating systems.")
    (license license:gpl2+)))

As we can see there's a reference to the GNU build system - (build-system gnu-build-system) - which tells us that during the build it will use configure and make. In the arguments section there are #:configure-flags that are passed into the build (e.g. --enable-smtp).

We can use the package transformation --with-configure-flag=package=flag to add additional configure flags to the build:

$ guix shell --container --nesting --development mutt --network --no-grafts nss-certs

[env]$ guix build mutt --with-configure-flag=mutt=--enable-compressed --no-substitutes --no-grafts
[... lots of build output ...]
successfully built /gnu/store/x6yd45pb1zzmwy8ggy9jrbzf584i8rf1-mutt-2.2.12.drv

Here, we've added the flag --enable-compressed and Guix builds the existing version from the archive, with the new flag added. We can check which configure flags were provided to the builder by looking at the derivation, and within that finding the mutt-builder derivation. In a standard shell do:

$ cat /gnu/store/x6yd45pb1zzmwy8ggy9jrbzf584i8rf1-mutt-2.2.12.drv

This shows the various inputs that were passed to the build process, and one of these is the reference to the mutt-builder derivation:

$ cat /gnu/store/ssni08hx6vq1dwrldhifyrxs9dl88psi-mutt-2.2.12-builder

This shows the arguments that went to the mutt builder, the part to look for is the configure-flags:

#:configure-flags (append (quote ("--enable-smtp" "--enable-imap" "--enable-pop" "--enable-gpgme" \
   "--enable-hcache" "--enable-sidebar" "--enable-autocrypt" "--with-ssl" "--with-sasl" "--with-sqlite3" \
   "--with-idn2" "--with-mailpath=/var/mail")) (quote ("--enable-compressed"))

As we can see the --enable-compressed option was provided as an additional flag (appended) to the ones that the standard package build uses.

We can also change the source of the package itself by patching it with the --with-patch package transformation. It's definitely a bit more complicated to organise this sort of transformation, I always find dealing with patches a bit complicated!

We're going to check that the patch we've selected applies cleanly to the pristine source (at the right version), and then we'll use the package transformation to build a package variant that includes the patch.

To find an appropriate patch I looked at The Debian Mutt package's patches, as Debian uses patches to the pristine source they often have good patches. We're going to check that we can apply the use_fqdn_from_etc_mailname to Mutt's 2.2.12 source code:

# clone mutt to an appropriate location
$ git clone
$ cd mutt

# checkout the 2.2.12 version of Mutt
$ git checkout -b mutt-with-patches mutt-2-2-12-rel

# download the patch from Debian
$ curl --remote-name

# apply the patch to the source
$ patch --input=882690-use_fqdn_from_etc_mailname.patch --verbose --backup
[... details of the patch ...]
patching file init.c
Using Plan A...
Hunk #1 succeeded at 170.
Hunk #2 succeeded at 4029.
Hunk #3 succeeded at 4038.

This proves that the patch can be applied cleanly to the Mutt source that's used in the Guix package. We're now ready to build the Guix package using the --with-patch package transformation:

$ guix shell --container --nesting --development mutt --network --no-grafts nss-certs wget

[env]$ curl --remote-name \

[env]$ guix build mutt --with-patch=mutt=./882690-use_fqdn_from_etc_mailname.patch --no-substitutes --no-grafts

And, that's it - we can now make changes to packages by adding our own custom patches.

Transformation limits

As we've seen package transformations are a great way of making a variety of changes that create a locally built package variant. It's really cool being able to update a package to use the latest source without having to edit anything.

There are some interesting implications to package transformations:

  • Package transformations use the package recipe that's in the archive
    A package transformation is a way to use the existing package recipe with some slightly different configuration (source code, inputs or some build settings). We can see those recipe's with guix edit <package>.
  • Package transformations can only alter existing Guix packages - not create new ones
    As the package transformation is using an existing recipe it can only deal with changes to how we want the recipe applied. If we have some source code without a recipe we can't use a package transformation. We can't use this to create new packages.
  • The <source location> needs to be processable by the package definition
    Each package definition specifies how the source is retrieved. Generally, it's best to use a package transformation that retrieves the source in a format that the package recipe is expecting. If the package recipe is using url-fetch then use --with-source or --with-latest. If the package definition uses git-fetch then try --with-git-url or -with-latest. It's not 100% certain, and we showed an example of an emacs packages that uses url-fetch but can be built from the git upstream.
  • The source can't have changed in some way that the package recipe isn't expecting
    If the updated package source needs an additional library, or requires some switch during the build then it will fail as the recipe in the archive doesn't know about this.

The limitation I don't like is that there's no way to rename a package or query which packages have been altered. I can't do a search for all packages that I've done a transformation on. In other packaging systems that I've used the way to track locally altered packages, and to prevent the packaging system from wiping out changes through upgrading was to rename them (or use a version suffix). There doesn't seem to be a way to do this with Guix's package transformations. To rename packages we have to use a local package recipe or a manifest, which we'll cover in a future post.

There are also other package transformations to explore (covered in the Guix manual). I'd love to hear if you're using Guix's package transformations, what you like about them and how this unique feature has been useful - send me an email through the contact form!

Posted in Tech Tuesday 05 December 2023
Tagged with tech ubuntu guix