We can speed up the downloading, building and updating of Guix packages by running a caching substitution server on the local network. In my set-up I have a fast workstation and then some slower laptops and VM's: all the clients pull their packages from the faster server.
Most users of Guix install binary packages (called 'substitutions') from an authorised Guix substitution server, rather than compiling the packages locally. Each system downloads their packages from the Internet - if we have a few Guix system this can really reduce the bandwidth available for streaming Netflix! To preserve bandwidth (and reduce the load on Guix's servers), we can download Guix's substitution binaries to one 'server' on the network, and then clients can get them from there. First, we'll cover how to run a set-up that downloads and distributes the binaries that the Guix servers have built. Then we'll go a step further, performing our own builds on the local server and distributing them to clients.
The guix publish command manages publishing a machines guix store (/gnu/store). There are two forms of doing this:
In each case the server only publishes substitutions that it has locally: we need to either download or build a package onto the server in order to make it available. We're not automatically making the whole of the Guix archive available.
On the server we create an archive key that's used to authenticate the substitution server to the client. We have to create the archive key logged in as the root user.
# login to the server as root user@server:$ sudo -i bash root@server:# guix archive --generate-key
A signing-key.pub and signing-key.sec are created and placed in the /etc/guix directory.
On the clients we need to add the --discover option to the guix-daemon so that it will look for substitution servers on the LAN. As I'm on a foreign distribution (Ubuntu) to change the way the guix daemon starts I edit the systemd guix-service definition. This is stored in /etc/systemd/system/guix-daemon.service
user@client:$ sudo systemctl stop guix-daemon.service # edit the /etc/system/system/guix-daemon.service user@client:$ sudo vim /etc/systemd/system/guix-daemon.service # add the --discover option to the line ExecStart=/var/guix/profiles/per-user/root/current-guix/bin/guix-daemon --build-users-group=guixbuild --discover --substitute-urls='https://ci.guix.gnu.org https://bordeaux.guix.gnu.org https://substitutes.nonguix.org'
We reload systemd and restart the guix-daemon service:
# Reload and restart the service user@client:$ sudo systemctl daemon-reload user@client:$ sudo systemctl start guix-daemon.service user@client:$ sudo systemctl status guix-daemon.service
The status shown is something like this, and we can see the --discover switch is there:
guix-daemon.service - Build daemon for GNU Guix Loaded: loaded (/etc/systemd/system/guix-daemon.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/guix-daemon.service.d └─override.conf Active: active (running) since Fri 2023-04-28 14:52:54 BST; 20h ago Main PID: 25043 (guix-daemon) Tasks: 7 (limit: 8192) CGroup: /system.slice/guix-daemon.service ├─25043 guix-daemon --build-users-group=guixbuild --discover --substitute-urls=https://ci.guix.gnu.org https://bordeaux.guix.gnu.org https://substitutes.nonguix.org └─25056 /gnu/store/61mbwhd3bcy3hdscnzwyavglql4vz94r-guile-wrapper/bin/guile --no-auto-compile /gnu/store/lw7ggbgs56c34bhppwv60paqhpm4qjm4-guix-command discover
Then we can do a guix pull to check if it knows about the new substitution server:
user@client:$ guix pull substitute: updating substitutes from 'http://192.168.1.20:8080'... 100.0% substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0% [ ... lots more output ... ]
This shows that the local substitution server (http://192.168.1.20:8080) has been found and is being used. Meanwhile, on the server we should see something like this:
GET /9ywk70631qsn5msxgizcxw40nkhaw0mn.narinfo -> GET /9ywk70631qsn5msxgizcxw40nkhaw0mn.narinfo: 404 GET /ywmjrbib2g7g3jbkalr1al82s7gpvf6p.narinfo GET /zhr8y15pjr597ibzmf0vmffynplf5i0h.narinfo GET /550pjvvwji7z7rfb516399zqdjdm1j2w.narinfo GET /f490h8y02krwf3yzf2c2zk3yd08gpnvn.narinfo GET /nar/gzip/f490h8y02krwf3yzf2c2zk3yd08gpnvn-module-import GET /nar/gzip/ywmjrbib2g7g3jbkalr1al82s7gpvf6p-module-import-compiled GET /nar/gzip/zhr8y15pjr597ibzmf0vmffynplf5i0h-module-import-compiled
This is all the various substitution binaries being indexed by the client during the guix pull.
To complete the test install something on the server, and then install the same package on a client machine. When we do the install on the client we will see the binaries being provided from the local substitution server. On the server do:
user@server:$ guix build vim
Note that we're doing guix build to download the official substitution binary into our servers /gnu/store: this is how we can get binaries into the store without having to install the package into a profile on the server.
On a client machine do:
user@client:$ guix package --substitute-urls='http://192.168.1.20:8080' --install vim The following package will be installed: vim 9.0.1303 substitute: updating substitutes from 'http://192.168.1.20:8080'... 100.0%
We're expressly setting the specific substitution server ('http://192.168.1.20:8080') so that it only tries that location. Doing the same thing without setting the --substitute-urls will also prefer the local server.
At this point anything that is an official binary can be downloaded (guix build X) into the server's Store and then client machines can install them from there.
Lets build a package on the server, after it's been compiled it will be available in the Guix Store for the clients to use. To do this we use the guix build command and the name of the package. To ensure that the package is built on our local server we prevent it from downloading any binary substitution from Guix's official servers with the --no-substitutes switch.
Lets build something really simple, a Tetris clone called Nudoku:
user@server:$ guix build nudoku --no-substitutes --dry-run The following derivations would be built: /gnu/store/k6a2dw2c12cnq3zjn7cchbi6r2ynacd5-nudoku-2.1.0.drv /gnu/store/4756irx013j06f1rpgmqs26v7rs408zn-gettext-minimal-0.21.drv /gnu/store/h2v8y37d500lxwa2dnnnndhihdwi0hqa-gettext-0.21.tar.gz.drv /gnu/store/n2nnsi19n67aihsp0acd8yxfr39big9i-nudoku-2.1.0-checkout.drv /gnu/store/615dj5fk4bys58cij9w93drsy9i8fpjd-tar-1.34.drv /gnu/store/9br2269lxl105x72yaqm96cd5v4w5bzk-tar-1.34.tar.xz.drv /gnu/store/47lx5q6dhyy0hmx71iyjipp8g5ka8k7k-tar-1.34.tar.xz.drv /gnu/store/rp0qp81s6dvacvb7ls858a7zjdlbaj24-autoconf-2.69.drv /gnu/store/2x3pab3l081nbm8f6nafzva3d1dp3wg5-autoconf-2.69.tar.xz.drv /gnu/store/gmar7sa1m0brpf1xlj89z3j5dbj46s02-m4-1.4.18.drv /gnu/store/d9qhnl0vyvxmjnm9frva5jh45vimj91n-m4-1.4.18.tar.xz.drv /gnu/store/klnp2p5n7xmim3pz91r2wqgqykbnaaww-m4-1.4.18.tar.xz.drv /gnu/store/zjwikg8mw4g56x19lkdw7vbdqy7f2hlx-automake-1.16.3.drv /gnu/store/khh2qq74kvx923bjddp9zkaak89wg8a2-automake-1.16.3.tar.xz.drv /gnu/store/y0s6n40hkmm6w0zkclq8zgh7pjqq4rf7-automake-1.16.3.tar.xz.drv /gnu/store/q6cc2l6g2dlxccalxf4wwi2rb27vf39c-autoconf-wrapper-2.69.drv
This shows us what will be built: in this case we are going from a clean install so it's going to build a set of required inputs like tar and then the Nudoku package itself. Remove the --dry-run option and run again to start the build.
Guix downloads the various packages and starts compiling them. When it's completed the whole thing the last line of output will be something like this:
phase `compress-documentation' succeeded after 0.0 seconds successfully built /gnu/store/k6a2dw2c12cnq3zjn7cchbi6r2ynacd5-nudoku-2.1.0.drv /gnu/store/1zscjb8lh3z46fmkvmvss43qh5mjd2p5-nudoku-2.1.0
In our client's terminal we can check the substitution is available on the server with:
client:$ guix weather --substitute-urls=http://dragon2.local:8080 nudoku computing 1 package derivations for x86_64-linux... looking for 1 store items on http://dragon2.local:8080... http://dragon2.local:8080 ☀ 100.0% substitutes available (1 out of 1) unknown substitute sizes 0.1 MiB on disk (uncompressed) 0.137 seconds per request (0.1 seconds in total) 7.3 requests per second (continuous integration information unavailable)
This tells us that Nudoku is available from the local substitution server. Notice we're now referring to the local substitution server by the name that was in the authorization file ('dragon2.local') rather than the IP address that we used earlier.
On the server we should see something like this:
GET /1zscjb8lh3z46fmkvmvss43qh5mjd2p5.narinfo GET /api/queue -> GET /api/queue: 404
Then we can install it on the client:
client:$ guix install --substitute-urls=http://dragon2.local:8080 nudoku
On the server we see:
This confirms that we're able to compile packages on the server and make them available as substitutes for the client systems. If you wan to experiment further packages like vitetris, angband and gnugo. Having built the package on the server, on the client do a plain guix install <package> to confirm that the local server is being preferred for substitutions.
If the local substitution server is not available then Guix will fall back to the official ones without a problem:
substitute: updating substitutes from 'http://dragon2.local:8080'... 0.0%guix substitute: warning: dragon2.local: connection failed: Connection refused
Rather than manually starting the substitution server we can set it to run all the time. If it's a guix system you can use system service. As I'm on a on a foreign distribution I need a systemd service. There's an example guix-publish systemd service in the Guix source. We can set this up as follows:
user@server:$ sudo wget https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-publish.service.in -O /etc/systemd/system/guix-publish.service # edit the file to change the port to 8080 user@server:$ sudo vim /etc/systemd/system/guix-publish.service #ExecStart=@localstatedir@/guix/profiles/per-user/root/current-guix/bin/guix publish --user=nobody --port=8181 ExecStart=/var/guix/profiles/per-user/root/current-guix/bin/guix publish --user=nobody --port=8080 #Environment='GUIX_LOCPATH=@localstatedir@/guix/profiles/per-user/root/guix-profile/lib/locale' LC_ALL=en_US.utf8 Environment='GUIX_LOCPATH=/var/guix/profiles/per-user/root/guix-profile/lib/locale' LC_ALL=en_US.utf8
Up to this point and in the manual we've been using port 8080, so to now have to change everything change this.
# Reload and restart the service user@server:$ sudo systemctl daemon-reload user@server:$ sudo systemctl start guix-publish.service user@server:$ sudo systemctl status guix-publish.service # check the service is serving something user@client:$ wget http://dragon2.local:8080 -O -
We can view the journal with journalctl:
user@server:$ journalctl -f -u guix-publish.service
For our final test lets go back to nature:
user@server:$ guix build cbonsai --no-substitutes [ ... lots of output ... ] successfully built /gnu/store/pypg88mgxl2hrz43l2flwnghh3x1q459-cbonsai-1.3.1.drv /gnu/store/mgc2i6yxm2zbqf8yx8x5f4ig4nbii2cv-cbonsai-1.3.1
On one of the clients we do:
user@client:$ guix weather --substitute-urls=http://dragon2.local:8080 cbonsai http://dragon2.local:8080 ☀ 100.0% substitutes available (1 out of 1) unknown substitute sizes 0.1 MiB on disk (uncompressed) (continuous integration information unavailable) user@client:$ guix package --install cbonsai --verbosity=3 The following package will be installed: cbonsai 1.3.1 substitute: updating substitutes from 'http://dragon2.local:8080'... 100.0% [... lots more output ...]
Using a single caching substitution server makes updates faster on my client systems and stops me hogging Internet bandwidth - it's always nice to use fewer resources if you can. The manual page is very complete, but the order of changes was quite confusing - so I hope this post is useful to others.