We can speed up the downloading, building and updating of Guix packages by running a caching substitution server on the local network. In my set-up I have a fast workstation and then some slower laptops and VM's: all the clients pull their packages from the faster server.
Most users of Guix install binary packages (called 'substitutions') from an authorised Guix substitution server, rather than compiling the packages locally. Each system downloads their packages from the Internet - if we have a few Guix system this can really reduce the bandwidth available for streaming Netflix! To preserve bandwidth (and reduce the load on Guix's servers), we can download Guix's substitution binaries to one 'server' on the network, and then clients can get them from there. First, we'll cover how to run a set-up that downloads and distributes the binaries that the Guix servers have built. Then we'll go a step further, performing our own builds on the local server and distributing them to clients.
The guix publish command manages publishing a machines guix store (/gnu/store). There are two forms of doing this:
In each case the server only publishes substitutions that it has locally: we need to either download or build a package onto the server in order to make it available. We're not automatically making the whole of the Guix archive available.
On the server we create an archive key that's used to authenticate the substitution server to the client. We have to create the archive key logged in as the root user.
# login to the server as root user@server:$ sudo -i bash root@server:# guix archive --generate-key
A signing-key.pub and signing-key.sec are created and placed in the /etc/guix directory.
Now we're going to start the guix publish service on the server and leave it running in a window so we can see the output. For testing purposes we'll do this directly from the root user account.
We provide the --advertise switch which advertises the availability of the new substitution server on the LAN. With a small piece of configuration on each client they will see the local substitution server and start using it.
root@server:# guix publish --advertise guix publish: warning: server running as root; consider using the '--user' option! guix publish: publishing /gnu/store on 0.0.0.0, port 8080 guix publish: using 'gzip' compression method, level 3 guix publish: Advertising guix-publish-dragon2
As we can see my server is called dragon2 and we're now publishing the /gnu/store on all interfaces. On a client (my laptop) run avahi-browse -alr to see the guix publish service being advertised:
= wlan0 IPv4 guix-publish-dragon2 _guix_publish._tcp local hostname = [dragon2.local] address = [192.168.1.20] port = [8080] txt = []
This confirms that the service is available from my server (dragon2.local). Now we prepare each client machine.
On the clients we need to add the --discover option to the guix-daemon so that it will look for substitution servers on the LAN. As I'm on a foreign distribution (Ubuntu) to change the way the guix daemon starts I edit the systemd guix-service definition. This is stored in /etc/systemd/system/guix-daemon.service
user@client:$ sudo systemctl stop guix-daemon.service # edit the /etc/system/system/guix-daemon.service user@client:$ sudo vim /etc/systemd/system/guix-daemon.service # add the --discover option to the line ExecStart=/var/guix/profiles/per-user/root/current-guix/bin/guix-daemon --build-users-group=guixbuild --discover --substitute-urls='https://ci.guix.gnu.org https://bordeaux.guix.gnu.org https://substitutes.nonguix.org'
We reload systemd and restart the guix-daemon service:
# Reload and restart the service user@client:$ sudo systemctl daemon-reload user@client:$ sudo systemctl start guix-daemon.service user@client:$ sudo systemctl status guix-daemon.service
The status shown is something like this, and we can see the --discover switch is there:
guix-daemon.service - Build daemon for GNU Guix
Loaded: loaded (/etc/systemd/system/guix-daemon.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/guix-daemon.service.d
└─override.conf
Active: active (running) since Fri 2023-04-28 14:52:54 BST; 20h ago
Main PID: 25043 (guix-daemon)
Tasks: 7 (limit: 8192)
CGroup: /system.slice/guix-daemon.service
├─25043 guix-daemon --build-users-group=guixbuild --discover --substitute-urls=https://ci.guix.gnu.org https://bordeaux.guix.gnu.org https://substitutes.nonguix.org
└─25056 /gnu/store/61mbwhd3bcy3hdscnzwyavglql4vz94r-guile-wrapper/bin/guile --no-auto-compile /gnu/store/lw7ggbgs56c34bhppwv60paqhpm4qjm4-guix-command discover
Then we can do a guix pull to check if it knows about the new substitution server:
user@client:$ guix pull substitute: updating substitutes from 'http://192.168.1.20:8080'... 100.0% substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0% [ ... lots more output ... ]
This shows that the local substitution server (http://192.168.1.20:8080) has been found and is being used. Meanwhile, on the server we should see something like this:
GET /9ywk70631qsn5msxgizcxw40nkhaw0mn.narinfo -> GET /9ywk70631qsn5msxgizcxw40nkhaw0mn.narinfo: 404 GET /ywmjrbib2g7g3jbkalr1al82s7gpvf6p.narinfo GET /zhr8y15pjr597ibzmf0vmffynplf5i0h.narinfo GET /550pjvvwji7z7rfb516399zqdjdm1j2w.narinfo GET /f490h8y02krwf3yzf2c2zk3yd08gpnvn.narinfo GET /nar/gzip/f490h8y02krwf3yzf2c2zk3yd08gpnvn-module-import GET /nar/gzip/ywmjrbib2g7g3jbkalr1al82s7gpvf6p-module-import-compiled GET /nar/gzip/zhr8y15pjr597ibzmf0vmffynplf5i0h-module-import-compiled
This is all the various substitution binaries being indexed by the client during the guix pull.
To complete the test install something on the server, and then install the same package on a client machine. When we do the install on the client we will see the binaries being provided from the local substitution server. On the server do:
user@server:$ guix build vim
Note that we're doing guix build to download the official substitution binary into our servers /gnu/store: this is how we can get binaries into the store without having to install the package into a profile on the server.
On a client machine do:
user@client:$ guix package --substitute-urls='http://192.168.1.20:8080' --install vim The following package will be installed: vim 9.0.1303 substitute: updating substitutes from 'http://192.168.1.20:8080'... 100.0%
We're expressly setting the specific substitution server ('http://192.168.1.20:8080') so that it only tries that location. Doing the same thing without setting the --substitute-urls will also prefer the local server.
At this point anything that is an official binary can be downloaded (guix build X) into the server's Store and then client machines can install them from there.
The next step is to authorise our local substitution server so that it can build packages itself and provide them to client machines. To ensure authenticity each client needs to be told that it can get substitutes which have been built on the substitution server. This is done by providing the public key of the archive to the client.
The substitution server automatically publishes the archive key as signing-key.pub.
user@client:$ wget http://dragon2.local:8080/signing-key.pub -O dragon2.local.pub user@client:$ sudo guix archive --authorize < dragon2.local.pub
Download the key from the local substitution server and then run the guix archive --authorize command providing the public key. The file looks something like this:
at dragon2.local.pub (public-key (ecc (curve Ed25519) (q #6B4106BE602E6C44F7E929F78FCEF608B59D2FDCB4EB8ED3184A790F4A210ACF#) ) )
The second step is to tell the guix-daemon on the client about the substitution server. Again we're going to change /etc/systemd/system/guix-daemon.service by putting the local substitution server at the start - this will ensure it's the preferred substitution server to use:
user@client:$ sudo systemctl stop guix-daemon.service # edit the /etc/system/system/guix-daemon.service user@client:$ $ sudo vim /etc/systemd/system/guix-daemon.service # add the --substitute-urls option on the guix-daemon command line and list the URLs of interest # add the local substitution server at the front ExecStart=/var/guix/profiles/per-user/root/current-guix/bin/guix-daemon --build-users-group=guixbuild --discover --substitute-urls='http://dragon2.local https://ci.guix.gnu.org https://bordeaux.guix.gnu.org https://substitutes.nonguix.org' # Reload and restart the service user@client:$ sudo systemctl daemon-reload user@client:$ sudo systemctl start guix-daemon.service user@client:$ sudo systemctl status guix-daemon.service
This time we should see something like this:
guix-daemon.service - Build daemon for GNU Guix
Loaded: loaded (/etc/systemd/system/guix-daemon.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/guix-daemon.service.d
└─override.conf
Active: active (running) since Fri 2023-04-28 14:52:54 BST; 20h ago
Main PID: 25043 (guix-daemon)
Tasks: 7 (limit: 8192)
CGroup: /system.slice/guix-daemon.service
├─25043 guix-daemon --build-users-group=guixbuild --discover --substitute-urls=http://dragon2.local:8080 https://ci.guix.gnu.org https://bordeaux.guix.gnu.org https://substitutes.nonguix.org
└─25056 /gnu/store/61mbwhd3bcy3hdscnzwyavglql4vz94r-guile-wrapper/bin/guile --no-auto-compile /gnu/store/lw7ggbgs56c34bhppwv60paqhpm4qjm4-guix-command discover
As we can see we're listing the LAN substitution server ('dragon2.local:8080') first.
Lets build a package on the server, after it's been compiled it will be available in the Guix Store for the clients to use. To do this we use the guix build command and the name of the package. To ensure that the package is built on our local server we prevent it from downloading any binary substitution from Guix's official servers with the --no-substitutes switch.
Lets build something really simple, a Tetris clone called Nudoku:
user@server:$ guix build nudoku --no-substitutes --dry-run The following derivations would be built: /gnu/store/k6a2dw2c12cnq3zjn7cchbi6r2ynacd5-nudoku-2.1.0.drv /gnu/store/4756irx013j06f1rpgmqs26v7rs408zn-gettext-minimal-0.21.drv /gnu/store/h2v8y37d500lxwa2dnnnndhihdwi0hqa-gettext-0.21.tar.gz.drv /gnu/store/n2nnsi19n67aihsp0acd8yxfr39big9i-nudoku-2.1.0-checkout.drv /gnu/store/615dj5fk4bys58cij9w93drsy9i8fpjd-tar-1.34.drv /gnu/store/9br2269lxl105x72yaqm96cd5v4w5bzk-tar-1.34.tar.xz.drv /gnu/store/47lx5q6dhyy0hmx71iyjipp8g5ka8k7k-tar-1.34.tar.xz.drv /gnu/store/rp0qp81s6dvacvb7ls858a7zjdlbaj24-autoconf-2.69.drv /gnu/store/2x3pab3l081nbm8f6nafzva3d1dp3wg5-autoconf-2.69.tar.xz.drv /gnu/store/gmar7sa1m0brpf1xlj89z3j5dbj46s02-m4-1.4.18.drv /gnu/store/d9qhnl0vyvxmjnm9frva5jh45vimj91n-m4-1.4.18.tar.xz.drv /gnu/store/klnp2p5n7xmim3pz91r2wqgqykbnaaww-m4-1.4.18.tar.xz.drv /gnu/store/zjwikg8mw4g56x19lkdw7vbdqy7f2hlx-automake-1.16.3.drv /gnu/store/khh2qq74kvx923bjddp9zkaak89wg8a2-automake-1.16.3.tar.xz.drv /gnu/store/y0s6n40hkmm6w0zkclq8zgh7pjqq4rf7-automake-1.16.3.tar.xz.drv /gnu/store/q6cc2l6g2dlxccalxf4wwi2rb27vf39c-autoconf-wrapper-2.69.drv
This shows us what will be built: in this case we are going from a clean install so it's going to build a set of required inputs like tar and then the Nudoku package itself. Remove the --dry-run option and run again to start the build.
Guix downloads the various packages and starts compiling them. When it's completed the whole thing the last line of output will be something like this:
phase `compress-documentation' succeeded after 0.0 seconds successfully built /gnu/store/k6a2dw2c12cnq3zjn7cchbi6r2ynacd5-nudoku-2.1.0.drv /gnu/store/1zscjb8lh3z46fmkvmvss43qh5mjd2p5-nudoku-2.1.0
In our client's terminal we can check the substitution is available on the server with:
client:$ guix weather --substitute-urls=http://dragon2.local:8080 nudoku computing 1 package derivations for x86_64-linux... looking for 1 store items on http://dragon2.local:8080... http://dragon2.local:8080 ☀ 100.0% substitutes available (1 out of 1) unknown substitute sizes 0.1 MiB on disk (uncompressed) 0.137 seconds per request (0.1 seconds in total) 7.3 requests per second (continuous integration information unavailable)
This tells us that Nudoku is available from the local substitution server. Notice we're now referring to the local substitution server by the name that was in the authorization file ('dragon2.local') rather than the IP address that we used earlier.
On the server we should see something like this:
GET /1zscjb8lh3z46fmkvmvss43qh5mjd2p5.narinfo GET /api/queue -> GET /api/queue: 404
Then we can install it on the client:
client:$ guix install --substitute-urls=http://dragon2.local:8080 nudoku
On the server we see:
GET /nar/gzip/pd6wvf6yg8bcrv74sd6zrsz5712bs6r3-nudoku-2.1.0
This confirms that we're able to compile packages on the server and make them available as substitutes for the client systems. If you wan to experiment further packages like vitetris, angband and gnugo. Having built the package on the server, on the client do a plain guix install <package> to confirm that the local server is being preferred for substitutions.
If the local substitution server is not available then Guix will fall back to the official ones without a problem:
substitute: updating substitutes from 'http://dragon2.local:8080'... 0.0%guix substitute: warning: dragon2.local: connection failed: Connection refused
Rather than manually starting the substitution server we can set it to run all the time. If it's a guix system you can use system service. As I'm on a on a foreign distribution I need a systemd service. There's an example guix-publish systemd service in the Guix source. We can set this up as follows:
user@server:$ sudo wget https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-publish.service.in -O /etc/systemd/system/guix-publish.service # edit the file to change the port to 8080 user@server:$ sudo vim /etc/systemd/system/guix-publish.service #ExecStart=@localstatedir@/guix/profiles/per-user/root/current-guix/bin/guix publish --user=nobody --port=8181 ExecStart=/var/guix/profiles/per-user/root/current-guix/bin/guix publish --user=nobody --port=8080 #Environment='GUIX_LOCPATH=@localstatedir@/guix/profiles/per-user/root/guix-profile/lib/locale' LC_ALL=en_US.utf8 Environment='GUIX_LOCPATH=/var/guix/profiles/per-user/root/guix-profile/lib/locale' LC_ALL=en_US.utf8
Up to this point and in the manual we've been using port 8080, so to now have to change everything change this.
# Reload and restart the service user@server:$ sudo systemctl daemon-reload user@server:$ sudo systemctl start guix-publish.service user@server:$ sudo systemctl status guix-publish.service # check the service is serving something user@client:$ wget http://dragon2.local:8080 -O -
We can view the journal with journalctl:
user@server:$ journalctl -f -u guix-publish.service
For our final test lets go back to nature:
user@server:$ guix build cbonsai --no-substitutes [ ... lots of output ... ] successfully built /gnu/store/pypg88mgxl2hrz43l2flwnghh3x1q459-cbonsai-1.3.1.drv /gnu/store/mgc2i6yxm2zbqf8yx8x5f4ig4nbii2cv-cbonsai-1.3.1
On one of the clients we do:
user@client:$ guix weather --substitute-urls=http://dragon2.local:8080 cbonsai http://dragon2.local:8080 ☀ 100.0% substitutes available (1 out of 1) unknown substitute sizes 0.1 MiB on disk (uncompressed) (continuous integration information unavailable) user@client:$ guix package --install cbonsai --verbosity=3 The following package will be installed: cbonsai 1.3.1 substitute: updating substitutes from 'http://dragon2.local:8080'... 100.0% [... lots more output ...]
Using a single caching substitution server makes updates faster on my client systems and stops me hogging Internet bandwidth - it's always nice to use fewer resources if you can. The manual page is very complete, but the order of changes was quite confusing - so I hope this post is useful to others.