Ubuntu upgrades: An approach to upgrading between Ubuntu releases

Introduction

An Update is Available For Your Computer cartoon

An Update is Available For Your Computer by Christiann Macauley

If you're running Ubuntu (Linux) as a desktop environment the general advice on the Internet to to reinstall every time there's a new release. It's certainly simpler, but it misses the opportunity to keep system configuration and to enjoy a piece of magic - the entire operating system and all applications upgrading from one version to another!

When I started using Linux, being able to upgrade was one of the main differences over Microsoft Windows. Perhaps due to the server heritage upgrades were the most popular methodology - plus getting hold of floppy disks and then CD's for a full fresh install wasn't always easy!

For many years I had very slow connection to the Net - it could take 24 hours to download the 7 floppy disks, nursing a 27.7k modem (relive the sound). Consequently, I've sat through many sweaty-palmed weekends hoping that the connection stayed up while the system tried to download a new glibc. It's fair to say then that my approach is battle tested!

It's not complex, but it is long-winded and conservative. Probably, due to that history!

Use-cases

There are two options for upgrading between one version of a distribution's release and another: fresh install and upgrade. If you do a fresh install then the installer will write over the system partition and install completely new versions of the packages. With an upgrade the installer leaves the partitions alone, installs new system packages and leaves the configuration files in place.

Upgrade from Trusty Tahr (14.04 LTS) to Xenial Xerus (16.04 LTS)

Trusty Tahr to Xenial Xerus

These instructions work for Ubuntu, and should work for Debian. I'm moving from one LTS release to another. With a little alteration the metholodology will work on other systems e.g. CentOS. In this case we're upgrade from Trusty Tahr (Ubuntu 14.04 LTS) to Xenial Xerus (16.04 LTS), but the instructions are applicable to any upgrade.

Strengths

  • Keep system configuration

If you've changed your system configuration then an upgrade will for the most part keep those configuration settings for you. Where I alter server daemons like SSH or hardware like power settings this is very useful.

  • Maintains installed applications

Upgrades keep all the applications that you've installed for you. If you do a plain install then you'll have to remember what you've installed. That impacts me a lot as I use a lot of non-standard applications.

  • Deal with major system changes

Between major releases there are often substantial system changes. The upgrade methodology keeps you aware of those alterations.

Weaknesses

  • Misses new defaults

Sometimes maintainers change to new defaults, particularly where there's been a major new version. Often we're over-riding those changes during a change so you miss them.

  • Complicated to undertake

Upgrades are more complicated to do than simply installing the system over the top.

  • Can fail if interrupted

Upgrades can fail badly if the Internet connection is interrupted, or the system gets stuck for some other reason. This is a general weakness with package management systems and filesystems where they can't revert the changes.

Keep Calm [And Switch to Linux]

Keep Calm, image by Miss Nixie, CC BY-NC-ND 2.0

a. Read the release notes

The first decision is when to upgrade. It's worth researching and understanding Ubuntu's upgrade paths. I use Ubuntu for work, so stay on the Long Term Support (LTS) releases. I wait for the .1 release which happens 3 months after the initial launch. Some bugs don't show up in the testing that the distributions software teams undertake, you only see them when mass usage takes place and users expand all the different types of use. This is particularly true for bugs that depend on interactions between different pieces of hardware. For the LTS releases I worked on we kept a lot of engineers working on it, resolving bugs for the first 3 months, so the .1 release stabilised the release a lot.

It's worth reading through the release notes which are part of the release announcement. It lists any major changes that the release team thinks users should know about, along with any common problems (and the workaround).

Bear in mind that LTS announcements only cover changes from the previous short release, not for the whole LTS to LTS cycle. To know everything that has changed we have to read the release notes for each interim release. For this upgrade that meant reading:

Computer Catalog : Consolidated/Convair Aircraft Factory San Diego Equipment

Read the Fscking Manual before upgrading! Image by SDASM Archives No known copyright restrictions

For Xenial's release:

Wily Werewolf:

Vivid Vervet:

Utopic Unicorn:

Note down any major system changes, significant bugs or new features. These are the ones that I noted down:

Major system affecting changes

  • Systemd (Vivid)

    Replaces Upstart with Systemd. This is a major change for how we manage system daemons (and a host of connected elements) on Linux systems. I did a bit of reading systemd on Ubuntu . As this so significant I also looked at the major systemd-boot bugs on Launchpad to see if there was anything that might impact me.

  • Python 3 default (Xenial)

    Xenial installs Python 3 and uses it as the default. This will impact my development environment.

  • Python 3 for vim (Xenial)

    Vim has been built with Python 3. Plugins that depend on Python 2 will struggle. There's a suggested workaround. I've taken the pain and gone to everything on Python 3.

  • Major SSH change (Xenial)

    The new version should be fine for my usage which is very basic.

Significant bugs

  • USB Creator

    There's been a major change of behaviour meaning that USB sticks created on 14.04 LTS won't work on 16.04 LTS. This is very painful for me as my upgrade process dictates I need to have a backup to boot the system. I did lots of research done to find out what was going on and what the recommended solution was: pretty inconclusive, confusing and annoying. Start with installing from USD, also the usb-creator bugs list.

  • Keyboard selection may be incorrect

    Possibility of a bad selection, where the workaround is to go to the keyboard selection control panel and change it back. This can impact me as I mess around with my Kinesis Advantage keyboard configuration.

New features

  • Snap packages (Xenial)

    A new packaging format for applications that can live in parallel to the traditional ones. Snap packages have been coming for a while. I'll probably ignore this for now.

  • Software Center replacement (Xenial)

    Sad to see this go as I worked on it. It's replaced by GNOME Software labelled Ubuntu Software. Doesn't impact me directly as I mostly use Synaptic.

  • GNOME Calendar (Xenial)

    Installed by default. Probably doesn't impact me at the moment.

  • Docker 1.10 (Xenial)

    This is the system installed version rather than using one provided directly by Docker.

  • Blueman 2.0 (Wily)

    Noted this down but didn't think much of it. Had some Bluetooth problems later where it was important to know more about the changes.

  • Ubuntu Make

    Upgraded Ubuntu Make which is interesting and noted so that I could take a look at it.

I has a new X.Org input driver

Be happy if you don't know why this is funny! And thankful for Keith Packard's contributions, image by Jacobo TarrĂ­o, CC BY-SA 2.0

b. Search for common problems

The benefit of waiting a couple of months before going to the new version is that the distribution developers and users should have solved many of the common problems. To make sure that we're not going to find anything terrible search the Web for the common tools and hardware that you use.

Commonly, I search using DuckDuckGo for the release names and version number a key area and terms like "problem" or "upgrade". For example, I'd search on "16.04 Ubuntu x200 Lenovo problem" to see if anything came up. To decide what key areas you should search for I'd choose any hardware, any commonly problematic area and anything unusual that you use.

My searches were:

  • "Upgrade 14.04 16.04 problem"

    Didn't turn up anything significant. A minor issue about DNS resolution.

  • "X200 Lenovo"

    This is my hardware platform so finding anything would be problematic. Nothing significant.

  • "LVM 16.04" and "LVM xenial"

    I use this to split my drive space. It is a common tool on server set-ups, but it's not used on desktop installs that much so worth checking. Nothing significant, we should continue to do the partition piece manually.

  • "Encfs xenial"

    Used to keep encrypted disk image archives. Not popular any more so worth checking. Showing as working.

  • "Xinit xenial"

    I often start X from the command line with xinit. This isn't a common use-case any more so worth checking as the code path may not be tested. Two problems showed in #1562219 and a thread on Ubuntu Forums #2328593

  • i3wm xenial

    i3 is a tiling window manager. It is used a lot but isn't tested by the distribution team. Nothing significant, one complaint in #595563

  • "Kernel 16.04 problem"

    It's very unlikely that Ubuntu will release anything with obvious problems, and the distribution carries patches that aren't in mainline. A quick search is warranted. Currently running linux-image-generic 4.4.0-53-generic SMP x86_64, nothing showed.

  • "Network manager 16.04" and "Network manager xenial"

    Network manager has had a lot of problems in the past. Losing network connectivity is a total pain, so a bit of searching for both network-manager and network-manager-applet is worth it. Nothing significant.

  • "Power management"

    Power management is a total pain and I use tlp. Nothing showed up this time.

c. Remove unused packages

In Debian we use apt to upgrade the system. This calculates the upgrades between Debian packages and then performs them in the most suitable way. In Ubuntu we use upgrade-manager which layers on an additional level of intelligence: the distribution engineers often add workarounds for known issues. We can help make the upgrade as simple as possible by removing any old or obsolete packages - it's a good opportunity to do some housekeeping.

Obsolete Packages

I use Synaptic as it's easy to find the Obsolete packages:

In the Sections area find:
Status
    'Installed (auto removable)'
    'Installed (local or obsolete)'
    'Not installed (residual config)'
Custom Filters
    Broken  # Check for broken applications
    Origin  # Check through for odd things

In each Status area and with Broken or an odd Origin Make for complete removal

  • Be careful with 'Local or obsolete'

    These may be applications that you use but have put in local installs. Note that Local installs are ones where Dpkg doesn't know the repository where this package came from.

  • Removing 'Not installed'

    To speed things up use <Shift>+<Arrow keys> to highlight a group and then right click to mark them for complete removal.

  • Remove old Kernel versions

    The /boot area can fill-up very easily. It's worth searching for linux-headers, linux-headers-generic and linux-image to find old version. Keep a couple around.

  • Remove applications that you don't use

    If we're not actively using an application it's worth removing them. The simpler the system the fewer libraries and the less complicated the upgrade calculation will be. I didn't do this as I was short of time.

Clean the system

There are a variety of tools that can examine the system directly, or through Apt's database and try to simplify or clean the set-up. These tools can be dangerous, so be careful.

  • Clean with Bleachbit

    Bleachbit tries to find files that are no longer required. I've used it in the past but it's pretty aggressive so skipped it this time. Basic instructions are:

    $ sudo apt-get install bleachbit
    $ sudo bleachbit
    
  • Use GTKorphan

    Gktorphan examines the Apt database and tries to find files that aren't required. It's worth running. I don't trust it enough to allow it to remove anything, I do that directly through Synaptic.

    $ sudo gtkorphan
    

It gives output a bit like this:

* compizconfig-backend-gconf
* compiz-plugins-main-default
* gwibber-service
* gwiber-service-facebook
* gwibber-service-identica
* gwibber-service-twiter
* libavahi-gobject0
* libavahi-ui-gtk3-o
[very long list redacted]

Note

It might be fine to do this directly from through gtkorphan but I'm just being cautious

  • Clean out the local caches

    The default is for Apt to maintain caches, if we want to free up space we can clean this out with.

    $ sudo apt-get clean
    
  • Use Cruft

    There are recommendations to use Cruft. I personally don't as it can be dangerous.

Note

This is too complicated and may be dangerous.

adequate --tag library-not-found --all

The output was this:

spotify-client: library-not-found /usr/bin/spotify => libcef.so
uno-libs3: library-not-found /usr/lib/libuno_cppuhelpergcc3.so.3 => libreglo.so
uno-libs3: library-not-found /usr/lib/libuno_cppuhelpergcc3.so.3 => libunoidllo.so
uno-libs3: library-not-found /usr/lib/libuno_cppuhelpergcc3.so.3 => libxmlreaderlo.so
libsvn1:amd64: library-not-found /usr/lib/x86_64-linux-gnu/libsvn_auth_kwallet-1.so.1.0.0 => libkdecore.so.5
libsvn1:amd64: library-not-found /usr/lib/x86_64-linux-gnu/libsvn_auth_kwallet-1.so.1.0.0 => libkdeui.so.5

You can then figure out what to do about each library problem. I couldn't figure out spotify, for libsvn I just removed the library and subversion as I'm not using them.

Not sure how useful it is for me as it took a long time and I didn't really do anything when it told me this. Could be because I'd already run GTKOrphan.

Fix packages

It's worth making sure that the package database is correct.

In Synaptic go to Edit and Fix Broken Packages, then click Mark all Upgrades button, finally hit the Apply button.

Apply updates

Make sure the whole system is up to date by updating all sources and doing any updates that are required.

Reboot the system

Probably best to reboot the system at this point to check that all software is fine: generally you'll have removed some kernels along the way!

Back then there was an install CD and a live CD for Ubuntu 5.04

Ubuntu installation media, Ubuntu 5.04 CD by Elizabeth K. Joseph, CC BY-NC-ND 2.0

d. Backup

We need backups of the whole system. It's worth doing this as a normal backup and an image. Disk images are much easier to recover with. We're also going to backup the whole system, including /boot, so that we can recover to this point.

If I'm short of time the alternative is to make sure I have the previous installer on a USB drive and to backup my home directory: since I previously installed this version it should be safe to depend on it.

Use LVM to snapshot

I'm using LVM which means I get snapshots "for free". I snapshot everything except for any swap LV's.

  • Make snapshots of everything

    Make snapshots of all the partitions.:

    $ sudo lvcreate -L10G -s -n rootsnapshot /dev/vgUbuntu/root-ubuntu-lv
    $ sudo lvcreate -L10G -s -n homesnapshot /dev/vgUbuntu/home-ubuntu-lv
    
  • Create a backup image of each partition

    We're using cat to create the images so they are full disk images, this means they take up a lot of space but can simply be copied back over should we have to recover. The whole thing takes a very long time, with the main /home backup taking over two hours to do!

    $ sudo sh -c "cat /dev/vgUbuntu/rootsnapshot | pv -s 15G | gzip > /mnt/backups/snapshots/`date +%Y%m%d`rootsnapshot.dd.gz"
    $ sudo sh -c "cat /dev/vgUbuntu/homesnapshot | pv -s 123G | gzip > /mnt/backups/snapshots/`date +%Y%m%d`homesnapshot.dd.gz"
    $ sudo sh -c "cat /dev/vgUbuntu/powersgsnapshot | pv -s 35G | gzip > /mnt/backups/snapshots/`date +%Y%m%d`powersgsnapshot.dd.gz"
    

Note

See this discussion on using cat instead of dd

  • Remove the temporary snapshots

    Before we copy the drive information to our backup we remove them so there's not confusing information in the backup.

    $ sudo lvremove /dev/vgUbuntu/rootsnapshot
    $ sudo lvremove /dev/vgUbuntu/homesnapshot
    $ sudo lvremove /dev/vgUbuntu/powersgsnapshot
    $ sudo lvs
    
  • Create a backup of your /boot partition

    We can't snapshot /boot because it's not within LVM. To guarantee the integrity of the backup the partition can't be written to during the backup. Depending on how conservative we are, either drop to runlevel 1 or boot the system from a USB drive.

    $ sudo sh -c "dd if=/dev/sda1 | gzip > /mnt/backups/lenovosg/snapshots/\`date +%Y%m%d\`bootsnapshot.dd.gz"
    
  • Record drive information

    Record various bits of drive set-up information which are useful when restoring.

    $ sudo sh -c "pvscan --uuid > /mnt/backups/snapshots/`date +%Y%m%d`-snapshotinfo.txt"
    $ sudo sh -c "sudo lvdisplay --maps >> /mnt/backups/snapshots/`date +%Y%m%d`-snapshotinfo.txt"
    $ sudo sh -c "sfdisk --dump /dev/sda > /mnt/backups/snapshots/`date +%Y%m%d`-partitions-backup-sda.sfdisk"
    

If you do the install and it all messes up then you can restore from snapshot.

e. Download Ubuntu

Download the new version and verify the image.

  • Download the new version

    Download the new version using the Web or a Torrent.

    $ mkdir ~/tmp/download
    $ transmission-gtk http://releases.ubuntu.com/16.04/ubuntu-16.04.2-desktop-amd64.iso.torrent?_ga=2.200209574.949229349.1493635885-657005342.1481661640
    
  • Download the SHA256 sums

    Both the SHA256SUMS and SHA256SUMS.gpg.

    $ cd ~/tmp/download
    $ wget http://releases.ubuntu.com/16.04/SHA256SUMS
    $ wget http://releases.ubuntu.com/16.04/SHA256SUMS.gpg
    
  • Add the public key

    If we don't have it already we have to add the official Ubuntu GPG key. Verify that the key has the correct fingerprint.

    $ gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys "8439 38DF 228D 22F7 B374 2BC0 D94A A3F0 EFE2 1092" "C598 6B4F 1257 FFA8 6632 CBA7 4618 1433 FBB7 5451"
    $ gpg --list-keys --with-fingerprint 0xFBB75451 0xEFE21092
    
  • Verify the SHA's are correct

    Verify that the SHA sums file is correct by checking the GPG signature on them.

    $ gpg --verify SHA256SUMS.gpg SHA256SUMS
    
  • Verify the download

    Check that the SHAs match with the file.:

    $ sha256sum -c SHA256SUMS 2>&1 | grep OK
    

f. Create a bootable USB

We create a bootable USB of the current system so that we can restore back to the current state. The easiest way is to use the disk creator in the menu.

$ lsblk
$ sudo ddrescue -D --force --verbose <ubuntu-15.04-image> /dev/sdb

Note

To generate a list of devices that can be used to boot press F12 at start-up and then select USB. To access the BIOS do F1

g. Upgrade preparation

Turn off screen lock and blank screen as these can mess up the upgrade process. In XFCE this means:

h. Start SSHD (Optional)

It's useful to run SSHD if you have access to another machine. If there's a problem we can login remotely and save the system properly. I haven't had to do this in a long, long time but it pays to be cautious.

Basic steps are:

$ sudo apt install openssh-client
$ sudo apt-get install openssh-server

Configure the server /etc/sshd_config:

Port 2222
#PasswordAuthentication yes
AllowUsers bob sarah
PermitRootLogin no

Restart the server:

$ sudo restart ssh
$ sudo systemctl restart ssh

Test that it's working:

$ ssh localhost

Note if you have strict host checking running ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no 127.0.0.1.

See:

i. Run Update Manager

With everything up to date, simplified and backed up it's time to run the upgrade process. In Ubuntu update-manager is a front-end to the upgrade process, it uses dpkg but also a set of heuristics to improve the upgrade process. In other words use it!

Note

the Ubuntu development team work on update-manager, making sure that it's aware of specific upgrade problems and enabling it to resolve problems. You should use it and not just plain apt upgrades

Open up a terminal and do:

$ update-manager -d

A lot of upgrades happen without a problem but it's not unusual to get some problems with the process. The upgrader is trying to find a conservative path through all the applications you've installed, updated and altered - sometimes it needs some help.

It's best to note down any changes you see in packages where you think you'd like to reinstall them later. I use the -d switch to figure out any changes I want to be aware of first. Here are some of the changes I noted down:

No longer needed

In many cases it will propose to remove applications that have been installed from repositories it doesn't know about, or if there are fundamental changes in the system. In this case I use LaTeX a little bit and the distribution version had changed. It notified me that it wanted to remove xscreensaver for some reason, I noted this down as I thought I'd need it later.

  • luatex
  • xchat-indicator
  • xscreenserver

No longer supported by Canonical

Changes in whether Canonical supports a component point to alterations in fundamental components. The interesting one was Consolekit as this was previously important.

  • consolekit?
  • lynx?
  • sshfs
  • ssh-askpass-gnome
  • texlive
  • texlive-latex-extra

Remove

The Removes are almost all programmes where Dpkg doesn't know the repository. The important one for me is Freemind, but changes in the JDK were also worth knowing about.

  • airvpn
  • bzr-fastimport
  • openjdk-6-jre; openjdk-7-jre
  • xchat
  • freemind

Changes

System programs have configuration files that are in the package. Dpkg will notify you if your version is different to the distributions packaged version. The trade-off is that my version has my configuration, but sometimes new versions will have important configuration changes.

For each change I inspect it and decide if I want the distribution version or my own. In this example it wanted to change /etc/init/lightdm.conf as my version had been configured along the way.

start on ((filesystem
-           and runlevel [!036]
+           and runlevel [!06]
            and started dbus
            and plymouth-ready)
           or runlevel PREVLEVEL=S)

 stop on runlevel [016]
Esoteric rites of penguin worship LOLZ

Strange apt commands, sometimes called the Esoteric rites of penguin worship, by LONnptech.org

j. Check proposed alterations

If you used the -d switch earlier then all the changes it proposes should be known.

After the upgrade is complete it asks to reboot. Before I do this I always update and apply any upgrades:

$ sudo apt-get update
$ sudo apt-get upgrade

I always do this before I reboot just in case there was some terrible kernel issue and an emergency update had to be sent out.

Next reboot and check that everything is fine.

k. Install any additional applications

Earlier we noted down applications that were being removed during upgrade. We now go back and re-install them. This will involve searching the Web and finding the PPA or repository that they came from.

In some cases we land up with applications where the dpkg database doesn't know the repository that the applications came from: common if you removed the repository from the deb configuration or if you installed it manually (e.g. dpkg install from the command line). To see these go into Synaptic in 'Installed (local or obsolete)' and then see if these are available from other sources as apt sources so that they're upgraded to the right versions.

The best way to do this is from within Synaptic, in the Settings->Repositories->Other Software section.

l. Test system hardware

We've changed the system from the top to the bottom so it's worth testing both the hardware and software to check that everything works. It's much better to catch a problem now rather than to be confused months later about some changed behaviour.

In reality, the upgrade process has already take a while so there may not be time (or energy) to test everything at once. I keep a file with everything in and then test it all over a couple of days. I've noted some of the things I test, but not all as this tutorial is already long enough!

For hardware tests, first install hardinfo, and then run it, you can also generate a report for later review. Mostly you just check through the main Summary information:

Element Test Status
Environment Tested under XFCE as it has some additional hardware information which is easy to access.  
Processor Double check that both CPU's are operating. 2x Intel Core 2 Duo CPU P8600 @ 2.40 Ghz Pass
Memory Check that it can address all 8GB. I always double check in top. Pass
Hard drive ATA Samsung SSD 840. You'll know about it if this isn't working. Pass
Screen

Mesa DRI Intel GM45 Express Chipset.

Open the 'Displays' applet in the Systems Settings and check. Resolution: 1920x1200 (16:10). The resolutions or dual-screen get messed up in upgrades some times.

Pass
Network interfaces
  • Do all the network interfaces work?
  • Ethernet
  • WIFI
  • Bluetooth: test with bluez
Pass
Audio
  • HDA-Intel
  • Check by starting Rythmnbox
  • Check the system settings - pavucontrol
Pass
Audio BT
  • Check that headphones work correctly with BlueTooth
  • blueman-manager
  • output: pavucontrol; rhythmbox
  • input: skype
FAIL
Camera

HD Pro Webcam C920 (/dev/video0)

  • Check it with Skype
  • Eventually, Skype took a long time to start
Pass
USB

On laptop (not on dock). Check list with lsusb

  • Mouse: standard USB
  • Drive
  • Trackpad
  • Pass
  • Pass
  • Pass
Keyboard

USB Kinesis Advantage Pro

  • Involved as I mess with the keyboard map when plugged in
Pass
Logitech

Logitech wireless receivers: software is Solaar. Small USB plugin and the Anywhere MX mouse

Works, selecting text is a bit broken but this is an old issue which is likely the hardware.

Pass
Laptop
  • Suspend / Resum - Triggering from the software on exit menu works
  • Hibernate: has to be configured
  • Pass
  • Pass
Trackpoint
  • TrackPoint
  • Sleep button (FN+F4): Disabled in power manager
  • Wifi button (FN+F5):
  • Lid button (suspend)
  • Lid button (resume)
  • Power button (short tap)
  • Pass
  • Pass
  • Pass
  • Pass
Dock
  • USB: test with a USB drive
  • Optical drive: HL-DT-ST DVDRAM GSA-U20N
  • Drive detected in hardinfo; didn't test as don't use
  • Pass
  • Pass
Printers Check printer shows: system-config-printer - HP LaserJet P2015 (NPI8261AA; 192.168.1.253); via HPLIP Pass

Bluetooth Sound

A good example of the sorts of issues that happen from time to time was that bluetooth sound appeared not to be working correctly. In the end it turned out to be a change in the way that Bluetooth drivers work and my own hardware configuration.

See: X220 Community page and Bluetooth on Ubuntu 16.04

m. System software tests

Check that all the applications have installed and are working.

Element Test Status
Display manager Test that the display manager (LightDM) shows and logs in the desktop environments I use (Xfce, i3, Unity). Pass
Resolution changes

Check that I can alter the X configuration; output to my main screen; turn-off the laptop screen.

xrandr --query
xrandr --output VGA1 --mode 1920x1200
xrandr --output LVDS1 --off
Pass
Desktop Env XFCE Check that XFCE desktop environment works properly Function - Change screen: GUI in accessories - Switch windows: <Super key>1-0 Pass
Desktop Manager

Check that i3 configuration works properly.

  • Change screen: laptop and flat panel (above)
  • Switch windows: <Super key>1-0
  • Top bar: <Super key>
  • Network manager: check it connects
  • Launcher: <Super>d
  • Launcher: <Super>g
  • Kill focused window: <Super key><Shift>q
  • External: FN<brightness buttons>
  • External: Screen lock: <Super><Ctrl>l
  • External: Filemanager: <Super>f
  • External: Power management: FN<sleep button> ; Lid switch
  • External: Exit session: <Super><Shift>e
Pass
Multiple logins Check that I can login on multiple VT's so I can run more one session at a time.  
Encrypted files Check that encryption is working correctly. Pass
Synaptic Check that the system software is all fine. Pass
SSH Check we can SSH to standard external sites. Pass
Password agent Check that password agent is working. Pass
Sudo Check that gksudo is working correctly. Pass
Power management Check power management is up and running. sudo tlp-stat -s Pass
Application containers Check that application containers are all fine. Pass
Email at (@) symbol illuminated sign

Email in lights by Chris Goldberg, CC BY-NC 2.0

n. Application software tests

Check through the most important applications and check that they work correctly. I haven't included the test scenarios for all the applications that I use as it would make this page too long (and they're pretty rough). Here's an example set of test cases for Mutt:

  • Version 1.5 in archive; changed to PPA for vim 1.7

  • Mbsync
    • Receive email
    • Check for email regularly (mbsync daemonize)
    • Delete email (mbsync sync back)
  • Mutt
    • Read email
    • Write email
    • Send email
  • msmtp
    • Send email (check .msmtp/log/msmtp.log and msmtp.queue.log)
    • Runs queue regularly (check .msmtp/log/msmtp-queu-daemonize-output.log)
  • Other
    • Keychain for ssh etc

Application software

The main application software that I test is:

Element Test Status
Browser - Firefox Browser checks. Pass
Email - Thunderbird Check that email is fine. Pass
Browser - Chromium Check that alternative browser is running. Pass
Email - Mutt Check that alternative email is running. Pass
Filemanager - Thunar Check that the file manager works. Pass
IRC - weechat Check that the IRC client is all fine. Pass
Music - Spotify Check we have music. Pass
Personal Cloud - Google Cloud Check we can store files online. Pass
Personal Cloud - Dropbox Alternative Cloud storage. Pass
Personal Cloud - SpiderOak Alternative Cloud storage. Pass
Text editor - Vim Primary text and programming editor. Pass
Note Taking - Vim Wiki Primary Note taking system. Pass
Office Productivity - LibreOffice LibreOffice office productivity Pass
Version control - Git Version control for files and programs Pass
Todo management - task Todo management for the command line Pass
Mindmaps - Freemind Mindmapping for organisation Pass

Development / Utility

The main utilities and development software that I check are:

Element Test Status
Python Python development environment Pass
Clojure Clojure development environment Pass
Diff Diff for comparison of code Pass
Bash Command line environment Pass
Tmux Terminal multiplexer Pass
Ranger Command line file manager Pass
nmcli Network manager cli client Pass

o. Specific configuration

There's generally some reconfiguration that has to be done at this stage. For example, resetting up ssh-agent and fixing hibernation. I keep a log of all changes I make, so it's easy to go back through this and double check that nothing has been missed.

p. Document upgrade

I learnt a long time ago that if it isn't written down then it didn't happen. So I keep a log of all changes that are made to the system and specifically for each upgrade I keep a file. It doesn't have to be snazzy but some notes about what you changed can really save you later when something goes wrong and you need to know the last few stages.

  • Document the version of the kernel that it initially boots with

    Keep this version, lock it in synaptic, as you know things work from here.

Common upgrade problems

Artist Trading Card - "Don't panic" - flaming laptop

Don't panic!, image by Sarah, CC BY 2.0

You're changing the entire operating system and applications so it's possible you might have some problems. But since this is open source there's always a way to solve them!

Extensive use of PPA's

Too many PPA's confusing the upgrade process. The problem with PPA's are that they are outside the Ubuntu official repositories so the upgrade paths and conflicts aren't tested. Either don't use them, or strip them out ahead of an upgrade (downgrading to the Ubuntu repository version) and then reintroduce them after the upgrade.

Lack of space

Not enough space to download and install. This happens to me quite a bit. When I ran the update-manager on my first upgrade I got this message:

The upgrade has aborted. The upgrade needs a total of 4,407 M free space on disk '/'. Please free at least an additional 747 M of disk space on '/'. Empty your trash and remove temporary packages of former installations using 'sudo apt-get clean'.

The problem was that my / directory didn't have enough space to download all the packages to for the upgrade process. I needed some temporary space for it to download the packages into.

  • Create a temporary TMP

    Create a new temporary location, in my case I'm using LVM so add a new filesystem and mount it, but I've also previously used a USB drive.

    sudo lvcreate -L10G -n aptstore vgUbuntu
    sudo mkfs.ext3 /dev/vgUbuntu/aptstore
    sudo mkdir /mnt/aptstore
    #sudo chmod 777 /mnt/aptstore
    
    sudo mount /dev/vgUbuntu/aptstore /mnt/aptstore
    sudo mkdir /mnt/aptstore/archives
    
  • Backup the APT archive locally:

    Copy the apt archive to the new location so that we don't have to redownload anything later.

    $ sudo cp -pR /var/cache/apt/archives/* /mnt/aptstore/archives/
    
  • Rebind

    We rebind the /var/cache/apt/archives location to our new filesystem.

    sudo mount --bind /mnt/aptstore/archives /var/cache/apt/archives
    
  • Run apt autoclean to make sure everything is fine

    The check command updates the package cache and checks for broken depenencies, it should catch any problems with the set-up. To be extra cautious run autoclean which will delete anything from the cache packages that can no longer be downloaded (e.g. because the apt configuration has changed). Finally run clean if you want to redownload everything.:

    $ sudo apt-get check
    #$ sudo apt-get clean
    $ sudo apt-get autoclean
    
  • Run the upgrade

    We can re-run the upgrade at this point and at but don't reboot at the end as we have to refix the apt cache location.

    $ update-manager -d
    # don't let it reboot when it comes to an end
    
  • Unbind the /var/cache/apt mount

    Restore /var/cache/apt so we're back to the smaller size.

    $ sudo umount /var/cache/apt/archives
    
  • Make sure apt is clean

    We don't want any old files so run apt-clean to remove any old downloaded packages from before the upgrade.

    $ sudo apt-get clean
    
  • Sync apt status back

    Move across any packages that are still in the cache. If there's not enough space this isn't really required.

    sudo rsync -av /mnt/aptstore /var/cache/apt
    
  • Run apt to check everything is fine

    Update and do any final updates before you reboot the system.

    $ sudo apt-get update
    $ sudo apt-get upgrade
    
  • Reboot

    Now you can reboot and check everything is OK.

The system is blown up

If the system has completely blown up and it's impossible to figure out what's happening then it's time to go back and do a clean install. This will wipe out everything except for home.

Technically, we have the backup which we can use to restore back to our previous state, I've never had to use this, but the commands are (roughly):

  • Boot the machine using a USB stick

    Boot the system using the Ubuntu USB stick that you created earlier.

  • Go to the rescue environment

    Ubuntu's installer has a rescue option.

  • Mount the partitions and chroot

    Get all the partitions mounted under the right environment and chroot into them:

    $ sudo mount -t ext3 /dev/mapper/vgUbuntu-root--ubuntu--lv /target
    $ sudo mount -t ext3 /dev/sda1 /target/boot
    $ sudo mount -t ext3 /dev/mapper/vgUbuntu-home--Ubuntu--lv /target/home
    $ sudo mount -t proc proc /target/proc
    $ sudo mount -t sysfs sysfs /target/sys
    $ mount -o bind /dev /newroot/dev
    $ sudo chroot /target
    
  • Restore the boot environment

    Grab the boot image from where you backed it up to and write it over the current boot.

    $ rm -Rf /boot/*
    $ gunzip /mnt/backups/lenovosg/snapshots/`date +%Y%M%d`rootsnapshot.dd.gz
    $ sudo dd if=/target/rootsnapshot.dd of=/dev/sda1
    $ sudo grub-install /dev/sda
    $ exit
    
  • Restore to the snapshots

    Restore the rest of the system. Umount all the LV's that are mounted so they can merge.

    $ sudo umount /target/*
    $ sudo lvconvert -merge homesnapshot -i -1
    $ sudo lvconvert -merge rootsnapshot -i -1
    

Final thoughts

Alright! We're all done with a new upgraded system - ready to enjoy the latest goodies! As you've seen this approach is cautious, trying to ensure success, but the cost is time and effort. Using this process makes it simple to move from each successive release of Ubuntu or from LTS-to-LTS with a working system and all your configurations intact. If you have tips or ideas on how to do upgrades, or you used this process, leave a comment below!

blog comments powered by Disqus