The goal of this post is set-up a NeoMutt mail configuration that locally mirrors an IMAP servers mailboxes. This will mean we'll be able to read and write email when offline. Any changes made to the local email, such as deleting an email, will also be made on the IMAP server.
The way this works is that the remote IMAP server's mailboxes are synchronised with the local system's mailboxes. New email that comes into the remote IMAP server is downloaded to the local email store (i.e the maildir directory). The user accesses the local mail store using NeoMutt as usual. Any changes, for example an email is deleted, are written to the local mailboxes. On the next sync the local mailbox's state is synchronised with the remote IMAP server, so in this example the email is deleted on the remote IMAP server as well.
An important element in this type of set-up is that it's two-way synchronisation, so changes made on either side will propagate to the other. For example, if you use Runbox's Webmail and delete an email on the IMAP server, the change will be synchronised to the local mailboxes. And, the same in reverse when using NeoMutt. That's why I'm calling it a mirrored IMAP set-up.
The two popular utilities for mirroring IMAP are offlineimap and Isync. Offlineimap was popular but it suffered a lack of maintenance. Consequently, isync is a bit more popular now and it's what this post covers. One thing to bear in mind with isync is that the actual binary is called mbsync so this is the name that's often used for it.
This is part 4 of the NeoMutt email series, you can find them all on the NeoMutt resources page. As before I'm using Runbox as my email provider - highly recommended - but this will work with any standard provider (including Gmail).
It's a pretty stable project and is widely available, to install it on Guix do:
guix shell isync
Isync is a little bit more out of date on Ubuntu, but it is available in the archive:
sudo apt install isync
Mbsync is configured by creating a configuration file, the default location is $XDG_CONFIG_HOME/isyncrc which for me is ~/.config/isyncrc. The older configuration option is to have a ~/.mbsyncrc file.
The model for thinking about how this works is that you have a local store and remotely there's a remote store (on an IMAP server), we tell mbsync how to connect the two together using a channel:
[local store] <---> [channel] <----> [remote store] <--> [remote server]
If there are a lot of channels then these can be put into a group. This means there are 5 steps to configuration:
A remote mail store defines the IMAP account where email is, this section contains the authentication and server details of the remote IMAP server.
IMAPAccount <account-name>-imap # Short name for the account Host <imap-server.somehost.com> # Hostname FQDN of the IMAP server User <userid@mailprovider.net> # Login userid Pass <somelongpassword> # Account password - use for testing only TLSType IMAPS # Type of SSL to use AuthMechs Login # Authorisation mechanism is password SystemCertificates yes # Use the systems TLS certificates
The IMAPAccount tells mbsync the name of the account and that the next section contains parameters for it. The <account-name> chosen will be used in other places to tie everything together.
The Host is the mail server address, for example for Runbox that is mail.runbox.com. The User is the userid to use for login, as we did before when configuring IMAP this is likely to be the user%domain but check with your mail provider's documentation.
For testing I just add the password using the Pass section. After completing testing it can be changed to using GnuPG and an encrypted file.
In newer versions of mbsync to specify the TLS type use the TLSType option with either STARTTLS or IMAPS. Additionally, specify TLSVersions with TLSv1.2. If you don't set TLSVersions it will use version 1.2 and above.
Mbsync automatically uses the system's certificate file so this doesn't need to be specified.
The AuthMechs Login provides a variety of authorisation options: to use passwords this has to be specified so that password authentication works.
The next step is to define the remote store which is the location on the IMAP server where the mail files reside. This is pretty short:
# Remote store IMAPStore <account-name>-remote Account <account-name>-imap
This says that we're defining an IMAPStore and we define a name for it. The general convention is to call it something with -remote on the end e.g. myprovider-remote. Then we tell mbsync that the location of this store is on the Account which we defined just above.
If you're wondering why we didn't have to specify a location for the mailboxes it's because mbsync "the reference point for relative Paths is the current working directory".
The local store is the location on the local file system where email will be synchronised to. In my case I'm using $HOME/.mail for all email storage. The first task is to create a subdirectory for all email for this account to be stored in:
# Create the local store dir mkdir -p ~/.mail/<account-name>
Going back to our mbsync configuration file we define the local storage:
# Local mail store MaildirStore <account-name>-local Path ~/.mail/<account-name>/ Inbox ~/.mail/<account-name>/INBOX Subfolders Verbatim
The MaildirStore is the name of a local store, again the convention is to use an account name with -local on the end this time. The Path is the path to the local mailstore, which is the account specific sub-directory: note that this must have a trailing forward slash to specify an entire directory. Inbox is the path to the inbox location (which is generally going to be in the MaildirStore path).
The Subfolders option specifies how subfolders are created for the local store: Verbatim converts them from the IMAP system which uses dots to using forward slashes so they're standard sub-directories.
Channels are the relationship between the local and remote mail stores, this is where everything is tied together.
# Channel connecting <account-name>-remote to <account-name>-local Channel <account-name> Far :<account-name>-remote: <mailbox> Near :<account-name>-local: <mailbox> Patterns * # Sync all folders Sync Pull # NOTE: Change to 'All' when testing is finished Expunge Near # NOTE: Change to 'Both' when testing is finished Create Near # NOTE: Change to 'Both' when testing is finished Remove Near # NOTE: SyncState * # not setting this defaults to writing state files in ~ CopyArrivalDate yes MaxMessages 50 # NOTE: useful for testing ExpireUnread yes # Remove old unread, to let up to 50 more in
The Channel parameter gives the channel a name, this will be used to run commands so should be a memorable name.
The Far and Near are the two mail-stores that are being connected together, which will be the remote and local ones from earlier, they are specified as :mailstore-name: using colons at the start and end. For example:
There are different ways to define which mailboxes to synchronise, which we'll discuss in more detail later. For now consider it in three steps:
The next option is the type of synchronisation to use, using Sync Pull for testing is a good way of checking everything is working as it only pulls down changes. When testing is complete it can be changed to Sync All which will bi-directionally sync changes including flag changes and deletions.
The Expunge option defines how email will be deleted from the mail stores, for testing this should be set to Near so that email deleted on the local side isn't deleted on the remote side. When testing is complete this can be changed to Expunge Both so that email deleted either remotely by another email client (e.g. Webmail) will also cause a deletion in the local store, and if email is deleted on the local side it's also deleted in the remote store.
The next two options Create and Remove define what happens when there are new mailboxes. By setting both to Near we're setting it so that the remote server is the master copy, and any mailboxes that appear there are also created in the local store. After testing this could be changed to Create Both so that if a new mailbox is created on the local system then it will appear on the remote one as well. Setting Remove Both might be a bit more of a worry if you're messing around with the configuration.
The SyncState option specifies where this channels synchronisation state files will be stored. For a Maildir local store using * means it stores the state file (.mbsyncstate) in the local directory.
The IMAP server records when the file was created, this is generally close to when the email actually arrived. The setting CopyArrivalDate yes records this time-stamp for you.
For testing set MaxMessages 50 which means that only 50 messages will be synchronised to the local store, along with ExpireUnread yes which means it will only sync 50 total messages even if the user has 100 new ones in a mailbox - it deletes the oldest messages to make way for newer ones. These can be removed after testing is complete.
There are a few options for choosing mail boxes to synchronise:
There are quite a lot of options, which is why a lot of examples in blog posts are subtly different.
The easiest rule is that if no mailbox is defined on the channels Far or Near lines then mbsync will look for INBOX. For example, something like this:
Channel mymail-inbox Far :mymail-remote: Near :mymail-local: Sync Pull Expunge Both
This will find the INBOX and will pull it down, but won't match any other mailbox. You don't need a Patterns line for this and it causes some odd issues if you set one.
For total granularity of matching specify a mailbox on the Far line and it will be synchronised to the mailbox that's specified on the Near line. This is good for matching individual mailboxes, with the downside being that it makes the configuration file longer.
This is what I use as it's very explicit: for each mailbox that I want to sync I have a channel, and then they're all grouped together in a group.
For example a pair would look something like this:
Channel mymail-drafts Far :mymail-remote:Drafts Near :mymail-local:Drafts
Where we're calling the channel mymail-drafts and then defining the Far as the remote server mymail-remote and the local mail store as my-mail-local. Mbsync will login to the remote server and find the Drafts folder and synchronise it into the local mail-store path, again into the Drafts folder.
Again, note that you don't need a Patterns option.
To match more than a single pair of mailboxes we can define a pattern to match. The easiest way to do this is to leave the mailbox section of the Far and Near lines empty. This means that mbsync will login to the root of the IMAP server and will then use the Patterns option to match:
Channel mymail-inbox Master :mymail-remote: Slave :mymail-local: Patterns * INBOX !Spam !Junk !Archive Sync Pull Expunge Both
This example matches the special folder inbox by specifying INBOX and everything else with *, it then excludes specific folders using ! for example the junk folder (!Junk). It's possible to write complex rules to match all email folders like this. Personally, I prefer to have simple rules for each folder and use a Group as it makes it easier to debug.
Finally, if there's a hierarchy of folders then it might be useful to use both mechanisms to match. For example, I have Archives with subfolders for each year (2015, 2016). To match this I use both:
Channel mymail-archives Master :mymail-remote:Archives Slave :mymail-local:Archives Patterns * # Match multiple levels e.g. Archives/<year> Sync All # Sync all email changes, including flags: bidirectionally Expunge Both # Deleted mail: bidirectional sync Create Both # Mailbox creation: bi-directional sync Remove Both # Mailbox removal: bi-directional sync Syncstate * # Put state file in maildir CopyArrivalDate yes # Use server date/time for email
In this example mbsync logs into the server and goes to the Archives folder location, it then matches all folders underneath this using Patterns *.
Groups allow selective synchronisation by listing a set of channels. This is useful because it's then easy to specify mailbox pairs in individual channels which are easy to test. And, if you only want some email on a particular device (e.g. only INBOX on a laptop) it's very easy to configure:
Group mymail-group Channel mymail-inbox Channel mymail-drafts Channel mymail-archives Channel mymail-sent Channel mymail-trash
A Group is assigned a name (e.g. mymail-group) which will be used on the command line. It can have one or more Channels in it.
Messing around with email is very scary with the risk that something will be deleted. Mbsync has some commands to show what's happening.
mbsync --list-stores --verbose <store-remote>
This command will check whether the server and remote store work. If they do it will list all the mailboxes on the remote store. It can also be used on the local-store.
To check that either a channel or group is configured correctly do:
mbsync --list --verbose <group-name>
This will show each channel, what it's matching and where it will synchronise to.
There's quite a few options to debug, for example I find --debug-net-all useful to see precisely what mbsync is seeing on the IMAP server.
To synchronise email run it like this:
mbsync --verbose --dry-run <group-name / channel-name>
This means that we can synchronise entire groups, or channels. For example, using the group and channels from above we could do:
mbsync mymail-group # sync everything in group mybsync mymail-inbox # sync inbox channel only mbsync mymail-inbox mymail-drafts # sync two channels
There's a lot of flexibility in the channels and groups concept. If storage is limited on a particular machine one strategy is to create a group with a couple of channels and only sync them.
When mbsync runs the output will show the channel or group and what is being matched, something like this:
Channels: 5 Boxes: 10 Far: +0 *4 #0 -0 Near: +23 *0 #0 -0
This says there are 5 Channels in the group, a total of 10 Boxes (mailboxes) were synchronised. On the far side there were 0 new messages (the plus sign) synchronised to it, 4 had flags changed (the * sign) and there were 0 trashed messages (the hash symbol) and 0 expunged (the minus symbol). On the Near side local email received 23 new messages, 0 had flags changed, 0 were trashed messages and 0 were expunged.
During testing the IMAP password's been in the configuration file unencrypted. We can fix that by storing the credentials using GnuPG. We already set these up for using NeoMutt previously, so it's a simple change in the IMAPAccount section:
PassCmd "gpg --quiet --for-your-eyes-only --no-tty --decrypt ~/sec.gpg"
Last time we configuring NeoMutt for multiple accounts using folder-hooks, with one account on Runbox and one on Gmail. This time, we update the mailboxes section so that NeoMutt uses the local mirrored mailboxes:
set folder = ~/.mail/runbox # set to the primary account's folder
set mbox_type = maildir
set spool_file = +INBOX # 'spool' is now pointing at the main inbox
set move = no # don't move mail from spool as it's pointing
# at the main inbox
# initial mailboxes
set record = +Sent
set trash = +Trash
set postponed = +Drafts
set mail_check = 120
set timeout = 30
set mail_check_stats = yes
set mail_check_stats_interval = 120
# Account 1: mailboxes and folder hook
mailboxes -label R:Inbox -notify -poll ~/.mail/runbox/INBOX \
-label R:Drafts -nonotify -poll ~/.mail/runbox/Drafts \
-label R:Sent -nonotify -poll ~/.mail/runbox/Sent \
-label R:Trash -nonotify -nopoll ~/.mail/futurile/Trash
folder-hook ~/.mail/runbox/ 'source ~/.config/neomutt/runbox-account-neomuttrc'
# Account 2: mailboxes and folder hook
mailboxes -label G:Inbox -notify -poll ~/.mail/gmail/INBOX \
-label G:Drafts -nonotify -poll ~/.mail/gmail/Drafts \
-label G:Sent -nonotify -poll ~/.mail/gmail/Sent \
-label G:Trash -nonotify -nopoll ~/.mail/gmail/Trash
folder-hook ~/.mail/gmail/ 'source ~/.config/neomutt/gmail-account-neomuttrc'
This should all be pretty straightforward at this point. Initially the $folder points to one of the accounts (e.g. your primary one) and sets up the initial mailboxes. One thing to note is that as we're not accessing the email store over IMAP all of those settings (e.g. $imap_idle) have been removed, and $mailcheck can be tuned to whatever is ideal.
The mailboxes command operates the same, this time it's set to find mailboxes in the local store (e.g. ~/.mail/runbox/). This explains the folder hook which matches against ~/.mail/runbox/ or ~/.mail/gmail/ and then runs the appropriate account-specific configuration.
Here's the shortened runbox-account-neomuttrc:
# vim: ft=muttrc fdm=marker foldcolumn=2 set folder = ~/.mail/runbox set spool_file = +INBOX set record = +Sent set trash = +Trash set from = user@email.domain set real_name = "Your Name" # set status: black (color0) text, yellow (color11) foreground color status color0 color39 # change status color for this account
Then the gmail-account-neomuttrc:
# vim: ft=muttrc fdm=marker foldcolumn=2
set folder = ~/.mail/gmail
set spool_file = +INBOX
unset record # Gmail automatically records sent email
# if $record is set it will duplicate mail
set trash = +[Gmail]/Trash
set posponed = +[Gmail]/Drafts
set from = user@gmail.com
set real_name = "Your Name"
# set status: black (color0) text, DeepSkyBlue1 (color39) foreground
color status color0 color11
As the default is to use the primary account, there's no need for a separate local configuration. When writing email in an account, any postponed email will be saved to the correct Drafts folder.
At this point we can locally mirror email by manually runnng mbsync. When testing is complete change the settings to bi-directional sync (Sync All in each Channel) and remove the limit on the number of messages to download (MaxMessages).
At the start of this series I wanted to cover three different ways to retrieve and interact with email using NeoMutt: local email on a Unix server, native IMAP and mirrored IMAP. Mirrored IMAP is a bit complex as we have a few moving parts, but it's the most flexibile.
To make our mbsync configuration sing we need to automate it so that new mail comes in automatically. We'll tackle that next time, using Goimapnotify and a Network Manager dispatcher script.