Syncing IMAP Accounts Between Servers

Due to a change in our mail server infrastructure, I had to move all emails from the old account to a new IMAP account. Obviously if you create the new account in your email client such as thunderbird, you can simply drag and drop the messages between the accounts. Unfortunately, timeouts and other errors prevent you from being sure that all emails have been moved between the servers correctly, if you rely on the mail client only.

Thanks to the tool offlineimap, this task of moving emails can be made much more reliable and it can be automated. The method I describe in this post can also be used for backing up your local machine and preserve your precious emails from being lost, for instance when your provider needs to close down or goes out of business. Offlineimap is a Python project actively developed on Github. Clone the latest version to your local machine as shown below and enter the directory:

git clone https://github.com/OfflineIMAP/offlineimap.git
cd offlineimap

The tool uses a config file where you need to specify your email accounts and settings. In the offlineimap directory, you will find two sample configuration files. The minimal config looks like this:

# Sample minimal config file.  Copy this to ~/.offlineimaprc and edit to
# get started fast.

[general]
accounts = Test

[Account Test]
localrepository = Local
remoterepository = Remote

[Repository Local]
type = Maildir
localfolders = ~/Test

[Repository Remote]
type = IMAP
remotehost = examplehost
remoteuser = jgoerzen

Copy this file to a location (e.g. /home/user/Offlineimap/offlineimaprc_migrate.conf) of your choice and open it in a editor.  For the example, consider that we have two mail servers. The server old.example.org currently contains all the mails that we need to move to the new server new.mailheaven.com. In the terminology of offlineimap, the remote repository is our source old.example.org. We will move the mails to the new server, which is denoted localrepository. This might be a bit confusing, as we plan to move the mails not to our local server, but to the server new.mailheaven.com. So keep this in mind, not to confuse the locations. This is the configuration that we will use in this example. It will copy all emails from the source to the target mailserver over a secure connection.

[general]
accounts = Email-Sync
maxsyncaccounts = 1

[Account Email-Sync]
remoterepository = old_server
localrepository = new_server

[Repository old_server]
type = IMAP
remotehost = old.example.org
remoteuser = olduser@old.example.org
remotepass = secretPassword
ssl = yes
maxconnections = 1
cert_fingerprint = 11111aaaaaaaaaa1111111111cccccccccc111111
folderfilter = lambda folder: folder not in ['Trash','Gesendete Elemente','Notizen','Kontakte','Sent','Junk-E-Mail','Postausgang','Entw&APw-rfe','Kalender','Journal','Gel&APY-schte Elemente','Aufgaben']
readonly = true
[Repository new_server]
type = IMAP
remotehost = new.mailheaven.com
remoteuser = newUser
remotepass = newSecretPassword
ssl = yes
cert_fingerprint = 222222222222222bbbbbbbbb22222222222222222
maxconnections = 1

The config file has several sections. The section general defines which accounts we want to synchronise. Here, we describe an offlineimap account, which is like a task. Do not confuse it with your actual email accounts. They are called repositories in offlineimap speech. In our example we have only one account called Email-Sync. The next section defines what we want to do with this account. We specify the source and the destination email accounts. As already said, the actual email accounts are denoted as repositories. Each repository is defined in its own section. This is the place to define the actual email accounts. Make sure to activate SSL. If offlineimap should complain about the CA or the certificate of the mail servers. it is very likely that you use self signed certificates and you need to tell offline imap that this is a trusted domain. We can achieve this by providing the fingerprint of the mail server certificate. You can use the following command to get the fingerprint from the mail server:

openssl s_client -connect example.org:993 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin | sed 's/./\L&/g' |  sed 's/\://g'```


This snippet retrieves the public certificate of the mail server and prints it in the terminal. The gmail mail server for instance has this fingerprint:

 openssl s_client -connect imap.gmail.com:993 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin SHA1 Fingerprint=E1:D6:E8:F1:69:72:75:CF:8F:1C:CF:E6:35:B0:42:30:C0:1D:62:C2```

Offlineimap expects the fingerprint in lower case letters and without the colon. The command above turns the fingerprint string into lower case and removes the colons. So again with the Gmail example, this is what the command does:

openssl s_client -connect imap.gmail.com:993 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin | sed 's/./\L&/g' |  sed 's/\://g'
sha1 fingerprint=e1d6e8f1697275cf8f1ccfe635b04230c01d62c2

You just need the later part of the line and copy it into the config file. Note that the configuration above stores your passwords in plain text, which is in general a bad idea. If you remove the lines with remotepass from the config file, offlineimap will prompt you for your passwords.

Create a Local Backup

Offlineimap is also suited very well for creating local backups of your IMAP accounts. You can store all your emails in the maildir format, which creates a folder structure with your messages on the hard drive. Use a config file similar file like the one below:

[general]
accounts = Mail-Server

[Account Mailserver-Local]
localrepository = Local-Backup
remoterepository = Mail-Server

[Repository Local-Backup]
type = Maildir
localfolders = /media/BackupPartition/Offline-Imap_BACKUP/
sep = /

[Repository Mail-Server]
type = IMAP
remotehost = mailserver.at
remoteuser = stefan
ssl = yes
maxconnections = 2
cert_fingerprint=33cccccccccccccccddddddddddddd4444444444
realdelete = no
readonly=true
folderfilter = lambda folder: folder not in ['Trash','Gesendete Elemente','Notizen','Kontakte','Junk-E-Mail','Postausgang','Entw&APw-rfe','Kalender','Journal','Gel&APY-schte Elemente','Aufgaben']

The realdelete=no flat option prevents that emails which have been deleted from your mail server will also be removed from the local backup.

Start Syncing

After you have your configuration ready, you can test the tool with the option –dry-run like this:

./offlineimap.py -c /home/user/Offlineimap/offlineimaprc_migrate.conf

Unfortunately, offlineimap throws errors in the dry-run, which is does not in the actual run. So if you can connect, you may need to run it directly. Note that we included the readonly flag for our source repository, in order to prevent changes there. Offlineimap can be a bit picky about folder names and translation rules. In the example above we excluded some folders from the transfer, as they caused problems either because they contained umlauts or other issues. Also there seems to be a race condition sometimes, leading to offlineimap complaining about existing folders. Repeating the process of creating the folders and waiting a few seconds seemed to resolve the problem. The documentation of offlineimap is quite extensive and a great source for finding out more on that topic.