Migrating from private IMAP to Google Apps Email

I have recently been moving several clients off of a basic email server to a Google Apps account (Free or Business, depending on the client), and several have had up to a couple years worth of email. I’ve tried several different techniques, but recently found some great command line kung fu that made the process extremely easy and much more accurate than the past attempts.

There are several different methods available for migrating email from various servers to Google Apps, including POP transfer from within each account’s settings, dragging and dropping via IMAP with a mail client like Thunderbird, or using the official email migration API from Google. However, each of these methods has its downsides: POP is slow and kludgy, dragging and dropping only works for low volume accounts, and the email migration API requires code be written and using a for-pay Apps account. What’s a geek to do? Turn to free and open source solutions!

A little bit of searching turns up that a command line tool known as imapsync does a fantastic job of, well, syncing IMAP accounts. I was tipped off by a blog post on Marius Ducea’s blog (lots of good stuff on that site by the way), which contains a basic introduction to imapsync and how to use it. The author of imapsync, Gilles Lamiral, continues to develop the software and is taking donations for different features, and also charges for the source and for support (visit Gilles’ site here). However, thanks to the licensing of imapsync (NSWF), it is also available for free from other hosted repositories. I was using CentOS, and got my copy from the following:

https://fedorahosted.org/imapsync/

After getting a copy of the Perl script, I ran perl -c imapsync to check dependencies as Ducea instructs, and found I was missing the Mail::IMAPClient module. That was a quick fix with

sudo cpan Mail::IMAPClient

Voila! The script ran just fine at that point. Ducea provides some good templates for running a sync between two servers, and it worked almost 100% right off the bat. However, I was getting issues with sent messages and deleted messages ending up in Google Apps with labels such as “[IMAP]/Sent Messages,” which obviously was not what I wanted. (Note: most such errors can be avoided by appending --dry --justfolders to the command and checking the output to see what imapsync discovers as folders/labels that already exist, and those which it plans on creating.)

A little more searching turned up Mark’s Pages of Stuff’s post about moving to Google Apps, and with a little modification there as well, I was able to get messages moved properly from the IMAP’s sent folder into Google Apps’s sent folder, trash, etc. On Mark’s posting, he uses folder labels like “[Google Mail] Bin,” but he notes that Google changes the name of labels based on locale. He just happens to be in England. However, running imapsync with --dry --justfolders will show you all the folders that exist on both the origin and target servers, so make sure that when you are planning your own migration that you inspect these values. For me, based in the US, the folders used the [Gmail] prefix and Trash instead of Bin.

Finally, after several test runs, I got the command line down to exactly what I needed, and I was off to the races. I set up a small bash script that would iterate over the several mailboxes I needed to migrate, pressed enter, and got about 30 messages in when the process died saying “Out of memory.” Out of memory, really? I was confused for a bit until I checked the message that it was dying on each time, and found that it had a rather massive attachment on it. The origin server, a small, shared hosting situation, must have capped the memory below the size of the attachment.

Luckily, imapsync has a --maxsize option, which will allow it to skip messages that total over a certain level of bytes. A little bit of experimenting on the server showed that I could only work with messages of about up to 2 MB in size, which was rather small, but doable in this situation. I added the –maxsize switch to the command, and ran through all of migrations again.

The final command that I used to migrate each account was as follows:

imapsync --syncinternaldates --host1 <ORIGIN MAIL SERVER> --port1 993 --ssl1 --user1 <ORIGIN USERNAME> --password1 <ORIGIN USER PASSWORD> 
--host2 imap.gmail.com --port2 993 --ssl2 --user2 <TARGET EMAIL ADDRESS> --password2 <TARGET USER PASSWORD> 
--useheader 'Message-Id' --skipsize --noauthmd5 --reconnectretry1 1 --reconnectretry2 1 --maxsize 2194304 \
   --folder "INBOX.Sent Messages" --prefix2 '[Gmail]/' --regextrans2 's/Sent Messages$/Sent Mail/g' \
   --folder "INBOX.Deleted Messages" --prefix2 '[Gmail]/' --regextrans2 's/Deleted Messages$/Trash/' \
   --folder "INBOX.Drafts" --prefix2 '[Gmail]/' --regextrans2 's/INBOX\.Drafts$/Drafts/'

(The three folder lines were unique to the IMAP configuration on the origin server, and will likely change for you. Also don’t forget --dry --justfolders while testing!)

After running the migrations on the underpowered origin server, I switched over to a VPS and ran the commands again, but this time without the –maxsize switch. No memory issues were encountered on the robust VPS, and I also was able to catch the last few newer emails that had been missed since the first migration.

Finally, after each account was migrated, I placed a forward on the email account on the origin server to redirect all new mail to <account>@<domain>.test-google-a.com, which acts as a secondary address for all Google Apps accounts before you switch over MX records. This allowed me to to a cutover from the old server to the new server without losing any new messages, and worked flawlessly.

All in all, the process took about 5 hours to transfer all of the data, and about an hour of research and playing around. Much better than dragging and dropping, or writing an entire mini-application to utilize the email migration API. Thanks to some great resources like imapsync and several helpful blogs, this was a much more pain-free experience than in the past!

Share and Enjoy

  • Twitter
  • Google Plus
  • LinkedIn
  • Email
  • RSS
  • HackerNews
  • Instapaper
  • StumbleUpon
  • Facebook

10 thoughts on “Migrating from private IMAP to Google Apps Email

  1. Mike Stroven

    Hi there,
    Love your blog! Hate Gmail… Was just wondering if you know of a way to use imapsync to migrate FROM gmail using the administrative OAUTH key, rather than individual passwords for every account?

    Reply
    1. Post author

      Whew, good question. I don’t recall seeing anything mentioned about OAuth support in imapsync, and a quick check on the source page doesn’t mention it either. Sounds like you might need to write some custom script to accomplish that. Sorry I can’t be of more help, but if you come up with a solution, definitely let us know!

      Reply
  2. Mobs

    Hello Joshua,
    Thanks for the post, just what I need however, since I am not a techie, this goes above my head. Where do I have to put the command, in terminal?
    He’re is my problem: Me and my wife use the same emailaccount on 2 different Macbooks. We pretty much bcc each other when sending email to keep each other updated. This in not the way – I know, but I haven’t got tha peace to set up an IMAP so we started like this and it’s been there for three years now. Recently we bought new MBP so we want to do it right now.

    We decided to use Google for business so we can use Google docs and Gmail. The thing is that we want to keep our emails we have accumulated tha past three/four years. So I just set up an IMAP account with the same emailaddress as what we’ve always used om both MBP. What next…

    Hope you can give some tips/clue.

    Cheers,
    Mobs

    Reply
    1. Post author

      Mobs,

      If you purchased a Google Apps for Business account and are not super technical, then you have a few other options as well (but yes, those commands are placed in the Terminal, so if you are feeling adventurous, go for it!).
      You may first want to try the Google Email Uploader for Mac if you have your messages stored in Apple Mail for example. Find it here: http://code.google.com/p/google-email-uploader-mac/
      You can also try the built-in Google Mail Fetcher, which is in the settings. Learn more here: https://support.google.com/mail/bin/answer.py?hl=en&answer=21288
      With a Business account, you can also use the Migration for Microsoft Exchange tool that Google provides, which despite its name, also works with old Gmail accounts. It may be a bit overkill for you, but would provide the closest thing to what the script I used in the post does. Check it out here: https://support.google.com/a/bin/answer.py?hl=en&answer=172212 You would need access to Windows to do it, but if you had Windows on your MBP or on a friend’s computer, that would work.
      Hope that gives you some good pointers.

      Reply
      1. Mobs

        Thnx Joshua, currently trying the Google Email Uploader – should take approx. 25 hrs so will keep you posted.

        Mobs

        Reply
  3. Pingback:

  4. Pingback:

  5. Pingback:

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>