{"id":159,"date":"2011-05-06T11:18:02","date_gmt":"2011-05-06T17:18:02","guid":{"rendered":"https:\/\/www.joshualyman.com\/?p=159"},"modified":"2011-05-07T14:01:16","modified_gmt":"2011-05-07T20:01:16","slug":"migrating-from-private-imap-to-google-apps-email","status":"publish","type":"post","link":"https:\/\/www.joshualyman.com\/2011\/05\/migrating-from-private-imap-to-google-apps-email\/","title":{"rendered":"Migrating from private IMAP to Google Apps Email"},"content":{"rendered":"

I have recently been moving several clients off of a basic email server to a Google Apps account (Free or Business, depending on the client), and several have had up to a couple years worth of email. I’ve tried several different techniques, but recently found some great command line kung fu that made the process extremely easy and much more accurate than the past attempts.<\/p>\n

There are several different methods available for migrating email from various servers to Google Apps, including POP transfer<\/a> from within each account’s settings, dragging and dropping via IMAP<\/a> with a mail client like Thunderbird, or using the official email migration API from Google<\/a>. However, each of these methods has its downsides: POP is slow and kludgy, dragging and dropping only works for low volume accounts, and the email migration API requires code be written and using a for-pay Apps account. What’s a geek to do? Turn to free and open source solutions!<\/p>\n

A little bit of searching turns up that a command line tool known as imapsync does a fantastic job of, well, syncing IMAP accounts. I was tipped off by a blog post on Marius Ducea’s blog<\/a> (lots of good stuff on that site by the way), which contains a basic introduction to imapsync and how to use it. The author of imapsync, Gilles Lamiral, continues to develop the software and is taking donations for different features, and also charges for the source and for support (visit Gilles’ site here<\/a>). However, thanks to the licensing of imapsync<\/a> (NSWF), it is also available for free from other hosted repositories. I was using CentOS, and got my copy from the following:<\/p>\n

https:\/\/fedorahosted.org\/imapsync\/<\/a><\/p>\n

After getting a copy of the Perl script, I ran perl -c imapsync to check dependencies as Ducea instructs, and found I was missing the Mail::IMAPClient module. That was a quick fix with<\/p>\n

sudo cpan Mail::IMAPClient<\/pre>\n
Voila! The script ran just fine at that point. Ducea provides some good templates for running a sync between two servers, and it worked almost 100% right off the bat. However, I was getting issues with sent messages and deleted messages ending up in Google Apps with labels such as “[IMAP]\/Sent Messages,” which obviously was not what I wanted. (Note: most such errors can be avoided by appending --dry --justfolders<\/code> to the command and checking the output to see what imapsync discovers as folders\/labels that already exist, and those which it plans on creating.)<\/p>\n
A little more searching turned up Mark’s Pages of Stuff’s post about moving to Google Apps<\/a>, and with a little modification there as well, I was able to get messages moved properly from the IMAP’s sent folder into Google Apps’s sent folder, trash, etc. On Mark’s posting, he uses folder labels like “[Google Mail] Bin,” but he notes that Google changes the name of labels based on locale. He just happens to be in England. However, running imapsync with --dry --justfolders<\/code> will show you all the folders that exist on both the origin and target servers, so make sure that when you are planning your own migration that you inspect these values. For me, based in the US, the folders used the [Gmail] prefix and Trash instead of Bin.<\/p>\n
Finally, after several test runs, I got the command line down to exactly what I needed, and I was off to the races. I set up a small bash script that would iterate over the several mailboxes I needed to migrate, pressed enter, and got about 30 messages in when the process died saying “Out of memory.” Out of memory, really? I was confused for a bit until I checked the message that it was dying on each time, and found that it had a rather massive attachment on it. The origin server, a small, shared hosting situation, must have capped the memory below the size of the attachment.<\/p>\n
Luckily, imapsync has a --maxsize<\/code> option, which will allow it to skip messages that total over a certain level of bytes. A little bit of experimenting on the server showed that I could only work with messages of about up to 2 MB in size, which was rather small, but doable in this situation. I added the –maxsize switch to the command, and ran through all of migrations again.<\/p>\n
The final command that I used to migrate each account was as follows:<\/p>\n
imapsync --syncinternaldates --host1 <ORIGIN MAIL SERVER> --port1 993 --ssl1 --user1 <ORIGIN USERNAME> --password1 <ORIGIN USER PASSWORD> \r\n--host2 imap.gmail.com --port2 993 --ssl2 --user2 <TARGET EMAIL ADDRESS> --password2 <TARGET USER PASSWORD> \r\n--useheader 'Message-Id' --skipsize --noauthmd5 --reconnectretry1 1 --reconnectretry2 1 --maxsize 2194304 \\\r\n   --folder \"INBOX.Sent Messages\" --prefix2 '[Gmail]\/' --regextrans2 's\/Sent Messages$\/Sent Mail\/g' \\\r\n   --folder \"INBOX.Deleted Messages\" --prefix2 '[Gmail]\/' --regextrans2 's\/Deleted Messages$\/Trash\/' \\\r\n   --folder \"INBOX.Drafts\" --prefix2 '[Gmail]\/' --regextrans2 's\/INBOX\\.Drafts$\/Drafts\/'<\/pre>\n<\/p>\n
(The three folder lines were unique to the IMAP configuration on the origin server, and will likely change for you. Also don’t forget --dry --justfolders<\/code> while testing!)<\/em><\/p>\n
After running the migrations on the underpowered origin server, I switched over to a VPS and ran the commands again, but this time without the –maxsize switch. No memory issues were encountered on the robust VPS, and I also was able to catch the last few newer emails that had been missed since the first migration.<\/p>\n
Finally, after each account was migrated, I placed a forward on the email account on the origin server to redirect all new mail to <account>@<domain>.test-google-a.com<\/code>, which acts as a secondary address for all Google Apps accounts before you switch over MX records. This allowed me to to a cutover from the old server to the new server without losing any new messages, and worked flawlessly.<\/p>\n
All in all, the process took about 5 hours to transfer all of the data, and about an hour of research and playing around. Much<\/em> better than dragging and dropping, or writing an entire mini-application to utilize the email migration API. Thanks to some great resources like imapsync and several helpful blogs, this was a much more pain-free experience than in the past!<\/p>\n","protected":false},"excerpt":{"rendered":"
Recently I have been moving several clients off of private IMAP servers onto Google Apps, and played with a lot of different migration techniques including POP sucking, email program drag and drop, and the command line. Using the Perl script imapsync, I was able to transfer several years worth of emails to Google Apps in just a few hours and with very little manual intervention. Great solution, read on for code and links.<\/p>\n
Continue readingMigrating from private IMAP to Google Apps Email<\/span><\/a><\/div>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[1,10],"tags":[84,74,73,81,80,22,83,82],"jetpack_publicize_connections":[],"yoast_head":"\nMigrating from private IMAP to Google Apps Email - Joshua Lyman.com<\/title>\n<meta name=\"description\" content=\"A great solution for moving email from a generic IMAP server to Google Apps using a free command line tool known as imapsync saved me tons of time!\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.joshualyman.com\/2011\/05\/migrating-from-private-imap-to-google-apps-email\/\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:title\" content=\"Migrating from private IMAP to Google Apps Email - Joshua Lyman.com\" \/>\n<meta name=\"twitter:description\" content=\"A great solution for moving email from a generic IMAP server to Google Apps using a free command line tool known as imapsync saved me tons of time!\" \/>\n<meta name=\"twitter:creator\" content=\"@jlyman\" \/>\n<meta name=\"twitter:site\" content=\"@jlyman\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Joshua\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.joshualyman.com\/2011\/05\/migrating-from-private-imap-to-google-apps-email\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.joshualyman.com\/2011\/05\/migrating-from-private-imap-to-google-apps-email\/\"},\"author\":{\"name\":\"Joshua\",\"@id\":\"https:\/\/www.joshualyman.com\/#\/schema\/person\/4679b2b5b142a2c904bb46f97a04bd9e\"},\"headline\":\"Migrating from private IMAP to Google Apps Email\",\"datePublished\":\"2011-05-06T17:18:02+00:00\",\"dateModified\":\"2011-05-07T20:01:16+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.joshualyman.com\/2011\/05\/migrating-from-private-imap-to-google-apps-email\/\"},\"wordCount\":936,\"commentCount\":11,\"publisher\":{\"@id\":\"https:\/\/www.joshualyman.com\/#\/schema\/person\/4679b2b5b142a2c904bb46f97a04bd9e\"},\"keywords\":[\"clients\",\"email\",\"gmail\",\"google apps\",\"imap\",\"linkedin\",\"migration\",\"servers\"],\"articleSection\":[\"General\",\"Programming\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.joshualyman.com\/2011\/05\/migrating-from-private-imap-to-google-apps-email\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.joshualyman.com\/2011\/05\/migrating-from-private-imap-to-google-apps-email\/\",\"url\":\"https:\/\/www.joshualyman.com\/2011\/05\/migrating-from-private-imap-to-google-apps-email\/\",\"name\":\"Migrating from private IMAP to Google Apps Email - Joshua Lyman.com\",\"isPartOf\":{\"@id\":\"https:\/\/www.joshualyman.com\/#website\"},\"datePublished\":\"2011-05-06T17:18:02+00:00\",\"dateModified\":\"2011-05-07T20:01:16+00:00\",\"description\":\"A great solution for moving email from a generic IMAP server to Google Apps using a free command line tool known as imapsync saved me tons of time!\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.joshualyman.com\/2011\/05\/migrating-from-private-imap-to-google-apps-email\/\"]}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.joshualyman.com\/#website\",\"url\":\"https:\/\/www.joshualyman.com\/\",\"name\":\"Joshua Lyman.com\",\"description\":\"Technologist, founder, and teacher\",\"publisher\":{\"@id\":\"https:\/\/www.joshualyman.com\/#\/schema\/person\/4679b2b5b142a2c904bb46f97a04bd9e\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.joshualyman.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/www.joshualyman.com\/#\/schema\/person\/4679b2b5b142a2c904bb46f97a04bd9e\",\"name\":\"Joshua\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.joshualyman.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/i1.wp.com\/www.joshualyman.com\/wp-content\/uploads\/2020\/12\/Headshot-2020-Square-Small.jpg?fit=1000%2C1000\",\"contentUrl\":\"https:\/\/i1.wp.com\/www.joshualyman.com\/wp-content\/uploads\/2020\/12\/Headshot-2020-Square-Small.jpg?fit=1000%2C1000\",\"width\":1000,\"height\":1000,\"caption\":\"Joshua\"},\"logo\":{\"@id\":\"https:\/\/www.joshualyman.com\/#\/schema\/person\/image\/\"},\"sameAs\":[\"https:\/\/www.joshualyman.com\",\"https:\/\/twitter.com\/jlyman\"],\"url\":\"https:\/\/www.joshualyman.com\/author\/jladmin\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Migrating from private IMAP to Google Apps Email - Joshua Lyman.com","description":"A great solution for moving email from a generic IMAP server to Google Apps using a free command line tool known as imapsync saved me tons of time!","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.joshualyman.com\/2011\/05\/migrating-from-private-imap-to-google-apps-email\/","twitter_card":"summary_large_image","twitter_title":"Migrating from private IMAP to Google Apps Email - Joshua Lyman.com","twitter_description":"A great solution for moving email from a generic IMAP server to Google Apps using a free command line tool known as imapsync saved me tons of time!","twitter_creator":"@jlyman","twitter_site":"@jlyman","twitter_misc":{"Written by":"Joshua","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.joshualyman.com\/2011\/05\/migrating-from-private-imap-to-google-apps-email\/#article","isPartOf":{"@id":"https:\/\/www.joshualyman.com\/2011\/05\/migrating-from-private-imap-to-google-apps-email\/"},"author":{"name":"Joshua","@id":"https:\/\/www.joshualyman.com\/#\/schema\/person\/4679b2b5b142a2c904bb46f97a04bd9e"},"headline":"Migrating from private IMAP to Google Apps Email","datePublished":"2011-05-06T17:18:02+00:00","dateModified":"2011-05-07T20:01:16+00:00","mainEntityOfPage":{"@id":"https:\/\/www.joshualyman.com\/2011\/05\/migrating-from-private-imap-to-google-apps-email\/"},"wordCount":936,"commentCount":11,"publisher":{"@id":"https:\/\/www.joshualyman.com\/#\/schema\/person\/4679b2b5b142a2c904bb46f97a04bd9e"},"keywords":["clients","email","gmail","google apps","imap","linkedin","migration","servers"],"articleSection":["General","Programming"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.joshualyman.com\/2011\/05\/migrating-from-private-imap-to-google-apps-email\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.joshualyman.com\/2011\/05\/migrating-from-private-imap-to-google-apps-email\/","url":"https:\/\/www.joshualyman.com\/2011\/05\/migrating-from-private-imap-to-google-apps-email\/","name":"Migrating from private IMAP to Google Apps Email - Joshua Lyman.com","isPartOf":{"@id":"https:\/\/www.joshualyman.com\/#website"},"datePublished":"2011-05-06T17:18:02+00:00","dateModified":"2011-05-07T20:01:16+00:00","description":"A great solution for moving email from a generic IMAP server to Google Apps using a free command line tool known as imapsync saved me tons of time!","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.joshualyman.com\/2011\/05\/migrating-from-private-imap-to-google-apps-email\/"]}]},{"@type":"WebSite","@id":"https:\/\/www.joshualyman.com\/#website","url":"https:\/\/www.joshualyman.com\/","name":"Joshua Lyman.com","description":"Technologist, founder, and teacher","publisher":{"@id":"https:\/\/www.joshualyman.com\/#\/schema\/person\/4679b2b5b142a2c904bb46f97a04bd9e"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.joshualyman.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/www.joshualyman.com\/#\/schema\/person\/4679b2b5b142a2c904bb46f97a04bd9e","name":"Joshua","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.joshualyman.com\/#\/schema\/person\/image\/","url":"https:\/\/i1.wp.com\/www.joshualyman.com\/wp-content\/uploads\/2020\/12\/Headshot-2020-Square-Small.jpg?fit=1000%2C1000","contentUrl":"https:\/\/i1.wp.com\/www.joshualyman.com\/wp-content\/uploads\/2020\/12\/Headshot-2020-Square-Small.jpg?fit=1000%2C1000","width":1000,"height":1000,"caption":"Joshua"},"logo":{"@id":"https:\/\/www.joshualyman.com\/#\/schema\/person\/image\/"},"sameAs":["https:\/\/www.joshualyman.com","https:\/\/twitter.com\/jlyman"],"url":"https:\/\/www.joshualyman.com\/author\/jladmin\/"}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7V8cO-2z","_links":{"self":[{"href":"https:\/\/www.joshualyman.com\/wp-json\/wp\/v2\/posts\/159"}],"collection":[{"href":"https:\/\/www.joshualyman.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.joshualyman.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.joshualyman.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.joshualyman.com\/wp-json\/wp\/v2\/comments?post=159"}],"version-history":[{"count":16,"href":"https:\/\/www.joshualyman.com\/wp-json\/wp\/v2\/posts\/159\/revisions"}],"predecessor-version":[{"id":193,"href":"https:\/\/www.joshualyman.com\/wp-json\/wp\/v2\/posts\/159\/revisions\/193"}],"wp:attachment":[{"href":"https:\/\/www.joshualyman.com\/wp-json\/wp\/v2\/media?parent=159"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.joshualyman.com\/wp-json\/wp\/v2\/categories?post=159"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.joshualyman.com\/wp-json\/wp\/v2\/tags?post=159"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}