Category Archives: Programming

Google went and took away my “Reader” link in the nav bar again today. Well, what happens when Google does that? I make an extension to get it back! I’ve updated my amazingly complex (not) “Back to the Reader” extension, available in the Chrome App Store, to account for the new navigation bar and changes.

It works again, and will stay activated in my browser until Google decides to put it back. For more information, see my page about it.

Share and Enjoy

  • Twitter
  • Google Plus
  • LinkedIn
  • Email
  • RSS
  • HackerNews
  • Instapaper
  • StumbleUpon
  • Facebook

A sad thing happened to my mother-in-law the other day. Somehow, one of her 1 TB hard drives that stores mountains of images (she is a professional photographer) got reformatted to a nice, fresh state, but certainly not what was wanted. She has a client that needs the images in just a couple of days, and so the race was on to see what I could do (remotely, using LogMeIn no less) to recover the data.

I turned to Photorec, and absolutely fantastic command line program that does a hard core scavenge of any disk, looking for known file headers, and tries to reconstruct a file based on its findings. I felt that, since the drive had simply been reformatted and not zeroed, with only one file being placed on the drive since the formatting, that the chances of finding the files was pretty good. Using another external hard drive as storage for recovered files, 15 hours later it had recovered 51,407 .JPGs, .NEFs, and .PSDs, woohoo!

Now one unfortunate byproduct of Photorec recovery is that, while much of the file’s metadata is still in place (modified/accessed timestamps, EXIF data, etc.), it didn’t recover filenames or folder structure. So I had over 51,000 images with random filenames in 130 different “recup” folders, mostly in chunks of the same creation date, but not entirely.

To solve this, I wrote a small little bash script to sort files into folders by modified date. I’m not super skilled at bash scripting, so it took me longer than it probably should of and is probably not very elegant, but it worked quite well in my case! I pointed the script at the recovered directories, ran it, and 3 minutes later it had created over 1300 folders based on the created/modified dates of the files, moved the files into the folders, and bam! I was pretty impressed with myself. :)

If anyone is looking for a similar script (couldn’t find one after an initial Google search last night) then feel free to take this one and modify to suit your needs. Obviously I can’t guarantee anything with it, but it shouldn’t eat your files if you set up the directories correctly. If you are nervous, change the “mv” command on line 27 to “cp” and you should be safer (although the script will of necessity run much slower). There are certainly some things that could be improved about the script (not hardcoding the 300 of ## number, not cd’ing into the source directory, etc.) but it worked in a pinch and didn’t need to be elegant. Hope it helps!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
#!/bin/bash
# Sort files into folders based on their creation date.
#
 
# Indicate a path to the source folder of data to be sorted, and a destination folder where you want the sorted folders to be.
# Note: this will work better if the sorted directory is NOT inside the source directory, otherwise recursion might occur and the world might blow up.
SOURCE_DIR=~/Desktop/testsrc/
SORTED_DIR=~/Desktop/sorted/
sorted_count=0
 
# go into the first folder of the source folder
cd "$SOURCE_DIR";
 
# for each item in this folder
find . -type f -print | while read file; do 
 
	# Get the creation date of the item and store it in a variable
	file_date=`stat -f '%Sm' -t "%F" "$file"`;
 
	# Does a folder with this date already exist?
	if [ ! -d "$SORTED_DIR/$file_date" ]; then
		# If no, create the folder
		mkdir -p $SORTED_DIR/$file_date
	fi
 
	# Now move the file to the folder
	mv "$file" "$SORTED_DIR/$file_date/"
 
        # Provide some feedback to the user on our progress
	let "sorted_count += 1"
	if [[ "$sorted_count % 100" -eq "0" ]]; then
		echo "Sorted $sorted_count of 51,407" # Obviously the of count is hard coded 
	fi
 
done
 
echo "Finished."

Share and Enjoy

  • Twitter
  • Google Plus
  • LinkedIn
  • Email
  • RSS
  • HackerNews
  • Instapaper
  • StumbleUpon
  • Facebook

I have recently been moving several clients off of a basic email server to a Google Apps account (Free or Business, depending on the client), and several have had up to a couple years worth of email. I’ve tried several different techniques, but recently found some great command line kung fu that made the process extremely easy and much more accurate than the past attempts.

There are several different methods available for migrating email from various servers to Google Apps, including POP transfer from within each account’s settings, dragging and dropping via IMAP with a mail client like Thunderbird, or using the official email migration API from Google. However, each of these methods has its downsides: POP is slow and kludgy, dragging and dropping only works for low volume accounts, and the email migration API requires code be written and using a for-pay Apps account. What’s a geek to do? Turn to free and open source solutions!

A little bit of searching turns up that a command line tool known as imapsync does a fantastic job of, well, syncing IMAP accounts. I was tipped off by a blog post on Marius Ducea’s blog (lots of good stuff on that site by the way), which contains a basic introduction to imapsync and how to use it. The author of imapsync, Gilles Lamiral, continues to develop the software and is taking donations for different features, and also charges for the source and for support (visit Gilles’ site here). However, thanks to the licensing of imapsync (NSWF), it is also available for free from other hosted repositories. I was using CentOS, and got my copy from the following:

https://fedorahosted.org/imapsync/

After getting a copy of the Perl script, I ran perl -c imapsync to check dependencies as Ducea instructs, and found I was missing the Mail::IMAPClient module. That was a quick fix with

sudo cpan Mail::IMAPClient

Voila! The script ran just fine at that point. Ducea provides some good templates for running a sync between two servers, and it worked almost 100% right off the bat. However, I was getting issues with sent messages and deleted messages ending up in Google Apps with labels such as “[IMAP]/Sent Messages,” which obviously was not what I wanted. (Note: most such errors can be avoided by appending --dry --justfolders to the command and checking the output to see what imapsync discovers as folders/labels that already exist, and those which it plans on creating.)

A little more searching turned up Mark’s Pages of Stuff’s post about moving to Google Apps, and with a little modification there as well, I was able to get messages moved properly from the IMAP’s sent folder into Google Apps’s sent folder, trash, etc. On Mark’s posting, he uses folder labels like “[Google Mail] Bin,” but he notes that Google changes the name of labels based on locale. He just happens to be in England. However, running imapsync with --dry --justfolders will show you all the folders that exist on both the origin and target servers, so make sure that when you are planning your own migration that you inspect these values. For me, based in the US, the folders used the [Gmail] prefix and Trash instead of Bin.

Finally, after several test runs, I got the command line down to exactly what I needed, and I was off to the races. I set up a small bash script that would iterate over the several mailboxes I needed to migrate, pressed enter, and got about 30 messages in when the process died saying “Out of memory.” Out of memory, really? I was confused for a bit until I checked the message that it was dying on each time, and found that it had a rather massive attachment on it. The origin server, a small, shared hosting situation, must have capped the memory below the size of the attachment.

Luckily, imapsync has a --maxsize option, which will allow it to skip messages that total over a certain level of bytes. A little bit of experimenting on the server showed that I could only work with messages of about up to 2 MB in size, which was rather small, but doable in this situation. I added the –maxsize switch to the command, and ran through all of migrations again.

The final command that I used to migrate each account was as follows:

imapsync --syncinternaldates --host1 <ORIGIN MAIL SERVER> --port1 993 --ssl1 --user1 <ORIGIN USERNAME> --password1 <ORIGIN USER PASSWORD> 
--host2 imap.gmail.com --port2 993 --ssl2 --user2 <TARGET EMAIL ADDRESS> --password2 <TARGET USER PASSWORD> 
--useheader 'Message-Id' --skipsize --noauthmd5 --reconnectretry1 1 --reconnectretry2 1 --maxsize 2194304 \
   --folder "INBOX.Sent Messages" --prefix2 '[Gmail]/' --regextrans2 's/Sent Messages$/Sent Mail/g' \
   --folder "INBOX.Deleted Messages" --prefix2 '[Gmail]/' --regextrans2 's/Deleted Messages$/Trash/' \
   --folder "INBOX.Drafts" --prefix2 '[Gmail]/' --regextrans2 's/INBOX\.Drafts$/Drafts/'

(The three folder lines were unique to the IMAP configuration on the origin server, and will likely change for you. Also don’t forget --dry --justfolders while testing!)

After running the migrations on the underpowered origin server, I switched over to a VPS and ran the commands again, but this time without the –maxsize switch. No memory issues were encountered on the robust VPS, and I also was able to catch the last few newer emails that had been missed since the first migration.

Finally, after each account was migrated, I placed a forward on the email account on the origin server to redirect all new mail to <account>@<domain>.test-google-a.com, which acts as a secondary address for all Google Apps accounts before you switch over MX records. This allowed me to to a cutover from the old server to the new server without losing any new messages, and worked flawlessly.

All in all, the process took about 5 hours to transfer all of the data, and about an hour of research and playing around. Much better than dragging and dropping, or writing an entire mini-application to utilize the email migration API. Thanks to some great resources like imapsync and several helpful blogs, this was a much more pain-free experience than in the past!

Share and Enjoy

  • Twitter
  • Google Plus
  • LinkedIn
  • Email
  • RSS
  • HackerNews
  • Instapaper
  • StumbleUpon
  • Facebook

My wife and I, like any fun newlywed couple (okay, maybe we’re getting a little old for the newlywed stage now), have a blog where we like to talk about the trips we go on, fun finds, time with family, etc. The time came the other day to make the blog private to allow access only to friends and family, instead of shouting our family matters to the entire Internet. It is run with WordPress, as are most of my personal and client sites, because of the fantastic power and flexibility that the platform provides. After searching for a few hours, I came up with what I believe is one a very good combo of plugins and settings to make a WordPress blog private.

Continue reading

Share and Enjoy

  • Twitter
  • Google Plus
  • LinkedIn
  • Email
  • RSS
  • HackerNews
  • Instapaper
  • StumbleUpon
  • Facebook

Update: The Reader link is back! Supposedly a mistake, but one that took several days to fix… All cheer for not needing an extra extension now!

Out of nowhere today, I noticed that Google decided that I didn’t need a link to Google Reader in the top left link bar that is present on all Google App pages, and instead thought that I should get in touch with my artistic side by replacing it with Photos. As much as I do love photography and art, I’ve never used the Photos (Picasa Web) service, and don’t plan to. But I’m definitely addicted to Google Reader! Why, Google, why?

So what’s a man to do? “Well heck,” I figured, “I’m smart, I’m a pseudo-developer, and I can just write my own extension to insert it back in! Take that Google!” Never mind the fact that I could access it in one more click by using the “more” link or just typing it in directly. No, I want it there, a regular link! So I created the “Back to the Reader!” Google Chrome extension, and voila, problem solved!

So you're tellin' me Doc, that Google Reader is back in the bar??

The extension is available to anyone, and the source is on GitHub. Now granted I did this in the most down and dirty quick way possible, and so I know it’s not super optimized and may probably break in some test cases I haven’t thought of. But I just wanted to get my Reader link back, and if it can benefit others, then fantastic. Please test it out and let me know what does and does not work for you. Long live Google Reader!

Download the Back to the Reader! extension

Share and Enjoy

  • Twitter
  • Google Plus
  • LinkedIn
  • Email
  • RSS
  • HackerNews
  • Instapaper
  • StumbleUpon
  • Facebook