random thoughts

I like to keep albums of recent photos on my iPhone, so I can show them to friends and family. Not only do I keep the photos that I took on the iPhone itself, but I also keep the photos I took using other cameras. Here, I will describe the method and tools I use to keep several months worth of photos conveniently in my pocket.

At first, this seems like a very simple problem, but there are a few subtle quirks, and I have worked around them using a script and some open source tools. In the end, I will have:

• photos from multiple cameras
• selected folders, but not necessarily everything
• photos are optimized for phone size, not wasting space
• photos are oriented correctly, not sideways or upside-down
• photos from multiple cameras appear in chronological order
• process is mostly automated

The end result looks like this:

iPhone “Photos” app showing albums

How my photos are organized

When I worked at Ericsson’s research lab in Singapore, they had done a study on how people store their digital photos and how they find a particular one in the bunch. They found that people tend to associate events with nearby dates, so they might say “Let’s see… Bill broke his leg skiing right before Janet’s wedding, so that must have been Winter of 2009”.

I definitely fall into that category, and so my pictures are stored in folders by date. It looks like this:

pictures     - top level
+ bulk       - raw unprocessed stuff, too big to browse
+ import     - a working area
+ y          - photo albums arranged by year/month/event
  + 2011
  + 2012
    + 2012-09
    + 2012-10
    + 2012-10_florida
    + 2012-10_halloween
    + 2012-11
    + 2012-11-12_ducks
    + 2012-11-18_sydney_birthday
    + 2012-11_NSL9
    + 2012-11_thanksgiving
  + 2013

I also want to note that some of my friends take a completely different approach… they pick a single tool and simply dump the photos into the tool. They don’t care to organize them at all and just rely on the tool. That’s certainly OK… I would guess these are the same people who don’t balance their checkbooks. If that describes you, then I suggest you stop reading here, because none of this will apply to you.

I have used the date-folder method since I got my first digital camera in 1998. It has migrated from Windows to Linux to MacOS. My data far outlasts any one tool.

So one of my assumptions is that my “pictures” directory is sacred. I do not want any tool to write anything to that directory (I am looking at you, iPhoto & iTunes).

Selecting folders (albums) to copy to the iPhone

So I set up two more directories.

• $HOME/iphone/pictures – where I say what photos to copy
• $HOME/itunes/pictures – a directory that iTunes can sync with

In the first one, $HOME/iphone/pictures, I set up a bunch of symlinks to point to the directories that I would like to copy. I do this so I can easily select some folders but leave out others.

2011-11_happy_crabby       -> ../../pictures/y/2011/2011-11_happy_crabby
2012-09                    -> ../../pictures/y/2012/2012-09
2012-10                    -> ../../pictures/y/2012/2012-10
2012-10_florida            -> ../../pictures/y/2012/2012-10_florida
2012-10_halloween          -> ../../pictures/y/2012/2012-10_halloween
2012-11                    -> ../../pictures/y/2012/2012-11
*** note - no ducks here ***
2012-11-18_sydney_birthday -> ../../pictures/y/2012/2012-11-18_sydney_birthday
2012-11_NSL9               -> ../../pictures/y/2012/2012-11_NSL9
2012-11_thanksgiving       -> ../../pictures/y/2012/2012-11_thanksgiving

For example, let’s say I took 1000 boring photos of ducks in November. I could easily skip 2012-11-12_ducks by not creating a symlink. I can also specifically include a very old folder by leaving the symlink in this directory. I have done that here with my Happy Crabby pictures from 2011.

Copying and processing photos

I have written a script that will go through each of these directories and perform a few clean-up tasks as it copies the photos to the itunes staging area.

• It copies the photos to a temporary working directory.

  We should never alter the original photos… always work with a copy.

• It rotates the image based on the EXIF orientation tag.

  I’m gonna start slamming on Apple, so fanbois should cover their ears. I have found that Apple handles image rotation differently than everyone else, and it took a while to get a handle on it. I can take pictures on my iPhone and email them to someone and they’ll show up as upside-down or sideways. The images look OK on the iPhone or on a Mac, but everyone else (Linux, Windows) sees them rotated. The same thing happens if I upload them to “gallery”, an online photo gallery. Sometimes, they would show the correct dimensions but the image would still be rotated. That is, a portrait would be taller than wide, but the person’s face would be sideways and extremely elongated. The solution I found is to process them using “renrot” (see my blog post about renrot for detail).

• It renames the file using the date & time in the EXIF data.

  When we get back from vacations, I find that we have photos from several different cameras, and it just seemed weird for my wife’s Nikon images (named DSCNxxxx.JPG) to appear before my iPhone images (named IMG_xxxx.JPG) and then my Panasonic images (named Pxxxxxxx.JPG) last. If I rename all of the images to (YYYY)(MM)(DD)(hh)(mm)(ss).JPG, they will appear in chronological order, regardless of which camera was used.

• It resizes the image to be no bigger than 1280 pixels high or wide.

  There is no need to me to waste space on my iPhone by storing each image in their 12MP glory (actually, I typically shoot 3MP for normal vacation photos). So I resize the images using imagemagick.

• It moves the files into the iTunes directory in numerical (chronological order).

At one point, I had a problem with iTunes arranging my photos in the order that they happened to appear in the directory instead of sorted by name or anything else. If you do a “find” command on Linux or a Mac, you’ll see the files in the order that they appear in the directory file. That’s OK under the hood, because “ls” and Mac Finder and Nautilus are all smart enough to show the files in sorted order. But no, not iTunes. So I move the files from the temporary work area to the iTunes sync directory one at a time, in chronological order.

Using the script

When it’s time to sync my iPhone (less and less frequent these days, and primarily driven by when I want to update my photo albums), I simply look over the symlinks, run the script and then sync.

Running the script takes a few minutes, and syncing all of the pictures instead of just the ones that have changed takes a few minutes as well. But I am very happy with the consistent results.

I’m sure that some folks will claim that I am “doing it wrong”, that I should not have to do so many work-arounds. But the bottom line is that I encountered real-world problems from the interaction between various cameras and the quirks of iTunes, and this script smooths out all of the wrinkles. It is really no trouble to run the pre-syncing script before syncing, and it does solve the problems of images being mis-rotated, taking too much space on my phone, and appearing out of order. It also makes it trivial to choose which album folders are copied to the iPhone and which ones are not.

The script itself appears below.

The pre-syncing script

#!/bin/bash
shopt -s nocaseglob
shopt -s nullglob

src="$HOME/iphone"
dest="$HOME/itunes"
if [ ! -d "$dest" ] ; then mkdir -p "$dest" ; fi

echo "=== PHOTOS ==="

wildcard="*.[Jj][Pp][Gg] *.[Jj][Pp][Ee][Gg] *.[Pp][Nn][Gg]"
stage="/tmp/itunes.$$.stage"
mkdir -p $stage
rm -rf "$dest/pictures"

# go through each directory, and do some processing
( ls -1 $src/pictures ) | while read d ; do

   # skip non-directories
   [ ! -d "$src/pictures/$d/." ] && continue

   echo "processing photos in [$d]"
   indent="   $d >> "

   # copy source photos to staging area
   ( cd $src/pictures/$d ; cp $wildcard $stage/ )

   # rotate
   echo " - rotating photos"
   ( cd $stage ; renrot $wildcard ) 2>/dev/null | sed -e "s/^/$indent/g"
   rm $stage/*_orig 2>/dev/null

   # resize
   echo " - resize photos"
   # iphone 4S screen is 640x960 (2xVGA)
   ( cd $stage ; mogrify -verbose -geometry '1280x1280>' $wildcard ) 2>/dev/null | sed -e "s/^/$indent/g"

   # move to itunes area, reorder as you go
   echo " - re-ordering files"
   mkdir -p "$dest/pictures/$d"
   ls -1 $stage | while read f ; do
      # Mac bash v3 does not have ${f^^*}, so we have to use 'tr'
      upper=$(echo $f | tr /a-z/ /A-Z/)  # was bash internal ${f^^*}
      mv -v "$stage/$f" "$dest/pictures/$(basename $d)/$upper" | sed -e "s/^/$indent/g"
   done

   echo ""
done

# clean up some special cases
xargsR='-r'
[ "$OSTYPE" == "darwin12" ] && xargsR=''  # Mac xargs does not have -r
find $dest/pictures/ -type d -print0 | xargs -0 rmdir 2> /dev/null # empty directories
find $dest/pictures/ -name \*_orig -print0 | xargs -0 $xargsR rm -v
find $dest/pictures/ -name MEDI\*.JPG -print0 | xargs -0 $xargsR rm -v
echo ""