When I was a kid, the state of the art for home movies was 8mm film. My parents had a movie camera that used a film cartridge that contained a 16mm film strip. They would insert the cartridge one way and record a few minutes of film, and then flip the cartridge over and record another few minutes. To develop the film, the processing company would open the film cartridge and split the long reel of 16mm wide film into two reels of 8mm wide film. They only took 25 reels of film over a span of 12 years. Back in 2012, I found a place that would convert the movies to DVD, and I kept an MP4 version on my computer, and I gave copies to everyone in my family. It was pretty awesome.<\/p>\n\n\n\n
By the time my kids came along, the world had moved to videotape, and we were just moving from analog to digital. We went top-of-the-line with a Sony MiniDV (NTSC) recorder. The NTSC DV video files record 720×480 pixels at 30fps\u2026 far from the 1080p videos that you\u2019d shoot on your iPhone today, but pretty hot for 2001. Like my parents, we ended up recording 20 MiniDV tapes over a period of 12 years, although our tapes were 60 minutes each instead of about 5.<\/p>\n\n\n\n
This is the story of converting those tapes to a modern video format.<\/p>\n\n\n\n
My 20 tapes sat on a shelf for several years, and then in 2012 I finally had the tools needed to copy the raw digital DV files to a computer hard disk. I used a tool called \u201cdvgrab\u201d on a laptop that had a FireWire connector. One side effect of dvgrab was that it saved every scene in a separate timestamped file. This turned out to be quite fortunate. When I was done, I had 250GB of raw DV files on an external hard drive. Unfortunately, I did not have enough disk space to do any processing on these files, so this external USB hard disk sat on a shelf for five years.<\/p>\n\n\n\n
In 2017 I re-discovered this USB hard disk, and I decided to finish the job. I wanted to encode them in MP4 format, and I wanted them to be separated into \u201cepisodes\u201d (such as \u201c2003 beach\u201d, \u201c4th birthday\u201d, \u201czoo\u201d and so on).<\/p>\n\n\n\n
The first step was to divide the 5000+ separate scene files into folders for episodes. I started with a script that looked at the timestamps in the filenames, and it moved them into folders based on that. This got me 90% of the way there. Here\u2019s the script. It used a single variable for the \u201cgap\u201d in time that would mean a break to the next episode.<\/p>\n\n\n\n
#!\/bin\/bash\n\nGAP=$((60*90))\n\nprevious_timestamp=0\nfor x in $(ls -1 ALL\/dvgrab-20*.dv | sort) ; do\n oldfolder=$(dirname $x)\n file=$(basename $x)\n # dvgrab-2011.02.12_19-04-58.dv\n year=${file:7:4}\n mon=${file:12:2}\n day=${file:15:2}\n hour=${file:18:2}\n min=${file:21:2}\n sec=${file:24:2}\n timestamp=$(date +%s -d \"$year-$mon-$day $hour:$min:$sec\")\n if [[ $(( $timestamp - $previous_timestamp )) -gt $GAP ]] ; then\n newfolder=$(date \"+%Y-%m-%d_%H-%M-%S\" -d \"@$timestamp\")\n mkdir $newfolder\n fi\n mv -v \"$x\" \"$newfolder\/$file\"\n previous_timestamp=$timestamp\ndone<\/pre>\n\n\n\nAfter I had the scenes grouped into episodes, I did an initial encoding of the entire thing. This ran overnight, but it could go unattended. The basic process was to concatenate the DV files and the use \u201cffmpeg\u201d to encode the episode into an MP4 file. I used the following script to do this in a loop.<\/p>\n\n\n\n
I ran this script many times, and it over time I tweaked the ffmpeg options to get a better output. This is the final cut.<\/p>\n\n\n\n
#!\/bin\/bash\n\nTMP=\"\/tmp\/encode\"\nsrcdir=\"\/media\/alan\/sandisk248GB\/MINIDV\"\ndestdir=\"\/home\/alan\/media\/videos\/minidv\"\nwildcard=\"20*\" # directories starting with a date from 2001 onwards\n\nffopts=\"\"\n\n# FILTERS\nffopts=\"$ffopts -vf yadif\" # de-interlacing\n\n# VIDEO ENCODING OPTIONS\nffopts=\"$ffopts -vcodec libx264\"\nffopts=\"$ffopts -preset medium\" # balance encoding speed vs compression ratio\nffopts=\"$ffopts -profile:v main -level 3.0 \" # compatibility, see https:\/\/trac.ffmpeg.org\/wiki\/Encode\/H.264\nffopts=\"$ffopts -pix_fmt yuv420p\" # pixel format of MiniDV is yuv411, x264 supports yuv420\nffopts=\"$ffopts -crf 23\" # The constant quality setting. Higher value = less quality, smaller file. Lower = better quality, bigger file. Sane values are [18 - 24]\nffopts=\"$ffopts -x264-params ref=4\"\n\n# AUDIO ENCODING OPTIONS\nffopts=\"$ffopts -acodec aac\"\nffopts=\"$ffopts -ac 2 -ar 24000 -ab 80k\" # 2 channels, 24k sample rate, 80k bitrate\n\n# GENERIC OPTIONS\nffopts=\"$ffopts -movflags faststart\" # Run a second pass moving the index (moov atom) to the beginning of the file.\n\nfor folder in $(cd $srcdir ; ls -1d $wildcard) ; do\n echo ; echo ; echo ; echo ; date ; echo $folder ; echo\n # do not overwrite existing files\n if [[ ! -f $destdir\/$folder.mp4 ]] ; then\n mkdir $TMP 2> \/dev\/null\n cat $srcdir\/$folder\/*.dv >> $TMP\/$folder.dv\n ffmpeg -i $TMP\/$folder.dv $ffopts $destdir\/$folder.mp4\n rm -frv $TMP\n else\n ls -l $destdir\/$folder.mp4\n fi\ndone<\/pre>\n\n\n\nThe next step was the most time-consuming (but fun) part. I wanted to curate all of the \u201cepisodes\u201d to make sure that they each contained a single subject in its entirety. I found a few variations:<\/p>\n\n\n\n
To do this curation step, I loaded up \u201cVLC\u201d video player with a playlist of all of the episodes, and I simply watched them at 4x speed. I\u2019d skip through predictable bits, and pay very close attention to the beginning and end of each episode. When I found something wonky, like a fade-out in its own separate directory, or a fade-out at the beginning of the next episode, I would find that DV file in the original directories and move it to the proper one.<\/p>\n\n\n\n
When I was done, I simply deleted the MP4 files ran the encoding script again.<\/p>\n\n\n\n
I noticed that the files would not play on my iPhone, and so I spent some time tweaking the ffmpeg options and re-encoding a few files (I limited it by changing the \u201cwildcard\u201d variable). Once I found the right options, I changed the wildcard back, deleted the MP4 files, and re-ran the encoder over all of the files again.<\/p>\n\n\n\n
When it was all over, I ended up with 349 \u201cepisode\u201d files in MP4 format, taking up 9.2 GB of disk space (much less than the 250 GB of the original DV files).<\/p>\n","protected":false},"excerpt":{"rendered":"
When I was a kid, the state of the art for home movies was 8mm film. My parents had a movie camera that used a film cartridge that contained a 16mm film strip. They would insert the cartridge one way and record a few minutes of film, and then flip the cartridge over and record […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2,5],"tags":[],"class_list":["post-1060","post","type-post","status-publish","format-standard","hentry","category-family","category-geek","count-0","even alt","author-alan","last"],"_links":{"self":[{"href":"https:\/\/alanporter.com\/blog\/wp-json\/wp\/v2\/posts\/1060","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alanporter.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/alanporter.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/alanporter.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/alanporter.com\/blog\/wp-json\/wp\/v2\/comments?post=1060"}],"version-history":[{"count":0,"href":"https:\/\/alanporter.com\/blog\/wp-json\/wp\/v2\/posts\/1060\/revisions"}],"wp:attachment":[{"href":"https:\/\/alanporter.com\/blog\/wp-json\/wp\/v2\/media?parent=1060"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alanporter.com\/blog\/wp-json\/wp\/v2\/categories?post=1060"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/alanporter.com\/blog\/wp-json\/wp\/v2\/tags?post=1060"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}