Q & A : What is "Safe" storage

To ask a question of The Photo Video Guy, send an email with your question to ross@thephotovideoguy.ca Rod writes to ask about storing images. "What's the best means of storage and archiving digital photos.  I read your article on thephotoguy.ca about memory cards.  I find the technical info helpful but I also read elsewhere that flash storage is not long-term reliable. "

Great question Rod, let's take a look at the subject.

There are potentially hundreds of opinions on how best to store and keep your digital photos, and videos.  In the old days of digital, owners would have to create their own image management system of directories, folders, naming conventions and the like, and hope that over time the thing was sustainable, manageable and the the owner could actually find something four years down the road.  It was not really viable then and is the worst option now.

The first rule of safe storage is that no single storage mechanism is "safe".  Storage professionals will tell you that no file is really safe until there are three copies of it.  The three copies include the master, the local backup and the offsite backup.

Physical Media

Before we get in to that architectural conversation, I want to address the comment you have heard that flash storage is not long-term reliable.  The real question is what constitutes "long-term".  In the early days of static column RAM integrity was measured in years without a power application to recharge the chips's ability to retain a bit instruction.  With today's flash memory cards such as high end Compact Flash or Secure Digital cards, reliability is superb.  There are numerous indicators where a card has spent six years underwater in the ocean and been readable.  No electronic storage medium is fully protected from data loss due to strong magnetics or degaussers, but reasonably maintained, the flash memory in storage cards will last a long time.  It's NOT suitable as a long term storage mechanism not because of reliability but because of the difficulty in cataloging.

Solid state drives use a different class of flash memory.  They are more reliable than media cards and as of yet, we have not seen any strong indicators of limited life.  They are extremely fast, robust and make an excellent storage medium but you pay a higher price for this sort of storage.

Classic spinning drives are the most common type of storage.  They are very cost-effective, come in enormous sizes and if you buy smart have superb integrity.  To a large extent you do get what you pay for.  Enterprise class spindles cost a LOT more than consumer grade drives.  They tend to be smaller in capacity, but are built to be used heavily.  An example would be Western Digital Velociraptor drives.  Fast, robust and pricey.

For home storage in desktop enclosures, Western Digital Caviar Black series are proven to be very reliable.  The newer Caviar Red drives vary their rotational speed but are built to be always on in Network Attached Storage.  As an archive target, Caviar Reds are good choices.  The less expensive Caviar Green drives are targeted at the "green" client.  I have lost two of five in two years and will never buy another one.  Similarly, I would never buy any drive from Seagate because of execrable reliability.  Hitachi was my go-to favourite drive but they were sold to Western Digital.  You can get these high quality drives in the external drives from G-Tech.

In laptop format drives, SSD is the way to go, albeit at a higher price.  If SSD is too expensive, then the WD high performance drives are a good choice.  If you are looking for a packaged solution, the G-Tech are excellent.  I used to use LaCie drives but I found them to be sufficiently unreliable such that I do not recommend their products to anyone.  Nice enclosures with really cheap drives.

Also popular are local RAID style drive enclosures.  Systems are available from Promise, CalDigit, Synology and Drobo.  RAID style systems can provide for protection from single drive failure by spanning data but recovery can be a real pain in the butt when something goes wrong.  Drobo made a name for themselves originally but having owned two and having had both units go bad more than once in and out of warranty, I would never suggest a Drobo to anyone.  Synology has developed a very good reputation and while their RAID is not proprietary as are some others, it is based on proven Linux models and works very well.  Be careful buying RAID because while it can look very pretty, upgradeability may be limited when you start to run out of space and be sure that data recovery is simple and proven.

The Storage Model

If you have a desktop computer, I recommend putting your photo / video library on a high performance external drive.  If you have a laptop, I recommend putting your photo / video library on a high performance external drive.  If you run Windows, at minimum your interface should be USB3, Thunderbolt if your system supports it.  If you run Macintosh, go Thunderbolt unless you have an older Mac Pro that has no Thunderbolt capability (like me), in which case get the CalDigit USB3 card and go USB3.  I tend not to depend on the internal drives to hold my libraries.  I capture images on the local laptop drive when in the field but transfer them to the library drive when I get home.  Laptop drives tend to be too small to be viable for long term library storage.

The backup drive should also be external, and use the fastest interface your computer can support.  If you use Macintosh, leverage the power of Time Machine to backup your library from the main external to this secondary drive.  If you use Windows, there are lots of backup applications to use, I am not a Windows user anymore, but I know that the Acronis people do very good software and it actually will perform a restore without dying.  Typically the backup drives are larger than the main drives so you can keep multiple versions.  This is a good use for RAID arrays.  They're usually a bit slower but make good backup targets.

The last piece of storage is offsite.  You can buy externals and move them to and from other sites or lockboxes, but the most effective way to do this is with secure cloud backup.  I have tried most of the services and recommend Crashplan above all others.  Your license will cover multiple computers and does not restrict backing up external drives.  Most other cloud services don't do external drives at all or charge extra for the privilege.  Carbonite gets lots of advertising but their pricing model is onerous and they treat external drives like lepers.

Now that you have a good storage model, let's get to cataloging.

In my opinion a photo editor is not necessarily a catalog system.  Photoshop comes with Bridge.  Bridge is a file browser not a catalog system.  It sucks and blows simultaneously.  While there are lots of editors, there are only two serious editors that also have very strong catalog functions and they are Aperture (Macintosh only) and Lightroom (Windows and Macintosh).  I recommend Lightroom to EVERYONE.  It works, there's tons of free training on the web and you can build your own catalog system and let the Lightroom engine do the majority of the work.

in my world, I set up my Lightroom catalog to store files by the date of capture.  This is the default so not a lot of work there.  At time of import, I have Lightroom make a second copy to a completely different drive as well (so I have three local copies plus cloud).  I used to have Lightroom convert to DNG at time of import, I no longer do this.  At time of Import, I use a preset to apply metadata information including my copyright and rights information as well as IPTC information into each file.  I choose to COPY the original into my Lightroom library so Lightroom can act as my organizer.  I use keywords on each import because I typically do an import after each shoot.  I don't leave cards in cameras for multiple shoots. Keywords help me find files after the fact so I include things like shoot information, lens and camera info, model names etc.  Anything that might apply to the entire import.  Once the import is done, I immediately create a collection for the entire import and give it a useful name.  I then create other collections to subdivide the images into easily findable subsets.  If I'm really bored, I will then specifically keyword select photos but this is a lot of work and hasn't made a real difference so I do it less and less.  Collections are for me the most useful of Lightroom's cross-hierarchy model.  The copy process means that I have a Lightroom library of photos, along with a separate Catalog file, both of which that get backed up to the local RAID array.  That original copy sits untouched on that other drive.  It's my last resort local backup.  I then set Crashplan to backup both my Catalog and my Lightroom library to the cloud.  I've done restores from Crashplan as a test and while not speedy, it works.  I've also done test restores from my local backups to be sure that they work as well.

Because of metadata, IPTC, keywords and EXIF, I can search Lightroom using multiple criteria and I can usually find what I am looking for no matter how old in a couple of minutes.  I don't have to build a structure in a folder system or directory because Lightroom does the job for me.  Transitioning to Lightroom is difficult for those who have built robust manual hierarchies because of the perceived loss of control.  It's all in the head of course, the Lightroom catalog is a much more powerful and much richer structure.

This also allows me to use Lightroom as the launch point for all other editing software.  I launch all plugins and Photoshop from within Lightroom.  This means that anything that goes out of Lightroom to an editor round trips back to Lightroom and updates the catalog.

I set Lightroom to optimize and backup the catalog on every exit.  This adds time to the close operation but keeps things running smoothly.  With over 50,000 images in my current library, performance is still good.

Lightroom does support multiple catalogs, but for my volume I don't need to use this function.  If you are making 150K retained images a year, maybe you do.  I also keep things clean by aggressive pruning of the library.  I make time to go through every import and use the X key to reject anything that I would not want to spend time editing right off the bat.  Then I delete the rejected photos.  Remember, I have that copy of everything that was made to the second drive at time of import if I ever really needed something (I never have) so by keeping my library to only good stuff it's fast and effective.

It's a longer answer to Rod's question but the subject warrants.  There are three considerations, physical media, storage architecture and catalog.  Hopefully this post helps you get a system set up that properly leverages all three.