Mark Lyn­der­say

It seems odd­ly rel­e­vant, on the Tues­day af­ter one of the most sa­cred week­ends on the Chris­t­ian cal­en­dar, to be talk­ing about back­up.It's a text I re­turn to oc­ca­sion­al­ly, some­times preach­ing from a place of deep and un­re­place­able loss, but like the cru­ci­fix­ion and res­ur­rec­tion, it's a nar­ra­tive that nev­er seems to get old. Or par­tic­u­lar­ly com­fort­able, come to think of it.

I've writ­ten this be­fore, and I'll write it again. There are on­ly two kinds of com­put­er users. Those who have lost da­ta and those who will lose da­ta.If you think the hu­man spir­it is eas­i­ly bro­ken, have a look at the ra­zor thin tol­er­ances and im­pos­si­bly small mech­a­nisms in­side your hard dri­ve.

What's shock­ing about dri­ve fail­ures isn't that they hap­pen, it's that they hap­pen with ac­cept­ably min­i­mal fre­quen­cy.

You may rest as­sured, how­ev­er, that dri­ves, whether me­chan­i­cal or sol­id state, will even­tu­al­ly fail and you have no as­sur­ance that such fail­ures will come with any fore­warn­ing at all. I run soft­ware on my serv­er that mon­i­tors each dri­ve's SMART sta­tus con­tin­u­ous­ly, but that didn't help with any fail­ures.

My para­noia about these mat­ters is per­va­sive, ab­solute and on­go­ing. Painful ex­pe­ri­ence has taught me that back­up should nev­er be con­sid­ered as a mat­ter of du­pli­cat­ing files; it must be­gin from the per­spec­tive of re­cov­ery.

Your strat­e­gy must be­gin by en­vi­sion­ing the hor­ror of los­ing every­thing on your com­put­er. The ef­fec­tive­ness of your back­up strat­e­gy will be mea­sured by how long it takes you to first, get back to work and then to re­store all your da­ta.My own sys­tem has four ac­tive tiers.

Every­thing on my imag­ing work­sta­tion/serv­er is con­tin­u­ous­ly backed up us­ing the built-in Mac­in­tosh in­cre­men­tal back­up ser­vice, Time Ma­chine.To cre­ate a big enough dri­ve for that, I've tak­en the risky mea­sure of cre­at­ing a striped raid, which cre­ates a vol­ume that's twice as like­ly to fail, so it's a safe­ty net made of fee­bler strands than I'd like.

I've al­ready re­cov­ered a 2TB in­ter­nal dri­ve from a Time Ma­chine back­up (the serv­er hosts sev­en dif­fer­ent dri­ves in­ter­nal­ly), which took around a day.The next tier is a du­pli­cat­ed pair of dri­ves hold­ing files that have moved from ac­tive use to archival sta­tus that sit in an ex­ter­nal dri­ve box (usu­al­ly switched off un­til need­ed) on my desk­top.

The third tier is a dri­ve that gets up­dat­ed an­nu­al­ly (it's phys­i­cal­ly switched with a new­er, more cur­rent dri­ve) with all of my dig­i­tal im­ages to date that sits in my sis­ter's house in Hous­ton.The next two tiers have proven far more elu­sive.

The fourth tier was meant to be a col­lec­tion of op­ti­cal disks which would have moved the im­ages to write once me­dia, which has some ad­van­tages over me­chan­i­cal dri­ves.

That strat­e­gy mi­grat­ed from DVD da­ta disks to Blu-Ray in short or­der, but while both me­dia and dri­ves are in place, the process de­mands a lev­el of per­fect­ed or­gan­i­sa­tion in fil­ing that I haven't been able to muster ei­ther the time or en­er­gy to ham­mer in­to place.So I've skipped to the fifth tier, mo­ti­vat­ed by slow­ly ris­ing up­load speeds avail­able from lo­cal broad­band providers.

That so­lu­tion is a cloud based strat­e­gy, mov­ing crit­i­cal files to an over­seas lo­ca­tion over an In­ter­net con­nec­tion.To be clear, I am cur­rent­ly man­ag­ing a 4TB dataset of mas­ter files and de­riv­a­tive files with thou­sands of hours worth of cor­rec­tive work in­vest­ed in them. This is ac­tu­al­ly a fair­ly small over­all dataset to work with for any­one who works with RAW files as orig­i­nals and re­tains TIFF im­ages as mas­ters for cor­rect­ed im­ages.

I know wed­ding pho­tog­ra­phers who han­dle much larg­er col­lec­tions and once you start repli­cat­ing across mul­ti­ple dri­ves for re­dun­dan­cy, things quick­ly es­ca­late to en­ter­prise class da­ta so­lu­tions.The eter­nal dilem­ma is to get max­i­mum re­dun­dan­cy for min­i­mum cost with ready ac­cess to crit­i­cal files that are kept up-to-date with fre­quen­cy.

For on­line back­up, I'd in­ves­ti­gat­ed Back­blaze and Crash­Plan, both pop­u­lar with my col­leagues, but even­tu­al­ly chose Ama­zon Glac­i­er, the most cost-ef­fec­tive on­line da­ta stor­age I've found so far.

To ac­cess it, I've been us­ing Arq (http://ow.ly/ZUQFX), a cross plat­form back­up tool cre­at­ed by a cloud ser­vice com­pa­ny that per­forms in­cre­men­tal back­up to a range of oth­er ser­vices, adding new files to an ex­ist­ing da­ta stor­age pool af­ter an ini­tial back­up with ex­cel­lent in­ter­rup­tion re­cov­ery.

The ini­tial back­up took around a month, and I can add around 50GB of da­ta overnight.

Ama­zon Glac­i­er (http://ow.ly/ZUQz5) is "cool" stor­age. Trans­fer rates are slow­er than the more pop­u­lar S3 stor­age ser­vices and the com­pa­ny makes its mon­ey off of Put, Get and List re­quests, charg­ing as you add or down­load in­for­ma­tion from their servers.Left alone dur­ing Feb­ru­ary, my charges for 4TB dropped to US$15 for the month though it cost US$40 per month dur­ing the ini­tial up­load.

Im­mo­la­tion lev­el da­ta loss (turns to knock wood des­per­ate­ly) sees me re­quest­ing the back­up dri­ve from Hous­ton via UPS while ac­cess­ing on­go­ing work from Glac­i­er.The pieces are in place and test well. I re­al­ly hope they nev­er have to be put in­to ac­tion.