Lexar Workflow DD512 SSD

OK, I'll admit it - I am a little paranoid about backing up images when I am travelling.  I probably take more precautions than many pros.  In my defence, I quite often travel alone and the photographic record is my travelling companion.  It is almost like that, if I lost it, it would be like I never went.  So I am always on the lookout for new backup tools that make this job faster and more secure without adding a lot of weight.

Lexar Workflow Hub

I had already invested in the Lexar Workflow hub.  I have the USB3 version but they also do it in Thunderbolt.  The hub has 4 slots and you can buy different slot-in modules with different kinds of interface - I had the SD and XQD readers.  The modules are also capable of being used as stand-alone reader as they have a USB3 slot which you can use with a cable.  

Recently, Lexar added SSD modules in two different capacities: 256GB and 512GB.  At first I didn't pay these too much attention as the price/performance at launch didn't appeal.  However, I stumbled on the 512GB model for £105 (including VAT) which was much more attractive.  The DD512 arrived formatted as exFAT which is a good option for cross system compatibility and large file handling.

At just 7cm x 6cm x 2cm the DD512 is very compact and, at 64g, very light.  Can it really be as fast as a "proper" SSD?  In truth, its not really as quick as the best 2.5in SATA drives.  Its scores using the Blackmagic Disk Test were still very respectable at 354MBps read and 237MBps write.

To put this into perspective.  I put this up against a 1TB Lacie Rugged drive which I have taken on previous trips.  This is a traditional spinning disk and, with its plastic armour, runs in at a lardy 334g - five times the wight of the DD512 (but with twice the capacity).  The ruggedness and the thunderbolt interface comes at a price too: £138 for the 1TB and £184 for the 2TB.  Do you need the ruggedness?  I told you about my paranoia right?  An SSD has an immediate advantage being solid state.  Do you need thunderbolt?  I tested the Lacie with the Blackmagic Tester and found no real difference in speed.  Using both interfaces the read speed was around 100MBps and the write speed about 95MBps.  So thunderbolt is not really valuable for performance.  It is really handy to have dual interfaces though.  I use a MacBook Air 11in for travelling and this has no built in card reader.  I typically will copy my cards to 2 different drives simultaneously using Shotput Pro and so I use one USB3 for the card reader and the other, plus the thunderbolt, for the target drives.  If you have a Windows laptop or a built in reader you could save some cash sticking to USB3.

Using the Blackmagic benchmarks, the DD512 has a clear lead in performance and in dimensions.  A spinning disk still wins on price per GB but the lead isn't as massive as it once was.

I decided to do some more real world tests.  For this I did what I usually need to do in the field which is to copy cards.  I used a card which contained 12.5GB of mixed media shot on the Sony A7s.  This contains a nice mix of chunky video files and raw images along with the plethora of small files that XAVC seems to like to produce.

Test	Time	MBps
Thunderbolt RAID to SD Card	3m17s	64
Thunderbolt RAID to DD512	56s	224
SD Card to DD512 (Workflow Hub)	2m22s	89
SD Card to DD512 (MacBook Air)	2m22s	89
DD512 to SSD (MacBook Air)	49s	257
DD512 to SSD (Mac Pro)	37s	340
SD to Lacie HDD	2m32s	83
Lacie HDD to SSD (Mac Pro)	2m01s	104

Let's analyse these results and try and add some perspective.  The SD card is a Transcend SDXC UHS1 Class 3 card which claims 90MBps read and 60MBps write.  My Blackmagic tests confirm this claim.  The first test was just to copy the card image back from my fast RAID array onto the card.  The card matched its claims in this real world test too.  I did the same copy to the DD512 out of interest.  Again it seems the Blackmagic tests are close to my real world scenario.

I copied the SD card onto the DD512 whilst both modules were in the hub.  I then repeated this test with them connected into separate ports on the MacBook Air.  Both tests scored 89MBps which is as fast as the card can read.  There was certainly no evidence of using the hub causing any contention - at least not at this data rate.

The next tests are the kind of thing you need to do back at base, copying off onto your primary machine.  For this I tried copying onto the Air's SSD and the more recent one in the Mac Pro.  The Mac Pro managed the full speed predicted by Blackmagic, the Air couldn't quite keep up but is still pretty speedy.

Finally, it was the Lacie's turn.  It was only slightly slower than the DD512 copying from the SD because the HDD write speed and SD read speed are similar.  Back at base though and its going to take you over three times longer to off-load the footage.  A full DD512 should take about 25 minutes to off-load against 82 minutes for the same data on the Lacie.

In Summary

For my own personal use case, I think the DD512 is a very good fit.  I already have the hub and the modules are going to be much easier to travel with.  In the triangle of performance, price and capacity they seem to be a good middle ground.  I will probably pair it up with a tried and tested HDD like the Lacie for its first trip while it proves itself in the field.

You could argue that the SD card is the limiting factor and I am not really gaining much performance.  The SD cards I use are top end cards but there are always some faster at the bleeding edge of performance at a hefty premium,  This tends to trickle down with time.  In the video field, 4K and RAW are pushing manufacturers onto newer platforms like XQD and Cfast 2.  When you are chewing up cards at upwards of 60MBps the ability to backup these expensive cards and reuse them can be important.

The DD512 is also quick enough to be used as an edit drive being on a par with a dual drive RAID0 HDD for speed.  If you were really confident in your ability to keep track of drives you could take the hub and build a mini RAID setup with something like Softraid.  At some point I think USB3 would start to give you diminishing returns though!

Movember - this time its personal

I have been doing Movember for a few years now off and on.  It was Philip Bloom whose campaigns brought this cause to my attention and I fundraise as part of his team.  In previous years it was because I was convinced by the aims of Movember but it was still an abstract concept.

This year things are a bit different.  My father’s PSA levels were high at his annual checkup at the beginning of this year.  He was referred for further tests.  This news hit us all hard, especially my Dad.  Any mention of the big C is always scary.  I thought it is worth telling our story for any who may face the same situation.

We had already hit the first problem with prostate cancer - diagnosis.  The PSA test is not very conclusive.  The doctors will talk about “cause for concern”.  The PSA suggests that there might be an issue but a biopsy is required to confirm it.  Even a biopsy is not a guaranteed indicator.  This is because the process of acquiring the sample is not just a sample in the medical sense but also in the statistical sense.  If the sample is taken from an infected area then you can have reasonable certainty that you require treatment.  However, if it doesn’t then you don’t know if you are clear or whether the biopsy just missed an infected area.

There was a touch of serendipity in our case. A few weeks before my father had his news I was listening to the radio when a story came on about prostate cancer.  I am not a big fan of medical stories but, on this occasion, I kept listening because of my experience with Movember.  The program was talking about a new diagnosis technique involving using an MRI scan to target areas on the prostate that looked more likely to be infected.  This has the added advantage that it looks at the prostate more holistically, where as the traditional biopsy is limited by the angle of entry.

I told my father about this so he had the opportunity to ask his specialist about it.  The specialist was familiar with this approach but it is still regarded as experimental in the UK so is not done under the NHS.

My father’s MRI scan showed some areas which were “cause for concern” so a multi-parametric biopsy was set based on the MRI results.  The biopsy was supposed to be an overnight stay in hospital, mainly so my Dad had support for getting used to the catheter required.  In the end it turned out to be a fair bit longer than that due to some conflicts with some of my Dad’s existing medications and some over-hydration.  Once my father was home there was nothing to do but wait for the results come through.  Even this isn’t a black and white decision.  The doctors classify the result in terms of risk.  Much of what they found was either clear or low risk.  There was one patch that was classified as medium risk and therefore “cause for concern”.  My father was given four options.  The first was to leave things alone and monitor the levels.  The second option was radio therapy.  The third was surgery on the prostate.  The final option, and the one my father elected to follow, was a high intensity ultrasound treatment.

This meant another trip back to hospital for an overnight stay.  Fortunately, we were much better prepared with the medications this time and everything went to plan and my Dad was back out on schedule.  So the actual treatment proved easier than the biopsy and was as a simple as you could hope when there is a catheter involved.

Several weeks following the treatment, Dad had to have another MRI scan to check for progress.  These all came back positive.  The final stage followed later and that was a retest of the PSA levels.  I am glad to report that these had dropped considerably and my father’s specialist was delighted with the results.

Of course, we will never know what the outcomes would have been if we had taken different treatment choices down the line but we are very happy with the outcome we had.  I like to think that Movember had a little part to play in the chain of events that informed our journey.  We would also like to thank all the staff at Yale Hospital Wrexham who were involved in my father’s treatment.

Our advice to anyone who finds themselves on a similar path is to talk to your doctors and make sure you are well informed on the treatments available to you.  Also to take comfort in our story that you can have a positive outcome especially if you get an early diagnosis.  It is for this reason I will continue to work for Movember for its work in raising awareness in the need to get tested.

The tache below is a link to my Movember page.  Please donate for this very worthy cause.

As part of Team Bloom 'Tache you can enter Phil's competition where there are some brilliant prizes for the filmmaker or photographer in your life.  If you don't have one then there is always me ;)  Follow this link to see the competition rules and don't wait too long.  I don't mind if you donate through me or directly as long as Movember gets the benefit. 

NAB 4K Update

NAB2013 has been and gone and its clear that 4K was a strong message from the show.  So has anything really major changed from my previous scriblings in the latest round of announcements?  I don't think so.  I don't think the overall trajectory has changed.  

​Blackmagic Production Camera 4K

The leading area at the moment is acquisition with a strong growth in the number of 4K capable devices from Sony, Blackmagic, Phantom, Kineraw etc.  The post world is keeping up with support for codecs, 6G SDI and Moore's Law helps with the processing.  We have a problem with storage though.  We are dealing with a double data whammy of RAW and 4K, but storage mediums are not increasing in density per cost much at the moment.

​We have the same problem with distribution.  Sony's announcements to support their 4K consumer devices are a bit of a drop in the ocean at the moment and not very compelling, even for AV enthusiasts like me.  My 1080p projector can reveal the difference between a blu-ray and a more compressed HD signal (download or satellite) or an up-scaled DVD.  Ultra HD has to be perceptibly better than HD blu-ray otherwise its just a waste of bits.

​Sony FMP-X1 4K media player

I am getting more tempted to look at 4K acquisition.  I do think there is a strong case for getting as good a "digital" negative as you can justify.  Even if your post workflow/pipeline is going to have to wait a while to catch up, there are probably enough options round dual record and proxies to make it viable now.

At some point, the infrastructural and technical challenges will be sorted and the sheer desperation ​of the consumer electronics business for you to buy new stuff will succeed.  Who knows, British Telecom may even realise that my rural exchange is still wired with string and replace it with something from the 21st century.  At that point I will be able to take a trip on my hover chair to a dusty cupboard and drag out those old 4K files.  

Film has been a pretty good archival medium.  Maybe not in its nitrate days when it was close to a WMD level of danger, but in the post war years.  I have pictures in my Aperture library going back to the turn of the millennium and my first digital camera - a 2MP Canon S10.  I have scans from the same era from slides made with a slide scanner.  Resolution, contrast, colour fidelity ... they have none of those things.  However, a scan of a slide now taken with my multifunction printer looks like it came from a DSLR.  There is little I can do now to my S10 files to make them better.  Digital files will always be limited by the technology they were acquired with.

​Zoom and Enhance - still largely Science Fiction

What the 4K!

Every so often my twitter stream will burst into life about 4K and the 4K Jets and the HD Sharks will get ready to rumble.  It may be a new 4k product announcement or a post on Red User which will act as the catalyst.  It seems far more polarising than say, 3D, as an issue.

I think there are a lot of aspects to this battle for hearts and minds - worth a little reflection I think.  If we look at acquisition then there just a couple of systems capable of 4k (or more acquisition).  Of these RED is the poster boy for 4K as Sony's F65 is only just starting to appear.  

F65 Prototype at NAB

Now most people would agree that more resolution is better (possibly not older actresses).  I saw a demo on the Sony stand at NAB2011 and it was deeply impressive despite the screen being huge and the audience being unrealistically close.  It is also true that it gives you more flexibility in post for reframing and cropping.  It certainly true that throwing resolution away usually works better than adding resolution.  Major motion pictures shot on film (though not all) tend to be mastered at 4K and many cinemas are equipping with 4K digital projectors.  RED claims this makes them the future-proof option.  30 years in IT leads me to believe that future-proof is a myth - perhaps future-resistant is more accurate.

The challenge at the moment on the acquisition and production side is the maths.  4K has four times more area roughly then 1080p.  That means 4x the data - all other things being equal.  That data is going to be present in your entire pipeline if you are going to master in 4K.  This has obvious implications in storing, movement and computation required in post.  This will increase costs and increase time.  Over time, improving technology will reduce this but it has significant implications now.

Currently, 4K equipped cinemas are the only place the public is likely 4K footage natively.  Will this change in the near future?  There are signs that the consumer electronic industry is going to put its faith in 4k as the next big thing after 3D.  CES this year had a smattering of 4K models.  CE Manufacturers always need a next big thing, its essential to their business model to persuade you that the consumer durable they sold you 2 years ago is no longer sufficient.  Interest in 3D is waning, they need a new flag to wave.

Toshiba 55" 4K TV

I happen to have a 3D set bought recently and a 3D projector waiting for an install.  Am I a 3D evangelist? Not really, I was really making the move to 1080p and 3D was just a bonus.  Its not that I don't like 3D, I enjoy the experience when its done well, but there is hardly anything to watch.  I have 2 sources of material: Sky and Blu-ray.  Sky has one channel which is about 70% repeated material (my impression, I am not sad enough to measure it) and a few blu-rays.  Even cinema releases which were 3D don't seem guaranteed to get a blu-ray disk.  If they do its normally dearer list price and less likely to be discounted.  

What have I got plugged into my setup.  I have a Blu-ray player (1080p), PS/3 (720p/1080p), SkyHD (720p/1080i) and AppleTV (720p). My amp is theoretically capable of handling and upscaling to 4k but compatibility is always in doubt till some standards emerge.  The signal path is via HDMI1.4 which can do 4k24p.  I do expect to see a jump in quality in my new 1080p projector over the old 720p one slightly offset by a bigger screen size.  However, I see no real difference in percieved resolution at my 3.5m viewing distance between my new 47" 1080p LED LG TV and my old 1024 x 720 Pioneer 42" plasma.  The step up in resolution to 4k is more marked than that between 1080 and 720 but I am not convinced it will be obvious at TV panel size.  How big do you have to go before you notice 4k.  In the world of AV consumerism there is the much feared WAF.  This is not a technical term but stands for wife acceptance factor.  Flat screen TV has faired well in WAF because the resulting sets maybe larger but they have taken up less space.  Get over 50" though and the sets are starting to draw to much attention to themselves and the WAF plummets relegating them to dedicated media rooms and bachelor pads.  This is probably less of an issue in the US where homes are typically more spacious, but any new format has got to succeed globally. 

I am the kind of early adopter guy who would buy 4K so what would it mean to me and how would I get it?  I think we can rule out broadcast.  In the UK, only the death of analgue has freed up enough space for a few HD over-air channels.  Sky too tends to worry about how many channels not the quality or we would not still be on 1080i.  Download services could distribute a file but how long are you prepared to wait.  In metropolitan areas it may be OK but I am limited to 5mbps out here in the sticks.  A single 4K movie not compressed to mush would probably exceed the typical UK monthly download limit for most punters.  So we are looking at some kind of disk or device to deliver the movie.  Blu-ray could do it but most current players arent going to be able to cope.  Even the ubiquitous and future-tolerant PS/3 is probably going to fail as its missing HDMI1.4.  If blu-ray can be stretched to fit (and in this I include at the pressing facilities) it just might gain a foothold with a slow burn dual play approach similar to 3D.  The disruption level is similar in that case.

I can't see a brand new physical format succeeding now.  DVD had the full weight of the industry behind it and its take off was slow in the UK taking a few years to hit critical mass and oust VHS.  Most of us early adopters have a drawer full of Region 1 disk we played on hacked DVD players.  We bought them so we had a decent choice back in the day and now we cant trade them in against blu-rays.  Blu-ray had a more troubled birth as it was fighting not just against DVD but HDDVD too.  It won the format war in the end but even after all this time it still has a smaller section in HMV than DVD.  Look round the rest of the store and you see racks of CDs.  CD is 30 years old and no higher definition format has come close to unseating it as a retail format.  The threat to CDs comes from a variety of formats that are worse - an interesting lesson for video.  Even HMVs future is far from assured - condemning video sources to online or a small selection at supermarkets. 

So winding right back to where we started with acquisition, how important is 4K?  If you are filming big budget features then it seems to make sense to feed the growing number of 4k projectors.  Not that I have seen any issues with Alexa films at my 4k equipped multiplex.  For lesser budget features then its a finely balanced decision.  What is going to provide the most production value up on the screen?  Any skimping you do on other crafts to feed that 4k pipeline are only going to be more visible in your end product.  For TV, Indie Films and Commericals I think its a long time before most of the 4k benefits start paying back with time being so critical.

"Worthy but Dull" time on Backup and Archive

I seemed to have been hearing or reading a lot about media data storage in the blogs, podcasts and courses I have been doing in recent times.  I have written before about my semi-paranoid approach to data on the road.  So this time I am going to concentrate on what happens when you get that all home.  I am not a leading authority on any of these technologies, I just have been looking for a decent solution for a long time and I have some experience which translates from the workplace.  

So lets get things in perspective first.  I am an enthusiast and my work has sentimental but no financial aspect bar the hardware it sits on.  This is a domestic environment and I only have ADSL access to the internet (5Mbps down 500Kbps up on a good day).  I have a Mac Pro which has 2 drives, one for Media and Apps and the other for data including my editing projects, music and photographs.  These are both 2Tb drives.  I have another 2Tb on a Firewire attached G-Tech RAID drive (RAID 0).  In addition to this I have a Windows Home Server (its a long story) with about a net 3Tb.  Lastly, I have a 2Tb Time Capsule and this is the only device not situated in my office. 

I have also have numerous external drives ranging from bus powered firewire drives mainly for travel to bare drives and a USB “Toaster”.  I have both DVD and Blu-Ray optical drives capable of writing and I have tried some off-line options.  I have Dropbox and the soon-to-die MobileMe.  I have tried Carbonite and CrashPlan offline services.

So I am no media facility but my levels of storage are probably into the top end of prosumer levels.  My set-up is no paragon and I am constantly in a state of angst about what to do about it.

The basis to my approach is the same process I would go through as an IT Professional for more commercial systems.  I have included a little mindmap that I used to bring things together that you might find handy. You have to look at risk and cost.  These are balancing factors and pretty much sit in opposition.  Reducing risk costs money and that cost is not linear.  The closer you try and get the risk to zero, the more the cost will spiral.  I also have a table of performance and costs.  This was compiled for my own use and should be taken as general guidance only.  Costs may vary over time and where you live and these are in good old pounds sterling.  The performance figures were based on multiple online reviews and I tried to stick to benchmarks which reflected the larger file sizes typical for AV users.

First of all we have to distinguish between backup and archive as these terms are often used interchangeably but they do have a separate purpose.  Backup is the one that is directly related to risk.  Your backup strategy is there to reduce risk.  It is primarily concerned with work that is in progress or highly likely to get accessed.  In a commercial environment this is often tied into a Disaster Recovery Plan or a Business Continuity Plan.  These differ mainly in scale.  Business Continuity looks at how can I get working again as soon as possible or keep working without your full resources for a short period.  DRP is more about recovering from vary large scale interruptions or total loss.  The business terms may sound over the top for a prosumer but the principle of risk analysis are just as valid outside big business.

You need to decide:

• How much work you are prepared to lose
• How long you are prepared to wait to get it back
• How much time you would be prepared to put into reloading things like software

Archiving is different in that it is an operational decision.  It is about treating material which you expect to use infrequently differently to that which is more likely to be needed.  Even in my simple set-up I have some storage that is speedier or more expensive or more difficult to extend than others.  I really don’t want to waste that by filling it full of stuff I do not expect to access.  I also don’t want to have to such through all that static data looking for what I am currently working on.  This is not stuff I will never look at.  The proper place for that is the trash can.  Original source files are a good example.  Its best practice to keep them but access will be rare.  You don’t need them handy but you do need to be able to get to them if you need to.  

Once you have placed data in an Archive this does not mean that you don’t also need to think about it in your Backup Strategy.  However, the approach you take will be probably be different to that in your working data.

Risks

Storage Failure

This is the most common form of data loss.  Hard disks are mechanical devices working on very fine tolerances.  They are vulnerable to shock and environmental issues as well as wear and tear.  Flash memory devices are more durable but have their own eccentricities which can affect longevity.  Optical disks are vulnerable to poor handling and decay.  Tape based media have a good reputation for durability but they are still a magnetic medium and they can wear or snag whilst being read.

The mitigation for storage failure is duplication.  If your data is on two drives the the mean time between failure (MTBF) is doubled.  That good but the real key is that the chances that both fail at exactly the same time become tiny.  You can duplicate using backup software to a different device, use folder synchronisation or Time Machine on Apple.  The most pure form of duplication is mirroring (also known as RAID 1).  This is either software or hardware controlled exact and real-time duplication of two drives.  I will come back to this in a bit more detail.  Mirroring is built into OSX at the software level and many PC’s have it at the BIOS level as a feature of their motherboard.

Data Corruption

This is not the failure of the data being read on the drive, this is an application error which writes bad data on to your drive.  Final Cut Pro 7 and Premiere Pro CS5 have both given me this issue in the past.  Both have an auto-save feature but it doesn’t always save you.

The mitigation for data corruption is versioning.  This is what auto-save is trying to do and is now built-in to OSX Lion.  It is also inherent in Time Machine and most backup software options.  However, unlike for storage failure, mirroring is not a mitigation, it will have two identically corrupt files.

Theft and Disaster

It seems an inevitable feature of the nature of the criminals that they are not going to leave that shiny drive array sat next to your Mac Pro when they are ransacking your place out of the goodness of their hearts.  Given time they are going to take anything plugged into your surge protectors and then they are going to nick those too!  This is the same with fire, flood

and other acts of God.  A wireless device like a Time Capsule may escape notice but you cant put it anywhere where the wireless signal will be poor.  A proper fire safe is probably beyond the reach of the prosumer.  The only option therefore is to get the data somewhere else.  This can be a physical or a virtual (cloud) location.

Virus & Security

Here the threat is to the data itself not the hardware it sits on.  Malicious code can brick your machine or force you to quarantine it from your own and public networks.  Hackers may seek to steal your data from your primary location or you backups.  If you have material of high sensitivity this may limit your backup options.  The mitigation here is to prevent the illegal access ever happening.  Once that firewall is breached then the mitigations are similar to Data Corruption

Running Out of Space

Is this really a risk?  Probably not literally but it tends to cause all kind of other issues.  The first problem is that things tend to break when the space runs out:  files get corrupted, background processes fail to run etc.  This may be something that is not immediately notice.  Secondly things tend to slow to a crawl as the system struggles to find a good place to put things.  There is also a tendency for you to start putting things where you know they shouldn’t be because there is no room in the right location - or is that just me.  One thing my long years in the IT game have taught me is that what seems a ludicrous amount of storage today will seem merely adequate in just a few years.

Devices

Internal Options

This is just about feasible on a Mac Pro but not the other Macs and MacBooks due to limited access and expansion capability.  Internally attached drives tend to be quicker than external single drives.  Anything even vaguely modern will be using SATA drives and this interface has been evolving through 1.5Gbps, 3Gbps and now 6Gbps.  However, you still only have 4 drive slots and your eggs very much in the one basket.  I would recommend concentrating more on performance on your PC and doing the backups elsewhere.

External Hard Drives

Absolutely massive choice of devices here so we need to split them into smaller categories.  They come in a variety of interfaces and may be pre-formatted for OSX or Windows although this is usually changeable.  They also can come with a variety of interfaces in order of speed USB2, FW400,FW800, USB3, eSATA, MiniSAS & Thunderbolt.

Bus powered

These are normally 2.5’ SATA drives in a protective case.  They are very portable and are typically in the 250MB to 1TB range.  USB2 is by far the most common but you also see USB3 & Firewire at a premium.

Western Digital My Passport Studio for Mac

Desktop

These are normally a 3.5” SATA drive in a housing.  The need plugging in somewhere and come in the 500MB to 3TB range.  Pretty much every interface type is available but faster speeds and multi-interface is at a premium.

Seagate FreeAgent GoFlex

Toaster

This is a drive interface and a SATA connection with a disk loading slot. Effectively the skeleton of Desktop External drive.  Most fit either size of SATA drive but for backup larger drives are the obvious choice.  It allows you to work with drives like you might have with tapes but remember tapes were born to be handled.  Bare drives are naked and vulnerable.

NewerTech Voyager

Flash Drives

These are stick like drives which are solid state.  Small and very portable they are limited to about 32GB currently.  USB2 is again by far the most common but a few USB3 sticks are beginning to turn up.

This is the entry level of backup solutions.  Bus powered and Flash Drives have limited capacity and are best seen as travel options and as good partners for Laptops where the primary storage itself is of limited capacity and probably slow.  For the same money a Desktop drive will be faster, bigger and with a better chance of a high speed interface.  This goes for Toasters too which offer access to excellent cost per MB figures providing you are a gentle disposition

Network Drives

There are a few single drive devices which are not direct attached.  In the Mac world the most obvious one is Apple’s Time Capsule.  This is actually a network router with a hard disk in one Mac Mini sized box.  Its a single drive of up to 3Tb capacity.  This is designed to integrate with the Time Machine service in OSX.  Most other similar drives are now designed with multimedia delivery in mind although they might be usable in this role with software or as a Time Machine Target.  Some modern routers have a USB port which allows a normal external drive to be attached.  This helps with sharing and with placement options but performance will take a bit hit compared to attaching that same drive direct.

Apple Time Capsule

Drive Arrays

I am calling anything with more than one drive an array but the breadth of devices that covers is pretty massive in terms of capacity, cost and implementation even if you rule out the massive SAS based storage arrays

If we look at them in terms of architecture first.  

Direct Attached

The simplest kind is a direct attached device.  In this format the drives attach just like a single drive external drive.  Intelligence in the device is limited.  Normally it is just adding a RAID controller card and some drive bays to an external drive.  Prosumer versions tend to look like stretched versions of a makers external drive and have just 2 drives.  Usually only RAID 1 (mirroring) and RAID 0 (striping) are offered.  RAID 0 gives you the full capacity of the dual drives and improved performance over a single drive but doubles the risk.  As such it is a poor choice for backup.  RAID 1 halves the risk but also the capacity so a maximum of 3TB.  If you can fit your backups in this size these drives in RAID 1 are safe pair of hands and good value but inflexible.  Slightly more expensive are versions which have removable drive caddies.  This opens up the world of drive rotation and they are a lot easier to get back online if you do have a drive go down inside.  Some of the manufacturers tie you into proprietary drives though which adds to the potential expense.  

Once you get beyond 2 drives new options become available.  Depending on just how many drives you have you can start to play RAID bingo.  Their are bewildering numbers of variants of RAID available each with its own subtle characteristics in terms of safety, performance in capacity.  Few devices support every possible option so its best to do your research on RAID levels, find the one which best matches your needs, then look for devices that support that RAID level.  To give an illustration I am going to look at the most commonly implemented on which gives good capacity and middling safety and performance - RAID 5.  In a RAID 5 array you lose one drives worth of capacity from the array and you need at least 3 drives.  So a 4 x 1TB RAID 5 array gives around 3TB of space.  The more drives you have the better the ratio of what you get to use.  For larger installations getting past the maximum physical drive size is essential.  Remember, backups may contain multiple versions of a file so you may need much more space for backup than you are using for your primary data.

Most multiple drive arrays have a choice of interfaces.  Its important to realise that this may be the deciding factor when it comes to performance.  It doesn’t matter how fast your drives are or which RAID level you picked, the data rate can’t exceed the bandwidth of the interface you use.

One thing to be aware of is that not all enclosures designed to hold arrays have the interface controllers built-in.  If you see an array described as a JBOD (Just a bunch of disks) then they will require a controller card (typically eSATA or MiniSAS) to be added to your PC.  For Mac users this means you will need a Mac Pro or a MacBook Pro with a PCI Express slot.  For any other Mac you will need one with the controller built in.

One last point about arrays - they can be touchy blighters!  All arrays have some hard maths going on in the background to make sure they OK.  If anything goes wrong then they have to either rebuild or go through diagnostics.  Now this is way better than losing a whole load of data but it will at the very least reduce performance while it happens or even take the device off-line.  This can take hours or even days.  I had a Windows NT machine with a bios based RAID controller.  One of the things that would upset it was the PC crashing or needing a forced reboot.  I said this was a Windows NT PC right?

G-Tech G-Speed QPromise Pegasus R6 (first Thunderbolt Array available)

Storage Robots

This term comes from Drobo and really the only thing that comes close in principle is my old Windows Home Server.  The main difference to the other device types is that Drobo and Microsoft did not go down the RAID route to achieve redundancy.  RAID is very fussy about disks.  Ideally RAID likes matched sets of the same type and capacity.  It will tolerate  a smaller capacity drive but at a big cost in overall capacity as everything works of multiple of the smallest drive leaving capacity unusable. Drobo & WHS will take anything you have handy and present it as a single drive but still retaining fault tolerance should a drive fail.  Using all your bays and need more space?  Take out a small drive and add a bigger one - the device does the rest.  This is impossible with a RAID array without changing all the drives and massive amounts of rebuilding and no small risk.  With RAID go as big as you can justify from the outset.  This approach focusses on easy of use and flexibility rather than performance.

Microsoft seems to have gone cold on WHS and the latest version has lost this feature so Drobo is now unique.  Its worth noting Drobo now offers NAS and SAN options as well as direct.  

Drobo S

Network Attached Storage (NAS)

A NAS is really a small server optimised for sharing data.  The RAID arrays are the same in principle as a direct RAID device but a NAS is seen as a completely separate device on your network where as a direct attached device appears as a local disk on your PC.  The benefit is that multiple computers can use the device as a backup target but network speeds rarely match those of direct interfaces in the real world as that bandwidth is shared with other traffic.  These are servers, so while they tend to look superficially the same, performance can vary widely due to CPU, memory, network and disk controllers.  There is also the quality of the interface (normally browser-based) to consider.  Strong competition in this sector keeps prices down so they offer good value if your priority is capacity and consolidation not performance.

Promise NS4600

Storage Area Network (SAN)

This is a bit of a specialised device.  Its really designed for big server rooms where multiple diskless servers are connected by a high speed fibre interface to a big box stuffed full of disks.  Each server is allocated a proportion of the disk and thinks it is local.  When you have a lot of disks the high cost of the interface hardware is offset by not having to provide a high level of redundancy and case room for each individual server.  There is a lower cost option called iSCSI that uses a network connection instead of the fibre.  But its still more a way of providing primary storage rather than a backup tool.

Optical Storage

We know Steve Jobs would like to kill it off but most computers still have some kind of Optical writing capability.  Most likely this is good old DVD.  Compared to magnetic and flash media DVDs are as cheap as chips.  They are also pretty easy to store and should last a while if you don’t go too cheap and handle with care.  But when a single memory card from your camera might have 32GB on it then 4.7GB is started to look a bit titchy.   Dual Layer disks help but are 10 times more expensive.  Blu-ray ups the capacity again to 25 or 50GB but again the cost rises dramatically and writers have never gained the kind of popularity which will drive the prices down to DVD levels and its doubtful they ever will.  Capacity is not the only issue - optical media is very slow to write and not very good for iterative saves.

Cloud Storage

This is a an internet service provision where your data is stored in a data centre operated by a third party.  This storage will be enterprise class, high redundancy and high availability.  All you need to do is set-up an account and run a client utility on your PC.  Most companies will offer different subscription models to suit users who have different size requirements or multiple devices.  They are split into two main types of service, synchronisation and backup.  

Dropbox is the best known of the synchronisers.  Rather than back up your whole disk you place data in a specified virtual folder structure.  The Dropbox client then replicates this in the cloud.  Any device you have Dropbox on, including mobile devices like smart phones, can access the data or you can go through a web client.  You can even elect to share folders with other people you nominate.  This is really designed for people who want to collaborate with others or tend to use multiple devices.  Its not really meant as a backup solution.  Its great for ad-hoc backups of relatively small amounts of data though.

The second type of server is aimed at the backup market.  Providers include people like Carbonite, Backblaze and CrashPlan.  My own experience of on-line backup as not been great but if I wanted a better one I would have to move.  The upstream speed on my ADSL is a paltry 400Kbps maximum.  If its at maximum it really seems to interfere with the downstream rate too.  With Carbonite in its low priority mode I only managed about 30Gb of backup over a month.  So you have to make sure that you link speed is going to make online viable for the size of data you need to backup.  Some vendors do offer a hard disk data take-on option.  This means that you only have to worry then about new data or changes being backed up.  They may also offer a hard disk restore option if the shipping time would be quicker than restoring over the net.

Arguably the biggest issue with a Cloud service is that you lose a degree of control over your data.  All the companies publish their policies on security and encryption but once it is in the Cloud you can never be truly be sure where it is or who can access it.  Unless you are a specialist it should be the case that they are better at security than you are but they are also a more obvious target.  Hackers or disgruntled staff are not the only risk - what if the company goes out of business?  If this is just a backup of your holiday movies from Majorca than its probably not an issue. If it is confidential work for clients then you need to take this into account. 

Other People Disks

This is a service that CrashPlan introduced me to.  Their client allows backups to other devices running CrashPlan.  This can be on a private network or via the internet.  The expectation is that this will be a “buddy” system where you agree to provide a reciprocal arrangement with someone you trust.  The performance limitation will be primarily the link you choose.  A private network should be much faster than an internet link.  A hard disk starter backup will probably be advisable.  It’s unlikely that your mate is going to have the same level of kit as a Cloud based service but at least you “know where they live”.

Tape

It may be old-school but tape is still a very effective option for both archiving and backup.  The standard that is best supported for big data volumes is called LTO.  If you do a bit of research you will see numbers after that and these are significant.  It refers to the generation of the standard.  Each generation has grown in tape capacity and data rate.  Backwards and forwards compatibility are enshrined in the standard as are the expected life and re-uses a tape should manage.  This is an enterprise class solution which means heavy-duty, fast, reliable and expensive.  Most of that cost though is up front, the tapes themselves are less expensive than other mediums.  You are going to need to have a lot of data though before you break even with disk based alternatives.

Quantum LTO5 Drives

Archiving

Archiving is “cut and paste” to Backup’s “Copy and Paste”.  If the data you are archiving still exists in its original place when you started then you didn’t really archive it, you just backed it up.

Why archive?

So if we haven’t improved our data security by archiving it why even bother?  Well there are several reasons why it is beneficial to do so.

Freeing up scarce resources

Perhaps you edit on a laptop in the field and to maximise your performance you have a fast but small SSD drive.  You would soon run out of space on that SSD if you leave old projects on there.  Some NLE’s have media manager functions that can reduce a project to just the assets used when they consolidate and move a project.

Move to safer environment

Resources like laptops, portable drives, fast RAID 0 arrays have inherently more risk attached to them.  Moving data to a less risk environment is advisable.

Confidentiality

If you are working with multiple clients you may want to avoid one client seeing any assets or folder names which may reveal the identity of another client.

Allowing a different backup regime

An archive isn't a copy.  So archives need backing up like everything else.  However, they are going to have a different risk profile and data change rate to projects in progress.  It makes no sense to tie up resources backing up the same file every night when there is only a tiny chance it will change.  If you do get to a situation where an old project needs substantial rework then it should be moved out of the archive back into the work-in-progress storage.

Cataloguing

You have to assume that by the time you need something in the archive that you will have forgotten all about it.  At my age, this is almost certainly true.  So how do you find stuff when people ask?  Perhaps you are just interested in finding something to use as stock footage in another project and just want to look on spec.

Folder and naming conventions

This is the place to start.  A degree of of orderliness is essential if you want to be able to find things.  How far you have to go with this probably depends on your working environment.  There are defined practices for movie and broadcast that you can use but these could be overkill and bring in a new level of jargon.  Some products may generate or impose the structure for you.  If you are using multiple software solutions sometimes you may have some conflicts.  My own general principles has to be to try and keep any software bin structures and physical folder structures in sync as much as possible.  Try and be descriptive with the project folder names.  If you are going to use any coding schemes then you need to be consistent and you need to make sure the key or index is accessible.

Metadata

With FCPX Apple moved to a metadata focused model from a bin based one much to the consternation of the editing community.  To me this wasn't too much of a shock to adjust to.  Not because of iMovie but because this is the same model that Aperture uses.  What did shock me was the poor integration with Aperture from FCPX but that is a different rant.  As a database guy, I appreciate that tagging is able to encode the multiple attributes that a clip possesses in a way bins can’t without a lot of duplication.  However, you have to be disciplined about it.  Video is a bit sparse for metadata compared to stills with their well established EXIF standard.  However, I think this will change over time and more metadata will arrive with the files themselves.  One of the interesting features of FCPX was autotagging based on analysis.  For example, it attempts to tag clips as wide shots or close-ups or for shakiness.  Adobe has some interesting technologies for voice recognition which could also find their way into metadata.

Off-line Cataloguing

If you use tapes or a tape-like workflow with other media, then you are going to need to generate a catalogue.  This is because your data is sat in a cupboard somewhere and is not available for searching.  So you need to capture the data structure and metadata into a stand-alone database that remains on-line.  You also need a physical indexing system so that when you search the catalogue you can go straight to the right disk or tape in the cupboard.

Retrieval

Whether its an archive tape or a backup disk its utterly useless if you can’t read it.  Some LTO tapes are guaranteed for 30 years.  30 years ago, in 1981, myself and three school friends won our school its first computer (a BBC Micro) in an inter-schools quiz on TV.  I think I would struggle to read files a BBC Micro saved onto cassette tape these days.  To be honest it was a bit hit and miss in 1981.  I have never really re-used my DV tapes and they acted as an archive once I had transcoded the footage onto to disk.  However, my aging DV camcorders are getting a bit flakey.  When the last one dies I lose my archive or will be scouring Ebay.

To ensure retrieval, you often need to transfer the data to a newer format prior to the old format being obsolete.  However, you need to be mindful of retaining as much quality as possible.  In the heyday of DV and DVD, burning to a DVD may have seemed a decent archive solution.  However, the transcode to MPEG2 within the DVD’s storage limit may have degraded the image.  However, a data DVD backup of AVI files created in the DV codec should be a true copy of what is on the tape.  But how long will that codec or file format be readable?  Always be on your guard for technical obsolescence whether it be hardware or software and be planning your transfer solution.

Summary and Recommendations

So my set-up is a paragon of virtue?

Err...not so much.  Part of the reason I am informed on this topic is I am perpetually in search of that ideal solution.  I am more of an experimenter than a completer finisher so I am always wrestling with finding a method and not quite as good at implementing it.  However, imperfect as it maybe, I do have some of the risks mitigated.  Lets go back to my basic questions and I will answer them for myself:

• How much work you are prepared to lose
  • Nothing, its not important to anyone else but I would hate to lose it.  My inner squirrel can’t cope with that.
• How long you are prepared to wait to get it back
  • No rush, as long as I know I can get it back time is not a big issue
• How much time you would be prepared to put into reloading things like software
  • I have a low tolerance for repeating things I have already done.  I have loads of software & plugins and the like.  I am prepared to lay out some cash to avoid that pain.

So how am I doing in mitigation of risks then?

Storage Failure

My main system drive is covered by Time Machine and my original media archive is on the Windows Home Server so they are OK.  However, my project drive is too big to fit on the same Time Capsule so I am having to do periodic back ups to an external drive.  This is not ideal.  I also have a few chunky things that have found their way onto my fast external firewire RAID 0 drive that I know shouldn’t be there but won’t fit anywhere else.

Data Corruption

Time Machine is a versioning solution so is good protection and the stuff on WHS is static so low risk.  I do have 2 generations of backup on the WHS too.  Again I am vulnerable on my other drive and the RAID.  My adhoc backups are overwrites so give less protection.

Theft and Disaster

I now use a Toaster to backup to bare drives which I transport in inexpensive plastic boxes.  I have 2 which I use alternatively and give to my parents who live 30 miles away.  I should do this a bit more often than I do but its a manual process and time flies.  I tried online backup but my rural ADSL is the limiting factor.

Virus and Security

I am a Mac user so the threat has traditionally been lower.  However, I have my doubts this will continue given Apple’s rise in market share and No1 company status.  I keep up-to-date with software updates on all devices and I have anti-virus and firewalls running on my PC’s and Mac Pro.  I don’t have significant Cloud based assets to be under threat.

Archiving

I don't have sufficient data or operational driver to make this necessary yet.

So if you know better, whats holding you back?

Well there is cash for a start.  Its money well spent but this is a hobby and there are so many, many things on my wish list... damn you Internet.

Secondly, I am that uncomfortable stage where 2TB drives are a bit under pressure but 3TB drives are still a hefty premium.  Once you scale up your primary storage everything needs to move up with it.  Also I am excited by the potential of Thunderbolt but we are still short on peripherals, including anyway to get it on my Mac Pro.

My non-media data is fine how it is.  I am not struggling for space and a Time Capsule is a good solution due to its risk and change profile.  With a bit of careful targeting I could probably even make the Cloud work for me whether this is through iCloud or another service we will have to see when Apple’s full plans are more evident.

My original media store is fine where it is.  My little WHS might be a technical dead-end but it does the job and has some growing room still.

My media drive I am hoping to replace with a large Thunderbolt array.  Thunderbolt is a fast enough interface that some of the old data organisation best practice designed to stop you hitting bandwidth bottlenecks just don’t apply.  This makes data organisation less complex.  I want to have 2 x 3TB drives in the array configured as RAID1 for backup.  Periodically I would swap one of the drives and keep my sneakernet arrangement with my Dad.  Fingers crossed they find a way of squeezing 4TB on a drive before 3Tb gets too small.

I could get the job done with something like a MiniSAS array now but I feel, given the level of investment, I should hang on a bit.  Without an option to retrofit Thunderbolt to my Mac Pro, I may not get the option and have to go that way.

Some helpful links:

Chris Fenwicks archive on a budget

16x9 Convergence podcast