I seemed to have been hearing or reading a lot about media data storage in the blogs, podcasts and courses I have been doing in recent times. I have written before about my semi-paranoid approach to data on the road. So this time I am going to concentrate on what happens when you get that all home. I am not a leading authority on any of these technologies, I just have been looking for a decent solution for a long time and I have some experience which translates from the workplace.

So lets get things in perspective first. I am an enthusiast and my work has sentimental but no financial aspect bar the hardware it sits on. This is a domestic environment and I only have ADSL access to the internet (5Mbps down 500Kbps up on a good day). I have a Mac Pro which has 2 drives, one for Media and Apps and the other for data including my editing projects, music and photographs. These are both 2Tb drives. I have another 2Tb on a Firewire attached G-Tech RAID drive (RAID 0). In addition to this I have a Windows Home Server (its a long story) with about a net 3Tb. Lastly, I have a 2Tb Time Capsule and this is the only device not situated in my office.

I have also have numerous external drives ranging from bus powered firewire drives mainly for travel to bare drives and a USB “Toaster”. I have both DVD and Blu-Ray optical drives capable of writing and I have tried some off-line options. I have Dropbox and the soon-to-die MobileMe. I have tried Carbonite and CrashPlan offline services.

So I am no media facility but my levels of storage are probably into the top end of prosumer levels. My set-up is no paragon and I am constantly in a state of angst about what to do about it.

The basis to my approach is the same process I would go through as an IT Professional for more commercial systems. I have included a little mindmap that I used to bring things together that you might find handy. You have to look at risk and cost. These are balancing factors and pretty much sit in opposition. Reducing risk costs money and that cost is not linear. The closer you try and get the risk to zero, the more the cost will spiral. I also have a table of performance and costs. This was compiled for my own use and should be taken as general guidance only. Costs may vary over time and where you live and these are in good old pounds sterling. The performance figures were based on multiple online reviews and I tried to stick to benchmarks which reflected the larger file sizes typical for AV users.

First of all we have to distinguish between backup and archive as these terms are often used interchangeably but they do have a separate purpose. Backup is the one that is directly related to risk. Your backup strategy is there to reduce risk. It is primarily concerned with work that is in progress or highly likely to get accessed. In a commercial environment this is often tied into a Disaster Recovery Plan or a Business Continuity Plan. These differ mainly in scale. Business Continuity looks at how can I get working again as soon as possible or keep working without your full resources for a short period. DRP is more about recovering from vary large scale interruptions or total loss. The business terms may sound over the top for a prosumer but the principle of risk analysis are just as valid outside big business.

You need to decide:

How much work you are prepared to lose
How long you are prepared to wait to get it back
How much time you would be prepared to put into reloading things like software

Archiving is different in that it is an operational decision. It is about treating material which you expect to use infrequently differently to that which is more likely to be needed. Even in my simple set-up I have some storage that is speedier or more expensive or more difficult to extend than others. I really don’t want to waste that by filling it full of stuff I do not expect to access. I also don’t want to have to such through all that static data looking for what I am currently working on. This is not stuff I will never look at. The proper place for that is the trash can. Original source files are a good example. Its best practice to keep them but access will be rare. You don’t need them handy but you do need to be able to get to them if you need to.

Once you have placed data in an Archive this does not mean that you don’t also need to think about it in your Backup Strategy. However, the approach you take will be probably be different to that in your working data.

Risks

Storage Failure

This is the most common form of data loss. Hard disks are mechanical devices working on very fine tolerances. They are vulnerable to shock and environmental issues as well as wear and tear. Flash memory devices are more durable but have their own eccentricities which can affect longevity. Optical disks are vulnerable to poor handling and decay. Tape based media have a good reputation for durability but they are still a magnetic medium and they can wear or snag whilst being read.

The mitigation for storage failure is duplication. If your data is on two drives the the mean time between failure (MTBF) is doubled. That good but the real key is that the chances that both fail at exactly the same time become tiny. You can duplicate using backup software to a different device, use folder synchronisation or Time Machine on Apple. The most pure form of duplication is mirroring (also known as RAID 1). This is either software or hardware controlled exact and real-time duplication of two drives. I will come back to this in a bit more detail. Mirroring is built into OSX at the software level and many PC’s have it at the BIOS level as a feature of their motherboard.

Data Corruption

This is not the failure of the data being read on the drive, this is an application error which writes bad data on to your drive. Final Cut Pro 7 and Premiere Pro CS5 have both given me this issue in the past. Both have an auto-save feature but it doesn’t always save you.

The mitigation for data corruption is versioning. This is what auto-save is trying to do and is now built-in to OSX Lion. It is also inherent in Time Machine and most backup software options. However, unlike for storage failure, mirroring is not a mitigation, it will have two identically corrupt files.

Theft and Disaster

It seems an inevitable feature of the nature of the criminals that they are not going to leave that shiny drive array sat next to your Mac Pro when they are ransacking your place out of the goodness of their hearts. Given time they are going to take anything plugged into your surge protectors and then they are going to nick those too! This is the same with fire, flood

and other acts of God. A wireless device like a Time Capsule may escape notice but you cant put it anywhere where the wireless signal will be poor. A proper fire safe is probably beyond the reach of the prosumer. The only option therefore is to get the data somewhere else. This can be a physical or a virtual (cloud) location.

Virus & Security

Here the threat is to the data itself not the hardware it sits on. Malicious code can brick your machine or force you to quarantine it from your own and public networks. Hackers may seek to steal your data from your primary location or you backups. If you have material of high sensitivity this may limit your backup options. The mitigation here is to prevent the illegal access ever happening. Once that firewall is breached then the mitigations are similar to Data Corruption

Running Out of Space

Is this really a risk? Probably not literally but it tends to cause all kind of other issues. The first problem is that things tend to break when the space runs out: files get corrupted, background processes fail to run etc. This may be something that is not immediately notice. Secondly things tend to slow to a crawl as the system struggles to find a good place to put things. There is also a tendency for you to start putting things where you know they shouldn’t be because there is no room in the right location - or is that just me. One thing my long years in the IT game have taught me is that what seems a ludicrous amount of storage today will seem merely adequate in just a few years.

Devices

Internal Options

This is just about feasible on a Mac Pro but not the other Macs and MacBooks due to limited access and expansion capability. Internally attached drives tend to be quicker than external single drives. Anything even vaguely modern will be using SATA drives and this interface has been evolving through 1.5Gbps, 3Gbps and now 6Gbps. However, you still only have 4 drive slots and your eggs very much in the one basket. I would recommend concentrating more on performance on your PC and doing the backups elsewhere.

External Hard Drives

Absolutely massive choice of devices here so we need to split them into smaller categories. They come in a variety of interfaces and may be pre-formatted for OSX or Windows although this is usually changeable. They also can come with a variety of interfaces in order of speed USB2, FW400,FW800, USB3, eSATA, MiniSAS & Thunderbolt.

Bus powered

These are normally 2.5’ SATA drives in a protective case. They are very portable and are typically in the 250MB to 1TB range. USB2 is by far the most common but you also see USB3 & Firewire at a premium.

Western Digital My Passport Studio for Mac

Desktop

These are normally a 3.5” SATA drive in a housing. The need plugging in somewhere and come in the 500MB to 3TB range. Pretty much every interface type is available but faster speeds and multi-interface is at a premium.

Seagate FreeAgent GoFlex

Toaster

This is a drive interface and a SATA connection with a disk loading slot. Effectively the skeleton of Desktop External drive. Most fit either size of SATA drive but for backup larger drives are the obvious choice. It allows you to work with drives like you might have with tapes but remember tapes were born to be handled. Bare drives are naked and vulnerable.

NewerTech Voyager

Flash Drives

These are stick like drives which are solid state. Small and very portable they are limited to about 32GB currently. USB2 is again by far the most common but a few USB3 sticks are beginning to turn up.

This is the entry level of backup solutions. Bus powered and Flash Drives have limited capacity and are best seen as travel options and as good partners for Laptops where the primary storage itself is of limited capacity and probably slow. For the same money a Desktop drive will be faster, bigger and with a better chance of a high speed interface. This goes for Toasters too which offer access to excellent cost per MB figures providing you are a gentle disposition

Network Drives

There are a few single drive devices which are not direct attached. In the Mac world the most obvious one is Apple’s Time Capsule. This is actually a network router with a hard disk in one Mac Mini sized box. Its a single drive of up to 3Tb capacity. This is designed to integrate with the Time Machine service in OSX. Most other similar drives are now designed with multimedia delivery in mind although they might be usable in this role with software or as a Time Machine Target. Some modern routers have a USB port which allows a normal external drive to be attached. This helps with sharing and with placement options but performance will take a bit hit compared to attaching that same drive direct.

Apple Time Capsule

Drive Arrays

I am calling anything with more than one drive an array but the breadth of devices that covers is pretty massive in terms of capacity, cost and implementation even if you rule out the massive SAS based storage arrays

If we look at them in terms of architecture first.

Direct Attached

The simplest kind is a direct attached device. In this format the drives attach just like a single drive external drive. Intelligence in the device is limited. Normally it is just adding a RAID controller card and some drive bays to an external drive. Prosumer versions tend to look like stretched versions of a makers external drive and have just 2 drives. Usually only RAID 1 (mirroring) and RAID 0 (striping) are offered. RAID 0 gives you the full capacity of the dual drives and improved performance over a single drive but doubles the risk. As such it is a poor choice for backup. RAID 1 halves the risk but also the capacity so a maximum of 3TB. If you can fit your backups in this size these drives in RAID 1 are safe pair of hands and good value but inflexible. Slightly more expensive are versions which have removable drive caddies. This opens up the world of drive rotation and they are a lot easier to get back online if you do have a drive go down inside. Some of the manufacturers tie you into proprietary drives though which adds to the potential expense.

Once you get beyond 2 drives new options become available. Depending on just how many drives you have you can start to play RAID bingo. Their are bewildering numbers of variants of RAID available each with its own subtle characteristics in terms of safety, performance in capacity. Few devices support every possible option so its best to do your research on RAID levels, find the one which best matches your needs, then look for devices that support that RAID level. To give an illustration I am going to look at the most commonly implemented on which gives good capacity and middling safety and performance - RAID 5. In a RAID 5 array you lose one drives worth of capacity from the array and you need at least 3 drives. So a 4 x 1TB RAID 5 array gives around 3TB of space. The more drives you have the better the ratio of what you get to use. For larger installations getting past the maximum physical drive size is essential. Remember, backups may contain multiple versions of a file so you may need much more space for backup than you are using for your primary data.

Most multiple drive arrays have a choice of interfaces. Its important to realise that this may be the deciding factor when it comes to performance. It doesn’t matter how fast your drives are or which RAID level you picked, the data rate can’t exceed the bandwidth of the interface you use.

One thing to be aware of is that not all enclosures designed to hold arrays have the interface controllers built-in. If you see an array described as a JBOD (Just a bunch of disks) then they will require a controller card (typically eSATA or MiniSAS) to be added to your PC. For Mac users this means you will need a Mac Pro or a MacBook Pro with a PCI Express slot. For any other Mac you will need one with the controller built in.

One last point about arrays - they can be touchy blighters! All arrays have some hard maths going on in the background to make sure they OK. If anything goes wrong then they have to either rebuild or go through diagnostics. Now this is way better than losing a whole load of data but it will at the very least reduce performance while it happens or even take the device off-line. This can take hours or even days. I had a Windows NT machine with a bios based RAID controller. One of the things that would upset it was the PC crashing or needing a forced reboot. I said this was a Windows NT PC right?

G-Tech G-Speed QPromise Pegasus R6 (first Thunderbolt Array available)

Storage Robots

This term comes from Drobo and really the only thing that comes close in principle is my old Windows Home Server. The main difference to the other device types is that Drobo and Microsoft did not go down the RAID route to achieve redundancy. RAID is very fussy about disks. Ideally RAID likes matched sets of the same type and capacity. It will tolerate a smaller capacity drive but at a big cost in overall capacity as everything works of multiple of the smallest drive leaving capacity unusable. Drobo & WHS will take anything you have handy and present it as a single drive but still retaining fault tolerance should a drive fail. Using all your bays and need more space? Take out a small drive and add a bigger one - the device does the rest. This is impossible with a RAID array without changing all the drives and massive amounts of rebuilding and no small risk. With RAID go as big as you can justify from the outset. This approach focusses on easy of use and flexibility rather than performance.

Microsoft seems to have gone cold on WHS and the latest version has lost this feature so Drobo is now unique. Its worth noting Drobo now offers NAS and SAN options as well as direct.

Drobo S

Network Attached Storage (NAS)

A NAS is really a small server optimised for sharing data. The RAID arrays are the same in principle as a direct RAID device but a NAS is seen as a completely separate device on your network where as a direct attached device appears as a local disk on your PC. The benefit is that multiple computers can use the device as a backup target but network speeds rarely match those of direct interfaces in the real world as that bandwidth is shared with other traffic. These are servers, so while they tend to look superficially the same, performance can vary widely due to CPU, memory, network and disk controllers. There is also the quality of the interface (normally browser-based) to consider. Strong competition in this sector keeps prices down so they offer good value if your priority is capacity and consolidation not performance.

Promise NS4600

Storage Area Network (SAN)

This is a bit of a specialised device. Its really designed for big server rooms where multiple diskless servers are connected by a high speed fibre interface to a big box stuffed full of disks. Each server is allocated a proportion of the disk and thinks it is local. When you have a lot of disks the high cost of the interface hardware is offset by not having to provide a high level of redundancy and case room for each individual server. There is a lower cost option called iSCSI that uses a network connection instead of the fibre. But its still more a way of providing primary storage rather than a backup tool.

Optical Storage

We know Steve Jobs would like to kill it off but most computers still have some kind of Optical writing capability. Most likely this is good old DVD. Compared to magnetic and flash media DVDs are as cheap as chips. They are also pretty easy to store and should last a while if you don’t go too cheap and handle with care. But when a single memory card from your camera might have 32GB on it then 4.7GB is started to look a bit titchy. Dual Layer disks help but are 10 times more expensive. Blu-ray ups the capacity again to 25 or 50GB but again the cost rises dramatically and writers have never gained the kind of popularity which will drive the prices down to DVD levels and its doubtful they ever will. Capacity is not the only issue - optical media is very slow to write and not very good for iterative saves.

Cloud Storage

This is a an internet service provision where your data is stored in a data centre operated by a third party. This storage will be enterprise class, high redundancy and high availability. All you need to do is set-up an account and run a client utility on your PC. Most companies will offer different subscription models to suit users who have different size requirements or multiple devices. They are split into two main types of service, synchronisation and backup.

Dropbox is the best known of the synchronisers. Rather than back up your whole disk you place data in a specified virtual folder structure. The Dropbox client then replicates this in the cloud. Any device you have Dropbox on, including mobile devices like smart phones, can access the data or you can go through a web client. You can even elect to share folders with other people you nominate. This is really designed for people who want to collaborate with others or tend to use multiple devices. Its not really meant as a backup solution. Its great for ad-hoc backups of relatively small amounts of data though.

The second type of server is aimed at the backup market. Providers include people like Carbonite, Backblaze and CrashPlan. My own experience of on-line backup as not been great but if I wanted a better one I would have to move. The upstream speed on my ADSL is a paltry 400Kbps maximum. If its at maximum it really seems to interfere with the downstream rate too. With Carbonite in its low priority mode I only managed about 30Gb of backup over a month. So you have to make sure that you link speed is going to make online viable for the size of data you need to backup. Some vendors do offer a hard disk data take-on option. This means that you only have to worry then about new data or changes being backed up. They may also offer a hard disk restore option if the shipping time would be quicker than restoring over the net.

Arguably the biggest issue with a Cloud service is that you lose a degree of control over your data. All the companies publish their policies on security and encryption but once it is in the Cloud you can never be truly be sure where it is or who can access it. Unless you are a specialist it should be the case that they are better at security than you are but they are also a more obvious target. Hackers or disgruntled staff are not the only risk - what if the company goes out of business? If this is just a backup of your holiday movies from Majorca than its probably not an issue. If it is confidential work for clients then you need to take this into account.

Other People Disks

This is a service that CrashPlan introduced me to. Their client allows backups to other devices running CrashPlan. This can be on a private network or via the internet. The expectation is that this will be a “buddy” system where you agree to provide a reciprocal arrangement with someone you trust. The performance limitation will be primarily the link you choose. A private network should be much faster than an internet link. A hard disk starter backup will probably be advisable. It’s unlikely that your mate is going to have the same level of kit as a Cloud based service but at least you “know where they live”.

Tape

It may be old-school but tape is still a very effective option for both archiving and backup. The standard that is best supported for big data volumes is called LTO. If you do a bit of research you will see numbers after that and these are significant. It refers to the generation of the standard. Each generation has grown in tape capacity and data rate. Backwards and forwards compatibility are enshrined in the standard as are the expected life and re-uses a tape should manage. This is an enterprise class solution which means heavy-duty, fast, reliable and expensive. Most of that cost though is up front, the tapes themselves are less expensive than other mediums. You are going to need to have a lot of data though before you break even with disk based alternatives.

Quantum LTO5 Drives

Archiving

Archiving is “cut and paste” to Backup’s “Copy and Paste”. If the data you are archiving still exists in its original place when you started then you didn’t really archive it, you just backed it up.

Why archive?

So if we haven’t improved our data security by archiving it why even bother? Well there are several reasons why it is beneficial to do so.

Freeing up scarce resources

Perhaps you edit on a laptop in the field and to maximise your performance you have a fast but small SSD drive. You would soon run out of space on that SSD if you leave old projects on there. Some NLE’s have media manager functions that can reduce a project to just the assets used when they consolidate and move a project.

Move to safer environment

Resources like laptops, portable drives, fast RAID 0 arrays have inherently more risk attached to them. Moving data to a less risk environment is advisable.

Confidentiality

If you are working with multiple clients you may want to avoid one client seeing any assets or folder names which may reveal the identity of another client.

Allowing a different backup regime

An archive isn't a copy. So archives need backing up like everything else. However, they are going to have a different risk profile and data change rate to projects in progress. It makes no sense to tie up resources backing up the same file every night when there is only a tiny chance it will change. If you do get to a situation where an old project needs substantial rework then it should be moved out of the archive back into the work-in-progress storage.

Cataloguing

You have to assume that by the time you need something in the archive that you will have forgotten all about it. At my age, this is almost certainly true. So how do you find stuff when people ask? Perhaps you are just interested in finding something to use as stock footage in another project and just want to look on spec.

Folder and naming conventions

This is the place to start. A degree of of orderliness is essential if you want to be able to find things. How far you have to go with this probably depends on your working environment. There are defined practices for movie and broadcast that you can use but these could be overkill and bring in a new level of jargon. Some products may generate or impose the structure for you. If you are using multiple software solutions sometimes you may have some conflicts. My own general principles has to be to try and keep any software bin structures and physical folder structures in sync as much as possible. Try and be descriptive with the project folder names. If you are going to use any coding schemes then you need to be consistent and you need to make sure the key or index is accessible.

Metadata

With FCPX Apple moved to a metadata focused model from a bin based one much to the consternation of the editing community. To me this wasn't too much of a shock to adjust to. Not because of iMovie but because this is the same model that Aperture uses. What did shock me was the poor integration with Aperture from FCPX but that is a different rant. As a database guy, I appreciate that tagging is able to encode the multiple attributes that a clip possesses in a way bins can’t without a lot of duplication. However, you have to be disciplined about it. Video is a bit sparse for metadata compared to stills with their well established EXIF standard. However, I think this will change over time and more metadata will arrive with the files themselves. One of the interesting features of FCPX was autotagging based on analysis. For example, it attempts to tag clips as wide shots or close-ups or for shakiness. Adobe has some interesting technologies for voice recognition which could also find their way into metadata.

Off-line Cataloguing

If you use tapes or a tape-like workflow with other media, then you are going to need to generate a catalogue. This is because your data is sat in a cupboard somewhere and is not available for searching. So you need to capture the data structure and metadata into a stand-alone database that remains on-line. You also need a physical indexing system so that when you search the catalogue you can go straight to the right disk or tape in the cupboard.

Retrieval

Whether its an archive tape or a backup disk its utterly useless if you can’t read it. Some LTO tapes are guaranteed for 30 years. 30 years ago, in 1981, myself and three school friends won our school its first computer (a BBC Micro) in an inter-schools quiz on TV. I think I would struggle to read files a BBC Micro saved onto cassette tape these days. To be honest it was a bit hit and miss in 1981. I have never really re-used my DV tapes and they acted as an archive once I had transcoded the footage onto to disk. However, my aging DV camcorders are getting a bit flakey. When the last one dies I lose my archive or will be scouring Ebay.

To ensure retrieval, you often need to transfer the data to a newer format prior to the old format being obsolete. However, you need to be mindful of retaining as much quality as possible. In the heyday of DV and DVD, burning to a DVD may have seemed a decent archive solution. However, the transcode to MPEG2 within the DVD’s storage limit may have degraded the image. However, a data DVD backup of AVI files created in the DV codec should be a true copy of what is on the tape. But how long will that codec or file format be readable? Always be on your guard for technical obsolescence whether it be hardware or software and be planning your transfer solution.

Summary and Recommendations

So my set-up is a paragon of virtue?

Err...not so much. Part of the reason I am informed on this topic is I am perpetually in search of that ideal solution. I am more of an experimenter than a completer finisher so I am always wrestling with finding a method and not quite as good at implementing it. However, imperfect as it maybe, I do have some of the risks mitigated. Lets go back to my basic questions and I will answer them for myself:

How much work you are prepared to lose
- Nothing, its not important to anyone else but I would hate to lose it. My inner squirrel can’t cope with that.
How long you are prepared to wait to get it back
- No rush, as long as I know I can get it back time is not a big issue
How much time you would be prepared to put into reloading things like software
- I have a low tolerance for repeating things I have already done. I have loads of software & plugins and the like. I am prepared to lay out some cash to avoid that pain.

So how am I doing in mitigation of risks then?

Storage Failure

My main system drive is covered by Time Machine and my original media archive is on the Windows Home Server so they are OK. However, my project drive is too big to fit on the same Time Capsule so I am having to do periodic back ups to an external drive. This is not ideal. I also have a few chunky things that have found their way onto my fast external firewire RAID 0 drive that I know shouldn’t be there but won’t fit anywhere else.

Data Corruption

Time Machine is a versioning solution so is good protection and the stuff on WHS is static so low risk. I do have 2 generations of backup on the WHS too. Again I am vulnerable on my other drive and the RAID. My adhoc backups are overwrites so give less protection.

Theft and Disaster

I now use a Toaster to backup to bare drives which I transport in inexpensive plastic boxes. I have 2 which I use alternatively and give to my parents who live 30 miles away. I should do this a bit more often than I do but its a manual process and time flies. I tried online backup but my rural ADSL is the limiting factor.

Virus and Security

I am a Mac user so the threat has traditionally been lower. However, I have my doubts this will continue given Apple’s rise in market share and No1 company status. I keep up-to-date with software updates on all devices and I have anti-virus and firewalls running on my PC’s and Mac Pro. I don’t have significant Cloud based assets to be under threat.

Archiving

I don't have sufficient data or operational driver to make this necessary yet.

So if you know better, whats holding you back?

Well there is cash for a start. Its money well spent but this is a hobby and there are so many, many things on my wish list... damn you Internet.

Secondly, I am that uncomfortable stage where 2TB drives are a bit under pressure but 3TB drives are still a hefty premium. Once you scale up your primary storage everything needs to move up with it. Also I am excited by the potential of Thunderbolt but we are still short on peripherals, including anyway to get it on my Mac Pro.

My non-media data is fine how it is. I am not struggling for space and a Time Capsule is a good solution due to its risk and change profile. With a bit of careful targeting I could probably even make the Cloud work for me whether this is through iCloud or another service we will have to see when Apple’s full plans are more evident.

My original media store is fine where it is. My little WHS might be a technical dead-end but it does the job and has some growing room still.

My media drive I am hoping to replace with a large Thunderbolt array. Thunderbolt is a fast enough interface that some of the old data organisation best practice designed to stop you hitting bandwidth bottlenecks just don’t apply. This makes data organisation less complex. I want to have 2 x 3TB drives in the array configured as RAID1 for backup. Periodically I would swap one of the drives and keep my sneakernet arrangement with my Dad. Fingers crossed they find a way of squeezing 4TB on a drive before 3Tb gets too small.

I could get the job done with something like a MiniSAS array now but I feel, given the level of investment, I should hang on a bit. Without an option to retrofit Thunderbolt to my Mac Pro, I may not get the option and have to go that way.

Some helpful links:

Chris Fenwicks archive on a budget

16x9 Convergence podcast