My experience of hiring a BMW i3 EV in Mallorca, Spain
Read moreThe rise of 4K and the Conundrum of the 1DC
Just a few years ago, I darkened the door of my first broadcast exhibition. There have been many since. Back in those days the big beast striding the halls was 3D. The massed ranks of the electronic industry were desperate to convince you that you needed a 3D strategy. If you only cared about 2D then your market was about to be flat. As I wandered the shiny things and flashing lights of this year's BVExpo in London 3D was notable mainly by its absence. So what are the cool kids playing with this year? Well that would be 4K sir. I wouldn't say that it had quite the same razzmatazz behind it as 3D did, but there was definitely an underlying theme.
Like 3D, 4K is not something you can easily see at home without the right equipment. So a show like BVE is the best place to get your own eyes on the effect. What 4K is about is resolution. With at least 4x the number of pixels as 1080p, 4K or UltraHD is a big jump in resolution. Technically it presents fewer curve balls than 3D, it is mainly a question of data and how to deal with it. However, there are still a lot more unknowns than there was with 3D. Even in those early days of 3D we knew a lot more about how it was going to reach our screens whether that was at the local multiplex or in our own lounge. Most of the cinemas I frequent are already projecting in 4K but the domestic arrangements are still barely pencilled in.
So if you are not producing your output for the Odeon or Vue, should you care? Is this another fad? My own feeling is that it is more than a fad. I personally think it is inevitable but that its rate of advance is far from certain. These are my latest thoughts based on the show floor and attending Philip Bloom's presentation on the Canon 1DC.
On the show floor, a number of stands had 4K monitors of around 50inches or less. The did look good but then so does a high end 1080p monitor. Even very close its very hard to see a pixel and this has a benefit on disguising noise. The problem is that you don't have to move very far from the screen for that to be also true of HD. I am not sure that I would be able to tell them apart on a 50in screen at say 4m - roughly what I have in my lounge. The same was pretty much true when I moved up from 720p to 1080p. This is very typical setup in the UK. Love it or hate it, 3D does offer an alternative experience. The gain from 4K in the living room seems marginal.
Philip Bloom made his presentation in the 4K theatre. This featured a commercial 4K projector but the screen was on a more domestic scale. It can be hard to judge in a much larger room but it looked quite similar to my own 8ft screen. All the 4K material Philip showed came from the Canon 1DC of which more later. So we are still at a domestic scale albeit a much higher end source. It didn't really change my opinion, once again it looked good, but not a huge upgrade over a good HD projector. I am sure 4K has a lot more in hand but I am still unconvinced the domestic scale quite warrants it. At the moment that is academic as there is still a gap which is the delivery mechanism. Sony have something planned to support their devices but I haven't seen any firm details. Just as I write this there are stories breaking about 4K streaming on the PS4 and talk of 100GB files. If you live in the sticks like I do then it might as well be delivered on gold bricks. RED also have plans but again details are sketchy and I wonder about what content they will have the rights to deliver. This is a notoriously difficult area to navigate, especially internationally. The data rates I have heard mentioned for RED are much lower, almost in the too good to be true bracket. They have some clever guys there but I have seen instances in the HD realm where data rates have not expanded to match resolution multiples to the detriment of overall image quality. There is a lot of hard maths involved in balancing data rate, resolution, frame rate, colour and compression. Any compromise here would undermine the 4K advantage and put me off entirely. I don't want to seem negative, I just have high expectations. I have seen films at the cinema recently like Skyfall and Lincoln projected in 4K that have looked sensational. Skyfall was not even acquired in 4K but it did not seem to impact the end result to my eyes.
Acquisition is perhaps the area we know most about. Available on RED from its inception and increasingly appearing in camera systems from Sony, Canon and others. The Canon 1DC is more or less the entry level option available now. There is this strange dichotomy where it is "cheap" for a 4K camera but very expensive for a DSLR. It is twice the cost of its very close relative, the 1DX. One advantage it has is that 4K recording is on board - something its more expensive relative the C500 can't manage. The downside is that the codec used is more limited. Compression is higher than alternative products particularly when it comes to colour space. So is it worth the money over the 1DX?
Philip Bloom at BVE London
One person who thinks so is Philip Bloom. Philip explained that he had all the reservations I have shared before using the camera. He had already been converted by the performance of his 1DX, used very effectively here in this Olly Knights music video. He still regarded himself as a 4K skeptic. However, after trying a 1DC he found himself trading up. The primary reason was the detail in the image. Comparing images from a still frame (say, from a time-lapse sequence) with a frame from a video sequence can be depressing. So much detail is lost in the moving frame. This is just not the case with 1DC. Philip equates the moving frame quality from the 1DC in 4K with a medium jpeg. This was very clear in the image blow-ups that Philip shared with us. This was the killer feature that convinced him, bolstered by the advantages of being able to reframe for HD or high quality masters. It's not the perfect camera, it still has most of the DSLR foibles and the codec is limited in speed and gradability. However, Philip feels that its improvement in image quality combined with traditional DSLR strengths of size, low light capabilty and stills ability is worth the investment. It was hard to disagree looking at a new music video project shot on the 1DC. Though I must admit that I soon forgot to pixel peep and just enjoyed the images and the song - just how it should be.
4K interface unit and recorder for FS700
For me personally the 1DC doesn't really fit my needs. Some time in July (allegedly), Sony will release the 4K upgrades for my FS700. Like with Phil's move from the X to the C there will likely be a substantial investment required. An investment difficult to justify on the basis of need (stop laughing - I know thats never stopped me before). It may also be sometime before I can even enjoy the full benefits of the improved acquisition. Even discounting my early adopter nature, I still have a nagging doubt that I may regret not adopting 4K in the long term. This coming from a man contemplating ordering a skip to dump my VHS tapes and driven to distraction by accidentally watching the DVD of Skyfall before he discovered where they had hidden the Blu-ray disk in the box.
"Worthy but Dull" time on Backup and Archive
I seemed to have been hearing or reading a lot about media data storage in the blogs, podcasts and courses I have been doing in recent times. I have written before about my semi-paranoid approach to data on the road. So this time I am going to concentrate on what happens when you get that all home. I am not a leading authority on any of these technologies, I just have been looking for a decent solution for a long time and I have some experience which translates from the workplace.
So lets get things in perspective first. I am an enthusiast and my work has sentimental but no financial aspect bar the hardware it sits on. This is a domestic environment and I only have ADSL access to the internet (5Mbps down 500Kbps up on a good day). I have a Mac Pro which has 2 drives, one for Media and Apps and the other for data including my editing projects, music and photographs. These are both 2Tb drives. I have another 2Tb on a Firewire attached G-Tech RAID drive (RAID 0). In addition to this I have a Windows Home Server (its a long story) with about a net 3Tb. Lastly, I have a 2Tb Time Capsule and this is the only device not situated in my office.
I have also have numerous external drives ranging from bus powered firewire drives mainly for travel to bare drives and a USB “Toaster”. I have both DVD and Blu-Ray optical drives capable of writing and I have tried some off-line options. I have Dropbox and the soon-to-die MobileMe. I have tried Carbonite and CrashPlan offline services.
So I am no media facility but my levels of storage are probably into the top end of prosumer levels. My set-up is no paragon and I am constantly in a state of angst about what to do about it.
The basis to my approach is the same process I would go through as an IT Professional for more commercial systems. I have included a little mindmap that I used to bring things together that you might find handy. You have to look at risk and cost. These are balancing factors and pretty much sit in opposition. Reducing risk costs money and that cost is not linear. The closer you try and get the risk to zero, the more the cost will spiral. I also have a table of performance and costs. This was compiled for my own use and should be taken as general guidance only. Costs may vary over time and where you live and these are in good old pounds sterling. The performance figures were based on multiple online reviews and I tried to stick to benchmarks which reflected the larger file sizes typical for AV users.
First of all we have to distinguish between backup and archive as these terms are often used interchangeably but they do have a separate purpose. Backup is the one that is directly related to risk. Your backup strategy is there to reduce risk. It is primarily concerned with work that is in progress or highly likely to get accessed. In a commercial environment this is often tied into a Disaster Recovery Plan or a Business Continuity Plan. These differ mainly in scale. Business Continuity looks at how can I get working again as soon as possible or keep working without your full resources for a short period. DRP is more about recovering from vary large scale interruptions or total loss. The business terms may sound over the top for a prosumer but the principle of risk analysis are just as valid outside big business.
You need to decide:
- How much work you are prepared to lose
- How long you are prepared to wait to get it back
- How much time you would be prepared to put into reloading things like software
Archiving is different in that it is an operational decision. It is about treating material which you expect to use infrequently differently to that which is more likely to be needed. Even in my simple set-up I have some storage that is speedier or more expensive or more difficult to extend than others. I really don’t want to waste that by filling it full of stuff I do not expect to access. I also don’t want to have to such through all that static data looking for what I am currently working on. This is not stuff I will never look at. The proper place for that is the trash can. Original source files are a good example. Its best practice to keep them but access will be rare. You don’t need them handy but you do need to be able to get to them if you need to.
Once you have placed data in an Archive this does not mean that you don’t also need to think about it in your Backup Strategy. However, the approach you take will be probably be different to that in your working data.
Risks
Storage Failure
This is the most common form of data loss. Hard disks are mechanical devices working on very fine tolerances. They are vulnerable to shock and environmental issues as well as wear and tear. Flash memory devices are more durable but have their own eccentricities which can affect longevity. Optical disks are vulnerable to poor handling and decay. Tape based media have a good reputation for durability but they are still a magnetic medium and they can wear or snag whilst being read.
The mitigation for storage failure is duplication. If your data is on two drives the the mean time between failure (MTBF) is doubled. That good but the real key is that the chances that both fail at exactly the same time become tiny. You can duplicate using backup software to a different device, use folder synchronisation or Time Machine on Apple. The most pure form of duplication is mirroring (also known as RAID 1). This is either software or hardware controlled exact and real-time duplication of two drives. I will come back to this in a bit more detail. Mirroring is built into OSX at the software level and many PC’s have it at the BIOS level as a feature of their motherboard.
Data Corruption
This is not the failure of the data being read on the drive, this is an application error which writes bad data on to your drive. Final Cut Pro 7 and Premiere Pro CS5 have both given me this issue in the past. Both have an auto-save feature but it doesn’t always save you.
The mitigation for data corruption is versioning. This is what auto-save is trying to do and is now built-in to OSX Lion. It is also inherent in Time Machine and most backup software options. However, unlike for storage failure, mirroring is not a mitigation, it will have two identically corrupt files.
Theft and Disaster
It seems an inevitable feature of the nature of the criminals that they are not going to leave that shiny drive array sat next to your Mac Pro when they are ransacking your place out of the goodness of their hearts. Given time they are going to take anything plugged into your surge protectors and then they are going to nick those too! This is the same with fire, flood
and other acts of God. A wireless device like a Time Capsule may escape notice but you cant put it anywhere where the wireless signal will be poor. A proper fire safe is probably beyond the reach of the prosumer. The only option therefore is to get the data somewhere else. This can be a physical or a virtual (cloud) location.
Virus & Security
Here the threat is to the data itself not the hardware it sits on. Malicious code can brick your machine or force you to quarantine it from your own and public networks. Hackers may seek to steal your data from your primary location or you backups. If you have material of high sensitivity this may limit your backup options. The mitigation here is to prevent the illegal access ever happening. Once that firewall is breached then the mitigations are similar to Data Corruption
Running Out of Space
Is this really a risk? Probably not literally but it tends to cause all kind of other issues. The first problem is that things tend to break when the space runs out: files get corrupted, background processes fail to run etc. This may be something that is not immediately notice. Secondly things tend to slow to a crawl as the system struggles to find a good place to put things. There is also a tendency for you to start putting things where you know they shouldn’t be because there is no room in the right location - or is that just me. One thing my long years in the IT game have taught me is that what seems a ludicrous amount of storage today will seem merely adequate in just a few years.
Devices
Internal Options
This is just about feasible on a Mac Pro but not the other Macs and MacBooks due to limited access and expansion capability. Internally attached drives tend to be quicker than external single drives. Anything even vaguely modern will be using SATA drives and this interface has been evolving through 1.5Gbps, 3Gbps and now 6Gbps. However, you still only have 4 drive slots and your eggs very much in the one basket. I would recommend concentrating more on performance on your PC and doing the backups elsewhere.
External Hard Drives
Absolutely massive choice of devices here so we need to split them into smaller categories. They come in a variety of interfaces and may be pre-formatted for OSX or Windows although this is usually changeable. They also can come with a variety of interfaces in order of speed USB2, FW400,FW800, USB3, eSATA, MiniSAS & Thunderbolt.
Bus powered
These are normally 2.5’ SATA drives in a protective case. They are very portable and are typically in the 250MB to 1TB range. USB2 is by far the most common but you also see USB3 & Firewire at a premium.
Western Digital My Passport Studio for Mac
Desktop
These are normally a 3.5” SATA drive in a housing. The need plugging in somewhere and come in the 500MB to 3TB range. Pretty much every interface type is available but faster speeds and multi-interface is at a premium.
Seagate FreeAgent GoFlex
Toaster
This is a drive interface and a SATA connection with a disk loading slot. Effectively the skeleton of Desktop External drive. Most fit either size of SATA drive but for backup larger drives are the obvious choice. It allows you to work with drives like you might have with tapes but remember tapes were born to be handled. Bare drives are naked and vulnerable.
NewerTech Voyager
Flash Drives
These are stick like drives which are solid state. Small and very portable they are limited to about 32GB currently. USB2 is again by far the most common but a few USB3 sticks are beginning to turn up.
This is the entry level of backup solutions. Bus powered and Flash Drives have limited capacity and are best seen as travel options and as good partners for Laptops where the primary storage itself is of limited capacity and probably slow. For the same money a Desktop drive will be faster, bigger and with a better chance of a high speed interface. This goes for Toasters too which offer access to excellent cost per MB figures providing you are a gentle disposition
Network Drives
There are a few single drive devices which are not direct attached. In the Mac world the most obvious one is Apple’s Time Capsule. This is actually a network router with a hard disk in one Mac Mini sized box. Its a single drive of up to 3Tb capacity. This is designed to integrate with the Time Machine service in OSX. Most other similar drives are now designed with multimedia delivery in mind although they might be usable in this role with software or as a Time Machine Target. Some modern routers have a USB port which allows a normal external drive to be attached. This helps with sharing and with placement options but performance will take a bit hit compared to attaching that same drive direct.
Apple Time Capsule
Drive Arrays
I am calling anything with more than one drive an array but the breadth of devices that covers is pretty massive in terms of capacity, cost and implementation even if you rule out the massive SAS based storage arrays
If we look at them in terms of architecture first.
Direct Attached
The simplest kind is a direct attached device. In this format the drives attach just like a single drive external drive. Intelligence in the device is limited. Normally it is just adding a RAID controller card and some drive bays to an external drive. Prosumer versions tend to look like stretched versions of a makers external drive and have just 2 drives. Usually only RAID 1 (mirroring) and RAID 0 (striping) are offered. RAID 0 gives you the full capacity of the dual drives and improved performance over a single drive but doubles the risk. As such it is a poor choice for backup. RAID 1 halves the risk but also the capacity so a maximum of 3TB. If you can fit your backups in this size these drives in RAID 1 are safe pair of hands and good value but inflexible. Slightly more expensive are versions which have removable drive caddies. This opens up the world of drive rotation and they are a lot easier to get back online if you do have a drive go down inside. Some of the manufacturers tie you into proprietary drives though which adds to the potential expense.
Once you get beyond 2 drives new options become available. Depending on just how many drives you have you can start to play RAID bingo. Their are bewildering numbers of variants of RAID available each with its own subtle characteristics in terms of safety, performance in capacity. Few devices support every possible option so its best to do your research on RAID levels, find the one which best matches your needs, then look for devices that support that RAID level. To give an illustration I am going to look at the most commonly implemented on which gives good capacity and middling safety and performance - RAID 5. In a RAID 5 array you lose one drives worth of capacity from the array and you need at least 3 drives. So a 4 x 1TB RAID 5 array gives around 3TB of space. The more drives you have the better the ratio of what you get to use. For larger installations getting past the maximum physical drive size is essential. Remember, backups may contain multiple versions of a file so you may need much more space for backup than you are using for your primary data.
Most multiple drive arrays have a choice of interfaces. Its important to realise that this may be the deciding factor when it comes to performance. It doesn’t matter how fast your drives are or which RAID level you picked, the data rate can’t exceed the bandwidth of the interface you use.
One thing to be aware of is that not all enclosures designed to hold arrays have the interface controllers built-in. If you see an array described as a JBOD (Just a bunch of disks) then they will require a controller card (typically eSATA or MiniSAS) to be added to your PC. For Mac users this means you will need a Mac Pro or a MacBook Pro with a PCI Express slot. For any other Mac you will need one with the controller built in.
One last point about arrays - they can be touchy blighters! All arrays have some hard maths going on in the background to make sure they OK. If anything goes wrong then they have to either rebuild or go through diagnostics. Now this is way better than losing a whole load of data but it will at the very least reduce performance while it happens or even take the device off-line. This can take hours or even days. I had a Windows NT machine with a bios based RAID controller. One of the things that would upset it was the PC crashing or needing a forced reboot. I said this was a Windows NT PC right?
G-Tech G-Speed Q
Promise Pegasus R6 (first Thunderbolt Array available)
Storage Robots
This term comes from Drobo and really the only thing that comes close in principle is my old Windows Home Server. The main difference to the other device types is that Drobo and Microsoft did not go down the RAID route to achieve redundancy. RAID is very fussy about disks. Ideally RAID likes matched sets of the same type and capacity. It will tolerate a smaller capacity drive but at a big cost in overall capacity as everything works of multiple of the smallest drive leaving capacity unusable. Drobo & WHS will take anything you have handy and present it as a single drive but still retaining fault tolerance should a drive fail. Using all your bays and need more space? Take out a small drive and add a bigger one - the device does the rest. This is impossible with a RAID array without changing all the drives and massive amounts of rebuilding and no small risk. With RAID go as big as you can justify from the outset. This approach focusses on easy of use and flexibility rather than performance.
Microsoft seems to have gone cold on WHS and the latest version has lost this feature so Drobo is now unique. Its worth noting Drobo now offers NAS and SAN options as well as direct.
Drobo S
Network Attached Storage (NAS)
A NAS is really a small server optimised for sharing data. The RAID arrays are the same in principle as a direct RAID device but a NAS is seen as a completely separate device on your network where as a direct attached device appears as a local disk on your PC. The benefit is that multiple computers can use the device as a backup target but network speeds rarely match those of direct interfaces in the real world as that bandwidth is shared with other traffic. These are servers, so while they tend to look superficially the same, performance can vary widely due to CPU, memory, network and disk controllers. There is also the quality of the interface (normally browser-based) to consider. Strong competition in this sector keeps prices down so they offer good value if your priority is capacity and consolidation not performance.
Promise NS4600
Storage Area Network (SAN)
This is a bit of a specialised device. Its really designed for big server rooms where multiple diskless servers are connected by a high speed fibre interface to a big box stuffed full of disks. Each server is allocated a proportion of the disk and thinks it is local. When you have a lot of disks the high cost of the interface hardware is offset by not having to provide a high level of redundancy and case room for each individual server. There is a lower cost option called iSCSI that uses a network connection instead of the fibre. But its still more a way of providing primary storage rather than a backup tool.
Optical Storage
We know Steve Jobs would like to kill it off but most computers still have some kind of Optical writing capability. Most likely this is good old DVD. Compared to magnetic and flash media DVDs are as cheap as chips. They are also pretty easy to store and should last a while if you don’t go too cheap and handle with care. But when a single memory card from your camera might have 32GB on it then 4.7GB is started to look a bit titchy. Dual Layer disks help but are 10 times more expensive. Blu-ray ups the capacity again to 25 or 50GB but again the cost rises dramatically and writers have never gained the kind of popularity which will drive the prices down to DVD levels and its doubtful they ever will. Capacity is not the only issue - optical media is very slow to write and not very good for iterative saves.
Cloud Storage
This is a an internet service provision where your data is stored in a data centre operated by a third party. This storage will be enterprise class, high redundancy and high availability. All you need to do is set-up an account and run a client utility on your PC. Most companies will offer different subscription models to suit users who have different size requirements or multiple devices. They are split into two main types of service, synchronisation and backup.
Dropbox is the best known of the synchronisers. Rather than back up your whole disk you place data in a specified virtual folder structure. The Dropbox client then replicates this in the cloud. Any device you have Dropbox on, including mobile devices like smart phones, can access the data or you can go through a web client. You can even elect to share folders with other people you nominate. This is really designed for people who want to collaborate with others or tend to use multiple devices. Its not really meant as a backup solution. Its great for ad-hoc backups of relatively small amounts of data though.
The second type of server is aimed at the backup market. Providers include people like Carbonite, Backblaze and CrashPlan. My own experience of on-line backup as not been great but if I wanted a better one I would have to move. The upstream speed on my ADSL is a paltry 400Kbps maximum. If its at maximum it really seems to interfere with the downstream rate too. With Carbonite in its low priority mode I only managed about 30Gb of backup over a month. So you have to make sure that you link speed is going to make online viable for the size of data you need to backup. Some vendors do offer a hard disk data take-on option. This means that you only have to worry then about new data or changes being backed up. They may also offer a hard disk restore option if the shipping time would be quicker than restoring over the net.
Arguably the biggest issue with a Cloud service is that you lose a degree of control over your data. All the companies publish their policies on security and encryption but once it is in the Cloud you can never be truly be sure where it is or who can access it. Unless you are a specialist it should be the case that they are better at security than you are but they are also a more obvious target. Hackers or disgruntled staff are not the only risk - what if the company goes out of business? If this is just a backup of your holiday movies from Majorca than its probably not an issue. If it is confidential work for clients then you need to take this into account.
Other People Disks
This is a service that CrashPlan introduced me to. Their client allows backups to other devices running CrashPlan. This can be on a private network or via the internet. The expectation is that this will be a “buddy” system where you agree to provide a reciprocal arrangement with someone you trust. The performance limitation will be primarily the link you choose. A private network should be much faster than an internet link. A hard disk starter backup will probably be advisable. It’s unlikely that your mate is going to have the same level of kit as a Cloud based service but at least you “know where they live”.
Tape
It may be old-school but tape is still a very effective option for both archiving and backup. The standard that is best supported for big data volumes is called LTO. If you do a bit of research you will see numbers after that and these are significant. It refers to the generation of the standard. Each generation has grown in tape capacity and data rate. Backwards and forwards compatibility are enshrined in the standard as are the expected life and re-uses a tape should manage. This is an enterprise class solution which means heavy-duty, fast, reliable and expensive. Most of that cost though is up front, the tapes themselves are less expensive than other mediums. You are going to need to have a lot of data though before you break even with disk based alternatives.
Quantum LTO5 Drives
Archiving
Archiving is “cut and paste” to Backup’s “Copy and Paste”. If the data you are archiving still exists in its original place when you started then you didn’t really archive it, you just backed it up.
Why archive?
So if we haven’t improved our data security by archiving it why even bother? Well there are several reasons why it is beneficial to do so.
Freeing up scarce resources
Perhaps you edit on a laptop in the field and to maximise your performance you have a fast but small SSD drive. You would soon run out of space on that SSD if you leave old projects on there. Some NLE’s have media manager functions that can reduce a project to just the assets used when they consolidate and move a project.
Move to safer environment
Resources like laptops, portable drives, fast RAID 0 arrays have inherently more risk attached to them. Moving data to a less risk environment is advisable.
Confidentiality
If you are working with multiple clients you may want to avoid one client seeing any assets or folder names which may reveal the identity of another client.
Allowing a different backup regime
An archive isn't a copy. So archives need backing up like everything else. However, they are going to have a different risk profile and data change rate to projects in progress. It makes no sense to tie up resources backing up the same file every night when there is only a tiny chance it will change. If you do get to a situation where an old project needs substantial rework then it should be moved out of the archive back into the work-in-progress storage.
Cataloguing
You have to assume that by the time you need something in the archive that you will have forgotten all about it. At my age, this is almost certainly true. So how do you find stuff when people ask? Perhaps you are just interested in finding something to use as stock footage in another project and just want to look on spec.
Folder and naming conventions
This is the place to start. A degree of of orderliness is essential if you want to be able to find things. How far you have to go with this probably depends on your working environment. There are defined practices for movie and broadcast that you can use but these could be overkill and bring in a new level of jargon. Some products may generate or impose the structure for you. If you are using multiple software solutions sometimes you may have some conflicts. My own general principles has to be to try and keep any software bin structures and physical folder structures in sync as much as possible. Try and be descriptive with the project folder names. If you are going to use any coding schemes then you need to be consistent and you need to make sure the key or index is accessible.
Metadata
With FCPX Apple moved to a metadata focused model from a bin based one much to the consternation of the editing community. To me this wasn't too much of a shock to adjust to. Not because of iMovie but because this is the same model that Aperture uses. What did shock me was the poor integration with Aperture from FCPX but that is a different rant. As a database guy, I appreciate that tagging is able to encode the multiple attributes that a clip possesses in a way bins can’t without a lot of duplication. However, you have to be disciplined about it. Video is a bit sparse for metadata compared to stills with their well established EXIF standard. However, I think this will change over time and more metadata will arrive with the files themselves. One of the interesting features of FCPX was autotagging based on analysis. For example, it attempts to tag clips as wide shots or close-ups or for shakiness. Adobe has some interesting technologies for voice recognition which could also find their way into metadata.
Off-line Cataloguing
If you use tapes or a tape-like workflow with other media, then you are going to need to generate a catalogue. This is because your data is sat in a cupboard somewhere and is not available for searching. So you need to capture the data structure and metadata into a stand-alone database that remains on-line. You also need a physical indexing system so that when you search the catalogue you can go straight to the right disk or tape in the cupboard.
Retrieval
Whether its an archive tape or a backup disk its utterly useless if you can’t read it. Some LTO tapes are guaranteed for 30 years. 30 years ago, in 1981, myself and three school friends won our school its first computer (a BBC Micro) in an inter-schools quiz on TV. I think I would struggle to read files a BBC Micro saved onto cassette tape these days. To be honest it was a bit hit and miss in 1981. I have never really re-used my DV tapes and they acted as an archive once I had transcoded the footage onto to disk. However, my aging DV camcorders are getting a bit flakey. When the last one dies I lose my archive or will be scouring Ebay.
To ensure retrieval, you often need to transfer the data to a newer format prior to the old format being obsolete. However, you need to be mindful of retaining as much quality as possible. In the heyday of DV and DVD, burning to a DVD may have seemed a decent archive solution. However, the transcode to MPEG2 within the DVD’s storage limit may have degraded the image. However, a data DVD backup of AVI files created in the DV codec should be a true copy of what is on the tape. But how long will that codec or file format be readable? Always be on your guard for technical obsolescence whether it be hardware or software and be planning your transfer solution.
Summary and Recommendations
So my set-up is a paragon of virtue?
Err...not so much. Part of the reason I am informed on this topic is I am perpetually in search of that ideal solution. I am more of an experimenter than a completer finisher so I am always wrestling with finding a method and not quite as good at implementing it. However, imperfect as it maybe, I do have some of the risks mitigated. Lets go back to my basic questions and I will answer them for myself:
- How much work you are prepared to lose
- Nothing, its not important to anyone else but I would hate to lose it. My inner squirrel can’t cope with that.
- How long you are prepared to wait to get it back
- No rush, as long as I know I can get it back time is not a big issue
- How much time you would be prepared to put into reloading things like software
- I have a low tolerance for repeating things I have already done. I have loads of software & plugins and the like. I am prepared to lay out some cash to avoid that pain.
So how am I doing in mitigation of risks then?
Storage Failure
My main system drive is covered by Time Machine and my original media archive is on the Windows Home Server so they are OK. However, my project drive is too big to fit on the same Time Capsule so I am having to do periodic back ups to an external drive. This is not ideal. I also have a few chunky things that have found their way onto my fast external firewire RAID 0 drive that I know shouldn’t be there but won’t fit anywhere else.
Data Corruption
Time Machine is a versioning solution so is good protection and the stuff on WHS is static so low risk. I do have 2 generations of backup on the WHS too. Again I am vulnerable on my other drive and the RAID. My adhoc backups are overwrites so give less protection.
Theft and Disaster
I now use a Toaster to backup to bare drives which I transport in inexpensive plastic boxes. I have 2 which I use alternatively and give to my parents who live 30 miles away. I should do this a bit more often than I do but its a manual process and time flies. I tried online backup but my rural ADSL is the limiting factor.
Virus and Security
I am a Mac user so the threat has traditionally been lower. However, I have my doubts this will continue given Apple’s rise in market share and No1 company status. I keep up-to-date with software updates on all devices and I have anti-virus and firewalls running on my PC’s and Mac Pro. I don’t have significant Cloud based assets to be under threat.
Archiving
I don't have sufficient data or operational driver to make this necessary yet.
So if you know better, whats holding you back?
Well there is cash for a start. Its money well spent but this is a hobby and there are so many, many things on my wish list... damn you Internet.
Secondly, I am that uncomfortable stage where 2TB drives are a bit under pressure but 3TB drives are still a hefty premium. Once you scale up your primary storage everything needs to move up with it. Also I am excited by the potential of Thunderbolt but we are still short on peripherals, including anyway to get it on my Mac Pro.
My non-media data is fine how it is. I am not struggling for space and a Time Capsule is a good solution due to its risk and change profile. With a bit of careful targeting I could probably even make the Cloud work for me whether this is through iCloud or another service we will have to see when Apple’s full plans are more evident.
My original media store is fine where it is. My little WHS might be a technical dead-end but it does the job and has some growing room still.
My media drive I am hoping to replace with a large Thunderbolt array. Thunderbolt is a fast enough interface that some of the old data organisation best practice designed to stop you hitting bandwidth bottlenecks just don’t apply. This makes data organisation less complex. I want to have 2 x 3TB drives in the array configured as RAID1 for backup. Periodically I would swap one of the drives and keep my sneakernet arrangement with my Dad. Fingers crossed they find a way of squeezing 4TB on a drive before 3Tb gets too small.
I could get the job done with something like a MiniSAS array now but I feel, given the level of investment, I should hang on a bit. Without an option to retrofit Thunderbolt to my Mac Pro, I may not get the option and have to go that way.
Some helpful links:
Thunderbolt
Thunderbolt - cheesy name but a really fascinating bit of tech. As I headed down to Converge, two pieces of news had just broken. The first was that Apple had just added Thunderbolt to it's new MacBook Pro range and the second was a rumour the Nikon D4 would get it too. Putting the two together gave me own Thunderbolt moment. I started to imagine what the still mythical Canon 5D MkIII could achieve with Thunderbolt. Now I am no more an electronics engineer than I am a cinematographer. I can build a PC from the components but that's about as far as it goes. Given that only that MacBook Pro has been announced with Thunderbolt so far, what follows is pure speculation on my part.
There are two digital ports on the MkII - USB2 and HDMI. USB has two purposes for the 5D. The first is data transfer for which the maximum speed is 480Mbps. USB rarely gets anywhere near that. When I connect to my Mac Pro with a Sandisk 480Mbps card I get 90Mbps whereas the same card in a FireWire 800 reader gets 460Mbps. The second use is as a bus for control instructions. The Canon tethering software and the Okii follow focus use this method to control the camera remotely. The second port is HDMI which is used to output to display devices. When Canon designed this they were thinking of playback to domestic tellies. HDMI is designed for domestic use and Canon only had room to squeeze the mini version of the plug on the body. HDMI is great in it's domestic environment but has issues in the camera implementation. Canon does not have the ability to give a clean, uncompressed output at 1080p while recording - the 5D is only 480p with camera indicators. The built-in LCD also becomes inoperable when HDMI is plugged in. HDMI has a lengthy handshake protocol which interrupts things when the signal changes. Finally, all HDMI connectors are hateful and mini is the worst. Not the end of the world in a fixed installation but frustrating in equipment that is constantly on the move. HDMI can shift a lot of data - HDMI1.4 tops out at around 10Gbps and can handle 4K resolution.
So where does Thunderbolt make a difference. Well the first difference is that it can do both jobs that USB and HDMI do at once. Like HDMI it has a 10Gbps capability but this is bidirectional. It's close relationship to PCIe, the internal bus used in PCs and Apples, should make it possible to produce a breakout cable that supports HDMI and USB for legacy use. It's so fast that transferring files is really going to going to be limited by the devices either end not the cable. For tethered shooting this will make a huge difference. No more jerky live view or long waits between shots or takes. Apple's implementation of Thunderbolt uses the display port connector. This is a small plug and the wire is relatively thin and flexible. Its not as secure as a locked in cable like a BNC type SDI connector but it is a better fit than mini HDMI.
Intel has a fibre optic version of Thunderbolt which is capable of being sent long distances. This could avoid the need to convert to SDI to get an image back to video village.
When the data comes off a sensor then a DSLR has to deal with it. First it has to process it, then it has to store it. With video, the Canon DIGIC4 chip has to first chuck loads of pixels away to get down to 2MP, then it has to encode into H264. That’s not an easy task, even modern PCs struggle to encode 1080p in realtime. Working that hard makes chips hot and there are not many places for that heat to go. Some of that heat makes its way into the sensor which it does not enjoy. Overheating and a rising noise floor are the things the DSLR shooter has to contend with. The ideal situation would be to have the option to get the data off the sensor and out of the camera altogether and process and record it in a separate box. When companies like RED and SI talk about “brains” this is what they mean. To shift RAW data like that you need a fast bus and it seems to me that is a role Thunderbolt could play. Whether the camera divisions that make these cameras ever want to go that far into the more esoteric end of the video market is debatable. However, at least it would mean stills photographers wouldn’t be coughing up for high-end video features the majority of them will never use. If the manufacturers decided to make the specs open then we could have 3rd parties like AJA or Codex provide the brain/recorders. Even if the processing is still done in camera, then Thunderbolt offers a more flexible way of getting data out than HDMI. Its bidirectional nature means that features like timecode, genlock and device control should be possible along with whatever video format the camera produces. So if Canon do switch to 1080p 4:2:2 to use their MPEG2 based codec from their video line, piping that out uncompressed to an external recorder is well within Thunderbolt’s bandwidth as it has more capacity than HDSDI. Thunderbolt based modularity seems to me like an elegant solution.
A fully rigged RED One looks like Borg technology - cool but complicated. With Thunderbolt, it looks like you may be just be able to plug into a hub like a USB without having to worry about the connection type. I know in reality things don’t tend to be that easy with plug and play but we can always dream. At the moment dreaming is all we can do till some devices appear and maybe the whole D4 thing was someone else's dream. One of my missions at NAB is to see how much of musings are feasible, likely or commercially viable.