4K Shared Storage — Considerations for Post, Color, VFX, and Archive

When Avatar was released in December of 2009, it was the most expensive movie ever made, in total budget and total storage costs. The VFX team needed over 1 petabyte (PB) of storage to complete the film—which out of a 162m runtime, was 60% CGI—and it wasn’t even shot in 4K. Five years later, in 2014 The Amazing Spider-Man 2 more than doubled that and generated roughly 2.4PB during the process of scanning 1.5 million feet of 35mm film to 4K.

Today, it’s relatively common for a feature production to surpass the 1PB mark, and even boutique production companies and corporate video teams are buying hundreds of terabytes worth of media storage. When almost all of the popular DSLR cameras and even cell phones are capable of shooting 4K, it’s no surprise that we see it in video workflows outside of film and TV. Even if you’re not working on a blockbuster with a $300m budget, the growing prevalence of 4K, HDR, HFR, and other in-demand formats is being felt everywhere, especially on storage. That’s why it’s more important than ever to consider your workflow and infrastructure carefully so that you can get the most out of your 4K shared storage investment.

Choosing Your Flavor

In any modern post-production workflow, shared storage is paramount. Having a centralized storage system enables teams to share and collaborate more effectively, create content faster, stay organized, and really utilize their media. With the high data rates common in 4K production, storage may be the most critical part of a 4K workflow, especially if you’re storing the raw camera masters and editing codecs. 4K media comes in a lot of different shapes and sizes, and depending on your delivery requirements, that can mean working with files that have bandwidth requirements ranging from 100 megabits per second to well over 1 gigabyte per second! With 4K media, you not only have to consider the demands on your network, but also the load on workstations. These variables can make it challenging to figure out how to set up your 4K post-production workflow.

For many people, the goal is to capture the best possible image, with the highest quality files, which means files that retain the most information and give the most flexibility in post-production. That means less compression, higher resolutions, and more bit-depth. In a perfect world, that means we’re always recording uncompressed 4K—or even better—8K. But some trade-offs need to be made, for example, if your project requires extensive VFX or graphics compositing, you’ll probably want to use an uncompressed image sequence, or high bit-depth codec that has no chroma subsampling, and an alpha channel. But you’ll need a powerful workstation and high-performance video editing shared storage to support it. In this case, you’ll probably also want to create more compressed (intermediary) files for editorial.

If you aren’t working on a VFX-heavy project, your best bet for a native 4K workflow might be to work only with a compressed codec like ProRes 422 (which has a data rate of ~500Mb/s) or even ProRes 422 Proxy, which only requires 155Mb per second.

Network Speeds vs. Throughput

The most common network connection is 1Gb Ethernet. Theoretically, 1Gb Ethernet can provide a maximum of 125MB/s to a single workstation. But that doesn’t mean you can just connect to a NAS and start editing 6 or 7 streams of 4K ProRes 422 Proxy. Instead, the read/write performance of the server, the networking stack, congestion, and many other factors are just as crucial to your final throughput. With the right shared storage server, you can edit multiple streams of compressed 4K over a standard Ethernet network—but for higher throughput requirements, like with multicam 4K ProRes RAW or ProRes 4444, you’ll need 10Gb Ethernet to do the trick. For image sequences, uncompressed media, or high frame rate and high bit depth formats, you can configure network storage with 25Gb, 50Gb, and even 100Gb Ethernet. But using higher bandwidth connections doesn’t always achieve higher throughput—there are frequently other bottlenecks in the chain that must first be overcome.

On average, one hour of HD footage (ProRes 422) requires approximately 51GB of storage; or roughly 850MB per minute. 4K, on the other hand, requires close to 226GB of storage—almost 4GB per minute. That’s a pretty significant jump, but the good news is, storage is becoming more affordable every year, and HDD manufacturers continue to release larger capacity drives. In many cases, it’s actually not necessary to keep all of your media on a single storage system. A smart approach to managing the large amount of data generated in a 4K workflow is to tier your storage so that you’re only keeping the most active media files on your high-performance storage, and your camera masters or older projects located on or nearline or cloud and LTO storage. A tiered storage approach is especially useful if you’re dealing with several stages of post-production that require a variety of file formats.

4K Editing

There are a lot of options for 4K editorial over shared storage. If you’re doing multicam or have several editors working with 4K media for a variety of projects, you can choose a highly compressed codec like ProRes LT that can easily stream over 1Gb or 10Gb Ethernet. If you have fewer editors and prefer to work with higher quality images, you can choose something like ProRes RAW—which requires more storage and more bandwidth—but won’t break your budget, even though it retains much better image quality. However, it’s important to keep in mind that in general, the more compressed the codec is, the more demanding it is on your workstation CPU and GPU.

4K Compositing/VFX

Compositing is the final step in the VFX pipeline, where a compositor arranges all of the VFX elements with the highest quality camera footage available to create a seamless picture. Compositing is, without a doubt, one of the most technologically-demanding parts of the post-production workflow. Why? Because completing a photo-realistic VFX shot relies on having as much visual information as possible, and that means the files are either A) completely uncompressed or B) as close as they can get. Most compositing (and color grading) workflows use image sequences instead of video files; in an image sequence, each frame is a single file that retains a ton of color information or can support multiple layers. One of the biggest hurdles in 4K compositing workflows is that they can require more than 1.2GB/s per second to playback a single sequence at 24p! Thankfully, with the development of flash-based storage arrays that use SSD instead of HDD disks, and with the advancement of 25Gb, 50Gb, and 100Gb Ethernet connectivity, these workflows are now more cost effective with shared storage. But because 4K DPX, OpenEXR, TIFF, and other types of image sequences are best handled by ultra-fast SSD disks, you’re limited by the lower capacities available with SSDs.

4K Color/Finishing

Like compositing, color grading requires working with the highest quality images available. Where compositing benefits from having multiple layers with alpha channels, color grading benefits most from retaining the maximum amount of color information (bit-depth) so that the colorist has the freedom to fine-tune every frame with pixel perfect precision. In some cases, a video codec like ProRes 4444 XQ can be used, because it still retains near lossless quality and color information, but comes with significantly lower data rate and storage requirements. For example, a one hour 16-bit 4K OpenEXR sequence (24p) would require roughly 1.2GB/s of bandwidth, and 4.3TB of storage capacity. In comparison, one hour of ProRes 4444 XQ requires approximately 234MB/s of bandwidth and about 845GB of storage capacity, a more achievable feat for HDD-based shared storage and standard 10Gb Ethernet networks.

4K Archival

Since most 4K post production workflows rely on creating compressed intermediary codecs or working with massive image sequences, it’s important that you’re smart about how (and where) you store your media. A good strategy is to keep your files on multiple tiers of storage. A good rule of thumb is to keep your most frequently accessed files (used in editorial, VFX, or color) on a separate high-speed shared storage server built for video workflows, and move your larger files (like camera masters) to archive storage or nearline storage.

Finding a shared storage server for your 4K workflow doesn’t need to be complicated. We’ve helped thousands of teams improve their post-production pipeline with 4K shared storage.

If your team is ready to get started, click here to schedule a demo with one of our workflow specialists >